Attractor Memory for Long-Term Time Series Forecasting: A Chaos Perspective
Authors: Jiaxi Hu
Yuehong HU
Wei Chen
Ming Jin
Shirui Pan
Qingsong Wen
Yuxuan Liang
Venue: NeurIPS 2024 poster (2024)
Keywords: State Space Model; Time Series; Chaos Theory
Primary area: machine_learning_for_other_sciences_and_fields

Abstract: In long-term time series forecasting (LTSF) tasks, an increasing number of works have acknowledged that discrete time series originate from continuous dynamic systems and have attempted to model their underlying dynamics. Recognizing the chaotic nature of real-world data, our model, Attraos, incorporates chaos theory into LTSF, perceiving real-world time series as low-dimensional observations from unknown high-dimensional chaotic dynamical systems. Under the concept of attractor invariance, Attraos utilizes non-parametric Phase Space Reconstruction embedding along with a novel multi-resolution dynamic memory unit to memorize historical dynamical structures, and evolves by a frequency-enhanced local evolution strategy. Detailed theoretical analysis and abundant empirical evidence consistently show that Attraos outperforms various LTSF methods on mainstream LTSF datasets and chaotic datasets with only one-twelfth of the parameters compared to PatchTST.

Attractor Memory for Long-Term Time Series Forecasting: A Chaos Perspective
Jiaxi Hu', Yuehong Hu', Wei Chen', Ming Jin', Shirui Pan', Qingsong Wen', Yuxuan Liang'
'The Hong Kong University of Science and Technology (Guangzhou)
Griffith University 'Squirrel Ai Learning, USA
[jhul 10, yhu322, wchenl 10) (mingjinedu, gingsongedu] @ @egmail.com s.pan @ wwiffith.eeu.aa. edu.au
Abstract
In long-term time series forecasting (LTSF) tasks, an increasing number of works have acknowledged that discrete time series originate from continuous dynamic systems and have attempted to model their underlying dynamics. Rec- ognizing the chaotic nature of real-world data, our model, Attraos, incorporates chaos theory into LTSF, perceiving real-world time series as low-dimensional observations from unknown high-dimensional chaotic dynamical systems. Un- der the concept of attractor invariance, Attraos utilizes non-parametric Phase Space Reconstruction embedding along with novel multi-resolution dynamic memory unit to memorize historical dynamical structures, and evolves by a requenccy-enhance local evolution strategy. Detailed theoretical analysis and abundant empirical evidence consistently show that Attraos outperforms var- ious LTSF methods mainstream LTSF datasets and chaotic datasets with only one-twelfth of the parameters compared to PatchTST. Code is available at htttps://githie http///tttuu.cco/iiiiitee Coom/CityMinng Law.iidddda///aii LabdddLrrr/rerrrr Attraos..
1 Introduction
In the intricate dance of time, time series unfold. Emerged from continuous dynamical systems [36, 11,39) 39] in the physical world, these series are meticulously collected at specific sampling frequencies. Like musical notes composition, they harmonize, revealing patterns that resonate through the symphony of temporal evolution. In this realm, Long-term Time Series Forecasting (LTSF) stands as one of the enduring focal points within the machine learning community, achieving widespread recognition in real-world applications, such as weather forecasting, financial risk assessment, and traffic prediction [28, 22, 30, 27, 18].
Building on the success of various deep LTSF models [49, 54, 52, 50, 29, 22], which primarily leverage neural networks learn temporal dependencies from discretely sampled data. Currently, researchers [46, 32] have been investigating the application of Koopman theory [48, 25] in recovering continuous dynamical systems, which applies linear evolution operators to analyze dynamical system characteristics in a sufficiently high-ddmeensioal Koopman function space. Nevertheless, the existence of Koopman space relies on the deterministic system, posing challenges given the chaotic nature of real-world time series data, evidenced through the Maximal Lyapunov Exponent in Appendix E.2.
In this paper, inspired by chaos theory [8], revisit LSTF tasks from chaos perspective: Linear or complex nonlinear dynamical systems exhibit stable patterns in their trajectories after sufficient evolution, known as attractors. As illustrated in Figure ((a), attractors be classified into four
"Y. Liang is the corresponding author. Email: yuxliang com
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) Different Types of Attractors
(b) Classical Chaotic System (Rössler, Thomas, Aizawa, Lorenz)
Fixed Point
Limited Cycle
(c) Real World Series (ECL, ETTml, Weather, Traffie)
Limited Torus
Strange
Figure I: (a): Classical chaotic systems with noise. (b): dynamical system structure real-world datasets. (c):
Different types of Attractors. See more figures in Appendix E.I.
types: Fixed Point, indicating stable, invariant systems; Limited cycle, representing periodic behavior; Limited Toroidal, exhibiting quasi-periodic behavior with non-intersecting rings a 2D plane, reflecting temporal distribution shifts; and Strange Attractor, characterized by nonlinear behavior and complex, con-topological shapes. Supported by chaos theory, we can transcend the limitations of deterministic dynamical systems to construct generalized dynamical system models. Figure ((b-c) showcases various classical chaotic dynamical systems and the dynamical trajectories of real-world LTSF datasets using the phase space reconstruction method [9]. Notably, the dynamical system trajectories in these LTSF datasets exhibit fixed structures akin to those in typical chaotic systems.
Given this chaos perspective, we consider real-world time series as stemming from an unidentified high- nigh-dimensional underlying chaotic system, broadly encompassing nonlinear behaviors beyond periodicity. Our focus centers recovering continuous chaotic dynamical systems from discretely sampled data for LTTF tasks, with the goal of predicting future time steps through the lens of attractor invariance. Specifically, this problem can be decomposed into two key questions: (i) how to model the underlying continuous chaotic dynamical system based on discretely sampled time series data; (ii) how to enhance forecasting performance by utilizing the attractors within the system.
In this context, Attraos emerges with the goal of capturing the underlying order within the seeming chaos via attractors. For tackling the first question, we employ struction method to recover the temporal dynamics and propose a Multi-resolution Dynamic Memory Unit (MDMU) to memorize the structural dynamics within historical sampled data. Specifically, as polynomials have been proven be universal approximators for dynamical systems [4], MDMU ex- pands upon the work of the State Space Model (SSM) [12, 10, 18] to different orthogonal polynomial subspaces. This allows for memorizing diverse dynamical structures that encompass various attractor patterns, while theoretically minimizing the boundary of attractor evolution error.
parameeeet phase space recon-
To address the second question, we devise frenuency-enhanceed local evolution strategy, which is built upon the recognition that attractor differences are amplified in the frequency domain, as observed in the field of neuroscience [5, 14, 6]. Concretely, for dynamical system components that belong to the same attractor, we apply a consistent evolution operator to derive their future states in the frequency domain. Our contributions can be summarized as follows:
A Chaos Lens on LTSF. We incorporate chaos theory into LTSF tasks by leveraging the concept of attractor invariance, leading principal way to model the underlying continuous dynamics
Efficient Dynamic Modeling. Our model Attraos employs non-parametric embedding to obtain high- nigh-dimensional dynamical representations, leverages MDMU to capture the multi-scale dynamical structure, and performs the evolution in the frequency domain. Remarkably, Attraos achieves this with only about one-twelfth the parameter count of PStchTST, Furthermore, we utilize the Blelloch algorithm [3] enable efficient computation of the MDMU.
Empirical Evidence. Various experiments validate the superior performance of Attraos. Besides Leveraging the properties of chaotic dynamical systems, we explore their extended applications in LTSF tasks, focusing chaotic evolution, modulation, representation, and reconstruction.

---

2 Preliminary
Attractor in Chaos Theory. In chaos theory, the interaction of three more variables exhibiting periodic behavior gives rise to complex dynamical system characterized by chaos. According to Takens's theorem [43, (43, assuming ideal dynamical system F .M- MM that "lives" on attractor A in manifold space M which locally C.V (N-times differentiable), time series data { E IR can be interpreted the observation of the unknown ideal dynamical system, we can employ the phase space reconstruction (PSR) method to establish an approximation K.: I + I. which lives in differential homomorphism attractor A in the Euclidean space with suitable dimension m [9]. The whole process illustrated Equation (I), where ) } are the sampled data from two dynamical systems. Strietly speaking, in our paper, the chaotic attractor structure Á = {A:,] we focused is in phase space IR" To facilitate understanding, we further provide a visual example of the Lorenz96 system in Appendix E.3.
by an unknown observation function h. To explore the properties
In the forecasting stage, the local prediction method emerges prominent one: u (()),, where the local evolution KC(') be either linear nonlinear neural network [45, 2, 40], with the parameter being shared among the points in the neighborhood of u, or belong to the same local attractor. Considering the universal approximation capabilities of polynomials for dynamical systems, we leverage the polynomial to describe the chaotic dynamical structures.
a. E A M +
a+t1 EACM M 4b M++ E R Prsn
<')) = Ax(t) + Bu(t)
(2a)
2 E R PrRR
(1)
xxt) (K * ()(
(2b)
K(t) = CAB
(2c)
M RD
M++1 E Ā R
Polynomial Projection with Measure Window. We only consider the first part the SSM (2a), which is parameterized map that transforms the input u(t) into an N -d-imensional latent space. According to Hippo [11], is mathematically equivalent to: given an input u(s), a set orthogonal polynomial basis 6n, (t, s) that J" 6m (t, (t,
s) (t, s)ds = 6m.n, and an inner product probability project the input u(s) onto the polynomial basis along time
measure s). This enables dimension (3), and we can combine on (t, s)w(t, s) as kernel (t, s) (4). When w(t, s) is defined in time window [[t,, & + 0j, it represents approximating the input over each window 0.
(u, On) = L u( u()))n (1, 8) s)ds
Ia (x)(t) = us)) $) s)(, s)ds
(4)
When the basis and measure are solely dependent time f, can be expressed in convolution form (2b). In this paper, we will utilize this property to project the dynamical trajectories {u,} } in phase space onto the polynomial spectral domain with kernel e B (2c) for characterization.
3 Theoretical Analysis & Methods
The overall structure of Attraos is illustrated in Figure In this section, we provide a comprehensive description of its components, including the Phase Space Reconstruction embedding, the Multi- Resolution Dynamic Memory Unit (MDMU), and the frequency-enhanceed local evolution, as well as the efficient computational methods employed.
3.1 Phase Space Reconstruction
According to chaos theory, the initial step involves constructing topologically equivalent dynamical structure through the PSR. The preferred embedding method is typically the Coordinate Delay Reconstruction [34], which does not rely on any prior knowledge the underlying dynamical system. By utilizing the discretely sampled data { and incorporating two hyper-parameter namely, embedding dimension m and time delay т, a high-dimensional dynamical trajectory {u.) in phase space be constructed by Eq. (5).
t; = (2i-(m- I)r: 24 (m
(5)
Kpatos = Unfold (K.,p.)
p, p)
(6)
3

---

Multi reselation Dynamic Memorr function
Polynomial Function Space
FrequeRcy enheuced
Imput time seties (x,, Xxd
output time series
T
Uo Xx Kwarl
Frequency Domain
$20
observation function
Up
Down !
PSR
82
$28
T Up
Down J
Operator X = {)
96
Gb
Initial DS state
future DS state
Figure 2: Overall architecture of Attros. Initially, the PSR technique is employed restore the underlying dynamical structures from historical data [z:]. Subsequently, the dynamical system trajectory is into MDMU, projected onto polynomial space 9 using time window 0 and polynomial order N. Gradually, hierarchical projection performed obtain more macroscopic memories of the dynamical system structure. Finally, local evolution operator KC()) in the frequency domain is employed to obtain future state, thereby for the prediction.
For multivariate time series data with C' variables, we have observed considerable variations in the Lyapunov exponents of each variable, hence channel-independent strategy [33] employed to construct unified dynamical system I. C IR".. To accelerate model convergence and reduce complexity, we apply non-overlapping patching to obtain Kpatch
(6). We denote the number of patches as L, using u E R Bx represent the tensor used for computing, where determination of m and is achieved by applying the CC method [23] as shown in Appendix C.I.
mp. The
Remark 3.1. This represents the pioneering non-parametric embedding in LTSF, effectively reducing the model parameters the embedding and output projection process (m is typically single-digit), Remark 3.2. a large body of dynamical literature [44, 41, 20, 25], local linear approximation serves effective method for modeling dynamical systems, providing basis for the effectiveness of the patching operations in this paper.
3.2 Dynamical Representation by MDMU
Proposition 1. A diag(-1,-11.... } which utilizes polynomial projection under a finite measure window (Lebesgue measure).
a rough approximation of normal Hippo-LegTi [11] matrix,
Remark 3.3. All proofs in this section can be found in Appendix B.
Adhere to Mamba [10]. we generate B and measure window 0 by linear layer, and propose a novel parameterized method (Proposition 1) for A instantiate Eq. (2a):
Broadcastp(diag(111 -1,..)
B Lin
A/O = softllus (u)),
(7)
where A R2 DxNN represents the dynamical characteristics of the system's forward evolution in the polynomial space. Due to its diagonal nature, its representational capacity is comparable to DDxNNN Matrix E R BxLNN controls the process of projecting u onto the polynomial domain like gate mechanism: The learnable approximation window A I Bx similar to attention mechanism, enables adaptive focus on specific dynamical structures (attractors).
Remark 3.4. In the Hippo theory, the measure window denoted by 6 while in SSMs [10, 12], the discrete step size is represented by A. These Iwo terms can be considered approximately equivalent
As shown in Figure 3(a), in practical computations, we need to discretize Eq. (2a) to fit the discrete dynamical trajectories. We apply zero-order hold (7OH) discretization [11] to matrix A, while opting for combination of Forward Euler discretization for B (instead of the commonly used

---

(a) Discretization Limited Measure Approximation
Sequential Computing OLL)
Ug
3/
142
M3
Bg
B2
discrettze
A
A2
Xg
X2
Xg
6
(b) Multi-scale Dynamic Memory Unit
Blelloch Scan Algorithm (llogL)
((+)
Ukp
14
U2
Mg
up projection
Rg
Ba
Pli Hla
B,ug
B,
Bg
A
A2
8
Up
Bzug
Se G2A e st
,",, e Să
A3
Down
Xg
+ B
8o0 G%o
X
Xg
X₃
Figure 3: (a) Discretization of continuous polynomial approximation for sequence data. g represents the optimal polynomial constructed from polynomial bases. (b) MDMU projects the dynamical structure onto different orthogonal subspaces G and S. (c) Sequential computation for Eq. (2a) in (T.) time complexity. (d) Blelloch tree scanning for Eq. (2a) in (logL) by storing intermediate results.
B (AA) -1 (exp(AA) - I) AB in SSMs), resulting in a more concise representation: (ZOH): A =
(Forward Euler) B AB.
(8)
Next, can project the dynamical trajectory u the polynomial domain using the discretized kernel to obtain dynamical representation æ E BBxLxDxN (9). This process can be achieved with (T.) complexity by employing sequential computation (Figure 3(c) or by utilizing the Blelloch scan (Figure 3())) to store intermediate results with (logg) complexity.
K = (B. BB AB A
B
x == K.
Up to this point, the construction the underlying continuous dynamical system and its application to discretely sampled data 2n have been established. However, in this scenario, we are still limited to a single representation of the dynamical structures with measure window The strange attractors, on the other hand, are often composed of multiple fundamental topological structures. Therefore, we require a multi-scale hierarchical representation of dynamical structures to capture their complexity.
To address this, as illustrated in Figure 3(b), we progressively increase the length of the window 0 by powers of 2. The region previously approximated by g 9 (left half) and g G (right half) will now be approximated by 2 9
Since the piecewise polynomial function space can be defined as the following form:
ttt = {g deg(g) < k, x (2-"1,2-*(
-0,
(10)
otherwise
with polynomial order &s N. piecewise scale z (0). and piecewise internal index (0, 10.,, 2" -1). it evident that dim() 2" implying that G possesses superior function capacity compared to -hh5- All functions in Gh are encompassed within the domain of G Moreover, since G and G% can be represented as space spanned by basis functions (øf (x)} and (x))
any function including the basis function within the Ghe, space can be precisely expressed as linear combination of basis functions from the G space with proper tilted measure μ28:
20 (x)
Σ H& (1
Σ H (x) (x) j=
(1I)
5

---

The projection coefficients on these two spaces can be mutually transformed using the linear projection matrix H and its inverse matrix H based on the odd or even positions along the L dimension in x.
xt 28 1
x 28 E ER. BxL//xXND
(12)
Remark 3.5. Although A is akin to an attention mechanism leads to different measure windows, ie., 01 * 02, it still maintains the linear projection property for up and down projection:Go, Go, <> G0,402. In our illustration, we have used unified measure window for simplicity.
Theorem 2. (Approximation Error Bound) Function f [0, 1] R. is k. times confinuously differentiable, the piecewise polynomial g E G approximates f with mean error bounded as:
Ilf all 2- 2
(()0
sup
4 NT [c(0,1]
Iteratively repeating this process enables us to model the dynamical structure from a more macroscopic perspective. Theorem 2 indicates that under this approach, the polynomial projection error has convergence of order N. Hyper-parameter analysis can be found in Figure 5.
Remark 3.6. When the function is uniformly equal across the dynamical structure, H and 0 are shared each projection level as the projection matrix for the left and right interval.
Al of evolution operator K. [K(O) is [ À
Theorem 3. The mean attractor evolution error
bounded by [|Co - A 1)E (-3V, + +), with the number of random patterns 2 V stored in the system by interaction paradigm & in ideal spherical retrieval paradigm.
Based on this hierarchical projection, we additionally want to uphold the constancy of attractor patterns throughout the evolution, which is equivalent to minimizing attractor evolution errors. According Theorem 3, it is imperative to ensure the separation between attractors, denoted as V. := minj [ A. A,) is sufficiently large. While there is intersection between the 9N and C.Ny spaces, which limits the attainment of sufficiently large V. To address this issue, define an orthogonal complement space 9Ø = S + ghy. to establish series of orthogonal function spaces (S S .... S. GoD We can extend Eq. (12)
20 x 01
== 20 = r + r
(13)
01 = 10 20 G 20
20 = 20 + G102
(14)
Theorem states that the coarsest- graiined G space, along with a series of orthogonal complement S spaces, approximate any given dynamical system structure with finite error bounds in Theorem 2.
Theorem 4. (Completeness in T.2 space) The orthonormal system By = {Øg : = 1,..., N} U
(u55 j ,,...,N:r- 0, 1.2.....: :- ... - -1} spans L [o,
Remark 3.7. H. H.G.H' HH,HH G NxN are obtained by applying Gaussian Quadrature to Legendre
polynomials [13]. The grudients of these matrices are subxequently utilized for adaptive optimization.
The hierarchical projection can be implemented explicitly using iterative display or can be efficiently computed through an implicit implementation proposed in the following Section 3.3.
3.3 System Evolution by Attractors
Following chaos theory, we employ a local evolution method x3+1 = KC (sx)) ) to forecast future state. Given dynamics representations s s/z, the subsequent step involves partitioning the attractor and utilizing the operator KC belonging to A; for system evolution. We present three evolution strategies:
Direct Evolution: We employ each representation zz/st E DxN as the feature to partition adjacent points in the dynamical system trajectory into the same attractor using the K-means method. Subsequently, we evolve these points using a local operator KC)))
Frequency-enhancce Evolution: Inspired by neuroscience [6], where attractor structures amplified in the frequency domain, we first obtain the frequency domain representation of the dynamical structure through Fourier transformation. Considering the dominant modes as attractors, we employ K()) to drive the system's evolution in the frequency domain.

---

Hopfield Evolution: Hopfield networks [16] are designed specifically for attractor memory retrieval (see Appendix A.2). In our approach, we utilize modern version [38] of the Hopfield network for the evolution of dynamical systems, employing cross-attention operations. We treat the trainable attractor library as the Key and Value, while different scales of dynamical structure representations
{s 28
2²0 8 2 serve as the Query, enabling sequence-to-sequenec evolution.
Our experimental results in Table 4 demonstrate that the frequency-enhanced evolution strategy outperforms others comprehensively, and we introduce two implementation approaches:
Explicit Evolution. Initially, the finest dynamical representation x" is obtained using K, and 2220 By applying the Fourier Transform, we select M low-frequency components the primary modes. Each mode undergoes linear evolution K00 W, CNXN, followed by back projection to the original scale.
then expanded to multiple scales s", s2 ...,..... 2²0
Implicit Evolution. However, explicit evolution methods inevitably increase the time complexity. To address this issue, inspired by the Blelloch algorithm and hierarchical projection, which both utilize tree-like computational graphs, we propose an implicit evolution method.
A Blelloch scan [3, 42] (Figure 3(d)) defines binary operator qi qj == (qj,a O bb,a: bi. &6,6 + (5,6)
used to compute the linear recurrence 2₁:- = A + Buy. We take L = 4 as an example:
Up sweep for even position
Down sweep for odd position
c2
A. Bui)
[A. (A.Bwa) = (A², ABuu Buz)
= (1,0) A, Bus)
(x. Bus)
41 = Cs ca.
A. Bus)
Bus) = (A ABus Bui)
A ABa, Buz) A, Bug)
A² ABu + Buz) AA* ABug Bus)
AA Baa + AB Bus)
A A'B A Buz ABus Bus)
JT, R2, 23, 24 [M. [M, ral. ra
The process commences by computing the values of variable r at even positions through an upward sweep. Subsequently, these even position values are employed during a downward sweep to calculate the values at odd positions. The corresponding value of x resides in the right node of r. Thus, we can modify the binary operator as 4 qj := (qj.a C hiaa H; @ (gj.a @ qi + )), thereby implicitly integrating hierarchical projection into the scanning operation. This leads to the scales of rı;
if
scale(I;-1)
+1 if is odd
scale(x;)
(i)
if i is even and power of 2
1
if is even and not power of 2
We simplify the original H1/00,, /0//02, H H1201/02,00 /t with just H E IR BxLNN which is generated directly through linear layer, and omit the reconstruction process. This approach directly sparsifies the kernels A B in different subspaces and using the linear layer to generate hierarchical space projection matrix. Afterwards, we learn the evolution using the data x in the frequency domain.
H Linear (u), Wont Li.. (u).
(15)
In this paper, we utilize this indirect and efficient hierarchical projection as the default setting. Ultimately, Attraos projects from the polynomial spectral space back to the phase space by employing another gating projection Wont E R BxLxNxI (15) to z E R Bx and derives the prediction results using an observation function parameterized by W,, E LDD (flattening the patches).
4 Experiments
In this section, we commence by conducting a comprehensive performance comparison of Attraos against other state-of of-the-r-t- models in seven mainstream LTSF datasets along with two typical chaotic datasets, namely Lorenz96-3d and Air-Convection, followed by ablation experiments per- taining to model architectures. Furthermore, leveraging the properties of chaotic dynamical systems, we explore their extended applications, including experiments on chaotic evolution, representation, reconstruction, and modulation (Appendix E). Finally, we provide the complexity analysis and robust- ness analysis. For detailed information regarding baseline models, dataset descriptions, experimental settings, and hyper- parameerr analysis, please refer Appendix D.
7

---

Table 1: Average results of long-term forecasting with an input length of 96 and prediction horizons of (96, 192, 336, 720). The best performance in Red, and the second hest in Blue. Full results are in Appendix E.S.
Model
Anraos
MambasTS
M-Mambu
RWKVITS
GPT-TS [SS]
Koopa [32]
InvTrm [3]]
PaicTTST
DEinear
(Ouns)
[Time Embb))
[
[12]
[33]
[S]]
Metric
MSE
MAE
MSH
MAE]
MSE
MAR]
MSE
MAE]
MSE
MAH
MSB
MAE
MSH
MAE]
MSH
MAE
MSF
MAR
ETThl
[0.423
0.420
0.444
0.438
|
0.459
0.453
[0.454
0.446
|
[0447
0:450
|
00450
0.443
[0:463
0.454
0.434
0.435
0.462
0.458
ETTh2
0.372
0.399
0.386
0.410
0.381
0.407
0..375
0.402
[0.389
0.414
[0:397
0:417
[0.383
0.107
0.380
0,406
0.561
0.520
ETTml
0,382
0.396
0.406
0.399 0.407
0.399
0.403
|0.396
0:401
[0:395
[0:407
0.412
0:403
0.403
ETTm2
00,280
0.324
0.299
0,345 | 0.2299
0.333
[0.285
0.330
|0.294
0.539
|
00.281
0.326
00.291
0.335
0.288
0.329
0.345
0.396
Exchange
0.349
0.395
0.364
0.405
0.364
0,407
[0,406
0.439
[0.37]
0.409
|0.390
0.424
[0.366
0.416
0.383
0.416
0.346
0.416
Crypto
0.187
0.157
0.193
0.162
| 0.199
0.163
0.190
0.159
|0.19
0.164
|0.19
0.165
00.196
0.164
0.192
0.161
0.201
0.176
Weather
0.246
0.271
0.258
0,280
0.2252
0.277
0.256
0.280
|0.279
0.279 | 00.247
0.277
00.260
0.280
0.258
0,280
0.266
0.319
Table 2: Prediction results on the artificial (Lorenz96-30) 3d) and real-world chaotic datasets (Air-convection) with various forecasting lengths. Red/Blue denotes the best/second performance.
Model
Altraus (Ours)
MT
S-Mamba [1]]
RWKV-TS
Knupa [32]
Tim
PatchTST
DLincar
(Time Emb.)
[1:]
[31]
[33]
[52]
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
0.892
0.721
0.894
0.722
0.891
0.786
0.929
0.750
Lorenz96-33
192
0.838
0.662
0.910
0.748
0.917
0.761
0.894
0.744
0.881
0.752
0.944
0.811
0.899
0.714
0.910
0.753
336
0.837
0.681
0.943
0.772
0.968
0.788
0.982
0.825
0.914
0.753
0.997
0,841
0.922
0.787
01893
0,737
0.739
0.996
0.814
1.058
1,129
0.955
0.971
0.806
AVG
0.847
0.692
0.935
0.764
0,986
0.808
0.957
0.803
0.919
0.761
1.008
0.848
0.930
0.771
0.903
0.762
96
0.437
0.303
0.451
0.314
0.468
0.329
0.447
0.308
0.443
0.307
0.470
0.337
0.465
0.331
0.441
0.525
192
0.455
0.321
0.172
0.331
0,481
0.340
0.467
0.328
0.451
0,329
0.485
0.349
0.477
0,341
0.460
0.358
Air
0.468
0.342
0,485
0.461
0.339
0.468
0.499
0.363
0.484
0.341
720
0.466
0.365
0,492
0.379
0.501
0.586
0.482
0. 0.367
0.488
0.369
0.516
0.401
0.304
0.392
0.474
0.359
AVG
0.454
0.328
0.471
0.342
0.484
0.352
0.464
0.336
0.463
0.337
0.4%3
0,363
0.483
0.354
0.459
0.341
4.1 Overall Performance
Mainstream LTSF Datasets. As depicted in Table 1, we can observe that: (a) Attraos consistently exhibits the best performance, closely followed by RWKV-TS and Koopa, which underscores the crucial role of modeling temporal dynamics in LTSF tasks. (b) The models based on state space models (Mamba4TS, S-Mamba) generally outperform the Transformer-bant InvTrm), indicating the potential superiority of state space models as fundamental frameworks for temporal modeling. (c) The performance of the GPT-TS model, which relies on pre-trained large language model, is relatively average, suggesting the inherent challenge in directly capturing the dynamics of temporal data using such models. A promising avenue for future research lies in training a dynamical foundational model from scratch on large-scale physical datasets or leveraging pre-training to obtain an attractor tokenizer that is better suited for inputs to the large language model.
models (ST
Chaotic Datasets. Table 2 presents the results both artificial and real-world chaotic datasets. It can be observed that: (a) Attraos exhibits superior performance on both datasets, thanks to the utilization of the PSR and MDMU modules, which effectively capture the multi-scale attractor structures. (b) In Lorenz96 dataset, where there is a prior knowledge about the phase space dimension, Attraos outperforms other models by significant margin. This highlights the importance of PSR in recovering the complete temporal dynamics. (c) Apart from Attraos. the linear model (DLinear) demonstrate the best predictive results due to their robustness. Conversely, deep learning models based on transformers cxhibit weaker performance and fail to model chaotic dynamics effectively.
4.2 Further Analysis
Ablation Studies. Next, we turn off each module of Attraos assess their individual effects. As shown in Table 3, we observe consistent performance decline of Attraos when deleting each module: (a) The removal of Phase Space Reconstruction exhibits the most severe performance degradation, indicating that the process of reconstructing the dynamical structure through PSR forms the foundation for Attraos' efficient capture of attractor structures. (b) Multi-scale hierarchical projection effectively captures the complex topological structure of the singular attractor, leading to improved performance. However, overfitting may occur in certain prediction lengths. (c) Time-varying B and W aut acts as a gating attention mechanism, allowing for more focused emphasis on dynamical structure segments
8

---

Table 3: Results of ablation study. "wlo" denotes without. PSR: Phase Space Reconstruction; MS: Multi-scale
hierarchical projection; TV: Time-varying B and W SPA: Specially initialized A FE: Frequency Evolution. Red/Blue denotes the performance improvement/decine.
Model
Attraos
Wio
w/o MS
w/a
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.292
0.348
0.301
0.357
0.299
0.355
0.299
0.354
0.294
0.351
0.297
0.352
ETTh2
192
0.374
0.386
0.405
0.381
0.384
0.388
336
0.420
0.432
0.427
0.438
0.426
0.436
0.424
0.430
0.425
0.436
0.427
0.435
720
0.418
0,431
0.431
0.450
0.425
0.437
0.416
0.427
0.421
0.433
0.427
0.437
AVG
0.376
0.399
0.387
0.413
0.380
0.405
0.380
0.402
0.478
0.401
0.382
0.403
96
0.159
0.205
0.171
0.215
0.163
0.210
0.167
0.214
0.162
0.209
0.164
0.211
Weather
192
0.212
0.266
0.263
0.253
0.222
0.249
0.253
336
0.265
0.288
0.282
0.304
0.271
0.295
0.277
0.299
0.263
0.288
0.270
0.294
730
0.347
0.340
0.358
0.351
0.346
0.338
0.355
0.346
0.347
0.342
0.355
0.356
AVG
0.246
0.271
0.259
0.283
0.250
0.274
0.255
0.282
0.247
0.272
0.251
0.279
Table 4: Results of various chaotic evolution strategies. Red/Blue denotes the best/second performance.
SSMs
Implicit
Explicit
Direct-CNi
Hopfield Hoppield-Ifmaiees Inmoides
Hompieldd44mndes 64mmdes
Metric
MSB
MAE
MSE
MAE
MSE
MAB
MSE
MAE
MSB
MAE
MSE
MAE
96
0.370
0.388
0.376
0.392
0.388
0.404
0.395
0.410
0.384
0.405
0.389
0,407
E17h1
192
0.416
0.418
0.419
0.423
0.441
0.439
0.444
0.437
0.430
0.446
0.427
01442
0.458
0.432
0.456
0.480
0.485
720
0.447
0.442
0.454
0.448
0.511
0.508
0.510
0.512
0.494
0.491
0,502
0,500
AVG
0.423
0.420
0.429
0.426
0.457
0.453
0.458
0.454
0.447
0.456
0.426
0.460
0.172
0.254
0.181
0.260
0.182
0.263
El1m2
192
0.242
0.301
0.247
0.308
0.264
0.331
0.265
0.334
0.255
0.312
0.259
0.314
336
0.34L3
0.340
0.310
0.349
0.325
0.359
0.319
0.354
0.315
0.347
0.312
01334
0.401
0.399
0.426
0.417
0.422
AVG
0,250
0.324
0.285
0.330
0.300
0.344
0,297
0.345
0.293
0.336
0.293
0.336
that potentially contain attractor structures, thereby enhancing performance. (d) The initialization method proposed for A matrix demonstrates marginal yet consistent improvements, underscoring the importance of prior inductive bias for machine learning models. (e) Frequency domain evolution methods significantly reduce temporal noise information and amplify attractor structures. We will further analyze the importance of frequency domain evolution in subsequent analysis.
Chaotic Evolution Strategy. We further compare various dynamical system evolution strategies mentioned in Section 3.3. From Table 4, it is evident that the frequency-ent enhannce evolution strategy outperforms the others. Moreover, our proposed efficient implicit evolution method can adaptively explore multi-scale dynamical structure information, avoiding redundant cyclic computations and mitigating overfitting, (b) An inherent characteristic of time series data is significant noise, making it challenging to capture the underlying dynamical structures in the time domain. Direct evolution do not yield satisfactory results.
strategies, whether linear or non-linear neural network-base Moreover, according the theorem 2 in FiLM [53], the recursion computation for dynamic projection further accumulates noise information. (c) Applying Hopfield networks in the time domain also proves to be unsatisfactory, and even adding more patterns (Key and Value) can have adverse effects. A potential solution is to apply Hopfield networks in the frequency domain instead.
Complexity Analysis. As depicted in Figure 4, we present a comprehensive visual analysis comparing Attraos with various baseline models in terms of their average performance on the ETTh1 dataset. The x-axis represents the training time, the y-axis represents the test loss, and the circle radius corresponds to the model parameters. In this analysis, we substituted GPT-TS with FiLM due to its limited relevance to this specific evaluation. The results clearly demonstrate that Attraos surpasses other models in both time and space complexity, maintaining a significant advantage. Notably, when compared to the PatchTST model with a hidden dimension of 256 (2.4M parameters), Attraos (0.2M parameters) possesses only one-twelfth of its parameter count.
Robustness Analysis. We add a 0.1 (O, 1) Gaussian noise to the training dataset to test the robustness of Attraos. As shown in Table 5, can be observed that Attraos exhibits strong robustness against noisy data, and increasing the level of noise can even lead to further performance improvement. This is attributed to the frequency domain evolution strategy, where we retain only the dominant modes as attractor structures, effectively removing the noise information. Furthermore, an interesting phenomenon has been observed in our experiments: as noise is introduced, the model's 's convergence speed increases. This discovery warrants further exploration in future studies.

---

Time vs. Loss vs. Model Size
Model
Attraos
Attrao-noise noise
Metric
MSE
MAE
MSE
MAE
Koops
96
0.370
0.388
0.360
0.390
DLinedFransformes Mamba WWKV-TS
192
0.416
0.418
0.413
0.415
1M
336
0,458
0,432
0.430
Loss
720
0.447
0.442
0,451
0.444
AVG
0.423
0.420
0.422
0.420
MambaaTs
96
0.172
0.254
0.170
0.251
HTTmm
192
0.242
0.301
0.238
0.297
Attraos Explicit
336
0.303
0.340
0.305
0.339
Attraos-ImplicR
720
0.401
0.399
0.398
0.392
AVG
0.280
0.324
0.278
0.320
Training Time
Table 5: Robustness analysis with additional noise. Red/Blue
Figure 4: Complexity analysis.
denotes the performance improvement/deeline.
Chaotic Reconstruction. As illustrated in the left of Figure 5, visualize the phase space forecasting results of Attraos on the Lorenz96 system. It can be observed that: Although some minor details may be missing due the sparsity introduced by the frequency domain evolution, Attraos successfully reconstructs the chaotic dynamic structure of Lorenz96. Moreover, modeling time series based on the dynamics structure of the phase space can be viewed as form of data augmentation, e.g., two-dimensional time figuring [50] or seasonal decomposition [54].
Chaotic Representation & Hyper-parameter Analysis. In the right of Figure 5, validated the impact of polynomial dimensions on the model performance. Noteworthy observations include: As noted in substantial literature on SSMs, polynomial dimension of 256 is generally required to approximate input time series signals with sufficient accuracy. However, we found that the patch operation effectively reduces this threshold linear fashion. For instance, the ETT dataset with a phase space dimension of 4, the required polynomial dimension drops to 256/4 = 64 dimensions.
0.34
Attraos Attraos, no. _acpsshooo_aahhhhheeeee
Dimensioo
0.32
0.30
30
200
400
600
200
600
32
64
128
256
512
Time Step
Time Step
Figure 5: Left: Chaotic Reconstruction for Lorenz96 system with 720 forecasting step. Right: Slyper-paramete analysis w.r.t. polynomial orders different model variants w.r.t. patching operation ETTm2 dataset.
Discussion. Recently, across various fields of machine learning, an increasing number of works have focused on the underlying physical properties hidden within real-world observational data [20, 21, 26, 51]. By leveraging physical priors, these models achieve significant improvements in generalization, accuracy, and interpretability. We hope Attraos introduces a fresh perspective to the LTSF community and encourages the emergence of physics-guided time series analysis models.
5 Conclusion and Future Work
LTSF tasks have long been a focal point research the machine-learning community. However, mainstream deep learning models currently overlook the crucial aspect that time series data is derived from discretely sampling underlying continuous dynamical systems. Inspired by chaotic theory, model, Attraos, considers time series generated by generalized chaotic dynamical system. By leveraging the invariance attractors, Attraos enhances predictive performance and provides empirical and theoretical explanations. In the future, will verify our proposed Attraos large- scale chaotic datasets and utilize the implicit neural representations for the phase space coordinates, enabling trainable embedding process address the instability of PSR techniques.
10

---

Acknowledgments and Disclosure of Funding
This work is mainly supported by the National Natural Science Foundation of China (No. 62402414). This work is also supported by the 2024A0310620), Guangzhou Municipal Science and Technology Project (No. 2023A03J001),, the Guangzhou Industrial Information and Intelligent Key Laboratory Project (No. 2024A03J0628), and a grant from State Key Laboratory of Resources Environmental Information System, and Guang- dong Provincial Key Lab of Integrated Communication, Sensing and Computation for Ubiquitous Internet of Things (No. 2023B 121 1212010077
Joint Funding Program (No.
References
[1] Bradley K Alpert. A class of bases in 1'2 for the sparse representation of integral operators. SIAM journal Mathematical Analysis, 24(1):246-262, 1993.
[2] Xueli An, Dongxiang Jiang, Chao Liu, and Minghao Zhao. Wind farm power prediction based on wavelet decomposition and chaotic time series. Expert Systems with Applications, 38(9): 11280-11285, 2011.
[3] Guy E Blelloch. Prefix sums and their applications. 1990.
[4] Erik Bollt. On explaining the surprising success of reservoir computing forecaster chaos? the universal machine learning dynamical system with contrast to var and dmd. Chaos: An Interdisciplinary Journal of Nonlinear Science. 31(1), 2021.
[5] Minyou Chen, Yonghui Fang, and Xufei Zheng. Phase space reconstruction for improving the classification of single trial eeg. Biomedical Signal Processing and Control, 11:10-16, 2014.
[6] Yen-Lin Chen, Yuan Chiang, Pei-Hsin Chiu, I-Chen Huang, Yu-Bai Xiao, Shu-Wei Chang, and Chang- Wei Huang. figh--dmensional phase space reconstruction with convolutional neural network for structural health monitoring. Sensors, 21(10):3514, 2021.
[7] Mete Demircigil. Judith Heusel, Matthias Löwe, Sven Upgang, and Franck Vermet. On a model of associative memory with huge storage capacity. Journal of Statistical Physics, 168:288-221 299,
2017.
[8] Robert Devaney. An introduction to chaotic dynamical systems. CRC press, 2018
[9] Ethan R Deyle and George Sugihara. Generalized theorems for nonlinear state space reconstruc- tion. Plos one, 6(3):e1829) 2011.
[10] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.0075: 2023.
[11] Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 33: 1474-1482 2020.
[12] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021.
[13] Gaurav Gupta, Xiongye Xiao, and Paul Bogdan. Multiwavelet-ban based operator learning for differential equations. Advances neural information processing systems, 34:24048 34:24048-24062. 2021.
[14] Varun Gupta, Monika Mittal, and Vikas Mittal. R-peak detection based chaos analysis of ecg signal. Analog Integrated Circuits and Signal Processing, 102:479-490, 2020.
[15] Florian Hess, Zahra Monfared, Manuel Brenner, and Daniel Durstewitz. Generalized teacher forcing for learning chaotic dynamics. arXiv preprint 2.Xiv:2306.044066 2023.
[16] John J Hopfield. Hopfield network. Scholarpedia, 2(5); 2007.

---

[17] Haowen Hou and F Richard Yu. Rwkv-ts: Beyond traditional recurrent neural network for time series tasks. arXiv preprint arXiv:2401.09093, 2024.
[18] Jiaxi Hu, Disen Lan, Ziyu Zhou, Qingsong Wen, and Yuxuan Liang. Time-ssm: Simplifying and unifying state space models for time series forecasting. arXiv preprint 2611:2200..1631, 2024.
[19] Jiaxi Hu, Qingsong Wen, Sijie Ruan, Li Liu, and Yuxuan Liang. Twins: Revisiting non- stationarity in multivariate time series forecasting. arXiv preprint arXiv:2406.037100 2024.
[20] Jiaxi Hu, Bowen Zhang, Qingsong Wen, Fugee Tsung, and Yuxuan Liang. Toward physies- guided time series embedding, arXiv preprint arXiv:2410.0665. 2024.
[21] Licheng Jiao, Xue Song, Chao You, Xu Liu, Lingling Li, Puhua Chen, Xu Tang, Zhixi Feng, Fang Liu, Yuwei Guo, et al. Ai meets physics: a comprehensive survey. Artificial Intelligence Review, 57(9):256, 2024.
[22] Ming Jin, Qingsong Wen, Yuxuan Liang, Chaoli Zhang, Siqiao Xue, Xue Wang, James Zhang, Yi Wang, Haifeng Chen, Xiaoli Li, et al. Large models for time series and spatio-temporal data: survey and outlook. arXiv preprint arXiv:2310. 10/96, 2023.
[23] H_S S Kim, R Eykholt, and JD Salas. Nonlinear dynamics, delay times, and embedding windows. Physica D: Nonlinear Phenomena, 127(112):48-66 60, 1999.
JD
[24] Witold Kinsner. Characterizing chaos through Yуаршпоw metries. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 36(2):141-151, 2006.
[25] Péter Koltai and Philipp Kunde. A koopman-takens theorem: Linear least squares prediction of nonlinear time series. Communications in Mathematical Physics, 405, 2024.
[26] Herbert Levine and Yuhai Tu. Machine learning meets physics: A two-way street, 2024.
[27] Yuxuan Liang, Haomin Wen, Yuqi Nie, Yushan Jiang, Ming Jin, Dongjin Song, Shirui Pan, and Qingsong Wen. Foundation models for time series analysis: A tutorial and survey, arXiv preprint 14345 2024.
[28] Bryan Lim and Stefan Zohren. Time-series forecasting with deep learning: survey. Philosoph- ical transactions. Series A, Mathematical, physical, and engineering sciences, 379:20200209, 02 2021. doi: (tt.0///sta.220.0200000
[29] Shengsheng Lin, Weiwei Lin, Wentai Wu, Feiyu Zhao, Ruichao Mo, and Haotong Zhang. Segrnn: Segment recurrent neural network for long-term time series forecasting. arXiv preprini arXiv:2308. 12000, 2023.
[30] Xu Liu, Junfeng Hu, Yuan Li, Shizhe Diao, Yuxuan Liang, Bryan Hooi, and Roger Zimmermann. Unitime: A language-empowered unified model for cross-domain time series forecasting. arXiv preprint ar0.0:2310.09751, 2023.
[31] Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang. Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.06625, 2023.
[32] Yong Liu, Chenyu Li, Jianmin Wang, and Mingsheng Long. Koopa: Learning non-stationary time series dynamics with koopman predictors. arXiv preprint arXiv:2305. 18803, 2023.
[33] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-temn forecasting with transformers. In the Eleventh International Conference on Learning Representations (ICLR), 2023.
[34] Lyle Noakes. The takens embedding theorem. International Journal of Bifuucation and Chaos, 1044):877-872, 1991.
[35] Frank WJ Olver. NIST handbook of mathematical fanctions hardbuck and CD-ROM. Cambridge university press, 2010.
12

---

[36] Sung Woo Park, Kyungjae Lee, and Junseok Kwon. Neural markov controlled sde: Stochastic optimization for continuous-time data, International Conference on Learning Representations. 2021.
[37] Milton Persson. The whitney embedding theorem, 2014.
[38] Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlovié, Geir Kjetil Sandve, et al. Hopfield networks is all you need. arXiv preprint arXiv:2008.02211, 2020.
[39] Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for iregullrly-samplle time series. Advances in neural information processing systems, 32, 2019.
[40] Shahrokh Shahi, Flavio H Fenton, and Elizabeth M Cherry. Prediction of chaotic time series using recurrent neural networks and reservoir computing techniques: A comparative study. Machine learning with applications, 8:100300, 2022.
[41] Charalampos Haris Skokos, Georg A Gottwald, and Jacques Laskar. Chaos detection and predictability, volume 1. Springer, 2016.
[42] Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. Simplified state space layers for sequence modeling, arXiv preprint arXiv:2208.0445 2022.
[43] Floris Takens. Detecting strange attractors in turbulence. In Dynamical Systems and Turbulence, Warwick 1980: proceedings of a symposium held at the University of Warwick 1979/80, pages Springer, 1980.
[44] Eugene Tan, Shamnon Algar, Débora Corrêa, Michael Small, Thomas Stemler, and David Walker. Selecting embedding delays: An overview of embedding techniques and a new method using persistent homology. Chaos: An Interdisciplinary Journal of Nonlinear Science, 33(3), 2023.
[45] I Vlachos and D Kugiumtzis. State space reconstruction for multivariate time series prediction. arXiv preprint arXiv:0809.2220. 2008
[46] Rui Wang, Yihe Dong, Sercan Ö Arik, and Rose Yu. Koopman neural forecaster for time series with temporal distribution shifts. arXiv preprint arXiv:2210.03675. 2022.
[47] Zihan Wang. Fanheng Kong, Shi Feng, Ming Wang, Han Zhao, Daling Wang, and Yifei Zhang, mamba effective for time series forecasting? arXiv preprint 2xXi:2443.11144: 2024.
[48] Matthew Williams, Ioannis Kevrekidis, and Clarence W Rowley. A driven approxima- tion of the koopman operator: Extending dynamic mode decomposition. Journal of Nonlinear Science, 25:1 1511311-336,, 2015.
[49] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long, Autoformer: Decomposition trans- formers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34:22419-22453 2021.
[50] Haixu WU, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Times- net: Temporal 2d-variation modeling for general time series analysis. arXiv preprint arXiv:2210.02/88 2022.
[51] Rose Yu and Rui Wang. Learning dynamical systems from data: An introduction to physics- guided deep learning. Proceedings of the National Academy of Sciences, 121((7):c2311808822 2024.
[52] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of AAAI, volume 37, pages 11121-1110 1228, 2023.
[53] Tian Zhou, Ziqing Ma, Qingsong Wen, Liang Sun, Tao Yao, Wotao Yin, Rong Jin, et al. Film: Frequency improved legendre memory model for long-term time series forecasting. Advances in Neural Information Processing Systems, 35:12677-12699 2022.
13

---

[54] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang. Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. International Conference Machine Learning, pages 27268-27286 PMIR, 2022.
[55] Tian Zhou, Peisong Niu, xue wang, Liang Sun, and Rong Jin. One fits all: Power general time series analysis by pretrained Im. In NeurlPS, 2023.
14

---

observe A Technical Background
A.1 Takens Theorem
Takens' theorem, introduced by Floris Takens in 1981, provides a framework for reconstructing the dynamics of chaotic system based conditions under which the state space of a dynamical system can be reconstructed from time-delay measurements of a single observable. The reconstructed system retains properties that invariant under smooth transformations, known as diffeomorphisms.
series of observations. The theorem establishes
In the context of discrete-time dynamical systems, Takens' theorem is commonly applied. Consider a dynamical system whose state space is a v-dimensional manifold M, with the evolution of the system governed by smooth map
F:M M.
Assume there exists a strange attractor A C M with a box-counting dimension da. According Whitney's embedding theorem [37], the attractor can be embedded into an m-dimmensional Euclidean space if
m 2d
This implies that there is a diffeomorphism e mapping the attractor A into RV such that the Jacobian matrix of is of full rank. In the delay embedding process, the embedding function is constructed using an observation function. This observation function h . + needs to be twice continuously differentiable and must assign real number to each point on the attractor A, ensuring that it is typical, meaning its derivative is of full rank without exhibiting any special symmetrics.
The delay embedding theorem states that the mapping
(x( (M()), (..,, (x)=)(x))
constitutes embedding of the strange attractor A into RV.
Now, consider d-dimensional state vector Tı, which evolves according unknown but determin- istic and continuous dynamic process. Suppose a one-dimensional observable y exists, which is smooth function of and is coupled to all the components of æ. At any given time, not only the current measurement y(t) but also measurements from past times separated by lag T: M:++; 2/+22,, and so on. Using m such lags results in an mn-dimensional vector. As the number of lags increases, the motion in the reconstructed space becomes more predictable, and in the limit as + oc, the dynamics could become deterministic. In reality, the deterministic nature of the dynamics is achieved at finite dimension, with the reconstructed dynamics being equivalent to the original system's 's dynamics, related by smooth, invertible change of coordinates (a diffeomorphism) The theorem specifically asserts that deterministic behavior emerges once the dimension reaches 2d + 1, with the minimal embedding dimension often being lower.
A.2 Hopfield Network
A.2.1 Classical Hopfield Network
Hopfield network a form of recurrent artificial neural network with binary neurons. It is characterized by:
Binary neurons with states +1 or -1.
Symmetric weight matrix with zero diagonal (no self-conncctionss)
Energy function that is minimized at stable states.
Asynchronous update of neuron states.
Dynamics
The state of each neuron is updated according to:
5t(t + 1) = sign
Σ wt)ss(t)
(16)
15

---

Where st 1) is the state of neuron i at time t + 1, Waj is the weight between neurons i and j. and 8y(t) is the state neuron j at time
The energy of the network is defined as:
E
Wij $48,
(17)
2 4d
Memory Storage and Retrieval
Memories stored in the network by adjusting the weights minimize the network energy, often using the Hebbian learning rule. The network can retrieve memory from a noisy or incomplete version by converging to a stored state. The memory capacity of Hopfield network depends on several factors, with the most significant one being the number of neurons in the network. John Hopfield proposed a rule in his original puper to estimate the memory capacity, stating that the network can effectively store approximately O. 1.15N independent memories, where N represents the number of neurons in the network. This means that for a network containing 100 neurons, can store approximately 15 patterns.
A.2.2 Modern Hopfield Network
In order integrate Hopfield networks into deep learning architectures, The Modern Hopfield Network allows for continuous state updating. It proposes a new energy function based on the associative memory model [7] and proposes new update rule that can be proven to converge to stationary points of the energy (local minima or saddle points), Specifically, the new Energy function is:
1 E - Ise (3,x"') 1 - - 8- log N M²
(18)
2
with (se(log sum exp) interaction function:
N
Ise(B, x) 13-1 --g Σ
exp (3x;)
(19)
Sering 2 Using softmax (BXT The novel update rule is:
nnew = f(E) = Xp X softmax (3XTE).
(20)
The new update rule can be viewed as the attention in Transformers. Firstly, N stored (key) patterns y: and S state (query) patterns r; that are mapped the Hopfield space of dimension dę- Then set x; = WT Yi. & Wor: ri and multiply the result of the update rule with Wv. The matrices Y (y....., ., YNd and the matrices X K YWWK =T =Q RWQ, and V YW Wy Wv. where WK E Xdd We E Xd, Wy E BRdexd,, 3 1/vda and softmax E INN is changed to row vector, the update rule multiplied by Wy is:
(ri,.... rs)" combine the y, and r; as row vectors. By defining
Z = softmax (I/V.QKT))
softmax (BRWOWTY") YWKWv.
In the Hopfield Evolution strategy of Attraos, The Query is settled as the dynamic structures and the Key/Value is settled as trainable vectors.
A.3 Orthogonal Polynomials
this section, we have selectively extracted key content from the appendix of Hippo [1]] that is pertinent to our work, for the convenience of the reader. We take Legendre polynomials as an example.
Note:
Under the usual definition of the canonical Legendre polynomial Pr. they are orthogonal with respect the measure we 1
2n +
[ Pr (x) 5mm
2
16

---

Also, they satisfy
(I((1) 1
(a(-1) = (-1)".
With respect to the measure gI[( 0, 4J. the normalized orthogonal polynomials are
(2n 1) 2 2²= +1+
In general, the orthonormal basis for any uniform measure consists of (2n + 1) times the corre- sponding linearly shifted version of Pa.
Derivatives of Legendre polynomials We note the following recurrence relations on Legendre polynomials:
(2n + 1)P P - -
Ph++ (n+ 11) + xP'
The first equation yields
P n+1 (2n + 1)Pr, + (2n - -3) + +...,
where the sum stops Po or P1. These equations directly imply
P (2n 1)P (2n 5)
and
(xr + (x) = Pi+ + PA - (n+
= =nP, + (2n - 1) (2n 3) Pn-2 P
To sum up, The Legendre polynomials are in
form.
B Proof
Proposition 5. [18] A = diag( (iagg-11--1 11,..... is rough approximation of shifted Hippo-LegT [1]] matrix.
Proof. Hippo 3 provides a mathematical framework for deriving the AB matrix for polynomial projection. The mainstream SSMs [10] typically initialize matrix A diag(-1, 22 -3, -........ }. representing the negative real diagonal elements of the normalized Hippo-LegS matrix (21). Since the LegS matrix is a mathematical approximation of exponentially decaying Legendre polynomials, initializing A d 1, 22 -3. - ... can be seen rough approximation of exponential decay. Similarly, for the normalized Hippo-LegTT matrix (22), which approximates a uniform measure, consider A = diagf-1. -1.... rough approximation of finite window of Legendre polynomials, which better for non-stationary time series as well [19]. The use of negative values for the elements ensure gradient stability during training.
1)2/2 + 1)2/2 k
(2n + 1)2 +1)
n < k, odd else
A
= k
(n 1) 1/2 + 1/2
k
(2n + 1) + 1) n > k, n odd
= A (N) rank(1).
A (D) = eig( (M))
A = A (N) rank(2),
A (D) = eig (A (N)
/Normal DPLR form of HiPPO-LegS)
/DMR
(Normal/ DPLR form of HiPPO-LegT)
(21)
(22)
Theorem 6. (Approximation Error Bound) Suppose that the function f : [0. 1] E R is times continuously differentiable, the piecewise potynomial g approximates with mean error bounded as follows:
2
Il - gll -rk
sup ((0 =E(0.1)
4
17

---

Proof. Similar to [1], we divide the interval [0. 1] into subintervals on which g is polynomial; the restriction of g to one such subinterval 4 the polynomial of degree less than k that approximates with minimum mean error. Also, the optimal can be regarded as the orthonormal projection
Q onto Biy We then use the maximum error estimate for the polynomial, which interpolates
at Chebyshev nodes of order k on I,J1 We define I,J- 2-- + 1)] for I 0, 0,,,..,,2 2 1, and obtain
11 f- -11 = S. [10N f) (x) f(x))] dx
ΣL [10," f) (x) -f(x)}² dr
$ Σ S [(0)))) (x) - f(x)] dx
2
2
Σ Ind
44
sup NET
dr
21-rk
[100) (0
sup
A
and by taking square roots we have bound (7). Here C.A.J denotes the polynomial of degree & which agrees with f at the Chebyshev nodes of order on Lrds, and we have used the well-known maximum error bound for Chebyshev interpolation.
The error of the approximation Q f of f therefore decays like 2-rk and, since S. has a basis of 2" elements, we have convergence of order k. For the generalization to m dimensions in the dynamic structure modeling, a similar argument shows that the rate of convergence is of order k/m.
Theorem 7. (Evolution Error Bound) By Jacobian value, the mean attractor evolution error K 1--A all of evolution operator K = [(X(()) is bounded by
K
A-A-Ai -|| (N - 1)8 (-BV,+1)
with the number of random patterns N 2 v stored in the system by interaction paradigm & in an ideal spherical retrieval paradigm.
Proof. Due to the numerous assumptions and extensive lemmas involved in the proof of this theorem, we will provide a brief exposition of its main ideas. For detailed proof, please refer to [38].
Firstly, the theorem defines the matching between patterns as:
Definition 1. (Pattern match). Assuming that around every pattern æ, a sphere S; is given. We say 2; is matched with & there is single fixed point x; E S; to which all points & E S converge.
As shown in Theorem 3 in [38], according to the upper branch of the Lambert W function [35], we can obtain the number of random patterns stored in a system is N 2 Vpcdt11
In our paper, we define the pattern1 as the Attractor Á in phase space, pattern2 as the future state evaluated by operator IC, noted as K. o Á. From Lemma A4 in [38], when the radius of the pattern matching sphere is M, we can describe the evolution process by jacobian value and get the matching error as:
||c A - A 2cM
where in the Equation (179) of [38]:
e = (N - 1) exp (-3 B (V, - 2 max Ao - -1111 M))
The Equation (404) of [38] says A - A.
2
I and
KC A A-
Al
2
so we can get:
<E(N 1)M exp (-3V.).
18

---

In our paper, we replace the exponential interaction function with an unknown function E() to finally get the version in Theorem 3.
Theorem (Completeness in L space) The orthonormal system Bş {Φj = 1,... 17 fug j = 1,....,,rrr 1,2. 11 -............................... 0...., - }] spans [,2 1].
Proof. We define the space GN to be the union of the b given by the formula:
GN
U r=0
g
(23)
and observe that GN¹ [2 1]. In particular, G contains the Haar basis for L 1], consisting of functions piecewise constant on each of the subintervals (2-11,2 2 + 1)). Here the closure GN defined with respect to the norm.
Il = (f. f)
where the inner product (f.g) is defined by the formula
(f.g) S f(z)g(z)dz.
Also, we have:
g:N = 9 @ S ① S @ ... .....5 S
(24)
and
S,N
linear span (1/ (x) 2/ (2'x x 1), =
knn ,, 0,...,2 -1.- (25)
let (01,.... Φx) denote an orthonormal basis for 9 in view of Equation 23, 24, and 25, the orthonormal system
for: ==1 ............ k} B u u j kr 0,1,2 : : 0, 2" -13
spans [].
Now we construct a basis for (R.) by defining, for E Z, the space G.N by the formula 9,N ( the restriction of to the interval (2-"n, 2 2n 1)) is a polynomial of degree less than k, for n E and observing that the space G.N V0, is spanned by the orthonormal set
WWr (x) = 2 (2'r
1) I,... k E 2
Thus (R()), which is contained in U, G.N. has orthonormal basis
{ ( : = 1,...,k; k; / 2}
C Model Details
C.1 Phase Space Reconstruction
Phase space reconstruction is crucial technique in the analysis of dynamical systems, particularly in the study of time series data. This method transforms one-dimensional time series into a multidimen- sional phase space, revealing the underlying dynamics of the system. The cross-correlation mutual
19

---

information (CC mutual information) method is statistical tool used to analyze the dependencies between two different time series. We provide a detailed mathematical description of these methods.
Phase space reconstruction involves transforming a single-variable time series into a multidimensional space unveil the dynamics of the system that generated the data. The technique is based on Takens" Embedding Theorem.
Takens' Embedding Theorem
Takens' Embedding Theorem allows the reconstruction dynamical system's phase space from sequence of observations. The theorem states that under generic conditions, map:
R.
(26)
can be constructed, where n is the dimension of the original phase space, and m is the embedding dimension, typically m 2n is sufficient to recover dynamics.
Time Delay Embedding
The most common approach for phase space reconstruction is the time delay embedding method. Given a time series {z(t)), the reconstructed phase space is:
(1) [z((), x(t 2T), x rlt (m 1)7)]
(27)
where 7 is the time delay, and m is the embedding dimension. In the context of coordinate delay reconstruction in complete scenario, it involves parameter called time window, which selectively uses discrete one-dimensional data for phase space reconstruction. In order to maximize the preser- vation of time series information, the time window parameter is typically set to 1. As result, this process is reversible, meaning that the original data can be reconstructed without loss of information.
The CC mutual information method is used to analyze the dependencies between two time series. It a measure the amount of information obtained about one time series through the other.
Mutual Information
Mutual information (( between two random variables X and Y is defined as:
Σ p(x,8) log p(x, y) p(z)p(y)
I(X;Y)=
(28)
ZEX.VEY
where p(z, y) is the joint probability distribution function of X and Y, and p(x) and p(y) the marginal probability distribution functions.
Cross-Correlation Mutual Information
For time series analysis, the mutual information is extended account for the time-lagged relation- ships:
Σ p(z, v(T) log p(z. v(T))
I();Y,T)=
(29)
p(z)p(u(T))
TEX
where is the time lag, and y(T) represents the time series Y shifted by Determining Time Delay (7)
Time delay (r) is an interval used in reconstructing the phase space of a time series. The right choice of is essential for revealing the dynamic properties of the system.
Calculating Mutual Information
For a given time series {x,), calculate the mutual information between the time series and its time-shifted version for different delays 7:
p(z,, Ie+T)
1(T) = Σ plan, Je+t) ) log
(30)
p(xx)p((ee4)
20

---

where p( 2(+7) is the joint probability distribution, and p(r;) and p(a+++) are the marginal probability distributions.
Selecting Time Delay
Plot the mutual information (()) against т. Choose the 7 at the first local minimum of this plot. This represents the delay where the series' points provide maximal mutual information.
Determining Embedding Dimension (m)
Embedding dimension (m) is the dimension of the reconstructed phase space. The correct ensures that trajectories the phase space do not intersect each other.
Calculating False Nearest Neighbors
For each dimension m, calculate the false nearest neighbors (FNN) F(m), which reflects the com- plexity of the trajectories reconstructed in m-dimensional space:
mm+++
F(m) = N
log Ci(m,r)
(31)
where (.m,,,)) r)) the count of points within a distance r from point in m-dimensional space, and N is the length of the time series.
Identifying the Saturation Point
As m increases, F F(m) typically increases and reaches saturation point. Choose the smallest m for which (m) does not significantly increase, indicating that increasing the dimension does not reveal more information about the dynamics.
D Experiments Details
D.1 Datasets
Our experiments are carried out on Five real-world datasets and two chaos datasets as described below:
ETT dataset are procured from two electricity substations over two years. They provide summary of load and oil temperature data across seven variables. For ETTml and ETTm2, the "m" signifies that data was recorded every 15 minutes, yielding 69,680 time steps. ETTh1 and ETTh2 represent the hourly equivalents of ETTml and ETTm2, each containing 17,420 time steps.
Exchange' dataset details the daily foreign exchange rates eight countries, including Australia, British, Canada, Switzerland, China, Japan, New Zealand, and Singapore from 1990 to 2016.
Weather dataset is a meteorological collection featuring 21 variates, gathered in the United States over four-year span.
Lorenz96 dataset is an artificial dataset. We simulate the data of 30,000 time steps with an initial dimension of 40d according to the Lorenz96 equation and map 3D by a randomly initialized Linear network to simulate the realistic chaotic time series generated from an unknown underlying chaotic system. More details are in Appendix E.3.
Crypots comprises historical transaction data for various cryptocurrencies, including Bit- coin and Ethereum. We select samples with Asset_ ID set to O, and remove the col- umn Count. We download the data from https://wwes kaggle cwm/ competitioon g g-research-crye crypto- www.rrrassttinaatttaaeeeeeeaeapa
Air Convection dataset is obtained by scraping the data from NOAA, and it includes the data for the entire year of 2023. The dataset consists of 20 variables, including air humidity, pressure, convection characteristics, and others. The data was sampled at intervals of 15 minutes and averaged over the course of the entire year.
3 http:///gghuu...om/////oaaaattt/ -itiseeiiatttiiin-eeeee-daaaa htnps://www.hge-jjee hhttps///www. www.sl.oaa.gg...
www.eena..am.moooe....... www.mmpg........e.. de/wetter
21

---

Lorenz96-3d: See Appendix E.3.
D.2
Baselines
Our baseline models include: Mamba4TS: Mamba4TS is a novel SSM architecture tailored for TSF tasks, featuring parallel scan this model adopts a patching operation with both patch length and stride set to 16. We use the recommended configuration our experimental settings with batch size of 32, and the learning rate is 0.0001.
Additionally,
S-Mamba [47]: S-Mamba utilizes linear tokenization of variates and a bidirectional Mamba layer to efficiently capture inter-variate correlations and temporal dependencies. This approach underscores its potential as a scalable alternative to Transformer technologies in TSF. We download the source code from: https//gihhb..cmm/wwwwwwwwwwwww wwwmwwwwwwwwzhh9211555 D-D-/----aMaaa and adopt the recommended setting as its experimental configuration.
RWKV-TS [17]: RWKV-TS is innovative RNN-based architecture for TSF that offers linear time and memory efficiency. We download the source code from: http:://ithub..cmm/ wwt../wwr/dohddoooooo hoom/howrrdhoo/RWWWW hou/RWWV. TS. We follow the recommended settings experimental configuration.
Koopa [32]: Koopa is a novel forecasting model that tackles non-stationary time series using Koopman theory differentiate time-variant dynamics. It features Fourier Filter and Koopman Predictors within a stackable block architecture, optimizing hierarchical dynamics learning. We download the source code from: https://gittubb. http://ggguuu..oo/uu///...CC com/thum//ooppa We set the lookback window to fixed values of (96, 192, 336, 720} instead of twice the output length as in the original experimental settings.
iTransformer [31]: iTransformer modifies traditional Transformer models for time series forecasting by inverting dimensions and applying attention and feed-forward networks across variate tokens. We download the source code from: https://githubbcco/ wwt.t/mmmm/aaaasttooooooooo We follow the recommended settings experimental configuration.
PSTchTST [33]: PStchTST introduces a novel design for Transformer- based models tailored to time series forecasting. It incorporates two essential components: patching and
structure. We obtain the source code from: http::/github.cco ctattuub..oomPPaccSSTT This code serves as our baseline for long-term forecasting, and we follow the recommended settings for our experiments.
LTSF-LLnear [52]] In LTSF-Lineat family, DLinear decomposes raw data into trend and seasonal components and NLinear is just single linear models to capture temporal relationships between imput and output sequences. We obtain the source code from: https://github.. using it as our long-term forecasting bascline and adhering to recommended settings for experimental configuration.
GPT4TS [55]] This study explores the application of pre-trained language models to time series analysis tasks, demonstrating that the Frozen Pretrained Transformer (FPT), without modifications to its core architecture, achieves state-of-the-art results across various tasks. We download the source code from: recommended settings as experimental configuration.
Stt////AMO---LL/NNaIISSS-S Fill We follow the
D.3 Experiments Setting
All experiments are conducted on the NVIDIA RTX3090-24G and A6000-48G GPUs. The Adam optimizer is chosen. A grid search is performed to determine the optimal hyperparameters, including the learning rate from (0.0001, 0.0005, 0.001) ), model layer from (1, 2) (typically 1), polynomial order from (32, 64, 256), patch length from (8, 16), projection level based on log and no more than 3. primary modes 16, and the input length 96. Due to the sensitivity of the CC method to numerical calculations, we take the average result from three iterations of the calculation.
22

---

Map
Figure 6: Dynamic structures of real-world data
E Supplemental Experiments
E.1 Dynamic Structures of Real-world Data
As shown in Figure present the phase space structures of various real-time series. Please note that due to our visualization limitations in three dimensions, the shapes of many attractors for these time series manifested in higher dimensions. We can only display slices of the attractors in the first three dimensions.
E.2 The Chaotic Nature of Datasets
Lyaponuy Exponent
Lyapunov exponents are set of quantities that characterize the rate of separation of infinitesimally close trajectories in a dynamical system. They play a crucial role understanding the stability and chaotic behavior of the system [24]. The formal definition of the Lyapunov exponent for trajectory of a dynamical system is given by:
1 In [|xX((1)] 2-+00 [16x((0)]
A lim
(32)
where (t((t) is the deviation vector at time t, and À is the Lyapunov exponent. The set of Lyapunov exponents for system provides spectrum, indicating the behavior trajectories each dimension of the system's 's phase space.
When the direction of the initial separation vector is different, the separation rate is also different. Thus, the spectrum of Lyapumov exponents exists, which have the same number of dimensions as the phase space. The largest of these is often referred the Maximal Lyapunov exponent (MLB). We reconstruct dimension m according to different phase spaces to get the maximum Lyapunov index. A positive MIEE is often considered indicator of chaotic behavior in the system, while negative value indicates convergence, implying stability in the system's 's behavior.
As shown in Table 6, calculate the maximum Lyapunov exponent for all mainstream LTSF datasets. Surprisingly, we find that their MLEs are all positive, indicating the presence chaos to varying degrees in these datasets. This directly supports the motivation proposed by Attraos. Among them, the weather dataset exhibits the strongest chaotic behavior, Note that the existence of at least positive Lyapunov exponent is sufficient to determine the presence of chaos. For example, the classical I Lorenz63 system has three Lyapunov exponents with values negative, zero, and positive respectively.
Remark E.1. For multivariate time series, take the average.
Table Maximal Lyapunov Exponents Various Datasets Multivariate Long-Term Term Forecasting
Dataset
ETThl
ETTh2
ETTml
ETTm2
Exchange
Weather
Electricity
Traffi
MLE
0.064437
0.059833
0.071673
0.082791
0.039670
0.242649
0.014613
0.189311
23

---

E.3
Simulation for Lorenz96 (Case Study)
Lorenz63 System
The Lorenz63 system, introduced by Edward Lorenz in 1963, simplified mathematical model for atmospheric convection. The model system of three ordinary differential equations now known as the Lorenz equations:
dx
x),
(33)
dt
dy
x(p =2) V
(34)
dl
dz
dt zy 32.
(35)
Here, y. and make the state of the system. The parameters p, and 3 represent the Prandtl number, Rayleigh number, and certain physical dimensions of the layer, respectively. Lorenz derived these equations to model the way air moves around in the atmosphere. This model is famous for exhibiting chaotic behavior for certain parameter values and initial conditions.
Lorenz96 System
The Lorenz96 system is mathematical model that was introduced by Edward N. Lorenz in 1996 as an extension of the original Lorenz model. It is commonly used to study chaotic dynamics in systems with multiple interacting variables.
The Lorenz96 system consists of set of ordinary differential equations that describe the time evolution of set of variables. In its simplest form, the system is defined as follows:
dx; dt (xi+1 24- 34
Here, xy represents the state variable at position i, and F is a forcing term that controls the overall behavior of the system. The nonlinear term (#i+1 -;-2 -3;-2);;-1 captures the interactions between neighboring variables, leading to the emergence of chaotic behavior. The Lorenz96 system exhibits a range of fascinating phenomena, including intermittent chaos, phase transitions, and the presence of multiple stable and unstable regimes. Its dynamics have been extensively studied to gain insights into the behavior of complex systems and explore the limits of predictability.
10
20
-10
10
(a) Lorena63
(b) Lorena96 40d mapping to 3d
(c) Disualization
Figure 7: Simulation for Lorenz96
To simulate time series generated from an unknown chaotic system, we first construct a 40- dimensional Lorenz96 system represent the underlying chaotic system. We then use randomly initialized linear neural network to simulate the observation function h. Through h, we map the Loren296-40d system, which resides in the manifold space, to 3-dimensional Euclidean space to obtain a multivariate time series dataset with three variables. As shown in the middle of Figure 7, in the observation domain, the dynamical system structure Lorenz96-40d difficult discern specific shapes, so we need to reconstruct it to 40 dimensions using the Phase Space Reconstruction (PSR) technique to study its dynamic characteristics in a topologically equivalent structure. is worth noting that on the right side of Figure 7, we visualize the last variable of the system and find striking similarities with real-world time series, even exhibiting some periodic behaviors, This further supports hypothesis that real-world time series generated by underlying chaotic systems.

---

E,4
Chaotic Modulation
1.4
0.4
Hippo-RNN Hippo-GTF
Hippo-RNN Hippo-GTF
1.2
0.2
0.8
0.6
96
192
336
720
96
192
336
720
Figure 8: Performance comparison about teaching forcing, measured by MSE. Left: Lorenzo96 dataset. Right:
Weather dataset.
In LTSF tasks, commonly observed that predictive performance deteriorates as the length of the forecasting window increases. This phenomenon bears striking resemblance to the inherent challenge in long-term predictions of chaotic dynamical systems, which are highly sensitive to initial conditions. Recent studies [15] in the field of chaotic dynamical systems have highlighted that, to address the issue of gradient divergence caused by positive maximal Lyapunov exponents indicative of chaos, the implementation of teaching forcing method to incrementally constrain the trajectory of the dynamical system presents straightforward yet effective framework. To enhance Hippo-RNN, have implemented the following modifications: å, (1 a) + aăt, & Fg (ž.-1), with I a S 1, where we intervene the evolution state z by utilizing the ground truth hidden state ž. As shown in Figure 8, this modification leads to notable improvements in both mainstream LTSF datasets and real-world chaotic datasets. However, the autoregressive nature the teaching forcing methods introduces additional computational overhead and may potentially reduce its generalization capabilities, which presents a challenge for integrating it into sequence-mapping models.
E.5 Full Results
As shown in Table 7, we showcase the full experiment results for mainstream LTSF datasets.
25

---

Table 7: Multivariate long-term term scries forecasting results in mainstream datasets with input length is 96 and prediction horizons are [96, 192, 336, 720}. The best model is in boldface, and the second best is underlined. The bottom part introduces the variable Kernel for multivariate variables.
Model
Anmos (Our)
MambacTS (Temporal Emh)
S M-Mamba
RWKVIS
GPT4TS [-]]
Koopa
In Trm
PaRATST
Diiieer
(Chemnel 1im (X])
[1]]
Merrc
MSL
MAD
MAB
M5B
MAD
3056
MSB
MAB
MAB
MAN
MAL
MS5
90
0.370
0.385
0.356
0.400)
0.388
0.407
0.383
0.401
0.39%
0.424
0.385
0,408
0.343
0.409
0.375
0.396
0.296
0.410
192
0.416
0.418
9.426
0.430
0.645
0.439
0.641
0.431
0.641
0.436
0.44]
0.431
0.485
0.442
0.429
0.426
0.349
0.354
336
0.458
0.433
0.656
0.451
0.892
0:467
0.893
0.465
0.692
1.866
0/474
0.654
0.39]
0.465
0.461
0.485
0.882
0.455
0.447
0.442
0.681
0.472
0.511
0.499
0.50
0.487
0.433
0.501
0.480
0.518
0.501
0.469
0.471
0.155
0.512
AVG
0.423
0.444
0.438
0.459
0.457
0.450
0.445
0.454
0.435
0.452
0.312
0.317
0.374
0,577
0.399
0,399
0.399
0.452
0.424
0.456
0.425
0.424
0.436
0,446
0.432
0.588
720
0.418
0.431
0.431
0.418
0.427
0.465
0.435
0.40
0.065
0.429
0.447
0.431
0.451
0.833
0.376
0.399
0.375
0,389
0.397
0,417
0.407
0.360
0.405
0.564
96
0.321
0,363
0.331
0.368
0.332
0.365
0.328
0.366
0.35
0.399
0333
0.360
0.343
0.377
0.336
0.365
0,345
0.372
ETTml
192
0.365
0,373
0.366
0.391
0.578
0.293
0.322
0.389
0374
03355
0335
0.395
0.379
03944
0.361
0.383
0,382
0.391
336
03390
0.395
0906
0413
0.409
0.144
00001
0.09
0907
0.000
13855
0:13
0/115
C315
0.3%
0.113
0.413
730
0/332
0460
0.453
0.76
0.453
0/46
04669
0.44
19455
0:41
1/85
03555
0-480
R1499
0.472
0.450
AVG
0.382
0.391
0.36
0.406
0.399
0.407
0.391
0400
0.3%
0.401
0330
0,403
0-00
0412
0-403
0.398
0.403
0.406
96
0.172
0.254
0136
0.204
0.182
0.267
0151
0264
0.190
0.25
0,00
0.261
0184
0:269
0.177
0.26
0.192
0.91
670ml
0.242
0261
0.300
0.248
0.309
0235
0255
0244
6:304
0:11
0246
0.00
0.244
0.203
0330
0.366
0.312
0.350
0.30
0321
0200
0:340
0350
0.43
0.371
720
0,801
0418
0.416
0.412
0.405
0.906
0.41
0,40
8.000
0.40
0,005
0.532
AVG
0.280
0.299
0.343
0.289
0255
0.94
0
0.326
0.335
0.329
0.345
Caype Exchong
0.171
0.082
0.200
0.244
0,180
0,087
0.129
0.155
0.091
0
0.093
0.215
0.313
0.309
0.222
0.319
0.185
0.094
0.531
0.455
0.30
0.423
0,330
0,477
0.300
0.328
0.417
0.371
0,445
0.416
0.33
0.51
0.330
750
0.800
0.669
0355
0.66
0.960
0.700
0883
0.044
0380
0.74
0.568
0.726
0385
6315
8.00
0311
6.774
(U633
AVG
0.349
0.365
0.364
0.405
0,364
0.407
0.406
0.450
0331
0.40
0,300
0.424
0.366
C4466
03333
0.16
0.346
0.416
96
0.174
0.443
0.179
0.449
0.187
0.147
6.176
0.139
0183
0.144
0.181
0.143
B.83
0.344
0.177
0.141
0.183
0155
102
0.182
0.149
0.188
0.154
0.191
0.153
0.186
11111
0.159
0.155
0.101
0.153
B.190
0.155
0.188
0.150
0.195
0.15
336
0.191
1.15%
0.197
0.166
0.197
0.163
0.192
IL162
0.01
0.168
0.208
0.173
B.199
0.367
0.195
0.164
0.206
0.180
720
0.201
1.179
0.207
0.186
0.216
0.190
0.286
1.184
0210
0.157
0.215
0.189
0212
0.189
0.208
0.185
0.219
0.201
AVG
0.187
0.157
0.195
0.162
0.19
0.163
0.190
0.159
0.196
0.164
0.199
0.165
0.196
0.364
0.192
0.161
0.201
9.176
0.139
0.215
0.165
0.208
0.175
0.205
0.158
0,203
0.316
0.176
0.217
0.197
0.212
0.257
0:215
0.254
0219
0.247
0.211
0.253
0.257
0.257
0.238
0.29)
0.275
0.275
0.297
0.20%
0.282
0.347
0.368
0.351
0.280
0.252
0.377
0.256
0.279
0.347
0.273
0.280
0.267
33
31
3
3
0
11
2
9
9
1
10
13
2
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification:
Attraos, considers time series generated by a generalized chaotic dynamic system. By leveraging the invariance of attractors, Attraos enhances time series prediction performance and provides empirical and theoretical explainations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes].
Justification:
In Section we discuss the limitations of this work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
27

---

Answer:
[Yes].
Justification:
All the proofs in this paper can be found Appendix B.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes].
Justification: We have open-sourced our code in https::////// //ann//////n/mssss//// 4open. sci enie/// x/ #/Attraos-40CK and provided detailed experimental settings in Appendix D to facilitate reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
28

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes].
Justification: We have open-soureed code and experimental settings in https:// // anonymous 4open wccienceer/Atraaee 40C2/ to facilitate reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips..
(oollc/guides/Coeeeee hetpoomisssoooooiiioeeoo onPolicy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. wctpublic//aeccccccccccccccc /gwii/gs///////eeer /code/uumiissiooooooo ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes].
Justification: We have provided detailed experimental settings in Appendix to facilitate reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No].
Justification: In time series forecasting (TSF) tasks, it is common not to provide error bars but instead directly calculate the average by conducting multiple experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
29

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes].
Justification: We provide experiment setting in Appendix D and complexity analysis in Section 4.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //nerrip///////e.eee/ cccppbcccc/eeeeccccceecccccaeeet htw.eettieeiiieeeeneeeeeenneeeee
Answer: [Yes].
Justification:
We follow the NeurIPS Code of Ethics in this paper.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes].
Justification: In Conclusion, we state that our goal is to offer the machine-learnin commu- nity fresh perspective and inspire further research on the essence of time series dynamics.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
30

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification: The datasets chosen in this paper are commonly used benchmark datasets for time series forecasting tasks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes].
Justification: Yes, we have.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
31

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes].
Justification: This paper follows CC 4.0, and the code is in an anonymized URL.. Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA].
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
32

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "pTEARgEDI5", "reviewer": "Reviewer_zUbZ", "summary": "The paper introduces a novel approach, named Attraos, for Long-term Time Series Forecasting (LTSF) based on treating the observed time series as high dimensional chaotic dynamical system. The model first estimates an embedding of the data through Phase Space Reconstruction (Takens embedding) and then utilizes a memory unit through state space model that represents the dynamical system through polynomials which allows to evolve the dynamics into future steps and forecast the data. The model is evaluated on multiple LTSF datasets and additional experiments such as ablations and robustness under noise are performed.", "questions": "See questions in W1 and W2 and also:\n\nQ1. The Phase Space Reduction approach seems to be the well known Takens embedding and authors mention Takens thm, however call the method differently. Is there something different that I'm missing in PSR that is not in the classical computation of Takens embedding?\n\nQ2. With local evolution I assumed that authors mean single step \"autoregressive\" forecasting. Is that the case? How would that change for multi-step?\n\nQ3. It would be informative if the benefit/limitation/motivation of frequency evol vs. direct evol be more elaborately explained.", "limitations": "See W1-W2 and Q1-Q3. More extended discussion of limitations would contribute to better evaluation of the work vs generalization.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "S1. The approach is proposing to forecast data evolution through learning the dynamical system that can generate such time series vs. to predict directly from data.\n\nS2. Novel approach for polynomial estimation through a state space model is proposed.\n \nS3. Fundamental properties of the approach are rigorously shown.\n\nS4. Incorporation of Belloch algorithm for speed up is used.\n\nS5. Experiments show that the approach performs better in many cases than existing approaches and ablation experiments are performed.", "weaknesses": "W1. There seems to be a detachment between the theorems and properties proved and the proposed system. There seems to be no discussion about how these properties lead to the particular setup in the paper. For example why this particular SSM was used? How the hyperparameters were chosen in light of the propositions? What are the cases that the model is limited and not expected to be effective? Also interpretation of the results is lacking.\n\nW2. For non toy experimental results exposition and interpretation of the resulting dynamical system is missing. What is the dynamical system/s that are being obtained for the benchmark and what is the distribution of polynomials?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for appreciating our technical novelty and the SOTA performance achieved by our method. We are really sorry for missing several details. Here we endeavor to address your questions.\n \n* **W1: Lack of discussion on model settings**: \n\n**[why using this particular SSM]:** The paper opts for an $A$ matrix structured as {-1, -1, -1...}, considering it a rough approximation of $Hippo-Leg$, which relies on finite measure window approximations utilizing Legendre polynomials (Line 478), aligning with our objective of approximating dynamical systems using windows $[t, t+\\theta]$. \n\n**[how the hyperparams are chosen]:** In the realm of SSM-related investigations, diverse $A$ matrices exhibit varying characteristics; for example, a diagonal matrix like {-1, -2, -3...} might emphasize past information, while a step matrix could function as a localized attention mechanism, treating specific time steps collectively. Table 3 illustrates the performance stability enhancement of our designated $A$ matrix compared to randomly initialized $A$ matrices. The revised article will encompass a more extensive variant analysis. \n\n**[Limitation]:** Irrespective of the chosen $A$ matrix, it's crucial that its diagonal elements are negative (following left-half plane control theory) to avert gradient explosions. Moreover, the diagonal state space may limit the expressive power of the model. This paper partially addresses this issue by introducing multiple orthogonal subspaces (MDMU block).\n \n* **W2: Missing additional analysis.**: \n\n**[non-toy experiment]:** In Appendix E, we have delved into Chaos Modulation (E4: rectifying trajectories with real values throughout model evolution), Chaos Reconstruction (E5: visualizing the constructed model's dynamical system alongside the actual dynamical system), and Chaos Representation (E6: exploring the impacts of truncating and non-truncating trajectories). Should you necessitate additional detailed analysis, kindly inform us, and we will promptly address your queries.\n \n**[the dynamics of other baselines]** As this paper is the first to restore the underlying dynamic structure in a time-series prediction benchmark using PSR technology and innovatively utilize polynomial estimation, whereas other baseline models are purely data-driven, it is challenging to access the dynamic systems built by other models. Detailed explanations regarding the real dynamical systems behind the datasets are provided in Appendix E1 and E2.\n \n**[Polynomial distribution]** Given that the state-space model is initialized from specific polynomials and then adapted through open matrix gradients, making it challenging to access the specific polynomial distributions behind the model. In recent work, the state-space model has been defined to adaptively update to the most suitable polynomial space for modeling the dynamics of the time series through gradient updates. (https://arxiv.org/pdf/2405.16312)\n \n* **Q1: Notation of PSR**: Takens embedding often refers to phase space reconstruction, while Takens theorem is broader, ensuring attractor structures can be recovered even in dimensions greater than twice the original sequence. These are roughly equivalent terms without a unified label in dynamic literature.\n \n* **Q2: Local Evolution**: Local evolution is not actually a single-step autoregression. In lines 81-83, the adjacent dynamic trajectory points belonging to the same attractor will undergo evolution using the same evolution operator $K_i$ (for example, time steps {1, 7, 8, 11} use the $K_1$ operator, and time steps {2, 5, 67, 68} use the $K_2$ operator). In the direct evolution strategy, we employ the KNN clustering algorithm to partition the points belonging to each attractor based on the Euclidean distance in phase space. In the frequency domain evolution strategy, we directly use dominant modes in the frequency domain to represent attractors.\n \n* **Q3: Frequency Evolution**: **[Motivation]:** We chose the frequency domain evolution strategy inspired by articles in the field of neuroscience, where attractors amplify in the frequency domain (lines 54-56).\n**[Limitations]:** Regarding interpretability, the frequency domain evolution strategy is less effective than direct evolution (dividing attractors based on Euclidean distance clustering).\n**[Advantages]:** Direct clustering can lead to incorrect attractor divisions due to the significant noise present in real-time series data. Frequency domain evolution, on the other hand, filters out some high-frequency noise, enhancing the model's robustness and consequently improving its performance.\n\nIn the revision, we have carefully incorporated your comments in the revised paper. Considering reviewers zqBV/qpQ4, all agreed that our paper has good merits such\nas **satisfactory novelty**, **good evaluation**, and **efficiency**, we believe our research findings are worth sharing with the research community. We sincerely hope that a revision is still considered.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2rJxkyfeTr", "reviewer": "Reviewer_zqBV", "summary": "The paper introduces chaos theory into a long-term time series forcasting (LTSF) model called Attraos (a play on the words \"attractor\" and \"chaos\"). They propose a Multi-resolution Dynamic Memory Unit (MDMU) which is inspired by (and looks a lot like) the State Space Models (SSMs) used in the Mamba family of models. However, unlike SSMs, Attraos assumes additional structure on top of time-series signals by utilizing attractor behavior from chaos theory in the phase space. Attraos outperforms many existing architectures (e.g., Mamba, RWKV-TS, PatchTST) on time-series tasks at a fraction of the computational complexity and cost.", "questions": "- Q1: The complete architecture for Attraos is not described. Do you just take Mamba and modify the SSM component? What is the complete set of learnable parameters? Where does the Hopfield Network come into play and how are its memories trained? What are the training hyperparameters for each experiment?", "limitations": "Limitations are adequately described in the conclusion.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- S1: **Outperforms existing architectures at a fraction of the cost**. Attraos is able to outperform models like Mamba, RWKV, and Transformers on many different time series tasks at a fraction of the computational cost (in terms of training time and parameter count).\n- S2: **Theoretically complete**. The paper provide rigorous definitions, theorems, and proofs throughout the main paper and appendix to justify their method. (Note: before this paper I was quite unfamiliar with chaos theory, though I have some experience in SSMs and time-series. I am unable to verify much of the math.)\n- S3: **Comprehensive empirical results**. The paper compares Attraos to many alternative architectures for time-series data (e.g., RWKV, Mamba, Transformers, MLPs, etc.) and shows that on many tasks Attraos outperforms alternative methods at a fraction of the computational cost. Additionally, the paper includes an ablation study for the different components of Attraos.", "weaknesses": "- W1: **Presentation of results could be clearer**. I was often confused by the choice of coloring in the tables. See the following for my suggestions:\n    1. [Table 1,2,4]. The \"Red/blue\" colorscheme is unfortunate, because most brains (including mine) associate the color \"red\" with \"poor performance\". I suggest using **bold** for best, <u>underline</u> for second best, as is often done in AI papers. \n    2. [Table 3,5]. For \"improvement/decline\" in performance, I suggest one of two options: (1) show \"better performance\" with blue, and \"worse performance\" with bold and red to emphasize decreased performance from an ablation, or (2) showing only the \"deltas\" with a \"stock ticker arrow\" (🔺 or 🔻, colored green for improvement, red for decline).\n- W2: **Complete architectural description is missing**. Most of the paper focuses on describing how to implement the special components of Attraos, but the complete picture of the architecture is missing. Additionally, there is no section describing the hyperparameters for the training setup for each experiment. See Q1 for specific questions.\n- W3: **Missing error bars**. Table 1 represents average results of long term forecasting across a swath of model classes, but these averages are not accompanied by reports of standard deviation. Thus, it is difficult to tell whether improved average performance is actually signficant. Improve by including error bars for all numerical results, which may require re-running some experiments under different random seeds.\n- W4: **Grandiose, non-academic language**. The paper starts with \"In the intricate dance of time\" [L15] 😂 and makes statements like \"we can transcend the limitations of deterministic dynamical systems\" [L34]. I suggest reworking these lines to preserve the professionalism of the rest of the work.\n\nThese weakenesses are admittedly small and easily fixable by the authors during the review process. My overall attitude towards the paper is that it is of high quality and should be accepted. However, a lot of the theory was beyond my ability to evaluate, and I am willing to adjust my score positively/negatively as I become more familiar with this work throughout the review process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for appreciating the theoretical design of our model and its efficiency. We apologize for missing several details and would like to clarify as follows:\n \n* **W1: Unclear presentation of results:** Your comment is really helpful. We have followed your comments to indicate the best/worst performance in our revision. Thank you.\n \n* **W2: Lack of several architectural details:** Thank you for pointing out this issue. Please see our reply in Q1.\n \n* **W3: Missing error bars:** Thank you for your correction. Subsequently, we chose 3 random seeds for experimentation. Due to time constraints, we have provided the standard deviation results （MSE）for select datasets. The comprehensive experimental error data and associated error bars will be included in the Appendix.\n||ETTh1|ETTm2|Weather|\n|---|---|---|---|\n|96|0.002|0.003|0.003|\n|192|0.003|0.003|0.001|\n|336|0.001|0.002|0.001|\n|720|0.005|0.003|0.002|\n\n\n \n* **W4: Several non-academic words**: We have revised this part based on your advice. Thanks for raising this concern.\n \n* **Q1.1: The complete architecture of Attraos**: We have streamlined the architecture by removing redundant modules from Mamba (such as gated multiplication and local convolution), keeping only the fundamental SSM kernel $K(t,s)=e^{tA}B$. This refinement enhances interpretability and recent studies suggest that redundant modules can introduce side effects in time series prediction tasks (https://arxiv.org/pdf/2406.04320, https://arxiv.org/pdf/2405.16312). To facilitate a deeper grasp of the Attraos framework, we present pseudocode below for your enhanced understanding, and a model pipeline structure diagram will be added in the appendix:\n \n| Input | Discrete historical data {z} in batch, phase space reconstruction hyperparameters {m, τ} |\n| --- | --- |\n|Initialize State Space Model| Obtain $B$ (B,L,N), $\\Delta$ (B,L,D), $H$ (B,L,N) through three linear layers, with special initialization of $A$ (D,N)|\n|Discretize State Space Model| $\\overline{A} = e^{\\Delta A}$ (B,L,D,N), $\\overline{B} = \\Delta B$ (B,L,D,N) |\n|1|Reconstruct dynamical trajectories in phase space (Equation 5) and obtain patch representation $u$ (B,L,D) (Equation 6)|\n|2| Improved Blelloch scan algorithm, obtaining dynamical system representation $x$ (B,L,D,N) through $u$, $H$, $\\overline{A}$, $\\overline{B}$ (line 205) |\n|3| Frequency domain evolution: $\\mathcal{F}\\circ\\mathcal{K}\\circ x \\circ\\mathcal{F}^{-1}$|\n|4| Flatten Tokens, obtaining prediction results through observation function $W_{h}$|\n| Output | Discrete future prediction data |\n \n * **Q1.2: Details about Hopfield Evolution**:\nThe Hopfield network evolution strategy is one of the three mainstream attractor evolution methods mentioned in the paper (line 178). It is positioned alongside frequency domain evolution and direct evolution strategies, used for system evolution on the dynamic representation built on multi-resolution dynamic memory units (MDMU). A detailed introduction to the Hopfield network is provided in **Appendix A2 on page 13**, and the modern version of the Hopfield network(https://arxiv.org/pdf/2008.02217) is essentially a cross-attention mechanism. By employing predefined trainable tokens as attractor memory banks (Key and Value in attention), and using data as queries, the overall energy function of the system is minimized during model training to achieve stability. The trainable tokens in the stable state can be considered as the system's attractor memory.\n \n * **Q1.3: Experimental hyperparameters**:\nThe experimental hyperparameters can be located in **Appendix D3 on page 21**. For further details concerning SSM and mathematical aspects, please refer to my response to the third reviewer (qpQ4).\n \nIn the revision, we have carefully incorporated your comments in the revised paper. Considering reviewers zUbZ/qpQ4, all agreed that our paper has good merits such as **satisfactory novelty**, **good evaluation**, and **efficiency**, we believe our research findings are worth sharing with the research community.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XgNetknNFK", "reviewer": "Reviewer_qpQ4", "summary": "This paper introduces Attraos, a new model for long-term time series forecasting (LTSF) that incorporates chaos theory and views time series data as observations from high-dimensional chaotic dynamic systems. Attraos utilizes attractor invariance, non-parametric Phase Space Reconstruction, and a multi-scale dynamic memory unit to effectively capture historical dynamics and forecast future states with substantially fewer parameters than existing models such as PatchTST. Empirical evidence demonstrates Attraos' superior performance across various LTSF and chaotic datasets, providing a new perspective on the underlying dynamics of time series data.", "questions": "1.\tL90 Is it a weighted integral $\\mu(ds)$ instead of $ds$? For polynomials $\\phi$, the integral with Lebesgue measure does not converge. \n2.\tIs $\\mu$ same as $\\omega$ in L94?\n3.\tIsn’t L94 giving the formula of projection to polynomial subspace and isn’t $K_n$ the kernel? How are they connected to $e^{tA}B$, where $A, B$ only appear in the induced dynamics of $x(t)$?\n4.\tDoes $\\mathcal{A}_i$ refer that there are multiple attractors in the dynamics? $\\mathcal{A}$ are sets, then what does transpose in L158 stand for?\n5.\tL112, 120, if $A$ is a (D,N) tensor, $B$ is a (B,L,N) tensor and $u$ is a $(B,L,D)$ tensor, how is $Ax(t)+Bu(t)$ in (2a) defined? \n6.\tCould the authors give more interpretation on what they intend to do $\\Delta$ and $\\theta$? I have no idea what the authors mean by ‘$\\Delta$ is similar to an attention mechanism’ and how this is related to the previous context.\n7.\tAgain, given the shape of $B$ and $\\Delta$, it is confusing what $\\Delta B$ refers to.\n8.\tHow is $H$ in L144 related to previous text?", "limitations": "Yes.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "-\tThe paper presents comprehensive experiment results and provide rigorous theoretical results to support their claims.\n-\tLeveraging results from chaotic theories for LTSF tasks is a novel and interesting idea.\n-\tThe method proposed in the paper needs much fewer parameters than previous methods.", "weaknesses": "-\tMost of the descriptions for problem setup (section 2), methods, and theoretical analysis (section 3) are very confusing and hard to follow. There are always notations suddenly coming out without much information. I suspect there are also several typos that cause difficulties in understanding and assessments. See **Questions** for details.\n-\tIf I understood correctly, the title is a little misleading since seemingly the key insight/perspective comes from the embedding theorem by Takens, a counterpart of Whitney theorem for attractors. Are there any other insights related to chaotic systems? If not, I would suggest concretizing this point in the title, abstract, introduction, and conclusion.\n-\tSection A.1 in appendix is entirely copied from Wikipedia, https://en.wikipedia.org/wiki/Takens%27s_theorem , except for some changes on notations. However the notations in this section are not coherent, e.g. $k$ and $N$ in L414.\n-\tSimilar to the first point, the paper does not provide enough background introduction.\n\nIf the authors worry containing all the necessary details will exceed the page limit, I would suggest presenting a detailed version of problem setup, backgrounds, analysis and methods in appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for appreciating the technical novelty and efficiency achieved by our method. We are really sorry for missing several details, most of which delve into some details of the state-space model. Please allow us to provide you with a detailed response to questions 1-8.\n* **Q1: $\\mu(ds)$ or $ds$?** Typically, describing two orthogonal bases involves the measure ($\\mu(ds)$). However, we have opted for the simplified representation seen in prior state space model literature, omitting the measure when the integral context is evident. In the revised version of the paper, we will incorporate the complete $\\mu(ds)$ and provide an explanation. Your observation is appreciated.\n* **Q2: The difference between $\\mu$ and $\\omega$:** In general, $\\omega$ denotes the weight function, and $\\mu(ds)= \\omega(s) ds$. Expressing it in this manner within state space models enables the adaptable manipulation of basis and weight functions, facilitating the derivation of various state space model variants.\n* **Q3: The connection between $K(t,s)$ and $e^{tA}B$?** Equations 2a-2c (line 86) represent the standard equations for state space models [1-6]. These formulas utilize the kernel $K$ to signify the integral transformation. For a differential equation of $x^{\\prime}(t)=Ax(t)+Bu(t)$, its general solution is:$$x(t)=e^{A (t-t_0)} x (t_0)+\\int_{t_0}^t e^{A(t-s)}Bu(s) \\mathrm{d}s.$$ When we assume the initial value is 0, i.e., $x(t_0)=0$, we can express $K(t,s)$ as $e^{tA}B$ based on the definition of matrix exponentiation.\n* **Q4: Details about $A$ and $\\nabla$**: You are correct that an attractor for a dynamical system should be multiple, so in our paper, we use sets to represent this. The symbol $\\nabla$ at line 158 signifies the distinctions between these attractor patterns. As per Theorem 3, it's crucial for $\\nabla$ to be sufficiently large to maintain the stability of these attractor patterns during training (invariance). This necessity leads us to establish multiple orthogonal subspaces to store each $A_i$.\n* **Q5: The computation of $Ax+Bu$**: We have adhered to the tensor shape representation of the state space model (Mamba), and **in our open-source code, we have meticulously annotated the tensor shapes at each step of the model**. Specifically, Equation 2a represents the expression for continuous states. Given $A, B, \\Delta$, we first obtain the discrete versions for actual computation using Equation 8 as $\\overline{A}=e^{\\Delta A}$ (B, L, D, N) and $\\overline{B}=\\Delta B$ (B, L, D, N). We then proceed with the calculations outlined in Formula 9 (also $x[k]=\\overline{A}[k]* x[k-1]+ \\overline{B}[k]*u[k], x[0]=0$ where the index is in L dimension).\n* **Q6: Details about $\\Delta$ and $\\theta$**:In the field of state space models, early studies such as LMU (NIPS 2019) and HIPPO (NIPS 2020) introduced the parameter $\\theta$ to represent the size of the measurement window, mainly in theoretical contexts. However, in practical coding, the $\\theta$ term was eventually removed through formula adjustments. Progressing from S4 (ICLR 2022), state space models began incorporating a trainable discrete step size $\\Delta$ as a dynamic measurement window $\\theta$, with additional insights provided in Figure 3 (a). **In Remark 3.4, we say $\\Delta$ as an attention mechanism.** This terminology is also used in HTTYH (ICLR 2023) and Mamba (ICML 2024). By structuring $\\Delta$ as (B, L, D), where each time step of the input signal has a unique discrete step size, the model can adaptively assign different weights to information at each step, revealing important sequence elements. This behavior resembles an attention mechanism (a gating mechanism) where attention scores are computed not through pairwise products but via a linear layer directly derived from the data itself (Line 120).\n* **Q7: Details about $\\Delta B$**: Given $B$ (B,L,N), $\\Delta$ (B,L,D), $\\Delta B$ is in shape (B,L,D,N). The corresponding code is **DeltaB = Delta.unsqueeze(-1) * B.unsqueeze(2)**, which means (B,L,D,1)*(B,L,1,N).\n* **Q8: Details about $H$**: $H$ denotes the hierarchical projection matrix we have defined. As detailed in line 136, the state space model allows us to estimate the dynamical trajectory using a series of $\\theta /\\Delta$. However, attractor structures exhibit diverse forms (points, loops, surfaces), necessitating multiple scales of $\\theta (e.g., 2\\theta, 4\\theta, 8\\theta)$ for accurate approximation. To address this, as illustrated in Figure 3 (b), we iteratively combine $\\theta$ in powers of 2 (repeatedly folding the sequence to acquire representations at varied scales). The matrix $H$ signifies the projection from the prior scale to the subsequent scale. Based on the distinctions between the left and right $\\theta$ intervals, it can be further segmented into $H^{\\theta1}$ and $H^{\\theta2}$. These concepts encompass piecewise polynomial approximation and wavelet transforms (line 466).\n\n* **W2: About Chaos System:** Chaos, a characteristic of nonlinear dynamical systems, has been a focal point of extensive study. The Takens and Whitney theorems serve as valuable tools for exploring chaos. Following your recommendation, we will change **Chaos perspective** to **Dynamic systems perspective** within the document.\n\n* **W1,W3,W4**: Paper Presentation:** We trust that the responses to **questions 1-8** have assisted in clarifying any uncertainties related to symbol interpretations. In the revised manuscript, we will rephrase sections related to the Takens theorem and integrate a more thorough background on state space models to ensure accessibility for readers across various disciplines.\n\nConsidering reviewers zUbZ/zqBV, all agreed that our paper has good merits such as **satisfactory novelty**, **good evaluation**, and **efficiency**, we believe our research findings are worth sharing with the research community. If your queries have been addressed, we kindly request an increase in the rating and confidence level.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We commence by thanking the reviewers for their insightful comments. We are pleased to see that all the reviewers agree with some strengths of our paper, such as technique novelty(**Reviewer zUbZ, zqBV, qpQ4**), comprehensive evaluation (**Reviewer zUbZ, zqBV, qpQ4**), and efficiency (**Reviewer zUbZ, zqBV, qpQ4**). Your expertise also helpsus to strengthen our paper significantly. 

We apologize for any inconvenience caused by the omission of certain details in the article and endeavor to respond to each comment. We sincerely hope that the responses can release the reviewers' concerns. For reference, we present a brief introduction of the response as follows.

### **In response to reviewer zUbZ:**
* We extensively examined the **SSM setting**, including the reason for choosing the {-1, -1, -1...} form of the A matrix, its appropriate applications, and practical constraints.
* We provided a further explanation on **local evolution**.
* We extensively examined the reasons, advantages, and limitations of the **frequency evolution strategy**.

### **In response to reviewer zqBV:**
* We provided the standard deviation (**error bar**) of the experimental results on several datasets.
* We presented a **pseudocode** for the model to aid in a better understanding of the architecture.
* We offered a further explanation on how the **Hopfield network stores attractor memories** during the training process.

### **In response to reviewer qpQ4:**
* We offered detailed explanations on 8 specific questions regarding the **SSM kernel and computational details**.

Given that **all reviewers have rated soundness and contribution as 3 (good)**, we will strive to enhance the presentation of this paper. We firmly believe that our novel approach/initiative can offer the community fresh perspectives and technical contributions.

Meta-review: The paper presents a novel model for time-series forecasting building on a chaotic dynamical systems view: the approach is original and robust, both empirically and theoretically. This has been widely recognized by the Reviewers. 
Issues were mainly associated with presentation aspects and they have been appropriately addressed and clarified through the rebuttal and the discussion period. 
The work can therefore be accepted pending that the Authors thoroughly revise and rewrite Section A.1 which appears to be a verbatim (and unacknowledged) copy of the Wikipedia page on Takens' theorem.
Decision: Accept (poster)