Title: Attractor Memory for Long-Term Time Series Forecasting: A Chaos Perspective
Venue: NeurIPS 2024
Accepted: yes
Keywords: State Space Model; Time Series; Chaos Theory
Abstract: In long-term time series forecasting (LTSF) tasks, an increasing number of works have acknowledged that discrete time series originate from continuous dynamic systems and have attempted to model their underlying dynamics. Recognizing the chaotic nature of real-world data, our model, Attraos, incorporates chaos theory into LTSF, perceiving real-world time series as low-dimensional observations from unknown high-dimensional chaotic dynamical systems. Under the concept of attractor invariance, Attraos utilizes non-parametric Phase Space Reconstruction embedding along with a novel multi-resolution dynamic memory unit to memorize historical dynamical structures, and evolves by a frequency-enhanced local evolution strategy. Detailed theoretical analysis and abundant empirical evidence consistently show that Attraos outperforms various LTSF methods on mainstream LTSF datasets and chaotic datasets with only one-twelfth of the parameters compared to PatchTST.

Attractor Memory for Long-Term Time Series Forecasting: A Chaos Perspective

Abstract

In long-term time series forecasting (LTSF) tasks, an increasing number of works have acknowledged that discrete time series originate from continuous dynamic systems and have attempted to model their underlying dynamics. Recognizing the chaotic nature of real-world data, our model, Attraos , incorporates chaos theory into LTSF, perceiving real-world time series as low-dimensional observations from unknown high-dimensional chaotic dynamical systems. Under the concept of attractor invariance, Attraos utilizes non-parametric Phase Space Reconstruction embedding along with a novel multi-resolution dynamic memory unit to memorize historical dynamical structures, and evolves by a frequency-enhanced local evolution strategy. Detailed theoretical analysis and abundant empirical evidence consistently show that Attraos outperforms various LTSF methods on mainstream LTSF datasets and chaotic datasets with only one-twelfth of the parameters compared to PatchTST. Code is available at https://github.com/CityMind-Lab/NeurIPS24-Attraos .

1 Introduction

In the intricate dance of time, time series unfold. Emerged from continuous dynamical systems [36, 11, 39] in the physical world, these series are meticulously collected at specific sampling frequencies. Like musical notes in a composition, they harmonize, revealing patterns that resonate through the symphony of temporal evolution. In this realm, Long-term Time Series Forecasting (LTSF) stands as one of the enduring focal points within the machine learning community, achieving widespread recognition in real-world applications, such as weather forecasting, financial risk assessment, and traffic prediction [28, 22, 30, 27, 18].

Building on the success of various deep LTSF models [49, 54, 52, 50, 29, 22], which primarily leverage neural networks to learn temporal dependencies from discretely sampled data. Currently, researchers [46, 32] have been investigating the application of Koopman theory [48, 25] in recovering continuous dynamical systems, which applies linear evolution operators to analyze dynamical system characteristics in a sufficiently high-dimensional Koopman function space. Nevertheless, the existence of Koopman space relies on the deterministic system, posing challenges given the chaotic nature of real-world time series data, evidenced through the Maximal Lyapunov Exponent in Appendix E.2.

In this paper, inspired by chaos theory [8], we revisit LSTF tasks from a chaos perspective: Linear or complex nonlinear dynamical systems exhibit stable patterns in their trajectories after sufficient evolution, known as attractors. As illustrated in Figure 1(a), attractors can be classified into four

(a)Different Types of Attractors

(b) Classical Chaotic System (Rossler, Thomas, Aizawa, Lorenz)

LimitedCycle

15

10

0.5

1.5

1.0

30

40

0.0

10

10

10

10.0

15_1.0_0.5 00

1.0

0.5

0.0

0.5

1.0

1.5

20

20

02-

10

10

20

30

(c) Real World Series (ECL, ETTm1, Weather, Traffic)

LimitedTorus

Strange

-1

1.21.20.5g.0 0.5

1.5

o't-

0.5

types: Fixed Point, indicating stable, invariant systems; Limited cycle, representing periodic behavior; Limited Toroidal, exhibiting quasi-periodic behavior with non-intersecting rings in a 2D plane, reflecting temporal distribution shifts; and Strange Attractor, characterized by nonlinear behavior and complex, non-topological shapes. Supported by chaos theory, we can transcend the limitations of deterministic dynamical systems to construct generalized dynamical system models. Figure 1(b-c) showcases various classical chaotic dynamical systems and the dynamical trajectories of real-world LTSF datasets using the phase space reconstruction method [9]. Notably, the dynamical system trajectories in these LTSF datasets exhibit fixed structures akin to those in typical chaotic systems.

Given this chaos perspective, we consider real-world time series as stemming from an unidentified high-dimensional underlying chaotic system, broadly encompassing nonlinear behaviors beyond periodicity. Our focus centers on recovering continuous chaotic dynamical systems from discretely sampled data for LTSF tasks, with the goal of predicting future time steps through the lens of attractor invariance. Specifically, this problem can be decomposed into two key questions: (i) how to model the underlying continuous chaotic dynamical system based on discretely sampled time series data; (ii) how to enhance forecasting performance by utilizing the attractors within the system.

In this context, Attraos emerges with the goal of capturing the underlying order within the seeming chaos via attractors. For tackling the first question, we employ a non-parametric phase space reconstruction method to recover the temporal dynamics and propose a Multi-resolution Dynamic Memory Unit (MDMU) to memorize the structural dynamics within historical sampled data. Specifically, as polynomials have been proven to be universal approximators for dynamical systems [4], MDMU expands upon the work of the State Space Model (SSM) [12, 10, 18] to different orthogonal polynomial subspaces. This allows for memorizing diverse dynamical structures that encompass various attractor patterns, while theoretically minimizing the boundary of attractor evolution error.

To address the second question, we devise a frequency-enhanced local evolution strategy, which is built upon the recognition that attractor differences are amplified in the frequency domain, as observed in the field of neuroscience [5, 14, 6]. Concretely, for dynamical system components that belong to the same attractor, we apply a consistent evolution operator to derive their future states in the frequency domain. Our contributions can be summarized as follows:

A Chaos Lens on LTSF . We incorporate chaos theory into LTSF tasks by leveraging the concept of attractor invariance, leading to a principal way to model the underlying continuous dynamics

Efficient Dynamic Modeling . Our model Attraos employs a non-parametric embedding to obtain high-dimensional dynamical representations, leverages MDMU to capture the multi-scale dynamical structure, and performs the evolution in the frequency domain. Remarkably, Attraos achieves this with only about one-twelfth the parameter count of PatchTST. Furthermore, we utilize the Blelloch scan algorithm [3] to enable efficient computation of the MDMU.

Empirical Evidence . Various experiments validate the superior performance of Attraos. Besides Leveraging the properties of chaotic dynamical systems, we explore their extended applications in LTSF tasks, focusing on chaotic evolution, modulation, representation, and reconstruction.

2 Preliminary

Attractor in Chaos Theory . In chaos theory, the interaction of three or more variables exhibiting periodic behavior gives rise to a complex dynamical system characterized by chaos. According to Takens's theorem [43, 34], assuming an ideal dynamical system F : M→M that 'lives' on attractor A in manifold space M which locally C N (N-times differentiable), time series data { z i } ∈ R can be interpreted as the observation of it by an unknown observation function h . To explore the properties of the unknown ideal dynamical system, we can employ the phase space reconstruction (PSR) method to establish an approximation K : R m → R m which lives in differential homomorphism attractor ˜ A in the Euclidean space with suitable dimension m [9]. The whole process is illustrated in Equation (1), where { z i } , { u i } are the sampled data from two dynamical systems. Strictly speaking, in our paper, the chaotic attractor structure ˜ A = { ˜ A i } we focused on is in phase space R m . To facilitate understanding, we further provide a visual example of the Lorenz96 system in Appendix E.3.

In the forecasting stage, the local prediction method emerges as a prominent one: u i +1 = K ( i ) ( u i ) , where the local evolution K ( i ) can be either linear or nonlinear neural network [45, 2, 40], with the parameter being shared among the points in the neighborhood of u i or belong to the same local attractor. Considering the universal approximation capabilities of polynomials for dynamical systems, we leverage the polynomial to describe the chaotic dynamical structures.

Polynomial Projection with Measure Window . We only consider the first part of the SSM (2a), which is a parameterized map that transforms the input u ( t ) into an N -dimensional latent space. According to Hippo [11], it is mathematically equivalent to: given an input u ( s ) , a set of orthogonal polynomial basis ϕ n ( t, s ) that ∫ t -∞ ϕ m ( t, s ) ϕ n ( t, s )d s = δ m,n , and an inner product probability measure µ ( t, s ) . This enables us to project the input u ( s ) onto the polynomial basis along time dimension (3), and we can combine ϕ n ( t, s ) ω ( t, s ) as a kernel K n ( t, s ) (4). When ω ( t, s ) is defined in a time window [ t, t + θ ] , it represents approximating the input over each window θ .

When the basis and measure are solely dependent on time t , it can be expressed in a convolution form (2b). In this paper, we will utilize this property to project the dynamical trajectories { u n } in phase space onto the polynomial spectral domain with kernel e t A B (2c) for characterization.

3 Theoretical Analysis & Methods

The overall structure of Attraos is illustrated in Figure 2. In this section, we provide a comprehensive description of its components, including the Phase Space Reconstruction embedding, the MultiResolution Dynamic Memory Unit (MDMU), and the frequency-enhanced local evolution, as well as the efficient computational methods employed.

3.1 Phase Space Reconstruction

According to chaos theory, the initial step involves constructing a topologically equivalent dynamical structure through the PSR. The preferred embedding method is typically the Coordinate Delay Reconstruction [34], which does not rely on any prior knowledge of the underlying dynamical system. By utilizing the discretely sampled data { z i } and incorporating two hyper-parameters, namely, embedding dimension m and time delay τ , a high-dimensional dynamical trajectory { u i } in phase space can be constructed by Eq. (5).

Multi-resolutionDynamic

Memoryinfunctionspace

PolynomialFunctionSpace

Sbe

Sbe

Frequency-enhanced

inputtimeseries

LocalEvolution

ioutputtimeseries

{x1,x2..,XxN}

↑Up

Down

个

{xN+1,xN+2.….,XN+T}

FrequencyDomain

m

m

Aio

Sk

()X

observationfunction

Ai

PSR

↑Up

Down

R

S2θ

↑Up

Down

个

{(2)} = X uozp.tado

G

G

initialDSstate

futureDsstate

For multivariate time series data with C variables, we have observed considerable variations in the Lyapunov exponents of each variable, hence a channel-independent strategy [33] is employed to construct a unified dynamical system K ⊂ R m . To accelerate model convergence and reduce complexity, we apply non-overlapping patching to obtain K patch (6). We denote the number of patches as L , using u ∈ R B × L × D to represent the tensor used for computing, where D = mp . The determination of m and τ is achieved by applying the CC method [23] as shown in Appendix C.1.

Remark 3.1. This represents the pioneering non-parametric embedding in LTSF, effectively reducing the model parameters in the embedding and output projection process ( m is typically single-digit).

Remark 3.2. In a large body of dynamical literature [44, 41, 20, 25], local linear approximation serves as an effective method for modeling dynamical systems, providing a basis for the effectiveness of the patching operations in this paper.

3.2 Dynamical Representation by MDMU

Proposition 1. A = diag {-1 , -1 , . . . } is a rough approximation of normal Hippo-LegT [11] matrix, which utilizes polynomial projection under a finite measure window (Lebesgue measure).

Remark 3.3. All proofs in this section can be found in Appendix B.

Adhere to Mamba [10], we generate B and measure window θ by linear layer, and propose a novel parameterized method (Proposition 1) for A to instantiate Eq. (2a):

where A ∈ R D × N represents the dynamical characteristics of the system's forward evolution in the polynomial space. Due to its diagonal nature, its representational capacity is comparable to R D × N × N ; Matrix B ∈ R B × L × N controls the process of projecting u onto the polynomial domain like a gate mechanism; The learnable approximation window ∆ ∈ R B × L × D , similar to an attention mechanism, enables adaptive focus on specific dynamical structures (attractors).

Remark 3.4. In the Hippo theory, the measure window is denoted by θ , while in SSMs [10, 12], the discrete step size is represented by ∆ . These two terms can be considered approximately equivalent.

As shown in Figure 3(a), in practical computations, we need to discretize Eq. (2a) to fit the discrete dynamical trajectories. We apply zero-order hold (ZOH) discretization [11] to matrix A , while opting for a combination of Forward Euler discretization for B (instead of the commonly used

(a) Discretization for Limited Measure Approximation

f(x)

f(x)

uo

u

u2

u3

θ =△

Bo

B1

B2

B3

discretize

uoiμo

A2

A3

xo

x1

x2

θ

(b) Multi-scale Dynamic Memory Unit

f(x)

f(x)

uo

u

u2

u3

up

97

B1

B2

B.3

uoiue

projection

μ2θ

xo

+Bu

B2u2+B3u3

A1

A2

θ

20

Up

x2

+B3u3

G = Ge 田S

g,g2 E G

A3

Down

x1

+Bu2

A2

G2e

0x

x1

X2

X3

B = (∆ A ) -1 (exp(∆ A ) -I ) · ∆ B

Next, we can project the dynamical trajectory u onto the polynomial domain using the discretized kernel to obtain dynamical representation x ∈ R B × L × D × N (9). This process can be achieved with O ( L ) complexity by employing sequential computation (Figure 3(c)) or by utilizing the Blelloch scan (Figure 3(d)) to store intermediate results with O ( logL ) complexity.

Up to this point, the construction of the underlying continuous dynamical system K and its application to discretely sampled data z n have been established. However, in this scenario, we are still limited to a single representation of the dynamical structures with measure window θ . The strange attractors, on the other hand, are often composed of multiple fundamental topological structures. Therefore, we require a multi-scale hierarchical representation of dynamical structures to capture their complexity.

To address this, as illustrated in Figure 3(b), we progressively increase the length of the window θ by powers of 2. The region previously approximated by g θ 1 ∈ G N θ (left half) and g θ 2 ∈ G N θ (right half) will now be approximated by g 2 θ ∈ G N 2 θ .

Since the piecewise polynomial function space can be defined as the following form:

with polynomial order k ∈ N , piecewise scale r ∈ Z + ∪ { 0 } , and piecewise internal index l ∈ { 0 , 1 , ..., 2 r -1 } , it is evident that dim ( G k ( r ) ) = 2 r k , implying that G k θ possesses a superior function capacity compared to G k 2 θ . All functions in G k 2 θ are encompassed within the domain of G k θ . Moreover, since G k θ and G k 2 θ can be represented as space spanned by basis functions { ϕ θ i ( x ) } and { ϕ 2 θ i ( x ) } , any function including the basis function within the G k 2 θ space can be precisely expressed as a linear combination of basis functions from the G k θ space with a proper tilted measure µ 2 θ :

The projection coefficients on these two spaces can be mutually transformed using the linear projection matrix H and its inverse matrix H † based on the odd or even positions along the L dimension in x .

̸

Remark 3.5. Although ∆ is akin to an attention mechanism leads to different measure windows, ie., θ 1 = θ 2 , it still maintains the linear projection property for up and down projection: G θ 1 , G θ 2 ↔ G θ 1 + θ 2 . In our illustration, we have used a unified measure window for simplicity.

Theorem 2. (Approximation Error Bound) Function f : [0 , 1] ∈ R is k times continuously differentiable, the piecewise polynomial g ∈ G N r approximates f with mean error bounded as:

Iteratively repeating this process enables us to model the dynamical structure from a more macroscopic perspective. Theorem 2 indicates that under this approach, the polynomial projection error has convergence of order N . Hyper-parameter analysis can be found in Figure 5.

Remark 3.6. When the weight function is uniformly equal across the dynamical structure, H θ 1 and H θ 2 are shared in each projection level as the projection matrix for the left and right interval.

Theorem 3. The mean attractor evolution error ∥ ∥ ∥ K◦ ˜ A-˜ A ∥ ∥ ∥ of evolution operator K = {K ( i ) } is bounded by ∥K◦ ˜ A-˜ A∥ ( N -1) E ( -β ∇ i +1) , with the number of random patterns N ≥ √ pc d -1 4 stored in the system by interaction paradigm E in an ideal spherical retrieval paradigm.

̸

Based on this hierarchical projection, we additionally want to uphold the constancy of attractor patterns throughout the evolution, which is equivalent to minimizing attractor evolution errors. According to Theorem 3, it is imperative to ensure the separation between attractors, denoted as ∇ i := min j,j = i ( ˜ A T i ˜ A i -˜ A T i ˜ A j ) , is sufficiently large. While there is an intersection between the G N θ and G N 2 θ spaces, which limits the attainment of a sufficiently large ∇ . To address this issue, we define an orthogonal complement space as G N θ = S N θ ⊕ G N 2 θ , to establish a series of orthogonal function spaces {S N θ , S N 2 θ , ..., S N 2 L θ , G N 2 L θ } . We can extend Eq. (12) as:

Theorem 4 states that the coarsest-grained G space, along with a series of orthogonal complement S spaces, can approximate any given dynamical system structure with finite error bounds in Theorem 2. Theorem 4. (Completeness in L 2 space) The orthonormal system B N = { ϕ j : j = 1 , . . . , N } ∪ { ψ rl j : j = 1 , . . . , N ; r = 0 , 1 , 2 , . . . ; l = 0 , . . . , 2 r -1 } spans L 2 [0 , 1] .

Remark 3.7. H , G , H † , G † ∈ R N × N are obtained by applying Gaussian Quadrature to Legendre polynomials [13]. The gradients of these matrices are subsequently utilized for adaptive optimization.

The hierarchical projection can be implemented explicitly using iterative display or can be efficiently computed through an implicit implementation proposed in the following Section 3.3.

3.3 System Evolution by Attractors

Following chaos theory, we employ a local evolution method x i +1 = K ( i ) ( x n ) to forecast future state. Given dynamics representations s/x , the subsequent step involves partitioning the attractor and utilizing the operator K i belonging to A i for system evolution. We present three evolution strategies:

Direct Evolution: We employ each representation x t /s t ∈ R D × N as the feature to partition adjacent points in the dynamical system trajectory into the same attractor using the K-means method. Subsequently, we evolve these points using a local operator K ( i ) .

Frequency-enhanced Evolution: Inspired by neuroscience [6], where attractor structures are amplified in the frequency domain, we first obtain the frequency domain representation of the dynamical structure through Fourier transformation. Considering the dominant modes as attractors, we employ K ( i ) to drive the system's evolution in the frequency domain.

Hopfield Evolution: Hopfield networks [16] are designed specifically for attractor memory retrieval (see Appendix A.2). In our approach, we utilize a modern version [38] of the Hopfield network for the evolution of dynamical systems, employing cross-attention operations. We treat the trainable attractor library as the Key and Value , while different scales of dynamical structure representations { s θ , s 2 θ , ..., s 2 L θ , x 2 L θ } serve as the Query , enabling sequence-to-sequence evolution.

Our experimental results in Table 4 demonstrate that the frequency-enhanced evolution strategy outperforms others comprehensively, and we introduce two implementation approaches:

Explicit Evolution . Initially, the finest dynamical representation x θ is obtained using K , and then expanded to multiple scales s θ , s 2 θ , ..., s 2 L θ , x 2 L θ . By applying the Fourier Transform, we select M low-frequency components as the primary modes. Each mode undergoes linear evolution K ( i ) = W i ∈ C N × N , followed by back projection to the original scale.

Implicit Evolution . However, explicit evolution methods inevitably increase the time complexity. To address this issue, inspired by the Blelloch algorithm and hierarchical projection, which both utilize tree-like computational graphs , we propose an implicit evolution method.

ABlelloch scan [3, 42] (Figure 3(d)) defines a binary operator q i · q j := ( q j,a ⊙ q i,a , q j,a ⊗ q i,b + q j,b ) used to compute the linear recurrence x k = A x k -1 + B u k . We take L = 4 as an example:

The process commences by computing the values of variable x at even positions through an upward sweep. Subsequently, these even position values are employed during a downward sweep to calculate the values at odd positions. The corresponding value of x resides in the right node of r . Thus, we can modify the binary operator as q i · q j := ( q j,a ⊙ q i,a , H i ⊗ ( q j,a ⊗ q i,b + q j,b )) , thereby implicitly integrating hierarchical projection into the scanning operation. This leads to the scales of x i :

We simplify the original H θ 1 /θ 2 , G θ 1 /θ 2 , H † θ 1 /θ 2 , G † θ 1 /θ 2 with just H ∈ R B × L × N , which is generated directly through a linear layer, and omit the reconstruction process. This approach directly sparsifies the kernels e A t B in different subspaces and using the linear layer to generate hierarchical space projection matrix. Afterwards, we learn the evolution using the data x in the frequency domain.

In this paper, we utilize this indirect and efficient hierarchical projection as the default setting. Ultimately, Attraos projects from the polynomial spectral space back to the phase space by employing another gating projection W out ∈ R B × L × N × 1 (15) to x ∈ R B × L × D × N , and derives the prediction results using an observation function parameterized by W h ∈ R LD × H (flattening the patches).

4 Experiments

In this section, we commence by conducting a comprehensive performance comparison of Attraos against other state-of-the-art models in seven mainstream LTSF datasets along with two typical chaotic datasets, namely Lorenz96-3d and Air-Convection, followed by ablation experiments pertaining to model architectures. Furthermore, leveraging the properties of chaotic dynamical systems, we explore their extended applications, including experiments on chaotic evolution, representation, reconstruction, and modulation (Appendix E). Finally, we provide the complexity analysis and robustness analysis. For detailed information regarding baseline models, dataset descriptions, experimental settings, and hyper-parameter analysis, please refer to Appendix D.

4.1 Overall Performance

Mainstream LTSF Datasets . As depicted in Table 1, we can observe that: (a) Attraos consistently exhibits the best performance, closely followed by RWKV-TS and Koopa, which underscores the crucial role of modeling temporal dynamics in LTSF tasks. (b) The models based on state space models (Mamba4TS, S-Mamba) generally outperform the Transformer-based models (PatchTST, InvTrm), indicating the potential superiority of state space models as fundamental frameworks for temporal modeling. (c) The performance of the GPT-TS model, which relies on a pre-trained large language model, is relatively average, suggesting the inherent challenge in directly capturing the dynamics of temporal data using such models. A promising avenue for future research lies in training a dynamical foundational model from scratch on large-scale physical datasets or leveraging pre-training to obtain an attractor tokenizer that is better suited for inputs to the large language model.

Chaotic Datasets . Table 2 presents the results on both artificial and real-world chaotic datasets. It can be observed that: (a) Attraos exhibits superior performance on both datasets, thanks to the utilization of the PSR and MDMU modules, which effectively capture the multi-scale attractor structures. (b) In Lorenz96 dataset, where there is a prior knowledge about the phase space dimension, Attraos outperforms other models by a significant margin. This highlights the importance of PSR in recovering the complete temporal dynamics. (c) Apart from Attraos, the linear model (DLinear) demonstrate the best predictive results due to their robustness. Conversely, deep learning models based on transformers exhibit weaker performance and fail to model chaotic dynamics effectively.

4.2 Further Analysis

Ablation Studies. Next, we turn off each module of Attraos to assess their individual effects. As shown in Table 3, we observe consistent performance decline of Attraos when deleting each module: (a) The removal of Phase Space Reconstruction exhibits the most severe performance degradation, indicating that the process of reconstructing the dynamical structure through PSR forms the foundation for Attraos' efficient capture of attractor structures. (b) Multi-scale hierarchical projection effectively captures the complex topological structure of the singular attractor, leading to improved performance. However, overfitting may occur in certain prediction lengths. (c) Time-varying B and W out acts as a gating attention mechanism, allowing for a more focused emphasis on dynamical structure segments

that potentially contain attractor structures, thereby enhancing performance. (d) The initialization method proposed for A matrix demonstrates marginal yet consistent improvements, underscoring the importance of prior inductive bias for machine learning models. (e) Frequency domain evolution methods significantly reduce temporal noise information and amplify attractor structures. We will further analyze the importance of frequency domain evolution in subsequent analysis.

Chaotic Evolution Strategy. We further compare various dynamical system evolution strategies mentioned in Section 3.3. From Table 4, it is evident that the frequency-enhanced evolution strategy outperforms the others. Moreover, our proposed efficient implicit evolution method can adaptively explore multi-scale dynamical structure information, avoiding redundant cyclic computations and mitigating overfitting. (b) An inherent characteristic of time series data is significant noise, making it challenging to capture the underlying dynamical structures in the time domain. Direct evolution strategies, whether linear or non-linear neural network-based, do not yield satisfactory results. Moreover, according to the theorem 2 in FiLM [53], the recursion computation for dynamic projection further accumulates noise information. (c) Applying Hopfield networks in the time domain also proves to be unsatisfactory, and even adding more patterns ( Key and Value ) can have adverse effects. A potential solution is to apply Hopfield networks in the frequency domain instead.

Complexity Analysis. As depicted in Figure 4, we present a comprehensive visual analysis comparing Attraos with various baseline models in terms of their average performance on the ETTh1 dataset. The x-axis represents the training time, the y-axis represents the test loss, and the circle radius corresponds to the model parameters. In this analysis, we substituted GPT-TS with FiLM due to its limited relevance to this specific evaluation. The results clearly demonstrate that Attraos surpasses other models in both time and space complexity, maintaining a significant advantage. Notably, when compared to the PatchTST model with a hidden dimension of 256 (2.4M parameters), Attraos (0.2M parameters) possesses only one-twelfth of its parameter count.

Robustness Analysis. We add a 0.1 * N (0, 1) Gaussian noise to the training dataset to test the robustness of Attraos. As shown in Table 5, it can be observed that Attraos exhibits strong robustness against noisy data, and increasing the level of noise can even lead to further performance improvement. This is attributed to the frequency domain evolution strategy, where we retain only the dominant modes as attractor structures, effectively removing the noise information. Furthermore, an interesting phenomenon has been observed in our experiments: as noise is introduced, the model's convergence speed increases. This discovery warrants further exploration in future studies.

Time vs.Loss vs.Model Size

FiLM

DLineaiTransformer

S-Mamba

1M

Loss

RWKV-TS

Mamba4TS

PatchTST

Attraos-Explicit

Attraos-lmplicit

Training Time

Chaotic Reconstruction. As illustrated in the left of Figure 5, we visualize the phase space forecasting results of Attraos on the Lorenz96 system. It can be observed that: Although some minor details may be missing due to the sparsity introduced by the frequency domain evolution, Attraos successfully reconstructs the chaotic dynamic structure of Lorenz96. Moreover, modeling time series based on the dynamics structure of the phase space can be viewed as a form of data augmentation, e.g., two-dimensional time figuring [50] or seasonal decomposition [54].

Chaotic Representation & Hyper-parameter Analysis. In the right of Figure 5, we validated the impact of polynomial dimensions on the model performance. Noteworthy observations include: As noted in substantial literature on SSMs, a polynomial dimension of 256 is generally required to approximate input time series signals with sufficient accuracy. However, we found that the patch operation effectively reduces this threshold in a linear fashion. For instance, in the ETT dataset with a phase space dimension of 4, the required polynomial dimension drops to 256/4 = 64 dimensions.

0

0

0.34

Attraos

Attraos_no_patch

10

10

Dimension

0.32

20

20

0.30

30

30

Q

200

400

600

200

400

600

32

64

128

256

512

TimeStep

TimeStep

Discussion. Recently, across various fields of machine learning, an increasing number of works have focused on the underlying physical properties hidden within real-world observational data [20, 21, 26, 51]. By leveraging physical priors, these models achieve significant improvements in generalization, accuracy, and interpretability. We hope Attraos introduces a fresh perspective to the LTSF community and encourages the emergence of physics-guided time series analysis models.

5 Conclusion and Future Work

LTSF tasks have long been a focal point of research in the machine-learning community. However, mainstream deep learning models currently overlook the crucial aspect that time series data is derived from discretely sampling underlying continuous dynamical systems. Inspired by chaotic theory, our model, Attraos, considers time series as generated by a generalized chaotic dynamical system. By leveraging the invariance of attractors, Attraos enhances predictive performance and provides empirical and theoretical explanations. In the future, we will verify our proposed Attraos on largescale chaotic datasets and utilize the implicit neural representations for the phase space coordinates, enabling a trainable embedding process to address the instability of PSR techniques.

Acknowledgments and Disclosure of Funding

This work is mainly supported by the National Natural Science Foundation of China (No. 62402414). This work is also supported by the Guangzhou-HKUST(GZ) Joint Funding Program (No. 2024A03J0620), Guangzhou Municipal Science and Technology Project (No. 2023A03J0011), the Guangzhou Industrial Information and Intelligent Key Laboratory Project (No. 2024A03J0628), and a grant from State Key Laboratory of Resources and Environmental Information System, and Guangdong Provincial Key Lab of Integrated Communication, Sensing and Computation for Ubiquitous Internet of Things (No. 2023B1212010007).

References

Bradley K Alpert. A class of bases in lˆ2 for the sparse representation of integral operators. SIAM journal on Mathematical Analysis , 24(1):246-262, 1993.

Xueli An, Dongxiang Jiang, Chao Liu, and Minghao Zhao. Wind farm power prediction based on wavelet decomposition and chaotic time series. Expert Systems with Applications , 38(9): 11280-11285, 2011.

Guy E Blelloch. Prefix sums and their applications. 1990.

Erik Bollt. On explaining the surprising success of reservoir computing forecaster of chaos? the universal machine learning dynamical system with contrast to var and dmd. Chaos: An Interdisciplinary Journal of Nonlinear Science , 31(1), 2021.

Minyou Chen, Yonghui Fang, and Xufei Zheng. Phase space reconstruction for improving the classification of single trial eeg. Biomedical Signal Processing and Control , 11:10-16, 2014.

Yen-Lin Chen, Yuan Chiang, Pei-Hsin Chiu, I-Chen Huang, Yu-Bai Xiao, Shu-Wei Chang, and Chang-Wei Huang. High-dimensional phase space reconstruction with a convolutional neural network for structural health monitoring. Sensors , 21(10):3514, 2021.

Mete Demircigil, Judith Heusel, Matthias Löwe, Sven Upgang, and Franck Vermet. On a model of associative memory with huge storage capacity. Journal of Statistical Physics , 168:288-299, 2017.

Robert Devaney. An introduction to chaotic dynamical systems . CRC press, 2018.

Ethan R Deyle and George Sugihara. Generalized theorems for nonlinear state space reconstruction. Plos one , 6(3):e18295, 2011.

Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 , 2023.

Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems , 33: 1474-1487, 2020.

Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396 , 2021.

Gaurav Gupta, Xiongye Xiao, and Paul Bogdan. Multiwavelet-based operator learning for differential equations. Advances in neural information processing systems , 34:24048-24062, 2021.

Varun Gupta, Monika Mittal, and Vikas Mittal. R-peak detection based chaos analysis of ecg signal. Analog Integrated Circuits and Signal Processing , 102:479-490, 2020.

Florian Hess, Zahra Monfared, Manuel Brenner, and Daniel Durstewitz. Generalized teacher forcing for learning chaotic dynamics. arXiv preprint arXiv:2306.04406 , 2023.

John J Hopfield. Hopfield network. Scholarpedia , 2(5):1977, 2007.

Haowen Hou and F Richard Yu. Rwkv-ts: Beyond traditional recurrent neural network for time series tasks. arXiv preprint arXiv:2401.09093 , 2024.

Jiaxi Hu, Disen Lan, Ziyu Zhou, Qingsong Wen, and Yuxuan Liang. Time-ssm: Simplifying and unifying state space models for time series forecasting. arXiv preprint arXiv:2405.16312 , 2024.

Jiaxi Hu, Qingsong Wen, Sijie Ruan, Li Liu, and Yuxuan Liang. Twins: Revisiting nonstationarity in multivariate time series forecasting. arXiv preprint arXiv:2406.03710 , 2024.

Jiaxi Hu, Bowen Zhang, Qingsong Wen, Fugee Tsung, and Yuxuan Liang. Toward physicsguided time series embedding. arXiv preprint arXiv:2410.06651 , 2024.

Licheng Jiao, Xue Song, Chao You, Xu Liu, Lingling Li, Puhua Chen, Xu Tang, Zhixi Feng, Fang Liu, Yuwei Guo, et al. Ai meets physics: a comprehensive survey. Artificial Intelligence Review , 57(9):256, 2024.

Ming Jin, Qingsong Wen, Yuxuan Liang, Chaoli Zhang, Siqiao Xue, Xue Wang, James Zhang, Yi Wang, Haifeng Chen, Xiaoli Li, et al. Large models for time series and spatio-temporal data: A survey and outlook. arXiv preprint arXiv:2310.10196 , 2023.

H_S Kim, R Eykholt, and JD Salas. Nonlinear dynamics, delay times, and embedding windows. Physica D: Nonlinear Phenomena , 127(1-2):48-60, 1999.

Witold Kinsner. Characterizing chaos through lyapunov metrics. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) , 36(2):141-151, 2006.

Péter Koltai and Philipp Kunde. A koopman-takens theorem: Linear least squares prediction of nonlinear time series. Communications in Mathematical Physics , 405(5):120, 2024.

Herbert Levine and Yuhai Tu. Machine learning meets physics: A two-way street, 2024.

Yuxuan Liang, Haomin Wen, Yuqi Nie, Yushan Jiang, Ming Jin, Dongjin Song, Shirui Pan, and Qingsong Wen. Foundation models for time series analysis: A tutorial and survey. arXiv preprint arXiv:2403.14735 , 2024.

Bryan Lim and Stefan Zohren. Time-series forecasting with deep learning: a survey. Philosophical transactions. Series A, Mathematical, physical, and engineering sciences , 379:20200209, 02 2021. doi: 10.1098/rsta.2020.0209.

Shengsheng Lin, Weiwei Lin, Wentai Wu, Feiyu Zhao, Ruichao Mo, and Haotong Zhang. Segrnn: Segment recurrent neural network for long-term time series forecasting. arXiv preprint arXiv:2308.11200 , 2023.

Xu Liu, Junfeng Hu, Yuan Li, Shizhe Diao, Yuxuan Liang, Bryan Hooi, and Roger Zimmermann. Unitime: A language-empowered unified model for cross-domain time series forecasting. arXiv preprint arXiv:2310.09751 , 2023.

Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.06625 , 2023.

Yong Liu, Chenyu Li, Jianmin Wang, and Mingsheng Long. Koopa: Learning non-stationary time series dynamics with koopman predictors. arXiv preprint arXiv:2305.18803 , 2023.

Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In the Eleventh International Conference on Learning Representations (ICLR) , 2023.

Lyle Noakes. The takens embedding theorem. International Journal of Bifurcation and Chaos , 1(04):867-872, 1991.

Frank WJ Olver. NIST handbook of mathematical functions hardback and CD-ROM . Cambridge university press, 2010.

Sung Woo Park, Kyungjae Lee, and Junseok Kwon. Neural markov controlled sde: Stochastic optimization for continuous-time data. In International Conference on Learning Representations , 2021.

Milton Persson. The whitney embedding theorem, 2014.

Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlovi´ c, Geir Kjetil Sandve, et al. Hopfield networks is all you need. arXiv preprint arXiv:2008.02217 , 2020.

Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for irregularly-sampled time series. Advances in neural information processing systems , 32, 2019.

Shahrokh Shahi, Flavio H Fenton, and Elizabeth M Cherry. Prediction of chaotic time series using recurrent neural networks and reservoir computing techniques: A comparative study. Machine learning with applications , 8:100300, 2022.

Charalampos Haris Skokos, Georg A Gottwald, and Jacques Laskar. Chaos detection and predictability , volume 1. Springer, 2016.

Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. Simplified state space layers for sequence modeling. arXiv preprint arXiv:2208.04933 , 2022.

Floris Takens. Detecting strange attractors in turbulence. In Dynamical Systems and Turbulence, Warwick 1980: proceedings of a symposium held at the University of Warwick 1979/80 , pages 366-381. Springer, 1980.

Eugene Tan, Shannon Algar, Débora Corrêa, Michael Small, Thomas Stemler, and David Walker. Selecting embedding delays: An overview of embedding techniques and a new method using persistent homology. Chaos: An Interdisciplinary Journal of Nonlinear Science , 33(3), 2023.

I Vlachos and D Kugiumtzis. State space reconstruction for multivariate time series prediction. arXiv preprint arXiv:0809.2220 , 2008.

Rui Wang, Yihe Dong, Sercan Ö Arik, and Rose Yu. Koopman neural forecaster for time series with temporal distribution shifts. arXiv preprint arXiv:2210.03675 , 2022.

Zihan Wang, Fanheng Kong, Shi Feng, Ming Wang, Han Zhao, Daling Wang, and Yifei Zhang. Is mamba effective for time series forecasting? arXiv preprint arXiv:2403.11144 , 2024.

Matthew O Williams, Ioannis G Kevrekidis, and Clarence W Rowley. A data-driven approximation of the koopman operator: Extending dynamic mode decomposition. Journal of Nonlinear Science , 25:1307-1346, 2015.

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems , 34:22419-22430, 2021.

Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. arXiv preprint arXiv:2210.02186 , 2022.

Rose Yu and Rui Wang. Learning dynamical systems from data: An introduction to physicsguided deep learning. Proceedings of the National Academy of Sciences , 121(27):e2311808121, 2024.

Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of AAAI , volume 37, pages 11121-11128, 2023.

Tian Zhou, Ziqing Ma, Qingsong Wen, Liang Sun, Tao Yao, Wotao Yin, Rong Jin, et al. Film: Frequency improved legendre memory model for long-term time series forecasting. Advances in Neural Information Processing Systems , 35:12677-12690, 2022.

Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International Conference on Machine Learning , pages 27268-27286. PMLR, 2022.

Tian Zhou, Peisong Niu, xue wang, Liang Sun, and Rong Jin. One fits all: Power general time series analysis by pretrained lm. In NeurIPS , 2023.

A Technical Background

A.1 Takens Theorem

Takens' theorem, introduced by Floris Takens in 1981, provides a framework for reconstructing the dynamics of a chaotic system based on a series of observations. The theorem establishes conditions under which the state space of a dynamical system can be reconstructed from time-delay measurements of a single observable. The reconstructed system retains properties that are invariant under smooth transformations, known as diffeomorphisms.

In the context of discrete-time dynamical systems, Takens' theorem is commonly applied. Consider a dynamical system whose state space is a v -dimensional manifold M , with the evolution of the system governed by a smooth map

Assume there exists a strange attractor A ⊂ M with a box-counting dimension d A . According to Whitney's embedding theorem [37], the attractor A can be embedded into an m -dimensional Euclidean space if

This implies that there is a diffeomorphism φ mapping the attractor A into R N , such that the Jacobian matrix of φ is of full rank. In the delay embedding process, the embedding function is constructed using an observation function. This observation function h : M→ R needs to be twice continuously differentiable and must assign a real number to each point on the attractor A , ensuring that it is typical, meaning its derivative is of full rank without exhibiting any special symmetries.

The delay embedding theorem states that the mapping

constitutes an embedding of the strange attractor A into R N .

Now, consider a d -dimensional state vector x t , which evolves according to an unknown but deterministic and continuous dynamic process. Suppose a one-dimensional observable y exists, which is a smooth function of x and is coupled to all the components of x . At any given time, we can observe not only the current measurement y ( t ) but also measurements from past times separated by a lag τ : y t + τ , y t +2 τ , and so on. Using m such lags results in an m -dimensional vector. As the number of lags increases, the motion in the reconstructed space becomes more predictable, and in the limit as m → ∞ , the dynamics could become deterministic. In reality, the deterministic nature of the dynamics is achieved at a finite dimension, with the reconstructed dynamics being equivalent to the original system's dynamics, related by a smooth, invertible change of coordinates (a diffeomorphism). The theorem specifically asserts that deterministic behavior emerges once the dimension reaches 2 d +1 , with the minimal embedding dimension often being lower.

A.2 Hopfield Network

A.2.1 Classical Hopfield Network

A Hopfield network is a form of recurrent artificial neural network with binary neurons. It is characterized by:

Binary neurons with states +1 or -1.

Symmetric weight matrix with zero diagonal (no self-connections).

Energy function that is minimized at stable states.

Asynchronous update of neuron states.

Dynamics

The state of each neuron is updated according to:

Where s i ( t +1) is the state of neuron i at time t +1 , w ij is the weight between neurons i and j , and s j ( t ) is the state of neuron j at time t .

The energy of the network is defined as:

Memory Storage and Retrieval

Memories are stored in the network by adjusting the weights to minimize the network energy, often using the Hebbian learning rule. The network can retrieve memory from a noisy or incomplete version by converging to a stored state. The memory capacity of a Hopfield network depends on several factors, with the most significant one being the number of neurons in the network. John Hopfield proposed a rule in his original paper to estimate the memory capacity, stating that the network can effectively store approximately 0.15N independent memories, where N represents the number of neurons in the network. This means that for a network containing 100 neurons, it can store approximately 15 patterns.

A.2.2 Modern Hopfield Network

In order to integrate Hopfield networks into deep learning architectures, The Modern Hopfield Network allows for continuous state updating. It proposes a new energy function based on the associative memory model [7] and proposes a new update rule that can be proven to converge to stationary points of the energy (local minima or saddle points). Specifically, the new Energy function is:

with lse ( log -sum -exp ) interaction function:

Sering 0E2 M 2 . Using p = softmax ( β X T ξ ) , The novel update rule is:

The new update rule can be viewed as the attention in Transformers. Firstly, N stored (key) patterns y i and S state (query) patterns r i that are mapped to the Hopfield space of dimension d k . Then set x i = W T K y i , ξ i = W T Q r i , and multiply the result of the update rule with W V . The matrices Y = ( y 1 , . . . , y N ) T and R = ( r 1 , . . . , r S ) T combine the y i and r i as row vectors. By defining the matrices X T = K = Y W K , Ξ T = Q = RW Q , and V = Y W K W V = X T W V , where W K ∈ R d y × d k , W Q ∈ R d r × d k , W V ∈ R d k × d v , β = 1 / √ d k and softmax ∈ R N is changed to a row vector, the update rule multiplied by W V is:

In the Hopfield Evolution strategy of Attraos, The Query is settled as the dynamic structures and the Key/Value is settled as trainable vectors.

A.3 Orthogonal Polynomials

Note : In this section, we have selectively extracted key content from the appendix of Hippo [11] that is pertinent to our work, for the convenience of the reader. We take Legendre polynomials as an example.

Under the usual definition of the canonical Legendre polynomial P n , they are orthogonal with respect to the measure ω leg = 1 [ -1 , 1] :

Also, they satisfy

The first equation yields and

With respect to the measure 1 θ I [ t -θ, t ] , the normalized orthogonal polynomials are

In general, the orthonormal basis for any uniform measure consists of (2 n +1) 1 2 times the corresponding linearly shifted version of P n .

Derivatives of Legendre polynomials We note the following recurrence relations on Legendre polynomials:

where the sum stops at P 0 or P 1 . These equations directly imply

To sum up, The Legendre polynomials are in closed-recursive form.

B Proof

Proposition 5. [18] A = diag {-1 , -1 , . . . } is a rough approximation of shifted Hippo-LegT [11] matrix.

Proof. Hippo 3 provides a mathematical framework for deriving the AB matrix for polynomial projection. The mainstream SSMs [10] typically initialize matrix A = diag {-1 , -2 , -3 , -4 , . . . } , representing the negative real diagonal elements of the normalized Hippo-LegS matrix (21). Since the LegS matrix is a mathematical approximation of exponentially decaying Legendre polynomials, initializing A = diag {-1 , -2 , -3 , -4 , . . . } can be seen as a rough approximation of exponential decay. Similarly, for the normalized Hippo-LegT matrix (22), which approximates a uniform measure, we can consider A = diag {-1 , -1 , . . . } as a rough approximation of a finite window of Legendre polynomials, which better for non-stationary time series as well [19]. The use of negative values for the elements is to ensure gradient stability during training.

Theorem 6. (Approximation Error Bound) Suppose that the function f : [0 , 1] ∈ R is k times continuously differentiable, the piecewise polynomial g approximates f with mean error bounded as follows:

Proof. Similar to [1], we divide the interval [0 , 1] into subintervals on which g is a polynomial; the restriction of g to one such subinterval I r,l is the polynomial of degree less than k that approximates f with minimum mean error. Also, the optimal g can be regarded as the orthonormal projection Q N r f onto G N ( r ) . We then use the maximum error estimate for the polynomial, which interpolates f at Chebyshev nodes of order k on I r,l . We define I r,l = [2 -r l, 2 -r ( l +1)] for l = 0 , 1 , . . . , 2 r -1 , and obtain

and by taking square roots we have bound (7). Here C N r,l f denotes the polynomial of degree k which agrees with f at the Chebyshev nodes of order k on I r,l , and we have used the well-known maximum error bound for Chebyshev interpolation.

The error of the approximation Q N r f of f therefore decays like 2 -rk and, since S N r has a basis of 2 r k elements, we have convergence of order k . For the generalization to m dimensions in the dynamic structure modeling, a similar argument shows that the rate of convergence is of order k/m .

Theorem 7. (Evolution Error Bound) By Jacobian value, the mean attractor evolution error ∥ ∥ ∥ K◦ ˜ A-˜ A ∥ ∥ ∥ of evolution operator K = {K ( i ) } is bounded by

with the number of random patterns N ≥ √ pc d -1 4 stored in the system by interaction paradigm E in an ideal spherical retrieval paradigm.

Proof. Due to the numerous assumptions and extensive lemmas involved in the proof of this theorem, we will provide a brief exposition of its main ideas. For a detailed proof, please refer to [38].

Firstly, the theorem defines the matching between patterns as:

Definition 1. (Pattern match). Assuming that around every pattern x i a sphere S i is given. We say x i is matched with ξ if there is a single fixed point x ∗ i ∈ S i to which all points ξ ∈ S i converge.

As shown in Theorem 3 in [38], according to the upper branch of the Lambert W function [35], we can obtain the number of random patterns stored in a system is N ≥ √ pc d -1 4 .

In our paper, we define the pattern1 as the Attractor ˜ A in phase space, pattern2 as the future state evaluated by operator K , noted as K◦ ˜ A . From Lemma A4 in [38], when the radius of the pattern matching sphere is M , we can describe the evolution process by jacobian value and get the matching error as:

where in the Equation (179) of [38]:

The Equation (404) of [38] says ∥ ∥ ∥ ˜ A ∗ i -˜ A i ∥ ∥ ∥ 1 2 βM and ∥ ∥ ∥ K◦ ˜ A-˜ A ∥ ∥ ∥ 1 2 βM , so we can get: ϵe ( N -1) M exp( -β ∇ i ) .

In our paper, we replace the exponential interaction function with an unknown function E ( · ) to finally get the version in Theorem 3.

Theorem 8. (Completeness in L 2 space) The orthonormal system B k = { ϕ j : j = 1 , . . . , k }∪{ ψ rl j : j = 1 , . . . , k ; r = 0 , 1 , 2 , . . . ; l = 0 , . . . , 2 r -1 } spans L 2 [0 , 1] .

Proof. We define the space G N to be the union of the G N ( r ) , given by the formula:

and observe that G N = L 2 [0 , 1] . In particular, G N contains the Haar basis for L 2 [0 , 1] , consisting of functions piecewise constant on each of the subintervals (2 -r l, 2 -r ( l +1)) . Here the closure G N is defined with respect to the L 2 -norm,

where the inner product ⟨ f, g ⟩ is defined by the formula

Also, we have:

and

We let { ϕ 1 , . . . , ϕ k } denote an orthonormal basis for G N 0 ; in view of Equation 23, 24, and 25, the orthonormal system

spans L 2 [0 , 1] .

Now we construct a basis for L 2 ( R ) by defining, for r ∈ Z , the space ˜ G N r by the formula ˜ G N r = { f : the restriction of f to the interval (2 -r n, 2 -r ( n +1)) is a polynomial of degree less than k , for n ∈ Z } and observing that the space ˜ G N r +1 \ ˜ G N r is spanned by the orthonormal set

Thus L 2 ( R ) , which is contained in ⋃ r ˜ G r N , has an orthonormal basis

C Model Details

C.1 Phase Space Reconstruction

Phase space reconstruction is a crucial technique in the analysis of dynamical systems, particularly in the study of time series data. This method transforms a one-dimensional time series into a multidimensional phase space, revealing the underlying dynamics of the system. The cross-correlation mutual

information (CC mutual information) method is a statistical tool used to analyze the dependencies between two different time series. We provide a detailed mathematical description of these methods.

Phase space reconstruction involves transforming a single-variable time series into a multidimensional space to unveil the dynamics of the system that generated the data. The technique is based on Takens' Embedding Theorem.

Takens' Embedding Theorem

Takens' Embedding Theorem allows the reconstruction of a dynamical system's phase space from a sequence of observations. The theorem states that under generic conditions, a map:

can be constructed, where n is the dimension of the original phase space, and m is the embedding dimension, typically m< = 2 n +1 is sufficient to recover dynamics.

Time Delay Embedding

The most common approach for phase space reconstruction is the time delay embedding method. Given a time series { x ( t ) } , the reconstructed phase space is:

where τ is the time delay, and m is the embedding dimension. In the context of coordinate delay reconstruction in a complete scenario, it involves a parameter called time window, which selectively uses discrete one-dimensional data for phase space reconstruction. In order to maximize the preservation of time series information, the time window parameter is typically set to 1. As a result, this process is reversible, meaning that the original data can be reconstructed without loss of information.

The CC mutual information method is used to analyze the dependencies between two time series. It is a measure of the amount of information obtained about one time series through the other.

Mutual Information

Mutual information I ( X ; Y ) between two random variables X and Y is defined as:

where p ( x, y ) is the joint probability distribution function of X and Y , and p ( x ) and p ( y ) are the marginal probability distribution functions.

Cross-Correlation Mutual Information

For time series analysis, the mutual information is extended to account for the time-lagged relationships:

where τ is the time lag, and y ( τ ) represents the time series Y shifted by τ .

Determining Time Delay ( τ )

Time delay ( τ ) is an interval used in reconstructing the phase space of a time series. The right choice of τ is essential for revealing the dynamic properties of the system.

Calculating Mutual Information

For a given time series { x t } , calculate the mutual information between the time series and its time-shifted version for different delays τ :

where p ( x t , x t + τ ) is the joint probability distribution, and p ( x t ) and p ( x t + τ ) are the marginal probability distributions.

Selecting Time Delay

Plot the mutual information I ( τ ) against τ . Choose the τ at the first local minimum of this plot. This represents the delay where the series' points provide maximal mutual information.

Determining Embedding Dimension ( m )

Embedding dimension ( m ) is the dimension of the reconstructed phase space. The correct m ensures that trajectories in the phase space do not intersect each other.

Calculating False Nearest Neighbors

For each dimension m , calculate the false nearest neighbors (FNN) F ( m ) , which reflects the complexity of the trajectories reconstructed in m -dimensional space:

where C i ( m,r ) is the count of points within a distance r from point i in m -dimensional space, and N is the length of the time series.

Identifying the Saturation Point

As m increases, F ( m ) typically increases and reaches a saturation point. Choose the smallest m for which F ( m ) does not significantly increase, indicating that increasing the dimension does not reveal more information about the dynamics.

D Experiments Details

D.1 Datasets

Our experiments are carried out on Five real-world datasets and two chaos datasets as described below:

ETT 2 dataset are procured from two electricity substations over two years. They provide a summary of load and oil temperature data across seven variables. For ETTm1 and ETTm2, the "m" signifies that data was recorded every 15 minutes, yielding 69,680 time steps. ETTh1 and ETTh2 represent the hourly equivalents of ETTm1 and ETTm2, each containing 17,420 time steps.

Exchange 3 dataset details the daily foreign exchange rates of eight countries, including Australia, British, Canada, Switzerland, China, Japan, New Zealand, and Singapore from 1990 to 2016.

Weather 4 dataset is a meteorological collection featuring 21 variates, gathered in the United States over a four-year span.

Lorenz96 dataset is an artificial dataset. We simulate the data of 30,000 time steps with an initial dimension of 40d according to the Lorenz96 equation and map to 3D by a randomly initialized Linear network to simulate the realistic chaotic time series generated from an unknown underlying chaotic system. More details are in Appendix E.3.

Crypots comprises historical transaction data for various cryptocurrencies, including Bitcoin and Ethereum. We select samples with Asset _ ID set to 0, and remove the column Count . We download the data from https://www.kaggle.com/competitions/ g-research-crypto-forecasting/data?select=supplemental_train.csv .

Air Convection 5 dataset is obtained by scraping the data from NOAA, and it includes the data for the entire year of 2023. The dataset consists of 20 variables, including air humidity, pressure, convection characteristics, and others. The data was sampled at intervals of 15 minutes and averaged over the course of the entire year.

Lorenz96-3d : See Appendix E.3.

D.2 Baselines

Our baseline models include: Mamba4TS: Mamba4TS is a novel SSM architecture tailored for TSF tasks, featuring a parallel scan (https://github.com/alxndrTL/mamba.py/tree/main). Additionally, this model adopts a patching operation with both patch length and stride set to 16. We use the recommended configuration as our experimental settings with a batch size of 32, and the learning rate is 0.0001.

S-Mamba [47]: S-Mamba utilizes a linear tokenization of variates and a bidirectional Mamba layer to efficiently capture inter-variate correlations and temporal dependencies. This approach underscores its potential as a scalable alternative to Transformer technologies in TSF. We download the source code from: https://github.com/wzhwzhwzh0921/S-D-Mamba and adopt the recommended setting as its experimental configuration.

RWKV-TS [17]: RWKV-TS is an innovative RNN-based architecture for TSF that offers linear time and memory efficiency. We download the source code from: https://github.com/howard-hou/RWKVTS. We follow the recommended settings as experimental configuration.

Koopa [32]: Koopa is a novel forecasting model that tackles non-stationary time series using Koopman theory to differentiate time-variant dynamics. It features a Fourier Filter and Koopman Predictors within a stackable block architecture, optimizing hierarchical dynamics learning. We download the source code from: https://github.com/thuml/Koopa. We set the lookback window to fixed values of {96, 192, 336, 720} instead of twice the output length as in the original experimental settings.

iTransformer [31]: iTransformer modifies traditional Transformer models for time series forecasting by inverting dimensions and applying attention and feed-forward networks across variate tokens. We download the source code from: https://github.com/thuml/iTransformer. We follow the recommended settings as experimental configuration.

PatchTST [33]: PatchTST introduces a novel design for Transformer-based models tailored to time series forecasting. It incorporates two essential components: patching and channel-independent structure. We obtain the source code from: https://github.com/PatchTST. This code serves as our baseline for long-term forecasting, and we follow the recommended settings for our experiments.

LTSF-Linear [52]: In LTSF-Linear family, DLinear decomposes raw data into trend and seasonal components and NLinear is just a single linear models to capture temporal relationships between input and output sequences. We obtain the source code from: https://github.com/cure-lab/LTSF-Linear, using it as our long-term forecasting baseline and adhering to recommended settings for experimental configuration.

GPT4TS [55]: This study explores the application of pre-trained language models to time series analysis tasks, demonstrating that the Frozen Pretrained Transformer (FPT), without modifications to its core architecture, achieves state-of-the-art results across various tasks. We download the source code from: https://github.com/DAMO-DI-ML/NeurIPS2023-One-Fits-All. We follow the recommended settings as experimental configuration.

D.3 Experiments Setting

All experiments are conducted on the NVIDIA RTX3090-24G and A6000-48G GPUs. The Adam optimizer is chosen. A grid search is performed to determine the optimal hyperparameters, including the learning rate from {0.0001, 0.0005, 0.001}, model layer from {1, 2} (typically 1), polynomial order from {32, 64, 256}, patch length from {8, 16}, projection level based on logL and no more than 3, primary modes is 16, and the input length is 96. Due to the sensitivity of the CC method to numerical calculations, we take the average result from three iterations of the calculation.

Original Time Series

Phase Space

Original Time Series

Phase Space

Original Time Series

Phase Space

200

400

Time

600

008

1000

200

400

Time

600

800

1000

200

400

Time

600

800

1000

Original Time Series

Phase Space

Original Time Series

Phase Space

2.5

Original Time Series

Phase Space

2.0

1.5

2

-1

0

0.5

0.0

1.0

0.5

E-

1.0

1.5

D

200

400

Time

600

008

1000

0

200

400

Time

600

800

1000

0

200

400

Time

600

800

1000

E Supplemental Experiments

E.1 Dynamic Structures of Real-world Data

As shown in Figure 6, we present the phase space structures of various real-time series. Please note that due to our visualization limitations in three dimensions, the shapes of many attractors for these time series are manifested in higher dimensions. We can only display slices of the attractors in the first three dimensions.

E.2 The Chaotic Nature of Datasets

Lyaponuv Exponent

Lyapunov exponents are a set of quantities that characterize the rate of separation of infinitesimally close trajectories in a dynamical system. They play a crucial role in understanding the stability and chaotic behavior of the system [24]. The formal definition of the Lyapunov exponent for a trajectory of a dynamical system is given by:

where δ X ( t ) is the deviation vector at time t , and λ is the Lyapunov exponent. The set of Lyapunov exponents for a system provides a spectrum, indicating the behavior of trajectories in each dimension of the system's phase space.

When the direction of the initial separation vector is different, the separation rate is also different. Thus, the spectrum of Lyapunov exponents exists, which have the same number of dimensions as the phase space. The largest of these is often referred to as the Maximal Lyapunov exponent (MLE). We reconstruct dimension m according to different phase spaces to get the maximum Lyapunov index. A positive MLE is often considered an indicator of chaotic behavior in the system, while a negative value indicates convergence, implying stability in the system's behavior.

As shown in Table 6, we calculate the maximum Lyapunov exponent for all mainstream LTSF datasets. Surprisingly, we find that their MLEs are all positive, indicating the presence of chaos to varying degrees in these datasets. This directly supports the motivation proposed by Attraos. Among them, the weather dataset exhibits the strongest chaotic behavior. Note that the existence of at least one positive Lyapunov exponent is sufficient to determine the presence of chaos. For example, the classical Lorenz63 system has three Lyapunov exponents with values negative, zero, and positive respectively.

E.3 Simulation for Lorenz96 (Case Study)

Lorenz63 System

The Lorenz63 system, introduced by Edward Lorenz in 1963, is a simplified mathematical model for atmospheric convection. The model is a system of three ordinary differential equations now known as the Lorenz equations:

Here, x , y , and z make up the state of the system. The parameters σ , ρ , and β represent the Prandtl number, Rayleigh number, and certain physical dimensions of the layer, respectively. Lorenz derived these equations to model the way air moves around in the atmosphere. This model is famous for exhibiting chaotic behavior for certain parameter values and initial conditions.

Lorenz96 System

The Lorenz96 system is a mathematical model that was introduced by Edward N. Lorenz in 1996 as an extension of the original Lorenz model. It is commonly used to study chaotic dynamics in systems with multiple interacting variables.

The Lorenz96 system consists of a set of ordinary differential equations that describe the time evolution of a set of variables. In its simplest form, the system is defined as follows:

Here, x i represents the state variable at position i , and F is a forcing term that controls the overall behavior of the system. The nonlinear term ( x i +1 -x i -2 ) x i -1 captures the interactions between neighboring variables, leading to the emergence of chaotic behavior. The Lorenz96 system exhibits a range of fascinating phenomena, including intermittent chaos, phase transitions, and the presence of multiple stable and unstable regimes. Its dynamics have been extensively studied to gain insights into the behavior of complex systems and to explore the limits of predictability.

100

80

50

60

30

40

20

30

40

uojsuawia

40

20

10

dno

20

0

10

20

-10

0

40

20

20

20

10

0

10

20

-20

-10

0

10

=40

20

Dimension 1

0

20

40

09

-40

20

1.

40

60

0

200

400

Time Steps

600

800

Dimension 1

Dimension 2

Dimension 3

1000

(a)Lorenz63

(b)Lorenz96-40d mappingto3d

(c)1dvisualization

To simulate time series generated from an unknown chaotic system, we first construct a 40dimensional Lorenz96 system to represent the underlying chaotic system. We then use a randomly initialized linear neural network to simulate the observation function h . Through h , we map the Lorenz96-40d system, which resides in the manifold space, to a 3-dimensional Euclidean space to obtain a multivariate time series dataset with three variables. As shown in the middle of Figure 7, in the observation domain, the dynamical system structure of Lorenz96-40d is difficult to discern specific shapes, so we need to reconstruct it to 40 dimensions using the Phase Space Reconstruction (PSR) technique to study its dynamic characteristics in a topologically equivalent structure. It is worth noting that on the right side of Figure 7, we visualize the last variable of the system and find striking similarities with real-world time series, even exhibiting some periodic behaviors. This further supports our hypothesis that real-world time series are generated by underlying chaotic systems.

E.4 Chaotic Modulation

1.4

0.4

Hippo-RNN

Hippo-RNN

Hippo-GTF

Hippo-GTF

1.2

0.3

1.0

0.2

0.8

0.6

0.1

96

192

336

720

96

192

336

720

In LTSF tasks, it is commonly observed that predictive performance deteriorates as the length of the forecasting window increases. This phenomenon bears a striking resemblance to the inherent challenge in long-term predictions of chaotic dynamical systems, which are highly sensitive to initial conditions. Recent studies [15] in the field of chaotic dynamical systems have highlighted that, to address the issue of gradient divergence caused by positive maximal Lyapunov exponents indicative of chaos, the implementation of teaching forcing as a method to incrementally constrain the trajectory of the dynamical system presents a straightforward yet effective framework. To enhance Hippo-RNN, we have implemented the following modifications: ˜ z t := (1 -α ) z t + α z t , z t = F θ (˜ z t -1 ) , with 0 ≤ α ≤ 1 , where we intervene the evolution state z by utilizing the ground truth hidden state z . As shown in Figure 8, this modification leads to notable improvements in both mainstream LTSF datasets and real-world chaotic datasets. However, the autoregressive nature of the teaching forcing methods introduces additional computational overhead and may potentially reduce its generalization capabilities, which presents a challenge for integrating it into sequence-mapping models.

E.5 Full Results

As shown in Table 7, we showcase the full experiment results for all mainstream LTSF datasets.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes] .

Justification: Attraos, considers time series as generated by a generalized chaotic dynamic system. By leveraging the invariance of attractors, Attraos enhances time series prediction performance and provides empirical and theoretical explainations.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes] .

Justification: In Section 5, we discuss the limitations of this work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes] .

Justification: All the proofs in this paper can be found in Appendix B.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes] .

Justification: We have open-sourced our code in https://anonymous.4open.science/ r/Attraos-40C2/ and provided detailed experimental settings in Appendix D to facilitate reproducibility.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes] .

Justification: We have open-sourced our code and experimental settings in https:// anonymous.4open.science/r/Attraos-40C2/ to facilitate reproducibility.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes] .

Justification: We have provided detailed experimental settings in Appendix D to facilitate reproducibility.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No] .

Justification: In time series forecasting (TSF) tasks, it is common not to provide error bars but instead directly calculate the average by conducting multiple experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes] .

Justification: We provide experiment setting in Appendix D and complexity analysis in Section 4.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes] .

Justification: We follow the NeurIPS Code of Ethics in this paper.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes] .

Justification: In Conclusion, we state that our goal is to offer the machine-learning community a fresh perspective and inspire further research on the essence of time series dynamics.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA] .

Justification: The datasets chosen in this paper are commonly used benchmark datasets for time series forecasting tasks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes] .

Justification: Yes, we have.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes] .

Justification: This paper follows CC 4.0, and the code is in an anonymized URL.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA] .

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA] .

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Attraos, a new model for long-term time series forecasting (LTSF) that incorporates chaos theory and views time series data as observations from high-dimensional chaotic dynamic systems. Attraos utilizes attractor invariance, non-parametric Phase Space Reconstruction, and a multi-scale dynamic memory unit to effectively capture historical dynamics and forecast future states with substantially fewer parameters than existing models such as PatchTST. Empirical evidence demonstrates Attraos' superior performance across various LTSF and chaotic datasets, providing a new perspective on the underlying dynamics of time series data.\n\nSTRENGTHS:\n-\tThe paper presents comprehensive experiment results and provide rigorous theoretical results to support their claims.\n-\tLeveraging results from chaotic theories for LTSF tasks is a novel and interesting idea.\n-\tThe method proposed in the paper needs much fewer parameters than previous methods.\n\nWEAKNESSES:\n-\tMost of the descriptions for problem setup (section 2), methods, and theoretical analysis (section 3) are very confusing and hard to follow. There are always notations suddenly coming out without much information. I suspect there are also several typos that cause difficulties in understanding and assessments. See **Questions** for details.\n-\tIf I understood correctly, the title is a little misleading since seemingly the key insight/perspective comes from the embedding theorem by Takens, a counterpart of Whitney theorem for attractors. Are there any other insights related to chaotic systems? If not, I would suggest concretizing this point in the title, abstract, introduction, and conclusion.\n-\tSection A.1 in appendix is entirely copied from Wikipedia, https://en.wikipedia.org/wiki/Takens%27s_theorem , except for some changes on notations. However the notations in this section are not coherent, e.g. $k$ and $N$ in L414.\n-\tSimilar to the first point, the paper does not provide enough background introduction.\n\nIf the authors worry containing all the necessary details will exceed the page limit, I would suggest presenting a detailed version of problem setup, backgrounds, analysis and methods in appendix.\n\nQUESTIONS:\n1.\tL90 Is it a weighted integral $\\mu(ds)$ instead of $ds$? For polynomials $\\phi$, the integral with Lebesgue measure does not converge. \n2.\tIs $\\mu$ same as $\\omega$ in L94?\n3.\tIsn’t L94 giving the formula of projection to polynomial subspace and isn’t $K_n$ the kernel? How are they connected to $e^{tA}B$, where $A, B$ only appear in the induced dynamics of $x(t)$?\n4.\tDoes $\\mathcal{A}_i$ refer that there are multiple attractors in the dynamics? $\\mathcal{A}$ are sets, then what does transpose in L158 stand for?\n5.\tL112, 120, if $A$ is a (D,N) tensor, $B$ is a (B,L,N) tensor and $u$ is a $(B,L,D)$ tensor, how is $Ax(t)+Bu(t)$ in (2a) defined? \n6.\tCould the authors give more interpretation on what they intend to do $\\Delta$ and $\\theta$? I have no idea what the authors mean by ‘$\\Delta$ is similar to an attention mechanism’ and how this is related to the previous context.\n7.\tAgain, given the shape of $B$ and $\\Delta$, it is confusing what $\\Delta B$ refers to.\n8.\tHow is $H$ in L144 related to previous text?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces chaos theory into a long-term time series forcasting (LTSF) model called Attraos (a play on the words \"attractor\" and \"chaos\"). They propose a Multi-resolution Dynamic Memory Unit (MDMU) which is inspired by (and looks a lot like) the State Space Models (SSMs) used in the Mamba family of models. However, unlike SSMs, Attraos assumes additional structure on top of time-series signals by utilizing attractor behavior from chaos theory in the phase space. Attraos outperforms many existing architectures (e.g., Mamba, RWKV-TS, PatchTST) on time-series tasks at a fraction of the computational complexity and cost.\n\nSTRENGTHS:\n- S1: **Outperforms existing architectures at a fraction of the cost**. Attraos is able to outperform models like Mamba, RWKV, and Transformers on many different time series tasks at a fraction of the computational cost (in terms of training time and parameter count).\n- S2: **Theoretically complete**. The paper provide rigorous definitions, theorems, and proofs throughout the main paper and appendix to justify their method. (Note: before this paper I was quite unfamiliar with chaos theory, though I have some experience in SSMs and time-series. I am unable to verify much of the math.)\n- S3: **Comprehensive empirical results**. The paper compares Attraos to many alternative architectures for time-series data (e.g., RWKV, Mamba, Transformers, MLPs, etc.) and shows that on many tasks Attraos outperforms alternative methods at a fraction of the computational cost. Additionally, the paper includes an ablation study for the different components of Attraos.\n\nWEAKNESSES:\n- W1: **Presentation of results could be clearer**. I was often confused by the choice of coloring in the tables. See the following for my suggestions:\n    1. [Table 1,2,4]. The \"Red/blue\" colorscheme is unfortunate, because most brains (including mine) associate the color \"red\" with \"poor performance\". I suggest using **bold** for best, <u>underline</u> for second best, as is often done in AI papers. \n    2. [Table 3,5]. For \"improvement/decline\" in performance, I suggest one of two options: (1) show \"better performance\" with blue, and \"worse performance\" with bold and red to emphasize decreased performance from an ablation, or (2) showing only the \"deltas\" with a \"stock ticker arrow\" (🔺 or 🔻, colored green for improvement, red for decline).\n- W2: **Complete architectural description is missing**. Most of the paper focuses on describing how to implement the special components of Attraos, but the complete picture of the architecture is missing. Additionally, there is no section describing the hyperparameters for the training setup for each experiment. See Q1 for specific questions.\n- W3: **Missing error bars**. Table 1 represents average results of long term forecasting across a swath of model classes, but these averages are not accompanied by reports of standard deviation. Thus, it is difficult to tell whether improved average performance is actually signficant. Improve by including error bars for all numerical results, which may require re-running some experiments under different random seeds.\n- W4: **Grandiose, non-academic language**. The paper starts with \"In the intricate dance of time\" [L15] 😂 and makes statements like \"we can transcend the limitations of deterministic dynamical systems\" [L34]. I suggest reworking these lines to preserve the professionalism of the rest of the work.\n\nThese weakenesses are admittedly small and easily fixable by the authors during the review process. My overall attitude towards the paper is that it is of high quality and should be accepted. However, a lot of the theory was beyond my ability to evaluate, and I am willing to adjust my score positively/negatively as I become more familiar with this work throughout the review process.\n\nQUESTIONS:\n- Q1: The complete architecture for Attraos is not described. Do you just take Mamba and modify the SSM component? What is the complete set of learnable parameters? Where does the Hopfield Network come into play and how are its memories trained? What are the training hyperparameters for each experiment?\n\nLIMITATIONS:\nLimitations are adequately described in the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel approach, named Attraos, for Long-term Time Series Forecasting (LTSF) based on treating the observed time series as high dimensional chaotic dynamical system. The model first estimates an embedding of the data through Phase Space Reconstruction (Takens embedding) and then utilizes a memory unit through state space model that represents the dynamical system through polynomials which allows to evolve the dynamics into future steps and forecast the data. The model is evaluated on multiple LTSF datasets and additional experiments such as ablations and robustness under noise are performed.\n\nSTRENGTHS:\nS1. The approach is proposing to forecast data evolution through learning the dynamical system that can generate such time series vs. to predict directly from data.\n\nS2. Novel approach for polynomial estimation through a state space model is proposed.\n \nS3. Fundamental properties of the approach are rigorously shown.\n\nS4. Incorporation of Belloch algorithm for speed up is used.\n\nS5. Experiments show that the approach performs better in many cases than existing approaches and ablation experiments are performed.\n\nWEAKNESSES:\nW1. There seems to be a detachment between the theorems and properties proved and the proposed system. There seems to be no discussion about how these properties lead to the particular setup in the paper. For example why this particular SSM was used? How the hyperparameters were chosen in light of the propositions? What are the cases that the model is limited and not expected to be effective? Also interpretation of the results is lacking.\n\nW2. For non toy experimental results exposition and interpretation of the resulting dynamical system is missing. What is the dynamical system/s that are being obtained for the benchmark and what is the distribution of polynomials?\n\nQUESTIONS:\nSee questions in W1 and W2 and also:\n\nQ1. The Phase Space Reduction approach seems to be the well known Takens embedding and authors mention Takens thm, however call the method differently. Is there something different that I'm missing in PSR that is not in the classical computation of Takens embedding?\n\nQ2. With local evolution I assumed that authors mean single step \"autoregressive\" forecasting. Is that the case? How would that change for multi-step?\n\nQ3. It would be informative if the benefit/limitation/motivation of frequency evol vs. direct evol be more elaborately explained.\n\nLIMITATIONS:\nSee W1-W2 and Q1-Q3. More extended discussion of limitations would contribute to better evaluation of the work vs generalization.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-24"}