Title: Koopa: Learning Non-stationary Time Series Dynamics with Koopman Predictors
Venue: NeurIPS 2023
Accepted: yes
Keywords: Time series forecasting

Deep learning
Abstract: Real-world time series are characterized by intrinsic non-stationarity that poses a principal challenge for deep forecasting models. While previous models suffer from complicated series variations induced by changing temporal distribution, we tackle non-stationary time series with modern Koopman theory that fundamentally considers the underlying time-variant dynamics. Inspired by Koopman theory of portraying complex dynamical systems, we disentangle time-variant and time-invariant components from intricate non-stationary series by Fourier Filter and design Koopman Predictor to advance respective dynamics forward. Technically, we propose Koopa as a novel Koopman forecaster composed of stackable blocks that learn hierarchical dynamics. Koopa seeks measurement functions for Koopman embedding and utilizes Koopman operators as linear portraits of implicit transition. To cope with time-variant dynamics that exhibits strong locality, Koopa calculates context-aware operators in the temporal neighborhood and is able to utilize incoming ground truth to scale up forecast horizon. Besides, by integrating Koopman Predictors into deep residual structure, we ravel out the binding reconstruction loss in previous Koopman forecasters and achieve end-to-end forecasting objective optimization. Compared with the state-of-the-art model, Koopa achieves competitive performance while saving 77.3% training time and 76.0% memory.

Koopa: Learning Non-stationary Time Series Dynamics with Koopman Predictors

Abstract

Real-world time series are characterized by intrinsic non-stationarity that poses a principal challenge for deep forecasting models. While previous models suffer from complicated series variations induced by changing temporal distribution, we tackle non-stationary time series with modern Koopman theory that fundamentally considers the underlying time-variant dynamics. Inspired by Koopman theory that portrays complex dynamical systems, we disentangle time-variant and timeinvariant components from intricate non-stationary series by Fourier Filter and design Koopman Predictor to advance respective dynamics forward. Technically, we propose Koopa as a novel Koop man forec a ster composed of stackable blocks that learn hierarchical dynamics. Koopa seeks measurement functions for Koopman embedding and utilizes Koopman operators as linear portraits of implicit transition. To cope with time-variant dynamics that exhibits strong locality, Koopa calculates context-aware operators in the temporal neighborhood and is able to utilize incoming ground truth to scale up forecast horizon. Besides, by integrating Koopman Predictors into deep residual structure, we ravel out the binding reconstruction loss in previous Koopman forecasters and achieve end-to-end forecasting objective optimization. Compared with the state-of-the-art model, Koopa achieves competitive performance while saving 77 . 3% training time and 76 . 0% memory. Code is available at this repository: https://github.com/thuml/Koopa .

1 Introduction

Time series forecasting has become an essential part of real-world applications, such as weather forecasting, energy consumption, and financial assessment. With numerous available observations, deep learning approaches exhibit superior performance and bring the boom of deep forecasting models. TCNs [4, 43, 47] utilize convolutional kernels and RNNs [12, 22, 37] leverage the recurrent structure to capture underlying temporal patterns. Afterward, attention mechanism [42] becomes the mainstream of sequence modeling and Transformers [31, 48, 53] show great predictive power with the capability of learning point-wise temporal dependencies. And the recent revival of MLPs [32, 51, 52] presents a simple but effective approach to exhibit temporal dependencies by dense weighting.

In spite of elaboratively designed models, it is a fundamental problem for deep models to generalize on varying distribution [1, 25, 33], which is widely reflected in real-world time series because of inherent non-stationarity. Non-stationary time series is characterized by time-variant statistics and temporal dependencies in different periods [2, 14], inducing a huge distribution gap between training and inference and even among each lookback window. While previous methods [16, 28] tailor existing architectural design to attenuate the adverse effect of non-stationarity, few works research on the theoretical basis that can be applied to deal with time-variant temporal patterns naturally.

<latexi sh

1_b

64="

ZGTV9HWj

F

m

nwRgY

>A

C0

S

N

D

r/qu

Q

I

y

U

+

B

5

zE

f

J

7

8

p

X

d

P

M

O

3

v

<latexi sh

1_b

64="XOnC

jqm/W

A

u8

ED2

vrRwM

>

H

c

V

LS

N

F

pf

Zd gk

I

U

G

9

J

0

+

P

5B

T

z

o

y

K

3

Y

Q

7

<latexi sh

1_b

64="Gu2

kpACW

M5

/f

7J

y

zv3w

>

0H

cjV

LS

N

FD

q

nQT

IKr

o

Y

U

m

O

X

8

B

PR

+

d

Z

9

E

g

<latexi sh

1_b

64="qn7P

urUI5

CY

ZL

M

c

8

A

>

yX

jVH

S

N

FD2

pf

d

gk

w

0

E

G

R

m

W

J

z

T

/

3+

Q

9

B

K

o

O

v

<latexi sh

1_b

64="v

W

7FgJnGM

N2QZB

L5

O

c

>A

C0H

jV

S

D

/q

u

T

IKrk

o

9

E

p

d

R

P+

8

z

3

U

f

m

y

X

Y

w

<latexi sh

1_b

64="f5w0

v

p9

M

Z

HCjrN

D

F

Q

>A

c

V

LS

2

dugk

RI

U

G

J

O

zE

3

q

X+

y

m

W

Y

P

o

K

/

n

B

T

8

7

<latexi sh

1_b

64="

IXm cyBqMvNYA

rwEu

UPV

W

Q

>

C

H

j

LS

FD2

pf

Zd gk

R

G

9

J

O

0

+

8

5

T

z

7

/

n

K

3

o

<latexi sh

1_b

64="wg0

WP5LqFk2T

v+

9

ZMER

>A

C

3

cjVH

S

N

D

r

om

KI

G

J

Y

B

Q

8

/

O

X

7

U

n

y

z

<latexi sh

1_b

64="v

W

7FgJnGM

N2QZB

L5

O

c

>A

C0H

jV

S

D

/q

u

T

IKrk

o

9

E

p

d

R

P+

8

z

3

U

f

m

y

X

Y

w

<latexi sh

1_b

64="Gu2

kpACW

M5

/f

7J

y

zv3w

>

0H

cjV

LS

N

FD

q

nQT

IKr

o

Y

U

m

O

X

8

B

PR

+

d

Z

9

E

g

<latexi sh

1_b

64="

ZGTV9HWj

F

2L

m

nwRgY

>A

C0

S

N

D

r/qu

Q

I

y

U

+

B

5

zE

f

J

7

8

p

X

d

P

M

O

3

v

K

Non-stationary Time series

Value

Time

Dynamics

K

1

K

2

K

3

g

g

-

1

Measurement

Function

Period 1

Period 2

Period 3

g

(

x

)

x

K

1

K

2

3

t

t

Linear System

Nonlinear System

From another perspective, real-world time series acts like time-variant dynamics [6]. As one of the principal approaches to analyze complex dynamics, Koopman theory [20] provides an enlightenment to transform nonlinear system into measurement function space, which can be described by a linear Koopman operator. Several pilot works accomplish the integration with deep learning approaches by employing autoencoder networks [40] and operator-learning [26, 49]. More importantly, it is supported by Koopman theory that for time-variant dynamics, there exists a coordinate transformation of the system, where localized Koopman operators are valid to describe the whole measurement function space into several subspaces with linearization [23, 36]. Therefore, Koopman-based methods are appropriate to learn non-stationary time series dynamics (Figure 1). Besides, the linearity of measurement function space enables us to utilize spectral analysis to interpret nonlinear systems.

In this paper, we disentangle non-stationary series into time-invariant and time-variant dynamics and propose Koopa as a novel Koop man forec a ster, which is composed of modular K oopman P redictors ( KP ) to hierarchically describe and advance forward series dynamics. Concretely, we utilize Fourier analysis for dynamics disentangling. And for time-invariant dynamics, the model learns Koopman embedding and linear operators to reveal the implicit transition underlying long-term series. As for the remaining time-variant components that exhibit strong locality, Koopa performs context-aware operator calculation and adaptation within different lookback windows. Besides, Koopman Predictor goes beyond the canonical design of Koopman Autoencoder without the binding reconstruction loss, and we incorporate modular blocks into deep residual architecture [32] to realize end-to-end time series forecasting. Our contributions are summarized as follows:

From the perspective of modern dynamics Koopman theory, we propose Koopa composed of modular Fourier Filter and Koopman Predictor , which can hierarchically disentangle and exploit time-invariant and time-variant dynamics for time series forecasting.

Based on the linearity of Koopman operators, the proposed model is able to utilize incoming series and adapt to varying dynamics for scaling up forecast horizon.

Compared with state-of-the-art methods, our model achieves competitive performance while saving 77 . 3% training time and 76 . 0% memory averaged from six real-world benchmarks.

2 Related Work

2.1 Time Series Forecasting with DNNs

Deep neural networks (DNNs) have made great breakthroughs in time series forecasting. TCN-based models [4, 43, 47] explore hierarchical temporal patterns and adopt shared convolutional kernels with diverse receptive fields. RNN-based models [12, 22, 37] utilize the recurrent structure with memory to reveal the implicit transition over time points. MLP-based models [32, 51, 52] learn point-wise weighting and the impressive performance and efficiency highlight that MLP performs well for modeling simple temporal dependencies. However, their practical applicability may still be constrained on non-stationary time series, which is endowed with time-variant properties and poses challenges for model capacity and efficiency. Unlike previous methods, Koopa fundamentally considers the complicated dynamics underlying time series and implements efficient and interpretable transition learners in both time-variant and time-invariant manners inspired by Koopman theory.

Recently, Transformer-based models have also achieved great success in time series forecasting. Initial attempts [19, 27, 48, 53] renovate the canonical structure and reduce the quadratic complexity

<latexi sh

1_b

64="

IXm cyBqMvNYA

rwEu

UPV

W

Q

>

C

H

j

LS

FD2

pf

Zd gk

R

G

9

J

O

0

+

8

5

T

z

7

/

n

K

3

o

for long-term forecasting. However, recent studies [16, 21] find it a central problem for Transformer and other DNNs to generalize on varying temporal distribution and several works [15, 16, 21, 28] tailor to empower the robustness against shifted distribution. Especially, PatchTST [31] boosts Transformer to the state-of-the-art performance by channel-independence and instance normalization [41] but may lead to unaffordable computational cost when the number of series variate is large. In this paper, our proposed model supported by Koopman theory works naturally for non-stationary time series and achieves the state-of-the-art forecasting performance with remarkable model efficiency.

2.2 Learning Dynamics with Koopman Operator

Koopman theory [20] has emerged over decades as the dominant perspective to analyze modern dynamical systems [6]. Together with Dynamic Mode Decomposition (DMD) [38] as the leading numerical method to approximate the Koopman operator, significant advances have been accomplished in aerodynamics and fluid physics [3, 9, 30]. Recent progress made in Koopman theory is inherently incorporated with deep learning approaches in the data science era. Pilot works [29, 40, 50] leverage data-driven approaches such as Koopman Autoencoder to learn the measurement function and operator simultaneously. PCL [3] further introduces a backward procedure to improve the consistency and stability of the operator. Based on the capability of Koopman operator to advance nonlinear dynamics forward, it is also widely applied to sequence prediction. By means of Koopman spectral analysis, MDKAE [5] disentangles dominant factors underlying sequential data and is competent to forecast with specific factors. K-Forecast [24] utilizes Koopman theory to handle nonlinearity in temporal signals and propose to optimize data-dependent basis for long-term time series forecasting. By leveraging predefined measurement functions, KNF [44] learns Koopman operator and attention map to cope with time series forecasting with changing temporal distribution.

Different from previous Koopman forecasters, we design modular Koopman Predictors to tackle time-variant and time-invariant components with hierarchically learned operators, and renovate Koopman Autoencoder by removing the reconstruction loss to achieve fully predictive training.

3 Background

3.1 Koopman Theory

A discrete-time dynamical system can be formulated as x t +1 = F ( x t ) , where x t denotes the system state and F is a vector field describing the dynamics. However, it is challenging to identify the system transition directly on the state because of nonlinearity or noisy data. Instead, Koopman theory [20] hypothesizes the state can be projected into the space of measurement function g , which can be governed and advanced forward in time by an infinite-dimensional linear operator K , such that:

Koopman theory provides a bridge between finite-dimensional nonlinear dynamics and infinitedimensional linear dynamics, where spectral analysis tools can be applied to obtain in-depth analysis.

3.2 Dynamic Mode Decomposition

Dynamic Mode Decomposition (DMD) [38] seeks the best fitted finite-dimensional matrix K to approximate infinite-dimensional operator K by collecting the observed system state (a.k.a. snapshot ). Although DMD is the standard numerical method to analyze dynamics, it only works on linear space assumptions, which can be hardly identified without prior knowledge. Therefore, eDMD [45] is proposed to avoid handcrafting measurement functions and harmonious incorporations are made with the learning approach by employing autoencoders, which yields Koopman Autoencoder (KAE). By the universal approximation theorem [13] of deep networks, KAE finds desired Koopman embedding g ( x t ) with learned measurement function in a data-driven approach.

3.3 Time Series as Dynamics

It is challenging to predict real-world time series because of inherent non-stationarity. But if we zoom in the timeline, we will find the localized time series exhibited weak stationarity. It coincides with Koopman theory to analyze large nonlinear dynamics. That is, the measurement

<latexi sh

1_b

64="YE2Vq8P

G

BF

dXg/UK+m5w

>A

cj

HLS

N

D

r

u

I

y

n

Mo

0

T

Jk

Q

f

R

O

v

p

Z

9

7

z

W

<latexi sh

1_b

64="P

2dU

QIy

Mc

m

nzT

0

qGA

>

C

X

jVHLS

N

FD3

97

+

R

Z

p

5

J

u

v

B

/

O

Kf

g

E

8

Y

<latexi sh

1_b

64="RYT

M

E

Fo

UI

z

X

j

03rk

>A

Cy

c

VHLS

N

D2

pf

Zdug

w

7

B/Q

m

9

q

8

+

G

n

J

O

5K

W

v

P

<latexi sh

1_b

64="Pn23v7qwoWrzkSRT

KAM

H+

>

C

X

cjV

N

FD

Gd

5

U

I

m

ZJ

E

f

9

0

B

p

/

O

Q

Y

u

y

<latexi sh

1_b

64="

3

EV2cvOU0do

+n7SH

Fq

>A

Cy

j

L

N

D

r

pf

Z

ugkW

m5KI

M

G

J

8

z

9w

/

X

Q

P

T

R

B

Y

<latexi sh

1_b

64="NWgE

O2Rj dA

+JfL

HZ

X

38

>

C

c

V

S

FD

r

p

z

y

9

T

Q

U

M

7

P

B

0

/

w

q

I

5

u

m

G

Yn

v

K

<latexi sh

1_b

64="

WODNA

KI

q

S

M9

r

Tny8Z

>

C2X

cjVHL

F

7/

5

YB

p

Q

o

d

E0

mk

+

3

P

gf

U

J

R

z

G

v

w

u

<latexi sh

1_b

64="wn3

O

T

cFRr0

k

JK

ND

B+vY

>A

C

H

jV

LS

2

pf

Zdug

I

U

G

9

zEy

Q

q

X

5M

m

W

P

o

/

8

7

<latexi sh

1_b

64="

IXm cyBqMvNYA

rwEu

UPV

W

Q

>

C

H

j

LS

FD2

pf

Zd gk

R

G

9

J

O

0

+

8

5

T

z

7

/

n

K

3

o

<latexi sh

1_b

64="G

E7v+BYcg

Hjf

2KTz

o

Z

A

>

C

n

V

LS

N

FD

r

p

du

k

w

I

9

Q

UP

J

m

W

038

/

q

5

y

R

O

X

M

<latexi sh

1_b

64="fZUNM

jFS

AP25

BL

u

+

>

C

n

c

VH

D

r

p

d

gk

w

RI

0

K

G8mEy

/

v

Q

J

O

3

9

zX

7

W

T

o

q

Y

<latexi sh

1_b

64="2W

UX

uPTMv5Bf3

mR

8q

KO

>A

C0

cjVHLS

N

FD

r

p

Zd gk

w

I

7

J

E

9

z

/o

+

G

Q

y

Y

n

<latexi sh

1_b

64="uy8m o+vLK3pH

/

EMd

F

C

>A

0X

cjV

S

N

D2

r

f

Z

gk

w

RI

7

UB

5JO

9

P

G

QW

z

n

Y

q

T

<latexi sh

1_b

64="uy8m o+vLK3pH

/

EMd

F

C

>A

0X

cjV

S

N

D2

r

f

Z

gk

w

RI

7

UB

5JO

9

P

G

QW

z

n

Y

q

T

<latexi sh

1_b

64="/o2y+c9Fp

CRKm

Q

wqJ

Vj

LT0

>A

z3

H

D

U

I

d

kZ

N

E

B

Y

n

uP

O

fG

8

7

S

v

M

X

r

W

g

5

<latexi sh

1_b

64="/o2y+c9Fp

CRKm

Q

wqJ

Vj

LT0

>A

z3

H

D

U

I

d

kZ

N

E

B

Y

n

uP

O

fG

8

7

S

v

M

X

r

W

g

5

<latexi sh

1_b

64="YrSA

QH2dZ

gL

0

7

V

PFWo

>

Cz3

cj

T

J

D

U

+I

m

k

q

p

N

E

B

R

n

u

O

fG

9

/8

v

M

y

K

X

w

5

<latexi sh

1_b

64="YrSA

QH2dZ

gL

0

7

V

PFWo

>

Cz3

cj

T

J

D

U

+I

m

k

q

p

N

E

B

R

n

u

O

fG

9

/8

v

M

y

K

X

w

5

<latexi sh

1_b

64="/o2y+c9Fp

CRKm

Q

wqJ

Vj

LT0

>A

z3

H

D

U

I

d

kZ

N

E

B

Y

n

uP

O

fG

8

7

S

v

M

X

r

W

g

5

Time-variant KP

Time-invariant KP

+

-

forecast

residual

· · ·

+

input

model

forecast

Snapshot Pairs (Series window)

Time-invariant forecast

Globally learned dynamics

Locally calcuted dynamics

Lookback Window

Forecast Window

t

t

1

t

2

model input

Time-variant forecast

X

var

X

inv

X

inv

residual

forecast

X

(

b

)

inv

Y

(

b

)

inv

X

(

b

)

Y

Y

(

b

)

var

ˆ

X

(

b

)

var

X

(

b

)

var

K

inv

K

inv

K

var

K

var

K

var

function space can be divided into several neighborhoods, which are discriminately portrayed by localized linear operators [23]. Therefore, we leverage Koopman-based approaches that tackle large nonlinear dynamics by disentangling time-variant and time-invariant dynamics. Inspired by Wold's Theorem [46] that every covariance-stationary time series X t can be formally decomposed as:

where η t denotes the deterministic component and ε t is the stochastic component as the stationary process input of linear filter { b j } , we introduce globally learned and localized linear Koopman operators to exploit respective dynamics underlying different components.

4 Koopa

We propose Koopa composed of stackable Koopa Blocks (Figure 2). Each block is obliged to learn the input dynamics and advance it forward for prediction. Instead of struggling to seek one unified operator that governs the whole measurement function space, each Koopa Block is encouraged to learn operators hierarchically by taking the residual of previous block fitted dynamics as its input.

Koopa Block As aforementioned, it is essential to disentangle different dynamics and adopt proper operators for non-stationary series forecasting. The proposed block shown in Figure 2 contains Fourier Filter that utilizes frequency domain statistics to disentangle time-variant and time-invariant components and implements two types of Koopman Predictor (KP) to obtain Koopman embedding respectively. In Time-invariant KP, we set the operator as a model parameter to be globally learned from lookback-forecast windows. In Time-variant KP, analytical operator solutions are calculated locally within the lookback window, with series segments arranged as snapshots. In detail, we formulate the b -th block input X ( b ) as [ x 1 , x 2 , . . . , x T ] ⊤ ∈ R T × C , where T and C denote the lookback window length and the variate number. The target is to output a forecast window of length H . Our proposed Fourier Filter conducts disentanglement at the beginning of each block:

Respective KPs will predict with time-invariant input X ( b ) inv and time-variant input X ( b ) var , and Timevariant KP simultaneously outputs the fitted input ˆ X ( b ) var :

<latexi sh

1_b

64="uy8m o+vLK3pH

/

EMd

F

C

>A

0X

cjV

S

N

D2

r

f

Z

gk

w

RI

7

UB

5JO

9

P

G

QW

z

n

Y

q

T

<latexi sh

1_b

64="

Y7BF

n

+Ij c0CXV

9L

ofk

>A

H

S

N

D2

r

p

Ey q5K

Mu

G

P

U

J

W

T

wm

d

/

g

Q

O

3

Z

v

8

z

R

<latexi sh

1_b

64="YrSA

QH2dZ

gL

0

7

V

PFWo

>

Cz3

cj

T

J

D

U

+I

m

k

q

p

N

E

B

R

n

u

O

fG

9

/8

v

M

y

K

X

w

5

<latexi sh

1_b

64="209V

5Yk

W

CM

j

D

Nd

U

>A

X

c

HLS

F

r

pf

Z

ug

w

RI

7K

JO

P

3

+

G

m

o

n

/v

q

z

T

B

E

Qy

8

<latexi sh

1_b

64="P

pFSA

JV

DK

MQ

wo

WI

0k

>

C

H

cj

L

N

2

r

f

Ey q5

u

G

7

U

m

g

Y

d

/

+

O

3

Z

v

9

8

n

R

X

B

z

T

<latexi sh

1_b

64="

PWBoX0VAjSm

J

ZGd

K7

r

M

>

C2

c

HL

N

FD

/q

5

Y

URE

u

w

T

+

3

y

gf

8

I

9

p

n

v

O

z

k

Q

<latexi sh

1_b

64="2W

UX

uPTMv5Bf3

mR

8q

KO

>A

C0

cjVHLS

N

FD

r

p

Zd gk

w

I

7

J

E

9

z

/o

+

G

Q

y

Y

n

<latexi sh

1_b

64="

TZ

IkX

N

FDRW

UnE

Mq

Hw

>A

C

cjV

LS

2

r

pf dug

0

+

B

K

OJ

ov

7

P/Q

m

85

9Y

3

y

G

z

<latexi sh

1_b

64="U

Og0yL8mQ

5CXdj

9vIR7pw

>A

zH

c

V

S

N

FD2

r

f

Z

u

k

oq

T

3

E

K

n

P

G

+

W

YJ

M

<latexi sh

1_b

64="JUN

rCRd uvV5

Wn

Om

3YXG

>A

cj

HLS

FD2

pf

7

BI

gq

Q

o

+

Ek

Pz

y

K0/

w

M

8

Z

9

T

<latexi sh

1_b

64="c

M

GV

d

+NS2

oOXk vqB

A

>

Cz3

j

HL

FD

r

pf

Z

ug

w

RI

0

7

WymQ

Y

5

U

/

E

K

P

T

8

9

n

J

<latexi sh

1_b

64="/o2y+c9Fp

CRKm

Q

wqJ

Vj

LT0

>A

z3

H

D

U

I

d

kZ

N

E

B

Y

n

uP

O

fG

8

7

S

v

M

X

r

W

g

5

<latexi sh

1_b

64="zX

NW

8J+j0

dnS

mgQ

E

>A

c

VHL

FD2

r

pf

Z

u

k

w

RI

B

K

O

U

v

7

P/

M

5

9Y

3

y

q

<latexi sh

1_b

64="

CWpYr

8fM2

NR

oc

PGDu

I

w

>A

z3

jVHLS

F

Zd gk

0

7

+

B

y

5

U

/Q

E

K

O

v

J

Xn

q

m

T

9

<latexi sh

1_b

64="OAPC9NoK2Y

I

vVJB

/5SFn

0k

>

zH

cj

L

D

r

pf

Zdug

w

R

q

U

E

3

7

Q

m

yX

G

8

+

M

T

W

<latexi sh

1_b

64="rXGCAungk

SB

L8yNPMZFTqf

>

3

cjVH

D2

/

d

mW

R

J

o

w

Qv

0

O

I

9Y

E

U

z

p7

+

K

5

<latexi sh

1_b

64="/o2y+c9Fp

CRKm

Q

wqJ

Vj

LT0

>A

z3

H

D

U

I

d

kZ

N

E

B

Y

n

uP

O

fG

8

7

S

v

M

X

r

W

g

5

<latexi sh

1_b

64="3rA

VK7yS

5Tvk

j

2NoJIQw

>

CzH

c

L

FD

G

q

m

u

U

EP

p

0

g

8

f

X

Rn

d

9

W

Y

M

O

+

Z

<latexi sh

1_b

64="q

P7

p

0

OL

yZd uYr

UVk

w

>A

Cz3

cj

H

S

N

FD2

f

g

RI

+

BW

J

8

/

X

o

m

v

G

E

Q

M

T

9

5

<latexi sh

1_b

64="w8CGQ

T

g+9

5

P3

0nYp

Z7

>A

cjVHLS

N

FD2

r

f

du

k

RI

B

K

E

qUX

z

/o

O

M

m

v

W

y

J

<latexi sh

1_b

64="90RAm3DBK2

q

T

C/Sn

PI

>

cjVHL

N

F

G

vWpdugk

w

7

fU

y

5

Y

J

o

Z

z

r

M

OE

+

8

X

Q

<latexi sh

1_b

64="/o2y+c9Fp

CRKm

Q

wqJ

Vj

LT0

>A

z3

H

D

U

I

d

kZ

N

E

B

Y

n

uP

O

fG

8

7

S

v

M

X

r

W

g

5

<latexi sh

1_b

64="Qy

3/8

A

BdnuT

R

v

j

g

w

>

C0X

c

VHLS

N

FD2

r

pf

Z

k

I

7K

U

5JO

E

9

qz

o

+

G

M

W

Y

P

m

<latexi sh

1_b

64="K

g

LR

wA

rf

UY

Fz

>

C

n

cjVH

S

N

D2

p

q

7dBIv

Qk

0

T

u

9

G

J

+

ZPE

/8M

3

m

5

X

W

o

O

y

<latexi sh

1_b

64="0RpKjWyXM

J

u

cvQZD9SL

8

>A

CzH

V

N

F

2

r

f

d

gk

w

I

oq

T

B

5O

UE

3/

7

m

n

P

Y

G

+

<latexi sh

1_b

64="+O

m8oKjc7

w

Jq fn3

W

9ZE

>A

C

VHLS

N

FD2

r/

ud

k

RX

G

T

Qv

0

P

I

M

Y

B

U

z

p

5

y

g

<latexi sh

1_b

64="/o2y+c9Fp

CRKm

Q

wqJ

Vj

LT0

>A

z3

H

D

U

I

d

kZ

N

E

B

Y

n

uP

O

fG

8

7

S

v

M

X

r

W

g

5

<latexi sh

1_b

64="/o2y+c9Fp

CRKm

Q

wqJ

Vj

LT0

>A

z3

H

D

U

I

d

kZ

N

E

B

Y

n

uP

O

fG

8

7

S

v

M

X

r

W

g

5

<latexi sh

1_b

64="/o2y+c9Fp

CRKm

Q

wqJ

Vj

LT0

>A

z3

H

D

U

I

d

kZ

N

E

B

Y

n

uP

O

fG

8

7

S

v

M

X

r

W

g

5

Encoder

ˆ

x

3

ˆ

x

2

ˆ

x

1

x

1

x

2

x

3

ˆ

x

4

Decoder

Encoder

Time-invariant KP

Time-variant KP

z

1

z

2

z

3

ˆ

z

1

ˆ

z

2

ˆ

z

3

eDMD

calculate

input

input

forecast

forecast

fitted input

ˆ

z

4

Encoder

Encoder

Decoder

Decoder

Decoder

Decoder

X

inv

Y

inv

X

var

ˆ

X

var

Y

var

K

inv

K

var

K

var

K

var

K

var

K

var

Z

=

{

z

1

, z

2

, z

3

}

K

var

Z

back

Z

fore

Unlike KAEs [45, 44] that introduce a loss term for rigorous reconstruction of the lookback-window series, we feed the residual X ( b +1) as the input of next block for learning a corrective operator. And the model forecast Y is the sum of predicted components Y ( b ) var , Y ( b ) inv gathered from all Koopa Blocks:

Fourier Filter To disentangle the series components, we leverage Fourier analysis to find the globally shared and localized frequency spectrums reflected on different periods. Concretely, we precompute the Fast Fourier Transform (FFT) of each lookback window of the training set, calculate the averaged amplitude of each spectrum S = { 0 , 1 , . . . , [ T/ 2] } , and sort them by corresponding amplitude. We take the top percent of α as the subset G α ⊂ S , which contains dominant spectrums shared among all lookback windows and exhibits time-invariant dynamics underlying the dataset. And the remaining spectrums are the specific ingredient for varying windows in different periods. Therefore, we divide the spectrums S into G α and its complementary set ¯ G α . During training and inference, FourierFilter( · ) conducts the disentanglement of input X (block superscript omitted) as

where F means FFT, F -1 is its inverse and Filter( · ) only passes corresponding frequency spectrums with the given set. We validate the disentangling effect of our proposed Fourier Filter in Section 5.2 by calculating the variation degree of temporal dependencies in the disentangled series.

Time-invariant KP Time-invariant KP is designed to portray the globally shared dynamics, which discovers the direct transition from lookback window to forecast window as F : X inv ↦→ Y inv . Concretely, we introduce a pair of Encoder : R T × C ↦→ R D and Decoder : R D ↦→ R H × C to learn the common Koopman embedding for the time-invariant components of running window pairs, where D denotes the embedding dimension. Working on the data-driven measurement function, we introduce the operator K inv ∈ R D × D as a learnable parameter in each Time-invariant KP, which regards the embedding of lookback and forecast window Z back , Z fore ∈ R D as running snapshot pairs. The procedure is shown in Figure 3 and TimeInvKP( · ) is formulated as follows:

Time-variant KP As time-variant dynamics changes continuously, we utilize localized snapshots in a window, which constitute a temporal neighborhood more likely to be linearized. To obtain semantic snapshots and reduce iterations, the input X var is divided into T S segments x j of length S :

We assume S is divisible by T and H ; otherwise, we pad the input or truncate the output to make it compatible. Time-variant KP aims to portray localized dynamics, which is manifested analytically as the segment-wise transition F : x t ↦→ x t +1 with observed snapshots. We utilize another pair of Encoder : R S × C ↦→ R D to transform each segment into Koopman embedding z j and Decoder : R D ↦→ R S × C to transform the fitted or predicted embedding ˆ z j back to time segments ˆ x j :

Given snapshots collection Z = [ z 1 , . . . , z T S ] ∈ R D × T S , we leverage eDMD [45] to find the best fitted matrix that advances forward the system. We apply one-step operator approximation as follows:

where Z † back ∈ R ( T S -1) × D is the Moore-Penrose inverse of lookback window embedding collection. The calculated K var ∈ R D × D varies with windows and helps to analyze local temporal variations as a linear system. With the calculated operator, the fitted embedding is formulated as follows:

To obtain a prediction of length H , we iterate operator forwarding to get H S predicted embedding:

Finally, we arrange the segments transformed by Decoder( · ) as the module outputs ˆ X var , Y var. The whole procedure is shown in Figure 3 and TimeVarKP( · ) can be formulated as Equation 8-13.

Forecasting Objective In Koopa, Encoder , Decoder and K inv are learnable parameters, while K var is calculated on-the-fly. To maintain the Koopman embedding consistency in different blocks, we share Encoder , Decoder in Time-variant and Time-invariant KPs, which are formulated as ϕ var and ϕ inv respectively, and use the MSE loss with the ground truth Y gt for parameter optimization:

Optimizing by a single forecasting objective based on the assumption that if reconstruction failed, the prediction must also fail. Thus eliminating forecast discrepancy helps for fitting observed dynamics.

5 Experiments

Datasets We conduct extensive experiments to evaluate the performance and efficiency of Koopa. For multivariate forecasting, we include six real-world benchmarks used in Autoformer [48]: ECL (UCI), ETT [53], Exchange [22], ILI (CDC), Traffic (PeMS), and Weather (Wetterstation). For univariate forecasting, we evaluate the performance on the well-acknowledged M4 dataset [39], which contains four subsets of periodically collected univariate marketing data. And we follow the data processing and split ratio used in TimesNet [47].

Notably, instead of setting a fixed lookback window length, for every forecast window length H , we set the length of lookback window T = 2 H as the same with N-BEATS [32], because historical observations are always available in real-world scenarios and it can be beneficial for deep models to leverage more observed data with the increasing forecast horizon.

Baselines We extensively compare Koopa with the state-of-the-art deep forecasting models, including Transformer-based model: Autoformer [48], PatchTST [31]; TCN-based model: TimesNet [47], MICN [43]; MLP-based model: DLinear [51]; Fourier forecaster: FiLM [54], and Koopman forecaster: KNF [44]. We also introduce additional specialized models N-HiTS [7] and N-BEATS [32] for univariate forecasting as competitive baselines. All the baselines we reproduced are implemented based on the original paper or official code. We repeat each experiment three times with different random seeds and report the test MSE/MAE. And we provide detailed code implementation and hyperparameters sensitivity in Appendix C.

5.1 Time Series Forecasting

Forecasting results We list the results in Table 1-2 with the best in bold and the second underlined. Koopa shows competitive forecasting performance in both multivariate and univariate forecasting. Concretely, Koopa achieves state-of-the-art performance in more than 70% multivariate settings and consistently outperforms other deep models in the univariate settings.

Notably, Koopa surpasses the state-of-the-art Koopman-based forecaster KNF by a large margin in real-world time series, which can be attributed to our hierarchical dynamics learning and disentangling mechanism. Also, as the representative of efficient linear models, the performance of DLinear is still subpar in ILI, Traffic and Weather, indicating that nonlinear dynamics underlying the time series poses challenges for model capacity and point-wise weighting may not be appropriate to portray time-variant dynamics. Besides, compared with painstakingly trained PatchTST with channelindependence mechanism, our model can achieve a close and even better performance with naturally addressed non-stationary properties of real-world time series.

ETTh2

Memory Footprint

4.0GB

DLinear

0.807GB, 7.0ms

PatchTST

0.977GB, 20.3ms

TimesNet

3.285GB, 205.4ms

MICN

2.535GB, 96.6ms

FiLM

1.993GB, 263.2ms

Autoformer

4.583GB, 167.2ms

KNF

1.459GB, 142.4ms

MSE

0.3

0.4

0.5

50

100

150

200

250

0

Training Time (ms/iter)

0

300

600

900

1200

1500

Training Time (ms/iter)

0.3

0.5

0.7

0.6

0.4

MSE

2.339GB, 58.8ms

DLinear

1.143GB, 47.3ms

PatchTST

79.674GB, 1692.2ms

FiLM

35.287GB, 1556.4ms

TimesNet

12.135GB, 1596.4ms

MICN

2.847GB, 140.9ms

Autoformer

5.051GB, 195.4ms

KNF

1.0GB

2.0GB

10GB

20GB

40GB

Memory Footprint

Traffic

Model efficiency We comprehensively evaluate the model efficiency from three aspects: forecasting performance, training speed, and memory footprint. In Figure 4, we compare the efficiency under two representative datasets with different variate numbers (7 in ETTh2 and 862 in Traffic).

Compared with the state-of-the art forecasting model PatchTST, Koopa saves 62.3% and 96.5% training time respectively in the ETTh2 and Traffic datasets with only 26.8% and 2.9% memory footprint. Concretely, the averaged training time and memory ratio of Koopa compared to PatchTST are 22.7% and 24.0% in all six datasets (see Appendix D.3 for the detail). Besides, as an efficient MLP-based forecaster, Koopa is also capable of learning nonlinear dynamics from time-variant and time-invariant components, and thus achieves a better performance.

1

2

3

4

Time Series

0.08

-

0.08

0

K

1

K

2

K

3

K

4

K

K

K

K

5.2 Model Analysis

Dynamics disentanglement To validate the disentangling effect of our proposed Fourier Filter, we divide the whole time series into 20 subsets of different periods and conduct respective linear regression on the components disentangled by Fourier Filter. The standard deviation of the linear weighting reflects the variation of point-to-point temporal dependencies, which works as the manifestation of time-variant property. We plot the value as Degree of Variation (Figure 5 Left). It can be observed that larger deviations occur in the time-variant component, which indicates the proposed module successfully disentangles two types of dynamics from the perspective of frequency domain.

Case study We present a case study on real-world time series (exchange rate) on the right of Figure 5. We sample the lookback window at the interval of one year and visualize the Koopman operators calculated in Time-variant KP. It can be clearly observed that localized operators can exhibit changing temporal patterns in different periods, indicating the necessity of utilizing varying operators to describe time-variant dynamics. And interpretable insights are also presented as series uptrends correspond to heatmaps with large value and downtrends are reflected with small value.

<latexi sh

1_b

64="

ZGTV9HWj

F

2L

m

nwRgY

>A

C0

S

N

D

r/qu

Q

I

y

U

+

B

5

zE

f

J

7

8

p

X

d

P

M

O

3

v

<latexi sh

1_b

64="

ZGTV9HWj

F

2L

m

nwRgY

>A

C0

S

N

D

r/qu

Q

I

y

U

+

B

5

zE

f

J

7

8

p

X

d

P

M

O

3

v

<latexi sh

1_b

64="Gu2

kpACW

M5

/f

7J

y

zv3w

>

0H

cjV

LS

N

FD

q

nQT

IKr

o

Y

U

m

O

X

8

B

PR

+

d

Z

9

E

g

<latexi sh

1_b

64="Gu2

kpACW

M5

/f

7J

y

zv3w

>

0H

cjV

LS

N

FD

q

nQT

IKr

o

Y

U

m

O

X

8

B

PR

+

d

Z

9

E

g

<latexi sh

1_b

64="v

W

7FgJnGM

N2QZB

L5

O

c

>A

C0H

jV

S

D

/q

u

T

IKrk

o

9

E

p

d

R

P+

8

z

3

U

f

m

y

X

Y

w

<latexi sh

1_b

64="v

W

7FgJnGM

N2QZB

L5

O

c

>A

C0H

jV

S

D

/q

u

T

IKrk

o

9

E

p

d

R

P+

8

z

3

U

f

m

y

X

Y

w

<latexi sh

1_b

64="7

cT

row9U

C

unI

S

q30Y

F

>A

H

jV

L

gRB

D

5

O

2

N

G

mZE

k

p

X

P

/

K

d

8

W

v

M

J

+

y

z

f

Q

<latexi sh

1_b

64="7

cT

row9U

C

unI

S

q30Y

F

>A

H

jV

L

gRB

D

5

O

2

N

G

mZE

k

p

X

P

/

K

d

8

W

v

M

J

+

y

z

f

Q

Ablation study We conduct ablations on Koopa. As shown in Table 3, Time-variant and Timeinvariant KPs perform as complementary modules to explore the dynamics underlying time series, and discarding any one of them will lead to the inferior performance. Besides, we evaluate alternative decomposition filters to disentangle time series dynamics. We find the proposed Fourier Filter conducts effective disentanglement, where the amplitude statistics of frequency spectrums from different periods are utilized to exhibit time-agnostic information. Therefore, Koopa tackling the right dynamics with complementary modules can achieves the best performance.

Avoiding rigorous reconstruction Unlike previous Koopman Autoencoders, the proposed Koopman Predictor does not reconstruct the whole dynamics at once, but aims to portray the partial dynamics evolution. Thus we remove the reconstruction branch, which is only utilized during training in previous KAEs. In our deep residual structure, the predictive objective function works as a good optimization indicator. We validate the design in Table 4, where the performance of sorely forecasting objective optimized model is better than with an additional reconstruction loss. Because the end-toend forecasting objective helps to reduce the optimization gap between training and inference, making it a valuable contribution of applying Koopman operators on end-to-end time series forecasting.

Learning stable operators Weturn to analyze our architectural design from the spectral perspective. The eigenvalues of the operator determine the amplitude of dynamics evolution. As most of nonstationary time series experience the distribution shift and can be regarded as an unstable evolution, the learned Koopman operator with the modulus far from the unit circle will cause non-divergent and even explosive trending in the long term, leading to training failures.

To tackle this problem generally faced by Koopman-based forecasters, we propose to utilize the disentanglement and deep residual structure. We measure the stability of the operator as the average distance of eigenvalues from the unit circle. As shown in Figure 6, the operator can become more stable by the above two techniques. The disentanglement helps to describe complex dynamics based on the decomposition and appropriate inductive bias can be applied. The architecture where each block is employed to fill the residual of the previously fitted dynamics reduces the difficulty of directly reconstructing complicated dynamics. Each block portrays the basic process driven by a stable operator within its power, which can be aggregated for a complicated non-stationary process.

5.3 Scaling Up Forecast Horizon

Most deep forecasting models work as a settled function once trained (e.g. inputT -outputH ). For scenarios where the prediction horizon is mismatched or long-term, it poses two challenges for the trained model: (1) reuse parameters learned from observed series; (2) utilize incoming ground truth for model adaptation. The practical scenarios, which we name as scaling up forecast horizon , may

<latexi sh

1_b

64="M5

py

WqJS

N32UPc

DI

m

8

>A

Bn jV

E

Gr

/ouL

7

k

+

g

w

F

0

Q

Xf

v

d

C

R

Z

z

T

H

O

Y

9

K

<latexi sh

1_b

64="

B

DUv8FJH

co53n

RC

27

>A

jVL

S

N

Gd

ZW

Y

qwpKI

m

E

k

P

X

/

z

Q

f

g

T

O

Mu

y

+

0

r

9

<latexi sh

1_b

64="

Pm pB

Du/NL

HyY

EkX

Fo

0

>A

n

cjV

S

3Gd

ZW

qw

r

KU

IJ

27v

R

z

C

O

8

f

T

9M

+

5

g

Q

-1

0

1

-1

0

1

Real

Imaginary

-1

0

1

-1

0

1

Real

Imaginary

-1

0

1

-1

0

1

Real

Imaginary

AV G

(

||

z

| -

1

|

) = 0

.

1092

AV G

(

||

z

| -

1

|

) = 0

.

0887

AV G

(

||

z

| -

1

|

) = 0

.

0613

(a)

(b)

(c)

lead to failure on most deep models but can be naturally tackled by Koopa. In detail, we first train Koopa with forecast length H tr and attempt to apply it on a larger forecast length H te .

Method Koopa scales up forecast horizon as follows: Since Time-invariant KP has learned the globally shared dynamics and Time-variant KP can calculate localized operator K var within the lookback window, we freeze the parameters of trained Koopa but only use the incoming ground truth to adapt K var. The naïve implementation uses incremental Koopman embedding with dimension D and conducts Equation 10 to obtain an updated operator, which has a complexity of O ( H te D 3 ) . We further propose an iterative algorithm with improved O (( H te + D ) D 2 ) complexity. The detailed method implementations and complexity analysis can be found in Appendix A.

Results As shown in Table 5, the proposed operator adaption mechanism further boosts the performance on the scaling up scenario, which can be attributed to more accurately fitted time-variant dynamics with incoming ground truth snapshots. Besides, the promotion becomes more significant when applied to non-stationary datasets (manifested as large ADF Test Statistic [10]).

6 Conclusion

This paper tackles time series as dynamical systems. With disentangled time-variant and timeinvariant components from non-stationary series, the Koopa model reveals the complicated dynamics hierarchically and leverages MLP modules to learn Koopman embedding and operator. Experimentally, our model shows competitive performance with remarkable efficiency and the potential to scale up the forecast length by operator adaptation. In the future, we will explore Koopa with the dynamic modes underlying non-stationary data using the toolbox of Koopman spectral analysis.

Acknowledgments

This work was supported by the National Key Research and Development Plan (2021YFC3000905), National Natural Science Foundation of China (62022050 and 62021002), Beijing Nova Program (Z201100006820041), and BNRist Innovation Fund (BNR2021RC01002).

References

Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, Jean-Christophe Gagnon-Audet, Yoshua Bengio, Ioannis Mitliagkas, and Irina Rish. Invariance principle meets information bottleneck for out-of-distribution generalization. NeurIPS , 2021.

O. Anderson and M. Kendall. Time-series. 2nd edn. J. R. Stat. Soc. (Series D) , 1976.

Omri Azencot, N Benjamin Erichson, Vanessa Lin, and Michael Mahoney. Forecasting sequential data using consistent koopman autoencoders. In International Conference on Machine Learning , pages 475-485. PMLR, 2020.

Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271 , 2018.

Nimrod Berman, Ilan Naiman, and Omri Azencot. Multifactor sequential disentanglement via structured koopman autoencoders. arXiv preprint arXiv:2303.17264 , 2023.

Steven L Brunton, Marko Budiši´ c, Eurika Kaiser, and J Nathan Kutz. Modern koopman theory for dynamical systems. arXiv preprint arXiv:2102.12086 , 2021.

Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza, Max Mergenthaler, and Artur Dubrawski. N-hits: Neural hierarchical interpolation for time series forecasting. arXiv preprint arXiv:2201.12886 , 2022.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2021.

Hamidreza Eivazi, Luca Guastoni, Philipp Schlatter, Hossein Azizpour, and Ricardo Vinuesa. Recurrent neural networks and koopman-based frameworks for temporal predictions in a low-order model of turbulence. International Journal of Heat and Fluid Flow , 90:108816, 2021.

Graham Elliott, Thomas J Rothenberg, and James H Stock. Efficient tests for an autoregressive unit root, 1992.

Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211 , 2013.

S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Comput. , 1997.

Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural networks , 4(2):251257, 1991.

Rob J Hyndman and George Athanasopoulos. Forecasting: principles and practice . OTexts, 2018.

Xiaoyong Jin, Youngsuk Park, Danielle Maddix, Hao Wang, and Yuyang Wang. Domain adaptation for time series forecasting via attention sharing. In International Conference on Machine Learning , pages 10280-10297. PMLR, 2022.

Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Reversible instance normalization for accurate time-series forecasting against distribution shift. In ICLR , 2022.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences , 114(13):3521-3526, 2017.

Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In ICLR , 2020.

Bernard O Koopman. Hamiltonian systems and transformation in hilbert space. Proceedings of the National Academy of Sciences , 17(5):315-318, 1931.

Vitaly Kuznetsov and Mehryar Mohri. Discrepancy-based theory and algorithms for forecasting nonstationary time series. Annals of Mathematics and Artificial Intelligence , 88(4):367-399, 2020.

Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In SIGIR , 2018.

Yueheng Lan and Igor Mezi´ c. Linearization in the large of nonlinear systems and koopman operator spectrum. Physica D: Nonlinear Phenomena , 242(1):42-53, 2013.

Henning Lange, Steven L Brunton, and J Nathan Kutz. From fourier to koopman: Spectral methods for long-term time series prediction. The Journal of Machine Learning Research , 22(1):1881-1918, 2021.

Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In ICCV , 2017.

Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895 , 2020.

Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In ICLR , 2021.

Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Rethinking the stationarity in time series forecasting. arXiv preprint arXiv:2205.14415 , 2022.

Bethany Lusch, J Nathan Kutz, and Steven L Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature communications , 9(1):4950, 2018.

Jeremy Morton, Antony Jameson, Mykel J Kochenderfer, and Freddie Witherden. Deep dynamical modeling and control of unsteady fluid flows. Advances in Neural Information Processing Systems , 31, 2018.

Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.14730 , 2022.

Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-BEATS: Neural basis expansion analysis for interpretable time series forecasting. ICLR , 2019.

Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. TKDE , 2009.

Adam Paszke, S. Gross, Francisco Massa, A. Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Z. Lin, N. Gimelshein, L. Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS , 2019.

Quang Pham, Chenghao Liu, Doyen Sahoo, and Steven CH Hoi. Learning fast and slow for online time series forecasting. arXiv preprint arXiv:2202.11672 , 2022.

Joshua L Proctor, Steven L Brunton, and J Nathan Kutz. Dynamic mode decomposition with control. SIAM Journal on Applied Dynamical Systems , 15(1):142-161, 2016.

David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting , 36(3):1181-1191, 2020.

Peter J Schmid. Dynamic mode decomposition of numerical and experimental data. Journal of fluid mechanics , 656:5-28, 2010.

Spyros Makridakis. M4 dataset, 2018.

Naoya Takeishi, Yoshinobu Kawahara, and Takehisa Yairi. Learning koopman invariant subspaces for dynamic mode decomposition. Advances in neural information processing systems , 30, 2017.

Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022 , 2016.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , 2017.

Huiqiang Wang, Jian Peng, Feihu Huang, Jince Wang, Junhui Chen, and Yifei Xiao. Micn: Multi-scale local and global context modeling for long-term series forecasting. In The Eleventh International Conference on Learning Representations , 2023.

Rui Wang, Yihe Dong, Sercan O Arik, and Rose Yu. Koopman neural forecaster for time series with temporal distribution shifts. arXiv preprint arXiv:2210.03675 , 2022.

Matthew O Williams, Ioannis G Kevrekidis, and Clarence W Rowley. A data-driven approximation of the koopman operator: Extending dynamic mode decomposition. Journal of Nonlinear Science , 25:1307-1346, 2015.

Herman Wold. A study in the analysis of stationary time series . PhD thesis, Almqvist & Wiksell, 1938.

Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. arXiv preprint arXiv:2210.02186 , 2022.

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with Auto-Correlation for long-term series forecasting. In NeurIPS , 2021.

Wei Xiong, Xiaomeng Huang, Ziyang Zhang, Ruixuan Deng, Pei Sun, and Yang Tian. Koopman neural operator as a mesh-free solver of non-linear partial differential equations. arXiv preprint arXiv:2301.10022 , 2023.

Enoch Yeung, Soumya Kundu, and Nathan Hodas. Learning deep neural network representations for koopman operators of nonlinear dynamical systems. In 2019 American Control Conference (ACC) , pages 4832-4839. IEEE, 2019.

Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? arXiv preprint arXiv:2205.13504 , 2022.

Tianping Zhang, Yizhuo Zhang, Wei Cao, Jiang Bian, Xiaohan Yi, Shun Zheng, and Jian Li. Less is more: Fast multivariate time series forecasting with light sampling-oriented mlp structures. arXiv preprint arXiv:2207.01186 , 2022.

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI , 2021.

Tian Zhou, Ziqing Ma, Qingsong Wen, Liang Sun, Tao Yao, Wotao Yin, Rong Jin, et al. Film: Frequency improved legendre memory model for long-term time series forecasting. Advances in Neural Information Processing Systems , 35:12677-12690, 2022.

A Scaling Up Forecast Horizon

In this section, we introduce the capability of Koopa to scale up forecast horizon. In detail, we train a Koopa model with forecast length H tr and attempt to apply it on a larger length H te . The basic approach conducts rolling forecast by taking the model prediction as the input of the next iteration until the desired forecast horizon is all filled. Instead, we further assume that after the model gives a prediction, the model can utilize the incoming ground truth for model adaptation and continue rolling forecast for the next iteration. It is notable that we do not retrain parameters during model adaptation, since it will lead to overfitting on the incoming ground truth and Catastrophic Forgetting [11, 18, 35].

Koopa can naturally cope with the scenario by learning Koopman embedding and operator K inv in Time-invariant KPs while calculating localized operator K var to describe the dynamics in the temporal neighborhood. Therefore, we freeze the parameters of Koopa but only use the incoming ground truth for operator adaptation of K var in Time-variant KPs.

A.1 Implementation of Operator Adaptation

At the beginning of rolling forecast, the Encoder in Time-variant KP outputs D -dimensional Koopman embedding for each observed series segment as [ z 1 , z 2 , . . . , z F ] , where F = H tr S is the segment number with S as the segment length. The operator K var in Time-variant KP is calculated as follows:

where Z back , Z fore ∈ R D × ( F -1) , K var ∈ R D × D . With the calculated operator, we obtain the next predicted Koopman embedding by one-step forwarding:

After decoding the embedding ˆ z F +1 to the series prediction, we can utilize the true value of incoming Koopman embedding z F +1 obtained by Koopa with frozen parameters. Instead of using K var to obtain the next embedding

ˆ z F +2 , we use incremental embedding collections Z back + , Z fore + ∈ R D × F to obtain a more accurate operator K var + ∈ R D × D to describe the local dynamics:

The procedure repeats for L times ( L ∝ H te) until the forecast horizon is all filled, we formulate it as Algorithm 1. And experimental results (Koopa OA) in Section 5.3 have demonstrated the promotion of forecasting performance due to more precisely fitted dynamics.

A.2 Computational Acceleration

The naïve implementation shown in Algorithm 1 repeatedly conducts Equation 17 on the incremental embedding collection to obtain new operators, which has a complexity of O ( LD 3 ) . We propose an equivalent algorithm with improved complexity of O (( L + D ) D 2 ) as shown in Algorithm 2.

Theorem. Algorithm 2 gives the same K var as Algorithm 1 in each iteration with O ( D 2 ) complexity.

Proof. We start with the first iteration analysis. By the definition of Moore-Penrose inverse, we have Z † back Z back = I F -1 , where I F -1 is an identity matrix with the dimension of F -1 . When the model receives the incoming embedding z F +1 , incremental embedding m = z F , n = z F +1 will be appended to Z back and Z fore respectively. Instead of calculating new K var + from incremental collections, we utilize calculated K var to find the iteration rule on K var+. Concretely, we suppose

where ∆ ∈ R ( F -1) × D , b ∈ R D are variables to be identified. By the definition of Moore-Penrose inverse, we have Z † back + Z back + = I F . By unfolding it, we have the following equations:

We suppose ∆ = δb ⊤ , where δ ∈ R F -1 , such that when b ⊤ Z back = ⃗ 0 , then ∆ Z back = 0 . Then we have Z † back m -δb ⊤ m = Z † back m -δ = ⃗ 0 , thus ∆ = Z † back mb ⊤ . Given equations that b ⊤ Z back = ⃗ 0 and b ⊤ m = 1 , we have the analytical solution of b :

Therefore, we find the equation between the incremental version K var + and calculated K var :

where m,n are the incremental embedding of Z back , Z fore and b can be calculated by Equation 20. We also derive the iteration rule on X = Z back Z † back to obtain b , which is formulated as follows:

By adopting Equation 21- 22 and permuting the matrix multiplication order, we reduce the complexity of each iteration to O ( D 2 ) . Therefore, Algorithm 2 has a overall complexity of O (( L + D ) D 2 ) . Since L ∝ H te , Algorithm 1-2 have O ( H te D 3 ) and O (( H te + D ) D 2 ) complexity respectively.

B Implementation Details

Koopa is trained with L2 loss and optimized by ADAM [17] with an initial learning rate of 0.001 and batch size set to 32. The training process is early stopped within 10 epochs. We repeat each experiment three times with different random seeds to obtain average test MSE/MAE and detailed results with standard deviations are listed in Table 6. Experiments are implemented in PyTorch [34] and conducted on NVIDIA TITAN RTX 24GB GPUs.

All the baselines that we reproduced are implemented based on the benchmark of TimesNet [47] Repository, which is fairly built on the configurations provided by each model's original paper or official code. Since several baselines adopt Series Stationarization from Non-stationary Transformers [28] while others do not, we equip all models with the method for a fair comparison.

C Hyperparameter Sensitivity

Considering the efficiency of hyperparameters search, we fix the segment length S = T/ 2 and the number of Koopa blocks B = 3 in all our experiments. We verify the robustness of Koopa of other hyperparameters as

Algorithm 1 Koopa Operator Adaptation.

Require: Observed embedding Z = [ z 1 , . . . , z F ] and successively incoming ground truth embedding [ z , . . . , z ] with each embedding z R D .

F +1 F + L i ∈

3: ˆ z F +1 = K var n

4: for l in { 1 , . . . , L } :

5: for m = z F + l -1 , n = z F + l

6: for Z back ← [ Z back , m ] , Z fore ← [ Z fore , n ]

7: for K var = Z fore Z † back

8: for ˆ z F + l +1 = K var n

9: End for

10: Return [ˆ z F +1 , . . . , ˆ z F + L +1 ]

Algorithm 2 Accelerated Koopa Operator Adaptation.

Require: Observed embedding Z = [ z 1 , . . . , z F ] and successively incoming ground truth embedding [ z F +1 , . . . , z F + L ] with each embedding z i ∈ R D .

3: ˆ z F +1 = K var n

4: for l in { 1 , . . . , L } :

5: for m = z F + l -1 , n = z F + l

6: for r = m -Xm

7: for b = r/ || r || 2

9: for X ← X + rb ⊤

10: for ˆ z F + l +1 = K var n

11: End for

12: Return [ˆ z F +1 , . . . , ˆ z F + L +1 ] ⊤

▷ z F + l comes successively

▷ m, n ∈ R D

▷ K var ∈ R D × D

▷ ˆ z F + l +1 ∈ R D

▷ Return predicted embedding

▷ z F + l comes successively

▷ m, n ∈ R D

▷ b ∈ R D

▷ r ∈ R D

▷ ˆ z F + l +1 ∈ R D

▷ X ∈ R D × D

▷ Return predicted embedding

MSE

0.1

0.2

0.3

0.4

Dimension of Koopman Embedding

1

2

3

4

0.1

0.2

0.3

0.4

Hidden Dimension

0.1

0.2

0.3

0.4

2 6

2 7

2 8

2 9

2 6

2 7

2 8

2 9

follows: the dimension of Koopman embedding D , the hidden layer number l and the hidden dimension d used in Encoder and Decoder. As is shown in Figure 7, we find the proposed model is insensitive to the choices of above hyperparameters, which can be beneficial for practitioners to reduce tuning burden.

Intuitively, a larger dimension of Koopman embedding D can bring about a lower approximation error. We further dive into it and find that stacking blocks can enhance the model capacity, and thus the performance is insensitive to D when the model is deep enough. To address the concern, we further check the sensitivity of D under varying block number B in Figure 8. It can be seen that a larger B generally leads to lower error even if D is small. And the performance can be sensitive to D when the model is not deep enough ( B = 1 , 2 ).

Besides, we find the proposed model can be insensitive to S on several datasets while sensitive on ECL and Traffic datasets (The difference is about 10%). There are many variables in these two datasets, but our current design shares the S for all variables. Since different variables with distinct evolution periods implicitly require different optimal S , the performance of the dataset with more variables is more likely to be influenced by S . Therefore, we set S = T/ 2 with relatively small performance fluctuation to deal with most situations.

2 6

2 7

2 8

2 9

Dimension of Koopman Embedding

MSE

0.12

0.13

0.14

0.15

0.1

0.2

0.3

0.4

0.5

MSE

T/8

T/4

T/2

T

Segment Length

D Supplementary Experimental Results

D.1 Full Forecasting Results

Due to the limited pages, we list additional multivariate benchmarks on ETT datasets [53] in Table 7, which includes the hourly recorded ETTh2 and 15-minutely recorded ETTm1/ETTm2, and the full univariate results of M4 [39] in Table 8, which contains the yearly, quarterly and monthly collected univariate marketing data. Notably, Koopa still achieves competitive performance compared with state-of-the-art deep forecasting models and specialized univariate models.

D.2 Full Ablation Results

We elaborately conduct model ablations to verify the necessity of each proposed module: Time-invariant KP, Time-variant KP, Fourier Filter and evaluate alternative choices to disentangle dynamics. As shown in Table 9, Koopa conducts effective disentanglement and tackles the right dynamics with complementary KPs, and thus achieves the best forecasting performance.

D.3 Model Efficiency

We comprehensively compare the forecasting performance, training speed, and memory footprint of our model with well-acknowledged deep forecasting models. The results are recorded with the official model configuration and the same batch size. We visualize the model efficiency under all six multivariate datasets in Figure 9- 11. In detail, compared with the previous state-of-the-art model PatchTST [8], Koopa consumes only 15.2% training time and 3.6% memory footprint respectively in ECL, 37.8% training time and 26.8% memory in ETTh2, 23.5% training time and 37.3% memory in Exchange, 50.9% training time and 47.8% memory in ILI, 3.5% training time and 2.9% memory in Traffic, and 5.4% training time and 25.4% memory in Weather, leading to the averaged 77.3% and 76.0% saving of training time and memory footprint in all six datasets. The remarkable efficiency can be attributed to Koopa with MLPs as the building blocks, and we find the budget saving becoming more significant on datasets with more series variables (ECL, Traffic).

Besides, as an efficient linear model, the performance of Koopa still surpasses other MLP-based models. Especially, Compared with DLinear [51], our model reduces 38.0% MSE (2.852 → 1.768) in ILI and 13.6% MSE (0.452 → 0.397) in Weather. And the average MSE reduction of Koopa compared with the previous state-of-the-art MLP-based model reaches 12.2% . Therefore, our proposed Koopa is efficiently built with MLP networks and shows great model capacity to exploit nonlinear dynamics and complicated temporal dependencies in real-world time series.

ETTh2

Memory Footprint

4.0GB

DLinear

0.807GB, 7.0ms

PatchTST

0.977GB, 20.3ms

TimesNet

3.285GB, 205.4ms

MICN

2.535GB, 96.6ms

FiLM

1.993GB, 263.2ms

Autoformer

4.583GB, 167.2ms

KNF

1.459GB, 142.4ms

MSE

0.3

0.4

0.5

50

100

150

200

250

0

Training Time (ms/iter)

0

300

600

900

1200

1500

Training Time (ms/iter)

0.3

0.5

0.7

0.6

0.4

MSE

2.339GB, 58.8ms

DLinear

1.143GB, 47.3ms

PatchTST

79.674GB, 1692.2ms

FiLM

35.287GB, 1556.4ms

TimesNet

12.135GB, 1596.4ms

MICN

2.847GB, 140.9ms

Autoformer

5.051GB, 195.4ms

KNF

1.0GB

2.0GB

10GB

20GB

40GB

Memory Footprint

Traffic

MSE

0.0

0.2

0.4

0.6

0

200

400

600

Exchange

Memory Footprint

Training Time (ms/iter)

4.0GB

1.0GB 2.0GB

0.977GB, 15.9ms

PatchTST

2.617GB, 191.7ms

TimesNet

2.820GB, 343.5ms

DLinear

0.807G, 8.2ms

MICN

1.163GB, 103.9ms

KNF

1.593GB, 122.5ms

FiLM

2.607GB, 268.9ms

Autoformer

4.585GB, 461.4ms

MSE

0.12

0.15

0.18

0.21

0.24

0

100

200

300

400

500

Memory Footprint

20GB

10GB

5GB

Training Time (ms/iter)

ECL

1.425GB, 29.0ms

PatchTST

TimesNet

DLinear

0.901G, 16.9ms

MICN

1.717GB, 41.5ms

KNF

23.220GB, 292.3ms

FiLM

7.561GB, 257.3ms

Autoformer

4.803GB, 164.2ms

Training Time (ms/iter)

MSE

0.1

0.2

0.3

0.4

0

100

200

300

Weather

Memory Footprint

4.0GB

1.0GB 2.0GB

PatchTST

3.877GB, 141.0ms

TimesNet

5.727GB, 289.9ms

DLinear

0.811G, 4.1ms

MICN

1.109GB, 13.7ms

KNF

4.525GB, 322.2ms

FiLM

4.687GB, 232.4ms

Autoformer

4.587GB, 151.8ms

Training Time (ms/iter)

ILI

MSE

0

100

200

300

400

500

1.6

2.0

2.4

2.8

3.2

3.6

Memory Footprint

4.0GB

2.0GB

1.0GB

0.963GB, 75.2ms

PatchTST

2.015GB, 147.7ms

TimesNet

1.835GB, 447.3ms

DLinear

0.783G, 54.6ms

MICN

2.725GB, 337.6ms

KNF

1.261GB, 329.4ms

FiLM

1.707GB, 455.6ms

Autoformer

2.481GB, 268.5ms

D.4 Training Stability

Since the eigenvalues of the operator determine the amplitude of dynamics evolution, where modulus not close to one causes non-divergent and explosive evolution in the long-term, we highlight that a stable Koopman operator only describes weakly stationary series well. While most of the Koopman-based forecasters can suffer from the operator convergence problem induced by complicated non-stationary series variations, we employ several techniques to stabilize the training process.

Operator initialization We adopt the operator initialization strategy, where the operator starts from the multiplication of eigenfunctions with standard Gaussian distribution and all-one eigenvalues.

Hierarchical disentanglement In our model, each block learns weak stationary process hierarchically and feeds the residual of fitted dynamics for the next block to correct. Thus Koopman Predictor aims not to fully reconstruct the whole dynamics at once, but to partially describe dynamics, so rigorous reconstruction is not forced in each block, reducing the difficulty of portraying the non-stationary series as dynamics.

Explosion checking We introduce an explosion checking mechanism that replaces the operator encountering nan number with the identity matrix when the exponential multiplication of multiple time steps is detected.

Based on the proposed strategies, we provide the model training curves in Figure 12 to check the convergence of our proposed model and other forecasters. The training curves of the proposed model in blue demonstrate a consistent and smooth convergence, indicating its effectiveness in converging toward an optimal solution.

E Broader Impact

E.1 Impact on Real-world Applications

Our work copes with real-world time series forecasting, which is faced with intrinsic non-stationarity that poses fundamental challenges for deep forecasting models. Since previous studies hardly research the theoretical basis that can naturally address the time-variant property in non-stationary data, we propose a novel Koopman

ETTh1

0.30

0.25

0.35

0.40

0.45

2

4

6

8

10

ETTh2

0.30

0.25

0.35

0.40

2

4

6

8

10

Weather

0.35

0.30

0.40

0.45

2

4

6

8

10

Training Loss

Epoch

Epoch

Epoch

forecaster that fundamentally considers the implicit time-variant and time-invariant dynamics based on Koopman theory. Our model achieves the state-of-the-art performance on six real-world forecasting tasks, covering energy, economics, disease, traffic, and weather, and demonstrates remarkable model efficiency in training time and memory footprint. Therefore, the proposed model makes it promising to tackle real-world forecasting applications, which helps our society prevent risks in advance and make better decisions with limited computational budgets. Our paper mainly focuses on scientific research and has no obvious negative social impact.

E.2 Impact on Future Research

In this paper, we find modern Koopman theory natural to learn the dynamics underlying non-stationary time series. The proposed model explores complex non-stationary patterns with temporal localization inspired by Koopman approaches and implements respective deep network modules to disentangle and portray time-variant and time-invariant dynamics with the enlightenment of Wold's Theorem. The remarkable efficiency and insights from the theory can be instructive for future research.

F Limitation

Our proposed model does not respectively considers dynamics in different variates, which leaves improvement for better multivariate forecasting with the consideration of various evolution patterns and series relationships. And Koopman spectral theory is still under leveraging in our work, which can discover Koopman modes to interpret the linear behavior underlying non-stationary data in a high-dimensional representation. Besides, Koopman theory for control considering factors outside the system can be promising for series forecasting with covariates, which leaves our future work.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the task of non-stationary time series forecasting with Koopman operator. Specifically, the authors propose Koopa Blocks, which uses Fourier transform to disentangle the time series into time-variant and time-invariant components, and leverages Koopman \noperator to model their dynamics respectively. The method is evaluated on six real-world benchmark datasets.\n\nSTRENGTHS:\n1. The paper is well-written and easy to follow.\n2. This study represents a pioneering endeavor in integrating Fourier transform and Koopman theory, with the aim of disentangling time-variant and time-invariant components within the Fourier spectral domain. The idea is both innovative and thought-provoking, setting a precedent for future investigations into the research of deep Koopman methods.\n3. The additional \"Scaling Up Forecast Horizon\" experiments demonstrate that Koopa is capable of accurately forecasting in situations involving length mismatches or long-term scenarios. In the realm of the Koopman method, these long-term prediction results are exceptional.\n4. The proposed model is memory and time efficient compared to previous methods.\n\nWEAKNESSES:\n1. The performance improvement of univariate forecasting is incremental. Also, the improvement of multivariate forecasting is also not so significant under most cases.\n2. The essence of the deep Koopman method lies in the spectrum of the Koopman operator, because the eigenvalues determine the model's behavior during long-term evolution. However, the author did not provide any analysis or visualized results regarding the eigenvalues. This lead to the following issues:\n- Given the considerable forecasting length (up to 192 in this case), it is remarkable how the model manages to achieve accurate predictions without encountering issues such as explosion or decay during the forward pass. Is it true that all the eigenvalues of the system have a modulus of one, ensuring stable and non-divergent behavior throughout the prediction process? \n- Intuitively, it would be expected that the time-variant Koopman operator possesses more eigenvalues with larger phase angles compared to the time-invariant operator. I wonder whether this is true in this case. What does the spectrum for these two operator looks like?\n\nQUESTIONS:\n1. The authors implement the DMD with torch.linalg.lstsq.  Were there any concerns regarding gradient explosion during the autograd process through this layer? Did the authors employ any techniques to stabilize the training process?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work propose a new architecture inspired by Koopman theory for time-series forecasting task. To address the non-stationarity problem, this work follow the idea of disentangle local and global time-series representation and forecasting model respectively. Particularly, this work propose to use Fourier filter on long sequence to extract more robust and stable patterns for global model and Koopa Block to hierarchically disentangle local and global representation in a residual structure. The proposed method achieves competitive performance when comparing with recent baselines.\n\nSTRENGTHS:\n1.  The structure is well organized. Figures 2 and 3 provide a clear visual representation of how the proposed method works.\n\n2. The experiments are comprehensive. The proposed method can beat the recent method PatchTST on the commonly used long sequence forecasting benchmark.\n\nWEAKNESSES:\n1. Although the paper's title revolves around the Koopman theory, the proposed method is generic and not closely related to it. \nFrom my understanding, most of technique contributions lie in how to extract local and global representation and design forecasting model for non-stationary time-series. The work solely follow the the conclusion of Koopman theory and model the latent dynamic with linear transition model. \n\n2. Technique novelty is limited. The general idea of using local and global presentation/model to tackle non-stationary time-series tasks has been used. This work builds upon these foundations and introduces several modifications by integrating some other methods used in recent deep time series forecasting, like residual block, Fourier filter.\n\n3.  Section 1 needs improvements. It is clear for me how Koopman operator address the non-linear dynamics, but section 1 does not clearly show how existing work of Koopman theory tackle non-stationary problems. Could the authors elaborate more on line 40-42? In line 52-54, could the authors explain the motivation of replacing reconstruction loss? In line 30-31, the authors introduce \"Non-stationary  ...... in different periods\". However, the proposed method do not show the capability to detect period boundary.\n\n4. The experiments are comprehensive, but it would be better to design experiments revolve around the topic and the proposed method. Ablation studies can successfully validate the effectiveness of the design of local and global model, but the experiment associated time and memory cost analysis might lose focus. The authors can also explore the effect of number of block to verify the capability of hierarchical disentanglement.  \n\n5. It would be better to discuss the related work of non-stationary time series forecasting, like regime switching state-space models, time-varying state-space models, etc.\n\nQUESTIONS:\nhow to set the hyperparameters  D?\n\nLIMITATIONS:\nN.A.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors address the issue that most real-world time-series are non-stationary. To tackle this problem they introduce Koopa, a globally learned (time-invariant) and localized (time-variant) linear Koopman operators, to exploit respective dynamics underlying different components. In practice, Koopa is composed of stackable Koopa Blocks, where each block learns operators hierarchically by taking the residual of previous block fitted dynamics as input. The authors evaluate the proposed model on variety of time-series data sets and compare their method with transformer and convolution based deep architectures. Koopa outperforms or performs on par with the existing methods, yet has a significantly lower computational costs. Moreover, the proposed method can be use in settings where the forecast horizon must be increased beyond the training set-up.\n\nSTRENGTHS:\nOriginality: \nThe related work section in well written. It covers both relevant model types in the field, for (a) time series forecasting, TCN, RNN and transformers are mentioned, as well as the authors position the current work in its context; for (b) Koopman operators, the authors provide an overview of the related work and how their work differs. \nThe work proposes a novel method leveraging Fourier analysis to disentangle time-series components, eDMD for estimating time-variant Koopman Predictor, and time-invariant Koopman Prediction as a learnable parameter. It differentiates itself from KAEs due to its block like structure, where the residual of the previous block is fed as input into the next block. \n\nQuality: \nThe submission is techinncally sound. The authors build upon Koopman operator theory and Wold's Theorem. In addition, experimentally they validate whether their proposed sub-components of Koopa, Fourier Filter, truly disentangles time-variant from time-invariant series. As well as provide an ablation on Koopa structure. \n\nClarity: \nThe work is well written providing sufficient relevant background knowledge on Koopman operator theory also for reader who is unfamiliar with this theory.  \n\nSignificance: \nThe obtained results are significant. The proposed method outperforms or performs on par with existing time-series forecasting methods. More importantly, the method is very efficient with respect to training time and memory costs compared to transformer or convolution based methods. Lastly, the authors showcase that their method is applicable for scaling up forecast horizon, where standard deep learning architectures fail. In all, based on the submission the author's suggest a very promising modelling technique, however, I am not an expert of Koopman operator theory (nor the works in this sub-domain).\n\nWEAKNESSES:\nClarity: \nWhen first mentioned, I would recommend using the complete name of the model type rather than only the acronym, for example, temporal convolutional network rather than just TCN, it makes it easier for the reader to read.\n\nQUESTIONS:\nFor time-variant KP how do you choose S, is the Time-variant KP sensitive to S?\n\nLIMITATIONS:\nThe authors have not addressed limitation of their work. I would suggest the authors to include such section in the final version of their work.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses non-stationary time series forecasting, in which case the temporal distribution of time series is changing over time. Koopman theory is introduced in this paper for its fundamental capacity on modeling dynamical systems and a Koopman forecaster called Koopa is proposed. Concretely, in this paper, time series is disentangled into time-variant and time-invariant components. For time-invariant component, Koopa introduces a learnable matrix as Koopman operator, and for time-variant component, Koopa leverages eDMD to find the best fitted matrix that advances forward the system. The proposed framework is technically sound but somewhat lacks novelty.\n\nSTRENGTHS:\n1. Non-stationary time series forecasting is challenging and it is reasonable to apply Koopman theory to tackle this task.\n2. Disentangling time series into variant and invariant components is intuitive.\n3. The methods of constructing Koopman operators for both time-variant and -invariant components are straightforward.\n\nWEAKNESSES:\n1. The proposed model lacks novelty. Both disentangling time series and applying Koopman theory are commonly used techniques in time series analysis. \n2. The technical contribution of this paper is limited. The extraction of time variant and invariant components is too simple and lacks analysis. The application of Koopman theory is simply the construction of Koopman operators.\n3. Hierarchically disentangling time series lacks intuition and motivation. What's the point of further disentangle variant component into invariant and variant components?\n4. The experiment does not involve highly non-stationary datasets.\n5. Koopman operators are highly correlated to measurement functions. In the branch of time variant component, although the model constructs diverse Koopman operators, the measurement, i.e., the encoder, is static. For more technical contributions, a deeper integration of Koopman's theory is needed.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nNot applicable.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studied time series forecasting problem for weather forecasting, energy consumption, and financial assessment. To propose a model that generalizes on varying distribution, the authors proposed Koopa which is composed of modular Koopman predictors to describe and advance forward series dynamics. The motivation is Koopman-based methods are appropriate to learn non-stationary time series dynamics. \nThe implementation is straightforward in the supplement material. The experimental results show the effective performance of the proposed  methods compared with state-of-the-art methods.\n\nSTRENGTHS:\nThe paper proposed a novel method to solve the time series forecasting problem. The introduction of Koopa Block is interesting. The experimental results show that the model outperforms existing methods with more efficient memory usage.\n\nWEAKNESSES:\nIt is interesting to see more analysis about Koopa blocks. For example on Figure 5, authors showed the visualized blocks are different which corresponds to different curves. But it is clear that the curves of K1, K2, and K2 share the similarity that they goes down first, then goes up. It is interesting to see the correlation analysis if there is some. \n\nThe experiments show the proposed method achieved better performance with efficient training time and memory usage. I am not sue if there is convergence analysis and what are the parameters, such as the number of epochs for training, batch size.\n\nQUESTIONS:\n1. Is there any correlation between Koopa blocks if the curves share similarity as shown in Figure 5 ?\n2. How is the convergence of the proposed method and compared methods?\n3. What are the parameters, such as the number of epochs for training, batch size for experiments?\n\nLIMITATIONS:\nThere is no theoretical analysis but the main contribution is to propose a novel architecture.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}