Augmented Invertible Koopman Autoencoder for long-term time series forecasting
Authors: Anthony Frion
Lucas Drumetz
Mauro Dalla Mura
Guillaume Tochon
Abdeldjalil AISSA EL BEY
Venue: Accepted by TMLR ()

Abstract: Following the introduction of Dynamic Mode Decomposition and its numerous extensions, many neural autoencoder-based implementations of the Koopman operator have recently been proposed. This class of methods appears to be of interest for modeling dynamical systems, either through direct long-term prediction of the evolution of the state or as a powerful embedding for downstream methods. In particular, a recent line of work has developed invertible Koopman autoencoders (IKAEs), which provide an exact reconstruction of the input state thanks to their analytically invertible encoder, based on coupling layer normalizing flow models. We identify that the conservation of the dimension imposed by the normalizing flows is a limitation for the IKAE models, and thus we propose to augment the latent state with a second, non-invertible encoder network. This results in our new model: the Augmented Invertible Koopman AutoEncoder (AIKAE). We demonstrate the relevance of the AIKAE through a series of long-term time series forecasting experiments, on satellite image time series as well as on a benchmark involving predictions based on a large lookback window of observations.

Published in Transactions on Machine Learning Research (05/2025)
Augmented Invertible Koopman Autoencoder for long-term time series forecasting
Anthony Frion
anthong.frionSherenn.cm
Institute of Coastal Systems- Analysis and Modeling Delmhollz-Zentrum Zentrum Hereon, Geesthachl, Germany
Lucas Drumctz
bucas. hhos.ddrmmmttahh---llaanninpppp
IMT Atlantique
Lab-STICC, UMR CNRS 6285, Brest, Pronce
Mauro Dalla Mura
Université Grenobles Alpes Grenoble INP
GIPSA-lab, Grenoble, France Institut Universitaire de France
Guillaume Tochon
guillaumee.cchoon@bddd.........
LRE EPITA, Le Kremlin-Bicêtre France
Abdeldjalil Aissa El Bey
abdeldjalil obeellicck.issalttxieinnteeteiie
IMT Atlontique
Lab-STICC, UMR CNRS 6285, Brest, Phonce
Reviewed on OpenReview: https: // opt.eevee.....eeeeeoeee net/ forum? id= o6ukhJLaaM
Abstract
Following the introduction of Dynamic Mode Decomposition and its numerous extensions, many neural autoencoder-base based implementations of the Koopman operator have recently been proposed. This class of methods appears to be of interest for modoling dynamical systems, either through direct long-lerm prediction of the evolution of the state or as a powerful embedding for downstream methods. In particular, a recent line of work has developed invertible Koopman autoencoders (TKAEs), which provide exact reconstruction of the input state thanks to their analytically invertible encoder, based coupling layer normalizing flow models. We identify that the conservation of the dimension imposed by the normalizing flows is limitation for the IKAE models, and thus we propose to augment the latent state with a second, non-invertible encoder network. This results in our new model: the Augmented Invertible Koopman AuloEncoder (AIKAE). We demonsttate the relevance of the ATKAE through a series of long-term time series forecasting experiments, on satellite image time series well as on benchmark involving predictions based on large lookback window of observations.
1 Introduction
A longstanding question in dynamical systems theory has been the ability to characterize the behavior of dynamical systems from which one does not have access to the equations that govern their evolution, but only data snapshots mcasured from them. With the incrcasing computational resources and the development of autodifferentiation frameworks, data-driven methods, and specifically deep neural networks, have taken increasing importance in dynamical systems modeling.

---

Published in Transactions on Machine Learning Research (05/2025)
Among these neural network methods, an increasing part has been designed based on Koopman operator theory, which means that they seek to find representation of the state from which the evolution through time can be described linearly. A popular class of such models is the Koopman autoencoder (Lusch et al.) 2018). which simply consists of a neural autoencoder along with a matrix that describes the linear dynamics in the latent space of the encoder. Many flavors of the Koopman auloencoder seek to improve the long-term stability of the linear latent dynamics through constrained parameterizations of its governing matrix (Bevanda et al. 2022 Fan al. 2022 Zhang et al., 2024) or additional loss function terms (Azencot et al., 2020: Frion al. 2023a Some works propose to use the representation learned by Koopman autoencoder in a broader computation pipeline, for example as embeddings for a Transformer model (Geneva & Zabaras 2022 Jin
et al. 2023) or in a data assimilation framework (Frion et al.) 2024 Singh et al. 2024). In the present work, we take interest in recent advancements (Meng et al. 2024 Jin el al. 2023) consisting in implementing the Koopman autoencoder with a coupling layer normalizing flow as the encoder and the analytical inverse of this flow as the decoder. We show that the induced constraint on the dimension of the latent space is detrimental to the ability of the model to find a Koopman invariant subspace. As a remedy, we propose to learn a second encoder in order to inflate the latent dimension of the model, without changing the architecture of the decoder. The resulting model is our Augmented Invertible Koopman AutoEncoder (AIKAE).
We perform long-term forecasting experiments with this model in two settings. First, we work on regularly- sampled time series with no missing observations, where one has access to numerous past observations in order to compute a forecast, For this setting, we show that a delayed AIKAE, i.e. a AIKAE from which the input space contains multiple consecutive observations rather than a single one, can obtain accurate results, challenging set of strong and recent. baselines. Then, we work on satellite image time series, where there are usually lot of missing observations, resulting in irregularly-sampled data. In this context, we use pre-trained AIKAE model as dynamical prior in a constrained variational data assimilation framework. We show that this model performs better in this task than other Koopman autoencoder variants. The codes associated to our experiments are available at htttss:/
cww http//aatooo-fffiion AAIKE
The remainder of this paper is organized as follows: in section 2 we review Koopman operator theory and the recent related neural network- bbased models. In section 3 we introduce our new architectures, including the AIKAE architecture in subsection 33.1 and the delayed Koopman autocncoders in subscction 3.2 In section we show how an AIKAE can be used as a dynamical prior in variational data assimilation cost. Our experiments on long-term forecasting with a fixed lookback window and on assimilating satellite image time series are respectively presented in sections and Section 7 concludes our work.
2 Background and related works
Originally introduced in Koopman (1331p, Koopman operator theory has known renewed interest in the last few decades, starting from the work of Mezić (2005). We refer the interested reader to Brunton et al. (2022) for an extensive review of Koopman operator theory and its applications. In a few words, this theory states that any dynamical system, regardless of its inherent complexity, can be described by a lincar operator, although at the cost of an infinite dimension in the general case. More precisely, let us introduce a (supposedly autonomous and deterministic) dynamical system from which the state at a given time can be described by a n-dimensional variable x E X R The system is defined by a discrcte-time cvolution operator F X + X. Then, assuming that the state of the system at an integer time is x, X, we define
x - F(xi).
The Koopman operator IC is such that, for any mcasurement function g:X- -+ and initial condition X
Kg(xx)) a (g ():) = g(x+1))
Thus, in theory, one would simply need to have access to the expression of XC for the canonical measurement functions (i.c. the projections of the full state x onto its n variables) to be able to exactly characterize the dynamical system F. However, the infinite dimension of the space of measurement functions means that the Koopman operator is itself infinite dimensional, and therefore often difficult to describe in practice.
2

---

Published in Transactions on Machine Learning Research (05/2025)
For this reason, most of the data-driven methods inspired by the Koopman operator consist in finding approximation of this operator one would require this set to be invariant by the Koopman operator. This would mean that, for any of the functions 9i, there would cxist coefficients ky such that, for any initial condition X, E X,
specific d-dimensional set of measurement functions (9I,..., 9d). Ideally,
Kg.(x;) 9:(((x)))) =
Σ (x)))
(3)
In this case, the action of the Koopman operator on the subspace spanned by (91,-~99a) could be simply described by matrix K E Rdxd, built with the coefficients kjj. For linear dynamical systems, the space spanned by the canonical measurement functions of the system (i.c. the functions constituting the state variables) is obviously invariant by the Koopman operator. For nonlinear dynamical systems, there are some cases in which a dimite-dimensional Koopman invariant subspace containing all the state variables (in addition to some 'augmentation' variables, required to obtain the linearity) are known. Examples such dynamical systems are detailed in e.g. Brunton et al. (2016) and Kutz al. (2016). However, most of the time the Koopman invariance has be approximated to certain degree, since in the general case there exists no finite-dimensional Koopman invariant subspace for the system. Once set of measurement functions (g1....,gd) ga) has been designed, one lypically looks for the matrix K that minimizes the residual error of the multiplication by matrix K. Formally, one can work with set of data X = (x1,..,, Xr), with a time-shifted version Y = (xi,...,x x1), where, for any index S t T, x = F(x.). Then, we seek to find
K" = arg min |g(())g(XX))) KeRaxd
- g(Y)
(4)
where we use the notations
(x(x)) 91 (X2)
91 (xr)
(x(x)) (x(x)
(x
g(X)=
g(Y)=
(5)
ga(xi)
(x2)
ga(xT)
sa(x()
ga(xb)
ga(x;))
The optimization problem of equation can be solved using the well-known least-squares solution:
K" ggY))
(6)
where denotes the Moore Penrose pseudoinverse. It should be noted that this solution only accounts for the advancement of one time step, i.e. one iteration of the discrete dynamics F. Hence, the obtained model will generally perform poorly in long-terna mergtermm predictions. For this reason, while early Koopman- based methods such as dynamic mode decompasition (Schmid) 2010) and extended dynamic mode docomposition (Williams) et al. 22115) compute the least-square solution of equation (or low-rank approximation of it), subsequent neural network-based implementations generally leverage trajectories with multiple time steps in order to train model that produces accurate long-term prodictions.
To sum up, many practical implementations of the Koopman operator consist in finding a set g X -> Rd of d measurement functions (gI,...,ga)) each from the state space X' to R, and a matrix K E Rddd that approximates the restriction of the Koopman operator to the subspace spanned by these functions. We have mentioned the importance of choosing a set of measurement functions that span an (approximately) Koopman invariant subspace. Another important aspect of these methods is the ability to faithfully roconstruct an input state E X' from its embedding g(x) RRd, and to do the same for the time-advanced embeddings Kg(x), in order to produce predictions for the evolution of the state vector from any initial condition. Thus, one must be able to define (possibly approximated) function f Rd + X such that the composition f is (approximately) equal to the identity function. The reconstruction abilities of the recently introduced classes of Koopman-based met are discussed in Jin et al. (2024). We defer a detailed discussion of older methods to appendix A and directly discuss Koopman autoencoders (KAEs), a class of methods introduced by Lusch et al. (2018) and extended by numerous subsoquent works, c.g. Otto Rowley (2019); Li et al. (2020 Azencot et (2020): [ermaan al. (2023); Frion et al. (2024) among many others. These methods model the Koopman invariant subspace through the means of neural autoencoder, which does not directly
3

---

Published in Transactions on Machine Learning Research (05/2025)
include the state variables. A neural autoencoder simply consists of two neural networks, ø and s) each with its set of trainable parameters, from which the composition is approximately equal to the identity function, (xx) s x. For KAE models, the encoder & : R= + g learns a non-trivial Koopman invariant set & l.e of measurement functions, while the decoder & : R + R" learns to reconstruct the state space from the latent representation of Φ. Depending on the implementations, the Koopman matrix K E R learned alongside the parameters of á and ta or obtained separately through the resolution of a least squares problem as in equation The predictions of KAE model after т time steps are computed as:
X(+r &ett cK (xx)).
While general neural oencodr models were originally introduced for the purpose of reducing the dimension of the input x (i.e. following the property d < n), in the context of finding a better representation of the Koopman operator, it might actually be beneficial to learn a latent representation with a higher dimension than the input (i.e. d > n). In practice, the latent dimension d should be regarded as an important parameter for the design of a KAE model.
Although the Koopman autoencoder framework enables for a high expressivity in the search of a suitable Koopman invariant subspace, it has the notable inconvenience that, in contrast to earlier methods, it com- putes an approximated rather than exact reconstruction of the input state. In practice, in the loss function for training a Koopman autoencoder, one should include a reconstruction term to ensure that the components @ and & indeed constitute an autoencoder. This loss function term is to be minimized in conjunction with the prediction loss term and to the linearity loss term (see e.g. Lusch et al. (2018)), ), which leads to a complex loss landscape, and possibly to difficultics in adjusting the relative weights of the loss function terms. For this reason, a recent line of work (Jin et al. 2023 Meng al.) 2024 Jin et al., 2024) has investigated the substitution of the neural autoencoder (ø, rb), by an analytically invertible d with its exact inverse Φ-1 Tn this case, the predictions of the model are given by:
(8) This results in subclass of methods which we call invertible Koopman autocncoders (IKAEs). More specifically, they proposed to implement & with coupling- normalizing flow models (Dinh et al. 2014 2017, Kingma & Dhariwal 2018). These models have several interesting properties. Notably, they indeed have an analytical inverse transformation, cnabling an (algobraically) exact reconstruction" of an encoded state x. In addition, their Jacobians are tractably computable, which gives them a potential for stochastic modeling. We provide more background on this and on normalizing flows in appendix B
X1+7 R x (KT'((T
=
Another important property of coupling-layer normalizing flows is that they always preserve the dimension of the input state, which is a necessity in order for the change of variable formula to be applicable. This may be detrimental for IKAE models since, as previously mentioned, one often needs to inflate the dimension of the state space in order to obtain a good approximation of the Koopman operator. To alleviate this issue, the authors of the existing IKAE models have proposed either before (Meng et al. 2024) or in-between (Jin et al. 2023) the coupling layers of the normalizing flow. However, this approach means that the resulting model will learn function that enables to reconstruct these added zeros by design, while only the reconstruction of the true stale variables is of interest. In addition, the variable
concatenate zeros to the state vector,
operation of concatenating zeros to the state vector prevents a direct application of the vhaige-ofvvrrialee formula from equation 19 hence reducing the possibilities for stochastic extensions.
3 Our proposed Koopman autoencoder architectures
3.1 Augmented invertible Koopman autoencoder
We mentioned in the previous section that the base IKAE architecture had the inconvenience of not enabling to learn a large enough set of measurement functions to obtain a sufficiently good approximation of the
It should be uoted that the algebraic invertibility does not guarantee stable and accurate roconstructions in practice, since normalizing flows can be subject numerical ill-conditioning: see e.g. Lee al. 2021 for an extensive discussion the conditioning of normalizing flow models.
4

---

Published in Transactions on Machine Learning Research (05/2025)
Z1
Φ -1
X Z
[+1
x
K
X
x
1
Figure 1: Graphical representation of the AIKAE architecture. is coupling layer normalizing flow with an analytical inverse o X is neural network (generally a simple multi-layer perceptron in practice) and K is a matrix.
perallei computation
2
X+
Φ
Z
2
Φ -
1
Xt+1
K
X
E+11
Zi-11
Zy+2
z 1+2
Φ
-1
X++2
K
L+2
Figure 2: Graphical representation of the prediction over 2 future timesteps with an AIKAE model. This can be casily generalized to long-term predictions. As can be seen, the forccasting computations are all performed in the latent space, and the predicted latent states can all be decoded in parallel for more efficiency.
Koopman operator. Thus, in this section, we propose a revised architecture in which the invertible encoder of these models is augmented with second neural encoder X. which enables to capture richer set of measurement functions while keeping the invertibility of the model. We call the resulting architecture an augmented invertible Koopman autoencoder (AIKAE). This architecture is represented graphically in figure [ Additionally, we represent the prediction over time steps in figure 2 and the generalization to an arbitrarily long T time steps prediction follows directly from it. Since x does not have to be invertible, it can be implemented with any neural network architecture.
The predictions performed by the AIKAE model over time steps from an observed initial condition x, can be described as follows:
zzj
()(x)) (
2,
=
(9)
5

---

Published in Transactions on Machine Learning Research (05/2025)
2i77 Kizt Z 8147 1 (0
244
(10)
(11)
The innovation of the AIKAE model in comparison to the IKAE model is that introduce second encoder X : R + RR, which we call the augmentation encoder. The latent state 2, corresponding to X is thus obtained by concatenating an augmentation encoding z = =x(x,) to the invertible encoding zj d(x;) produced by the unchanged normalizing flow model ø IR" + IR", as summarized in equation g Then, the linear latent dynamics is defined by the multiplication of the full latent state 2t by the matrix K, which is now of size == n + p, as summarized in equation 10 Hence, by adding an augmentation part to the encoding, we indeed increase the number of measurement functions included in the latent space, and the dimension of the approximated Koopman operator K as consequence. Finally, in order to go back to the input space after any desired number r of itcrations, one can decode the invertible part z of Ze+r; as shown in equation 11 Note that, with our notations, the operations z and z respectively correspond to projections on the first n or the last variables of z E RR+P.
From studying these equations, one can see that the augmentation part z of the initial latent state has no influence on the direct reconstruction & (i.e. the case where 7 0), which is still algebraically exact thanks to the analytical inverse  of Φb However, evidenced by equation 10 z; has an impact on the subsequent invertible parts of the encoding through the multiplications by K if and only if z is not in the nullspace of the upper-right block of K. An immediate corollary of this observation is that the upper-right block of K should be nonzero in order for some information to flow from zz to 2 and subsequent invertible encodings. In fact, should the last p columns of K be zero, then the augmentation part zi would have no influence on the predictions in the state space, which means that the whole model would be equivalent to non-augmentedd IKAE model. Thus, we have the intuitive result that the AIKAE architecture is generalization of the IKAE architccture. In addition, one may interpret the invertible and augmentation parts of encoding 2, 8s disenlanglement between the 'static features' and the "dynamical features'. This interpretation is particularly interesting when performing data assimilation using the methods of section 4 In practice, the output size p of x determines the latent size d = n of an AIKAE, making it an important hyperparameter, similarly to the latent dimension d itsolf for non-invertible KAE models.
In order to characterize the predictions by an AIKAE more compact way, we introduce the global encoder Φ R + Rd which corresponds to the concatenation of the invertible encoder & and the augmentation (xx)). Correspondingly,
encoder X, i.e. the application of h-1 on the invertible part (i.e. the first n variables) of a latent vector. Thus, equations 9 to 11 can now be summarized as
have that the global decoder 4-1 : Rd - R consists in
8E+ = x (K<)((x)))
(12)
As the notations suggest, --1 is still an analytical left inverse of 4, as h- o 4 corresponds to the identity function. One should however be aware that the reversed composition $ o Φ-1 is not an identity function since the information on the augmentation part of the encoding is dismissed when computing -- Thus, s is not right inverse of +.
3.2 Delayed Koopman autoencoders
We now discuss delayed Koopman autoencoders, which simply consist in KAE models that take as their input state concatenation of m consecutive observed states from a dynamical system rather than state vector corresponding to specific time index. We refer to this approach as delay embedding. Formally, when observing a long time series (Xo,... ... XT) xT+,, rather than directly using a state x, E R² as the input to a KAE, one may alternatively use
Xtm
Y: =
E R
(13)
Xtm+m-1
as the input to the model. Then, the dimension of the input space of the model will be n' nm. Thus, in order to avoid manipulating very high-dimensional input state (taking into account the fact that the
6

---

Published in Transactions on Machine Learning Research (05/2025)
2;
14+1
Vo+1
Nire
X
X 1lee
1
K
X
2 L+11
Figure 3: Graphical representation of an ATKAE model with delay embedding of the state. The model is the same as in figure I except that the input (and output) of the model now contains multiple consecutive states x, stacked along single dimension.
latent space should be at least as big as the input space in order for the encoder to be invertible), it is more convenient to do so when the original dimension n of the input space is small. In this regard, the case of univariate time scries (i.c. n = 1) is of particular interest. A graphical representation of an AIKAE model using this delay embedding strategy is shown figure 3 This representation can obviously be generalized to predictions over multiple time steps (of the variable y, each representing m time steps for the original
variable x), as in figure 2
In practice, using such a delay embedding of the state may increase the predictability when the know ledge of a single observation x, is not sufficient to predict the subsequent states, i.e. in cases where x, is not actually the state variable of dynamical system. The use of delay embedding is commonplace data-driven signal processing, as the well-known Takens theorem (Takens 1981), guarantees an increased predictability of the system when the size of the delay embedding increases.
In particular, the use of delay embedding for DMD was proposed by Tu et al. (0014). Along with the subse- quent works of e.g. Le Clainche & Vega (2017); Kamb et al. 2020);; Yuan et al. 2021], they demonstrated the ability to model a higher number of Koopman modes, and an increased robustness to noise in the observed data. However, to the best of our knowledge, our work is the first to propose using a large delay embedding for neural network-based implementation of the Koopman operator. Experiments involving this approach are presented in section
4 AIKAE as a variational data assimilation prior
Variational data assimiliation consists in inferring the full state of system time, by leveraging set of partial and noisy observations as well as some prior knowledge on the dynamical behavior of the system, often in the form of a dynamical model. Concretely, the assimilated state is obtained by minimizing a variational cost that comprises a term of fidelity to the observed data and a term of fidelity to the prior knowledge. This cost. is minimized using some form of gradient descent. algorithm. Traditionally, the prior knowledge comes the form of a complex physical model, which can be differentiated using adjoint methods (see e.g. [Bannister (2017) ). A rich line of work has recently investigated the minimization of a variational cost using autodifferentiation frameworks, either by re-implementing physical models in such frameworks (see Gelbrecht et al. (2023) for a review) or by substituting this physical prior by a data-driven one (Nonnennacher & Greenberg 2021 Fablet et al. 2021). We refer the interested reader to Cheng et al. 2023h for a review of the methods combining machine learning and data assimilation.
Here, we show how to use a pre-trained AIKAE model as prior for variational data assimilation, taking inspiration from the work of Frion et al. (2024). In few words, this method consists in finding the initial latent state of the model that enables to most closely fit. set of observed states with associated timestamps.
7

---

Published in Transactions on Machine Learning Research (05/2025)
Although it was originally introduced for regular (non-invertible) KAE models, it can be straightforwardly adapted to TKAE and ATKAF models, and thus we hereafter explicit the ATKAF case only.
Suppose that we have at disposal a trained AIKAE model, with its components (é, x) - & and K. In addition, we observe a trajectory of data through a set of T points "(,,)). with the associated time indexes (fo....,IT) E NT++1 supposed to be arranged increasing order with 40- = 0 for convenience. Note that the timestamps could be chosen to be non-integers if we use the matrix logarithm of K, as explained in Frion et al. 2024 In order to fit the observed datapoints, one can solve the following optimization problem:
min Zo E
Φ 1 (K' - -x,,12.
(14)
This method corresponds to strong-constraing variational data assimilation scheme, where the chosen dynamical prior is the pro-trained AIKAE model. In practice, it can be solved using autodifferentiation, leveraging the fact that the prior is fully differentiable. Once the (approximated) solution zg is found, one can query the predicted state at any time t by simply computing
& - =
(15)
Then, depending on the time steps for which we are interested in the predictions, this framework may enable to solve denoising, interpolation, forocasting or all these tasks once. In particular, the denoising capabilities of well-trained model stem from the assumption that any trajectory produced by this model is physically consistent. Thus, the model is expected to be unable to exactly fit set of noisy observations (Xto...,,,, xt,) through equation [4. but to instead produce the physically consistent trajectory that best matches these points, which should remove the noise in the observations.
In order to adapt this method to a pre-trained IKAE model with components & and K, one would simply have to substitute & to & equations [4 and 15 Interestingly, since the latent space of an IKAE is in bijection with the state space, the exact equality of its trajectory to an observation at one given timestamp deterministically gives the remaining of the time series. To illustrate this remark, suppose that constrain the equality of the predicted initial state to the initial observation in equation 14 Then, we can solve
T
2 = min Σ
(K*z - x1.
E
(16)
1=0
s.t.
(z)) X0.
For AIKAE, we have that the constraint (z)( xe is respected if and only if z d(xu). Thus, equation To is equivalent to unconstrained oplimization problem on x E IRP. If K has a nonzero upper- right block (i.e if different values of z can influence the invertible parts of the subsequent latent states in different ways), then multiple trajectories are admissible, making this problem nontrivial. In contrast, when adapting equation 16 for an IKAE, since there is no angmentation encoder, the only possible value for Zu is zj = Φ(xo), which is the same one as in the direct inference in equation g taking no account of any of the subsequent observations. Overall, one can see that an AIKAE can produce several different trajectories that exactly match an observed initial state while IKAE is not able to do so.
5 nong-term time series forecasting experiments
In this section, present experiments set of popular long-term time scries forecasting datasets. Some- times called the 'Informer benchmark' as reference to the work of Zhou et al. (2021) that popularised it, it is comprised of the ETT datasets (including the subsets ETTh1, ETTh2, ETTm1, ETTm2), ECL, Exchange, Traffic and Weather. These datasets have been extensively used in the last few years to evaluate the performance of the recently introduced long-term time series forecasting models, including Transform- ers (Zhou et al. 2022 Nie et al.) 2002),, convolution-baseo methods (Wu al.) 2023) and linear models (Zeng et al. (023). We refer the interested reader to Wang et al. (2024) for a recent assessment of the rapidly evolving state of the art this benchmark. The long-term time series forecasting task consists in predicting
8

---

Published in Transactions on Machine Learning Research (05/2025)
the state of time series over prediction length of Tp timesteps, using as input lookback window of Tr. preceding states. In practice, Tr and Tp are typically in the order of 100 time steps. Although the considered datasets consist in multivariate time series, it has been observed that using the information of a single variable over all time steps in a lookback window enables to obtain better performance than when considering the information of all variables at a single time slep. For this reason, some of the best performing methods consist. in either only one single univariate model that used on every variable of the dataset (Zeng et al., 2023 Li et al.) 2023), or in one univariate model for each variable (Nie et al., 2023). In particular, it has been repeatedly observed (see e.g. Zeng et al. (2023) Li et al. (2023); Toner & Darlow (2024] Han et al. (2024) that simple linear models significantly outperform early Transformer models such as Informer (Zhou ett al.
2021) on this benchmark. In addition, as underlined by Wang et al. (2024), sequential models such
as long short-term memory networks (LSTM, Hochreiter (197)) typically struggle to caplure the long-term relationships compared to models that process the lookback window all at once. Using these insights, we propose to solve the long-tene time scrics forccasting task with univariate delayed Koopman autocncoders, as described in subsection 3.2 rather than with classical KAE that would use a single (multivariate) obser- vation as its state space. Interestingly, delayed KAE model may be seen generalization of the simplest linear model proposed by Zeng et al. (2023). In few words, this linear model consists in directly finding a matrix W E RTTxTT representing a linear relationship between the observed lookback window X E R and the corresponding output. Y E RTr__ Although W is found through stochastic gradient descent, this met hod is reminiscent of DMD with a delay embedding. Indeed, when supposing TŁ = Tp, these two approaches are equivalent. In this regard, our delayed KAE approach is an additional generalization where the linear relationship is computed a latent space defined by a nonlinear encoding through & of the delay embedded state, rather than directly on this state. Thus, it will be of particular interest to assess whether the addi- tion of nonlinear encoder with an IKAE or AIKAE model enables to improve the forecasting performance, knowing that linear models have been observed to perform surprisingly well for the datasets that we consider.
We compare our delayed IKAE and AIKAE models against a set of strong and recent basclines representing several popular classes of models for long-term time series forecasting:
The DLinear model (Zeng el. al. 2023 is a variant of the previously discussed linear model, which leverages trend-season decomposition of the lookback observations rather than the direct lookback window of observations.
PatchTST Nic et al. 2023 is a Transformer model, which decomposes the input time scries into patches each containing information on several time indexes. It also treats each channel of the multivariate time series independently instead of mixing their information.
Timesnet (w et al. 2023 is a convolution- based method. It consists in building 2-dimensional representations of the time series by reshaping the input data according to its main frequencies, and processing these representations using convolutional neural networks.
iTransformer (Liu et al. 2024 is a Transformer model in which the feature and time dimensions are switched, which has been shown to enable better performance than all of the previously proposed variants of the Transformer model.
Following standard evaluation conditions (see e.g. Wu et al. (2023): Liu et al. (2024)) ), we test our IKAE and ATKAE models with lookback window of size Ty = 96, and 4 lengths Tr of the prediction window: 96, 192, 336, 720. Thus, the size of the invertible part of the latent space is Tr, 96. For AIKAE, the augmentation part of the latent space is of size 32, leading to global latent space of size 128. We use reversible instance normalization (RevIN, Kim et al. 2021 for IKAE and AIKAE, as it reported to improve the performance of multiple long-term time series forccasting models. RevIN performs channel- wise normalization of the input data, with 2 learnable parameters for each channel. In order to ensure the reproducibility of our results, we use a fixed random seed to initialise all IKAE and AIKAE models. The training performed with the Adam algorithm with a learning rate of 10-3 and momentum parameters 3 (0.9, 0.999). A detailed account of the architectures and hyperparameter search for our models is deferred to appendix] D.
9

---

Published in Transactions on Machine Learning Research (05/2025)
Table 1: Forecasting mean squared errors (MSEs) and mean absolute errors (MAFs) for various models and
long-term forocasting tasks. For cach datasct, we use a lookback window of size Tr 96 and prediction horizons Tp of sizes 96, 192,336, 720. IKAE and AIKAE are our own implementations, while we use the
results reported by Liu et al. (2024) for all other models. For each task and metric, the best result is in bold
and the sccond best rosult is underlined.
Model
IKAE
AIKAE
iTransformer
PatchTST
TimesNet
DLinear
Mutric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.159
0.252
0.153
0.247
0.148
0.240
0.181
0.270
0.168
0.272
0.197
0.282
ECL
192
0.173
0.265
0.167
0.259
0.162
0.253
0.188
0.274
0.184
0.289
0.196
0.285
336
0.189
0.282
0.184
0.277
0.178
0.269
0.204
0.293
0.198
0.300
0.209
0.301
720
0.227
0.313
0.223
0.310
0.225
0.317
0.246
0.324
0.220
0.320
0.245
0.333
96
0.460
0.313
0.131
0.297
0.395
0.268
0.462
0.295
0.593
0.321
0.650
0.396
ETTm2 ETTml ETTh2 ETTh1 Weather Traffic
192
0.476
0.317
0.449
0.302
0.417
0.276
0.466
0.296
0.617
0.336
0.598
0.370
336
0.496
0.327
0.467
0.311
0.433
0.283
0.482
0.304
0.629
0.336
0.605
0.373
720
0.524
0.343
0.499
0.329
0.467
0.302
0.514
0.322
0.6400
0.350
0.645
0.394
96
0.157
0.209
0.160
0.211
0.174
0.214
0.177
0.218
0.172
0.220
0.196
0.255
192
0.193
0.245
0.195
0.245
0.221
0.2541
0.225
0.259
0.219
0.261
0.237
0.296
336
0.237
0.281
0.237
0.281
0.278
0.296
0.278
0.297
0,280
0.306
0.283
0.335
720
0.317
0.333
0.317
0.333
0.358
0.347
0.354
0.348
0.365
0.359
0.345
0.381
96
0.386
0.399
0.386
0.398
0.386
0.405
0.414
0.419
0.384
0.402
0.386
0.400
192
0.437
0.433
0.435
0.432
0.441
0.436
0.460
0.445
0.436
0.429
0.437
0.432
336
0.480
0.457
0.482
0.959
0.487
0.458
0.501
0.466
0.491
0.469
0.481
0.459
720
0.575
0.522
0.561
0.516
0.503
0.491
0.500
0.488
0.521
0.500
0.519
0.516
96
0.301
0.348
0.300
0.346
0.297
0.349
0.288
0.338
0.340
0.374
0.333
0.387
192
0.365
0.394
0.365
0.394
0,380
0.400
0.388
0.400
0.402
0.414
0.477
0.476
336
0.394
0.422
0.387
0.416
0.428
0.432
0.426
0.433
0.452
0.452
0.594
0.541
720
0.414
0.443
0.429
0.450
0.427
0.445
0.431
0.446
0.462
0.468
0.831
0.637
96
0.328
0.364
0.322
0.360
0.334
0.368
0.329
0.367
0.338
0.375
0.345
0.372
192
0.365
0.389
0.364
0.387
0.377
0.391
0.367
0.385
0.374
0.387
0.380
0.389
336
0.390
0.405
0.390
0.406
0.426
0.420
0.399
0.410
0.410
0.411
0.413
0.413
720
0.463
0.443
0.465
0.443
0.491
0.459
0.454
0.439
0.478
0.450
0.474
0.453
96
0.184
0.266
0.185
0.267
0,180
0.264
0.175
0.259
0.187
0.267
0.193
0.292
192
0.255
0.313
0.246
0.307
0,250
0.309
0. 0.241
0.302
0,249
0.309
0.284
0.362
336
0.297
0.343
0.303
0.344
0.311
0.348
0.305
0.343
0.321
0.351
0.369
0.427
720
0.382
0.393
0.385
0.393
0.412
0.407
0.402
0.400
0.408
0.403
0.554
0.522
The full results are reported in table 1 The Exchange datasct is excluded from these results since it is the only dataset for which none of the tested models is able to clearly beat the persistence, which is a trivial method consisting in copying the last observed value from the lookback window to the whole prediction window. Classically, models that do not perform better than persistence are considered to contain no relevant information for the task at hand. Thus, we consider comparisons between methods that do not beat the persistence to be irrelevant. Extended results including the Exchange dataset and the persistence baseline can be found in appendix E.1
From table 1 one can see that the AIKAE model clearly outperforms the IKAE model on the datasets for which the amount of training data the highest, ie.. F.C.. and Traffic. In contrast, for the other datascts, AIKAE and IKAE obtain very similar results, which suggests that inflating the latent space with an augmentation encoding brings no improvement to the IKAE. Besides, both the IKAE and AIKAE models obtain very competitive performance against the other methods, ranking first or second for mumerous tasks.
Our results scem promising considering the low numbers of parameters and overall simplicity of our methods. We emphasize that the number of parameters of our models depend on the lookback window length Tr,, but not on the prediction length Tp, since longer predictions are simply obtained by autoregressive multiplications in the latent space. Thus, our models evaluated in table | have at most 110K parameters for IKAE and 175K parameters for AIKAE (for some tasks the parameter count is lower, due to the hyperparameter search described in appendix D). This is two orders of magnitude below most of the Transformer models, which have
10

---

Published in Transactions on Machine Learning Research (05/2025)
tens of millions of parameters (sce e.g. Zeng et al. (0223) and comparable to linear models such as DLinear, which has around 140K parameters for Tp = 720. Besides, extensions of our models with classical time series processing tools such as the Fourier transform (Zhou et al. 2022) or seasonal-trend decomposition Zeng et al.
(0023) might be interesting directions to study in order to further improve the results.
Additional results on this benchmark can be found in the appendices. Specifically, in appendix F.2 we study the influence of RevIN and of different encoder architectures, notably establishing the superiority of AIKAE to the IKAE with zero padding proposed in Meng et al. (2024). In appendix E.3 study the performance of IKAE and AIKAE with varying lookback window sizes and show that, like linear models and in contrast to many Transformer models, their performance consistently improves with an increasing lookback window size. Our observation that the performance of older Transformer methods stagnates or even decreases as the lookback window size increases is consistent with the results of numerous previous works, e.g. [Zngg : Nie et al. (2023): Liu et al. (2024). In appendix E.44 we study the sensitivity of the TKAE and AIKAE models to the random initialization of their parameters, and show that this sensitivity is moderate. In appendix E.5 we discuss the sensitivity of the performance with regards to some hyperparameters, namely the learning rate, the number of coupling layers of the invertible encoder and the width of these layers. Finally, in appendix [E.6] we plot some predictions made by our models, along with the associated context windows and groundtruth.
et al. (2023):
6 Variational data assimilation on satellite image time series
We now move on to a training task involving long-term forecasting of satellite image time series at the pixel level. For reasons which will be shortly explained, this kind of data often involves missing observations, making it more challenging to handle than the time series data from section 5 Thus, contrarily our approach for these previous experiments, we will not make use of the delay embedding strategy explained in section 3.2 Instead, we will train more classical Koopman autoencoder models, which we will then use as dynamical priors in the testing phase, using the assimilation approach described in section [
We work with a dataset of Sentinel-2 image time series, introduced by Frion et al. (2023b) and used as a variational data assimilation benchmark by Frion et al. (2024). These data differ from the time series datasets of the previous section in several ways. Most importantly, satellite images have multiple missing observations that are due to the presence of clouds between the observation satellite and the surface of the Earth. Since we are usually solely interested in modeling the surface, find ourselves with the dilemma of either directly processing as a pre-processing step. In the first case, the time series will be significantly more difficult to process. In particular, one cannot directly observe a lookback window of many previous observations in order to make a long-term prodiction. In the second case, the time scries is significantly casier to process, yet it is made partly synthetic by the interpolation pre-processing step, which will be learned by the model alongside the true distribution of the satellite data. Fortunately, as underlined by Frion al. (2024), the Koopman autocncoder framework is more flexible than most time series processing methods thanks to its ability to learn underlying continuous representation of the modeled system. However, in order to retain this flexibility, one cannot work with large delay embedding as described in section 3.2 or use the other models from the benchmark of section 5 Instead, we will work with a more classical model, from which the input space is built out of only two consecutive observations, the second of which being used to compute a first order derivative. Indeed, as underlined in Frion et al.] (2024), the access to a first order derivative enables to more casily compute short-term predictions since, when the cvolution is smooth enough, one can already obtain a reasonably good approximation by using it to compute an explicit Euler scheme over one time step. Machine learning- based autoregressive forecasting models based on two previous observations are commonplace for tasks such as weather prediction: see o.g. Lam et al. (2023) and Oskarsson et al. (0024).
irregularly sampled time series or interpolating the available observations
2024), 1, on which we will We now describe the forecasting benchmark that was introduced by Frion et al.
test several variants of Koopman autoencoder models including our new AIKAE architecture. We have at disposal satellite images from two spatial areas: the forest of Fontaineblcau and the forest of Orléans. The data from Fontainebleau, which is used as a training area, is regularly sampled in time thanks to a pre- processing Cressman interpolation step. To train KAE model as dynamical prior, we use Ttrain 242
11

---

Published in Transactions on Machine Learning Research (05/2025)
time steps of data, from area of 150 150 pixels. The Sentinel-2 images that compose the dataset are multispectral, which means that they contain richer spectral information than classical RGB images. Namely, the available information for each pixel and time step is reflectance vector of size L = 10, including the classical rod-green-blue spectral bands as well as 7 bands in the infrared domain. We emphasize that we work at the pixel level, which means that the input space of model corresponds to the reflectance vector (and its first order derivative) of a single pixel, and that the pixel trajectories are assumed to all correspond to same dynamical system.
After training, the trained model is used dynamical prior in variational data assimilation framework, as discussed in section E The objective is to accurately prodict the Trest = 100 steps of data that follow the window of training data, by leveraging the observations of this training window. This task is declined on the two areas discussed before. For the Fontainebleau training area, the time series is again regularly sampled in time, and only the capacity of the model to extrapolate to unseen time indexes assossed. For the Orléans area, the data are not interpolated as pre-processing stop, and are thus irrogularly sampled in time. In this case, all observations correspond to actual satellite measurements. The extrapolation lask on this area tests not only the ability of the trained KAE model to extrapolate in time, but also to transfer its learned knowledge to new spatial area with differing dynamics. Besides, the irregular sampling pattern of the observed data for this task is the precise reason for which one cannot resort to model with delay embedding in this experiment.
It should be noted that directly training a model on irregularly-sampled data is preferable and that is possible to do so with a KAE model, as demonstrated by Frion et al. (2024). However, we restrain our study to the case where training is performed on interpolated data in order to match the conditions of the main benchmark proposed by the authors. For this benchmark, we compare 4 variants of the Koopman autoencoder model:
The base KAE model with 2 multilayer perceptron (MLP) networks as ils encoder e and decoder bi as described in Frion et al. (2024
An IKAE model leveraging a coupling layer normalizing flow model as its analytically invertible encoder, as proposed by Meng et al. 2024).. More precisely, the encoder is implemented with the NICE architecture (Dinh et al. [0144). Note that substituting the NICE model by a non-volume preserving encoder such as RealNVP (Dinh et al. 2017) resulted in less stable training procedure, constraining reduction of the learning rate and leading to worse performance.
An IKAE with zero padding, as suggested by Meng et al. 2024 (abbreviated IKAE-zp). This model is identical to the previous one except for the concatenation of zcros to the input state before entering the normalizing flow encoder, hence inflating the dimension of the latent space.
Our AIKAE model, as described in subsection means of learning a second, non- non-invertible encoder X (implemented as a MLP) rather than applying a fixed zero padding.
3.1
where the latent dimension is inflated by the
The size of the input space is n = 2L = 20, and the dimension is augmented by 16, leading to a latent space of size d = 36, for IKAE-zp and AIKAE. The latent dimension is set to d = 32 for KAE, and constrained to d = n by design for IKAE. For each of these models, we train five instances corresponding to five parameter initializations with fixed random sceds. Following the recommendations of Frion et al. (2024). we design loss functions based on 4 terms: the prediction term, the reconstruction term, the linearity term and an additional orthogonality term. While the first 3 of these loss terms were proposed by Lusch et al. (2018) and are standardly used by multiple KAE implementations, the orthogonality term was proposed as way to improve the long-term stability of the predictions, by ensuring that the norms of the latent states stay approximately constant through time. As previously mentioned, the 3 tested invertible models are trained without the reconstruction loss term since their reconstructions are exact by design. The training is performed with the Adam algorithm, with a learning rate of 10-3 3 We use weight decay with a cocfficient of 10-6 6 for training the IKAE-zp and AIKAE models. For the other 2 models, we present results obtained with no weight decay since the usage of weight decay did not improve the performance.
12

---

Published in Transactions on Machine Learning Research (05/2025)
Table 2: Mean squared errors (MSEs) and mean absolute errors (MAEs) obtained by averaging the perfor-
mance of 5 instances of cach model. The models are pre-trained on the Fontainebleau area, and then used as variational data assimilation priors, following equation 17 The KAE model was the only one to be subject
to overfitting when assimilating the Orléans data, hence the additional row 'Orléans (overfit)'.
Model
KAE
IKAE
IKAE-zp
AIKAE
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Fontainebleau
0.00112
0.0213
0.00128
0.0224
0.00106
0.0212
0.00108
0.0204
Orléans (optimal)
0.00346
0.0384
0.00382
0.0399
0.00324
0.0366
0.00297
0.0356
Orléans (overfit)
0.00403
0.0419
N/A
N/A
N/A
N/A
N/A
N/A
The lesting procedure leverages the methods of section | by using the pre-trained model as a variational prior for data assimilation, with the motivation of producing long-term forecast from an observed trajectory.
Typically, for evaluating trained AIKAE instance with components & and K on long-tern meg-term forecasting the
Fontainebleau data, we instantiate equation
14 as
Tirasn
z min
Σ || (K'zo) xx11?
(17)
Z ZR
where E LRNNNN corresponds to the reflectance over an area of N = 100 100 pixels, which is included in the training data. Concretely, since the prior is pixelwise model, this can be seen as N x N separate optimization problems. One could however perform a joint optimization with an additional spatial coherence prior, as proposed by Frion et al. (2024). After z is obtained, the forecasting mean squared error is computed as
Terain Trest tast
MSE
1
Σ
1 (K'z)) -xel?
(18)
Tiest Tiran+++1
and the mean absolute error is computed in an analogous way. This procedure can be simply adapted to the other KAE variants by replacing and x-1 by o, 1b or o-1 when necessary. The procedure is similar for the Orléans area, except that the assimilation cost and the metrics are computed only over time indexes where groundtruth observations are available.
Table 2 displays the obtained results. Additionally, in appendix F.3 we display and analyze forecasting re- sults associated to this task for a randomly selected pixel. When extrapolating on the training Fontainebleau area, one can see that the performances of the models are relatively even, except for the IKAE model. Its worse performance may be attributed to the reduced size of its latent space, which limits its ability to find a proper linear representation of the system. The IKAE-zp variant. seems to partially alleviate this issue as it obtains significantly better performance. The test Orléans area exhibits a stronger contrast between the tosted models. On this area, one can see that the AIKAE model performs bost, followed by the IKAF-zp.
Interestingly, the base KAE model was the only variant which was observed to overfit on its assimilated data on the Orléans area. In other words, the latent initial state z from equation 17 is not necessarily the best initial state for minimizing the forecasting error of equation 18. Concretely, as explained in section 4 we find an approximation to 2 by minimizing the associated variational cost using automatic differentiation with the Adam optimizer. We observe that the best extrapolation performance is obtained by using a relatively small learning rate and fewer gradient descent steps, resulting in a suboptimal minimization of the variational cost of equation [17] In contrast, for all other models, the best extrapolation result is always obtained by minimizing the variational cost, which means that the assimilation scheme does not overfit the assimilated data. To illustrate this difference, we report two results for the KAE model on the Orléans area: the first one (mplimal) is with the optimal variational assimilation hyperparameters, as reported by Frion el al. (2024). The second one (overfit) is with the hyperparameters that minimize the cost of equation [7 The tendency to overfit on the assimilated data is a critical flaw in practice since, in real conditions, one cannot fit the assimilation hyperparameters using future data which are actually not known, and one will simply choose the hyperparameters that best fit the available data. Since the conditions for training and testing all 4 models
13

---

Published in Transactions on Machine Learning Research (05/2025)
are very similar, the fact that the other models do not overfit the assimilation data should be attributed to increased regularity enabled by their invertible encoders.
Additional results on AIKAE models for satellite image time series forecasting are presented in appendix F More precisely, we study the influence of the size of the augmentation encoding प in section F.1 and variants of the lincarity loss in section F.2 In a few words, the conclusions of these experiments are that. 1) the performance of AIKAE models increases with the augmentation size until it eventually decreases due to overfitting, and 2) it is important to compute the linearity loss over both the invertible and the augmentation parts of the AIKAE encoding, with unweighted least squares implementation.
7 Conclusion
We have experimentally shown that, for the recently introduced invertible Koopman autoencoder (IKAE) models, the analytical invertibility of the encoder can limit the practical ability to learn Koopman invari- ant subspace, i.c. a latent space in which the system dynamics can be described lincarly. Consequently, we proposed to augmeent these models with a new non-invertible encoder, resulting in our model: augmented invertible Koopman autoencoder (ATKAF). We have shown how recently proposed variational data assimi- lation schemes leveraging Koopman autoencoder models can be easily extended to the AIKAE architecture, enabling to work in difficult contexts where the observed data may be incomplete and noisy. Additionally, we proposed to design Koopman autoencoder models with a delay embedding, in order to solve long-term time series forecasting tasks in ideal settings where the data is regularly sampled with no missing information, and a large number of past states are observed. We showed that the AIKAE model performs equally or better than the IKAE, both in these ideal settings and in more difficult settings related to satellite image time series. Additionally, we showed that our ATKAE with delay embedding performs competitively with recent concurrent methods on a popular long-term time series forecasting benchmark.
A potential direction for future work would be to design stochastic Koopman autoencoder models, leverag- ing the likelihood computation abilities brought by the coupling-layer normalizing flows that compose the (augmented) invertible Koopman autoencoders. Besides, regarding our delay embedding strategy, it would be of interest to lest alternative approaches where the input to the model is a condensed representation of several time steps (e.g. using mean pooling or convolution) rather than a direct stack of these steps, in order to reduce the computational cost as well as increase the robustness to missing data.
References
Omri Azencot, Benjamin Erichson, Vanessa Lin, and Michael Mahoney. Forecasting sequential data using consistent koopman autoencoders. In International Conference on Machine Learning, pp. 475 485. PMLR, 2020.
Ross Bannister. A review of operational methods of variational and ensemble-variational data assimilation. Quarterly Journal of the Royal Meteorological Society,
2017.
Nimrod Berman, Iann Naiman, and Omri Azencot. Multifactor sequential disentanglement via structured koopman autoencoders. In The Eleventh International Conference on Learning Representations, 2023. URL https::/ nett/forumiid--ftPTeeeee
Pelar Bevanda, Max Beier, Sebastian Kerz, Armin Lederer, Stefan Sasnowski, and Sandra Hirche. Diffeo- morphically learning stable koopman operators. IEEE Control Systems Letters, 6:3427 13422-3432, 2022.
Steven L Brunton, Bingni W Brunton, Joshua L Proctor, and J Nathan Kutz. Koopman invariant subspaces and finite linear representations of nonlinear dynamical systems for control. PloS one, 11())ee0150111, 2016.
Steven L Brunton, Marko Budišić, Eurika Kaiser, and Nathan Kutz. Modern koopman theory for dynamical systems. STAM Review, 64(2):229 340, 2022. doi: 10.1137/21M1401222
14

---

Published in Transactions on Machine Learning Research (05/2025)
Sibo Cheng, César Quilodrán-Casas, Said Ouala, Alban Farchi, Che Liu, Pierre Tandeo, Ronan Fablet, Didier Lucor, Bertrand Tooss, Julien Brajard, et al. Machine learning with data assimilation and uncertainty quantification for dynamical systems: a review. IEEE/CAA Journal of Automatica Sinica, 10(6):1361
1387, 2023.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXie:1410.8516, 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In International Conference Learning Representations, 2017. URL
Ronan Fablet, Bertrand Chapron, Lucas Drumetz, Etienne Mémin, Olivier Pannekoucke, and François Rousseau. Learning variational data assimilation models and solvers. Journal of A Advances in Model- ing Earth Systems, 13(10))c22211 20022 2021.
Fletcher Fan, Bowen Yi, David Rye, Guodong Shi, and lan R Manchester. Learning stable koopman embed- dings. In 2022 American Control Conference (ACC), pp. 7747-7747. IEEE, 2022.
Anthony Frion, Lucas Drumetz, Mauro Dalla Mura, Guillaume Tochon, and Abdeldjalil Aissa-El-Bey. Lever- aging neural koopman operators to learn continuous representations of dynamical systems from scarce data. In ICASSP 2029-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1 5. IEEE, 2023a.
Anthony Frion, Lucas Drumetz, Guillaume Tochon, Mauro Dalla Mura, and Abdeldjalil Aissa Bey. Learn- ing sentinel-2 reflectance dynamics for data-driven assimilation and forecasting. In 2023 31st. European Signal Processing Conference (EUSIPCO), pp. 1390-1394. TEEE, 2023b.
Anthony Frion, Lucas Drumetz, Mauro Dalla Mura, Guillaume Tochon, and Abdeldjalil Aissa El Bey. Neural koopman prior for data assimilation. IEEE Transactions on Signal Processing, 2024.
Maximilian Gelbrecht, Alistair White, Sebastian Bathiany, and Niklas Boers Differentiable programming for earth system modeling. Geoscientific Model Development, 16(11):3113-31 2023.
Nicholas Geneva and Nicholas Zabaras. Transformers for modeling physical systems. Newral Networks, 146: 272 289, 2022.
Lu Han, Han-Jia Ye, and De-Chuan Zhan. independent stteeg for multivariate time series forecasting. IEFE Transactions Knowledge and Dala Engineering, 2024.
The capacity and robustness trade-off: Revisiting the channel
S Hochreiter. Long short-term memory. Neural Computation MIT-Press, 1997.
Yuhong Jin, Lei Hou, Shun Zhong, Haiming Yi, and Yushu Chen. Invertible koopman network and its application in data-driven modeling for dynamic systems. Mechanicol Systems and Signal Processing, 200: 110604, 2023.
Yuhong Jin, Lei Hou, and Shun Zhong. Extended dynamic mode decomposition with invertible dictionary learning. Neural Networks, 173:106177, 2024.
Mason Kamb, Eurika Kaiser, Steven L Brunton, and J Nathan Kutz. Time-delay observables for koopman: Theory and applications. STAM Journal on Applied Dynasmical Systems, 19(2):886 917, 2020.
Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Reversible instance normalization for accurate time-serics forecasting against distribution shift. In International Conference on Learning Representations, 2021.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. Advances in neural information processing systems, 31, 2018.
15

---

Published in Transactions on Machine Learning Research (05/2025)
Bernard O Koopman. Hamiltonian systems and transformation in hilbert space. Proceedings of the National Academy of Sciences, 17(5):315-315 1931.
Nathan Kutz, Joshua L Proctor, and Steven L Brunton. Koopman theory for partial differential equations. arXiv preprint wxXic1607.000070 2016.
Remi Lam, Alvaro Gonzall,, Matthew Willson, Peter Wirnsberger, Meire Fortunato, Ferran Alet, Suman Ravuri, Timo Ewalds, Zach Eaton-Rosen Weihua Hu, et al. Learning skillful medium-range global weather forecasting. Science, 312(6667):11116-1422, 2023.
Soledad Le Clainche and José M Vega. Higher order dynamic mode decomposition. SIAM Journal on Applied Dynamical Systems, 16(2):882-925, 2017.
Holden Lee, Chirag Pabbaraju, Anish Prasad Sevekari, and Andrej Risteski. Universal approximation using well-conditonned normalizing flows. Advances in Neural Information Processing Systems, 11:12700-12777 2021.
Qianxiao Li, Felix Dietrich, Erik M Bollt, and Ioannis G Kevrekidis. Extended dynamic mode decomposition with dictionary learning: A data-driven adaptive spectral decomposition of the koopman operator. Chaos: An Interdisciplinary Journal of Nonlinear Science, 27(1)), 2017.
Yunzhu Li, Hao He, Jiajun Wu, Dina Katabi, and Antonio Torralba. Learning compositional koopman
operators for model- based control. In International Conference on Learning Hepresentations, 2020. URL
httpss:/openrevvie net/forumiid-iie H11dzA4tPP
Zhe Li, Shiyi Qi, Yiduo Li, and Zenglin Xu. Revisiting long-term time series forecasting: An investigation on linear mapping. arXiv preprint 1rXiv:3305100911 2023.
Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers effective for time series forecasting. In The Twelfth International Conference Learning Represeniations, 2024. URL htttp:://operrevieee http:///pppppeeeeeeeaeee//o//aaa www..etefoouumm.ddeeeeAfaaaa
Bethany Lusch, J Nathan Kutz, and Steven I, Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature commumications, 9(1):4950, 2018.
Andrew L Maas, Awni Y Hanmun, Andrew Y Ng. et al. Rectifier nonlinearities improve neural network acoustic models. In Proc. ieml, volume 30, pp. 3. Atlanta, GA, 2013.
Yuhuang Meng, Jianguo Huang, and Yue Qiu. Koopman operator learning using invertible neural networks. Journal of Computational Physics, 501:112795, 2024.
Tgor Mezić. Spectral properties of dynamical systems, model reduction and decompositions. Nonlinear Dynamics, 41:309 325, 2005.
Vinod Nair and Geoffrey E Hinton. Rectified lincar units improve restricted boltzmann machines. Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807 814, 2010.
Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference Learning Repre- sentations, 2023. URL httpss:/ net
Marcel Nonnemmacher and David S Greenberg. Deep emulators for differentiation, forecasting, and parametrization in earth science simulators. Journal of Advances in Modeling Earth Systems, 13(7): c2021M5002554 2021.
Joel Oskarsson, Tomas Landelius, Marc Peter Deiscnroth, and Frodrik Lindsten. Probabilistic weather forecasting with hicrarchical graph neural networks. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL. httpes///ppenreviie http:///wweraaaaaaaa/////a////// met/forrunidduullppppl/
16

---

Published in Transactions on Machine Learning Research (05/2025)
Samuel E Otto and Clarence W Rowley. Linearly recurrent autoencoder networks for learning dynamics. STAM Journal on Applied Dynamical Systems, (35(1)558-58) 593, 2019.
Peter J Schmid. Dynamic mode decomposition of numerical and experimental data. Journal of fluid me- chanics, 656:5-28, 2010.
Ashutosh Singh, Ashish Singh, Tales Imbiriba, Deniz Erdogmus, and Ricardo Borsoi. Koda: data-driven recursive model for time series lorecasting and data assimilation using koopman operators. ar preprint WwXiv:2409.19514 2024.
Floris Takens. Detecting strange attractors in turbulence. Dynamical Systems and Turbulence, 1981.
William Toner and Luke Nicholas Darlow. An analysis of linear time series forecasting models. In Forty- first International Conference Machine Learning, 2024. URL htctps//oopmmeriee..nne/fooo
x18TCcbYaT
Jonathan H Tu, Clarence Worth Rowley Dirk M Luchtenburg, Steven I. Brunton, and J Nathan Kutz. On dynamic mode decomposition: Theory and applications. Journal of Computational Dynamics, 1(2): 391 391-421, 2014.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Yuxuan Wang, Haixu Wu, Jiaxiang Dong, Yong Liu, Mingsheng Long, and Jianmin Wang. Deep time series models: A comprehensive survey and benchmark. arXiv preprint xrXft:2477.12776 2024.
Matthew O Williams, Ioannis G Kevrekidis, and Clarence W Rowley. A data-driven approximation of the koopman operator: Extending dynamic mode decomposition. Journal of Nonlinear Science, 25:1307-1346, 2015.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long, Autoformer: Decomposition transformers with auto-correlaionn for long-term series forecasting. Aduances in neural information processing systems, 34: 22410, 2021.
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d- variation modeling for general time series analysis. In The Eleventh International Conference on Learning Representations, 2023. URL
Uq
Enoch Yeung, Soumya Kundu, and Nathan Hodas. Learning deep neural network represenlations for Koop- man operators of nonlinear dynamical systems. In American Control Conference (ACC), pp. 4832-488 4839. IEEE, 2019.
Yuan Yuan, Kaiwen Zhou, Wenwu Zhou, Xin Wen, and Yingzheng Liu. Flow prediction using dynamic mode decomposition with time-delay embedding based on local measurement. Physics of Fluids, 33()), 2021.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference artificial intelligence, volume 37, pp. 11121-11128 2023.
Jingdong Zhang, Qunxi Zhu, and Wei Lin. Learning hamiltonian neural koopman operator and simultanc- ously sustaining and discovering conservation laws. Physical Review Research, 6(1::001,03,, 2024.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang, In- former: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 11106-11115, 2021.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International conference machine learning. pp. 27268 27286. PMLR, 2022.
17

---

Published in Transactions on Machine Learning Research (05/2025)
A Reconstruction abilities of older Koopman-based methods
For Dynamic Mode Decomposition (DMD, Schmid 2010)) ) which is the most popular and well-ostablished method for linding an approximation to the Koopman operator, the set of measurement functions is simply chosen to be the set of canonical measurement functions constituting the state variable. Hence, this method implicitly assumes that the dynamical system under study evolves linearly, or at least that accurate short- term predictions can be made by a linear model. In this case, the predictions are made directly in the state space, making its reconstruction unnecessary. A notable extension of DMD is the so-called extended Dynamic Mode Decomposition (Williams et al. 2015), which consists in manually designing a set of mcasurement functions that is likely to yield an approximate invariance by the Koopman operator. Common choices for these measurement functions are sets of polynomials of the state variables up to a chosen degree and sets of radial basis functions. The canonical measurement functions of the state are usually included in the hand-designed dictionary. This enables to trivially link the set of measurement functions to the state space by projecting on the appropriate variables. Although exlended DMD is generally applied on low- dimensional dynamical systems, a high number of measurement functions is usually required to obtain accurate predictions, which is a limiting factor of this method in practice. Thus, some subsequent works (Li et al. 2017 Yeung et al. 2019) have proposed lo replace the hand-designed dictionary of measurement functions by a lover-dimensional dictionary that is automatically learned by neural network. In these models, the inferred Koopman invariant subspace is a concatenation of the fixed canonical measurement functions and of the ones that are learned by the neural network.
Although the inclusion of the state variables in the Koopman invariant subspace again enables to easily reconstruct the state vector after multiplication by K, it may be detrimental to the actual linearity of the model. Indeed, depending the dynamical system under study, there might not exist an (approximately) Koopman invariant subspace of low dimension that contains the state variables. This flaw has motivated the introduction of Koopman autoencoders Lusch et al. (2018), which do no constrain a direct inclusion of the input variables in their latent space.
B Stochastic modeling abilities of invertible Koopman autoencoders
Let us here outline some possible adaptations of invertible Koopman autoencoders for stochastic modeling, although we do not present associated experiments in the paper.
The determinant of the Jacobian matrix corresponding to a coupling-layer normalizing flow & can be easily computed in practice, enabling perform likelihood computations thanks to the well known change of variable formula:
det 0b(x)
(x(x) py (o(x))
(19)
where Y is defined in the latent space of the model.
Using these properties, coupling-layer normalizing flows were originally introduced as a generative model, enabling to link a complex probability distribution px, supposodly corresponding to a set of observed data, to a simple probability distribution py, often chosen be a standard Gaussian. Although the current existing works on invertible Koopman autoencoders only consider deterministic settings, these properties may be used to train invertible Koopman autocncoders in stochastic context.
For example, one may estimate the probability distribution function of an advanced state X++7 knowing that the state x, is observed with an uncertainty corresponding to a Gaussian white noise with a covariance Σ, E Rax,, In this case, we would have that x, in fact random variable, defined as
X ~ N E,).
(20)
Using equation 19 one can accordingly compute the probability density function of the associated encoding z, = (xx)). From there, Zeer is obtainod as the multiplication of z, by the square matrix K", enabling for another easy change of variable since the Jacobian of this linear transformation K itself. Finally, one
can go back to the input space by simply reversing equation
19 As a conclusion, one may compute the
18

---

Published in Transactions on Machine Learning Research (05/2025)
Table 3: Summary of the characteristics of the datasets of the Informer benchmark. The "Dataset length"
entry contains the lengths of the training, validation and test subsets of data.
Datasct
Channels
Sampling period
Datasct length
Information
ETTh1, ETTh2
7
hour
(8545, 2881, 2881)
Electricity
ETTml, ETTm2
7
15 minutes
(34465, 11521, 11521)
Electricity
Weather
21
10 minutes
(36792, 5271, 10540)
Weather
ECL
321
1 hour
(18317, 2633, 5261)
Electricity
Traffic
862
hour
(12185, 1757, 3509)
Transportation
Exchange
8
1 day
(5120, 665, 1422)
Economy
probability density function of X++r when X, follows a known Gaussian distribution, which can enable to design loss function that leverages the likelihood of subsequent stales.
Conversely, following classical usage of normalizing flow models, one may consider the latent distribution be Gaussian while the distribution in the imput space is more complex. In this case, one may consider variant of the IKAE where variance encoder is trained along with the invertible normalixing flow encoder Ø, so that Z, is random variable following:
21 ~ N(x(x))),
(21)
where {(x{)= E R" corresponds to the (positive) diagonal coefficients of a diagonal covariance matrix. In this case, Z++7 K'z, 24 also Gaussian as linear transformation of Gaussian variable (one might consider adding Gaussian noise each propagation step too), and one can again retrieve the probability density function of X with final change of variable. Thus, one may construet stochastic dynamical model of the considered system, trained with a likelihood criterion.
C Description of the long-term time series datasets
Here, provide brief descriptions of the datasets constituting the benehmark from section The ETT datasots (Zhou et al. 2021 constituted from factors of clectricity transformers, including load and oil lemperalures, recorded from July 2016 to July 2018. There are four subsets: ETTh1, ETTh2, ETTml and ETTm2. ETThh and ETTh2 have a sampling period of one hour, while ETTml and ETTm2 have a sampling period of 15 minutes. Weather (Wu et al. 2021) is constituted from 21 metcorological indicators, recorded every 10 minutes by the weather station the Max Planck Biogeochemisttry Institute in 2020. FCL. (Wu et al., 2021) (also called "Flectricity"" constituted from the hourly electricity consumptions of 321 clients, recorded from 2012 2014. Traffic (Wu al. (222)) is composed of hourly occupancy rates of 863 roads in the San Francisco bay area, recorded by the California Department of Transportation from January 2015 to December 2016. Exchange (Wu et. al. 2021 contains the daily exchange rates of eight different countries, ranging from 1990 to 2016.
In Table 3 we summarize the numbers of channels, sampling periods, time series lengths (including training, validation and test subsets), and nature of the information of these datasets. From this table, one can clearly see that the ECL and Traffic datascts contain much more channels than all of the other considered datascts. Concerning the loading and separation into train, validation and test splits for each dataset, we use the code from Zeng et al. (2023). corresponding to the same settings for the benchmarks of Wi et al. (2023) and Liu et al. (2024) that the bascline methods use.
As explained in the main text, each of these datasets contains several simultaneously recorded channels of information, and thus one can theoretically combine the information of these channels in order to increase the predictive capabilitics of forccasting model. However, some recent methods (Zeng et al.) 2023 Nic et al. 2023 Li al., 2023) (along with the ones introduced in this paper) rely on a single univariate model, with performance challenging the strongest multivariate time series processing models.
19

---

Published in Transactions on Machine Learning Research (05/2025)
D Implementation details for delayed Koopman autoencoders
Here, we describe in more detail the architectures of the delayed IKAE and delayed AIKAE models that produced the results on the first 2 columns of table [ As explained in section 3.2 we make use of the delay embedding strategy, which means that the invertible encoding contains information multiple time steps. Besides, we adopt the channel independence principle (Nic et al. 2023), so that an input state contains only information on one specific variable of the time series. Thus, since we use an input sequence length of 96 time steps for all experiments, the dimension of our input (and invertible encoding) is always n = 96 = 96. The main component of both the IKAE and AIKAE is their invertible encoder o, which in these experiments is implemented as NICE normalizing flow (Dinh et al., 2014p. Using the terminology from Dinh et al. (2014), the NTCE model is composed of k coupling layers, each using the additive coupling law and MLP network with one hidden layer of width and leaky rectified linear unit nonlincarity (Maas eet al. 2013). Following common modeling choices, the input to the MLP is composed of half of the input state variables, i.e. it is of size n/2 = 48. Thus, neglecting the bias parameters, the hidden layer of the MLP contains about (n/2) w parameters and its output layer contains w (n/2) parameters, for a total of approximately wn parameters, and finally around kwn trainable parameters when adding all the coupling layers of Φ. The decoder, being analytically obtained from this encoder, does not involve any additional trainable parameters. For the AIKAE models only, one additionally trains an augmentation encoder X, which is implemented as MPL network. This MLP network comprises 3 linear layers, of width fixed to [256, 128, 32). A ReLU nonlinearity Nair & Hinton (2010) is applied after the first 2 of these layers. Nole that the width of the final layer is the size of the augmentation encoding, i.e. p = 32. This augmentation encoder contains approximately 60K trainable parameters. Finally, the Koopman matrix K adds d² parameters to the models, where d = n = 96 for the IKAE models and d = n + = 128 for the AIKAE models.
A hyperparameter search is performed for cach dataset and prediction length on the number of coupling layers (set in the range & E {3, 4)) and the width w of the hidden layer of the coupling functions (set in the range w {128, 256}). Thus, the number of parameters for the IKAE models varies approximately from 50K to 110K. Conversoly, the number of trainable parameters for the AIKAE models varies approximately from 115K to 175K.
As mentioned in the main text, all models are trained with the Adam algorithm with a learning rate of 10 and parameters 3 = (0.9, 0.999). As we observed a high sensitivity of the final performance on the batch size of the dataloader, for each task we search for the best batch size among the values {4, 128, 512].
E
Additional long-term time series forecasting results
E.1 Extended forecasting results
Table 4 extends the results from table [ by adding the persistence baseline and the exchange dataset. As mentioned in the main text, it shows that the tested models significantly outperform the persistence baseline on all datascts except for Exchange.
E.2 Ablation study
In order to get insight on the performance of the delayed IKAF and AIKAE models, we now perform an ablation study. We focus on the influence of two components of our models: the nonlinear encoder and the usage of RevIN.
Concretely, we test eight different models. The IKAE and AIKAE models with RevIN correspond to the results reported in table I For cach of these models, we train variant where we do not use RevIN. In order to infer the interest of inflating the latent space with a second learned encoder rather than with zero padding as proposed by Meng et al. (2024), we also test IKAE models with zero padding, which are referred to as IKAE-zp, with or without RN The size of the zero padding is 32, corresponding to the size of the augmentation encoding in AIKAE models. In addition, we test simple linear models, where the nonlinear
encoder & + is simply replaced by an identity function, with (Li et al. 2023 without ZZng et al.
20

---

Published in Transactions on Machine Learning Research (05/2025)
Table 4: Forecasting mean squared errors (MSEs) and mean absolute errors (MAEs) for various models and long-term forecasting tasks. For each dataset, we use a lookback window of size Tr 96 and prediction horizons Tp of sizes 96, 192, 336. 720. IKAE and AIKAE are our own implementations, while we use the
results reported by Zeng ct al. (2023) for the persistence bascline and by Liu et al. (2024) for all other
models. For each lask and metric, the best, result is in bold and the second best result is underlined.
Model
IKAE
AIKAE
iTransformer
PSt
TemosNet
DLinewr
Persistence
Metric
MSE
MAF
MSF
MAF
MSF
MAF
MSF
MAF
MSF
MAF
MSR
MAF
MSF
MAF
96
0.159
0.252
0.153
0.247
0.148
0.240
0.181
0.270
0.168
0.272
0.197
0.282
1.588
0.946
ECL
192
0.173
0.265
0.167
0.259
0.162
0.253
0.188
0.274
0.184
0.289
0.196
0.285
1,595
0,950
336
0.189
0.282
0.184
0.277
0.178
0.269
0.204
0.293
0.198
0.300
0.209
0.301
1.617
0.961
720
0.227
0.313
0.223
0.310
0.225
0.317
0.246
0.324
0.220
0.320
0.245
0.333
1.647
0.975
96
0.081
0.202
0.080
0,201
0.086
0.206
0.088
0.205
0.107
0.234
0.088
0.218
0.081
0.196
Exch.
192
0.159
0.290
0.163
0.295
0.1
0.299
0.176
0.299
0.226
0.344
0.176
0.315
0.167
0.289
336
0.331
0.435
0.320
0.429
0.331
0.417
0.301
0.397
0.367
0.448
0.313
0.427
0.305
0.396
720
0.854
0,700
0.848
0.698
0.847
0.691
0.901
0.714
0.964
0.746
0. 1389
0.695
0.823
0.681
96
0.460
0.313
0.431
0.297
0.395
0.268
0.462
0.295
0.593
0.321
0.650
0.396
2.723
1.079
Traffic
192
0.476
0.317
0.449
0.302
0.417
0.276
0.466
0.296
0.617
0.336
0.598
0.370
2.756
1.087
336
0.496
0.327
0.467
0.311
0.433
0.283
0.482
0.304
0.629
0,336
0,605
0.373
2.791
1,095
720
0.524
0.343
0.499
0.329
0.467
0.302
0.514
0.322
0.640
0.350
0.645
0.394
2.811
1.097
mm' ETTml ETTh2 ETTh1 Weath her
0.157
0.209
0,160
0.211
0.174
0.214
0.177
0.218
0.172
0.220
0.196
0.255
0.259
0.2541
192
0.193
0.245
0.195
0.245
0.221
0.254
0.225
0,259
0.219
0.261
0.237
0.296
0,309
0.292
336
0.237
0.281
0.237
0.281
0.278
0.296
0.278
0.297
0.280
0.306
0,283
0.335
0.377
0.338
720
0.317
0.333
0.317
0.333
0.358
0.347
0.354
0.348
0.365
0.359
0.345
0.381
0.465
0.394
0.386
0.399
0.386
0.398
0.386
0.405
0,414
0,419
0.384
0,402
0.386
0,400
1.295
0.713
192
0.437
0.433
0.435
0.432
0.441
0.436
0.460
0.445
0.436
0.429
0.437
0.432
1.325
0.733
336
0.480
0.457
0.482
0.459
0.487
0.458
0.501
0.466
0.491
0.469
0.481
0.459
1.323
0.744
720
0.375
0.522
0.561
0.516
0.503
0.491
0.500
0.488
0.521
0.500
0.519
0.516
1.339
0.756
96
0.301
0.348
0.300
0.346
0.297
0.349
0.288
0.338
0.340
0.374
0.333
0.387
0.432
0.422
192
0.365
0.394
0.365
0.394
0.380
0.400
0.388
0.400
0.402
0.414
0.477
0.476
0.534
0.473
336
0.394
0.422
0.387
0.441
0.428
0.432
0.426
0.433
0.452
0.452
0.594
0.541
0.591
0.508
720
0,414
0.443
0.429
0.450
0.427
0.445
0.431
0.446
0.462
0.468
0.831
0.657
0.588
0.517
96
0.328
0.364
0.322
0.360
0.334
0.368
0.329
0.367
0.338
0.375
0.345
0.372
1,214
0.665
192
0.365
0.389
0.364
0.387
0.377
0.391
0.367
0.385
0.374
0.387
0.380
0.389
1.261
0.690
336
0.390
0.405
0.390
0.406
0.426
0.420
0.399
0.410
0.410
0.411
0.413
0.413
1,283
0.707
720
0.463
0.443
0.465
0.443
0.491
0.439
0.454
0.439
0.478
0.450
0.474
0.453
1.319
0.729
0.184
0.266
0.185
0.267
0.180
0.264
0.175
0.259
0.187
0.267
0.193
0.292
0.266
0.328
192
0.255
0.313
0.246
0.307
0.250
0.309
0.241
0.302
0.249
0.309
0.284
0.362
0.340
0.371
336
0.297
0.343
0.303
0.344
0.311
0.348
0.305
0.343
0.321
0.351
0.369
0.427
0.412
0.410
720
0.382
0.393
0.385
0.393
0.412
0.407
0.402
0.400
0.4088
0.403
0.554
0.522
0.521
0.465
21

---

Published in Transactions on Machine Learning Research (05/2025)
Table 5: Forecasting mean squared errors (MSEs) and mean absolute errors (MAFs) of different models
on three datascts, with lookback window Tz 96 and forccasting horizon Tp 96. For cach datasct and metric, the best result is in bold and the second best result is underlined.
Dataset
Traffic
Weather
FTTml
Metric
MSE
MAE
MSE
MAE
MSE
MAE
with RevIN
0.450
0.301
0.171
0.216
0.322
0.360
AIKAE
without RevIN
0.497
0.299
0.167
0.226
0.350
0.386
with RevIN
0.452
0.304
0.174
0.219
0.331
0.365
IKAE-zp
without RevIN
0.507
0.300
0.170
0.227
0.349
0.380
with RevIN
0.460
0.313
0.174
0.220
0.328
0.364
IKAE
without RevIN
0.517
0.317
0.168
0.225
0.342
0.377
with RevIN
0.644
0.390
0.194
0.234
0.349
0.369
Linear
without RevIN
0.649
0.397
0.201
0.266
0.345
0.377
2023 RevIN. We work with Tr. Tp = 96, which means that the linear model without RevIN may be seen as a dynamic mode decomposition [Schmmid 2010) with a delay embedding of size 96. We perform the ablation study on three datasets: Traffic, Weather and ETTml. For each model, the architecture is kept the same for each dataset. The IKAE and AIK AE have an invertible encoder comprising 4 coupling layers of width 256. For the AIKAE, the augmentation encoder has the same architecture as in the main results, The optimizer is Adam with a learning rate of 10-3, and the batch size is 4 for the ETTml dataset and 32 for the two other datasets.
The results obtained by the eight described models are reported in table 5 Although we use our own imple- mentation of the linear models in order to limit the risk of differing implementation choices influencing the study, we obtain consistent results with the implementations of ZZng et al. (2023) and Li et al. (2023). From these results, one can see that the addition of RevIN often (though not always) improves the performance of all backbone models. In addition, the gains obtained by using a more complex embodding appear to be complementary to the gains of RevIN. In particular, the delay AIKAE model without RevIN outperforms the delay IKAE without RevIN, which itself outperforms the linear model without RevIN. Thus, this study shows that resorting to an invertible nonlincar embedding of the input data improves the results compared to a simple linear model, and that the results are further improved when additionally increasing the di- mension of this embedding with an AIKAF. Besides, the superiority of IKAF-zp to IKAE (either with or without RevIN) cannot be clearly established, and thus AIKAE remains the strongest of the tested KAE architectures in this benchmark.
E.3 Influence of the lookback window size
Tt has been repeatedly observed in previous works (e.g. Zeng et al. (2023 Nie et al. (2023 Liu et al. 2024 that many baansformer-based models for long-term time series forecasting do not benefit from an increased size Tr, of the lookback window. Indeed, for many of these models, the forecasting performance stagnates or even decreases as the length of the lookback window increases, which has been attributed to a distracted attention over the input. In contrast, simple linear models have been shown to greatly benefit from a longer window of observations. Thus, we now assess the performance of the IKAE and AIKAE models as the size of the lookback window increases. We work in the same setting as Zeng et al. (2023). where we evaluate the forecasting performance for prediction window of size Tp 720 according
varying input sizes Tz from 48 to 720. For each lookback length, we train our two models as well as the DLincar model of Zeng et al. (2023) and 4 Transformer- basedd models: the base Transformer (Vaswani] 2017]. Informer (Zhou el al. 2021). Autoformer (Wu et al. 2021) and FEDformer (Zhou et al.) 2022). For the DLinear and Transformer models, we use the code of Zeng et al. (2023). For our IKAE and AIKAE models, order to keep the experiment as fair as possible, we use the same architocture and training hyperparameters for every lookback length, i.c. 4 coupling layers of width 256 and batch size of 512.
22

---

Published in Transactions on Machine Learning Research (05/2025)
AIGAE IKAE
Duncar Tranffrmer
Infomer Autoformer
FED
AIAE KAE
DL DLinear
Informrr Autofomer
PEDMomet
Tranfommr
0.425
NOE
0.30
0.375
0.25
0.325
0.29
72
96
120
1.4
168
192
336
504
672
720
48
96
120
144
168
152
336
504
672
720
langth
Lookbuukk tungin
Figure 4: Mean squared error (left) and mcan average errors (right) obtained by different models for predic-
tions of Tp 720 time steps the ECL dataset, function of the lookback window size Tz.
Table 6: Mean and standard deviation of the MSE and MAE, over 5 random initializations of the parameters, for TKAE and ATKAE models on the ECL and Weather datasets.
Model
IKAE
AIKAE
Metric
MSE
MAE
MSE
MAE
96
0.1588 + 0.0005
0.2518 + 0.0006
0.1533 + 0.0010
0.2469 + 0.0011
192
0.1727 + 0.0004
0.2652 + 0.0003
0.1667 + 0.0003
0.2592
0.0003
ECL
336
0.1892 + 0.0004
0.2818 + 0.0004
0.1838 + 0.0006
0.2768
0.0004
720
0.2267 + 0.0001
0.3134 + 0.0006
0.2235 + 0.0027
0.3101
0.0020
96
0.1573 + 0.0007
0.2085
0.0008
0.1603 0.0020
0.2110
0.0009
192
0.1932 0.0018
0.2450 = 0.0016
0.1946 + 0.0028
0.2451 + 0.0027
Weather
336
0.2374 0.0011
0.2810
0.0010
0.2374 = 0.0013
0.2806 = 0.0011
720
0.3168 + 0.0014
0.3328 = 0.0013
0.3167 = 0.0013
0.3335 = 0.0008
The results of this experiment are summarrizd in figure From this figure, one can see that none of the Transformer models is characterized by consistently decreasing error metrics as the size of the lookback window increases. Only the AIKAE, IKAE and DLinear models exhibit this behavior. While the AIKAE and IKAE models outperform the DLincar models for shorter lookback windows (as could be seen from the main results in (able 1. DLinear performs best for longer lookback windows. Thus, this experiments shows that the delayed Koopman autoencoder models do not share the same flaws many Transformer models, but still struggle to compete with linear models when very large window of past observations is available.
E.4 Sensitivity to the initialization
Here, we study the sensitivity of the delayed IKAE and AIKAE models to the initialization of their parameters before training. We focus on two datasets: ECL and Weather. For each prediction length of these datasels, we separately train instances of IKAE and AIKAE models. These instances differ only by their initializations, each associated to different fixed random seed. We report the mean and the standard deviations of the mean squared errors and mean absolute errors over these 5 seeds in table 6 From these results, one can see that the performance of our models are very stable across different. initializations, since the standard deviations of the MSE and MAE are small in comparison their associated means.
E.5 Sensitivity to hyperparameter choices
Here, we study the performance of our models according to different training hyperparameters. Namely, we study the influence of the learning rate, the number of coupling layers in the invertible encoder é, and the hidden dimension of the MLP used in each of these coupling layers. We study learning rates of value
23

---

Published in Transactions on Machine Learning Research (05/2025)
Learning rate, FTTh2
Coupling layers, FTTh2
Layers width, ETTh2
0.44
0.4
0.42
0.40
0.38
0.38
0.31
0.38
0.36
0.54
0.34
0.34
0.52
0.32
0.32
0.30
0.0005
0.001
256
Learning rate. Traffic
Coupling layers. Traffic
Layers width, Traffic
0.54
0.52
0.50
100
0.48
0.46
0.46
0.44
0.44
0.44
0.42
256
192
336
Figure Summary of hyperparameter sensitivity analysis for dclayed AIKAE and IKAE, on the ETTh2 (lop) and Traffic (boltom) datasets. We study the influence of three hyperparameters: the learning rate (left), the number of coupling layers (middle) and the width of the layers (right). The full lines represent AIKAE models while the dashed lines represent IKAE models.
{0.0002, 0.0005, 0.001, 0.002] (as a reminder, all of our models in the main results are trained with learning rate 0.001). We consider numbers of layers in the range {3, 4, 6}, and hidden dimensions values {128, 256, 512}. We focus our study on two datasets: ETTH2 and Traffic, for which we run our models in the same conditions as in the main results, with prediction lengths Tp E {96, 192, 336, 720). For the lcarning rale study, we always use the architecture which performed the best in the main results. Conversely, for the two other studies, the hyperparameters other than the one under study are fixed to their values in the best found configuration of the hyperparameter search detailed in section D
The test MSE That we obtain are summarized in figure 5 These results show a relative robust ness of the models to the choice of the learning rate. In fact, it seems that, although our main results are all obtained with a learning rate of 0.001, increasing this value to 0.002 (or including it in the hyperparameter search) might further improve the performance. Besides, one can see that increasing the number of parameters of the models (through the number width of the layers) generally improves the performance dataset, but not necessarily the ETTh2 dataset, which is consistent with the results of e.g. Liu et al. (2024). When specifically comparing AIKAE to IKAE models (respectively represented by full and dashed lines), one can see that the advantage of AIKAE is not clear for ETThh but significant for Traffic, which is consistent with the results of table I Interestingly. it appears that an AIKAE with layer width 128 performs slightly better than an IKAE with layer width 512 on the Traffic data. These models have respective parameter counts of around 125K and 210K, and thus one can say that the augmentation encoder enables a much greater parameter efficiency for this dataset.
the Traffic
E.6
Case studies for long- term time series forecasting
In order for the reader to have a more intuitive idea of the quality of our model predictions, we now display some results obtained on randomly selected samples of the test subsets of datasets from the long-term time series forocasting benchmark. On figures and ? we respectively plot predictions on test samples from the ECL dataset and from the Weather dataset. In both cases, we show the predictions made over 96 time steps (left)) and 336 time steps (right), with both IKAE model and AIKAE model, using the same context
24

---

Published in Transactions on Machine Learning Research (05/2025)
6.5
-0.5-
-0.5
-1.0
-1.
Context
Context
Gmundrruth
Gmundrruth
prediction
IKAE prediction
1.5
prediction
1.5
25
50
1.00
1.0
200
300
Figure 6: Predictions with AIKAE and IKAE models from same context window in the test subset of the ECL dataset, over 96 time steps (left) and 336 time steps (right).
1.6
1.6.
1.4
1.4
1.0
1.0
Groundinth
Grourdruth
IAE prediction
IKAE predition
0.8
AIKAE prediction
0.8
AIKAE prediction
50
75
100
125
150
175
200
100
200
300
400
Figure 7: Predictions with AIKAE and IKAE models from same context window in the test subset of the Weather dataset, over 96 time steps (left) and 336 time steps (right).
window of Ty. = 96 time steps. From those figures, one can see that the predictions the AIKAE and IKAE are qualitatively very similar.
F
Additional results for satellite image time series forecasting
F.1 Influence of the size the augmentation encoding
As stated the main text, the worse results of the IKAE model compared to other Koopman autoencoder variants may be explained by the fact that the size of the input state is too low to obtain a good enough Koopman invariant subspace of the system. Here, seek to obtain empirical assessment of how the size of the augmentation encoding for AIKAE models affects the satellite image time series forecasting performance. As reminder, the augmentation encoding for the AIKAE model results in table 2 is of size = 16, resulting in a global latent space of size d = n + = 36. Besides, the IKAE model may be seen as a special case an AIKAE where = 0. Building on these 2 cases, we train new AIKAE models with augmentation encoding sizes of 2, 4, 8 and 32. For all of these models, the other training hyperparametors are kept the same as in the AIKAE model from the main results. The reported results correspond to means over random initializations of the parameters of the models. The full results reported in table 7
25

---

Published in Transactions on Machine Learning Research (05/2025)
Table 7: Mean squared errors (MSEs) and mean absolute errors (MAEs) obtained by averaging the perfor-
mance of instances of each model. The models are pre-trained on the Fontaineblean area, and then used
as variational data assimilation priors, following equation 17 We study varying sizes of the augmentation encoding in the AIKAE, reminding that an augmentation size of 0 corresponds to an IKAE models.
Dataset
Fontainebleau
Orléans
Metric
MSE
MAE
MSE
MAE
Augmentation size 0
0.00128
0.0224
0.00382
0.0399
Augmentation size 2
0.00110
0.0216
0.00362
0.0392
Augmentation size 4
0.00109
0.0212
0.00359
0.0388
Augmentation size 8
0.00105
0.0210
0.00314
0.0365
Augmentation size 16
0.00108
0.0204
0.00297
0.0356
Augmentation size 32
0.00118
0.0214
0.00365
0.0388
Several conclusions can be drawn from table 7 First, even a very low-dimensional augmentation encoding (i.e. augmentation size 2) can have a significant impact on the results. A potential explanation for this is that the augmentation variables of the latent space may represent more complex quantities since they are not structurally in bijection with the imput. In addition, one can see that the performance gradually improves with increasing sizes of the augmenlation encoding until the optimal value p = 16, and then degrades for the size 32. This shows that a large augmentation encoding might lead to an overfitting model. Overall, one can see that the performance of the model is very sensitive to p. which should therefore be chosen carefully in order to attain optimal results.
F.2 Influence of the linearity loss on the augmentation encoding
In theory, in order for an AIKAE model to actually learn a Koopman invariant subspace of the studied dynamical system, one should have that the time propagation of an encoding at time t by t time steps leads to the encoding of the state at time t + r. This observation the reason for the usage of a lincarity loss function for Koopman autoencoder models, explained e.g. by Lusch et al. (2018). With the notations of the AIKAE, the linearity loss, which we indeed use for training all Koopman autoencoder variants, can be written
Lio ( K) = Ex.,7 K" (x;;) - +(x++>)11
(22)
However, for the AIKAF particular, we have noted in section 3.1 that only the invertible part of the encoding directly related to the input space, and that the augmentation part of the encoding might be interpreted as containing the 'static features' of the state. Thus, one might question the importance of actually ensuring that the augmentation part of the encoding follows a linear dynamics. Inspired by this observation, one may generalize equation 22 to weighted version as
Liina +b K) = Ex,-- ||lk|ll|K|//// (x, (xx)) -(x+++))|1++
(X===]] x(x+++)11.
(23)
In this variant, the case where a corresponds to the unweighted version of equation 22 while a = 0 means that only the invertible part of the encoding is expected to evolve lincarly in time. In between these two cases, one might imagine choosing any value of between 0 and to determine the relative importance of the linearity of the invertible and augmentation parts of the encoding. A value > 1 would mean that it is more important for the augmentation part z than for the invertible part 2 to follow lincar dynamics, which seems quite counter-intuitive.
In order to get some insight on these possible variants of the linearity loss function for the AIKAE model, we now examine three cases:
the case a = 1, as presented in the main results of table
the case a = 0, where only the lincarity of the invertible part z is required,
the case a = 1/2, where the linearity of of is twice as important as the linearity of 24.
26

---

Published in Transactions on Machine Learning Research (05/2025)
Table 8: Mean squared errors (MSEs) and mean absolute errors (MAEs) obtained by averaging the perfor- mance of instances of each model. The models are pro-trained on the Fontaineblean area, and then used variational data assimilation priors, following equation 17 The models diller by the expression of the pa-
rameter in their linearity loss term during training, following equation
23
The
1, corresponding
to the main results of table 2 clearly leads to the best performance.
Dataset
Pontainebleau
Orléans
Metric
MSF
MAE
MSE
MAE
a =
0.00108
0.0204
0.00297
0.0356
= 1/2
0.00114
0.0211
0.00335
0.0377
a = 0
0.00131
0.0223
0.00349
0.0382
0.8
0.6
Reflectance
KAE
0.2
IKAE-20
AIKAE
extrapolated data
0.0
250
500
750
1000
1250
1500
1750
Time (days)
Figure 8: Assimilated trajectories by different models on the B7 spectral band, for randomly selected pixel of the test Orléans area. The dashed vertical line marks the limit between the window of assimilated data (on the left) and the extrapolation window (on the right).
For each of these three cases, we train 5 instances of the AIKAE model starting from the exact same 5 random initializations, and otherwise following the same training procedure as in the main results. The means the mean squared errors and mean absolute errors for each variant on the Fontainebleau and Orléans areas are listed in table g
From the results of table one where the linearity loss is implemented as unweighted least squares calculation. The loss of accuracy when deviating from this case is significant. For the considered task, this result may be explained by the fact that the testing task is to perform a very long-term forecasting using an assimilation of the initial latent state while the training task involves naive forecasting on a relatively shorter timespan. promotion of the lincarity of the augmentation part of the encoding might be seen a regularization of the model, bringing a significant boost the test performance, One might expect that, in simpler setups, this observation might. not always be true.
that the best performance obtained with 1, i.c. in the case
this context, the
27

---

Published in Transactions on Machine Learning Research (05/2025)
F.3 Graphical results for satellite image time series forecasting
On figure 8 some assimilated trajectories are plotted on the B7 spectral band (i.e. the most energelic one for our data). The trajectories result from an assimilation of the data snapshots before the dashed line, using the best trained instance respectively the KAE, IKAE, IKAE-zp and AIKAE models. On this example, the IKAE model clearly does not fit the assimilated data as well as the other models, which confirms our observation that the limited latent dimension might. hurt the expressive power of this model. When comes to extrapolating beyond the assimilated datapoints, the AIKAE model clearly performs best, followed by the IKAE-zp, which is consistent with the global results of table 2
28


Reviews:
[{"review_id": "nP4IHZ3jZW", "reviewer": "Reviewer_FaCp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The paper is very clearly written and this reviewer appreciates the nice exposition of the DMD framework and its extension. \n- The introduced ideas are well-motivated, and the introduced method is explained clearly without burdening the readers with too much technicalities.\n\nWeaknesses\n\n1. I felt the introduction of augmented observables could be motivated better. If the invertible mapping is already transforming all the necessary information into latent variables, why is the augmentation necessary? The authors provide base this on learning behavior depending on the latent variable dimension in order to increase the redundacy in the data. However, if the mapping is invertible, such augmented information must be available in the latent variables $\\mathbf{z}^i_t$ already. Further explanation would be helpful.\n\n2. There are some mathematical observations that are incorrect, at least according to my first reading. The authors seem to be under the impression that because the autoencoder is invertible algebraically, the inversion can be done to machine precision (page 4: \"enabling an exact reconstruction of an encoded state $\\mathbf{x}$ up to machine precision). This is not true. When floating point arithmetic is used, even for exact algebraic expressions, conditioning affects the accuracy of the inverted result. For the authors' claim to be true, it should be back up by guarantees in the conditioning of the inverse.\n\n3. The authors claim that, when the DMD matrix $\\mathbf{K}$ learned from the latent data $[ \\mathbf{z}^i_t, \\mathbf{z}^a_t]$, \"the augmented variables $\\mathbf{z}^a_t$ influences the subsequent invertible parts of the encoding through the multiplication by $\\mathbf{K}$ as long as the upper-right block of $\\mathbf{K}$ is nonzero.\" This is a sufficient condition but not a necessary one; the upper-right block can be non-zero but $\\mathbf{z}^a_t$ could lie in the nullspace. Therefore, having upper-right block of $\\mathbf{K}$ non-zero does not imply that the augmented variables are being used at all. Furthermore, it is not difficult (mathematically speaking) to construct simple examples where the influence is non-zero, but is trivial. \n\n4. The computational results can benefit from more details. The authors seems to be showing that, somehow the large transformer models underperform compared to the AIKAE (or IKAE) and that errors do not necessarily decrease when the \"lookback length\" is increased. This phenomena is a bit confusing. In general, longer time-series data used for prediction leads to better prediction results. These models appear to have more parameters, more room for hyperparameter tuning, etc, and it is not clear a prior whether these results (e.g. Figure 2) are exhibiting real bottlenecks in the learning procedure, or whether they have just not been fine-tuned enough.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce Augmented Invertible Koopman AutoEncoder (AIKAE) which is an extension of Dynamic Mode Decomposition (DMD) technique by incorporating (1) invertible autoencoder and (2) further augmentation via additional observables. Both the autoencoder and observable mappings are certain types of deep learning models. By introducing both an invertible autoencoder and augementation, the authors show that AIKAE achieve roughly the state-of-the-art results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The authors should verify the numerical precision of the inversion as in Weaknesss 2, either theoretically or numerically, or adjust the statement.\n\n2. The authors should revisit the linear algebraic properties of the augmented DMD procedures, as observed in the Weakness 3. I recommend the authors analyze this further by analyzing the linear algebraic properties of the DMD augmentation.\n\n3. I recommend the authors provide full details on how the compared models (transforms, convnets, etc) were trained and fine-tuned. If these were available in the literature, the authors should provide a broad summary. I believe the numerical results would be much more convincing if these details show that these models cannot be fine-tuned to yield better accuracy (or at least it is difficult to do so).\n\n4. Page 2: \"However, most of the time the Koopman invariance has to be approximated to a certain degree\" is a bit unclear.\n\n5. Page 4: \"IKAE architecture had the inconvenient\" : \"inconvenience\"?\n\n6. Page 7: I recommend the authors provide a brief description of what the time-series dataset is. It is omitted in the current manuscript, I believe.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "E03RJlAl0U", "reviewer": "Reviewer_gBJs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n(1)\tThis work combines invertible and non-invertible encoders to expand latent space without sacrificing reconstruction accuracy, and grounded in Koopman operator theory, addressing invariance and linear dynamics approximation.\n\n(2)\tIt validated on both regularly sampled time series and irregularly sampled satellite imagery. And it matches or surpasses recent baselines (e.g., Transformers, DLinear) in long-term forecasting tasks.\n\n(3)\tThis work has publicly released code that enhances reproducibility and community adoption.\n\nWeaknesses: \n\n(1)\tThe description in the methodology section is not sufficiently clear. Figure 1 only illustrates the transition from Xt to Xt+1, but fails to show feature extraction and fusion across different time steps. Additionally, the overall mapping from multiple input steps through feature extraction to multiple output steps is not clearly depicted. Details regarding each sub-module used in the method are also missing, making it challenging to grasp the methodology clearly at either the global or detailed level.\n\n(2)\tThe paper's topic of long-term time series forecasting, but the motivation behind introducing the \"Variational data assimilation on satellite image time series\" part is insufficiently articulated. Satellite image time series appear inadequately connected to the general theme of long-term time series forecasting (such as topics mentioned in related work).\n\n(3)\tIn the experimental results compared with baselines, too few indicators achieve comparable state-of-the-art performance.\n\n(4)\tNo case study is provided to verify the prediction performance of the model in practical scenarios. Also, there is limited discussion on the sensitivity of the model's performance to hyperparameter choices, which is important for practical implementation.\n\n(5)\tNo discussion of training/inference costs or scalability comparison with the listed baselines.\n\n(6)\tThe potential for stochastic Koopman modeling is mentioned but not demonstrated.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces the Augmented Invertible Koopman AutoEncoder (AIKAE), addressing the dimension limitation of existing invertible Koopman autoencoders (IKAEs) by augmenting the latent space with a non-invertible encoder. The AIKAE retains exact reconstruction via its invertible component while expanding the Koopman operator’s expressivity. Experiments on long-term forecasting benchmarks (e.g., ETT, Traffic, Weather) demonstrate AIKAE’s competitiveness with state-of-the-art methods like iTransformer and DLinear, particularly showing improved performance over IKAEs. Additionally, AIKAE excels in variational data assimilation for satellite image time series with missing observations, outperforming other Koopman variants. The model’s delayed embedding approach leverages large lookback windows, aligning with linear models’ scalability advantages. Code availability and theoretical grounding in Koopman operator theory further strengthen its contribution. However, gains over baselines are often marginal, and computational costs or hyperparameter sensitivity remain underexplored.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "(1)\tAs for weakness point 2, It’s necessary to clarify why list the \"Variational data assimilation on satellite image time series\" part.\n\n(2)\tAs for weakness point 3, hyperparameters tuning or other techniques may help.\n\n(3)\tAs for weakness point 4 and 5, it will be better if case study, hyperparameters analysis and complexity analysis provided.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pixD7kwz5b", "reviewer": "Reviewer_kcG2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n   * The presentation of the paper is exceptionally good. I felt that the logic flows very smoothly, and the proposed model is well-motivated.\n   * I personally found the proposed strategy very interesting. The idea of augmenting the encoder but only exploiting the invertible bit of the Koopman output is clever.\n\nWeaknesses:\n   * The paper has limited theoretical insights into the proposed model. In particular, I am interested in how the augmented state helps improve the approximation power of the Koopman unit $\\mathbf{K}$ from an approximation theoretic point of view. I also have some questions regarding the new model listed below.\n   * An ablation study would strengthen the paper, where you show how the performance depends on $p$, the output dimension of the augmented encoder $\\chi$. I suspect that as $p$ gets large enough, the model will no longer benefit from increasing $p$.\n   * The results on the Informer benchmark are promising but not extremely exciting.\n\nOverall, I think this paper has good quality and look forward to a revised version.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new Koopman-operator-based time-series processing model called Augmented Invertible Koopman Autoencoder (AIKAE). The model uses an invertible encoder based on coupling layer normalizing flows. It then augments this encoder with an additional trainable encoder and decodes only the channels that correspond to the coupling layer normalizing flows. In augmenting the encoder, the Koopman operator may be discretized in a much higher-dimensional space, which was a limitation of using only the coupling layer normalizing flows. Empirical results on the Informer Benchmark and a satellite image processing task show the promise of the model.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. On page 4, when you discussed the coupling-layer normalizing flows, the part that starts with \"In addition, the determinant of the Jacobian matrix\" and the next paragraph that starts with \"Using these properties\" sound a bit decoupled from the flow of the narratives. I think the paper can stand without them, and they do not serve very well as an introduction to readers unfamiliar with normalizing flows either. Please consider either compressing them or expanding them for better clarity.\n2. For the model, my main question is as follows: As you mentioned earlier in the paper, the discretized Koopman matrix $\\mathbf{K}$ does not exactly solve the equation in (4). I wonder, when you augment the state vector to have a dimension $d = n + p$, if there are differences between the errors on the first $n$ states and the last $p$ states. Intuitively, correctly matching the first $n$ states is more important than doing so for the last $p$ states because the first $n$ of them are what you used for decoding. In that sense, perhaps you can even explore a weighted least-square that favors matching some states over others. Maybe some discussions and empirical explorations along this direction can be insightful.\n3. For the delayed autoencoder, you wrote \"it is more convenient to do so when the original dimension $n$ of the input space $\\mathcal{X}$ is small. In this regard, the case of univariate time series (i.e. $n = 1$) is of particular interest.\" Have you thought about other embedding ideas as opposed to simply stacking the previous inputs, such as using a mean pooling or a convolution?\n4. In section 4, you mentioned that the model can be used for \"denoising, interpolation, forecasting.\" It is straightforward to see how it can be used for interpolation and forecasting. Can you explain in the manuscript how denoising is possible? Does that relate to the implicit regularization of the model?\n5. Also in section 4, you explained that the trajectory associated with the AIKAE model is nonunique due to the lack of injectivity from $\\mathbf{z}$ to $\\mathbf{x}$. In that case, is there a way to choose which trajectory to use? For example, does a different trajectory give you a different inductive bias of the model?\n6. Minor issues:\n   1. On page 3, please use $\\cdot^+$ instead of $.^+$. Please use a double dash in the term \"Moore--Penrose inverse.\"\n   2. In (14), does this $d$ cause an overhead with the $d$ you used for the dimension of $\\mathbf{z}$?", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work meets the two major requirements for TMLR acceptance and (uniquely) has full support from the reviewers (who were high quality reviewers). In addition, all reviewers believe that the work is ready for publication and have had their comments fully addressed. For this reason, I recommend accept as is.
Decision: Accept (poster)