Structured Matrix Basis for Multivariate Time Series Forecasting with Interpretable Dynamics
Authors: Xiaodan Chen
Xiucheng Li
Xinyang Chen
Zhijun Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Time Series Forecasting
Structured Matrix Basis
Interpretable Dynamics
Primary area: machine_learning_for_other_sciences_and_fields
TL;DR: In this paper, we propose a dynamic multivariate time series forecasting model with a structured matrix basis.

Abstract: Multivariate time series forecasting is of central importance in modern intelligent decision systems. The dynamics of multivariate time series are jointly characterized by temporal dependencies and spatial correlations. Hence, it is equally important to build the forecasting models from both perspectives. The real-world multivariate time series data often presents spatial correlations that show structures and evolve dynamically. To capture such dynamic spatial structures, the existing forecasting approaches often rely on a two-stage learning process (learning dynamic series representations and then generating spatial structures), which is sensitive to the small time-window input data and has high variance. To address this, we propose a novel forecasting model with a structured matrix basis. At its core is a dynamic spatial structure generation function whose output space is well-constrained and the generated structures have lower variance, meanwhile, it is more expressive and can offer interpretable dynamics. This is achieved via a novel structured parameterization and imposing structure regularization on the matrix basis. The resulting forecasting model can achieve up to $8.5\%$ improvements over the existing methods on six benchmark datasets, and meanwhile, it enables us to gain insights into the dynamics of underlying systems.

Structured Matrix Basis for Multivariate Time Series Forecasting with Interpretable Dynamics
Xiaodan Chen', Xiucheng Li² (24), Xinyang Chen', Zhijun Li (x<)
School of Computer Science and Technology, Harbin Institute of Technology
2 School of Computer Science and Technology. Harbin Institute of Technology, Shenzhen {21 (21B303004@stu. lixiuchenge, chenxinyange, lizhijun_ bizhijun_ss@hhiee hitt.it.... edu.
Abstract
Multivariate time series forecasting of central importance in modern intelligent decision systems. The dynamics of multivariate time series are jointly characterized by temporal dependencies and spatial correlations. Hence, it is equally important to build the forecasting models from both perspectives. The real-world multivariate time series data often presents spatial correlations that show structures and evolve dynamically. To capture such dynamic spatial structures, the existing forecasting approaches often rely on two-stage learning process (learning dynamic series representations and then generating spatial structures), which is sensitive to the small time-window input data and has high variance. To address this, we propose a novel forecasting model with structured matrix basis. At its core is a dynamic spatial structure generation function whose output space well-constrained and the generated structures have lower variance, meanwhile, it is more expressive and can offer interpretable dynamics. This achieved via a novel structured parameterization and imposing structure regularization
the matrix basis. The resulting forecasting model can achieve up to 8.5% improvements over the existing methods six benchmark datasets, and meanwhile, it enables us gain insights into the dynamics of underlying systems.
Introduction
Multivariate time series forecasting plays a pivotal role in a wide range of fields, such as traffic flow management, electricity consumption, and weather prediction. Multivariate time series data records quantities of interest from N series spanning over time steps, and its underlying dynamics are jointly characterized by the temporal correlations (intra-series dependencies) and spatial structures (inter-series dependencies). Inspired by the advancements in Natural Language Processing, substantial research has been proposed to apply RNNs and Transformers to capture the underlying temporal dependencies [41, 32, 44, 45, 30]. Besides, the convolution paradigm also exhibits promising temporal correlation modeling capability and excels in long-term forecasting [33, 36, 29).
As the underlying dynamics are jointly described by the intra- and inter-correlations, it is equally important to explore the spatial structures for an ideal forecasting model design. Many proposals employ dense connection to capture the spatial correlations implicitly [31, 24]. However, the dense connection lacks clear structures and is prone introduce noise from uncorrelate spatial dimensions. The development Graph Neural Networks (GNNs) [18, 17, 7] offers an effective solution model non-Euclidean structure data. DCRNN [22] builds graphs based on spatial proximity and conducts graph convolution to capture spatial correlations for traffic forecasting. To apply GNNs to more general scenarios where graph structures are unavailable, the forecasting methods propose to learn the graph adaptively through learnable node embeddings [37, 1, 38, 15], which significantly enhances the forecasting performance.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Despite the progress achieved, the spatial structures remain static across time steps for the aforemen- tioned forecasting methods, which may not reflect the actual inter-series correlation. Because many scenarios, the spatial correlations are also changing dynamically, for example, the traffic speeds of certain road segments manifest correlation only in peak hours. To relax this restriction, many dynamic graph-based methods have been proposed [43, 39, 42, 28, 34] to learn the spatial structures within a short time window. The intuition is that the spatial correlations in real-world applications often evolve continuously and tend to be stable over a short period of time. The general idea is to learn each series a dynamic representation via a nonlinear transformation fayam by taking as input the current time window data, and it then generates the spatial structures by pairwise interacting the dynamic representations via a transformation fpair. the composition of two transformations forms the spatial structure generation function fepatial fpair fdyam- The dynamic representation function fayum is often implemented as MLP ([43, 42, 28, 34 or RNN ([39)) whereas fpair is mostly instantiated by attention mechanism or inner product. One severe issue of these methods is that the output space of Jaymm is not well constrained and unbounded, this makes the learned dynamic representations very sensitive the short time window input data and the unboundedness will be exaggerated by the inner product operation in fpair. which will lead to the outputs of fapatial fluctuate drastically and have high variance. The issue becomes even more severe in the presence of anomaly patterns. To reduce the variance, TPGNN [26] first learns a static spatial structure A (adjacency matrix) and then generates the dynamic spatial structures with a matrix polynomial A. O'm A nt where O.m. is determined by the timestamps of the current time window. However, it has two drawbacks: 1) the coefficient a' solely depends on the timestamps and cannot adapt to the current window data, and 2) the matrix power basis A, A², ..... A Al overly restricted and has limited expressive capability. In addition, the existing forecasting methods often lack interpretable dynamics.
In this paper. we propose a dynamic multivariate time series forecasting model with a structured matrix basis. Instead of relying on the two-stage spatial structure learning process (learning dynamic series representations and then generating spatial structures), we directly parameterize /spatial with a learnable matrix basis B1, B2..... BM and represent any spatial structure with a convex combination
ΣM O. Bm. 0mm 2 0 and Σ m=1 O... 1. To learn the matrix basis effectively, we propose novel structured matrix parameterizatiion method and impose structure regularization on the basis to enhance parameter efficiency and reduce complexity. In contrast to the two-stage spatial structure learning methods, the output space of our proposed Sapatial is well constrained. Consequently, the generated spatial structures have lower variance and the resulting model is easier to learn. In comparison to TPGNN, our matrix basis is more expressive since it is not limited by the matrix power constraint; the coefficient a can also be computed adaptively via the interaction of current time-window data and the basis. In addition, the coefficient a offers a fashion to track the spatial structure evolution and enables us to gain insights into the underlying dynamics. Thus, the resulting model is more interpretable.
In summary, our proposed fapatial has the following appealing properties: 1) lower variance and easier to learn, 2) it is more expressive, and 3) it can yield more interpretable results. This is achieved through novel structured matrix parameterization and structure regularization. By integrating Sspatial into the forecasting framework, we evaluate the efficacy of the proposed method on six benchmark datasets, it achieves up to 8.5% improvements over existing forecasting methods across various prediction lengths and can also offer interpretable dynamics.
2 Related Work
Temporal Dependency Modeling Early deep sequential methods adopt recurrent neural networks to capture nonlinear temporal dynamics [41, 32. 5]. Motivated by the wide receptive fields of attention mechanism, various Transformer-based methods have been developed to capture the long- term temporal dependencies in forecasting. To reduce the quadratic complexity of the vanilla attention, Log Trans [21], Informer [44], Autoformer [35], and FEDformer [45] have been proposed successively. Non-stationary Transformer [27] attempts to mitigate the difficulties caused by non- stationarity in modeling temporal correlation. PSt [30] explores the strategies of patch-level semantic modeling and shannel-independence. VQ-MTM [13] explores the well-defined semantic units for the Transformer architecture in time series modeling. In state-space models, the transition matrix employed to model long-term temporal dependencies, and their recent representative works include Hippo [10]. LSSL [11]. To reduce the computational complexity, S4 [12] and Mamba [9]
2

---

have been proposed successively. The convolution -bssed methods have also shown promising results in time series modeling, MICN [33] explores isometric convolution to capture non-local temporal patterns. TimesNet [36] reshapes 1D signals into 2D by aligning according to inherent multiple periodicity and it then employs the 2D kernels to capture both intraperiod- and interperiod-variatinn Modern CN [29] utilizes large kernels to model long-term dependencies. In addition, the Fourier basis parameterization has also been proposed to model long-term dependencies in time series imputation [25].
Static Spatial Correlation Modeling The spatial correlation plays an equally important role in time series modeling. DeepAR [31] and Pyraformer [24] employ dense connection to model the spatial dependencies. However, the dense connection fails to explore the underlying structure and may introduce noises from unrelated series. The advancement of GNNs offers an effective way to model non-Fuclidean structure data. DCRNN [22] constructs the graph by using spatial proximity and proposes to fuse the spatial information via graph convolution operation, and thus it is only applicable when the underlying graph structures are easily accessible. To sidestep this limitation, the adaptive GNNs-based methods [1, 38, 15, 14, 16] propose to learn each series a representation and then generate the spatial structure via the interaction of the representations. BiTGraph [4] further develops the method to account for missing patterns in message passing to handle the time series with missing values. However, these approaches implicitly assume the series representations are shared across the entire time steps, and hence the underlying spatial graphs remain static over time.
Dynamic Spatial Correlation Modeling In real-world applications, the inter-series correlations or spatial structures are often evolving dynamically. To adapt to these scenarios, many dynamic spatial structure methods are proposed, which share a similar two-stage spatial structure learning process as the static adaptive GNNs-based methods (as discussed in Section 1). The difference is that the series representations are generated by conditioning a small time window rather than the entire timeline data, and hence, the underlying spatial structures can change dynamically over time. GMAN [43], iTransformer [28), Crossformer [42], and Card [34] implement the dynamic node representation function via MLP, whereas ESG [39] adopts the RNN. As the output spaces of their Sapatial are not well constrained, the learned spatial structures are very sensitive to the change of windww data and hence have high variance. To reduce the variance, TPGNN [26] proposes to represent the dynamic graphs with matrix polynomial. As the polynomial coefficients are solely determined by the timestamps, such method fails to utilize the current time-window data. Moreover, the matrix power basis is overly restricted and has weak expressive capability. The proposals [40, 8] attempt to build dynamic graphs by merging temporal and spatial dimensions. Nevertheless, the entanglement of temporal correlations and spatial dependencies makes the models hard to optimize. In addition, the existing multivariate time series forecasting methods also lack interpretability regarding the underlying dynamics.
Methodology
Notation From generative perspective, the multivariate time series X E NxXXD records D-dimensional physical quantities of interest generated by N series (i.e., sensors or instances) over T time steps. We use (()) E RT* to represent the observations from n-th sensor and X, Nx to indicate the observations at the t-th timestamp. The slice notation X,-H:t E Nx HDD denotes the values within window spanning from the time interval |t- 1 H, t). The operator diag: VxN - RVV takes the diagonal elements of a square matrix and returns it as vector, the operator vec reshapes matrix or tensor into a vector.
Overview and Pipeline Figure 1-(a) presents the architecture of our proposed Sumba (dynamic multivariate time series forecasting with structured matrix basis), which comprises L blocks. Each block contains two primary modules: the Multi-Scall TCN and Dynamic GCN modules. The Multi-
( Scale TCN module in the f-th block takes as the input HH:H Nx and generates the
intermediate representation (e-1) N which is fed the Dynamic GCN module H:to
Hitg E xxx The Multi-Scale TCN captures the temporal dependencies by performing multi-scale temporal convolution operation in a channel-independent manner, we choose the kernel sizes x 2, x3, 1 x 6, and 1 7 in this paper. The Dynamic GCN module comprises two functions, namely, the spatial structure generation function fepatial and graph convolution function fgen- Our proposed fspatial generates the dynamic spatial structure A, E p N (adjacency matrix)
to produce ( Z
3

---

(b) Dynamic GCN Modale
V E NXFxD
Matching
Mutching
Profictor Layer
Time 3
U
A Hels
Dynamic GCN Module
xL
Zig-Nit
B
Bar
Mulli-Seallc TCN Module
GCN
Xt,-Mi6 Mido
2
Figure (a) The framework of our proposed Sumba. (b) the detailed structure of the Dynamic GCN module.
by conditioning the intermediate representation 2((-1) E R-NxD time step I [lo H, 10), which is obtained by transforming the current time-window data Xto-Hita- Given the generated dynamic graph A,, fgen further fuses the spatial information of Z!(-11) across different series to yield ( by performing the graph convolution operation. Note that the spatial and temporal dimensions are kept unchanged during the entire transformation in the L blocks.
3.1 Adaptive Dynamic Spatial Structure Generation
The core layer of our proposed forecasting method is the spatial structure generation function fspatial- It serves to infer the optimal graph structure A, that best characterizes the present spatial correlations from the intermediate representation Z (we drop the layer index to keep the notation uncluttered in this subsection). As mentioned in Section 1, the existing methods all adopt the two-stage learning process, which results in unconstrained output function space and high graph structure variance. To address this, we propose to directly parameterize learnable matrix basis B = (Bm } of dimension M where Bm E NNN and define the spatial structure generation function as
M
M
Sepatial (Z)
at.m Bm:
at,m 20,
acm = 1.
(I)
m==
m==1
Here, we choose the convex combination instead of the linear combination to better control the output space. This is reasonable since the basis B is free to optimize in the training stage. The intuition that the basis B can be shared and optimized globally across different time windows, and to infer the spatial structure dynamically, we only need to adaptively compute the coefficient O2 E R M by conditioning on Z!.
behind Eq.
However, two challenges remain in adopting the spatial structure generation function in Eq. 1. I) The number of parameters be learned equals M which grows quadratically to N. As these M matrices lack connection and constraints, learning by brute force will sooner become infeasible even for medium-size N. This actually the reason why the existing methods resort the two-stage learning process, i.e., learning each series an embedding whose learnable parameters are ND with D being the embedding dimension. 2) Intuitively, the best am should be simultaneously determined by Z' and B,, but identifying B., with 2 parameters will make O. hard to compute when N is large.
Structured Parameterization and Regularization To circumvent the two challenges. we propose to parameterize the basis matrices in a structured manner and impose additional structure regularization on the basis. The idea is to represent each basis matrix B,, in its SVD (singular value decomposition) factor product form B., = U Em VT. m and then parameterize the factors U 2m. Vm. where Um, Vm. N are orthogonal matrices, and Σ,, a diagonal matrix consisting of the singular values of B One benefit of such structured parameterization is that it permits us to establish connections between the basis matrices and impose constraints, and consequently, enhance the parameter efficiency and ease of the model learning. To be specific, we impose the constraint that all Bn, for m = 1, .2.....6 . share the same parameterized orthogonal matrices U, and each matrix has its unique Em: i.e.,
B a V V ..... ...UME VT).
(2)

---

The rationality behind such a choice stems from the geometry interpretation of orthogonal matrix- vector multiplication, i.e.. left multiplying vector by an orthogonal matrix equivalent to coordinate transformation, which is provided in Appendix A. Hence, we implicitly require that all basis matrices share the same pair of coordinate transformations, which can be considered sort of implicit regularization since it reduces extra freedom and guides the model to find the coordinate transformations (U and V) that best suit the basis.
Dynamic Coeflicient Generation Such a parameter sharing regularization mechanism along with the structured parameterization also brings another benefit, allows us treat diag(Em) R as a fingerprint to identify each B. Hence, we can compute the a to infer the dynamic spatial structure by simultaneously conditioning on Z; and diag(Em) (m To this end, we design an adaptive matching module that takes as input Z E I NND,, and [diag (ii)g yields the coefficient an the time step, as follows.
diag(2) (MM) ]}. and it
D A (diag E1 (iiag(E2),.... (iag(EM) E NXM
_ vec (7) + Timeenncoiin(())
E ERND.
0r softmax
(W.D) Waz/Vd)
E RRM
where TimeFncoding is the timestamp encoding function, Wa and W, are used to match the dimension, i.e., mapping D and z into the Rd space. Given the structured basis in Eq. 2, each represents a dynamic spatial structure or graph at step t as
M
A, = Sspatial (Z,)
Σ 1m
ar,m U. VT.
(4)
Given A, at each time step, we can perform graph convolution operation aggregate information from the spatial dimension. The process is illustrated in Figure 1-(b). The output space of fspatial is well constrained on the premise that E,m are well bounded, which is given by the following theorem.
Theorem 3.1. The output space of fapatial in Eq. 4 is bounded by the sum of the maximum Em (the maximum singular value of B.n) for m = 1,2.... M in terms of the Eg norm i.e., |I S
M Em=r max(2m).
The proof is presented in Appendix A.2 and Theorem 3.1 states that the variance of the learned structures controllable by restricting the maximum value of Em. which is easy to achieve since En, can simply be parameterized by vector with nonnegative values. Besides, by tracking the change of a₂ over time, our proposed method enables us to gain insight into the underlying dynamies of the system, and thus offers additional interpretability, as we will show in Section 4.4.
Orthogonality To impose the orthogonal constraint, one may attempt to apply orthogonality penalty, i.e., by adding the penalty term | - I|| + V - I|| to the optimized objective. However, such hard penalty cannot guarantee genuine orthogonality and the extra penalty term also increases the learning difficulty. Hence, rather than relying on the hard penalty, we opt for the orthogonal parameterization. In particular, we restrict our attention the special orthogonal group
SO(N) & (A E NxN A T A = I, (dee((A) -1.
(5)
which are flexible enough to represent the coordinate transformation. A nice property of SO(N) is that it is both a compact Lie group and a smooth manifold [6]. The tangent space of the Lie group at the identity forms vector space equipped with Lie bracket operation, namely, its Lie algebra. The Lie algebra of SO(N) is the set of all skew-symment matrices, denoted by so(N), i.e.,
(N & {A N A -A}.
(6)
In differentiable manifold and Lie group, well-known result is that the matrix exponential exp so > SO establishes the connection between Lie group and its Lie algebra, i.e., for any E so we have exp(g) E SO(N). The matrix exponential is defined as
20
1 A 2
exp(A) A
A I+ A
k=0
5

---

The computation of the matrix exponential map is costly but for the special orthogonal group it has cheap first order approximation, also known as the Clay map [3, 2].
(+12)
I (1- 21
((A) - I
(8)
The Clay map can be implemented in parallel by the Gaussian elimination algorithm in a numerically stable way.
Low Rank Approximation By using the Clay map, the number of parameters required to parame- terize U and V N(N -) (two skem-symmetrie matrices N (NN 1)/2 + NNN - 1)/2). Thus the total number of parameters required by the basis in Eq. 2 is N(N - 1) MN. To further reduce real-worle applications, the spatial the parameter count, we can apply the low rank approximation. structures (adjacency matrices) of multivariate time series are often low-ranked. Suppose the rank is K, we can only preserve the first K columns U and V as well as parameterize each E.m with a length K vector, which leads to a parameter count upper bound NK + M K. Since K is often much less than N in practice, the low rank approximation can enhance both the parameter and computation efficiency when N is large.
3.2 Hierarchical Architecture
By stacking L blocks, the Sumba significantly enhances its capability to model temporal correlations and spatial dependencies effectively. We initialize ( with the original input X RR The output of L-th block Z (4) E IRN xgN*HxD'' x produces the multi-step prediction Y to:to+F through a linear transformation. The model is optimized by minimizing mean absolute error (MAE):
En
1o+F=1 F-1
(n)
Va (m)
MMAA Es Y xl &
t=
Dt
NxF
Experiments
In this section, we evaluate our approach Sumba against time series forecasting methods on six benchmark datasets (Section 4.2 and Appendix C). The ablation studies are presented in Section 4.3. We demonstrate the interpretability of our method with case studies Sec- tion 4.4. The sensitivity of hyperparameters is provided in Section 4.5 and Appendix D; the computational cost is empirically studied in Appendix E. The code of Sumba is available at: https://gth thub. com/
4.1 Experimental Setup
Datasets We conduct experiments on six commonly adopted public datasets including: (1) Elec- tricity [44] contains hourly electricity consumption of 321 clients.
(2) Weather [35] includes 21
meteorological factors collected every 10 minutes from the weather station of the Max Planck Biogeo- chemistry Institute. (3) PEMS [23] records traffic data of 358 variates in California sampled every
5 minutes. (4) ETTh2 [44] contains hourly data from electricity transformers. (5) Traffic [35] measures the hourly road occupancy rates of 862 sensors San Francisco Bay area freeways. (6)
Solar-Energy [20] records the solar power production, which is sampled every 10 minutes from 137 PV plants.
Baselines We compare our method with the following baselines: (1) TCN- based methods:
MICN [33], ModernTCN [29]; (2) Transformer-based methods: PatchTST [30]. FEDformer [45],
Autoformer [35], Reformer [19]; (3) Static graphs-based methods: MTGNN [38], MegaCRN [15];
(4) Dynamic graphs-based methods: |Transformer [28]. Crossformer [42], Card [34], ESG [39].
TPGNN [26], FourierGNN [40]; (5) Structured State Space model: S4 [12]. More details of
baselines are provided in Appendix B.
Implementation details The number of blocks L of Sumba is set to 3, the dimension of structured basis M is set to 5, and the rank K is set to min(N, (,,()) 30) in all our experiments. The batch size is 32, the learning rate 0.0001. We split the datasets into training, validation, and test datasets with the ratio 0. 0.6/0.200 2/0.2 chronologically. The future window size F is set to 3, 6, 12, and 24 for all

---

Table 1: The forecasting results with prediction horizons of 3 and 6 on Electricity, Weather, etc.
Method
Electricity RMSE
Weather RMSE
(F = 3)
MAE
MAPE(S)
MAE
MAPE(%)
MICN [33]
0.217 + 0.001
U.323 + 0.001
2.380 + 0.012
0.0812 + 0.001
0.217 + 0.002
1.563 + 0.017
ModernTON [29)
0.172 = 0.001
0.259 = 0.001
1.629 0.008
0.0727 0.000
0.213 0.001
1.695 +: 0.056
PatchTST [30]
0.173 + 0.001
0.266 = 0.002
1.359 = 0.006
0.0642 + 0.000
0.214 0.000
2.511 0.223
FEDformer [45]
0,268 + 0.002
0,350 = 0.002
1444=00001.
0.236 + 0.001
0.364 0.001
4.832 4 0.039
Autoformer [35]
0.271 +: 0.003
0.383 = 0.001
2.543 = 0.005
0.238 + 0.001
0.386 0.002
7.133 4: 0.693
Reformer [19]
0.324 + 0.002
0,452 + 0.002
2.921 0.010
0.0821 + 0.001
0.220 + 0.002
1,533 + 0.022
S4 [12]
0.331 + 0.002
0.479 + 0.002
2.792 + 0.015
0.0921 + 0.001
0.292 +: 0.003
4.782 + 0.055
MTGNN [38]
0.153 + 0.001
0.245 = 0.002
1.340 = 0.004
0.0693 = 0.000
0.219 0.001
1.624 + 0.018
MegaCRN [15]
0.272 0.001
0.394 0.002
2.429 0.026
0.0674 0.000
0.215 0.001
1.458 1 0.032
Transformer [28]
0.158 + 0.001
0.248 + 0.000
1.472 = 0.007
0.0644 + 0.000
0.216 +: 0.001
1,628 + 0.026
CrossFormer [42]
0.156 + 0.000
0.216 = 0.001
1.370 = 0.008
0.0643 - 0.002
0.210 + 0.003
1417 + 0.035
Card [34]
0.152 + 0.001
0.244 0.001
1.480 0.013
0.0641 0.000
0.215 0.001
1.523 0.007
ESG [39]
0.153 = 0.002
0.247 = 0.001
1.434 = 0.005
0.0753 = 0.000
0.215 + 0.001
1.837 + 0.072
FourlerGNN [40]
0.213 + 0.002
0.327 = 0.003
1.619 = 0.006
0.0859 - 0. 0.0
0.229 + 0.002
1.732 + 0.031
TPGNN [26]
0,206 + 0.003
0.324 0.001
1.677 = 0.009
0.0652
- 0.000
0.219 + 0.001
1.538 + 0.024
Sumba
0.148 + 0.001
0.237 = 0.000
1,331 = 0.005
0.0587 - 0.000
0.208 1 0.000
1.381 + 0.011
Method
ETTh2 RMSE
Traffic RMSE
(F ("=33 =3
MAE
MAPE(%)
MAE
MAPE(%)
MICN [33]
0.225 = 0.002
0.341 = 0.001
1.126 0.008
0.253 = 0.001
0.577
+: 0.001
2.794 + 0.013
ModemTON [29]
0.212 : 0.002
0.340 0.002
1153
0.239 0.002
0.562 1002:000 0.002
3.210 : 0.036
PatchTST [30]
0,182 : 0.001
0.285 = 0,001
0.854 0.003
0.226
0.001
0.532 0.002
2.410 1 0.009
FEDformer [45]
0.332 1 0.012
0.480 = 0.006
1.522 1: 0.023
0.346 0.006
0.009
4.010 + 0.041
Autoformer [35]
0.299 = 0.003
0.438 0.004
1.344 0.017
0.361 = 0.003
0.728:0.00
3.956 : 0.026
Reformer [19]
0.442 + 0.006
0.663 0.011
2.091 + 0.025
0.322 0.007
0.757 0.014
3.351 + 0.038
S4 [12]
0.441 = 0.004
0.617 = 0.018
2.749 + 0.017
0.295 0.003
0.734 0.003
2.647 = 0.011
MTGNN [38]
0.185 + 0.001
0.289 0.001
0.855 + 0.004
0.203
0.002
0.525 + 0.003
2.332 + 0.008
MegaCRN [15]
0.237 0.002
0.352 0.001
1.177 : 0.005
0.326 = 0.002
0.735 0.007
4.0G4 = 0.028
iTranstormer [28]
0.188 + 0.000
0.294 = 0.002
0.906 + 0.001
0.195 0.001
0.492 0.001
2.155 + 0.007
CrossFormer [42]
0.206 + 0.001
0.325 = 0.002
0.927 + 0.003
0.198 0.002
0.602 - 0.006
2.079 + 0.021
Card [34]
0,182 + 0.000
0.287 = 0.000
0.809 + 0.006
0.197 = 0.001
0.496
2.147 = 0.007
ESG [39]
0.217 = 0.001
0.335 0.009
0.909
1 0.015
0,1
0.001
0.588 1 0. 0.003
2.086 : 0.004
FourierGNN [40]
0.232 + 0.003
0.376 0.005
1.436
+ 0.016
0,264
0.002
0.585 + 0.002
2.934 + 0.019
TPGNN [26]
0.207 + 0.001
0.329 = 0.002
0.986-
1 0.004
0.288
0.003
O. 0.665-2 1 0.004
3.137 + 0.022
Sumba
0.179 = 0.000
0.286 = 0.001
0.850 + 0.001
0.192
0.001
0.521 + 0.002
2.042 = 0.015
Method
Electricity RMSE
Weather RMSE
(F 6)
MAE
MAPE($)
MAE
MAPE(S))
MICN [33]
0,235 0.002
0.317 0.001
1.890
0.009
0.111 = 0.001
0.250 + 0.001
2.886 :: 0.013
ModernTON [29]
0,191 0.002
0.256 1 0.001
1.719
0.005
0,0934 + 0.000
0.211 + 0.001
1.712
PatchTST [30]
0.188 + 0.001
0.290 + 0.001
1.547 + 0.009
0.0746 + 0.000
0.239 + 0.001
3.138 + 0.096
FEDformer [45]
0.271 0.002
0.384 0.001
2.541 0.008
0.230 = 0.001
0.364 + 0.003
4.118 0.027
Autolormer [35]
0,269 0.004
0.378 0.002
2.643
0.007
0.246 - 0.002
0.399 : 0.003
12.782 1.039
Reformer [19]
0.348 + 0.001
0.483 00 0.001
3.180 = 0.006
0.117 = 0.001
0.263 + 0.002
3.611 + 0.025
$4 [12]
0.341 0.003
0.489
0.002
3.265
0.009
0.0926 + 0.001
U.285 + 0.003
3.932 0.037
MTGNN [38]
0.169
0.268 0.003
1.636 0.003
0,0817 + 0.000
0.256 1: 0.001
1.987 0.017
MegaCRN [15]
0,292 = 0.002
=.119
2.743 1443300000 0.011
0.0819 + 0.000
0,212 0.001
1.972 + 0.019
Transformer [28]
0.171 + 0.002
0.269 0.001
1.578 + 0.005
0.0804 + U.000
U.243 + 0.002
1.774 + 0.023
CrossFormer [42]
0.169
0.270 0.001
1.541 0.006
0.0882 + 0.000
0.237 +: 0.002
2.931 0.059
Card [34]
0,166 0.001
0.267 + 0.001
1.576 1176100 0.002
0.0778 + 0.000
0,212 1: 0.002
1.748 + 0.009
ESG [39]
0.168 + 0.002
0.275 1 0.003
1.549 + 0.003
0.0932 + 0.005
U.242 + 0.001
2.322 + 0.026
FourierGNN [40]
0,237 0.003
0.369
0.002
1.978
0.007
0.0935 + 0.004
0.257
:: 0.002
2.869 0L038
TPGNN [26]
0.234 0.002
0.367
0.002
2.042
0.007
0.0782 + 0.000
U.246
+ 0.002
1.731 0.030
Sumba
0.163 0.001
0,260
0.001
1,530
0.005
0.0704 + 0,000
0.234 + 0.002
1.663
0.013
Method
ETTh2
Traffic RMSE
(F = 6)
MAE
RMSE
MAPE(%)
MAB
MAPE(S)
MICN [33]
0,281 :: 0.001
0.405 0.001
1.692 + 0.006
0.275 = 0.002
0.571 # 0.001
3.369 + 0.019
Modem [29]
0.272 = 0.002
0.380 = 0.003
1.819
+ 0.015
0.261 = 0.002
0.597 + 0.001
3.219 + 0.028
PatchTST [30]
0.210 :: 0.002
0.328 0.002
0.956
+ 0.005
0.231 = 0.001
0.554 0:64:00 0.002
2.465 : 0.012
FEDformer [45]
0.349 + 0.006
0.508 0.007
1.523
+ 0.021
0.344 0.003
0.718 0.007
3.845 + 0.033
Autoformer [35]
0.322 1: 0.002
0.473 = 0.003
1.350 + 0.010
0.360 0.002
0.730 + 0.003
3.891 1 0.018
Reformer [19]
0.488 + 0.012
0.690 0.007
2.604
+ 0.031
0.338 0.002
0.775 + 0.006
3.613 + 0.025
S4 [12]
0.562 + 0.003
0.747 0.009
2.755 + 0.021
0.301 = 0.002
0.738 0.003
2.952 + 0.020
MTGNN [38]
0.212 + 0.001
0.323 0.002
0.994 + 0.006
0.227 0.002
0.551 + 0.004
2.562 + 0.007
MegaCRN [15]
0.287 + 0..04
0.411 + 0.003
1.437 + 01.009
0.338 0.002
0.754 + 0.002
3.994 + 0.015
Transformer [28]
0.212 = 0.001
0.329 = 0.001
1.029 + 0.003
0.211
0.002
0.524 0.001
2.269 - 0.005
CrossFormer [42]
0.249 0.002
0.387 0.001
1.097 + 0.006
0.216
0.002
0.634 0.003
2.140 0.014
Card [34]
0.209 + 0.001
0.319 0.001
0.951 + 0.004
0.215 0.001
0.545 0.002
2.311 - 0.006
ESG [39]
0.254 + 0.006
0.371 = 0.013
1.027 + 0.012
0.221 = 0.002
0.626 -: 0.002
2.248 - 0.003
FourierGNN [40]
0.265 + 0.002
0.393 = 0.006
1.933 + 0.008
0.291 = 0.001
0.665
1 0. 0.002
3.608 + 0.028
TPGNN [26]
0.236 - 0.001
0.379 0.003
L.092 : 0.012
0.268 0.002
0.726 0.004
3.185 + 0.035
Sumba
0.204 1 0.001
0.323 0.001
0.935 1 0.003
0.210
0.002
0.529 0.002
2. 2.196 + 0.013
7

---

Table 2: The forecasting results with prediction horizons of 3 and 6 on PEMS and Solar datasets.
Method
PEMS RMSE
Solar-Energy RMSE
(F = =3
MAE
MAPE(%)
MAE
MAPE(%)
MICN [33]
0.155 = 0.001
0.228
0.002
1.180 + 0.004
0.077 = 0.000
0.136 0.001
0.782 = 0.003
ModemTCN [29]
0.143 : 0.001
0.211 0.000
1.102 : 0.003
0.065
0.000
0.125
0.001
0,608 : 0.002
PatchTST [30]
0.179 + 0.001
0.242 0 0.001
1.137 0.003
0.082
0.000
0.142 1:42:00 0.001
0.617 : 0.002
FEDformer [45]
0.246 + 0.002
0.349 - 0.003
1.765 1: 0.005
0.169
0.001
0.247 1: 0.001
1.224 + 0.006
Autoformer [35]
0.462 = 0.003
0.670 0.002
2.466 0.008
0.213
0.001
0.290 1: 0.002
1.340 = 0.004
Reformer [19]
0.190 + 0.001
0.300 0.002
1.506
+: 0.003
0.068
0.000
0.147 + 0.001
0.818 0.004
S4 [12]
0.232 +: 0.002
0.331 0.003
2.029 1 0.007
0.092
0.001
0.139 + 0.001
0,974 + 0.003
MTGNN [38]
0.145 + 0.000
0:215
0.000
1.127 + 0.004
0.060
0.000
0.135
0.001
0.629 + 0.002
MegaCRN [15]
0.154 0.001
0.232 0.003
1.209
0.004
0.057
0.000
0.129 0.001
0.658 0.002
iTransformer [28]
0.146 + 0.001
0.215 = 0.001
1.147 + 0.007
0.060
0.000
0.126 + 0.001
0.609 + 0.002
CrossFormer [42]
0,144 0.001
0.219 0.001
1.139 + 0.002
0.047
0.000
0.116 - 0.001
0.553 - 0.003
Card [34]
0.155 = 0.001
0.226 = 0.001
0.001
1.139 + 0.002
0.057 = 0.000
0.134 0.001
0.684 = 0.002
ESG [39]
0.142 - 0.001
0.212 0,001
1.089 0.003
0.050
0.000
0.122 0.001
0,606 0.002
FourierGNN [40]
0,154 = 0.001
0.236
0.002
1.292 + 0.003
0.063
0.000
0.129 + 0.002
0.617 = 0.003
TPGNN [26]
0.149 + 0.001
0.230 = 0.001
1.157 + 0.005
0.000 0.059
0.138 + 0.002
0.612 + 0.002
Sumba
0.137 = 0.001
0.204 = 0.000
1.000 + 0.002
0.046 0.000
0.114 + 0.002
0.541 = 0.002
Method
PEMS RMSE
Solar-Energy RMSE
(F =P=00
MAE
MAPB(%)
MAE
MAPE(%)
MICN [33]
0.176 = 0.001
0.239 = 0.001
1.355 # 0.003
0.094 = 0.000
0.175 + 0.001
0.889 + 0.003
ModemTTNN [29]
0.154 = 0.001
0.228 0.003
1.218 0.006
0.086 = 0.000
0,161 0.002
0.796 : 0.003
PatchTST [30]
0.189 0.001
0.262 0.002
1.226 0.003
0.096 0.000
0.183
0.841 0.002
FEDformer [45]
0,244 = 0.001
0.348 0.002
1.761 + 0.002
0.169 0.001
0.253 1 0.001
1.305 + 0.002
Autoformer [35]
0.515 0.002
0.729 = 0.004
2.881 + 0.003
0.246 = 0.002
0.333 - 0.002
1.528 : 0.002
Reformer [19]
0.202 + 0.001
0.316 0.002
1.655
+: 0.003
0.083
0.000
0.175 1: 0.001
0.919 + 0.004
$4 [12]
0,214 = 0.003
0.344 0.005
2.177 :: 0.006
0,101
0.001
0.158 0.002
1.156 : 0.004
MTGNN [38]
0.158 + 0.001
0.234 + 0.001
1.229
+ 0.003
0.083
0.000
0.169 + 0.001
0.810 + 0.002
MegaCRN [15]
0.167 0.002
0.254 + 0.003
1.356 1 0.005
0.087
0.000
0.172 1 0.002
0.924 + 0.005
iTrasformer [28]
0.157 + 0.001
0.233 0.002
0.003
0.071
0.000
0.001
0.823
CrossFormer [42]
0.151 + 0.001
0.233 0.002
1.234 + 0.002
0.063
0.001
0.156 0.002
0.739 + 0.003
Card [34]
0.152 = 0.002
0.228 = 0.003
1.182 + 0.004
0.062
0.000
0.160 + 0.002
0.775 = 0.003
ESG [39]
0.151 + 0.001
0.226 0.001
1.159 + 0.004
0.068
0.000
0.158 + 0.001
0.757 + 0.005
FourierGNN [40]
0.182 = 0.003
0.267
0.002
13377 + 0.003
0.096
0.001
0.177 0.002
0.872 + 0.002
TPGNN [26]
0,169 + 0.001
0.256
0.002
1.292 + 0.003
0.090
0.000
0,196 + 0.002
0.872 + 0.002
Sumba
0.147 = 0.001
0.222 0.002
1,155 + 0.102
0.062 0.000
0,147 0.001
0.734 = 0.002
methods, and the history window size H is set to 168. All methods are trained on Nvidia V100 GPUs. Our method is implemented with PyTorch 2.0 and we use the source codes publicly released by the authors for all baseline methods. We adjust the hyperparameters of baseline methods to obtain the best performance on each dataset, and evaluate the performance of different methods in terms of Mean Absolute Error (MAE). Root Mean Square Error (RMSE), and Mean Absolute Percentage Error (MAPE).
4.2 Forecasting Performance
Table and 2 present the forecasting performance with F E {3, 6) on six benchmark datasets, with results evaluated using three different random seeds. The best results are highlighted in bold, while the second-best results are underlined. PStchTST and ModernTCN demonstrate the best performance among methods without explicitly modeling spatial correlations. Besides, graph-based spatio-temporal methods yield better results than those that do not account for spatial structures. In addition, Card, Crossformer, iTransformer, and ESG outperform static graph- based methods such as MTGNN and MegaCRN, highlighting the significance of explicitly modeling dynamic spatial structures. Our method Sumba achieves state-of- ofate-of-the-arr offtthe-rt art performance in most cases, demonstrating an improvement up to 8.5% over the best baseline on the Weather dataset. This superior performance is attributed to its capability to produce dynamic graph structures with low variance and high expressiveness using the proposed structured matrix basis. The forecasting results for additional prediction horizons are provided in Appendix C.
8

---

Table 3: Ablation study.
Electricity RMSE
Weather
PEMS
==3
MAE
MAPE(%)
MАE
RMSE
MAPE(%)
MAE
RMSE
MAPE(S)
w/o. dynamic
0.153
0.245
1.340
0.0693
0.219
1.624
0.145
0.215
1.127
w/. Um. Vm
0.148
0.238
1,251
0.0659
0.214
2.255
0.138
0.205
1.054
w/o. orthogonality
0.151
0.242
1.337
0.0633
0.213
1.605
0.142
0.209
1.094
Sumba
0.148
0.237
1,331
0.0587
0.208
1.381
0.137
0.204
1.060
Electricity
Weather
PEMS
6=
MAE
RMSE
MAPE(S)
MAE
RMSE
MAPE(%)
MAE
RMSE
MAPE(%)
w/o. dynamic
0.169
0.268
1.636
0.0817
0.256
1.987
0.158
0.234
1.229
w/. Um. V
0.164
0,262
1.537
0.0722
0.234
2.116
0.149
0.227
1.162
w/o. orthogonality
0.167
0.265
1.578
0.0706
0.239
1.690
0.151
0.226
1.179
Sumba
0.163
0.260
1,530
0.0704
0.234
1.663
0.147
0.222
1.155
Figure The change of a over one week.
4.3 Ablation Study
We conduct the ablation study on the Electricity, Weather, and PEMS datasets with prediction horizons F of 3 and 6 to verify the efficacy of the proposed modules. particular, we consider the following replace the Dynamic GCN module with the vanilla GCN module in [38]. (2) w/. Um, Vn. we use distinct Um, Vn m for each basis matrix remove the orthogonal parameterization
variants of our proposed model: (1) w/o. dynamic:
U and V. As shown in Table 3, the performance w/o. dynamic declines rapidly by up to 13. 13.83%, 8. 8.59%, and 16.31% on three metrics, which verifies the efficacy of the proposed dynamic GCN model. The absence of structure regularization (w/. Un and the elimination of orthogonal parameterization (w/o. orthogonality) structured matrix basis lead to averaged performance drops of 4.95% and 3.05%, respectively. This proves the effectiveness of the proposed structure regularization and orthogonal constraint strategy.
Bm (3) w/o. orthogonality:
Interpretable Dynamics
As discussed in Section 3.1, one appealing feature of our proposed Sumba is that it enables to gain insights into the underlying time series dynamics by tracking the change of the matrix basis coefficient
time. To verify this, present the heatmap of the change of Solar-Energy and Electricity in Figure 2, in which the x-axis denotes the time (hourly), the y-axis represents the no. of 5 basis matrices, and the color indicates the weight of a. can be observed that most of the weights are concentrated on two basis matrices, no. and no. 4, which implies that there are dominant spatial structures on the two datasets. Furthermore, the two dominant spatial structures appear alternatively and regularly, which actually corresponds to the day and night for the Solar-Enes Energy dataset. This aligns well with our intuition that solar energy observations should manifest different spatial correlations as the light intensity varies. For the Electricity dataset, there a spatial correlation (no. 4) that spans two hours and only emerges in midnight, this reveals an interesting electricity consumption pattern, which is unnoticed by previous methods. Therefore, our proposed method is able to offer more interpretable dynamics the underlying systems through tracking
datasets-

---

Pacsscy
Veather
PUUS
0365
01.8
1148
0:44
01020
0.46
CTTY
0000
0.96
6040
6040
6047
01
Figure 3: The sensitivity of the dimension of basis M.
4.5 Parameter Sensitivity Analysis
We evaluate the impact of M on performance of our model, where the M ranges from 1, 2, 5, to 10. M corresponds to the static graph. As shown in Figure 3, the performance of our model improves significantly when M > 1. When M = 2, our method achieves the best performance on Electricity and Solar-Energy datasets, this coincides with the fact that the two datasets have two dominant spatial structure patterns, i.e., day and night. The experiments empirically show that the best setting of M on ETTh2, Traffic, and PEMS datasets and on the Weather dataset. We hypothesize this is due to the Weather dataset having more complex dynamics, and thus it requires more basis matrices to cover the various patterns. The sensitivity analysis of L, K and H presented in Appendix D.
5 Conclusion
In this paper, we propose a time series forecasting method with the structured matrix basis, Sumba, to capture dynamic spatial structures. To this end, propose novel structured parameterization and impose structure regularization on the basis to enhance parameter efficiency, the output space of the spatial structure function is thus well constrained and the generated spatial structures have lower variance. Our proposed method offers us a manner to gain insights into the dynamics of the underlying systems, and thus is more interpretable. The experiments verify the superiority of our proposed method. Our extensive ablation studies prove the effectiveness of each proposed component. In addition, the case study shows that our method can offer desirable interpretability. In the future, we would like to explore how to better regularize the learned matrix vector space and set the dimension of basis M in a data-driven manner. We will also explore the possibility of integrating our proposed spatial structure modeling with other temporal encoders such as Transformers, and Structured State Space models, and apply our interpretable dynamics into more datasets to discover more interesting and hidden patterns.
six benchmark datasets
Acknowledgements
This work supported by National Natural Science Foundation of China under Grant No. 62206074, Grant No. 62306085, Grant No. 62072137, Shenzhen College Stability Support Plan under Grant No. GXWI2002081113233300 and the National Key R&D Program of China under Grant No. 2023YFB4503100.
10

---

References
[1] Lei Bai, Lina Yao, Can Li, Xianzhi Wang, and Can Wang. Adaptive graph convolutional recurrent network for traffic forecasting, In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[2] Mario Lezcano Casado. Trivializations for gradient-based optimization on manifolds. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[3] Mario Lezcano Casado and David Martínez-Rubic Cheap orthogonal constraints in neural networks: A simple parametrization of the orthogonal and unitary group. International Conference Muchine Learning (ICML), 2019.
[4] Xiaodan Chen, Xiucheng Li, Bo Liu, and Zhijun Li. Biased temporal convolution graph network for time series forecasting with missing values. In International Conference on Learning Representations (ICLR), 2023.
[5] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv: /412.3555 2014.
[6] Jean Gallier and Jocelyn Quaintance. Differential Geometry and Lie Groups: A Computational Perspective. Springer, 2020.
[7] Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. In International Conference on Learning Representations (ICLR), 2019.
[8] Jake Grigsby, Zhe Wang, Nam Nguyen, and Yanjun Qi. Long-range transformers for dynamic spatiotemporal forecasting, arXiv preprint arXiv:2109.1222 2021.
[9] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint urXiv:2312.007555 2023.
[10] Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[11] Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-time models with linear state space layers. In Advances in Neural Information Processing Systems (NeurlPS), 2021.
[12] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations (ICLR), 2022.
[13] Haokun Gui, Xiucheng Li, and Xinyang Chen. Vector quantization pretraining for eeg time series with random projection and phase alignment. In International Conference Machine Learning (ICML). 2024.
[14] Shengnan Guo, Youfang Lin, Huaiyu Wan, Xiucheng Li, and Gao Cong. Learning dynamics and heterogeneity of spatial-temporal graph data for traffic forecasting. IEEE Trans. Knowl. Data Eng., 2022.
[15] Renhe Jiang, Zhaonan Wang, Jiawei Yong, Puneet Jeph, Quanjun Chen, Yasumasa Kobayashi, Xuan Song. Shintaro Fukushima, and Toyotaro Suzumura. Spatio-temporal meta-graph learning for traffic forecasting. In Association for the Advancement of Artificial Intelligence (AAAI), 2023.
[16) Yue Jiang, Xiucheng Li, Yile Chen, Shuai Liu, Weilong Kong, Antonis F. Lentzakis, and Gao Cong. A scalable adaptive graph diffusion forecasting network for multivariate time series forecasting, In 40th IEEE International Conference Data Engineering (ICDE), 2024.
[17] Gilmer Justin, S. Schoenholz Samuel, F. Riley Patrick, Vinyals Oriol, and E. Dahl George. Neural message passing for quantum chemistry. International Conference on Machine Learning (ICML), 2017.
11

---

[18] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. International Conference on Learning Representations (ICLR), 2016.
[19] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations (ICLR), 2020.
[20] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. International ACM SIGIR conference on research & development in information retrieval, 2018.
[21] Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In Advances in neural information processing systems (NeurIPS), 2019.
[22] Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural net- work: Data-driven traffic forecasting. In International Conference on Learning Representations (ICLR), 2018.
[23] Minhao LIU, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia LAI, Lingna Ma, and Qiang Xu. Scinet: Time series modeling and forecasting with sample convolution and interaction. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[24] Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International Conference on Learning Representations (ICLR), 2021.
[25] Shuai Liu, Xiucheng Li, Gao Cong, Yile Chen, and Yue Jiang. Multivariate time-series imputation with disentangled temporal representations. In International Conference Learning Representations (ICLR), 2023.
[26] Yijing Liu, Qinxian Liu, Jian-We Zhang, Haozhe Feng, Zhongwei Wang, Zihan Zhou, and Wei Chen. Multivariate time-series forecasting with temporal polynomial graph neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[27] Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Exploring the stationarity in time series forecasting. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[28] Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. In International Conference on Learning Representations (ICLR), 2024.
[29] Donghao Luo and Xue Wang. Modernten: A modern pure convolution structure for general time series analysis. In International Conference on Learning Representations (ICLR), 2024.
[30] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-tern forecasting with transformers. In International Conference on Learning Representations (ICLR). 2023.
[31] David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks, International Journal of Forecasting, 2020.
[32] Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. Lstm neural networks for language modeling. In Interspeech, 2012.
[33] Huiqiang Wang, Jian Peng, Feihu Huang, Jince Wang, Junhui Chen, and Yifei Xiao. Mien: Multi-scale local and global context modeling for long-term series forecasting. In International Conference on Learning Representations (ICIR), 2023.
[34] Xue Wang, Tian Zhou, Qingsong Wen, Jinyang Gao, Bolin Ding, and Rong Jin. Card: Channel aligned robust blend transformer for time series forecasting. In International Conference on Learning Representations (ICLR), 2024.
12

---

[35] Haixu Wu, Jiehui Xu, Jianmin Wang. and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[36] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Times- net:temporal 2d-variation modeling for general time series analysis. In International Conference on Learning Representations (ICLR), 2023.
[37] Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. Graph wavenet for deep spatial-temporal graph modeling. In International Joint Conference on Artificial Intelligence (IJCAI), 2019.
[38] Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. Connecting the dots: Multivariate time series forecasting with graph neural networks. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining (SIGKDD). 2020.
[39] Junchen Ye, Zihan Liu, Bowen Du, Leilei Sun, Weimiao Li, Yanjie Fu, and Hui Xiong. Learning the evolutionary and multi-scale graph structure for multivariate time series forecasting. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining (SIGKDD), 2022.
[40] Kun Yi, Qi Zhang, Wei Fan, Hui Hc, Liang Hu, Pengyang Wang. Ning An, Longbing Cao, and Zhendong Niu. Fouriergnn: Rethinking multivariate time series forecasting from a pure graph perspective. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
[41] Wojciech Zaremba, llya Sutskever, and Oriol Vinyals. Recurrent neural network regularization. arXiv preprint arXiv:1409.220 1409.2329,99 2014.
[42] Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizzig cross-dimension dependency for multivariate time series forecasting. In International Conference on Learning Representa- tions (ICLR), 2023.
[43] Chuanpan Zheng, Xiaoliang Fan, Cheng Wang, and Jianzhong Qi. Gman: A graph multi- attention network for traffic prediction. In Association for the Advancement of Artificial Intelligence (AAAI), 2020.
[44] Haoyi Zhou, Shanghang Zhang. Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Association for the Advancement of Artificial Intelligence (AAAI), 2021.
[45] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. International Conference on Machine Learning (ICML), 2022.
13

---

A Background and Proof of the Theorem
A.1 Geometry Interpretation of Matrix Multiplication
Recall that given a matrix A = UEVT, the multiplication Ax can be interpreted as 1) transforming x into the new coordinate system x' _ V x, 2) doing the elementwise product the new coordinate Ex', and 3) transforming the coordinate system again y Uy'.
Ax UEV UEx Uy'.
(10)
A.2 Proof of the Theorem
Theorem A.1. The output space of fspatial Eq. is bounded by the sum of the maximum Em (the maximum singular value of B,)) for m = 1, 2, M in terms of the Ey norm i.e, Iasattialll S
of
Em max(Em)
M
Proof.
||
Σ
am
m VT
(11)
m=1
22
Omm
UEmm V 1l2
(triangle inequality)
(12)
m=1
Σ
Um V 112
(0 < am <1)
(13)
| 112
(14)
m=1
M
Σ
Em ll2
(U, are orthogonal matrices)
(15)
m=1
Σ
max max(Em) (Em).
(16)
The proof is completed.
B
Implementation Details
B.1 Baselines
our experiments, we compare our proposed Sumba with state-of-the-arl (SOTA) baseline models.
MICN: It utilizes temporal convolution networks and isometric convolution networks to capture local and global temporal correlations, respectively.
MCdernTCN: It leverages large kemnlls build long-term temporal dependencies.
PatchTST: It introduces patch-based attention mechanism and chamnel-independent
strategy to establish long-term temporal correlations.
FEDformer: It leverages eiqueency-enaance attention capture long-term temporal
dependencies and employs frequency sampling to reduce the complexity.
Autoformer: It designs auto-correlation module to discover the sub-series similarity based periodicity and aggregates similar sub-series from underlying periods.
Reformer: It utilizes locality sensitive hashing strategy to reduce the complexity of the attention mechanism.
S4: It captures long-term temporal dependencies via structured transition matrices.
14

---

It builds an adaptive directed graph using learnable node embedding and aggre- gates information along spatial dimensions through mix-hop propagation.
MTGNN:
MegaCRN: It exploits memory-enhanced node embedding to build the graph structure.
iTransformer: embeds the whole time series into spatial token and captures spatial correlations using the self-attention mechanism.
constructs both temporal and spatial dependencies using the attention mechanism and introduces a router mechanism to decrease the complexity of the spatial attention module.
Crossformer:
Card: It introduces summarized spatial tokens to decrease the complexity of the spatial attention module and employs token blend mechanism in the temporal attention module to extract local temporal correlations.
ESG: It designs multi-scale evolving node embedding based on Gated Recurrent Unit to capture dynamic spatial dependencies.
FourierGNN: builds dynamic graphs by merging spatial and temporal dimensions and designs the Fourier Graph Operator to decrease complexity.
TPGNN: It regards the dynamic spatial correlations as the matrix polynomial graph where the time-varying coefficients are determined by timestamps.
B.2 Metrics
The evaluation metrics including MAE, RMSE, and MAPE are as follows:
MAE
Eijen 1ui - Hisl [92]
RMSE
Eijen (Vv - )1y)2
MAPE
Σ ejen
1yai
(17)
[S2|
[22] |8kr]
where Q denotes the index set along temporal and spatial dimensions.
C More Forecasting Results
Table provides the results with prediction lengths of 12 and 24. The results underscore the significant role of explicitly capturing spatial correlations in improving forecasting accuracy. Furthermore, dynamic graphs exhibit greater ability to enhance the forecasting performance than modeling static spatial correlations. Among the baseline methods, Card, Crossformer, and ESG demonstrate superior forecasting accuracy. Notably, our model achieves state-of-the performance. yielding improvements more than 20% in terms of MAE and 25% in terms of MAPE over the existing methods on the Solar-Enere Energy dataset. addition, we conduct experiments with forecasting horizon of 96, and as Table 5 shows, Sumba gives rise to favorable performance in comparison to PatchTST, FEDformer, and iTransformer which are designed specifically for long-term forecasting.
D Hyperparameters Sensitivity
We evaluate the sensitivity of hyperparameters including the number of blocks the rank K, and the history window size H. Number of Blocks: As shown in Figure 4, Sumba achieves its best performance when L 3. Further increasing does not lead to performance improvements. We hypothesize that this is due to the over-smoothing issue associated with Graph Neural Networks. Rank: Figure 5 shows how performance varies as the rank K changes from 20 50. The optimal setting for K is 30 for the Electricity, PEMS, and Solar-Energy datasets, while K = 50 yields the best results for the Traffic dataset. We propose that this attributed to the larger number of nodes the Traffic dataset, which requires higher rank to effectively represent the dynamic spatial correlations. History window: Figure 6 demonstrates that increasing the history window size enhances model performance, especially for the Electricity dataset.
15

---

Table The forecasting results under forecasting length 12 and 24.
Method
Electricity
Weather RMSE MAPE(%)
PEMS
(F = 12)
MAR
RMSE
MAPE(%)
MAE
MAR
RMSF
MAPE(S5)
MICN [33]
0,278
2.633
0.123
2.970
0.277
0.404
2.021
ModernTCN [29]
0.202
0.307
1.796
0.111
0.268
4.541
0.169
0.252
1.368
PatchTST [30]
0.201
0.311
1.765
0.0958
0.270
6.108
0.211
0.293
1.426
FEDlormer [45]
0.274
0.390
2.535
0.251
0.394
6.789
0.262
0.369
1.931
Autaformer [35]
0,284
0.402
2.822
0.216
0.368
8.119
0.585
0.732
2.736
Reformer [19]
0.376
0.521
3.495
0.185
0.349
4.815
0.217
0.338
1.801
S4 [12]
0,361
0.512
3.314
0.213
0.347
5.443
0.246
0.346
2.184
MTGNN [38]
0.184
1.704
0.0899
4.549
0.177
0.263
1.391
MegaCRN [15]
0.366
0.520
2.715
0.115
0.279
3.757
0.188
0.286
1.572
TTransfommer [28]
0.188
0.296
1.703
0.0978
0.273
4.879
0.173
0.258
1.402
CrossFormer [42]
0.183
0.289
1.715
0.122
0,270
4.656
0.164
0.256
1.370
Card [34]
0,186
0.285
1.610
0.0892
0.268
3.829
0.168
0.254
1.347
ESG [39]
0.189
0.295
1.691
0.0969
0.264
3.513
0.164
0.248
1.313
FourierGNN [40]
0.275
0.396
2.694
0.121
0,276
4,932
0,199
0.305
1.832
TPGNN [26]
0,259
2.165
0.104
0.186
0.287
Sumba
0.178
0.283
1.560
0.0884
0.263
3.794
0.162
0.245
1.295
Method
ETTh2
Traffic
Solar- Solar-Energy
(F
12)
MAE
RMSE
MAPE(%)
RMSE
MAPE(S)
MAE
RMSE
MAPE(S)
MICN [33]
0.290
0.414
1.492
0.281
0.629
3,479
0.105
0.207
1.000
ModernTCN [29]
0.320
0.445
1.564
0.271
0.612
3.219
0.118
0.210
0.974
PatchTST [30]
0.237
0.374
1.135
0.250
0.582
2.559
0.122
0.243
1.054
FEDformer [45]
0.349
0.512
1,550
0.350
0.727
3.840
0.204
0.292
1.289
Autoformer [35]
0,360
0.526
1,481
0.361
0.735
4.007
0.309
0.414
1.680
Reformer [19]
0.579
0.728
4.289
0.354
0.789
3.608
0.113
0.229
1.125
$4 [12]
0.634
0.846
2.768
0.307
0.749
3.079
0.137
0.210
1.585
MTGNN [38]
0.245
0.381
1.172
0.247
0.577
2.762
0.0975
0.214
1.007
MegaCRN [15]
0.332
0.465
1.519
0.367
0.784
4.397
0.116
0.226
1 1.088
iTranfformer [28]
0.242
0.376
1.121
0.237
0.559
2.582
0.0974
0.211
0.991
CrossFormer [42]
0.307
0.446
1.467
0.238
0.666
2.497
0.0881
0.197
0.936
Card [34]
0.238
0.378
1.076
0.228
0,576
2,455
0.0850
0.210
0.996
ESG [39]
0.313
0.454
1.179
0.269
0.667
2.868
0.0935
0.204
0.962
FourierGNN [40]
0.309
0.454
1,551
0.382
0.792
4.159
0.119
0.208
1.098
TPGNN [26]
0.269
0.427
1.283
0.302
0.780
2.507
0.107
0.205
0.976
Sumba
0.234
1.045
0.569
2.358
0.0845
0.191
Method
Flectricity RMSE
Weather
PEMS
(F 24)
MAE
MAPE(%)
MAE
RMSE
MAPE(S)
MАE
RMSE
MAPE(S)
MICN
0.261
0.375
2.595
0.149
0.309
5.026
0.7
0.292
1.646
ModernTCN [29]
0.222
0.331
1.945
0.151
0.306
7.100
0.197
0.290
1.582
PatchTST [30]
0.211
0.329
1.859
0.128
0.309
10,480
0.246
0.345
1.611
FEDformer [45]
0.281
0.400
2.782
0.254
0.407
10.152
0.282
0.394
1.983
Autoformer [35]
0.278
0.398
2,932
0,234
0.389
9.728
0.455
0.601
2.277
Reformer [19]
0.402
0.554
3.829
0.248
0.422
8,590
0.263
0.400
2.435
[12]
0.348
0.499
3,258
0.269
8.225
0.259
0.361
2.403
MTGNN [38]
0.194
0.311
1.777
0.119
0.331
8.730
0.209
0.307
1.617
MegaCRN [15]
0.337
0.483
2.593
0.144
0.311
7.949
0.230
0.343
1.937
Transformer [28]
0.195
0.310
1.774
0.126
0.313
8.254
0.200
0.296
1.552
CrossFormer [42]
0.196
0.310
1.944
0.160
0.309
7.186
0.9
0.283
1.565
Card [34]
0.197
0.315
1.746
0.115
0.303
8.007
0.196
0.294
1.528
ESG [39]
0.196
0.314
1.729
0.127
0.294
5.227
0.182
0.296
1.507
FourierGNN [40]
0.316
0.396
2,409
0.133
0.259
11,606
0.245
0.367
1,518
TPGNN [26]
0.262
0.355
2.121
0.125
0.296
5.238
0.226
0.319
1.903
Sumba
0.189
0,303
1.696
0.115
0.295
8.593
0.186
0.281
1,488
Method
ETTh2
Traffic
Golar-Energy RMSE
(F
MAE
RMSE
MAPE(%)
MAE
RMSE
MAPE(%)
MAE
MAPE($))
MICN [33]
0.313
0.464
1.345
0.294
0.693
3.107
0.146
0.284
1.216
ModernTCN [29]
0.479
0.631
3.095
0.281
0.626
3.261
0.160
0.279
1.176
PST [30]
0.281
0.440
1.233
0.258
0.594
2.613
0.166
0.310
1.219
FEDformer [45]
0.368
0.542
1.651
0.353
0.740
4.142
0.229
0.345
1.533
Autoformer [35]
0.398
0.579
1.690
0.354
0.743
3.519
0.355
0.502
1.929
Reformer [19]
0.754
0.955
5.485
0.378
0.815
3,982
0.189
0.328
1,585
S4 [12]
0.6378
0.841
2.844
0.313
0.754
3.291
0.174
0.272
2.124
MTGNN [38]
0.288
0.452
1.267
0.266
3.117
0.286
1.224
MegaCRN [15]
0.363
0.516
1.738
0.415
0.833
4.990
0.151
0.296
1,508
TTrnsformer [28]
0.276
0.436
1.251
0.247
0,577
2.762
0.137
0.290
1.210
CrossFormer [42]
0.407
0.588
1.774
0.248
0.676
2.675
0.121
0.278
1.221
Card [34]
0.274
0.439
1.196
0.247
0.578
2,481
0.122
0.294
1.278
ESG [39]
0.308
0.457
1.271
0.275
0.689
2.715
0.124
0.271
1.166
FourierGNN [40]
0.326
0.515
1.374
0.427
0.794
4.189
0.162
0.307
1,571
TPGNN [26]
0.293
0.445
1.271
0.349
0.728
3.319
0.277
1.253
Sumba
0.273
0.433
1.217
0.243
0.589
2.669
0.121
0.264
1.119
16

---

Table 5: The results with forecasting horizon 96.
Electricity
Weather
PEMS
Method
MAE
RMSE
MAPE (%)
MAE
RMSE
MAPE (%)
MAE
RMSE
MAPE (%)
PatchTST [30]
0.241
0.378
2.383
0.206
0.404
13.692
0.354
0.497
1.906
FEDformer [4S]
0.308
0.438
3.223
0.346
0.525
8.949
0.482
0.631
3.182
Autoformer [35]
0.303
0.433
2.923
0.339
0.518
21.389
0.721
0.927
3.447
Reformer [19]
0.389
0.549
3.675
0.470
0.695
11.815
0.403
0.534
3.151
iTransformer [28]
0.229
0.367
2.248
0.213
0.411
13.009
0.311
0.446
2.055
Crossformer [42]
0.232
0.366
2.465
0.211
0.384
7.745
0.271
0.405
2.421
Card [34]
0.225
0.364
2.310
0.191
0,389
9.293
0,283
0,414
1.871
Sumba
0.234
0.373
2.675
0.185
0.383
11,561
0.253
0.377
1,700
0.200
Destriat
0.5
2,00
0:175
PEMS
1.75
ETT-2
0,150
1,50
0.125
0.3
125
1.00
0.075
0.75
0.1
0.50
Figure The sensitivity of our method to the number of blocks I.
0,200
0.5
2.00
Beettchy
FEMS
0.175
1.75
Solar
Tiaffe
0.150
0.4
0.3
125
0.100
1.00
0.075
0.2
0.75
0.060
20
30
40
50
20
Figure 5: The sensitivity of our method to rank K.
Becrioly
0.176
0.5
200
Weathe
HEAS
0.150
0.4
1.50
So
0.125
125
160
075
0.050
0.53
160
Figure 6: The sensitivity of our method the history window H.
17

---

Figure 7: The training curves on Electricity and Weather datasets with prediction length F
Table 6:
The complexity of different methods.
Method
Complexity
MTGNN [38]
O(N2D)
MegaCRN [15]
O(NDD)
iTransformer [28]
O(N=D)
CrossFormer [42]
O(NrD)
Card [34]
O(NrD)
ESG [39]
O(N3D)
FourierGNN [40]
ONN H(NHlog(HH))
TPGNN [26]
O(NDD)
w/n. Common U-V basis
O(MNKD)
Sumba
O(NKD)
E Convergence Speed and Computational Complexity
Convergence Speed Figure illustrates the training curves for the first 10 epochs on the Electricity and Weather datasets. Sumba, ModernTCN, and ESG exhibit fast convergence, achieving it within epochs. In contrast, methods such as Card, FourierGNN, and TPGNN take several tens epochs to reach convergence.
Computational Complexity As shown in Table we analyze the computational complexity of different methods in capturing spatial dependencies, where N is the number of nodes, M is the dimension the matrix basis, is the dimension node embedding or hidden representation, & N and K & N. MTGNN, MegaCRN, iTransformer, ESG, and TPGNN build the graph structure via inner product of static or dynamic node embedding. leading to a computation com- plexity of (NN D). Crossformer and Card employ router mechanism and summarized token, respectively, in spatial correlation modeling, reducing the complexity to O(Nrd). FourierGNN has complexity of ON H log( [oo[[N H)) due to building hypervariable graph. Our proposed Sumba leverages low-rank approximation and common coordinate transformations strategy, achieving the complexity of OKKND) whereas the computational cost for the model without common U-V basis OMKND). Thus, our model has low complexity, especially when N is large.
Limitations
Although our proposed Sumba demonstrates superior performance in multivariate time series forecasting, the dimension of matrix basis M is empirically tuned in our experiments. Besides, the long-term forecasting capability of the proposed model requires further enhancement.
G Broader Impact
In this paper, we propose a novel time series forecasting method to capture dynamic spatial corre- lations with structured matrix basis. Our research aims to contribute the advancement of the relevant community without any negative social impact.
18

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
19

---

Justification:
See Appendix A.2.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Section 3 and Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification:
Our code available at: https://anonynnn sapen.sciencc////Suublb/
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
wwwssssssicccyooioo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/CodeSubmie / //iips.co.. wttpcccccppccceeeeecec.eccccccct cubbiicgguiies /utia//uuiieeeeoeuuuuiiiiiiii cotommsssionn www.nnnolllo...eeeee..... oniooicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Section 4 and Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We repeat each experiment 3 times and report the mean results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
21

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that NeurIPS Code of Ethics https: //tpr//eerpp//ee/ee cc ctt.ccc/pp.ac ct/ppubiic//tiiicccd.........
didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer:
[Yes]
Justification: We make sure preserve anonymity.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix G.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
22

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited, and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer:
[Yes]
Justification: We cite the original paper that produced the code package or dataset. See Section 2 and Section 4.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
23

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "GdBBrKVLdM", "reviewer": "Reviewer_ftFS", "summary": "This paper presents a method to learn dynamic spatial structures in spatio-temporal forecasting tasks. Specifically, it proposes to parametrize the dynamic structures with a convex combination of fixed matrix bases, and the bases are further confined to be in the same coordinate system. Beyond that, it also imposes low-rank assumption on the coordinates to further reduce complexity. Empirical evidences show the proposed method achieve impressive accuracy in a number of spatio-temporal forecasting benchmarks, and the found spatial structures are highly interpretable.", "questions": "1. Why is TCN selected for temporal encoding?\n2. Is there any issue that the learned bases degenerate to be trivial or converge to be close?", "limitations": "Limitations are discussed, and I agree that the selection of $M$, i.e. the number of bases, is highly empirical and ad-hoc.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written with clear introduction of motivation behind major proposals and solid theoretical ground. \n- Extensive experiments are provided to showcase the advantages of the proposed method empirically that addresses both efficacy and efficiency concerns.", "weaknesses": "The forecasting horizon in the experiments is quite different from the setting in some baselines, such as PatchTST and FEDformer. While long-term forecasting is not major claim of the paper, I wonder if the method scales well with prediction length.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Q1 The forecasting horizon in the experiments is quite different from the setting in some baselines, such as PatchTST and FEDformer. While long-term forecasting is not major claim of the paper, I wonder if the method scales well with prediction length.\n\n\nThanks for your comments. Indeed, long-term forecasting is not the main focus of our proposed method, and the present version of Sumba is not aimed at addressing long-term predictions. But **we do conduct experiments with a forecasting horizon of 96 and as Table 1 in the supplied PDF file shows,  Sumba still gives rise to favorable performance** in comparison to PatchTST, FEDformer, and iTransformer which are designed specifically for long-term forecasting. We will leave it for our future work to further enhance its long-term forecasting capability and add this discussion to the limitation section. \n\n>Q2 Why is TCN selected for temporal encoding?\n\nSince the main focus of the paper is to effectively capture the spatial structures under the GCN framework, TCN is chosen for temporal encoding **due to its simplicity and can be easily integrated with the GCN operation**. In addition, **TCN architecture** (with appropriate modifications) [1] **also proved very effective in time series analysis** recently. We are also exploring the possibility of integrating our proposed spatial structure modeling with other temporal encoders such as Transformers, Structured State Space models, etc.\n\n[1] ModernTCN: A Modern Pure Convolution Structure for General Time Series Analysis, ICLR 2024.\n\n> Q3 Is there any issue that the learned bases degenerate to be trivial or converge to be close?\n\n**We do not empirically run into this issue** in our experiments (conducted on six public datasets from various domains). We hypothesize that **the degenerated issue may arise when the spatial structures are static** and can be represented by a single graph structure. In such a case, the model would only attend to one of the matrices in the basis (that is, all weights are concentrated in one entry while the others are zeros in the coefficient $\\alpha$). This will not have a negative impact on the model's performance.\n\n> Q4 Limitations are discussed, and I agree that the selection of  $M$, i.e. the number of bases, is highly empirical and ad-hoc.\n\nYes, it will be our future work to explore the adaptive selection of $M$ as well as enhancing the model’s long-term forecasting capability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HxcdtwPeVY", "reviewer": "Reviewer_aCCB", "summary": "This paper presents a multivariate forecasting model underlined by a dynamic spatial structure generation function, enabled by SVD-based parameterization and theoretically-bounded output space. Experiments on six commonly benchmark datasets in comparison to several existing baselines demonstrated the overall improved forecasting performance of the presented method as measured by MAE/RMSE/MAPE. Additional ablation studies further demonstrated the benefits of dynamic coefficient generation and the structured parameterization.", "questions": "Clarification on the observed margin of performance improvements and statistics/error bars to the results will be appreciated. \n\nBetter clarification and empirical results on relation with switching dynamics systems will be appreciated. \n\nWhat is the computational cost difference in the ablation with or without common U-V basis (i.e., vs. w/ Um,Vm)?", "limitations": "The authors did not include sufficient discussion about the limitation or potential negative societal impact of the presented work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The concept of dynamic generation of spatial structure function is of novelty and has practical value for adapting to the varying spatial structure underlying time series data.\n\nThe SVD-based parameterization and low-rank approximation provides theoretical-based solutions to the challenge of identifying time-varying spatial structure functions. \n\nThe experiments considered a large number of common benchmarks and representative existing forecasting models. The performance was overall favorable, and the ablation study thorough.\n\nThe interpretability results in section 4.4 and Fig 2 are interesting, especially in uncovering the dynamic patterns underlying a dataset.", "weaknesses": "The obtained performance gain (Table 1) was overall marginal (up to 2-3 decimal points). The practical implication of such margin of improvements is not clear and need to be clarified. The effect of hyperparameters and random initialization on such margin of improvements also needs to be examined — adding statistics to the results over different random seeds of experiments is important.\n\nIn switching dynamic systems, it is also common to model the transition matrix over time as a linear combination of several global matrices, with time-varying mixing coefficients (without considering the proposed SVD decomposition and parameterization). It’d strengthen the paper to add discussion about the relation with this line of works, as well as experimental comparison.\n\n\nBased on the results in Fig 3, it appears that increasing M from 1 to 2 in general introduced small difference in performance except in the Electricity dataset. This again raises some question on the significance of the dynamic spatial structure introduced in this paper. Stronger clarification on performance improvement and adding error bars to the results in Fig 3 will be important for verifying the contribution of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Q1 The obtained performance gain (Table 1) was overall marginal (up to 2-3 decimal points). Adding statistics to the results over different random seeds of experiments is important.\n> Clarification on the observed margin of performance improvements and statistics/error bars to the results will be appreciated.\n\n\nThank you for your comments and suggestions. We would like to point out that the performance gains of 2-3 decimal points are often considered to be **notable improvements in time series forecasting literature**, e.g., the prior works iTransformer [1] and Card [2] also achieve similar gains. To further demonstrate it more clearly, we provide the **improvements (percentage)** over the best baseline methods in the following table. \n| Dataset    | Electricity | Weather | PEMS| ETTh2 | Traffic | Solar |\n|:------:|:------:|:------:|:------:|:------:|:------:|:------:|\n| **Improvement**| 2.06%   | 3.97%   | 3.17%  | 1.06%   | 1.66%   | 2.15%  |\n\n In summary, our proposed method achieves an **average performance improvement of 2.35%** over the best baselines. This enhancement is significant in the field of time series forecasting and verifies the effectiveness of our proposed model.\nRegarding the effect of random initialization, as mentioned in line 263 of our manuscript, our reported results are actually averaged over three runs with different random seeds. We will further add the statistics (standard deviation) into the tables as well as error bars in our revised manuscript. In addition, **the error bars are also provided in Figure 1 in the supplied PDF file**.\n\n[1] iTransformer: Inverted Transformers are Effective for Time Series Forecasting, ICLR 2024.\n\n[2] Card: Channel Aligned Robust Blend Transformer for Time Series Forecasting, ICLR 2024.\n\n\n>Q2 In switching dynamic systems, it is also common to model the transition matrix over time as a linear combination of several global matrices, with time-varying mixing coefficients. Better clarification and empirical results on relation with switching dynamics systems will be appreciated.\n\nThank you for your suggestion. In switching dynamic systems, **the transition matrix is used to model the temporal dependencies**. The recent representative works include Hippo [1], LSSL [2], S4 [3], and Mamba [4]. In contrast, **the structured matrix basis is used to capture the spatial structures** in our proposed method. We have conducted the additional experiments to **compare with S4 in Table 2 in the supplied PDF file** and will also add the experimental results as well as the corresponding discussion in our revised manuscript.\n\n[1] Hippo: Recurrent Memory with Optimal Polynomial Projections, NeurIPS 2020.\n\n[2] Combining Recurrent, Convolutional, and Continuous-time Models with Linear State Space Layers, NeurIPS 2021.\n\n[3] Efficiently Modeling Long Sequences with Structured State Spaces, ICLR 2022.\n\n[4] Mamba: Linear-time Sequence Modeling with Selective State Spaces, arXiv 2023.\n\n\n>Q3 Based on the results in Fig 3, it appears that increasing M from 1 to 2 in general introduced small difference in performance except in the Electricity dataset.\n> Stronger clarification on performance improvement and adding error bars to the results in Fig 3 will be important for verifying the contribution of the paper.\n\nThank you for your suggestion. To further demonstrate the effectiveness of our proposed dynamic spatial structure, we present the **improvements (percentage)** in the table below as $M$ increases from 1 to 2. The results show that our method achieves an **average performance improvement of 5.66%** across six different datasets. This significant improvement underscores the effectiveness of our dynamic spatial structure design. We also include the **error bars in Figure 2 in the supplied PDF file** to show these more clearly. \n| Dataset    | Electricity | Weather | PEMS| ETTh2 | Traffic | Solar |\n|:------:|:------:|:------:|:------:|:------:|:------:|:------:|\n| **Improvement**| 10.77%   | 5.45% | 3.10%| 2.20%  | 2.80%  | 9.63%|\n\n\n\n\n>Q4 What is the computational cost difference in the ablation with or without common U-V basis (i.e., vs. w/ Um,Vm)?\n\nLet $M$ be the dimension of the matrix basis, $N$ the number of nodes, $K$ the rank, and $D$ the feature dimension. As shown in Equation 10 of our manuscript, if each matrix basis has specific coordinates $\\mathbf{U}_m$ and $\\mathbf{V}_m$, then $\\mathbf{V}_m^T \\mathbf{x}$ and $\\mathbf{U}_m \\mathbf{y}^\\prime$ will be calculated $M$ times. Consequently, the computational costs are $\\mathcal{O}(MNKD)$ and $\\mathcal{O}(NKD)$ for the models without and with a common $\\mathbf{U}$-$\\mathbf{V}$ basis, respectively. The computational cost difference will be included in our revised manuscript.\n\n\n\n>Q5 The authors did not include sufficient discussion about the limitation or potential negative societal impact of the presented work.\n\nThanks for your suggestion. We will add more discussion on the limitation including, the model performance on **long-term forecasting tasks and hyperparameter selection**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cjjw8rv7QC", "reviewer": "Reviewer_VUH9", "summary": "This paper proposes a novel approach for effectively capturing spatial and temporal correlations in multivariate time series forecasting and enhancing the interpretability of prediction models. The aim is to address the shortcomings of existing methods, which often fail to adequately reflect dynamic spatial correlations and exhibit high variability. To tackle this issue, the paper introduces a method that bypasses the traditional two-stage learning process and directly generates dynamic structures using a structured matrix basis. Furthermore, the basis matrices are parameterized through singular value decomposition (SVD), and all basis matrices share the same orthogonal matrices to improve the efficiency of model training.", "questions": "1. The proposed model may be complex to implement due to the use of structured matrix bases and singular value decomposition, potentially leading to a steep initial learning curve in practical applications. Does the appendix indicate similar results for datasets other than Electricity?\n2. While the model has been validated on various datasets, it remains to be seen how well it performs on very specialized domain data or data with unclear characteristics (i.e., weak temporal dependencies or spatial correlations). How does the model address these scenarios?", "limitations": "Including more analysis and comparison with graph-based spatio-temporal models would enhance the paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method of bypassing the two-stage learning process and directly generating dynamic spatial structures effectively overcomes the limitations of existing models.\n2. By providing interpretability of the model, it offers users greater insights and increases the reliability of the results. This is a crucial aspect that is often required in many time series forecasting research papers.\n3. This paper effectively addresses various limitations that arise in traditional time series forecasting problems.\n4. This paper is well-organized theoretically.", "weaknesses": "1. The proposed model may be complex to implement due to the use of structured matrix bases and singular value decomposition, potentially leading to a steep initial learning curve in practical applications. Does the appendix indicate similar results for datasets other than Electricity?\n2. While the model has been validated on various datasets, it remains to be seen how well it performs on very specialized domain data or data with unclear characteristics (i.e., weak temporal dependencies or spatial correlations). How does the model address these scenarios?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Q1 The proposed model may be complex to implement due to the use of structured matrix bases and singular value decomposition, potentially leading to a steep initial learning curve in practical applications. Does the appendix indicate similar results for datasets other than Electricity?\n\n\nWe would like to clarify that our method actually is very easy to implement. Note that the **singular value decomposition only serves to offer a parameterization form, we do not need to implement it in practice**. Instead, we only need to implement the parameterization of two orthogonal matrices $\\mathbf{U}$ and $\\mathbf{V}$ via the Clay map. As we mentioned in the paper, the Clay map can be implemented in a numerically stable way **in PyTorch with one line code by the  `torch.linalg.solve` function**. The learning curves of our proposed method are also very stable across all six datasets, and we will add more of them to the revised manuscript.\n\n\n\n>Q2 While the model has been validated on various datasets, it remains to be seen how well it performs on very specialized domain data or data with unclear characteristics (i.e., weak temporal dependencies or spatial correlations). How does the model address these scenarios?\n\n\nThanks for your comments. To our knowledge, the temporal dependency assumption is necessary for all time series forecasting methods. If a dataset has only **weak temporal dependencies**, it will impede the performance of all existing time series forecasting methods. For datasets with **weak spatial correlation**, it means that different channels (series) are independent and our proposed method will consequently learn an identity matrix for the spatial structure. This will not cause any negative impact on the forecasting performance. We will add the corresponding experiments to verify this by synthesizing the datasets with independent channels in the revised manuscript. \n\n\n>Q3 Including more analysis and comparison with graph-based spatio-temporal models would enhance the paper.\t\n\nThanks for your suggestion. In Table 1 of our manuscript, MTGNN, MegaCRN, iTransformer, Crossformer, Card, ESG, FourierGNN, and TPGNN are all GNN-based methods. First, we note that the graph-based spatio-temporal methods often produce better results than the methods without considering the spatial structures. Second, the dynamic graph-based methods such as iTransformer, Card, and ESG often give rise to better performance than the static graph-based methods such as MTGNN and MegaCRN. However, the existing graph-based spatio-temporal models all rely on a two-stage spatial structure learning process that is prone to yielding high variance and impeding the final performance, which is verified by the experiments. We will add more comparisons and analyses in the next version of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for their precious comments and valuable suggestions. Our major points of view are summarized as follows.

- We conducted **additional experiments with a forecasting horizon of 96**. 
- We provide explanations on **the choice of temporal encoding function and the potential degeneration issue**.
- We clarify the **performance improvements** and include the error bars to show them more clearly.
- We include **S4 as a new baseline method** and the corresponding discussion is also presented. 
- We provide the **computational cost** of the models with/without a common $\mathbf{U}$-$\mathbf{V}$ basis.
- We further clarify the **implementation details** and add a more **detailed analysis and comparison with graph-based spatiotemporal models**.

Meta-review: Thanks for your submission to NeurIPS! The reviewers have reach a
consensus, and it is clear that the reviewers are positive with its
quality. The recommended decision from the AC is accept. The authors may
take into account the feedbacks of the reviewers to further improve the
paper. There could be open or extension questions that may not be easy
to answer while possible comments (probably in the supplemental) could
help.

Bests

The AC
Decision: Accept (poster)