Interpretable Sparse System Identification: Beyond Recent Deep Learning Techniques on Time-Series Prediction
Authors: Xiaoyi Liu
Duxin Chen
Wenjia Wei
Xia Zhu
Wenwu Yu
Venue: ICLR 2024 poster (2024)
Keywords: time series
sparse system identification
long term prediction
Primary area: general machine learning (i.e., none of the above)
TL;DR: A sparse identification method without using neural network, achieving much higher accuracy with significantly lower computational cost in multi-variable long-term predictions compared with recent SOTA deep learning methods.

Abstract: With the continuous advancement of neural network methodologies, time series prediction has attracted substantial interest over the past decades. Nonetheless, the interpretability of neural networks is insufficient and the utilization of deep learning techniques for prediction necessitates significant computational expenditures, rendering its application arduous in numerous scenarios. In order to tackle this challenge, an interpretable sparse system identification method which does not require a time-consuming training through back-propagation is proposed in this study. This method integrates advantages from both knowledge-based and data-driven approaches, and constructs dictionary functions by leveraging Fourier basis and taking into account both the long-term trends and the short-term fluctuations behind data. By using the $l_1$ norm for sparse optimization, prediction results can be gained with an explicit sparse expression function and an extremely high accuracy. The performance evaluation of the proposed method is conducted on comprehensive benchmark datasets, including ETT, Exchange, and ILI. Results reveal that our proposed method attains a significant overall improvement of more than 20\% in accordance with the most recent state-of-the-art deep learning methodologies. Additionally, our method demonstrates the efficient training capability on only CPUs. Therefore, this study may shed some light onto the realm of time series reconstruction and prediction.

Published as a conference paper at ICLR 2024
INTERPRETABLE SPARSE SYSTEM IDENTIFICATION: BEYOND RECENT DEEP LEARNING TECHNIQUES ON TIME-SERIES PREDICTION
Xiaoyi Liu', Duxin Chen',, Wenjia Wei', Xia Zhu² & Wenwu Yu!
School of Mathematics, Southeast University, Nanjing 210096, China
2 Network Laboratory, 2012 Lab, Huawei Technologies Co. Ltd, Shenzhen 518129, China
x exiaoyiliiu chendx, wwyu)eseu.c eew.euuueeeeuuu........ edu. cn we weiwenjiaa zhuxial] wawawaal@@hawwe..comm
ABSTRACT
With the continuous advancement of neural network methodologies, time series prediction has attracted substantial interest over the past decades. Nonetheless, the interpretability of neural networks is insufficient and the utilization of deep learning techniques for prediction necessitates significant computational expendi- tures, rendering its application arduous in numerous scenarios. In order to tackle this challenge, an interpretable sparse system identification method which does not require time-consuming training through back- pack-propagation is proposed in this study. This method integrates advantages from both knowledge-based and data-driven approaches, and constructs dictionary functions by leveraging Fourier basis and taking into account both the long-term trends and the short-term fluc- tuations behind data. By using the I norm for sparse optimization, prediction results can he gained with an explicit sparse expression function and an extremely high accuracy. The performance evaluation of the proposed method is conducted on comprehensive benchmark datasets, including ETT, Exchange, and ILI. Re- sults reveal that our proposed method attains a significant overall improvement of more than 20% in accordance with the most recent state-of-the the-thh-rrt deep learning methodologies. Additionally, our method demonstrates the efficient training ca- pability on only CPUs. Therefore, this study may shed some light onto the realm of time series reconstruction and prediction.
INTRODUCTION
Long-term prediction with less input data plays crucial role in the contemporary of big data. Numerous domains, including electricity, exchange-rate, and disease (Wu et al. (2021); Zhou et al. (002); Zhang& & Yan (2022); Zeng al. (2023); Bi et al. (2023), ), necessitate accurate predictions over long time horizons. Recently, the advancement of deep learning has led to the prominence of neural network methods, particularly Transformer-based approaches (Vawwani et al. al.) (2017)) such as Crossformer (Zhang & Yan (2022)) and FEDformer (Zhou et al.] (2022)). Nevertheless, the heavy reliance on GPU resources deep learning has turned time series prediction into a competition of computational capabilities, rendering its challenge for ordinary laboratory setups or personal computers to cope with the demands (Wu et al. (2021). Is it truly inconceivable achieve long- term forecasting without the utilization of neural networks? This study endeavors to explore an innovative approach rooted in system identification, which ensures compatibility with CPU-based execution while maintaining high level of accuracy comparable to that of deep learning methods.
Traditional time series prediction methods like ARIMA (Shumway et al. (017 SVM Sapankevych & Sankar (2009)) and Temporal Regularized Matrix Factorization TRMF Yu et al.
*Duxin Chen is corresponding author. This work is supported by the National Key R&D Program of China under Grant No. 20222D012003,, the Zhishan Youth Scholar Program, the National Natural Science Foundation of China under Grant Nos. 62233004, 62273090, 62073076. the Huawei Technical Cooperation Project under Grant No. TC20220329025, and the Jiangsu Provincial Scientific Research Center of Applied Mathematies under Grant No. BK20233002.

---

Published as a conference paper at ICLR 2024
2016) are some commonly used classical statistical algorithms. However, these methods are typ- iallyy limited to make short-term predictions and encounter challenges when applied to long-temm prediction tasks due to the accumulation of errors over time. The introduction of deep learning methods can be traced back to the emergence of RNNs models, which leverage their hidden states to summarize and retain past information. Prominent examples of RNN-based architectures include LSTM (Hchreiterr & Schmidhuber (1997), GRU (Chung et al. (2014), and LSTNet (Lai et al. (2118) Additionally, there are deep learning approaches that build upon classical algorithms, such as DeepGLO (Sen et al.] (2019)) and Temporal Latent
(TLAE, Nguyen & Quanz (202T) which integrate deep learning techniques into the TRMF framework for enhanced predic- tion performance. Furthermore, there exist many research endeavors that build upon the concept of Temporal Convolutional Networks (TCN, Aksan & Hilliges ( Yan et al. (2020); [ea et al.) (2077). These approaches employ stacked arrangement of one-dimensional convolutional layers and residual connections to effectively capture both the local and global characteristics inherent in time series data. Despite demonstrating advancements over traditional methods in short-term pre- diction, these algorithms still face limitations when it comes to accurate long-term prediction tasks.
In recent years, the advent of the Transformer (Vaswani et al. (017)) has given rise to series of Transformer- -based models that tackle the challenges associated with long-term prediction, continu- ally pushing the boundaries of state-of-the-ar performance in this domain. These examples include Crossformer (Zhang& & Yan (2022)) FEDformer (Zhou et al. (2022), ), Autoformer (Wu al. (2021) ), Informer (Zhou al. (2021) ). Pyraformer (Liu et al. (2021)) p). LogTrans (Li al. (2019). and so on. Despite efforts to optimize algorithmic complexity, it remains evident that current long-term predic- tion methods necessitate significant computational resources. Moreover, as the prediction horizon increases, the training time becomes intolerable for small-scale laboratories. Accordingly, our ob- jective is to embrace lightweight approach wherein the computational requirements are reduced while upholding high prediction accuracy. Successful instances of this approach are exemplified by DLinear (Zeng et al. (2023)) and TiDE (Das et al. (2023). which deviate from the conventional Transformer framework and achieve remarkable performance solely by employing simple linear models, thereby mitigating training time. Building upon this foundation, we attempt to abandon deep learning frameworks and explore machine learning-based approach that enables long-term time series prediction on CPUs. Our goal is to develop a method that guarantees precision while remaining insensitive to the increase in prediction horizon, meaning that the training time of the model does not significantly increase as the prediction horizon grows.
Thus, we here propose the Global-Local Identification and Prediction (GLIP) model, which com- bines the framework of identification models (Brunton et al. (216); Yuan et al. (2019); Gao & Yan (2022) with insights from deep learning training methods. Specifically, The prediction model uti- lizes system identification methods and involves two main stages of identification and prediction. In the first stage of identification, three types of basis functions are constructed using Discrete Fourier Transform (Lu et al. (2021)) for prediction. These include global basis obtained from the training set, stored basis, and local basis from the test set. Note that prediction with explicit model instead of an entire black-box neural network approach, can also be referred to N-BEATS (Oreshkin et al. (2019)). In the second stage of identification, we further explore the relationships between variables through identification to improve prediction performance. In the entire procedure, it is apparent that GLIP abstains from employing any back propppropngt mechanisms, opting instead for exclusive reliance elementary machine learning techniques. Occam's Razor (Blumer et al. (1987) told us Entities should not be multiplied without necessity. For the same prediction task, our model is obviously more concise and efficient. The primary contributions can be summarized as follows:
1). A novel identification model for long-term prediction is introduced, which operates efficiently on CPUs instead of relying on GPUs Additionally, the training time of the GLIP model remains largely unchanged the prediction horizon increases. This significantly expands the applicability of the model, enabling long-term prediction even on personal computers. By offering a fresh direc- tion amidst the prevalence of Transformer-baser models in the realm of long-term prediction, this research opens new avenues for exploration.
2). The utilization of Fourier transform- bunsform-bbsed basis functions in global identification and local pre- diction allows for precise capture of both the knowledge of long-term trends and short-term fluctua- tions in time series data. This approach integrating both knowledge and data driven ways effectively addresses the limitation of neural networks in global perspective for prediction tasks.
2

---

Published as a conference paper at ICLR 2024
3). In the context of local basis function prediction, the incorporation of Fourier transforms part of the basis for each local segment data enables accurate extraction potential future trends even when the imput-output ratio is very small. This capability surpasses the limitations of traditional sparse identification methods.
4). Remarkable prediction performance was achieved on the four benchmark datasets, surpassing the current state-of-the-an results. The proposed model exhibited notable 23.64% improvement in Mean Squared Error (MSE) compared to neural network approaches. This breakthrough highlights the efficacy of non-neural network models in long-term prediction tasks.
2 NOTATIONS
When performing time-series prediction, the predicted multivariate time series can be represented as X. Here, X a two-dimensional vector, where the number of rows corresponds to the count of variables, and the number of columns represents the length of the time series. The i-th row of X is denoted as X[i, :]. while the i-th column of X is denoted as X(: :]. Furthermore, the subset of rows (columns) from the i-th j-th positions in X can be denoted as X(i j.:] (X[:, i j)) Matrices with the same number of rows or columns can he concatenated either horizontally or vertically. The horizontal concatenation is denoted as X, YJ. while the vertical concatenation is denoted [X; Y-- For instance, time series X comprising m variables can be represented as [21; #2; I.m. where H; denotes specific time series. The linspace function is used to create a vector with equally spaced elements. linspace (z1, Z2, n.) generates row vector containing n elements that are evenly spaced between I1 and I22 The symbol represents the Hadamard product, which denotes the element- wise multiplication of corresponding elements in vectors or matrices. A scalar can be added to or subtracted from a vector. When a scalar is added to or subtracted from a vector, the scalar will undergo the corresponding operation with each element of the vector. In addition, if we want to explore higher-order relationships among variables in a multivariate matrix X, such as a quadratic relationship, we denote it as X which signifies:
x[I.:|X X(, X(2, (33, .im. X|m. -17-
In addition, this paper also distinguishes between global prediction and local prediction, the signifi- cance of which can be further explored in Appendix,
METHODOLOGY
In this section, thorough exposition of the proposed Global-Local Identification and Predic- tion (GLIP) model will be presented. Analogous to deep learning methodologies, the time series will be partitioned into distinct sets, which are training set, validation set, and testing set, with a par- titioning ratio of ::1:2. In the stage of the entire prediction framework, we first construct global basis and storage basis in the training set for global prediction. Secondly. we evaluate the performance of global prediction in the validation set to determine its suitability for local rolling prediction. Fi- nally, in the test set, the amalgamation of all preceding information will be leveraged to execute local rolling prediction. The detailed procedures of global prediction and local rolling prediction are illustrated in Figure
3.1 GLOBAL IDENTIFICATION
The propose of global identification is to investigate the macroscopic variations exhibited in the given time series. We will employ Fourier transformation and sparse identification techniques on the training set. This approach facilitates the extraction and design of the global basis and storage basis, which serve as preparatory components for subsequent local rolling identification. Furthermore, the utilization the global basis enables preliminary estimation of global trend, thereby enabling long-term predictions surpassing 1000 data points.
3.1.1 GLOBAL BASIS AND GLOBAL PREDICTION
Global identification involves initially applying Fourier transformation to seek potential frequencies periods behind the training data. Subsequently, global basis is constructed by utilizing these
3

---

Published as a conference paper at ICLR 2024
inferred periods exhibited in the frequency domain to form a basis of trigonometric functions. Sparse identification is then performed using this global basis, followed by global prediction. Specifically, based on the Discrete Fourier Transformation (DFT), denoted as: A, DFT(X), where A, f are amplitude and frequency of DFT, respectively. For further global prediction, frequency with high amplitude are selected (see Appendix. C). By utilizing the frequency-periie formula f 1/T, these frequencies are converted into corresponding periods, unveiling the underlying potential periodicity T* [Tr, .... These potential periods are employed construct the global basis function
θ [sin Ctt), cos (Ct)), 1],
(1)
where C₁ = 2 2T and t is a column vector represent the time of train length. The following sparse identification process can be performed using the global basis.
arg min /. [( (1X- dt + A E
There exist several well- well-established algorithms for solving the aforemmentined optimization prob- lem (equation 2). such as OMP (Schnass (2018), LASSO (Ranstam & Cook (2018). SBM (Jacobs et al. (2018)), and others. In this context, we employ the 7 norm for sparsity regularization and utilize the typical coordinate descent method similar to LASSO for optimization. After obtaining 2g one can set È = Luriin + 1, Lerain + 2, ....(eurain is the length of training set) to predict the data in the validation and test sets, recording the predicted result as Xap- Furthermore, to capture the global inter-dependency among variables, we can concatenate the results obtained from the univariate pre- dictions with the training set, denoting as X, = [ : train], X9pl. Thus constructing a new basis function library Ø for further network relationship identification, i.e.,
θ = [gg. X x3 9 sin (g). cos (Xg.........
(3)
By substituting O into equation 2 and optimizing it, we can obtain the updated variable predictions gg with network coupling relationships. This represents the outcome of global identification and prediction. Appendix. B describes the introduction to system identification.
The schematic diagram of the process for establishing the global basis can be seen in the upper part of the "Global Prediction" section in Figure [ The global identification and prediction presents two notable advantages. Firstly, sparse identification acts as a robust mechanism to mitigate the risk of overfitting, facilitating the discermment of authentic underlying patterns or latent cycles. Notably, the direct utilization of the extracted high-frequency feature by DFT for reconstruction restricts the identification efficacy to the training set, with limited the generalizability to the validation and prediction sets. Secondly, this module, by exclusively employing for prediction during the initial identification phase, circumvents the common issue of error accumulation in long-term prediction encountered by traditional identification methods.
3.1.2 STORAGE BASIS
The storage basis serves as a preparatory foundation for local rolling prediction. Given that this model does not employ backpropagation for parameter updates. when conducting local rolling pre- diction on the test set, it is challenging to fully leverage the data feature from the training set, except for the solely obtained results of global identification and prediction. Considering our primary focus on prediction from an identification perspective, it is crucial to construct set of basis, known as the storage basis, that simultaneously relevant to both the training set and local rolling prediction.
If the length of a batch is denoted as Leatch (i.e., the length of input and output sequences), we continuously slide a window of length Leatch over the test set with an interval of Linerval During each sliding window, we perform the DFT on each variable of the small window's time series and record the high-frequency components in the frequency domain. We utilize a set to store these high- frequency components to ensure there is no duplication. Ultimately, this process yields a set of periods T.. Leveraging these periods, similar to the global basis, we can construct the following storage basis:
e- [ssn cos(C2t)))
(4)
where, C2 = 2TT 2 T The illustration of the storage basis is located below the "Global Prediction" section in Figure D The storage basis functions will have significant impact local rolling prediction, addressing the challenge faced by traditional methods in effectively using information from the training set during local rolling prediction.
4

---

Published as a conference paper at ICLR 2024
Covariao
Frediction
Predicted Na iakee
Sornact
Gloval Besi
[
Toun Ehra
X](,0: [
Radss
DFT
Predictio
XiL .1]
X11- :4
A.Ji
B,0T")
Cycle
Boois
Dorn loaker
X(10:4
Spii
Wittern Repertion
Cycle
Scrroge
Bans
Validation
Tce
Lacal Prodiction
XfE Asga
Weishiis
[6:22
Weiahtiss
Prodiction
Cycle
Locel
Figure 1: The GLIP model architecture
3.2 VALIDATION IDENTIFICATION
The objective of validation identification is to evaluate the efficacy of conducting local rolling pre- diction on the validation set, utilizzig two global identification mentioned in the previous sections as reference. This assessment serves preparatory step for subsequent local rolling prediction.
For the sake of simplicity and clarity, we denote X generic representation of batch, where X refers to X(; i Loathh] Meanwhile, we assume that the output derived from the local rolling prediction model denoted as Xpodd Its significance is similar to that of X, representing a generic representation of batch.
3.2.1 GLOBAL IDENTIFICATION VALIDATION
Validating global identification is to compare whether utilizing local features for prediction yields superior performance compared to global prediction. Assume that the predicted values of global prediction for specific batch are denoted as global- To evaluate the quality of global prediction, necessary to observe whether there is a significant improvement in prediction accuracy for each variable i when utilizing local features. i.e..
X global [i, Tamput :] (i, Tamgut :] < lr,
(5)
No
Sred X [ú, Linput :]
N.
where N₁ is the number of validation batches. is hyperparameter, and can be adjusted based on the desired level of improvement. In the subsequent local rolling prediction, we will determine whether it necessary to further incorporate local basis prediction by observing whether the vari- ables satisfy equation
3.2.2 VARIABLES RELATIONSHIP VALIDATION
The validation of variable relationships aims to verify whether, after conducting local rolling pre- dictions, the coupling relationships between different variables can further enhance the prediction effectiveness. To accomplish this, assuming that the output of coupling relationships after local hyperparameter, 2g. By substituting X
rolling prediction is X Meanwhile, for X giobbal replacing 2n with hy and swapping the numerator and denominator in the summation symbol in equation []] we can use the modified equation determine whether further coupling of variables is needed in the test set improve prediction performance.
introduce
5

---

Published as a conference paper at ICLR 2024
3.3 LOCAL IDENTIFICATION
This section will primarily focus on local identification and utilize the results obtained from global identification and validation identification to enhance the shortcomings of traditional methods in prediction. We aim to achieve local rolling prediction through local identification.
3.3.1 LOCAL IDENTIFICATION CURVE
In the context of local rolling prediction, both conventional approaches and prevalent neural network methods commonly employ the raw values of the input for prediction. Nonetheless, it is important to acknowledge that the input values may exhibit outliers or abrupt changes, thereby exerting a substantial influence on the prediction accuracy. Consequently, prior to the local prediction, this section necessitates the implementation of a straightforward preprocessing procedure for the input data, which will draw upon the insights gained from global identification and prediction.
We aim to construct a new input curve that is a weighted sum of both the global prediction and the real time series. This approach allows to capture both the global trend and local fluctuations in the data. Assuming that the input data denoted as I, and the corresponding results obtained from global identification are denoted as y. Now, construct
input x= follows:
(6) < B 1. Typically, a and / are determined based on the quality global prediction. If the global prediction is good. smaller values are assigned to a and 18, indicating a higher weight for global prediction and resulting in a prediction that leans more towards the global trend. Conversely, larger values are assigned a and 3, allowing more emphasis local information. ? reveals the pace of weight changes and is usually set to 1. By adjusting the parameters to appropriate values, we can construct the local identification curve. In ablation experiments, we have demonstrated that utilizing x² for prediction yields superior results compared to directly using x.
x w o x (1 w) y.
where = (linspace(a"/1, ////
Lbacch)) In general, 0 S a <
3.3.2 LOCAL PREDICTION
The local prediction consists of three steps. The first step is to determine if local identification is necessary. a variable demonstrates good performance in the validation identification, satisfying equation we do need to proceed with further prediction and can directly use the results of global prediction for local prediction. Otherwise, we proceed to the second step.
The second step involves selecting the local basis. If certain variables fail meet the criterion spec- ified by equation 5 it suggests that the performance of global identification is inadequate and needs further enhancements. Leveraging the locally derived identification curve obtained from equation 6 we adopt a methodology akin to global identification and conduct DFT. Subsequently, we identify the high-frequency components and transform them into potential periods denoted T
+ (Ctt), cos (Cst),,
where, - Tr The "Local Prediction" section in Figure illustrates the process of selecting the local basis. However, relying solely this basis is insufficient for local rolling prediction. This is because the input for local rolling prediction is limited in terms of available data, and the absence of global information can potentially lead to sigmificant prediction biases.
Therefore, the third step involves integrating the basis and performing local prediction. We combine the previously constructed basis with the local basis functions to obtain comprehensive represen- tation of global information. This incorporates three types of basis: the global basis obtained from global identification (equation the storage basis constructed from additional computations on the test set (equation 4. and the local basis (equation 7 derived from the local identification curve. By merging these three types of basis, we obtain the final basis functions for local identification and prediction, i.c.,
(8) By utilizing this integrating basis to replace E in equation 2 and performing sparse identification, we can obtain Epred- Letting Linput + 1, Linput + 2, .... enables us to make local rolling predictions and obtain Xped It is worth mentioning that Xrred is obtained in a similar manner during the validation identification phase.
pred = 1 +., O

---

Published as a conference paper at ICLR 2024
Models
IMP:
GLIP
DLincar
FEDiormer
Autolomer
Informer
Reformer
Pynaformer
LegnTrans
Metric
MSE
MSB
MAB
MSE
MAE
MSB
MAE
MSE
MAE
MSE
MAE
MSE
MAB
MSE
MAE
MSB
MAB
24.91%
0.332
0.289
0.353
0.358
0.397
3.755
1.525
0.693
0.645
0.597
2.116
192
34, 46%
0.251
0.364
0.383
0.418
0.429
0.439
0.456
0.452
5.602
1.911
0.588
0.741
0.788
0.685
4.315
1,635
336
38.39%
0.276
0.383
0.418
0.465
0.196
0.187
0.482
0.486
4.721
1,835
1011
0.787
0.907
0.747
L.121
16601
720
22. 18%
0.314
0.408
0.605
0.551
0.463
0.474
0.515
0.511
3.647
1.625
1.458
0.957
0.963
0.783
3.188
1.540
96
10.34%
0.148
0.260
0.167
0.360
0.203
0.287
0.255
0.339
0.365
0.453
0.658
0.619
01435
0.507
0.768
0.642
E17m
192
17.86%
0.184
0.302
0.224
0.503
0.269
081
0.340
0.533
0.563
1078
0.827
0.730
0.673
0.989
0.757
21.35%
0.281
0.342
0.339
0.372
1.363
0.887
1.549
1.201
0.845
720
34.00%
0.262
0.360
0.397
0:421
0.121
0.115
0.433
0.432
3.379
1,336
2,631
1.242
3,625
1.451
3.048
1.328
Exchange
96
11.11%
0.072
0.202
0.081
0.203
0,148
0.278
0.197
0.525
0.8447
0,752
1.065
0.829
0.576
1,105
0.968
0.812
192
20. 34939
0.125
0.275
0.157
0.295
0.271
0.380
0.300
0.309
1.204
0.895
1.188
0.506
1.748
1.151
1,040
0.851
336
31.48%
0.209
0.348
0.305
0.414
0.460
0.500
0.509
0.524
1.672
1,036
1.357
0.976
1.874
1.172
1.659
1.081
31.10%
0.643
0.601
1.447
0.941
2,478
1.310
1.943
1.206
1.941
24
18.65%
0.908
2,215
1.081
3,228
1,760
1.287
5.764
1.677
4.366
1,420
4,480
1.444
36
9.52%
1.766
0.943
1.963
0.963
2.679
1 0,800
3,103
1,148
4.755
1,467
4.446
1.389
7.394
2,031
4.799
1.467
48
18.73%
1.731
0.952
2,130
1.024
2,622
1,078
2.669
1.085
4.763
1,469
4.572
1.436
7,551
2,057
4.800
1.168
60
22. 808
1.82N
0.992
2.568
1.096
2,857
1.157
2.770
1.125
5.264
1.564
4.741
1.487
7.662
2,100
5.278
1.560
Table 1: Local rolling prediction errors terms of MSE and MAE The results from other models
sourced from (Zeng et al. (2023); Wu et al. (202 1 Kitaev al. (2020)). For ILI, the input length for local rolling prediction is 24, with output lengths 24, 36, 48, 60. For other datasets, the input length is 96, with output lengths 96, 192, 336, 720. The best results are indicated by bold numbers in the table, while the second-best results are marked with horizontal lines.
3.3.3 LOCAL VARIABLES RELATIONSHIP
Lastly, we consider the relationships among different variables in local prediction. During the val- idation identification phase, we assess which variables are suitable for representation using other variables as basis. We extract these variables and construct basis similar to equation 2 Similar to the global identification process, we perform a second identification using these basis. By utilizing the identification results, we can derive further prediction results for each variable. This predicted outcome, denoted as Xpedd represents the final prediction result for local rolling prediction. In Appendix. we provide pseudo-code for the entire algorithmic process.
4 EXPERIMENTS
We will show the results of global prediction and local rolling prediction. Furthermore, we will highlight the advantages of GLIP in terms of computational efficiency and long-term prediction compared to neural networks. The experimental settings can be found in Appendix.
4.1 PERFORMANCE COMPARISON
4.1.1 LOCAL ROLLING PREDICTION
As shown in Table. [ GLIP achieved excellent results in local rolling prediction across the four benchmark datasets. Specifically, it achieved a 32.49% MSE improvement on the ETTh dataset, a 21.14% MSE improvement on the ETTm dataset, 23.52% MSE improvement on the Exchange dataset, and 17.42% MSE improvement on the ILI dataset. Overall, compared to neural network methods, GLIP achieved a total improvement of 23.64%. We observed that the improvement tends to be greater when the O/I is larger. For instance, when the O/I is 7.5, the MSE improvement is consistently above 30%. This indicates that GLIP has the ability to predict long-term data based on short-term data in local rolling prediction, addressing the challenge of long-tern prediction that traditional methods cannot handle. Figure 2 (d)-(f) showcase the visualization results of the three datasets when performing local rolling prediction with the longest output.
It is noteworthy that in the context of local rolling prediction, GLIP attains remarkable performance by exclusively employing composite of Fourier basis, subsequently coupled with sparse identifi- cation. While sharing some conceptual resemblance with FEDformer-f (Zhou et al. (2022)) GLIP distinctively circumvents the reliance neural network architecture and instead directly conducts
7

---

Published as a conference paper at ICLR 2024
2001 WAS DH DASI TAKE 193
708 X50 BAS 736
Tea Sxg
Time Dae
2000
(6) ETTh Prodction
(h) Exchurge Glutel Profiction
(3) Global Predlcction
0.27
(4) RTT12 Local Prodiction
(1) Eschongo Local Prodiction
(S Local Predicton
Figure 2: The visualization results depict the prediction of specific variable in the ETT, Exchange, and ILI datasets. Figures (a), (b), and (c) represent global predictions, while Figures (d), (e), and (f) illustrate slice of local rolling predictions. The labels in the figure hold the following implica- tions: "Ground Truth" signifies the veritable time series data; "Global Prediction" denotes the visual representation of global prediction, aligning with the output outcomes from the global identification module; "Local Identification" represents the result of local identification after the weighted com- bination of global predictions and local input data; "Local Prediction" encompasses the outcomes of local rolling prediction; and the red vertical line demarcates the division between the training set and test set.
the prediction, thereby yielding superior outcomes. This observation underscores the viability of non-neural network methodologies for time series prediction in certain scenarios.
4.1.2 GLOBAL PREDICTION
Global prediction is distinctive advantage of GLIP, which sets apart from neural network-based methods that necessitate an ample number of training samples and are therefore incapable of ac- complishing global prediction. This implies that neural networks may not consistently capture the genuine trends of time series from a macroscopic perspective. In certain scenarios, GLIP can pro- vide reference for global prediction, as depicted in Figure (c). However, it is important to note that global predictions may not always be accurate and further local predictions may be required.
An example of accurate global prediction is the oil temperature variable (OT) in ETTh and ETTm, (a). Through identification, we have observed that global identification and prediction yield better results compared to local rolling prediction. Therefore, in local rolling prediction, we directly utilize the results of global prediction depicted in Figure (d).
as illustrated in Figure 2
4.1.3 ENVIRONMENT AND EFFICIENCY
Considering that GLIP only uses a few machine learning techniques for implementation, does not require GPU environment for experiment. Each of the aforementioned experiments can be completed in matter of seconds to minutes on a personal computer running on a CPU environment. This greatly reduces our computational requirements. According to the principle of Occam's ''s razor, GLIP undoubtedly stands as a superior prediction model.
Furthermore, considering the structure of GLIP, its efficiency in learning the local rolling prediction model is essentially independent of the prediction length. Once the model structure is determined, can directly perform predictions without being affected by the increase in prediction length. Specifi- cally, the time required for predictions with length of 720 is comparable to that of predictions with length of 96 (see Supplementary Material).
4.1.4 INTERPRETABILITY AND PARAMETER ADJUSTMENT
The interpretability of GLIP lies in the following three aspects. Firstly, GLIP employs DFT to explicitly identify potential periods within the time series for both global and local identification,
8

---

Published as a conference paper at ICLR 2024
incorporating them into the basis functions. This can be utilized interpret the main components extracted from the frequency domain of the time series. Secondly, in comparison with neural net- works, GLIP has explicit mathematical function expression for time series prediction tasks. The function expression provides insights into the predicted trends with an explicit white-box form. Thirdly, the hyperparameters, namely k1; kz, a, B, A, and parameters E have practical significance, e.g., = indicates the cycles and magnitudes between variables, which can be used directly to interpret the model. Comparatively, parameters such as the number of hidden layers and neurons in neural networks or the number of multi-heads in Transformer may not have straightforward interpretability. Furthermore, owing to the interpretability of the parameters and the expeditious computational effi ciency of the model, the process of parameter adjustment is markedly simpler in contrast to neural networks (see Appendix. D
4.2 ABLATION EXPERIMENT
While GLIP consists several components in- tegrated together for prediction, it is crucial
Loupput
Metric
GLIP
Case
Case
Case
to emphasize that each component has been meticulously constructed and holds irreplace- able significance. Removing or altering any of these components may potentially render the identification process ineffective. To demon- strate the significance of these components, excluded different components and performed ablation experiment. The validation results the Exchange dataset are presented in Table
MSE
0.072
1.912
0.087
0.091
MAE
0.202
1.151
0.212
0.220
MSH
0.125
1.186
0.2016
0.232
192
MAR
0.275
1.135
0.328
0.318
MSE
0.209
1.182
0.485
0.792
336
MAE
0.348
1.122
0.496
0.453
MSE
0.443
1.759
0.602
2.450
720
MAE
0.516
1.014
0.599
0.757
Table 2: Results for ablation experiments on the Exchange dataset. Case 1 omits sparse identifica- tion, Case 2 lacks local identification curves, and Case 3 excludes storage basis.
Effect of Sparse Identification: If high- frequency Fourier bases are directly used for prediction without undergoing sparse identifi- cation, it is possible that the resulting predic- tions on the training set will closely resemble the original dataset. However, this approach may lead to overfitting. We refer to the absence of the sparse identification component as Case 1. From Table 2 is evident that Case has no predictive capability.
Effect of Local Identification Curve: During the process of local rolling identification, if the input values are directly utilized for prediction, may overlook the global trend information. Additionally, it is highly probable to encounter overfitting issues when dealing with input points that exhibit an abrupt or anomalous behavior, consequently leading to prediction errors. The absence of local identification curves is denoted Case 2. From Table 2 it can be noted that Case 2 exhibits weaker predictive capabilities compared to GLIP.
Effect of Storage Basis: Storage bases crucial components in prediction tasks they maximize the utilization of information provided by the training set. Without storage bases, our training is confined local context, limiting the ability leverage historical information from global per- spective. Case 3 is designated to represent the scenario where trend storage bases are absent. From Table be seen that Case demonstrates close proximity GLIP in short-term predictions, but loses its predictive capability in long-term prediction.
5 CONCLUSION AND FUTURE WORK
In this study, we have provided a detailed account of the GLIP model to global prediction and local rolling prediction. By ingeniously constructing basis functions extracting knowledge from the frequency domain much possible and employing sparse identification techniques, accomplish long-term prediction tasks within CPU environment. This breakthrough overcomes the limitations of traditional methods with heavy computational burden and opens up direction for long-term prediction.
able to
In future work, plan to enhance both the scale of the data and the scope of the predictions. Additionally, we aim to integrate the identification methods with neural network approaches, striving to achieve superior prediction performance within shorter time frames.

---

Published as a conference paper at ICLR 2024
REFERENCES
Emre Aksan and Otmar Hilliges, Sten: Stochastic temporal convolutional networks. ArXiv Preprint arXiv:1902.06550 2019.
Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. Accurate medium- range global weather forecasting with 3d neural networks. Nature, pp. 1-6, 2023.
Anselm Blumer, Andrzej Ehrenfeucht, David Haussler, and Manfred K Warmuth. Occam's razor. Information Processing Letters, 24(6):377-380, 1987.
Steven L Brunton, Joshua L Proctor, and Nathan Kutz. Discovering governing equations from data by sparse identification nonlinear dynamical systems. Proceedings of the National Academy of Sciences, 11((1)):3332-395) 2016.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv Preprint
2014.
Abhimanyu Das, Weihao Kong, Andrew Leach, Rajat Sen, and Rose Yu. Long-term forecasting with tide: Time-series dense encoder. ArXiv Preprint ArXiv:2304. 08424, 2023.
Urban Fasel, Nathan Kutz, Bingni W Brunton, and Steven L Brunton. Ensemble-sindy: Robust sparse model discovery in the low-data, high-noise limit, with active learning and control. Pro- ceedings of the Royal Society A, :7702260))202299044 2022.
Ting-Tine Gao and Gang Yan. Autonomous inference of complex network dynamics from incom- plete and noisy data. Nature Computational Science, 260):160-168, 2022.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neurul Computation, 9(8): 1735-1780. 1997.
William R Jacobs, Tara Baldacchino, Tony Dodd, and Sean Anderson. Sparse bayesian nonlinear system identification using variational inference. IEEE Transactions on Automatic Control, 63 (12):4172-4187, 2018.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. ArXiv Preprint arXiv:2001.04455 2020.
Guokun Lai, Wei-Cheng Chang, Yiming Yang. and Hanxiao Liu. Modeling long and short-term temporal patterns with deep neural networks. In The s1 International ACM SIGIR Conference on Research Development in Information Retrieval, pp. 95-104, 2018.
Colin Lea, Michael Flynn, Rene Vidal, Austin Reiter, and Gregory D Hager. Temporal convolu- tional networks for action segmentation and detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 156-165, 2017.
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer time series forecasting, Advances in Neural Information Processing Systems, 32, 2019.
Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and fore- casting, In International Conference on Learning Representationss 2021.
Tianjian Lu, Yi-Fan Chen, Blake Hechtman, Tao Wang, and John Anderson. Large-scale discrete 2021.
fourier transform on tpus. IEEE Access,
Nam Nguyen and Brian Quanz. Temporal latent auto-encoder: A method for probabilistic multi- variate time series forecasting. Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 91 9117-9125, 2021.
Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. ArXiv Preprint ArXiv: 10437, 2019.
10

---

Published as a conference paper at ICLR 2024
Jonas Ranstam and JA Cook. Lasso regression. Journal of British Surgery, (16(10))1348-1348 2018.
Nicholas Sapankevych and Ravi Sankar. Time series prediction using support vector machines: a survey, IEEE Computational Intelligence Magazine, 4(2):24-38, 2009.
Karin Schnass. Average performance of orthogonal matching pursuit (omp) for sparse approxima- tion. IEEE Signal Processing Letters, 1225)::888888666 2018.
Rajat Sen, Hsiang- Fu Yu, and Inderjit S Dhillon. Think globally, act locally: A deep neural network approach to high-dimensional time series forecasting. Advances in Neural Information Processing Systems, 32, 2019.
Robert H Shumway, David Stoffer, Robert H Shumway, and David S Stoffer. Arima models. Time Series Analysis and Its Applications: With R Examples, pp. 75-163, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances Neural Informa- lion Processing Systems, 30, 2017.
Ines Wilms, Sumanta Basu, Jacob Bien, and David S Matteson. Sparse identification and estima- tion of large-scale vector autoregressive moving averages. Journal of the American Statistical Association, 118(541)5511-582 2023.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long, Autoformer: formers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 01-22419-22432 2021.
Decomposition trans-
Jining Yan, Lin Mu, Lizhe Wang, Rajiv Ranjan, and Albert Y Zomaya. Temporal convolutional networks for the advance prediction of enso. Scientific Reports, 10(1):8055, 2020.
Hsiang-Fu Yu, Nikhil Rao, and Inderjit S Dhillon. Temporal regularized matrix factorization for high-dimensional time series prediction. Advances in Neural Information Processing Systems, 29, 2016.
Ye Yuan, Xiuchuan Tang, Wei Zhou, Wei Pan, Xiuting Li, Hai-Tao Zhang, Han Ding, and Jorge Goncalves. Data driven discovery of cyber physical systems. Nature Communications, 10(1): 1-9, 2019.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 11121-11128, 2023.
Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The Eleventh International Conference on Learning Representations, 2022.
Haoyi Zhou, Shanghang Zhang. Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 1 11106-11115, 2021.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-tern long-term series forecasting. In International Conference PMLR, 2022.
Machine Learning, pp.
11

---

Published as a conference paper at ICLR 2024
A GLOBAL AND LOCAL PREDICTION OF TIME SERIES
Time series prediction can be broadly classified into global prediction and local rolling prediction. Global prediction involves predicting the entire time series by partitioning the temporal data into distinct sets, namely the training set, validation set, and testing set. Global Prediction (GP) model utilizes the training and validation sets to directly predict the remaining whole data points within the time series. However, GP models (especially neural network models) usually failed when confronted with lengthy time series due to limited availability of training samples. Conversely, local rolling pre- diction has emerged as prevalent methodology in contemporary deep learning practices. This way entails segmenting the training, validation, and testing sets into multiple batches, typically repre- senting localized segments within the time series, with each batch comprising input (training) and output (prediction) segments. The output-input ratio is denoted as O/I. In the context of local pre- diction, O/I ratio larger than is generally deemed indicative of long-term prediction. Presently, several models (Wu et al. (202T); Liu et al.] (2021); Zeng et al. (2023) employed for long-term pre- diction within local rolling settings have achieved notable O/I ratios of 7.5. However, despite such advancements, the achievable prediction length often remains limited to less than 1000 due to inher- ent constraints associated with local predictions. In contrast, global prediction scenarios frequently permit prediction lengths surpassing 1000 or even larger for extended time series. Currently, almost no methods exist to effectively address the challenges presented by these scenarios.
B INTRODUCTION TO SPARSE SYSTEM IDENTIFICATION
Sparse system identification (Wilms et al. 2023 Fasel et al. (002 : Brunton et al.) (2016) is a data-driven methodology employed for the inference of dynamic equations or algebraic equations governing a system based on observed data. The core idea behind sparse identification to first establish a basis function library, typically polynomial series, trigonometric sequence etc. By mini- mizing the sparsity penalty term within the equation, the objective is to identify equations from the basis function library that possess the fewest non-zero terms. This process facilitates the elimination of extraneous terms, thereby engendering a more succinct and comprehensible dynamic model. The fundamental formulation of sparse identification is represented
x Ot, X)E +N,
where, θ represents the basis function library, is the prediction time, X represents the 1, 2, 1-step data of all systems before the prediction data in the 1-order dynamic equation, and / N (1, In some instances, sparse identification successfully discerns the latent system dynamics concealed within the data, yielding outcomes of remarkable precision (Gao & Yan 2022); : Yuan
al. (2019)). Nonetheless, direct utilization of this method with no strategies designing the basis functions to capture the knowledge behind data may engender cumulative errors, thereby rendering it unsuitable for long-term prediction. Consequently, it becomes imperative to develop innovative identification models capable of addressing the challenges associated with long-term identification and prediction.
C INTRODUCTION TO DISCRETE FOURIER TRANSFORM (DFT)
Discrete Fourier Transformation (DFT) can be denoted as:
A. DFT(X).
The amplitude A and frequency f obtained through DFT are typically utilized for analyzing the frequency domain characteristies of time series. A and f are vectors of equal length, maintaining a one-to-one correspondence. Generally speaking. the larger the amplitude, the greater the impact of the corresponding frequency on the time series.
In this instance, Our objective is to select, based on the amplitude magnitude, the frequencies most likely to construct the main components of the time series. Subsequently, these selected frequencies transformed into basis functions for consideration, which will be further screened through sparse identification. Therefore, our candidate periodical terms T are determined by the selection process involving A and f.
12

---

Published as a conference paper at ICLR 2024
D EXPERIMENTAL SETTINGS
D.1 DATASETS
We selected four benchmark datasets, namely ETTh, ETTm, Exchange. and ILI ZZng et (2023) for validation and testing purposes. These datasets are wildely used and consist of multivariate time series data that cover various real-world domains such as temperature, exchange, and disease. One can refer to (Wu et al. (2021): Zeng et al. (2023)) for detailed descriptions of these data.
D.2 BASELINES
Considering the limitations of traditional methods in long-term predictions, our comparative analysis primarily focuses neural network-based methods. Specifically, concentrate on local rolling prediction and compare it with several prominent models, including DLinear (Zeng et al. 2023
FEDformer (Zhou et al. ),022), Autoformer (Wu et al. (202T)), Informer (Zhou et al. 2021
Reformer (Kitev et al. (2020)), Pyraformer (Liu et al. (2021)) and LogTrans (Li et al. (2019)) In
instances where multiple models are available for specific method (e.g., DLinear and FEDformer),
we select the model with higher accuracy for the comparative evaluation. We will provide a concise overview these models:
1) DLinear Zeng et al. 2023) is a simple neural network prediction model that combines time series decomposition and linear layers. a typical network model that operates independently of the Transformer framework. It is worth noting that recent models such as Crossformer (Zhang & Yan )022)) shares similar levels of accuracy with DLinear. To maintain brevity in the discussion, detailed experimental comparison will not be presented in this paper. The source code is available at htttts::///gg htaps:://qithbbe com/ cure- lab waa.aaaTsSinneeeeee
2) FEDformer (Zhou 2022) is Transformer- based model that addresses long-term forecast- ing tasks by utilizing Fourier and wavelet bases. FEDformer is characterized by time complexity of O(L), where L represents the sequence length, and its implementation code can found at https: http://hhwbbbom//////ggggggg////
3) Autoformer (Wu et al. 2021) is a Transformer-based model that incorporates autocorrelation
mechanisms to capture the inherent autocorrelation structure within a sequence. By introducing autocorrelation mechanisms, the model can effectively leverage the periodic information within the time series, thereby enhancing prediction accuracy. The source code available httpp::/
g thub. com/thuml Autof
4) Informer (Zhou et al. (2021) is a Transformer-basee model with ProbSparse self-attention. The time complexity of Informer OT log L.). The source code is available at ht.pt:::p//............ ///.tth com/ honl/hhohaoyy//oe
5) Pyraformer (Liu et al. (222)) is a Transformer-based model that introduces pyramid attention mechanisms for information propagation. This model has time complexity of O(L). The source code of Pyraformer can be found at ht /tap::///gtthhii com/ allaalpayy.. http//aaaaaayyaaaaaaaaaa
6) Reformer (Liu al. (2021) is a Transformer based model that incorporates Locality Sensitive Hashing (LSH) attention mechanism and RevNet to reduce computational complexity. This model has a time complexity of OL log L), where L represents the sequence length. The source code of Reformer can be found at httpt::://// /tt::///itti liggttttuubbo com/ http:///oogggeettra/aee////attee
modelss/eefi to
7) LogTrans (Li et (0119) is modification of the Transformer model that incorporates Log- sparse attention and Restart attention mechanisms to reduce computational complexity. By utiliz- ing these mechanisms, LogTram achieves time complexity of OL L)2), where represents the sequence length. The source code for LogTrans is avaliable at https: //gi thub. com/
ml mlpotter/Trans forternnsforrrrrrmmm Teie__eieeeee_e____eeeee__e Series
D.3 EVALUATION METRICS
To ensure consistency with the aforementioned approach, still employ the Mean Absolute Error (MAE) and Mean Square Error (MSE) evaluation metrics to determine the predictive accuracy of the time series. Furthermore, ensure complete alignment with the aforementioned approaches
13

---

Published as a conference paper at ICLR 2024
regarding the proportions of the training, validation, and test sets, well the input and output lengths. In Table we present the improvement in MSE. Here, we briefly provide the average MAE improvement for the four datasets: ETTh: 15.6%, ETTm: 4.51%, Exchange: 9.18%, ILI: 8.65%.
D.4 HYPERPARAMETER, ADJUSTMENT
GLIP provides default tuning methods and manual tuning methods regarding hyperparameter tum- ing. default tuning methods include three hyperparameters tuning modes. The first mode involves keeping the hyperparameters unchanged. Most hyperparameters can be directly determined be- fore model training and do not require tuning. For instance, the order of O is generally set to 2, and 11 2/2 0.8, 7 1. The second mode is the conditional tuning method. It adjusts pa- rameters based on the difference between the input data and global predictions, i.c., a, B. When the mean difference between global predictions and input exceeds half of the current input data's semi-range, indicating a significant global prediction error that may affect local predictions, we set = 0.8, 3 = 0.9. Otherwise, set 0.1, 3 0.9. The third mode is the adaptive tuning method. is in I regularization. Considering that being too large or too small will not yield good predictive results, we define feasible range for 7, such as [le 7, le 1]. We employ binary search on the validation set to find potentially optimal value. Moreover, GLIP provides manual tuning methods that you can also choose adjust the parameters according your needs.
E PSEUDO-CODE OF GLIP
Algorithm 1: Global Local Identification and Prediction (GLIP)
Input: Time-series X, Jutput length 1, Output length O, hyperparameter a, 3,
Output: Global predicction X Xp Local prediction Xpred
Global Basis and Global Prediction */
Do DFT on the entire training set and get A, f. then get T
3 Create global basis O and compute equation 2 then get global predict Xap-
Create O with X yp and compute equation again, and get X output.
1* Storage Basis #/
6 Create a set T
for Each batch in training set do
Conduct DFT on each batch and obtain A2, f2, then gain T
Create storage basis O, with Tü
11 * Validation Process */
Validate the effectiveness of global predictions and the relationships between variables the validation set and determine their suitability for further application in local predictions.
Local Basis and Local Prediction */ Global prediction is validated then
14
15
Use global prediction as local prediction and get Xpred-
else
17
for Each batch in the test set do Weight input data with equation
18
19
Conduct DFT on the weighted input data and obtain local basis O
20
Create basis O pred with equation & for local prediction and obtain X Xpred-
The relationships between variables are validated then Use the relationships between variables to form a basis for identification and once again, resulting in local prediction outputs X pred-
22
else
Output the local prediction results Xpred-
14


Reviews:
[{"review_id": "TPqLg8A1n9", "reviewer": "Reviewer_sU4J", "summary": "The paper presents  Global-Local Identification and Prediction (GLIP), a new approach for long-term time-series prediction based on sparse system identification. The proposed approach utilizes discrete Fourier transformation in global identification and local prediction and the experiments show significant improvement over the baselines, in particular in long-horizon prediction.", "questions": "I would appreciate the authors' response to the weaknesses listed above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "Strengths:\n- Novel approach for long-term time-series prediction\n- Very promising results showing significant improvement over the state of the art\n- The method performs particularly well for longer horizons, with the higher improvement compared to baselines on the longer horizons", "weaknesses": "I found the description of the method very hard to follow with many details either missing or unclear.\n- Section 3.1.1:\n\t* while the section describes a Discrete Fourier Transformation (DFT) on X, it is not clear where in this section the resulting A,f are used.\n\t* it is not clear what T/T* are or how are they computed? Is \\Theta_g includes exactly three items or multiple sin/cos waves? (this question also applied to Section 3.1.2)\n\t* \"from the univariate predictions with the test set\" - should probably be the training set?\n\t* X^*_g is not really used in Eq. (3), different from the text. Also, not clear what order of expressions were considered.\n- Section 3.2: it is not clear to me if this is part of the model/training or a description of how to validate the model's performance (e.g., the paper mentioned it is used to \"evaluate\" or \"conduct an assessment\")?\n- Overall, I think the paper would significantly benefit from a pseudo-code describing training and inference procedures.\n\nOne significant weakness is related to interpretability. The paper positions the proposed approach as an interpretable approach (including the title of the paper and in the motivation described in the abstract). However, the paper presents no results on interpretability in the paper and there is only a short description that says that the parameters are \"amenable to interpretation\". To make strong claims regarding the interpretability of the proposed approach, some results are needed (i.e., what are these parameters, are they weights associated with interpretable features, how many are there, etc). Some example(s) for the interpretability of results would be very useful as well.\n\nExperiments:\n- Datasets: the number of datasets used in the paper seems significantly smaller compared to comparable works. For example, in [2] and [3], other datasets such as weather, electricity, and traffic were included.\n- Global prediction is presented as an advantage of the proposed approach, but without any baseline, it is very hard to judge whether the results we observe are good (e.g., in Exchange and ILI where the predicted patterns deviate from the ground truth). It would be useful to include some time-series forecasting baseline so that we can get a sense of how well the proposed approach is doing by comparison.\n- Reproducibility: missing details as described above (e.g., the construction of \\theta_g). In addition, the chosen values for the hyper-parameters (e.g., \\alpha, \\beta) are not described (as well as the procedure of tuning them).\n\nLiterature: the work should probably mention other models that have focused on basis expansion, such as N-BEATS [1].\n\nMinor issues:\n- Section 3: \" in the testing set for global prediction\" -> should be training set?\n- page 6: \"prediction. we aim\" -> \"prediction. We aim\"\n\n\n[1] Oreshkin, B. N., Carpov, D., Chapados, N., & Bengio, Y. (2019, September). N-BEATS: Neural basis expansion analysis for interpretable time series forecasting. In International Conference on Learning Representations.\n[2] Wu, H., Xu, J., Wang, J., & Long, M. (2021). Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34, 22419-22430.\n[3] Zeng, A., Chen, M., Zhang, L., & Xu, Q. (2023, June). Are transformers effective for time series forecasting?. In Proceedings of the AAAI conference on artificial intelligence (Vol. 37, No. 9, pp. 11121-11128).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ley8zc3OYO", "reviewer": "Reviewer_jLT6", "summary": "The paper proposes a time series forecasting approach based on sparse identification of global, storage, and local basis functions.\nIn contrast to the recent transformer-based advances, this approach runs on CPU and is computationally highly competitive to other models. It also outperforms other Transformer-based methods by a large margin.\n\nOverall, it integrates knowledge about the challenges in time series forecasting (concretely, incorporating global and local aspects of time series datasets). This is further highlighted by additional design decisions like extracting local basis functions, through a prediction-data weighted input time series to mitigate outliers and data-variations. \nAn ablation study stresses the importance of each of the designed components and justifies each aspect of the complex model design (Fig 1).\n\nIn my opinion and after reading the method and appendix carefully, I see this as an excellent paper that demonstrates how to obtain state-of-the-art results in an orthogonal way to mainstream Transformer deep-learning approaches recently proposed.\n\nWhile the overall presentation is complete, well justified, and technically sound, the language seems over complicated at places and could/should be revised. For instance, the sentence \n> \"The parameters Ξ, procured through the sparse identification process, exhibit the capacity to signify the intensity of latent cycles and the magnitude of inter-variable influences. By virtue of these parameters, a more profound comprehension of the model can be attained. \"\n\ncould be rewritten in simpler (and more understandable) English by\n\n> \"The parameters Ξ indicate the cycles and magnitudes between variables, which can be used directly to interpret the model.\"\n\nSimilarly, pointing to individual panels in Figure 1 while describing the method would also greatly support the understanding of the method.\nThe current version describes the process in text and just references the figure as a whole.", "questions": "* what do the authors mean exactly with \"reasonable\" in their statement: code is available upon **reasonable** request? Will the source code be published on GitHub? If not, what is the motivation behind not providing the code open source?\n* eq 3: how many higher-order polynomials ($X^2$, $X^3$, etc) were integrated in the global features?\n* Can the authors please provide a detailed description of how the hyperparameters were obtained? The appendix/SI does not provide any information in this regard. Were they manually set or obtained by random search? What was the training/validation/evaluation split? How long did the hyperparameter tuning process take? How sensitive is the overall model to the choice of parameters? Were one set of hyperparameter set for every dataset or were some identical hyperparameters used across all benchmark datasets? This information is not in the current appendix!", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* strong state-of-the-art performance across different forecasting benchmarks\n* strong computational efficiency (without GPUs)\n* extensive ablation study to justify each of the (many) components in the approach", "weaknesses": "* complex language makes the text more arduous to understand than necessary\n* no details on hyperparameter tuning provided in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x2xzT86hLy", "reviewer": "Reviewer_fA8m", "summary": "The work proposes Global-Local Identification and Prediction (GLIP) for time series forecasting. The proposed method moves away from deep learning (often transformer-based) methods, instead utilising lightweight approaches that can be run on a CPU (system identification and Fourier transformation). This has the further advantage of making training time (mostly) independent of the prediction horizon. In an evaluation over four datasets, GLIP shows improved performance for both MSE and MAE compared to seven other methods.", "questions": "1. How are values for $\\alpha$ and $\\beta$ selected? The work explains the what high/low values mean and when they should be chosen, but how were they determined in practice?   \n2. Section 4.1.3 gives a vague measure of \"seconds to minutes\" for running the experiments. Are the authors able to provide concrete training/inference times for GLIP, and compare this to the same for the baselines?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The proposed method is a strong combination of global prediction (forecasting the entire time series) with local information (rolling batches). As shown in Figure 2, the method is able to weight global/local information when most appropriate. Results compared to baselines in Table 1 are strong across all output lengths and datasets. This is especially impressive given the speed of the method (\"seconds to minutes\").", "weaknesses": "**Clarity**  \nC1. Inline citations are not in brackets, which sometimes makes the text hard to follow.  \nC2. Acronyms are often not defined.  \nC3. There are some typos throughout the work.  \n\n**Quality**  \nQ1. Percentage improvement is only given for MSE. Additional analysis of percentage improvement for MAE would be helpful (potentially in the appendix).   \nQ2. I have a small concern regarding data leakage from the test set. I believe the test batch information is only used for local prediction (constructing the storage basis), but in Figure 1 there is an arrow going from the test batch back into the global prediction (predicted length). Could the authors please verify that there is no data leakage that would affect the global prediction/overall results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)