BasisFormer: Attention-based Time Series Forecasting with Learnable and Interpretable Basis
Authors: Zelin Ni
Hang Yu
Shizhan Liu
Jianguo Li
Weiyao Lin
Venue: NeurIPS 2023 poster (2023)
Keywords: time series forecasting
basis learning
self-supervised learning

Abstract: Bases have become an integral part of modern deep learning-based models for time series forecasting due to their ability to act as feature extractors or future references. To be effective, a basis must be tailored to the specific set of time series data and exhibit distinct correlation with each time series within the set. However, current state-of-the-art methods are limited in their ability to satisfy both of these requirements simultaneously. To address this challenge, we propose BasisFormer, an end-to-end time series forecasting architecture that leverages learnable and interpretable bases. This architecture comprises three components: First, we acquire bases through adaptive self-supervised learning, which treats the historical and future sections of the time series as two distinct views and employs contrastive learning. Next, we design a Coef module that calculates the similarity coefficients between the time series and bases in the historical view via bidirectional cross-attention. Finally, we present a Forecast module that selects and consolidates the bases in the future view based on the similarity coefficients, resulting in accurate future predictions. Through extensive experiments on six datasets, we demonstrate that BasisFormer outperforms previous state-of-the-art methods by 11.04% and 15.78% respectively for univariate and multivariate forecasting tasks. Code is
available at: https://github.com/nzl5116190/Basisformer.

BasisFormer Attention-based Time Series Forecasting with Learnable and Interpretable Basis
Zelin Ni
Hang Yu' Ant Group Hangzhou, China hyu. www.hngo@antgrrup.ccom
Shanghai Jiao Tong University
Shanghai, China mz151161900sjtur www.9900sttuuuuu..... edu. cn
Shizhan Liu
Jianguo Li! Ant Group Hangzhou, China
Weiyao Lin'
Shanghai Jiao Tong University
Shanghai Jiao Tong University Shanghai, China wylinesjtu. edu.edu.com cn
Shanghai, China
shanluzuode@sjtu. edu. cn
Abstract
Bases have become an integral part of modern deep learning-basee models for time series forecasting due to their ability to act feature extractors future references. To be effective, basis must be tailored to the specific set of time series data and exhibit distinct correlation with each time series within the set. However, current saate-o--th--ase art methods are limited in their ability to satisfy both of these requirements simultaneously. To address this challenge, an end-to-end time series forecasting architecture that leverages learnable and interpretable bases. This architecture comprises three components: First, we acquire bases through adaptive self-supervised learning, which treats the historical and future sections of the time series two distinct views and employs contrastive learning. Next, we design Coef module that calculates the similarity coefficients between the time series and bases in the historical view via bidirectional cross- attention. Finally, we present Forecast module that selects and consolidates the bases in the future view based the similarity coefficients, resulting in accurate future predictions. Through extensive experiments on six datasets, we demonstrate that BasisFormer outperforms previous state of-the-art methods by 1 11.04% and 15.78% respectively for univariate and multivariate forecasting tasks. Code is available at:
propose BasisFormer,
com wwt.:/1111111111 1tw.///111111////sssoorroe
Introduction
Bases, such trends and seasonalities, indispensable for time series modeling and forecasting, since they capture the underlying temporal patterns and serve as the key factors driving changes in the data over time. For example, seasonalities product or a service, while trends can reflect the long-term growth or decline a market or industry. Incorporating these bases into time series models can improve the understanding and prediction of future behaviors. Indeed, all commonly used deep learning models for time series forecasting can be reimagined as basis-driving models. N-BEATS M. N-HiTS 2 and FiLM all resort explicit bases, such as Fourier and Legendre basis. More generally, the linear (or convolution) layers in MLPs (or CNNs (5]) can be regarded implicit bases, as they act as filter banks decompose the time series. In addition, the covariate embedding (a.k.a. global timestamp embedding) in RNNs 6
capture the regular fluctuations in demand for
"Equal contribution. This work was done when Zelin Ni was a research intern at Ant Group. t Cooresponniig authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

and Transformers 17 is another form of basis, since they provide reference points for predicting
future sequences.
To apply bases for time series forecasting, three steps are required. First and foremost, an appropriate basis should be chosen or learned for the set of time series hand. In practice, the full space of the basis can often be very large, whereas the patterns of the time series in a set are often similar. It therefore desirable to learn basis that is tailored to the specific characteristics (e.g., periods or frequencies) of the time series data. This can help reduce the complexity of the forecasting model, as well as make it more accurate and interpretable. Second, each time series in the set is decomposed according to the basis. This involves computing coefficients or weights that determine the similarity or projection energy of the time series w.r.t. (with respect to) each vector (i.e., filter) in the basis. Note that these coefficients are supposed to vary across different time series, since each time series also exhibits unique patterns and characteristics. For instance, the Fourier coefficients corresponding to different time series will be dissimilar. Finally, the prediction is determined by the weighted aggregation of the future part of the basis.
Unfortunately, the aforementioned state-of-the-art methods fall short in satisfying the requirements in the first two steps simultaneously. On one hand, methods relying on classical bases, such as N- BEATS M. N-HiTS (2) and FiLM [3], often assume that the basis is not learnable and instead learn the coefficients of each time series in flexible manner when projecting to the basis. However, such a basis may not effectively account for temporal patterns since there is no guarantee that the given basis includes all periods or frequencies corresponding to the set of time series. On the other hand, approaches that aim to adaptively learn the basis from data, such as MIP A CNN [.) RNN (6) Transformer and their variants (7-3), often overlook the need for flexible associations between the basis and each individual time series. Specifically, although Transformer and its variants learn the covariate embeddings, they add or concatenate the same embeddings to different time series embedding in restricted manner. For MLP and CNN, they adopt the same linear and convolution layers for all time series.
To effectively tackle the aforementioned quandary, it is imperative to obtain basis that can accurately reflect the distinct traits of the dataset, and to devise a predictive network that can selectively utilize relevant vectors in the basis for forecasting purposes. To move forward to this goal, we propose Basierormer, a time series forecasting architecture with learnable and interpretable basis. As a first step, we acquire the basis through adaptive self-supervised learning from the data. This involves treating the historical and future sections of time series as two distinct views and employing contrastive learning to learn the basis, assuming that the selection of basis for a time series should be consistent across both views. Subsequently, we design the Coef module that measures the similarity between the time series and the basis in the historical view via bidirectional cross-attention, facilitating the flexible association between individual time series and the basis. Finally, we develop a Forecast module that consolidates vectors from the basis in the future view according to the similarity yielded by the Coef module, leading to accurate future predictions. We emphasize that the above three parts are trained in an end-to-end fashion. In summary, the key contributions of our work comprise:
We propose a self-supervised method for basis learning by treating the historical and future sections of the time series as two distinct views and employing contrastive learning, which ensures that the selection of basis for time series consistent across both views.
We design the Coef and Forecast module that chooses and merges relevant basis in the future view based on the coefficients that measure the similarity between the time series and the basis in the historical view.
We conduct extensive experiments on six datasets, and find that our model outperforms previous SOTA methods by 1 11.04% for univariate forecasting tasks and 15.78% for multivariate forecasting tasks.
2 Related works
Time series forecasting models recent years, deep learning methods have emerged as the pre- dominant technique for time series forecasting. As illustrated in the introduction, these deep learning methods typically resort to bases to facilitate the prediction of the future. Depending on the types of bases used in the network, forecasting models fall into two categories: those using classical orthogonal bases and those using learnable bases. The first group involves N-BEATS M. N-HITS (2) and FiLM 33. N-BEATS and N-HiTs typically utilize the Fourier basis and then learn the coefficients
2

---

for this basis in recursive network such that the basis helps decompose the historical part of the time series into different components and these components can be further aggregated to predict the future. FiI.M approximates the historical part via the Legendre polynomial basis and further removes the noise with the help of the Fourier basis. The major drawback with the group of methods that the basis is predefined, thus giving rise to the problem of which type of basis to choose (e.g., Fourier or Legendre polynomial) and further which vectors in the basis to choose (e.g., which frequency components we choose from the Fourier basis), On the other hand, the models based on learnable bases, such as Dlinear 4) TCN [5), Deepar ( Log Trans M.. Informer 77]. AutoFormer [8], FedFormer [9], etc, use a learnable linear convolution layer, covariate embeddings as the basis. Although these bases are adaptable to the time series, the relationship between the basis and the time series is fixed for all time series. For example, the covariate embedding is added or concatenated to the embedding of different time series in the same way, without considering the unique frequencies and periodic patterns of each series. In our paper, we propose method that allows for both a learnable basis and a flexible correlation between the basis and each time series for more accurate predictions.
Basis learning for time series analysis Apart from time series forecasting, learnable bases have also been explored for other time series-related tasks, such as time series classification. Note that basis learning is distinct from the representation learning of time series, the former aims to capture the pattern of a set of time series using a common basis, while the latter aims to extract features from individual time series. Moreover, the basis can help extract features from time series as in Dlinear 44] basis is exploited, such as Fourier and wavelet basis.
and TCN [5. Traditionally, a non-learnabe However, there exist handful of works that overcome this limitation and enable the use learnable basis. The learnable group transform [14] generalizes the filter banks in the wavelet transformer and allows nonlinear transformation from the mother wavelet to obtain flexible filter banks that can better extract features from time series. Along this direction, Balestriero et al. [15] propose a learnable Gaussian filter over the Wigner-Ville transform with few interpretable parameters, and prove that the resulting filter banks can interpolate between classical ones, including Fourier, wavelet, and chirplet basis. Similar works have also been proposed for audio signal processing 16 17). suggesting that a learnable basis is more effective feature extractor than a fixed basis. Thus, we utilize a learnable basis in our work and demonstrate its usefulness for time series forecasting. should be noted that DEPTS [8]] tackles the challenges posed by intricate dependencies and multiple periodicities in periodic time series through the implementation of a deep expansion learning framework. However. the complex initialization and optimization strategies employed by DEPTS, as well as its limitation of only being applicable to periodic sequences, have motivated us to develop a simpler and more universally applicable basis learning framework. Concretely, we propose a novel method for basis learning based on self-supervised contrastive learning.
Self-supervised time series representation learning Since we employ self-supervised representation learning techniques to learn the basis, it is necessary to examine related works in this domain. One prevalent method in this field is the contrastive predictive coding (CPC) [19] which implements contrastive learning to obtain time series representations by treating subsequent future time series as positive samples and random non-subsequent future time series as negative samples. Another method, TS-TCC (20,, augments the data with two types of perturbations to obtain two views and performs cross-view prediction task contrastively, similar to CPC. As an alternative, TS2VEC [21 generates positive samples via timestamp masking or random cropping without leveraging the future information. Note that all these methods seek to establish a common representation for time series from various views. Unlike these approaches, our goal is to preserve the consistency of the relationship between the time series and the basis. In other words, while the representation of the time series in the historical and future view may differ, their relationships with the corresponding basis should remain consistent.
3 BasisFormer
Suppose we have collection of time series with a dimension of C, implying that C correlated time series require simultaneous prediction. Each time series in this set is characterized by its history x = (x), .z7) and future y = (y),.... Wo), where and O correspond to the input and output sequence lengths, respectively. Our primary objective is to learn a basis that can account for the behavior of all time series in the group, and further exploit it for predicting y given æ. Correspondingly, can also be split into the historical component zn and the future component zy.
3

---

Basis Module
MILP
Module
Time Sentes
BCAB
AL
Spiit
BCAB
Multheea
Multihead Cross
Coof Module
Stere Asseecies
Module
Coef
Cross
BCAB
Histrrck Perricaal Miew
View
Veer
FFN
Bottleneck MLP
Split
MSE
InfoNCE loss
Figure I: The architecture of BasisFormer, consisting of ( the Coef module, ④ the Forcast module, and O the Basis modulc. The green and blue lines denote the data flow of the set of time scries and basis vector repespetively. The cyan diamond denotes tensor dot product. Note that the dot-dash linc, which denotes the data flow the future part of the time series, only included during training but removed during inference.
As mentioned in the introduction, three steps are necessary to employ bases for time series forecasting: learning appropriate basis, computing the coefficient of the time series with respect to cach basis vector, and predicting based on weighted aggregation of the future portion of the basis. The proposed BasisFormer facilitates these three steps in highly versatile manner. As demonstrated in Figure [1. the overall architecture of BasisFormer also contains three parts: ! the Coef module, which compares time series to each basis vector to determine the corresponding coefficient; ? the Forecast module, which predicts the future based on the coefficients and the future section of the basis; and ( the Basis module, which learns the basis by aligning the relationships between the basis and the time series from the historical and future perspectives. We will now elaborate on each of the Formerormer components.
3.1 Coef module for similarity comparison between time series and basis
The Coef Module is designed to measure the similarity between of time series and set of basis vectors. Since our focus is on the relations between the two sets, rather than within each set, we exploit a bipartite graph to represent such relations, where the nodes one set represent the time series (see green nodes in Figure MD and the nodes in the other set represent the basis vectors (see blue nodes Figure Tp. As a result, the strength of the edge connecting two nodes in the graph is equivalent to the similarity coefficient between them (see red edges in Figure I]. To obtain the edge strength, we require representations of the nodes the graph. We accomplish this by developing bidirectional cross-attention block (BCAB) to learn the node representations via cross-attention, in similar vein to the graph attention network
As first step, given two sets of inputs ( and 6((), let us define the cross attention block (CAB) as:
= LayerNorm (MA (t) ((0) (()) a (9))
a (f+1) = (a (0) TayerNorm(FFNu))
à),
where MA denotes multihead attention with H heads whose query is given by Q W WGa, key by K Wib. and value by V W.b. By exchanging information mutually between a and (6) we can construct the BCAB as:
(a++)) (6+1)) = (a (o))),
(3)
where
(6+1) (a b(0), ((+1) CAB (b()) (0).
Note that the parameters in CAB& for computing a (6+1) and (i++) may differ, to capture the heterogeneity in the relations from a to (1) and from (() to (

---

Correspondingly, given C' time series E R and the basis 25 E R NXI of size N, we can get their representations by stacking M layers of BCABH, namely, aMM RC and Zg (M) R x. where De represents the hidden dimension for each head in BC Note that the cross attention computed between the time series and the basis, instead of across time which is frequently found in Transformer based models 7 13 Additionally, the attention mechanism is used to allow for flexible associations between the time series and basis vectors. This approach ensures that each time series can selectively attend to the most relevant basis vectors, and likewise, each basis vector can selectively attend to the most relevant time series.
Finally, the Coef module calculates the "coefficient" of each time series w.r.t. each basis vector as the inner product of their representations z. () and Z (M) for each of the H heads, resulting in the coefficient tensor c E RCXNxH1
3.2 Forecast module for aggregation and future prediction
After obtaining the coefficients, we take advantage of them for the sake of forecasting. We begin by projecting the future part of the basis vectors Zy into a space where they can be linearly aggregated using the coefficients. As the Coef module computes the coefficients for H heads, the projection of z, should also have H heads to maintain consistency. To this end, we employ four- layer Multi- Layer Perceptron (MLP) with a bottleneck to map By E F xO to by E R R.NX xx, which is then split into H heads, each with size N x (O/H), and is denoted as ž, E E xgNxxx(O// ((OHH)
For each head, we aggregate the N basis vectors by computing the coefficients weighted sum of a over the dimension of N, that is,
N U hl = Σ cli.j. M :.
(5)
where h E {1,.... , denotes the head index, and the size of ÿ C' x H X (O/H).
Next, we concatenate the H heads together and pass them through another four-layer MLP with a bottleneck, in order to exchange information between different heads. This is because different heads may have captured different aspects of the input sequence and the fusion MLP can help combine the information and improve the overall prediction performance.
is noteworthy that the bottleneck layers in the above module are used to reduce the dimensionality of the input features before projecting them higher- sighe--dieensionll space. This helps to reduce the computational complexity of the projection operation and prevent overfitting. Additionally, using a bottleneck layer can also assist in extracting more informative features by forcing the model to learn a compressed representation of the input, thus improving the prediction accuracy.
Finally, we compare the predicted values ý with the true values y via a Mean Squared Error (MSE) loss function, that is, Epredd MSF( y).
3.3 Basis module for basis learning
In this subsection, we present our approach for learning a data-driven basis in a self-supervised manner. The goal is to obtain a basis that satisfies three essential properties.
First, the relation between the basis vectors and the time series should be consistent across time, such that we can predict the future by combining the future part of the basis using the coefficients obtained from the historical part of the basis and the time series. Specifically. given time series (x, y) and the corresponding basis (Zz, 2y), the coefficients (i.c., edge strength) between the time series and the basis should be consistent across the historical view (xx, z.) and the future view (y zy)) In other words, the relevance of given time series to a particular basis vector in the historical view should be retained in the future view. To achieve this, we pass both (æ, z2) and (y, zy) through the Coef module to get the coefficient tensor respectively for the two views, C, and cy. both with size C x N x H. For each time series, we then perform contrastive learning by regarding the coefficient w.r.t. each basis vector in C, as the anchor point, the coefficient w.r.t. the corresponding basis vector By as the positive sample, and the coefficient w.r.t. the remaining basis vectors in Cy as the negative sample. We optimize the InfoNCE loss to maximize the mutual information between e, and Cy.
5

---

which is given by
exp(cxli,3.1 cylū,
Tealign
CN Σ log
(6)
Σ exp(cali, Cyli, k,:)e)
where represents the temperature used to adjust the smoothness of alignment distribution.
In addition, require the basis to be interpretable, which means that gain insights into the underlying patterns captured by the basis. To achieve interpretability, we promote smoothness across time with regularization term, that is,
Lamooth |
where concatenate the basis vectors for the historical and future views (Zz, Fy) along the last dimension to form and the smoothness matrix S E -(=00)((++-2) can be expressed as:
2 1
S
(8)
1 2 1
apparent by multiplying with S, we compute || 12(=,4-1) 22|3, + 1]12: which the curvature over time (23). One advantage of using S is that the addition of constant and linear function time makes the loss invariant. Therefore, the above smoothness loss can accommodate
the change of the overall mean level as well as the linear trend.
Finally, the basis should funetion of the timestamp. skip connection between the input and the output of the second layer. The input to the network is the normalized timestamp associated with the first time point the historical window. Suppose that the overall length of time series in the dataset is T. then the normalized timestamp is defined as T = 1/T, where E {0, .... T - 1). The output of the networks is an N (7 11+0) O) tensor, which the basis for the current time window.
such, we develop four-layer MLP with
Overall, the loss we optimize can be expressed as:
I p Tralign
Tmmooth
We find that the performance of BasisFormer is robust to the weights in front of the terms in (9) Therefore, we set the weights to be one in all our experiments. Sensitivity analysis of the weights in the loss function can be found in the Appendix A.4
4 Experiments
To assess the effectiveness of model, conducted comprehensive experiments six datasets from real-world scenarios, using the experimental setup identical to that in [5] 7-91. Below we summarize the experimental setup, datasets, models, and compared models.
The length of the historical input sequence is maintained at 96 (or 36 for the illness dataset), whereas the length of the sequence to be predicted is selected from range of values, ie., {96, 192, 336, 720} ((24, 36, 48, 60) for the illness dataset). Note that the input length is fixed to be 96 for all methods for fair comparison.
Experimental setup:
The six datasets used this study comprise the following: 1) ETT 7 which consists of temperature data of electricity transformers; 2) Electricity, which includes power consumption data of several customers; 3) Exchange [24). containing financial exchange rate within a specific time range; 4) Traffic, comprising data related to road traffic; 5) Weather, which involves various weather indicators; and 6) Illness, consisting of recorded influenza-like illness data. Note that ETT is further divided into four sub-datasets: ETTh1, ETTh2, ETTml, and ETTm2, and the results in Table are based only on the ETTm2 sub-dataset. The results for the remaining three sub-datasets can be found in the Appendix.
Datasets:
Models for comparison: this study, we compare our proposed model against the following state-of-the-art models: four transformer-based models, namely FEDformer [9] Autoformer [8]. Pyraformer [3]: one MI.P-based model, i.e., Dlinear 44: and one CNN-based model, i.e., TCN
[5]. We also consider two recently proposed models, e.g., N-Hits 2 and FiLM B]. Due to space

---

Table 1: Multivariate results for six datasets were obtained using an input length of I 96 (or I = 36 for the illness dataset) and output lengths of (96, 192, 336, 720) (or (21,36, 48, 60) for the illness dataset). In all experiments, lower MSE values indicate better model performance, and we present the best results in boldface.
Models
Fedformer
Autoformer
N-HITS
Film
Dlincur
TCN
Basisformer
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAB
MSE
MAE
0.203
0.287
0.255
0.339
0.192
0.265
0.183
0.266
0.193
0.292
3.041
1.330
0.184
0.266
192
0.269
0.328
0
0.340
0.287
0.329
0.247
0.305
0.284
0.362
3.072
1.339
0.248
0.307
ETT
336
0.325
0.366
0.339
0.372
0.389
0.389
0.309
0.343
0.369
0.554
3.105
1.348
0.321
0,355
720
0,421
0.415
0.422
0.419
0.591
0.491
0.407
0.399
0.554
0.522
3.135
1.354
0.410
0,404
96
0.193
0.308
0.201
0.317
1.748
1.020
0.199
0.276
0.199
0.284
0.985
0.813
0.165
0.259
192
0.201
0.315
0.222
0.334
1.743
1.018
0.198
0.279
0.198
0.287
0.996
0.821
0.178
0.272
electricty
336
0.214
0.329
0.231
0.338
0.217
0.301
0,210
0.302
1,000
0.824
0.189
0.282
720
0.246
0.355
0.254
0.361
0.280
0.358
0.245
0.335
1.438
0.784
0.223
0.311
96
0.148
0.278
0.197
0.323
1.685
1.049
0.083
0.201
0.088
0.218
3.004
1.432
0.085
0,205
0.380
0.300
0.369
1.658
1.023
0.179
0.300
0.176
0.315
3.048
1.444
0.177
0.299
exchunge
336
0.460
0.500
0,509
0.524
1.566
0.988
0.337
0.416
0.313
0.427
3.113
1.459
0.336
0.421
0.841
1.447
0.941
1.809
1.055
0.642
0.610
0.839
0.695
3.150
1.458
0.854
96
0.587
0.366
0.613
0.388
2.138
1,026
0.652
0.395
0.650
0.396
1.438
0.784
0.444
0.315
192
0.604
0.373
0.616
0.382
0.605
0.371
0,605
0.378
1.463
0.794
0.460
0.316
traffic
336
0.621
0,383
0.622
0.387
0.615
0.372
0.612
0.382
1.479
0.799
0,471
0.317
720
0.626
0.382
0.660
0.408
0.692
0.428
0.645
0.394
1.499
0.804
0.486
0.318
96
0.217
0.296
0.266
0.336
0.648
0.492
0.193
0.234
0.196
0.255
0.615
0,589
0.173
0.214
192
0.276
0.336
0.307
0.367
0.616
0.479
0.238
0.270
0,237
0.296
0.629
0.600
0.223
0.257
weather
336
0.339
0,380
0,359
0.395
0,579
0.462
0.288
0,304
0.283
0.335
0.639
0.608
0.278
0.298
720
0.403
0.428
0.419
0.428
0,541
0.447
0.358
0.350
0,343
0,383
0.639
0.610
0.355
0.347
24
3.228
1.260
3,486
1.287
3,297
1.679
2.198
0,911
2.398
1.040
6.624
1.830
1,550
0.814
36
2.679
1,080
3. 103
1. 148
2.379
1 1.441
2.267
0.926
2.646
1.088
6.858
1,879
1.516
0.819
illness
2.622
1.078
2.669
1.085
3,34]
1,751
2.348
0.989
2.614
1.086
6.968
1.892
1.877
0.907
60
2,857
1,157
2,770
1.5
2.278
1.493
2,508
1.038
2,804
1,166
7.127
1.918
1.878
0.902
Experiment with 1.' means it reported an out-of-memory emor on computer with 80G memory.
constraints, we present the results of a selected number of models in this paper, according to their
performance and diversity. Interested readers can refer to the supplementary material for a more comprehensive comparison.
4.1 Main results
The results of the multivariate time series forecasting are presented in Table I. Very frequently, the proposed BasisFormer outperformed the comparison models across all six datasets, achieving the best results. Furthermore, Basisformer exhibited a 21.79% improvement over the ttate-of-the-art method Fedformer. We also observed that when compared to recently proposed models, such FiLM 3 and Dlinear [4, BasisFormer significantly improved the average MSE performance by a large margin of 10.78% and 14.78%, respectively. It is worth noting that Basis is vastly superior to the other methods for the traffic dataset, probably because Traffic dataset is a highly periodic dataset and our model can learn periodic representation well.
Multivariate results:
Univariate results: The results of the univariate time series forecasting are presented in Table 2 Our model is on par with the state-of-the-ard methods. Specifically, compared to the sota method such as FEDformer 19, our proposed model improved the average MSE performance by 15.36%. Compared to the recent models such as FiLM [3] and Dlinear [4]. we also achieved better performance with the increment of 1.6% and 16.17% separately
4.2 Ablation studies
Effect of learnable basis: In order to demonstrate the effectiveness of learnable bases, we replaced the learnable basis part in our model with three commonly used types of fixed bases: fixed sine/cosine basis that covers all possible frequencies within the input length, random sine/cosine basis selected in a wide range of frequencies, and covariate embedding which is generally used Transformer- based models. The results are shown in Table The substitution of the learnable bases resulted in, at minimum, a 5% average decrease in performance. It is pertinent to mention that although
Additional experimental details and results are presented in the Appendix section.
7

---

Table 2: Univariate results for six datasets were obtained using an input length of 96 (or I 36 for the illness dataset) and output lengths (96, 192, 336, 720] (or (21,36, 48, 60) for the illness dataset). In all experiments, lower MSE values indicate better model performance, and we present the best results in boldface.
Models
Fedformer
Autoformer
Informer
Dlinear
FILM
Basisformer
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.072
0.206
0.065
0,189
0.080
0.217
0.070
0.191
0.071
0.193
0.070
0.191
192
0.102
0.245
0.118
0.256
0.112
0.259
0.104
0.238
0.101
0.236
0.101
0.238
ETT
336
0.130
0.279
0.154
0.305
0.166
0.314
0.135
0.278
0.129
0.273
0.131
0.276
720
0.178
0.325
0.182
0.335
0.228
0.380
0.188
0.332
0.179
0.329
0.181
0.330
96
0.253
0.370
0.341
0.438
0.258
0.367
0.374
0.439
0.400
0.453
0,333
0.408
192
0.282
0.386
0.345
0.428
0.285
0.388
0.352
0.423
0.380
0.438
0.371
0.427
electricty
336
0.346
0.431
0.406
0.470
0.336
0.423
0.378
0.441
0.419
0.464
0.413
0.455
720
0.422
0.484
0.565
0.581
0.607
0.599
0.419
0.479
0.472
0.506
0.471
0.498
96
0.154
0.304
0.241
0.387
1.327
0.944
0.094
0.236
0.103
0.249
0.108
0.242
192
0.300
0.369
1,258
0.185
0.238
0.386
0.229
0.364
exchange
336
0.511
0.555
0.509
0.524
2.179
1.296
0.340
0.458
0.355
0.489
0.436
0.499
720
1.301
1.260
0.867
1,280
0.953
0.618
0.624
0.589
0.610
1.130
0.815
96
0.207
0.312
0.246
0.346
0.257
0.353
0.303
0.398
0.246
0.320
0.175
0.265
192
0.205
0.312
0.266
0.370
0.299
0.376
0.256
0.345
0.208
0.284
0.169
0.259
traffic
336
0.219
0.323
0.263
0.371
0.312
0.387
0.242
0.330
0.203
0.284
0.162
0.254
720
0.244
0.344
0.269
0.372
0.366
0.436
0.288
0.368
0.231
0.316
0.183
0.272
96
0.0062
0.062
0.0110
0.081
0.0040
0.044
0.0050
0.057
0.0013
0.026
0,0014
0.027
192
0.0060
0.062
0.0075
0.067
0,0020
0.040
0.0058
0.063
0.0015
0.029
0.0016
0.030
weather
336
0.0041
0.050
0.0063
0.062
0.0040
0.049
0.0063
0.067
0,0017
0.030
0.0016
0,030
720
0.0055
0.059
0.0085
0.070
0,0030
0.042
0.0064
0.066
0.0022
0.035
0.0022
0.035
24
0.708
0.627
0.948
0.732
5.282
2,050
0.815
0.736
0.674
0.643
0.798
0.620
36
0.584
0.717
0.634
0.650
4.554
1.916
0.914
0.817
0.758
0.731
0.563
0.605
illness
48
0.717
0.697
0.791
0.752
4.273
1.846
0.956
0.854
0.774
0.753
0.596
0.632
60
0.855
0.774
0.874
0.797
5.214
2.057
1.065
0.907
0.913
0.828
0.695
0.686
Table 3: Comparison of learnable and other basis on the Electricity dataset. The best results are marked in bold.
basis
learnable(ours)
fixed sine/cusine
random sine/cosine
covariate embedding
Metric
MSF
MAF
MSF
MAF
MSF
MAF
MSR
MAR
96
0.165
0.259
+6.8%
+5.4%
+7.2%
+5.5%
+5.2%
+4.5%
192
0.178
0.272
+4.7%
+3.2%
+6.6%
+4.3%
+3.8%
+3.5%
336
0.189
0,282
+9.2%
+6.1%
+10.1%
+6.5%
+5.2%
+4.6%
720
0.223
0.311
+16.4%
+10.4%
+12.4%
+7.5%
+7.1%
+5.6%
avg
0.189
0.281
+9.3%
+6.3%
+9.1% 1%
+6.0%
+5.3%
+4.6%
Table 4: The Impact of multi-head Operations Basis. In this experiment, k represents the number of heads,
with & E 81,8, 32). We used the Electricity dataset for this experiment. The best results are marked in bold.
heads
4
8
16
32
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
0.177
0.273
0.168
0.263
0.165
0.259
0.170
0.264
192
0.185
0.278
0.180
0.273
0.178
0.272
0.176
0.269
336
0.196
0.291
0.194
0.288
0.189
0.282
0.198
0.289
720
0.248
0.331
0.221
0.311
0.223
0.311
0.225
0.312
avg
0.201
0.293
0.191
0.284
0.189
0.281
0.199
0.283
sinusoidal types of bases have good generalization ability, they are inadequate in terms of adaptability for specific data. For covariate embeddings, despite having learnable parameters and containing additional sequence information, they do not account for the distinct correlation between the basis and different time series.
The Impact of multi-head mechanism: We further check the impact of the number of heads in the Coef module on performance. The results are displayed in Table The results show that performance presents an upward trend when the number of heads is increased within a specific range. However, beyond certain number, further augmentation of heads may result in decline in performance. Consequently, we set H = 16 in our experiments.
Ablation study of the Basis module: We finally employ ablation experiments to isolate the two different loss functions employed in the Basis module. The results are recorded in Table 5 It can be observed that the InfoNCE loss contributes significantly to the good performance of BasisFormer.
8

---

Table 5: The contribution of each loss term in the sell-supervised module performance. The dataset used in experiment was Electricity. The best results are marked bold.
superviston
none
infonce
smoolth
infoncetsmooth
Metric
MSE
MAB
MSF
MAE
MSE
MAR
MSE
MAE
96
0.177
0.273
-4.7%
-3.5%
-1.1%
-0.4%
-6.8%
-5.0%
192
0.188
0.282
-3.8%
-3.0%
-0.9%
-0.3%
-5.5%
-3.6%
336
0.205
0.298
-5.5%
-4.0%
-2.8%
-1.6%
-8.0%
-5.3%
720
0.243
0.327
-7.0%
-4.4%
-3.3%
-1.5%
-8.4%
-5.1%
avg
0.204
0.295
-5.2%
-3.7%
-2.0%
-0.9%
-7.2%
-4.8%
Table 6: The impact of the numher of bases N on the performance of the model. The electricity dataset is employed in this experiment. We present the best results in boldface.
basis number
15
20
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
0.173
0.269
0.171
0.265
0.166
0.259
0.168
0.263
0.168
0.263
0.183
0.277
0.178
0.270
0.176
0.270
0.176
0.269
0.176
0.268
0.196
0.289
0.192
0.284
0.190
0.283
0.192
0.285
0.193
0.285
0.231
0.317
0.229
0.314
0.218
0.306
0.220
0.308
0.224
0.311
avg
0.196
0.288
0.192
0.283
0.187
0.279
0.189
0.281
0.190
0.282
basb: 2
0.-
144
112
(b) basis vector 2 Figure 2: Two highly correlated basis vectors when the number of basis vectors N is large.
(a) basis vector
The smoothness loss also makes positive contributions. Remarkably, the employment of both loss functions in conjunction yields the most optimal result. Specifically, the InfoNCE loss advances the bases' capability to acquire the representation of time series, while the smoothness loss function mitigates overfitting to noise within the data.
Impact of the number of basis vectors: We present the performance of the proposed model under varying numbers of basis vectors N in Table where N is set to 1, 5, 10, 15, and 20. The results demonstrate that the model's performance remains stable over wide range of N, indicating its ability to adaptively adjust to the number of basis vectors. Notably, when N increases beyond a certain threshold, some of the basis vectors may become redundant. To further explore this, we visualize a subset of the learned basis vectors when N 20 in Figure 2 Interestingly, we observe a high cosine similarity of -0.93 between two of the bases. suggesting that some basis vectors may be necessary for accurate prediction. Thus, in practical applications, we set N to 10 for all datasets to reduce computational complexity without compromising performance.
4.3 Other studies
The adaptability of the self-supervised module to other models: To demonstrate the generalizabil- ity of our self-supervised basis learning module, we treat the covariate used in the FEDformer 9 and Autoformer g as learnable bases and supervise them using our self-supervised framework. is noteworthy that only provide supervision to the covariate employed in these models, without introducing any additional parameters the prediction pathway. The results are presented in Table 7 We can tell that the addition of self-supervision to the covariate exclusively causes performance improvement of approximately 5%. This phenomenon suggests that the learnable basis provides a more reliable reference for the future than the given covariates.
The interpretability of the bases: order to elucidate the impact of basis in a more comprehensible manner, we visualize the time series and the corresponding learned basis for the traffic dataset in Figure (())andd Figure3(b) Specifically, we choose time series and basis vectors at random. Upon observation of the figures, several key points can be discerned: Firstly, the traffic data have strong periodic patterns, with roughly four peak values in a sinusoidal shape within length of 96. The positions of the peaks not evenly distributed over time though. Correspondingly, the learned basis

---

Table 7: The performance comparison of the self-superyi supervveeee module used in Transformer- models. The 'origin' represents no modifications made to the original model and the "(+)coef modulc' represents applying our designed supervisdd network to supervise the covariate in the models. The dataset used in this experiment was Electricity. The best results are marked in bold.
buseline
Fedformer
Autoformer
comparison
origin
(+) coef module
origin
(+) onef module
metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.193
0.308
0.183
0.301
0.201
0.317
0.193
0.305
192
0.201
0.315
0.196
0.312
0.222
0.334
0.200
0.312
336
0.214
0.329
0.209
0.324
0.231
0.338
0.211
0.325
720
0.246
0.355
0.229
0.340
0.254
0.361
0.252
0.359
avg
0.214
0.327
0.204
0.319
0.227
0.338
0.214
0.325
sorics
serees
Scrios
-0.5
-1.0
56
1.44
192
152
1.64
392
48
96
144
192
(a) Visualization of time series on the Traffic dataset
Basis
basis
basis
-0.4
xy
1.0
555
0.5
-0.5
.65
(b) Visualization of learned basis on the Traffic dataset
Figure 3: The visualization of time series and learned basis on the Traffic dataset: The solid line indicates the historical series and the dashed line indicates the future series. For this visualization, we set the input length I to 96 and the output length O to 96.
follows this periodic pattern and effectively capture the salient features of the data. Secondly, the basis vectors given by our approach smooth, indicating that they not corrupted by the noise in the data. Note that the noise is not predictable, and so the basis that drives the change in the future is preferred to be smooth. Thirdly, it is evident that the learned bases have varying heights and intervals, thereby providing diversity to characterize different features of the time series. Finally, the obtained bases are consistent from both past and future perspectives, thus facilitating the prediction of future trends based on the coefficient similarity between a time series and the basis in the historical part. More visualized results can be found in the Appendix EE]
Conclusion
This paper presents BasisFormer, novel solution to alleviate two significant limitations that impede the efficacy of the existing SOTA methods. Through the utilization of BasisFormer, automatic learning of adjustijn basis can be achieved. Moreover, given the learned basis, BasisFormer also allows different time series to be correlated with distinct subsets of basis vectors. Our experimental findings provide compelling evidence of the superiority of BasisFormer over existing methods.
Acknowledgements
The paper is supported in part by the following grants: National Key Research and Development Program of China Grant (No.2018,) 00000000000000000 National Natural Science Foundation of China (No. 62325109, U21B2013, 61971277).
10

---

References
[1] Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint 190. 2019.
[2] Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza, Max Mergenthaler, and Artur Dubrawski. N-hits: Neural hierarchical interpolation for time series forecasting. arXiv preprint arXiv:2201. 2022.
[3] Tian Zhou, Ziqing Ma, Qingsong Wen, Liang Sun, Tao Yao, Wotao Yin, Rong Jin, et al. Film: Frequency improved legendre memory model for long-term time series forecasting. Advances in Neural Information Processing Systems, 35:12677-12690,, 2022.
[4] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? arXiv preprint arXiv:2205. 13504, 2022.
[5] Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv: 1803.0127, 2018.
[6] David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 1613)1181-111,, 2020.
[7] Haoyi Zhou, Shanghang Zhang. Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11106-11115
2021.
[8] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition trans- formers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34:22419 22430, 2021.
[9] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang. Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-tern series forecasting. arXiv preprint 12270 2022.
[10] Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Rethinking the stationarity in time series forecasting. arXiv preprint crXiv:2205.14415 2022.
[11] Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in neural information processing systems, 32, 2019.
[12] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXhv:2001.04451/ 2020.
[13] Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International Conference on Learning Representations, 2021.
[14] Romain Cosentino and Behnaam Aazhang. Learnable group transform for time-series. In International conference machine learning, pages 2 2164-2173, PMLR, 2020.
[15] Randall Balestriero, Hervé Glotin, and Richard Baranuik. Interpretable and learnable super- resolution Lime-frequence representation. In Mathematical and Scientific Machine Learning. pages 152-152. PMLR, 2022.
[16] Hendrik Purwins, Bo Li, Tuomas Virtanen, Jan Schlüter, Shuo- Yiin Chang, and Tara Sainath. Deep learning for audio signal processing. IEEE Journal of Selected Topics in Signal Processing, 13(2):206-22 -19, 2019.
[17] Stefan Lattner, Monika Dörfler, and Andreas Arzt. Learning complex basis functions for invariant representations of audio. arXiv preprint arXiv:1907.059922, 2019.

---

[18] Wei Fan, Shun Zheng, Xiaohan Yi. Wei Cao, Yanjie Fu, Jiang Bian, and Tie- Yan Liu. Depts: deep expansion learning for periodic time series forecasting. arXiv preprint vrXiv:2203.0768/ 2022.
[19] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1rxhvv:807.03740 2018
[20] Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting. arXiv preprini arXiv:2106. 10t::220..11112,.. 2021.
[21] Zhihan Yue, Yujing Wang. Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8980 8987, 2022.
[22] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv: 1710. 10903, 2017.
[23] Hang Yu and Justin Dauwels. Modeling spatio-tempore extreme events using graphical models. IEEE Transactions on Signal Processing, 64(5):1101-1116, 2015.
[24] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The sl international ACM SIGIR conference on research development in information retrieval, pages 95-104, 2018.
[25] Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. arXiv preprint arXiv:2202. 01500.0.1555.... 575, 2022.
[26] Patrick Schäfer. The boss concerned with time series classification in the presence of noise. Data Mining and Knowledge Discovery, 29: 1505--1-5,,, 2015.
[27] Juntang Zhuang. Tommy Tang, Yifan Ding, Sekhar C Tatikonda, Nicha Dvornek, Xenophon Papademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. Advances in neural information processing systems, 33:18795-2 18806, 2020.
12

---

Appendix
A Additional Experiments"
A.1
Experiments on the ETT datasets
In the main body, we present a comparison of the benchmark methods on the ETTm2 dataset. In this section, we extend our analysis to the remaining three ETT datasets, namely ETTh1, ETTh2, and ETTml, as summarized in Table g Our experimental results reveal that Basisformer outperforms all other methods in terms of MSE and MAE. Specifically, Basisformer demonstrates a superior average MSE reduction of 1.32%. 6.74% and 9.23% when compared to FiLM, Fedformer and DLinear, respectively.
Table 8: Multivariate results for the remaining three ETT datasets using input length of / = 96 (or / 36 for the illness dataset) and output lengths of O E (96, 192, 336, 720) (or O E (24, 36, 48, 60) for the illness dataset). In experiments, lower MSE values indicate better model performance, and we present the best results in boldface.
Models
Fediormer
Autoformer
N-HITS
HiLM
Dlinear
Informer
Basiserreer
Metric
MSF
MAF
MSF
MAF
MSR
MAE
MSF
MAF
MSF
MAF
MSF
MAF
MSR
MAF
96
0.376
0.419
0.449
0.459
0.419
0.413
0.388
0.401
0.386
0.400
0.865
0.713
0.394
0.411
192
0.420
0.448
0.500
0.482
0.468
0.443
0.443
0.439
0.437
0.432
1,008
0.792
0.442
0.437
ETThi
336
0.459
0.465
0.521
0.496
0.551
0.489
0.484
0.461
0.481
0.459
1.107
0.809
0.473
0.451
720
0,506
0,507
0,514
0,512
0.669
0.559
0.525
0.519
0,519
0,516
1,181
0.865
0.460
0.465
96
0.358
0.397
0.346
0.388
0.374
0,383
0,292
0.341
0.333
0.387
3.755
1.525
0.312
0.356
192
0,429
0.439
0.456
0.452
0.476
0,446
0.378
0.396
0.477
0.476
5.602
1,931
0.382
0,401
ETTh2
336
0,496
0,487
0.482
0.486
0.472
0,446
0.426
0.438
0.594
0.541
4,721
1.835
0.418
0.431
720
0.463
0.474
0.515
0.511
0.932
0.636
0.443
0.455
0.831
0.657
3.647
1.625
0,418
0.438
96
0,379
0.419
0,505
0.475
0.324
0.349
0.357
0.373
0.345
0.372
0.672
0.571
0.342
0.374
192
0.426
0,44]
0.553
0.496
0.376
0.379
0.387
0.385
0.380
0.389
0.795
0.669
0.380
0.392
ETTml
336
0.445
0.459
0.621
0.537
0.409
0,405
0.420
0.407
0.413
0.413
1.212
0.871
0.420
0.418
720
0.543
0.490
0.671
0.561
0.472
0.443
0.478
0.439
0.474
0.453
1.166
0.823
0.492
0.458
A.2
Experimental results with longer length input setting
Throughout our research, we maintain consistency in our experimental settings by fixing the input length to be 96 (with a reduced input length of 36 for the illness dataset), instead of using longer length. The main rationale behind this decision is that, in practical scenarios where the model is deployed as an online service and tasked with predicting long range of the future at granular level of minutes or hours, collecting lengthy history (i.e.. spanning 720 timestamps) for a large number of time series in real-time can be quite challenging. Therefore, the adoption of an input length of 96 proves to be more practical and feasible.
Given that certain recent methods utilize longer input lengths yield better performance, irrespective of the length, we present supplementary comparison outcomes with extended input lengths in Table 9 Specifically, Fedformer, Autoformer, and TCN exhibit decline in performance with increase in input length, and hence, we retain their original outcomes at an input length of 96. In contrast, Dlinear employs an input length of 336 (104 for the illness dataset) by default, FiLM utilizes an input length that is at most four times of the output length, and N-HiTS adopts an input length that is five times of the output length. To enable a fair comparison, we standardize our input length for longer inputs to 192 (72 for the illness dataset).
The experimental results yield several notable findings. Firstly, those methods that benefit from longer inputs, namely Dlinear, FiLM, and N-HiTS, exhibit a significant performance decline when the input length is reduced from longer settings an input length of 96. Concretely, Dlinear, FiLM, and N-HiTS show performance declines of 25.82% 19.48%, and 330.42%, respectively. Conversely, our approach maintains most of its performance with slight deterioration of 6.23%, evident in Table and Table 9 Secondly, concerning longer inputs, our method surpasses recent approaches such as Dlinear, FiLM, and N-HiTS, with an average MSE performance improvement of 1.35%. 0.63%, and 7.75%, respectively, and corresponding evaluation MAE performance improvement of 3.15% 2.33%, and 4. 4.06%, reseeeeeesseeeeeee respectively. It is noteworthy that our approach requires an input length
'All the six datasets can be downloaded from

---

Table 9: Multivariate results for six datasets using longer input length. Lower MSE indicate superior model performance, and the best results are presented boldface.
Models
Fediormer
Autoformer
N-IITS
FIMM
Dlineur
TCN
Basisformer
Metric
MSE
MAE
MSB
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAB
MSE
MAE
0.203
0.255
0.339
0.176
0.255
0.256
0.167
0.185
0.270
192
0.269
0.328
0.281
0.340
0.245
0.305
0.222
0.296
0.224
0.303
3.072
1.339
0.247
0.307
ETT
0.339
0.372
0.295
0.346
0.281
3.105
0.341
720
0.421
0.415
0.422
0.419
0.401
0.416
0.371
0.389
0.399
0.421
3.135
1.354
0.381
0.393
0.193
0.308
0.201
0.317
0.147
0.249
0.154
0.267
0.140
0.237
0.985
0.813
0,145
0.245
192
0.201
0.315
0.222
0.334
0.167
0.269
0.164
0,255
0.153
0.249
0.996
0.821
0.165
0.263
electricty
0.329
0.231
0
0.186
0.290
0
0.169
1,000
0,178
0.276
720
0.246
0.355
0.254
D.361
0.243
0.340
0.236
0.332
0.203
0.301
1.438
0.784
0,219
0.310
96
0.148
0.278
0.197
0.323
0.092
0.211
0.079
0.204
0.081
0.203
3.004
1.432
0.084
0.205
192
0.271
0.380
0,300
D. 0.369
0.208
0.322
0.159
0.292
0.157
0.293
3,048
1,444
0.172
0.298
exchange
0.460
0.500
0.509
0.524
0.422
0.270
0.398
0.305
0.414
3.113
1.459
0.303
0.403
720
1,195
0.841
1.447
0.941
0,888
0,723
0.536
0.574
0.643
0.601
3,150
1.458
0,781
0.668
96
0.587
0.366
0.613
0.388
0.402
0.282
0.416
0.294
0.410
0.282
1.438
0.784
0.403
0.293
192
0.604
0.373
0.616
0.382
0,420
0.297
0,408
0,288
0.423
0.287
1.463
0.794
0.421
0.301
traffic
336
0.621
0.383
0.622
0,448
0.313
0.425
0.298
0.436
0.296
1.479
0.799
0.418
0.298
720
0.626
0,382
0.660
0.408
0,5339
0,353
0.520
0,353
0.466
0.315
1.499
0.804
0.464
0.312
96
0.217
0.296
0.266
D.336
0.158
0.195
0.199
0,262
0.176
0.237
0.615
0.589
0.168
0.215
192
0.276
0.336
0.307
0.367
0.211
0.247
0.228
0.288
0.220
0.282
0.629
0.600
0,213
0.257
weather
336
0.339
0.380
0.359
0.395
0.274
0.300
0.267
0,323
0.265
0.319
0.639
0.608
0.263
0.292
720
0.403
0.428
0.419
0.428
0.351
0,353
0.319
0,36
0.323
0.362
0.639
0.610
0,343
0.346
24
3.228
1,260
3,486
1.287
1.862
0.869
1.970
0.875
2,215
1.081
6.624
1,830
1.427
0.778
36
2.679
1.080
3.103
1.148
2.071
0.969
1.982
0,859
1.936
0.963
6.858
1.879
1.464
0.813
illness
48
2,622
1.078
2.669
1.085
2.184
0.999
1.868
0.896
2.130
1.024
6.968
1.892
1.660
0.862
60
2.857
1.157
2.770
1.125
2.507
1,060
2.057
0.929
2.368
1.096
7.127
1.918
1.853
0.917
of 192 (72 for the illness dataset), which is at least 40% lower than the input length of the other three methods. Furthermore, for even longer input lengths, our model's performance can be further enhanced, signifying that our approach can leverage limited data more efficiently.
We simultaneously study the inference time of our model for various input lengths, denoted as I E (96, 192, 336, 720], and output lengths, denoted as O E (96, 192, 336, 720]. The results are shown in Table 10 These measurements are based on the "exchange" dataset.
Table 10: Inference time for various input/output settings
O=96
O=192
O=336
O=720
I
0.000833
0.001211
0.001419
0.00211
I= 191
0.000884
0.001285
0.001437
0.002139
1=336
0.000893
0.001338
0.001469
0.002194
I=720
0.000941
0.001364
0.001547
0.002246
The table above illustrates the average inference time per instance of our algorithm under different configurations, measured in seconds. As observed, our algorithm exhibits notable speed, averaging at the millisecond level. Furthermore, when comparing the increase in output length to the extension of the input length, the additional inference time incurred by augmenting the input length is minimal. This is attributed to the preprocessing step where the input sequence, regardless of its length, is projected into a fixed-length (usually 100) sequence using linear layer. As result, extending the input length does not significantly amplify the time consumption in our method.
A.3 Additional abalation study
Impact of the number of the BCAB layers: The ablation study the number of BCABs is shown in Table IT The findings indicate that stacking a certain number of BCAB modules can enhance the performance of the model. However, exceeding certain threshold can lead to overfitting, resulting in a decline in performance. Hence, we recommend the use of two layers of BCABs in practical experiments to achieve optimal performance without overfitting the model.
Impact of the bottleneck in the forecast module: The performance of the proposed model under varying bottleneck settings is presented in Table I2 The results demonstrate that employing

---

Table 11: The impact of the number of stacked BCAB on the performance of the model. The electricity dataset employed in this experiment. We present the best results in boldface.
BCAB number
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.186
0.273
0.166
0.260
0.166
0.259
0.168
0.263
0.171
0.266
192
0.187
0.274
0.176
0.270
0.176
0.268
0.176
0.269
0.179
0.272
336
0.208
0.297
0.187
0.280
0.190
0.283
0.190
0.283
0.191
0.284
720
0.244
0.325
0.228
0.313
0.218
0.306
0.234
0.319
0.237
0.319
avg
0.206
0.292
0.189
0.281
0.187
0.279
0.192
0.283
0.194
0.285
bottleneck architecture with a width of 48 can significantly reduce the number of model parameters without degrading the performance significantly, as opposed to not using bottleneck architecture.
Table 12: The impact of the MLP bottleneck in the forecast module. The electricity datasct employed in this experiment. Setting the bottleneck dimension to 96 is equivalent to not using a bottleneck since the input length is 96. The best results are highlighted in bold. The second best is underlined.
bottleneck
96
48
32
24
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.163
0.257
0.166
0.259
0.172
0.267
0.172
0.269
192
0.172
0.265
0.176
0.268
0.182
0.273
0.186
0.279
336
0.186
0.279
0.190
0.283
0.194
0.286
0.197
0.289
720
0.217
0.305
0.218
0.306
0.230
0.316
0.233
0.317
avg
0.184
0.276
0.187
0.279
0.195
0.286
0.197
0.288
A.4
Sensitivity analysis of the weights for the losses in Eq.19)
Our model utilizes three distinet loss functions: the supervised MSE loss for prediction Lyred, the self-supervised InfoNCE loss for basis learning Latign, and the smoothness loss for smoothing the basis over time Taaign- During training, we directly combine these loss functions as the model's performance is not significantly impacted by the relative weights of the individual losses within a certain range. This assertion is supported by the performance evaluation presented Figure 4 which investigates the impact of differrnnt weight combinations of the three loss functions. In setting, we fix the weight of the predicted loss function to be 1, and then fix either the weight of the contrast loss function the smoothness loss function to be I, while the other one varies within the range of (0.2, 0.4, 0.6, 0.8, 1.0, 1.2, 1.4, 1.6). To explore the inflection point of the effect, we take the middle point of two points and calculate finer range again. Our results indicate that the contrast loss function is essentially stable between the weight range of 0.6- while the smoothness loss function is similarly stable between the weight range of 0.9-1.5.
Smooth
0.2350
InfoNCE
0.2325
0.2300
MSE
0.2275
0.2250
0.2225
0.2200
0.2
0.4
weight
Figure 4: MSE for the testing data a function of the weight for the smoothness (the red line) and the infoNCE loss(the blue line).
3

---

A.5 Comparison with self-supervised methods
Our proposed method adopts contrastive learning, here we provide comparative experiments with other self-supervised learning method namely Cost [25]. For the experiments, fix the imput length at 96 and vary the output length from 96 to 720. Table 13 and [able|14illustren that Basisformer outperforms Cost in terms of both MSE and MAE across all cases. It is important note that Cost is trained using a two-step process: self-supervised training followed by supervised ridge regression. In contrast, Basisformer trains the basis, coef, and forecast modules end-to-end. This enables Basisformer to learn a basis that is specifically tailored for forecasting, potentially leading to improved performance.
Table 13: Multidimensional prediction comparative experiments three datasets,
ETT
Electricity
exchange
Models
Metric
96
192
336
720
96
192
336
720
96
192
336
720
MSE
0.184
0.248
0.321
0.410
0.165
0.178
0.189
0,223
0.085
0.177
0.336
0.854
Basisforner
MAE
0.266
0.307
0.355
0.404
0.259
0.272
0.282
0.311
0.205
0.299
0.421
0.670
MSE
0.280
0,480
0.805
1.562
0.199
0.212
0.246
0.263
0.464
0.833
1.992
Cost
MAE
0.375
0.506
0.676
0.955
0.290
0,292
0.307
0.338
0.393
0.521
0.691
0.871
Table 14: Multidimensional prediction comparative experiments another three datasets.
Iraffic
weather
illness
Models
Metric
96
192
336
720
96
192
336
720
96
192
336
720
MSE
0.444
0.460
0.471
0.486
0.173
0.223
0.278
0.355
1.550
1.516
1.877
1.878
Basisformer
MAE
0.315
0.316
0.317
0.318
0.214
0.257
0.298
0.347
0,814
0.819
0.907
0.902
MSE
0.576
0.546
0.555
0.591
0.372
0.528
0.835
1.394
2.330
2.497
2,650
2,829
Cost
MAE
0,377
0.359
0.363
0.379
0.415
0.517
0.666
0.894
0.923
0.962
1.032
1.062
A.6 Comparison with time series discretization
To compare with time series discretization, we have chosen an method, namely Boss (26], which utilizes the Bag-
method for feature extraction.
In order evaluate the effectiveness of our approach, have conducted classification task using several UCR datasets. The description of the datasets is concluded in Table 15
Table 15:
The description of selected UCR datasets
Dataset
Train Size
Test Size
Length
Classes
Is predictable
Description
Mallat
55
2345
1024
Y
simulated dataset
Rock
20
50
2844
Y
rock examples from ASTER spectral librury Each series is extracted from the segmented audio rotationally aligned version of facial outline
Phoneme
214
1896
1024
39
N
FaceUCR
200
2050
131
14
N
To adapt our method for classification, we followed these steps:
We partitioned the sequence into past and future parts, uniformly dividing them in 6:4 ratio for all datasets. Different partitioning methods be explored in future research improve the model's performance.
We used self-supervised approach for training. reserving 10% of the original training data for validating self-supervised performance. The remaining data was used for training, and the self- supervised loss function included prediction, alignment, and smoothness losses. Early termination based on validation set performance was done with a patience 3.
From the well-trained self sepervuper model, we extracted the aggregation coefficient matrix, specifi- cally from the past perspective. This matrix was flattened to create sequence features, which were then fed into random forest classifier for final classification. Notably, during selffsuuervised training, both past and future sequences were used for consistency, but only the past coefficient matrix was utilized in the classifier.

---

We conducted a fair comparison by extracting features using both Boss and our model, ensuring that our feature parameter count did exceed Boss's. We employed a random forest classifier with 100 features and maximum depth of 30 for classification, and the results are summarized in the Table T6
Table 16; The comparison results of BOSS and Basisformer. We present the best results in boldface.
Model
Boss
Basisformer
Metric
acc
acc
valid loss
Mallat
0.83
0.87
0.12
Rock
0.56
0.72
0.18
Phoneme
0.20
0.07
1.27
FaceUCR
0.68
0.41
1.68
The applicability of our method to self-supervision relies on predictability and consistency between past and future data. The validation set loss in the table indicates that datasets lacking predictability have high validation losses, posing challenges for loss function optimization. The Phoneme and FaceUCR datasets lack predictability. The Phoneme dataset includes speech segments from different individuals with random content before and after, while the FCceUCR dataset consists of flattened one-dimensional vectors of rotated face images, both lacking inherent predictability. These datasets require a holistic understanding of the entire sequence for meaningful interpretation, and as result, our proposed Basisformer struggles to extract useful features, leading lower performance compared to Boss.
On the other hand, datasets like Mallat and Rock, exhibiting predictability and low validation losses, allow our approach to achieve superior performance over Boss. Surprisingly, we achieve this performance using only the representation of the past sequence input for the classifier.
A.7 Uncertainty of the results
To assess the stability of our proposed method, we performed repeated experiments and calculated the standard deviations for all methods, as presented in Table 17 Notably, our method exhibits relatively small variance within the table, indicating its high degree of stability.
B Implementation Details
The training and testing of BasisFormer are conducted on an NVIDIA GeForce RTX 3090 graphics card with 24268MB of VRAM. During the trainin process, we adopt the Adabelief optimizer [27 for optimization. We train the model for 30 epochs with the patience of 3 epochs. All experiments are averaged over 5 trials.
To implement the multi-head mechanism, we calculate the multi-head attention for each CAB separately, and then restore it to the original dimension through multiplication. concatenation, and linear layer. In the last layer of the network, mapping layer was utilized to map it to H heads, and the dot product outputs the final coefficients.
To promote the learning of bases and ensure consistency of time series across different dimensions, normalized the time series during training and performed inverse normalization when outputting the results.
For the other models compared in the table, we utilized their original code and conducted experiments by only varying the input length.
C Analysis of the Limitations of BasisFormeet
BasisFormer demonstrates proficiency in learning effective representations and capturing the rela- tionship between bases and time series. However, this proficiency is contingent upon the multi- dimensional time series being the same feature scale, which necessitates normalization of the time series during training and inverse normalization when outputting results. Despite this, the normalization and inverse normalization operations introduce changes to the original distribution of
5

---

Table 17: Results for benchmark datasets with standard deviations in the brackets.
Models
Fodiormer
Autoformer
N-HiTS
FiLM
Dlincar
TCN
Basisformer
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
0,203
0.287
0.255
0.339
0.192
0.265
0.183
0.266
0.193
0.292
3,041
1.350
0.184
0.266
96
(0.002)
(0.001)
(0.020)
(0.020)
(0.003)
(0.002)
(0.000)
(0.000)
(0.004)
(0.006)
(0,000)
(0.000)
(0.002)
(0.002)
ETT
192
(0.006)
0.269
(0,005)
0.328
(0.027)
0.281
(0.025)
0.340
(0.004)
0.287
(0.001))
0.329
(0.000)
0.247
(0.000)
0.305
(0.016)
0.284
(0.016)
0.362
(0.002)
3.072
(0.001)
1.339
(0.004)
0.248
(0.002)
0.307
336
0.325
0.366
0.339
0.372
0.389
0.389
0.309
0.343
0.369
0.554
3.105
1.348
0.321
0.355
(0.002)
(0.003)
(0.018)
(0.015)
(0.005)
(0.003)
(0.000)
(0.000)
(0,006)
(0.002)
(0.005)
(0.005)
(0.005)
(0.004)
0.421
0,415
00.22
0A19
0.591
0.407
0.299
5:185
1.354
0,410
0.404
(0.018)
(0.012)
(0.015)
(0.010)
(0.011)
(0.002)
(0.001)
(0.000)
(0.037)
(0.026)
(0.021)
(0.005)
(0.007)
(0.004)
96
(0.001)
0.193
(0,001)
0,308
(0.005)
0.201
(0.004)
0.317
(0,005)
1,748
(0.001)
1,020
(0.000)
0.199
(0.001)
0.276
(0,000)
0,199
(0.000)
0.284
(0.006)
0.985
0.813
(0.001)
0,165
(0.001)
0.259
electiiey
192
0.201
0,315
0.222
0.334
1.743
1.018
0.198
0.279
0.198
0.287
0.996
0.821
0.178
0.272
(0.005)
(0.006)
(0.003)
(0.004)
(0.008)
(0.003)
(0.000)
(0.001)
(0.000)
(0.000)
(0.008)
(0.007)
(0.001)
(0.001)
0.214
0.329
0.231
0.338
1.677
1.000
0.217
0.301
0,210
0.302
1.000
0.824
0.189
0.282
(0:001)
(0.002)
(0.006)
(0.004)
(0.010)
(0.003)
(8:001)
(0.001)
(0.001)
(0.001)
(0.004)
(0.003)
(0.001)
((,001)
0.246
0.355
0.254
0.361
0.280
0.358
0.245
0.335
1,438
0.784
0.223
0.311
720
(0.003)
(0.003)
(0.007)
(0.008)
(0.000)
(0.000)
(0.000)
(0.000)
(0.006)
(0.005)
(0.002)
(DA0001)
0.148
0,278
0.197
1.049
0.083
0.201
0.085
3.004
0.085
0.205
(0.004)
(0.004)
(0.019)
(0.012)
(0.042)
(0.017)
(0.003)
(0.0033
(0.004)
(0.008)
(0.128)
(0,070)
(0.004)
(0.005)
etclange
192
(0.012)
0.271
(0.010)
0,380
(0.020)
0.300
0.369
(0.015)
1.658
(1.00))
1.023
(0.005)
0.179
(0.002)
0.300
(0.005)
0.176
(0.006)
0.315
(0.020)
3,048
(0.008)
1.444
(0,005)
0.177
(0.006)
0.299
336
(0.009)
0.460
(0.007)
0,500
(0.041)
0.509
(0.016)
0.524
(0.037)
1.566
(0.015)
0,988
(0.005)
0.337
(0.003)
0.416
(0.008)
0,313
(0.006)
0.427
(0.082)
3.113
(0.021)
1.459
(0.011)
0.336
(0.007)
0.121
L.195
0,841
1.447
0,941
18809
1.055
0.642
0.610
0.839
0.695
3.150
L455
0.854
0.670
(0.042)
(0.017)
(0.084)
(0.028)
(0.052)
(0.018)
(0.040)
(0.029)
(0.027)
(0.012)
(0.237)
(0.063)
(0.024)
(0.011)
96
(0.010)
0.587
(0.008)
0,366
(0.028)
0.613
(0.012)
0
(0.016)
2.138
(0.006)
1 1.026
(0.001)
0.652
(0.003)
0.395
(0.001)
0.650
(0.001)
0.396
(0.001)
1,438
(0.001)
0.784
(0.003)
0.444
(0.003)
0.315
traffic
192
(0.012)
0.604
(0.009)
0,373
(0.042)
0.616
(0.020)
0.382
(0.015)
2.101
(0.007)
1.015
(0.001)
0.605
(0.003)
0.371
(0.002)
0.605
(0.001)
0.378
(0.0322
1,463
(0.010)
0.794
(0.004)
0.460
(0.002)
0.316
336
0.621
0,383
0.622
0.387
0.615
0.372
0.612
0.382
1,479
0.799
0.471
0.317
(0.008)
(0.008)
(0:009)
(0.003)
(0.001)
(0.001)
(0.003)
(0.00-1)
(0.003)
(0.002)
(0.005)
(0.004)
720
0.626
0,382
0.660
0,408
0.692
0428
0.645
0.394
1,499
0.804
0.486
0,318
(0.0044)
(0.003)
(0:025)
(0.015)
(0.000)
(0.000)
(0.001)
(0.001)
(0.010)
(0.005)
(0.005)
(0.004)
0.217
0.29%
0.266
0.336
0.648
0.492
0.193
0.234
0.196
0:258
0.615
0.589
0.173
0.214
(0.018)
(0.019)
(0.0077
(0.006)
(8.001)
(0.000)
(0.002)
(0.001)
(0.001)
(0.003)
(0.002)
(0.002)
(0.003)
(0.003)
westteer
192
(0.015)
0.276
(0.017)
0,336
(0.024)
0.307
(0.022)
0.367
(0.003)
0.616
(0.001)
0.479
(0.000)
0.218
(0.001)
0.270
(0.001)
0.237
(0.002)
0.296
(0.023)
0.629
(0.009)
0.600
(0.002)
0.223
(0.001)
0.257
336
(0:014)
0.339
(0.015)
0,380
(0.035)
0.359
(0.031)
(0.0072
0.579
(0.01)
0.462
(0.001)
(0.000)
0.304
(0.0022)
0,283
(0.0044)
(0.050)
0.639
(0.017)
(0.0011)
0.278
(B-I00)
720
(0.009)
0.403
(0.008)
0.428
(0.017)
0.419
(0.014)
0.428
(0.001)
0.541
(0.000)
0.447
(0.001)
0.358
(0.000)
0.350
(0.020)
0.343
(0.020)
0.383
(0.050)
0,639
(0.018)
0.610
(0.001)
0.355
(0.001)
0.347
3,228
1,260
3,486
1.287
3.297
1.679
2.198
0.911
2.398
1.040
6.624
1.830
1,550
0.814
24
(0.020)
(0.019)
(0.107)
(0.018)
(0:007)
(0.000)
(0.138)
(0.058)
(0.0K5)
(0.032)
(0L550)
(0.194)
(0.087)
(0.024)
2.679
1.080
3.103
1,148
2.379
1,441
2.267
0.926
2,646
1.088
6.858
1,879
1.516
0.819
illuess
(0.018)
(0.005)
(0.139)
(0.025)
(0.136)
(0.043)
(0.077)
(0.059)
(0.137)
(0.064)
(0.216)
(0.1352)
(0.130)
(0.055)
48
(0.010)
2.622
(0.002)
1.078
(0.151)
2.669
(0.037)
1,085
(0.092)
3,341
(0.030)
1,751
(0.115)
2.348
(0.037)
0.989
(0.140)
2.614
(0.029)
1.086
(0.032)
6.968
(0.008)
1.892
(0.110)
1.877
(0.032)
0,907
2.857
2.770
1.125
2.278
1.493
2.508
1.038
2,804
7.127
1.918
1.778
0,902
60
(0.011)
(0.003)
(0.085)
(0.019)
(0.187)
(0.064)
(0.130)
(0.018)
(0.049)
(0.009)
(0.134)
(0.025)
(0.098)
(0.024)
Experiment with means it reported an out-of-memory envor on a computer with 128G memory.
the time series, making it arduous to fit certain distributions. As such, future work could explore alternative approaches to training on datasets with considerably different feature scales, eliminating the need for normalization and inverse normalization. Possible avenues for investigation include identifying appropriate mathematical methods or neural network transformations to map data to a suitable and universal feature space.
D Relation to Meta-learning
From a meta-learning standpoint, the learnable basis in our model is tantamount to meta- weta-knowledge for all time series within the same window. The coefficients, which are derived from the similarity between each time series and the foundation, represent distinctive knowledge for each time series. Consequently, our model can be perceived as a manifestation of meta-learning. Notwithstanding. we departed from conventional meta-learning approaches by forgoing a two-stage inner-outer loop optimization method, instead opting for an end-to-end training method.
E Additional Visualization
We have incorporated the visualization of the attention map of the BCAB module on the traffic datasct, as depicted in Figure This visualization demonstrates that different time sequences have

---

distinct attention scores for different the same set of basis vectors. It can also be seen that the attention mechanisms exhibit significant similarities between the past and future sequences. This demonstrates the consistency in the base elements between the two.
36
001
-1225
170
170
265
236
1175
1150
015
015
223
326
033
0.1
832
1100
385
660
043
780
215
518
(a) attention map in the past
(b) attention map in the future
Figure 5: corresponding set attention maps from past and future perspectives
Additionally, we have provided a visualization of specific time sequence alongside the features corresponding to the highest and lowest attention scores, shown in Figure 6 The highest attention score is 0.2316, while the lowest attention score 0.03371. Figure R2 highlights that the representa- tion with total of 8 sets of main peaks (Figure ((c) more comprchensively captures the patterns of the data compared to the configuration with only 2-3 main peaks (Figure 6(b)). This indicates a correlation between the attention scores and the relationship between time sequences and features.
important to note two key points. Firstly, since bases represent condensed patterns of time sequences, it is unlikely for a base to be identical to any single time sequence, especially when N small. Secondly, after the bases are processed through multiple linear and nonlinear layers the network, they correspond to predicted sequences. Therefore, the numerical values of the bases serve as reference points only. The focus should be on patterns exhibited by the bases.
825
823
6-
06-
00-
-33
35
(a) series
(b) hasis with low attention (c) hasis with high attention score
Figure 6: Time series and bases with higher and lower attention scores
Analysis of the Model Complexity
Suppose that the input and output length in BasisFormer is and O respectively when forecasting single time series. Note that the time and space complexity of BasisFormer are of the same order. Therefore, refer both of them as complexity in the sequel.
With regards to the coef module, the complexity is primarily determined by the cross-attention mechanism. Within our approach, BCAB utilizes attention the channel dimension, and

---

encode the time sequence dimension to specified hidden length D. & O via a linear layer during computation. Consequently, the complexity of this module O(N), where N is the number of bases fixed hyperparameter which is usually not large. In this step, we omit the number of BCAB stacks M, since M is also a fixed hyperparameter. As previously mentioned in Appendix A.3 to limit overfitting, M is typically set to 2.
The prediction module incorporates Multilayer Perceptron (MLP) networks, which are employed for separating and concatenating different heads. Both MLP networks have bottlenecks with constant values, and they carry complexity of O(O). In terms of the aggregation of different base vectors, the complexity also is O(0). Therefore, the cumulative complexity of this module is O(O).
In summary, the total complexity of model is O Table 18 provides comparison of the computational complexity among different models, and BasisFormer achieves the lowest complexity among them.
Table 18: Comparison of computational complexity for different models.
Methods
TIME
MEMORY
Fedformer
O(0)
O(0)
Autoformer
O log O)
l00 O)
N-HiTS
O r / -))
0 - -r r((1-))r//(111) r)
FiL.M
(
0
Dlinear
0
TCN
((0)
0
LogTrans
(O O)
O(0
Reformer
O(0 log O)
O log O)
Informer
( log O) ((0)
O(0 log O)
Basisformer
((0)


Reviews:
[{"review_id": "tZ8xwGTEsT", "reviewer": "Reviewer_u9Fd", "summary": "The authors of the paper propose a time series forecasting architecture with a self-supervised method for basis learning, called BasisFormer. Their assumption is that the selection of basis for a time series is consistent across both historical and future sections of the time series. They introduce a Coef module that measures the similarity between the time series and the basis in the historical view via bidirectional cross-attention, and a Forecast module that consolidates vectors from the basis in the future view according to the similarity yielded by the Coef module. In their evaluation they demonstrate improvements in forecasting tasks.", "questions": "1. What happens if bases are not learnable directly from the time-series, but require a domain expert's KB?\n", "limitations": "The novelty of this work is limited to cases where bases do exist and are detectable automatically in the input time-series, where in many cases the provided input is shorter than the length required for identifying cases of seasonality, for example.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written with clear explanations of the proposed architecture.\n\n2. The empirical results, including the ones that were presented in the supplementary material, support the original claims across the manuscript.\n", "weaknesses": "1. In order to further validate the claims presented in this work, I would expect seeing another comparison to methods involve discretization of time-series, such as [Moskovitch, R. and Shahar, Y., 2015. Classification of multivariate time series via temporal abstraction and time intervals mining. Knowledge and Information Systems].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Q1 – Comparison with time series discretization*\n\nWe appreciate your suggestion to include a comparison with methods involving the discretization of time-series, such as Moskovitch and Shahar (2015). Unfortunately, we were unable to find the corresponding code and data for comparison. Therefore, we have chosen an alternative method for comparison, namely Boss [R1], which utilizes the Bag-Of-SFA-symbols method for feature extraction.\n\nIn order to evaluate the effectiveness of our approach, we have conducted a classification task using several UCR datasets. The description of the datasets is as follows:\n\n| Dataset | Train Size | Test Size | Length | Classes | Type      | is_predictable | description                                       |\n|---------|------------|-----------|--------|---------|-----------|----------------|---------------------------------------------------|\n| Mallat  | 55         | 2345      | 1024   | 8       | SIMULATED | Y              | a simulated dataset                               |\n| Rock    | 20         | 50        | 2844   | 4       | SPECTRO   | Y              | rock examples from the ASTER spectral library     |\n| Phoneme | 214        | 1896      | 1024   | 39      | SOUND     | N              | Each series is extracted from the segmented audio |\n| FaceUCR | 200        | 2050      | 131    | 14      | IMAGE     | N              | rotationally aligned version of facial outline.   |\n\nTo adapt our method for classification, we followed these steps:\n\n1.\tWe partitioned the sequence into past and future parts, uniformly dividing them in a 6:4 ratio for all datasets. Different partitioning methods can be explored in future research to improve the model's performance.\n\n2.\tWe used a self-supervised approach for training, reserving 10% of the original training data for validating self-supervised performance. The remaining data was used for training, and the self-supervised loss function included prediction, alignment, and smoothness losses. Early termination based on validation set performance was done with a patience of 3.\n\n3.\tFrom the well-trained self-supervised model, we extracted the aggregation coefficient matrix, specifically from the past perspective. This matrix was flattened to create sequence features, which were then fed into a random forest classifier for final classification. Notably, during self-supervised training, both past and future sequences were used for consistency, but only the past coefficient matrix was utilized in the classifier.\n\nWe conducted a fair comparison by extracting features using both Boss and our model, ensuring that our feature parameter count did not exceed Boss's. We employed a random forest classifier with 100 features and a maximum depth of 30 for classification, and the results are summarized in the table below.\n|         | Boss        | Basisformer | Basisformer |\n|---------|-------------|-------------|-------------|\n|         | acc         | acc         | valid_loss  |\n| Mallat  | 0.83         | 0.87        | 0.12 |\n| Rock    | 0.56        | 0.72        | 0.18 |\n| Phoneme | 0.20    | 0.07        | 1.27|\n| FaceUCR | 0.68    | 0.41        | 1.68|\n\n**The applicability of our method to self-supervision relies on predictability and consistency between past and future data.** The validation set loss in the table indicates that datasets lacking predictability have high validation losses, posing challenges for loss function optimization.\nThe Phoneme and FaceUCR datasets lack predictability. The Phoneme dataset includes speech segments from different individuals with random content before and after, while the FaceUCR dataset consists of flattened one-dimensional vectors of rotated face images, both lacking inherent predictability. These datasets require a holistic understanding of the entire sequence for meaningful interpretation, and as a result, our proposed Basisformer struggles to extract useful features, leading to lower performance compared to Boss.\n\nOn the other hand, datasets like Mallat and Rock, exhibiting predictability and low validation losses, allow our approach to achieve superior performance over Boss. **Surprisingly, we achieve this performance using only the representation of the past sequence as input for the classifier.**\n\n*Q2- non-learnable basis*\n\nOur motivation in this research is to demonstrate the superiority of learnable bases compared to manually chosen bases. **We validated this by replacing the learnable basis with commonly used bases, such as sine-cosine encoding and covariate embedding**, in Table 3 (Page 8). However, we recognize that if a domain expert possesses comprehensive knowledge of the time series characteristics and designs a basis specifically for that set, the manually chosen basis may be comparable or even superior to a learnable basis.\n\n*Q3- input length shorter than a period*\n\nWe respectfully disagree with the mentioned comment. In our research, the notion of the basis expands beyond the sole identification of seasonality. The basis serves as a condensed representation or summary of the inherent shape or pattern within a collection of time series data. As the basis is learnable, it can encompass any relevant information that aids in forecasting.\n\nIt is important to note that our approach involves learning both the historical and future components of the basis from the training data. During inference, these learned components are fixed. **As a result, our forecasting process solely requires understanding how these components map to each other in order to predict the future.**\n\n[R1] Schäfer, Patrick., 2015. The BOSS is concerned with time series classification in the presence of noise. DMKD"}, {"review_id": "eqDNjIDb4A", "reviewer": "Reviewer_twjH", "summary": "This paper proposed BasisFormer which is an end-to-end time series forecasting model that leverages learnable and interpretable bases. BasisFormer treats the historical and future sections of the time series as two distinct views and using contrastive learning. By making use of Coef module and Forecast module, the BasisFormer outperforms previous state-of-the-art methods for univariate and multivariate forecasting tasks.", "questions": "At this point, I don't have specific questions to ask. The paper is clearly written.", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Contrastive learning objective is applied for basis learning which guarantees the consistency between the historical and future sections of the time series. And when applying the SSL module to other frameworks, there is a performance improvement of approximately 5%, which suggests the general application of the learnable basis.\n\n2. Based on the experiment results on six datasets, the proposed BasisFormer model outperforms previous SOTA methods on univariate and multivariate forecasting tasks.\n\n3. The network architecture of BasisFormer is carefully designed and well analyzed through ablation studies and model comparison. The paper is well written and easy to understand.\n", "weaknesses": "In 4.3, the author analyzed the interpretability of the learned bases by visualizing the time series and the corresponding learned basis. An additional visualization of the attention distribution in BCAB module can be helpful for checking different weights assigned to each basis in different attention heads, and thus understanding the network behavior.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Q1 - Visualization*\n\nThanks for pointing this out! We have incorporated the visualization of the attention map of the BCAB module on the traffic dataset, as depicted in Figure R1 in the PDF file attached to the global response. This visualization demonstrates that different time sequences have distinct attention scores for different the same set of basis vectors.\n\nAdditionally, we have provided a visualization of a specific time sequence alongside the features corresponding to the highest and lowest attention scores, as shown in Figure R2. The highest attention score is 0.2316, while the lowest attention score is 0.03371. Figure R2 highlights that the representation with a total of 8 sets of main peaks (Figure R2(c)) more comprehensively captures the patterns of the data compared to the configuration with only 2-3 main peaks (Figure R2(b)). This indicates a correlation between the attention scores and the relationship between time sequences and features.\n\nIt is important to note two key points. Firstly, since bases represent condensed patterns of time sequences, it is unlikely for a base to be identical to any single time sequence, especially when N is small. Secondly, after the bases are processed through multiple linear and nonlinear layers in the network, they correspond to predicted sequences. Therefore, the numerical values of the bases serve as reference points only. **The focus should be on the patterns exhibited by the bases.**"}, {"review_id": "cpI4iCcpxU", "reviewer": "Reviewer_z6WM", "summary": "This paper studies basis learning for time series forecasting for which the past and future basis representations are aligned. Contrastive learning is used to build the time series basis and similarity between the past values and basis is used for time series prediction. The experiments on several time series forecasting datasets show improvement, especially for multivariate forecasting, when the input length (history) is short.\n", "questions": "- Basisformer seems to have more advantages over baselines for the multivariate time series forecasting, how do you explain this?\n- How models are trained for the longer input sequence? Is it only inference time change?\n- Increasing the input length gives more advantage to the baselines. Do you conduct further analysis of the inference time versus performance for different length of input? \n- Do you plan to share the implementation?\n", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper is overall well written\n- The idea of using contrastive learning to learn basis is sound \n- The proposed approach improves over baselines especially for multivariate case\n", "weaknesses": "- There is a strong emphasis on the consistency of the representations for past and future but this is directly experimented/evaluated \n- Reproducibility is questionable without source code and scripts to run as the approach composed of several components and steps of training \n- Proposed approach is particularly improves over baselines when the history to predict is shorter, however the motivation/need for a shorter history is not supported \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Q1 - past & future consistency*\n\nThank you for bringing this to our attention. **We have shown the consistency of representations for the past and future sequences in our model in Figure R1 in the PDF file attached to the general response**, where the attention mechanisms exhibit significant similarities between the past and future sequences. This demonstrates the consistency in the base elements between the two.\n\n**Moreover, we have validated the significance of this past and future consistency in Table 5 (Page 8) and Table 6 (Page 9). **In Table 5, by ensuring such consistency through the InfoNCE loss, our proposed Basisformer achieves an average performance improvement of 5.2%. Table 6 further supports this finding by showing that consistency in Autoformer [8] and Fedformer [9] also leads to performance enhancement.\n\n*Q2 - reproducibility*\n\nAs mentioned at the end of Abstract, full code of our model will be available at the time of publication.\n\n*Q3 - motivation for a short history*\n\nAs mentioned in Lines 392-393 in the supplemtary material, \"Throughout our research, we maintain consistency in our experimental settings by fixing the input length to be $96$ (with a reduced input length of $36$ for the illness dataset), instead of using a longer length. The main rationale behind this decision is that, in practical scenarios where the model is deployed as an online service and tasked with predicting a long range of the future at a granular level of minutes or hours, collecting a lengthy history (i.e., spanning 720 timestamps) for a large number of time series in real-time can be quite challenging. Therefore, **the adoption of an input length of 96 proves to be more practical and feasible.\"**\n\nIndeed this is a common practice in  N-beats[1], Autoformer[8], and Fedformer[9].\n\nOn the other hand, to further demonstrate the superiority of Basisformer for longer input sequences, we have conducted experiments in Section A.2 in the supplementary material. As discussed in Lines 412-418, \"concerning longer inputs, our method surpasses recent approaches such as Dlinear, FiLM, and N-HiTS, with an average MSE performance improvement of 1.35\\%, 0.63\\%, and 7.75\\%, respectively, and a corresponding evaluation MAE performance improvement of 3.15\\%, 2.33\\%, and 4.06\\%, respectively. It is noteworthy that our approach requires an input length of 192 (72 for the illness dataset), which is at least 40\\% lower than the input length of the other three methods. **Furthermore, for even longer input lengths, our model's performance can be further enhanced, signifying that our approach can leverage limited data more efficiently.\" **\n\n*Q4 - performance is better for multivariate time series*\n\nThe advantages of Basisformer over baselines for multivariate time series forecasting can be explained by considering the nature of multidimensional time series and the role of our model's basis. \n\nThe basis functions in our model play a crucial role in capturing the primary patterns within the time series, akin to the principles of principle component analysis (PCA) to some extent. It is important to note that PCA performs better when the observed data are more correlated. \n\nMultidimensional time series, in comparison to single-dimensional ones, tend to have higher levels of correlation. For instance, traffic datasets often exhibit peaks during morning and evening hours, indicating correlated patterns. Consequently, the presence of such correlations allows our model to effectively learn the basis functions, leading to improved forecasting performance.\n\n**In summary, the utilization of basis functions in Basisformer, combined with the presence of correlated patterns in multidimensional time series, allows for better noise mitigation and identification of key patterns. **This ultimately enhances the overall performance of our approach for multivariate time series forecasting.  \n\n*Q5 - how to train the model for longer inputs*\n\nOur model is trained using an end-to-end approach. When dealing with longer input sequences, we follow a similar methodology as N-hits [2] and Dlinear [4]. Specifically, we retrain the model using the extended input sequences along with their corresponding output sequences. Consequently, both the training and inference times are modified to accommodate the longer input sequences.\n\n*Q6 - time vs input length*\n\nWe have provided a table below that displays the inference times for various input lengths, denoted as $I={96,192,336,720}$, and output lengths, denoted as $O={96,192,336,720}$. These measurements are based on the \"exchange\" dataset.\n|    | O=96     | O=192     | O=336     |     O=720      |\n|-------|-----------|-----------|-----------|-----------|\n| **I=96**  | 0.000833  | 0.001211  | 0.001419  | 0.002110  |\n| **I=192** | 0.000884  | 0.001285  | 0.001437  | 0.002139  |\n| **I=336** | 0.000893  | 0.001338  | 0.001469  | 0.002194  |\n| **I=720** | 0.000941  | 0.001364  | 0.001547  | 0.002246  |\n\nThe table above illustrates the average inference time per instance of our algorithm under different configurations, measured in seconds. As observed, our algorithm exhibits notable speed, averaging at the millisecond level. Furthermore, when comparing the increase in output length to the extension of the input length, the additional inference time incurred by augmenting the input length is minimal. This is attributed to the preprocessing step where the input sequence, regardless of its length, is projected into a fixed-length (usually 100) sequence using a linear layer. **As a result, extending the input length does not significantly amplify the time consumption in our method.**"}, {"review_id": "lQtR6qUyIF", "reviewer": "Reviewer_CQBS", "summary": "This paper addresses the problem of finding effective bases for time series forecasting models. Current methods are limited in their ability to satisfy the requirements of being tailored to specific time series data and exhibiting distinct correlation with each time series. To tackle this challenge, the authors propose BasisFormer, an end-to-end architecture that leverages learnable and interpretable bases. Bases are obtained through adaptive self-supervised learning, where historical and future time series sections are treated as distinct views and contrastive learning is used. The proposed architecture includes a Coef module that calculates similarity coefficients between time series and bases using bidirectional cross attention, and a Forecast module that selects and consolidates bases for accurate future predictions. Extensive experiments on six datasets demonstrate that BasisFormer outperforms previous methods for both univariate and multivariate forecasting tasks, achieving considerable improvements in performance.", "questions": "see weakness", "limitations": "n/a", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper has studied a time series forecasting problem. The paper is well written. The significance of this work lies in addressing the limitations of existing methods in time series forecasting by proposing BasisFormer, an architecture that leverages learnable and interpretable bases. This approach allows for tailored modeling of specific time series data and distinct correlations with each time series, leading to improved forecasting accuracy.", "weaknesses": "However, I still have several concerns towards this paper. First, the illustration and motivation for the basis/bases is not very clear. What are the basis in essence? How can you define basis in time series? Are basis really needed for time series forecasting? What good effects can basis learning bring up? How to verify it? Second, in the main contribution, though the authors combine several techniques together, including self-supervised learning, contrastive learning, cross attention, basis selection. However, I don't find the necessity for each component. Third, the experimental results are less convincing. The proposed method adopts contrastive learning, but it does not compare the performance with other self-supervised learning methods, such as [1] [2]. Other works adopt periodic basis are also not discussed in related work [3].\n\n[1] Cost: contrastive learning of disentangled seasonal-trend representations for time series forecasting. ICLR 2022.\n[2] Time-Series Representation Learning via Temporal and Contextual Contrasting. IJCAI 2021.\n[3] DEPTS: Deep Expansion Learning for Periodic Time Series Forecasting. ICLR 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Q1 - motivation for baisis*\n\nAs mentioned in Lines 20-22 on Page 1, bases are defined as **sequences that capture the underlying temporal patterns for a set of time series and serve as the key factors driving changes in the data over time**. They may encompass trends, seasonalities, and other vital elements that aid in modeling and forecasting time series data.\n\n\nRegarding the necessity of bases, **it is worth noting that nearly all time series forecasting models can be viewed as basis-driven models.** For example, Transformer-related models [8] [9] rely on covariate encoding, akin to bases, and some models explicitly depend on specific bases such as Fourier or Legendre bases [2] [3]. Experimental validation in this regard has been carried out previously, as exemplified by the following table citing experimental results from Dlinear [4]. In the presented experiments, the absence of bases (i.e., covariates) represented by timestamps (denoted as \"wo/Temp\") leads to a significant decrease in model performance, underscoring the importance of bases. It is important to note that the values in the table are presented in MSE format, where smaller values indicate better performance.\n\n| Methods    | Embedding | 96    | 192    | 336   | 720   |\n|:----------:|:---------:|:-----:|:------:|:-----:|:-----:|\n| Fedformer  | ALL       | 0.597 | 0.606  | 0.627 | 0.649 |\n|            | wo/Temp   | 0.613 | 0.623  | 0.65  | 0.677 |\n| Autoformer | ALL       | 0.629 | 0.647  | 0.676 | 0.638 |\n|            | wo/Temp   | 0.681 | 0.665  | 0.908 | 0.769 |\n| Informer   | ALL       | 0.719 | 0.696  | 0.777 | 0.864 |\n|            | wo/Temp   | 0.754 | 0.780  | 0.903 | 1.259 |\n\n\nThe objective of Basisformer is to further show that learnable basis is preferred to manually specified basis for time series forecasting, and the usefulness of learnable basis is demonstrated in Table 5 (Page 8) and Table 6 (Page 9). In Table 5, by learning the basis adaptively from the data via the InfoNCE loss (cf. Section 3.3), our proposed Basisformer achieves an average performance improvement of 5.2%. Table 6 further supports this finding by showing that using a learnable basis instead of the manually specified covariates in Autoformer [8] and Fedformer [9] also leads to performance enhancement.\n\n*Q2 - contribution of contrastive learning and cross attention*\n\nAs discussed in the second paragraph of Section I, the application of bases for time series forecasting involves three essential steps. Firstly, an appropriate basis is learned for the set of time series under consideration. Secondly, each time series in the set is decomposed based on the learned basis, which entails calculating coefficients or weights that represent the similarity or projection energy of the time series with respect to each vector in the basis. Finally, the prediction is determined by aggregating the future part of the basis using the computed weights.\n\nTo this end, we employ self-supervised learning, specifically contrastive learning, to learn the basis, as described in Section 3.3. Cross attention is then utilized to perform basis projection or selection, which corresponds to the second step, as explained in Section 3.1.\n\n**To demonstrate the necessity of the self-supervised basis learning module, we conducted an ablation study (Lines 283-289, Page 7).** The results show that employing only the standalone self-supervised loss, namely the infoNCE loss, leads to an average performance improvement of 5.2%. \n\nFurthermore, we conducted additional experiments to validate the effectiveness of cross attention. In these experiments, we removed cross attention and basis selection by setting the number of stacked BCABs (Bidirectional Cross-Attention Blocks) in the model to 0. The experimental results are presented in the table below.\n| BCAB number | 0     | 1     | 2     | 3     | 4     |\n|-------------|-------|-------|-------|-------|-------|\n| Metric      | MSE   | MSE   | MSE   | MSE   | MSE   |\n| 96          | 0.186 | 0.166 | 0.166 | 0.168 | 0.171 |\n| 192         | 0.187 | 0.176 | 0.176 | 0.176 | 0.179 |\n| 336         | 0.208 | 0.187 | 0.190 | 0.190 | 0.191 |\n| 720         | 0.244 | 0.228 | 0.218 | 0.234 | 0.237 |\n| avg         | 0.206 | 0.189 | 0.187 | 0.192 | 0.194 |\n\n**The table highlights that, without altering other conditions, the absence of BCAB (i.e., cross attention for basis selection) resulted in an average performance decrease of 10.16% compared to the best-performing configuration with 2 layers of BCAB.** This emphasizes the effectiveness of cross attention in the model.\n\n*Q3 - comparison with self-supervised methods*\n\nDue to time constraints, we are currently able to present comparative results with only one method, Cost. For the experiments, we fix the input length at 96 and vary the output length from 96 to 720. **Table R1 in the PDF attached to the general response illustrates that Basisformer outperforms Cost in terms of both MSE and MAE across all cases.** It is important to note that Cost is trained using a two-step process: self-supervised training followed by supervised ridge regression. In contrast, Basisformer trains the basis, coef, and forecast modules end-to-end. This enables Basisformer to learn a basis that is specifically tailored for forecasting, potentially leading to improved performance.\n\n**Regarding DEPTS, we will discuss it in Section 2 as follows: **\"DEPTS tackles the challenges posed by intricate dependencies and multiple periodicities in periodic time series through the implementation of a deep expansion learning framework. However, the complex initialization and optimization strategies employed by DEPTS, as well as its limitation of only being applicable to periodic sequences, have motivated us to develop a simpler and more universally applicable basis learning framework.\"\n\n"}, {"review_id": "OZSm3KDJ6i", "reviewer": "Reviewer_yNQz", "summary": "To be effective, a basis must be tailored to the specific set of time series data and exhibit distinct correlation with each time series within the set. As far as we know, the state-of-the-art methods are limited in their ability to satisfy both of these requirements simultaneously. To address this issue, the authors propose BasisFormer, an end-to-end time series forecasting architecture that leverages learnable and interpretable bases. Firstly, the authors acquire bases through adaptive self-supervised learning, which treats the historical and future sections of the time series as two distinct views and employs contrastive learning. Secondly, the authors design a Coef module that calculates the similarity coefficients between the time series and bases in the historical view via bidirectional cross-attention. Finally, the authors present a Forecast module that selects and consolidates the bases in the future view based on the similarity coefficients, resulting in accurate future predictions. ", "questions": "The authors addressed all my concerns. So, I have no further comments.", "limitations": "Yes", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors acquire bases through adaptive self-supervised learning, which treats the historical and future sections of the time series as two distinct views and employs contrastive learning. \n2. The authors design a Coef module that calculates the similarity coefficients between the time series and bases in the historical view via bidirectional cross-attention. \n3. The authors present a Forecast module that selects and consolidates the bases in the future view based on the similarity coefficients, resulting in accurate future predictions. ", "weaknesses": "1. The authors claim that this is a self-supervised learning model, and according to the reports of existing studies, the performance should be weaker than that of supervised models. I hope the authors can compare some time series prediction models based on self-supervised learning. Forgive me for being skeptical about the effect at the moment.\n\nminor comment:\n\n2. I would like the authors to be more clear about Eq. 7, why explainability is related to the smooth term, and to use the common l2-norm.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Q1 - comparison with self-supervised methods*\n\nFirst, We would like to clarify that our model should not be classified as a self-supervised model in a strict sense. This is because our model utilizes a supervised loss function for predictions, while the alignment loss function and smoothing loss function are self-supervised. It is worth noting that our approach differs from classical self-supervised algorithms that involve a two-step process of self-supervised training followed by a supervised ridge regression. In contrast, we adopt an end-to-end training approach by integrating the self-supervised module with the prediction task.\n\nTo address the reviewer's request, we have compared our model with several recent self-supervised methods, namely [1] and [2]. The comparative results are presented in the following table. For the experiments, we fix the input length at 96 and vary the output length from 96 to 720. **The table below illustrates that Basisformer outperforms Cost in terms of both MSE and MAE across all cases.** It is important to note that Cost is trained using a two-step process: self-supervised training followed by supervised ridge regression. In contrast, Basisformer trains the basis, coef, and forecast modules end-to-end. This enables Basisformer to learn a basis that is specifically tailored for forecasting, potentially leading to improved performance.\n\n| Models      |     |ours |        | Cost   |        |\n|:-----------:|:---:|:------:|:------:|:------:|:------:|\n| Metric      |   output_length  | MSE    | MAE    | MSE    | MAE    |\n| ETT         | 96  | 0.184  | 0.266  | 0.280  | 0.375  |\n|             | 192 | 0.248  | 0.307  | 0.480  | 0.506  |\n|             | 336 | 0.321  | 0.355  | 0.805  | 0.676  |\n|             | 720 | 0.410  | 0.404  | 1.562  | 0.955  |\n| electricity | 96  | 0.165  | 0.259  | 0.199  | 0.290  |\n|             | 192 | 0.178  | 0.272  | 0.199  | 0.292  |\n|             | 336 | 0.189  | 0.282  | 0.212  | 0.307  |\n|             | 720 | 0.223  | 0.311  | 0.246  | 0.338  |\n| exchange    | 96  | 0.085  | 0.205  | 0.263  | 0.393  |\n|             | 192 | 0.177  | 0.299  | 0.464  | 0.521  |\n|             | 336 | 0.336  | 0.421  | 0.833  | 0.691  |\n|             | 720 | 0.854  | 0.670  | 1.192  | 0.871  |\n| traffic     | 96  | 0.444  | 0.315  | 0.576  | 0.377  |\n|             | 192 | 0.460  | 0.316  | 0.546  | 0.359  |\n|             | 336 | 0.471  | 0.317  | 0.555  | 0.363  |\n|             | 720 | 0.486  | 0.318  | 0.591  | 0.379  |\n| weather     | 96  | 0.173  | 0.214  | 0.372  | 0.415  |\n|             | 192 | 0.223  | 0.257  | 0.528  | 0.517  |\n|             | 336 | 0.278  | 0.298  | 0.835  | 0.666  |\n|             | 720 | 0.355  | 0.347  | 1.394  | 0.894  |\n| illness     | 96  | 1.550  | 0.814  | 2.330  | 0.923  |\n|             | 192 | 1.516  | 0.819  | 2.497  | 0.962  |\n|             | 336 | 1.877  | 0.907  | 2.650  | 1.032  |\n|             | 720 | 1.878  | 0.902  | 2.829  | 1.062  |\n\n\n\n\n*Q2 - I would like the authors to be more clear about Eq. 7, why explainability is related to the smooth term, and to use the common l2-norm.*\n\nThe incorporation of a loss term related to smoothness serves two significant purposes. **Firstly, it plays a crucial role in mitigating the risk of the learned basis fitting the noise present in the data, which would result in the emergence of high-frequency patterns.** If the basis is corrupted by noise, it becomes difficult to extract meaningful information such as trends or seasonality. Therefore, ensuring a smooth basis is essential for obtaining interpretable results.\n\n**Secondly, the inclusion of the smoothness loss term directly impacts the overall performance of the model.** This is evident from the results of ablation experiments involving different loss functions, as presented in Table 5. As our primary objective is long-range future forecasting, it is advantageous to have smoothly changing patterns in the basis.\n"}]

Author rebuttal:
**General Response to All Reviewers**

We sincerely thank all the reviewers for their valuable suggestions. We are delighted by the unanimous recognition of our work and appreciate the reviewers' positive feedback on the carefully designed network architecture and the use of contrastive learning in BasisFormer.

We have thoroughly reviewed each of the reviewers' questions and suggestions, and we are grateful for their patience and diligence. In response, we have conducted additional experiments, introduced a new baseline, provided visualizations to address any lingering questions, and emphasized the significance of our work.

In the following rebuttal, we address each reviewer's comments individually. The reviewer’s comments are shown in italics. The paragraph(s) following them is the authors’ response. Unless otherwise specified, all references to pages, equations, sections, and citations refer to the original paper. Additionally, figures, tables, and citations prefixed with "R" (e.g., [R1]) are newly added citations in this rebuttal. All newly added images and a table are enclosed within a separate single-page PDF attached to this general response. We will incorporate the suggested revisions into the final camera-ready version to enhance the clarity and persuasiveness of our paper.

Once again, we would like to express our gratitude to the reviewers for their insightful feedback, which has helped us identify areas for improvement and refine our work. We welcome any further insights or concerns that would contribute to enhancing the paper according to the reviewers' perspectives.

Meta-review: The paper presents a new method that leverages similarities between time series in a contrastive learning framework, with the main selling point being the considerable improvement in performance over state of the art methods. The reviewers praised the clarity of the writing, the explanations concerning the design decisions, the empirical results, and the analysis through ablation studies that yielded valuable insights. Three of the reviewers indicated acceptance, 2 of which gave very high scores. On the other hand, Reviewer z6WM had concerns about the consistency of the representations, the model performance on longer time series and reproducibility of the experiments. Meanwhile, Reviewer CQBS issued concerns about the motivation for bases and each of the components. The authors have, in my opinion, responded adequately to the concerns raised by Reviewers z6WM and CQBS. As these two reviewers have not read the authors' response, I will go with my own assessment and consider most of their comments addressed.
All things considered, I recommend acceptance of this paper.
I do, however, urge the authors to incorporate their responses on consistency and length of time series in the discussion part of their camera-ready paper, bolstering the explanations that they already have provided on these topics in the current manuscript.
Decision: Accept (poster)