Title: BasisFormer: Attention-based Time Series Forecasting with Learnable and Interpretable Basis
Venue: NeurIPS 2023
Accepted: yes
Keywords: time series forecasting

basis learning

self-supervised learning
Abstract: Bases have become an integral part of modern deep learning-based models for time series forecasting due to their ability to act as feature extractors or future references. To be effective, a basis must be tailored to the specific set of time series data and exhibit distinct correlation with each time series within the set. However, current state-of-the-art methods are limited in their ability to satisfy both of these requirements simultaneously. To address this challenge, we propose BasisFormer, an end-to-end time series forecasting architecture that leverages learnable and interpretable bases. This architecture comprises three components: First, we acquire bases through adaptive self-supervised learning, which treats the historical and future sections of the time series as two distinct views and employs contrastive learning. Next, we design a Coef module that calculates the similarity coefficients between the time series and bases in the historical view via bidirectional cross-attention. Finally, we present a Forecast module that selects and consolidates the bases in the future view based on the similarity coefficients, resulting in accurate future predictions. Through extensive experiments on six datasets, we demonstrate that BasisFormer outperforms previous state-of-the-art methods by 11.04% and 15.78% respectively for univariate and multivariate forecasting tasks. Code is
available at: https://github.com/nzl5116190/Basisformer.

BasisFormer: Attention-based Time Series Forecasting with Learnable and Interpretable Basis

Abstract

Bases have become an integral part of modern deep learning-based models for time series forecasting due to their ability to act as feature extractors or future references. To be effective, a basis must be tailored to the specific set of time series data and exhibit distinct correlation with each time series within the set. However, current state-of-the-art methods are limited in their ability to satisfy both of these requirements simultaneously. To address this challenge, we propose BasisFormer, an end-to-end time series forecasting architecture that leverages learnable and interpretable bases. This architecture comprises three components: First, we acquire bases through adaptive self-supervised learning, which treats the historical and future sections of the time series as two distinct views and employs contrastive learning. Next, we design a Coef module that calculates the similarity coefficients between the time series and bases in the historical view via bidirectional crossattention. Finally, we present a Forecast module that selects and consolidates the bases in the future view based on the similarity coefficients, resulting in accurate future predictions. Through extensive experiments on six datasets, we demonstrate that BasisFormer outperforms previous state-of-the-art methods by 11.04% and 15.78% respectively for univariate and multivariate forecasting tasks. Code is available at: https://github.com/nzl5116190/Basisformer .

1 Introduction

Bases, such as trends and seasonalities, are indispensable for time series modeling and forecasting, since they capture the underlying temporal patterns and serve as the key factors driving changes in the data over time. For example, seasonalities can capture the regular fluctuations in demand for a product or a service, while trends can reflect the long-term growth or decline of a market or industry. Incorporating these bases into time series models can improve the understanding and prediction of future behaviors. Indeed, all commonly used deep learning models for time series forecasting can be reimagined as basis-driving models. N-BEATS [1], N-HiTS [2], and FiLM [3] all resort to explicit bases, such as Fourier and Legendre basis. More generally, the linear (or convolution) layers in MLPs [4] (or CNNs [5]) can be regarded as implicit bases, as they act as filter banks to decompose the time series. In addition, the covariate embedding (a.k.a. global timestamp embedding) in RNNs [6]

and Transformers [7-13] is another form of basis, since they provide reference points for predicting future sequences.

To apply bases for time series forecasting, three steps are required. First and foremost, an appropriate basis should be chosen or learned for the set of time series at hand. In practice, the full space of the basis can often be very large, whereas the patterns of the time series in a set are often similar. It is therefore desirable to learn a basis that is tailored to the specific characteristics (e.g., periods or frequencies) of the time series data. This can help reduce the complexity of the forecasting model, as well as make it more accurate and interpretable. Second, each time series in the set is decomposed according to the basis. This involves computing coefficients or weights that determine the similarity or projection energy of the time series w.r.t. (with respect to) each vector (i.e., filter) in the basis. Note that these coefficients are supposed to vary across different time series, since each time series also exhibits unique patterns and characteristics. For instance, the Fourier coefficients corresponding to different time series will be dissimilar. Finally, the prediction is determined by the weighted aggregation of the future part of the basis.

Unfortunately, the aforementioned state-of-the-art methods fall short in satisfying the requirements in the first two steps simultaneously. On one hand, methods relying on classical bases, such as NBEATS [1], N-HiTS [2], and FiLM [3], often assume that the basis is not learnable and instead learn the coefficients of each time series in a flexible manner when projecting to the basis. However, such a basis may not effectively account for temporal patterns since there is no guarantee that the given basis includes all periods or frequencies corresponding to the set of time series. On the other hand, approaches that aim to adaptively learn the basis from data, such as MLP [4], CNN [5], RNN [6], Transformer and their variants [7-13], often overlook the need for flexible associations between the basis and each individual time series. Specifically, although Transformer and its variants learn the covariate embeddings, they add or concatenate the same embeddings to different time series embedding in a restricted manner. For MLP and CNN, they adopt the same linear and convolution layers for all time series.

To effectively tackle the aforementioned quandary, it is imperative to obtain a basis that can accurately reflect the distinct traits of the dataset, and to devise a predictive network that can selectively utilize relevant vectors in the basis for forecasting purposes. To move forward to this goal, we propose BasisFormer, a time series forecasting architecture with learnable and interpretable basis. As a first step, we acquire the basis through adaptive self-supervised learning from the data. This involves treating the historical and future sections of a time series as two distinct views and employing contrastive learning to learn the basis, assuming that the selection of basis for a time series should be consistent across both views. Subsequently, we design the Coef module that measures the similarity between the time series and the basis in the historical view via bidirectional cross-attention, facilitating the flexible association between individual time series and the basis. Finally, we develop a Forecast module that consolidates vectors from the basis in the future view according to the similarity yielded by the Coef module, leading to accurate future predictions. We emphasize that the above three parts are trained in an end-to-end fashion. In summary, the key contributions of our work comprise:

We propose a self-supervised method for basis learning by treating the historical and future sections of the time series as two distinct views and employing contrastive learning, which ensures that the selection of basis for a time series is consistent across both views.

We design the Coef and Forecast module that chooses and merges relevant basis in the future view based on the coefficients that measure the similarity between the time series and the basis in the historical view.

We conduct extensive experiments on six datasets, and find that our model outperforms previous SOTA methods by 11.04% for univariate forecasting tasks and 15.78% for multivariate forecasting tasks.

2 Related works

Time series forecasting models In recent years, deep learning methods have emerged as the predominant technique for time series forecasting. As illustrated in the introduction, these deep learning methods typically resort to bases to facilitate the prediction of the future. Depending on the types of bases used in the network, forecasting models fall into two categories: those using classical orthogonal bases and those using learnable bases. The first group involves N-BEATS [1], N-HiTS [2], and FiLM [3]. N-BEATS and N-HiTs typically utilize the Fourier basis and then learn the coefficients

for this basis in a recursive network such that the basis helps decompose the historical part of the time series into different components and these components can be further aggregated to predict the future. FiLM approximates the historical part via the Legendre polynomial basis and further removes the noise with the help of the Fourier basis. The major drawback with the group of methods is that the basis is predefined, thus giving rise to the problem of which type of basis to choose (e.g., Fourier or Legendre polynomial) and further which vectors in the basis to choose (e.g., which frequency components we choose from the Fourier basis). On the other hand, the models based on learnable bases, such as Dlinear [4], TCN [5], Deepar [6], LogTrans [11], Informer [7], AutoFormer [8], FedFormer [9], etc, use a learnable linear or convolution layer, or covariate embeddings as the basis. Although these bases are adaptable to the time series, the relationship between the basis and the time series is fixed for all time series. For example, the covariate embedding is added or concatenated to the embedding of different time series in the same way, without considering the unique frequencies and periodic patterns of each series. In our paper, we propose a method that allows for both a learnable basis and a flexible correlation between the basis and each time series for more accurate predictions.

Basis learning for time series analysis Apart from time series forecasting, learnable bases have also been explored for other time series-related tasks, such as time series classification. Note that basis learning is distinct from the representation learning of time series, as the former aims to capture the pattern of a set of time series using a common basis, while the latter aims to extract features from individual time series. Moreover, the basis can help extract features from time series as in Dlinear [4] and TCN [5]. Traditionally, a non-learnable basis is exploited, such as Fourier and wavelet basis. However, there exist a handful of works that overcome this limitation and enable the use learnable basis. The learnable group transform [14] generalizes the filter banks in the wavelet transformer and allows nonlinear transformation from the mother wavelet to obtain flexible filter banks that can better extract features from time series. Along this direction, Balestriero et al. [15] propose a learnable Gaussian filter over the Wigner-Ville transform with a few interpretable parameters, and prove that the resulting filter banks can interpolate between classical ones, including Fourier, wavelet, and chirplet basis. Similar works have also been proposed for audio signal processing [16, 17], suggesting that a learnable basis is a more effective feature extractor than a fixed basis. Thus, we utilize a learnable basis in our work and demonstrate its usefulness for time series forecasting. It should be noted that DEPTS [18] tackles the challenges posed by intricate dependencies and multiple periodicities in periodic time series through the implementation of a deep expansion learning framework. However, the complex initialization and optimization strategies employed by DEPTS, as well as its limitation of only being applicable to periodic sequences, have motivated us to develop a simpler and more universally applicable basis learning framework. Concretely, we propose a novel method for basis learning based on self-supervised contrastive learning.

Self-supervised time series representation learning Since we employ self-supervised representation learning techniques to learn the basis, it is necessary to examine related works in this domain. One prevalent method in this field is the contrastive predictive coding (CPC) [19] which implements contrastive learning to obtain time series representations by treating subsequent future time series as positive samples and random non-subsequent future time series as negative samples. Another method, TS-TCC [20], augments the data with two types of perturbations to obtain two views and performs a cross-view prediction task contrastively, similar to CPC. As an alternative, TS2VEC [21] generates positive samples via timestamp masking or random cropping without leveraging the future information. Note that all these methods seek to establish a common representation for a time series from various views. Unlike these approaches, our goal is to preserve the consistency of the relationship between the time series and the basis. In other words, while the representation of the time series in the historical and future view may differ, their relationships with the corresponding basis should remain consistent.

3 BasisFormer

Suppose we have a collection of time series with a dimension of C , implying that C correlated time series require simultaneous prediction. Each time series in this set is characterized by its history x = ( x 1 , · · · , x I ) and future y = ( y 1 , · · · , y O ) , where I and O correspond to the input and output sequence lengths, respectively. Our primary objective is to learn a basis z that can account for the behavior of all time series in the group, and further exploit it for predicting y given x . Correspondingly, z can also be split into the historical component z x and the future component z y .

Coef

Module

Bottleneck

MLP

Bottleneck

MLP

...

BCAB

Coef

Module

FFN

FFN

Multihead

Cross

Attention

Multihead

Cross

Attention

BCAB

Split

Concat

Split

MSE loss

③

Basis Module

②

Forecast Module

①

BCAB

Time Series

Basis

Historical View

Future View

Q

K

V

Q

K

V

Share

Parameters

MLP

...

...

BCAB

As mentioned in the introduction, three steps are necessary to employ bases for time series forecasting: learning an appropriate basis, computing the coefficient of the time series with respect to each basis vector, and predicting based on a weighted aggregation of the future portion of the basis. The proposed BasisFormer facilitates these three steps in a highly versatile manner. As demonstrated in Figure 1, the overall architecture of BasisFormer also contains three parts: 1 the Coef module, which compares a time series to each basis vector to determine the corresponding coefficient; 2 the Forecast module, which predicts the future based on the coefficients and the future section of the basis; and 3 the Basis module, which learns the basis by aligning the relationships between the basis and the time series from the historical and future perspectives. We will now elaborate on each of the BasisFormer components.

3.1 Coef module for similarity comparison between time series and basis

The Coef Module is designed to measure the similarity between a set of time series and a set of basis vectors. Since our focus is on the relations between the two sets, rather than within each set, we exploit a bipartite graph to represent such relations, where the nodes in one set represent the time series (see green nodes in Figure 1) and the nodes in the other set represent the basis vectors (see blue nodes in Figure 1). As a result, the strength of the edge connecting two nodes in the graph is equivalent to the similarity coefficient between them (see red edges in Figure 1). To obtain the edge strength, we require representations of the nodes in the graph. We accomplish this by developing a bidirectional cross-attention block (BCAB) to learn the node representations via cross-attention, in a similar vein to the graph attention network [22].

As a first step, given two sets of inputs a ( i ) and b ( i ) , let us define the cross attention block (CAB) as:

where MA H denotes multihead attention with H heads whose query is given by Q = W q a , key by K = W k b , and value by V = W v b . By exchanging information mutually between a ( i ) and b ( i ) , we can construct the BCAB as:

where

Note that the parameters in CAB H for computing a ( i +1) and b ( i +1) may differ, so as to capture the heterogeneity in the relations from a ( i ) to b ( i ) and from b ( i ) to a ( i ) .

Correspondingly, given C time series x ∈ R C × I and the basis z x ∈ R N × I of size N , we can get their representations by stacking M layers of BCAB H , namely, x ( M ) ∈ R C × D c × H and z ( M ) x ∈ R N × D c × H , where D c represents the hidden dimension for each head in BCAB H . Note that the cross attention is computed between the time series and the basis, instead of across time which is frequently found in Transformer based models [7, 13]. Additionally, the attention mechanism is used to allow for flexible associations between the time series and basis vectors. This approach ensures that each time series can selectively attend to the most relevant basis vectors, and likewise, each basis vector can selectively attend to the most relevant time series.

Finally, the Coef module calculates the 'coefficient' of each time series w.r.t. each basis vector as the inner product of their representations x ( M ) and z ( M ) x for each of the H heads, resulting in the coefficient tensor c ∈ R C × N × H .

3.2 Forecast module for aggregation and future prediction

After obtaining the coefficients, we take advantage of them for the sake of forecasting. We begin by projecting the future part of the basis vectors z y into a space where they can be linearly aggregated using the coefficients. As the Coef module computes the coefficients for H heads, the projection of z y should also have H heads to maintain consistency. To this end, we employ a four-layer Multi-Layer Perceptron (MLP) with a bottleneck to map z y ∈ R N × O to ˆ z y ∈ R N × O , which is then split into H heads, each with size N × ( O/H ) , and is denoted as ˜ z y ∈ R N × H × ( O/H ) .

For each head, we aggregate the N basis vectors by computing the coefficients weighted sum of ˜ z y over the dimension of N , that is,

where h ∈ { 1 , · · · , H } denotes the head index, and the size of ˜ y is C × H × ( O/H ) .

Next, we concatenate the H heads together and pass them through another four-layer MLP with a bottleneck, in order to exchange information between different heads. This is because different heads may have captured different aspects of the input sequence and the fusion MLP can help combine the information and improve the overall prediction performance.

It is noteworthy that the bottleneck layers in the above module are used to reduce the dimensionality of the input features before projecting them to a higher-dimensional space. This helps to reduce the computational complexity of the projection operation and prevent overfitting. Additionally, using a bottleneck layer can also assist in extracting more informative features by forcing the model to learn a compressed representation of the input, thus improving the prediction accuracy.

Finally, we compare the predicted values ˆ y with the true values y via a Mean Squared Error (MSE) loss function, that is, L pred = MSE(ˆ y , y ) .

3.3 Basis module for basis learning

In this subsection, we present our approach for learning a data-driven basis in a self-supervised manner. The goal is to obtain a basis that satisfies three essential properties.

First, the relation between the basis vectors and the time series should be consistent across time, such that we can predict the future by combining the future part of the basis using the coefficients obtained from the historical part of the basis and the time series. Specifically, given a time series ( x , y ) and the corresponding basis ( z x , z y ) , the coefficients (i.e., edge strength) between the time series and the basis should be consistent across the historical view ( x , z x ) and the future view ( y , z y ) . In other words, the relevance of a given time series to a particular basis vector in the historical view should be retained in the future view. To achieve this, we pass both ( x , z x ) and ( y , z y ) through the Coef module to get the coefficient tensor respectively for the two views, c x and c y , both with size C × N × H . For each time series, we then perform contrastive learning by regarding the coefficient w.r.t. each basis vector in c x as the anchor point, the coefficient w.r.t. the corresponding basis vector in c y as the positive sample, and the coefficient w.r.t. the remaining basis vectors in c y as the negative sample. We optimize the InfoNCE loss to maximize the mutual information between c x and c y ,

which is given by

where ϵ represents the temperature used to adjust the smoothness of alignment distribution.

In addition, we require the basis to be interpretable, which means that we can gain insights into the underlying patterns captured by the basis. To achieve interpretability, we promote smoothness across time with a regularization term, that is,

where we concatenate the basis vectors for the historical and future views ( z x , z y ) along the last dimension to form z , and the smoothness matrix S ∈ R ( I + O ) × ( I + O -2) can be expressed as:

It is apparent by multiplying z with S , we compute ∥ z [: , t -1] -2 z [: , t ] + z [: , t +1] ∥ 2 2 , which is the curvature over time [23]. One advantage of using S is that the addition of a constant and a linear function of time makes the loss invariant. Therefore, the above smoothness loss can accommodate the change of the overall mean level as well as the linear trend.

Finally, the basis should be a function of the timestamp. As such, we develop a four-layer MLP with a skip connection between the input and the output of the second layer. The input to the network is the normalized timestamp associated with the first time point in the historical window. Suppose that the overall length of a time series in the dataset is T , then the normalized timestamp is defined as τ = t/T , where t ∈ { 0 , · · · , T -1 } . The output of the networks is an N × ( I + O ) tensor, which is the basis for the current time window.

Overall, the loss we optimize can be expressed as:

We find that the performance of BasisFormer is robust to the weights in front of the terms in (9). Therefore, we set the weights to be one in all our experiments. Sensitivity analysis of the weights in the loss function can be found in the Appendix A.4.

4 Experiments

To assess the effectiveness of our model, we conducted comprehensive experiments on six datasets from real-world scenarios, using the experimental setup identical to that in [5, 7-9]. Below we summarize the experimental setup, datasets, models, and compared models.

Experimental setup : The length of the historical input sequence is maintained at 96 (or 36 for the illness dataset), whereas the length of the sequence to be predicted is selected from a range of values, i.e., { 96 , 192 , 336 , 720 } ( { 24 , 36 , 48 , 60 } for the illness dataset). Note that the input length is fixed to be 96 for all methods for a fair comparison.

Datasets : The six datasets used in this study comprise the following: 1) ETT [7], which consists of temperature data of electricity transformers; 2) Electricity, which includes power consumption data of several customers; 3) Exchange [24], containing financial exchange rate within a specific time range; 4) Traffic, comprising data related to road traffic; 5) Weather, which involves various weather indicators; and 6) Illness, consisting of recorded influenza-like illness data. Note that ETT is further divided into four sub-datasets: ETTh1, ETTh2, ETTm1, and ETTm2, and the results in Table 1 are based only on the ETTm2 sub-dataset. The results for the remaining three sub-datasets can be found in the Appendix.

Models for comparison : In this study, we compare our proposed model against the following state-of-the-art models: four transformer-based models, namely FEDformer [9], Autoformer [8], Pyraformer [13]; one MLP-based model, i.e., Dlinear [4]; and one CNN-based model, i.e., TCN [5]. We also consider two recently proposed models, e.g., N-Hits [2] and FiLM [3]. Due to space

Experiment with '-' means it reported an out-of-memory error on a computer with 80G memory.

constraints, we present the results of a selected number of models in this paper, according to their performance and diversity. Interested readers can refer to the supplementary material for a more comprehensive comparison.

4.1 Main results

Multivariate results : The results of the multivariate time series forecasting are presented in Table 1. Very frequently, the proposed BasisFormer outperformed the comparison models across all six datasets, achieving the best results. Furthermore, Basisformer exhibited a 21.79% improvement over the state-of-the-art method Fedformer. We also observed that when compared to recently proposed models, such as FiLM [3] and Dlinear [4], BasisFormer significantly improved the average MSE performance by a large margin of 10.78% and 14.78%, respectively. It is worth noting that BasisFormer is vastly superior to the other methods for the traffic dataset, probably because Traffic dataset is a highly periodic dataset and our model can learn periodic representation well.

Univariate results : The results of the univariate time series forecasting are presented in Table 2. Our model is on par with the state-of-the-art methods. Specifically, compared to the sota method such as FEDformer [9], our proposed model improved the average MSE performance by 15.36%. Compared to the recent models such as FiLM [3] and Dlinear [4], we also achieved better performance with the increment of 1.6% and 16.17% separately. 3

4.2 Ablation studies

Effect of learnable basis : In order to demonstrate the effectiveness of learnable bases, we replaced the learnable basis part in our model with three commonly used types of fixed bases: fixed sine/cosine basis that covers all possible frequencies within the input length, random sine/cosine basis selected in a wide range of frequencies, and covariate embedding which is generally used in Transformerbased models. The results are shown in Table 3. The substitution of the learnable bases resulted in, at minimum, a 5% average decrease in performance. It is pertinent to mention that although

sinusoidal types of bases have good generalization ability, they are inadequate in terms of adaptability for specific data. For covariate embeddings, despite having learnable parameters and containing additional sequence information, they do not account for the distinct correlation between the basis and different time series.

The Impact of multi-head mechanism: We further check the impact of the number of heads in the Coef module on performance. The results are displayed in Table 4. The results show that performance presents an upward trend when the number of heads is increased within a specific range. However, beyond a certain number, further augmentation of heads may result in a decline in performance. Consequently, we set H = 16 in our experiments.

Ablation study of the Basis module: We finally employ ablation experiments to isolate the two different loss functions employed in the Basis module. The results are recorded in Table 5. It can be observed that the InfoNCE loss contributes significantly to the good performance of BasisFormer.

basis number

1

5

10

15

20

Metric

MSE

MAE

MSE

MAE

MSE

MAE

MSE

MAE

MSE

MAE

96

0.173

0.269

0.171

0.265

0.166

0.259

0.168

0.263

0.168

0.263

192

0.183

0.277

0.178

0.270

0.176

0.270

0.176

0.269

0.176

0.268

336

0.196

0.289

0.192

0.284

0.190

0.283

0.192

0.285

0.193

0.285

720

0.231

0.317

0.229

0.314

0.218

0.306

0.220

0.308

0.224

0.311

avg

0.196

0.288

0.192

0.283

0.187

0.279

0.189

0.281

0.190

0.282

0

48

96

144

192

t

0.2

0.1

0.0

0.1

basis 1

z_x

z_y

(a) basis vector 1

0

48

96

144

192

t

0.1

0.0

0.1

0.2

basis 2

z_x

z_y

(b) basis vector 2

The smoothness loss also makes positive contributions. Remarkably, the employment of both loss functions in conjunction yields the most optimal result. Specifically, the InfoNCE loss advances the bases' capability to acquire the representation of time series, while the smoothness loss function mitigates overfitting to noise within the data.

Impact of the number of basis vectors : We present the performance of the proposed model under varying numbers of basis vectors N in Table 6, where N is set to 1, 5, 10, 15, and 20. The results demonstrate that the model's performance remains stable over a wide range of N , indicating its ability to adaptively adjust to the number of basis vectors. Notably, when N increases beyond a certain threshold, some of the basis vectors may become redundant. To further explore this, we visualize a subset of the learned basis vectors when N = 20 in Figure 2. Interestingly, we observe a high cosine similarity of -0 . 93 between two of the bases, suggesting that some basis vectors may not be necessary for accurate prediction. Thus, in practical applications, we set N to 10 for all datasets to reduce computational complexity without compromising performance.

4.3 Other studies

The adaptability of the self-supervised module to other models: To demonstrate the generalizability of our self-supervised basis learning module, we treat the covariate used in the FEDformer [9] and Autoformer [8] as learnable bases and supervise them using our self-supervised framework. It is noteworthy that we only provide supervision to the covariate employed in these models, without introducing any additional parameters in the prediction pathway. The results are presented in Table 7. We can tell that the addition of self-supervision to the covariate exclusively causes a performance improvement of approximately 5%. This phenomenon suggests that the learnable basis provides a more reliable reference for the future than the given covariates.

The interpretability of the bases: In order to elucidate the impact of basis in a more comprehensible manner, we visualize the time series and the corresponding learned basis for the traffic dataset in Figure 3(a) and Figure 3(b). Specifically, we choose 4 time series and basis vectors at random. Upon observation of the figures, several key points can be discerned: Firstly, the traffic data have strong periodic patterns, with roughly four peak values in a sinusoidal shape within a length of 96. The positions of the peaks are not evenly distributed over time though. Correspondingly, the learned basis

baseline

Fedformer

Autoformer

comparison

origin

(+) coef module

origin

(+) coef module

metric

MSE

MAE

MSE

MAE

MSE

MAE

MSE

MAE

96

0.193

0.308

0.183

0.301

0.201

0.317

0.193

0.305

192

0.201

0.315

0.196

0.312

0.222

0.334

0.200

0.312

336

0.214

0.329

0.209

0.324

0.231

0.338

0.211

0.325

720

0.246

0.355

0.229

0.340

0.254

0.361

0.252

0.359

avg

0.214

0.327

0.204

0.319

0.227

0.338

0.214

0.325

0

48

96

144

192

t

1

0

1

2

3

4

series 1

x

y

0

48

96

144

192

t

1

0

1

2

series 2

x

y

0

48

96

144

192

t

1

0

1

2

3

4

series 3

x

y

0

48

96

144

192

t

1.0

0.5

0.0

0.5

1.0

1.5

series 4

x

y

(a) Visualization of time series on the Traffic dataset

0

48

96

144

192

t

0.5

0.0

0.5

1.0

1.5

2.0

basis 1

z_x

z_y

0

48

96

144

192

t

1.0

0.5

0.0

0.5

1.0

1.5

2.0

basis 2

z_x

z_y

0

48

96

144

192

t

1

0

1

2

3

4

5

basis 3

z_x

z_y

0

48

96

144

192

t

1.4

1.2

1.0

0.8

0.6

0.4

basis 4

z_x

z_y

(b) Visualization of learned basis on the Traffic dataset

follows this periodic pattern and effectively capture the salient features of the data. Secondly, the basis vectors given by our approach are smooth, indicating that they are not corrupted by the noise in the data. Note that the noise is not predictable, and so the basis that drives the change in the future is preferred to be smooth. Thirdly, it is evident that the learned bases have varying heights and intervals, thereby providing diversity to characterize different features of the time series. Finally, the obtained bases are consistent from both past and future perspectives, thus facilitating the prediction of future trends based on the coefficient similarity between a time series and the basis in the historical part. More visualized results can be found in the Appendix E.

5 Conclusion

This paper presents BasisFormer, a novel solution to alleviate two significant limitations that impede the efficacy of the existing SOTA methods. Through the utilization of BasisFormer, automatic learning of a self-adjusting basis can be achieved. Moreover, given the learned basis, BasisFormer also allows different time series to be correlated with distinct subsets of basis vectors. Our experimental findings provide compelling evidence of the superiority of BasisFormer over existing methods.

References

Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint arXiv:1905.10437 , 2019.

Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza, Max Mergenthaler, and Artur Dubrawski. N-hits: Neural hierarchical interpolation for time series forecasting. arXiv preprint arXiv:2201.12886 , 2022.

Tian Zhou, Ziqing Ma, Qingsong Wen, Liang Sun, Tao Yao, Wotao Yin, Rong Jin, et al. Film: Frequency improved legendre memory model for long-term time series forecasting. Advances in Neural Information Processing Systems , 35:12677-12690, 2022.

Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? arXiv preprint arXiv:2205.13504 , 2022.

Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271 , 2018.

David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting , 36(3):1181-1191, 2020.

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 11106-11115, 2021.

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems , 34:22419-22430, 2021.

Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. arXiv preprint arXiv:2201.12740 , 2022.

Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Rethinking the stationarity in time series forecasting. arXiv preprint arXiv:2205.14415 , 2022.

Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in neural information processing systems , 32, 2019.

Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451 , 2020.

Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International Conference on Learning Representations , 2021.

Romain Cosentino and Behnaam Aazhang. Learnable group transform for time-series. In International conference on machine learning , pages 2164-2173. PMLR, 2020.

Randall Balestriero, Hervé Glotin, and Richard Baranuik. Interpretable and learnable superresolution time-frequency representation. In Mathematical and Scientific Machine Learning , pages 118-152. PMLR, 2022.

Hendrik Purwins, Bo Li, Tuomas Virtanen, Jan Schlüter, Shuo-Yiin Chang, and Tara Sainath. Deep learning for audio signal processing. IEEE Journal of Selected Topics in Signal Processing , 13(2):206-219, 2019.

Stefan Lattner, Monika Dörfler, and Andreas Arzt. Learning complex basis functions for invariant representations of audio. arXiv preprint arXiv:1907.05982 , 2019.

Wei Fan, Shun Zheng, Xiaohan Yi, Wei Cao, Yanjie Fu, Jiang Bian, and Tie-Yan Liu. Depts: deep expansion learning for periodic time series forecasting. arXiv preprint arXiv:2203.07681 , 2022.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting. arXiv preprint arXiv:2106.14112 , 2021.

Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 8980-8987, 2022.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903 , 2017.

Hang Yu and Justin Dauwels. Modeling spatio-temporal extreme events using graphical models. IEEE Transactions on Signal Processing , 64(5):1101-1116, 2015.

Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval , pages 95-104, 2018.

Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. arXiv preprint arXiv:2202.01575 , 2022.

Patrick Schäfer. The boss is concerned with time series classification in the presence of noise. Data Mining and Knowledge Discovery , 29:1505-1530, 2015.

Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C Tatikonda, Nicha Dvornek, Xenophon Papademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. Advances in neural information processing systems , 33:18795-18806, 2020.

Appendix

A Additional Experiments 4

A.1 Experiments on the ETT datasets

In the main body, we present a comparison of the benchmark methods on the ETTm2 dataset. In this section, we extend our analysis to the remaining three ETT datasets, namely ETTh1, ETTh2, and ETTm1, as summarized in Table 8. Our experimental results reveal that Basisformer outperforms all other methods in terms of MSE and MAE. Specifically, Basisformer demonstrates a superior average MSE reduction of 1.32% , 6.74% and 9.23% when compared to FiLM, Fedformer and DLinear, respectively.

A.2 Experimental results with longer length input setting

Throughout our research, we maintain consistency in our experimental settings by fixing the input length to be 96 (with a reduced input length of 36 for the illness dataset), instead of using a longer length. The main rationale behind this decision is that, in practical scenarios where the model is deployed as an online service and tasked with predicting a long range of the future at a granular level of minutes or hours, collecting a lengthy history (i.e., spanning 720 timestamps) for a large number of time series in real-time can be quite challenging. Therefore, the adoption of an input length of 96 proves to be more practical and feasible.

Given that certain recent methods utilize longer input lengths to yield better performance, irrespective of the length, we present supplementary comparison outcomes with extended input lengths in Table 9. Specifically, Fedformer, Autoformer, and TCN exhibit a decline in performance with an increase in input length, and hence, we retain their original outcomes at an input length of 96. In contrast, Dlinear employs an input length of 336 (104 for the illness dataset) by default, FiLM utilizes an input length that is at most four times of the output length, and N-HiTS adopts an input length that is five times of the output length. To enable a fair comparison, we standardize our input length for longer inputs to 192 (72 for the illness dataset).

The experimental results yield several notable findings. Firstly, those methods that benefit from longer inputs, namely Dlinear, FiLM, and N-HiTS, exhibit a significant performance decline when the input length is reduced from longer settings to an input length of 96. Concretely, Dlinear, FiLM, and N-HiTS show performance declines of 25.82%, 19.48%, and 330.42%, respectively. Conversely, our approach maintains most of its performance with a slight deterioration of 6.23%, as evident in Table 1 and Table 9. Secondly, concerning longer inputs, our method surpasses recent approaches such as Dlinear, FiLM, and N-HiTS, with an average MSE performance improvement of 1.35%, 0.63%, and 7.75%, respectively, and a corresponding evaluation MAE performance improvement of 3.15%, 2.33%, and 4.06%, respectively. It is noteworthy that our approach requires an input length

of 192 (72 for the illness dataset), which is at least 40% lower than the input length of the other three methods. Furthermore, for even longer input lengths, our model's performance can be further enhanced, signifying that our approach can leverage limited data more efficiently.

We simultaneously study the inference time of our model for various input lengths, denoted as I ∈ { 96 , 192 , 336 , 720 } , and output lengths, denoted as O ∈ { 96 , 192 , 336 , 720 } . The results are shown in Table 10. These measurements are based on the "exchange" dataset.

The table above illustrates the average inference time per instance of our algorithm under different configurations, measured in seconds. As observed, our algorithm exhibits notable speed, averaging at the millisecond level. Furthermore, when comparing the increase in output length to the extension of the input length, the additional inference time incurred by augmenting the input length is minimal. This is attributed to the preprocessing step where the input sequence, regardless of its length, is projected into a fixed-length (usually 100) sequence using a linear layer. As a result, extending the input length does not significantly amplify the time consumption in our method.

A.3 Additional abalation study

Impact of the number of the BCAB layers : The ablation study on the number of BCABs is shown in Table 11. The findings indicate that stacking a certain number of BCAB modules can enhance the performance of the model. However, exceeding a certain threshold can lead to overfitting, resulting in a decline in performance. Hence, we recommend the use of two layers of BCABs in practical experiments to achieve optimal performance without overfitting the model.

Impact of the bottleneck in the forecast module: The performance of the proposed model under varying bottleneck settings is presented in Table 12. The results demonstrate that employing a

bottleneck architecture with a width of 48 can significantly reduce the number of model parameters without degrading the performance significantly, as opposed to not using a bottleneck architecture.

A.4 Sensitivity analysis of the weights for the losses in Eq. (9)

Our model utilizes three distinct loss functions: the supervised MSE loss for prediction L pred , the self-supervised InfoNCE loss for basis learning L align , and the smoothness loss for smoothing the basis over time L align . During training, we directly combine these loss functions as the model's performance is not significantly impacted by the relative weights of the individual losses within a certain range. This assertion is supported by the performance evaluation presented in Figure 4, which investigates the impact of different weight combinations of the three loss functions. In our setting, we fix the weight of the predicted loss function to be 1, and then fix either the weight of the contrast loss function or the smoothness loss function to be 1, while the other one varies within the range of { 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 , 1 . 0 , 1 . 2 , 1 . 4 , 1 . 6 } . To explore the inflection point of the effect, we take the middle point of two points and calculate a finer range again. Our results indicate that the contrast loss function is essentially stable between the weight range of 0.6-1.2, while the smoothness loss function is similarly stable between the weight range of 0.9-1.5.

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

weight

0.2200

0.2225

0.2250

0.2275

0.2300

0.2325

0.2350

MSE

Smooth

A.5 Comparison with self-supervised methods

Our proposed method adopts contrastive learning, here we provide comparative experiments with other self-supervised learning method namely Cost [25]. For the experiments, we fix the input length at 96 and vary the output length from 96 to 720. Table 13 and Table 14 illustrate that Basisformer outperforms Cost in terms of both MSE and MAE across all cases. It is important to note that Cost is trained using a two-step process: self-supervised training followed by supervised ridge regression. In contrast, Basisformer trains the basis, coef, and forecast modules end-to-end. This enables Basisformer to learn a basis that is specifically tailored for forecasting, potentially leading to improved performance.

A.6 Comparison with time series discretization

To compare with time series discretization, we have chosen an method, namely Boss [26], which utilizes the Bag-Of-SFA-symbols method for feature extraction.

In order to evaluate the effectiveness of our approach, we have conducted a classification task using several UCR datasets. The description of the datasets is concluded in Table 15:

To adapt our method for classification, we followed these steps:

We partitioned the sequence into past and future parts, uniformly dividing them in a 6:4 ratio for all datasets. Different partitioning methods can be explored in future research to improve the model's performance.

We used a self-supervised approach for training, reserving 10% of the original training data for validating self-supervised performance. The remaining data was used for training, and the selfsupervised loss function included prediction, alignment, and smoothness losses. Early termination based on validation set performance was done with a patience of 3.

From the well-trained self-supervised model, we extracted the aggregation coefficient matrix, specifically from the past perspective. This matrix was flattened to create sequence features, which were then fed into a random forest classifier for final classification. Notably, during self-supervised training, both past and future sequences were used for consistency, but only the past coefficient matrix was utilized in the classifier.

We conducted a fair comparison by extracting features using both Boss and our model, ensuring that our feature parameter count did not exceed Boss's. We employed a random forest classifier with 100 features and a maximum depth of 30 for classification, and the results are summarized in the Table 16.

The applicability of our method to self-supervision relies on predictability and consistency between past and future data. The validation set loss in the table indicates that datasets lacking predictability have high validation losses, posing challenges for loss function optimization. The Phoneme and FaceUCR datasets lack predictability. The Phoneme dataset includes speech segments from different individuals with random content before and after, while the FaceUCR dataset consists of flattened one-dimensional vectors of rotated face images, both lacking inherent predictability. These datasets require a holistic understanding of the entire sequence for meaningful interpretation, and as a result, our proposed Basisformer struggles to extract useful features, leading to lower performance compared to Boss.

On the other hand, datasets like Mallat and Rock, exhibiting predictability and low validation losses, allow our approach to achieve superior performance over Boss. Surprisingly, we achieve this performance using only the representation of the past sequence as input for the classifier.

A.7 Uncertainty of the results

To assess the stability of our proposed method, we performed 5 repeated experiments and calculated the standard deviations for all methods, as presented in Table 17. Notably, our method exhibits a relatively small variance within the table, indicating its high degree of stability.

B Implementation Details

The training and testing of BasisFormer are conducted on an NVIDIA GeForce RTX 3090 graphics card with 24268MB of VRAM. During the trainin process, we adopt the Adabelief optimizer [27] for optimization. We train the model for 30 epochs with the patience of 3 epochs. All experiments are averaged over 5 trials.

To implement the multi-head mechanism, we calculate the multi-head attention for each CAB separately, and then restore it to the original dimension through multiplication, concatenation, and a linear layer. In the last layer of the network, a mapping layer was utilized to map it to H heads, and the dot product outputs the final coefficients.

To promote the learning of bases and ensure consistency of time series across different dimensions, we normalized the time series during training and performed inverse normalization when outputting the results.

For the other models compared in the table, we utilized their original code and conducted experiments by only varying the input length.

C Analysis of the Limitations of BasisFormer

BasisFormer demonstrates proficiency in learning effective representations and capturing the relationship between bases and time series. However, this proficiency is contingent upon the multidimensional time series being on the same feature scale, which necessitates normalization of the time series during training and inverse normalization when outputting results. Despite this, the normalization and inverse normalization operations introduce changes to the original distribution of

Experiment with '-' means it reported an out-of-memory error on a computer with 128G memory.

the time series, making it arduous to fit certain distributions. As such, future work could explore alternative approaches to training on datasets with considerably different feature scales, eliminating the need for normalization and inverse normalization. Possible avenues for investigation include identifying appropriate mathematical methods or neural network transformations to map data to a suitable and universal feature space.

D Relation to Meta-learning

From a meta-learning standpoint, the learnable basis in our model is tantamount to meta-knowledge for all time series within the same window. The coefficients, which are derived from the similarity between each time series and the foundation, represent distinctive knowledge for each time series. Consequently, our model can be perceived as a manifestation of meta-learning. Notwithstanding, we departed from conventional meta-learning approaches by forgoing a two-stage inner-outer loop optimization method, instead opting for an end-to-end training method.

E Additional Visualization

We have incorporated the visualization of the attention map of the BCAB module on the traffic dataset, as depicted in Figure 5. This visualization demonstrates that different time sequences have

distinct attention scores for different the same set of basis vectors. It can also be seen that the attention mechanisms exhibit significant similarities between the past and future sequences. This demonstrates the consistency in the base elements between the two.

(a) attention map in the past

(b) attention map in the future

xattn_map

136

0.25

170

204

238

306

-0.20

340

30

s

442

476

0.15

510

544

578

612

-0.10

646

680

714

748

782

0.05

816-

850

-

0

1

2

3

4

5

6

7

8

の

basis

y attn_map

048

102

0.225

136

170

204

-0.200

238

272

306

-0.175

340

374

series

408

0.150

442

476

510

0.125

544

578

612

-0.100

646

680

714

0.075

748

782

816

0.050

850

0

2

3

4

5

6

7

8

6

basis

Additionally, we have provided a visualization of a specific time sequence alongside the features corresponding to the highest and lowest attention scores, as shown in Figure 6. The highest attention score is 0.2316, while the lowest attention score is 0.03371. Figure R2 highlights that the representation with a total of 8 sets of main peaks (Figure 6(c)) more comprehensively captures the patterns of the data compared to the configuration with only 2-3 main peaks (Figure 6(b)). This indicates a correlation between the attention scores and the relationship between time sequences and features.

It is important to note two key points. Firstly, since bases represent condensed patterns of time sequences, it is unlikely for a base to be identical to any single time sequence, especially when N is small. Secondly, after the bases are processed through multiple linear and nonlinear layers in the network, they correspond to predicted sequences. Therefore, the numerical values of the bases serve as reference points only. The focus should be on the patterns exhibited by the bases.

(a) series

(b) basis with low attention score (c) basis with high attention score

0.15

0.15

1.0

0.10

0.10

0.5

S00

0.05

-000

00

0.00

0.05

0.5

0.10

0.05

0.15

0.10

1.0

0.20

0.15

25

50

75

100

125150175200

0

25

50

75

100

125

150

175200

0

25

50

75

100

125

150

175200

F Analysis of the Model Complexity

Suppose that the input and output length in BasisFormer is I and O respectively when forecasting a single time series. Note that the time and space complexity of BasisFormer are of the same order. Therefore, we refer to both of them as complexity in the sequel.

With regards to the coef module, the complexity is primarily determined by the cross-attention mechanism. Within our approach, BCAB utilizes attention on the channel dimension, and we

encode the time sequence dimension to a specified hidden length D c ≪ O via a linear layer during computation. Consequently, the complexity of this module is O ( N ) , where N is the number of bases - a fixed hyperparameter which is usually not large. In this step, we omit the number of BCAB stacks M , since M is also a fixed hyperparameter. As previously mentioned in Appendix A.3, to limit overfitting, M is typically set to 2 .

The prediction module incorporates two Multilayer Perceptron (MLP) networks, which are employed for separating and concatenating different heads. Both MLP networks have bottlenecks with constant values, and they carry a complexity of O ( O ) . In terms of the aggregation of different base vectors, the complexity also is O ( O ) . Therefore, the cumulative complexity of this module is O ( O ) .

In summary, the total complexity of our model is O ( O ) . Table 18 provides a comparison of the computational complexity among different models, and BasisFormer achieves the lowest complexity among them.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo be effective, a basis must be tailored to the specific set of time series data and exhibit distinct correlation with each time series within the set. As far as we know, the state-of-the-art methods are limited in their ability to satisfy both of these requirements simultaneously. To address this issue, the authors propose BasisFormer, an end-to-end time series forecasting architecture that leverages learnable and interpretable bases. Firstly, the authors acquire bases through adaptive self-supervised learning, which treats the historical and future sections of the time series as two distinct views and employs contrastive learning. Secondly, the authors design a Coef module that calculates the similarity coefficients between the time series and bases in the historical view via bidirectional cross-attention. Finally, the authors present a Forecast module that selects and consolidates the bases in the future view based on the similarity coefficients, resulting in accurate future predictions.\n\nSTRENGTHS:\n1. The authors acquire bases through adaptive self-supervised learning, which treats the historical and future sections of the time series as two distinct views and employs contrastive learning. \n2. The authors design a Coef module that calculates the similarity coefficients between the time series and bases in the historical view via bidirectional cross-attention. \n3. The authors present a Forecast module that selects and consolidates the bases in the future view based on the similarity coefficients, resulting in accurate future predictions.\n\nWEAKNESSES:\n1. The authors claim that this is a self-supervised learning model, and according to the reports of existing studies, the performance should be weaker than that of supervised models. I hope the authors can compare some time series prediction models based on self-supervised learning. Forgive me for being skeptical about the effect at the moment.\n\nminor comment:\n\n2. I would like the authors to be more clear about Eq. 7, why explainability is related to the smooth term, and to use the common l2-norm.\n\nQUESTIONS:\nThe authors addressed all my concerns. So, I have no further comments.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of finding effective bases for time series forecasting models. Current methods are limited in their ability to satisfy the requirements of being tailored to specific time series data and exhibiting distinct correlation with each time series. To tackle this challenge, the authors propose BasisFormer, an end-to-end architecture that leverages learnable and interpretable bases. Bases are obtained through adaptive self-supervised learning, where historical and future time series sections are treated as distinct views and contrastive learning is used. The proposed architecture includes a Coef module that calculates similarity coefficients between time series and bases using bidirectional cross attention, and a Forecast module that selects and consolidates bases for accurate future predictions. Extensive experiments on six datasets demonstrate that BasisFormer outperforms previous methods for both univariate and multivariate forecasting tasks, achieving considerable improvements in performance.\n\nSTRENGTHS:\nThis paper has studied a time series forecasting problem. The paper is well written. The significance of this work lies in addressing the limitations of existing methods in time series forecasting by proposing BasisFormer, an architecture that leverages learnable and interpretable bases. This approach allows for tailored modeling of specific time series data and distinct correlations with each time series, leading to improved forecasting accuracy.\n\nWEAKNESSES:\nHowever, I still have several concerns towards this paper. First, the illustration and motivation for the basis/bases is not very clear. What are the basis in essence? How can you define basis in time series? Are basis really needed for time series forecasting? What good effects can basis learning bring up? How to verify it? Second, in the main contribution, though the authors combine several techniques together, including self-supervised learning, contrastive learning, cross attention, basis selection. However, I don't find the necessity for each component. Third, the experimental results are less convincing. The proposed method adopts contrastive learning, but it does not compare the performance with other self-supervised learning methods, such as [1] [2]. Other works adopt periodic basis are also not discussed in related work [3].\n\n[1] Cost: contrastive learning of disentangled seasonal-trend representations for time series forecasting. ICLR 2022.\n[2] Time-Series Representation Learning via Temporal and Contextual Contrasting. IJCAI 2021.\n[3] DEPTS: Deep Expansion Learning for Periodic Time Series Forecasting. ICLR 2022.\n\nQUESTIONS:\nsee weakness\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper studies basis learning for time series forecasting for which the past and future basis representations are aligned. Contrastive learning is used to build the time series basis and similarity between the past values and basis is used for time series prediction. The experiments on several time series forecasting datasets show improvement, especially for multivariate forecasting, when the input length (history) is short.

STRENGTHS:
- The paper is overall well written
- The idea of using contrastive learning to learn basis is sound 
- The proposed approach improves over baselines especially for multivariate case

WEAKNESSES:
- There is a strong emphasis on the consistency of the representations for past and future but this is directly experimented/evaluated 
- Reproducibility is questionable without source code and scripts to run as the approach composed of several components and steps of training 
- Proposed approach is particularly improves over baselines when the history to predict is shorter, however the motivation/need for a shorter history is not supported

QUESTIONS:
- Basisformer seems to have more advantages over baselines for the multivariate time series forecasting, how do you explain this?
- How models are trained for the longer input sequence? Is it only inference time change?
- Increasing the input length gives more advantage to the baselines. Do you conduct further analysis of the inference time versus performance for different length of input? 
- Do you plan to share the implementation?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-11

comments: SUMMARY:
This paper proposed BasisFormer which is an end-to-end time series forecasting model that leverages learnable and interpretable bases. BasisFormer treats the historical and future sections of the time series as two distinct views and using contrastive learning. By making use of Coef module and Forecast module, the BasisFormer outperforms previous state-of-the-art methods for univariate and multivariate forecasting tasks.

STRENGTHS:
1. Contrastive learning objective is applied for basis learning which guarantees the consistency between the historical and future sections of the time series. And when applying the SSL module to other frameworks, there is a performance improvement of approximately 5%, which suggests the general application of the learnable basis.

2. Based on the experiment results on six datasets, the proposed BasisFormer model outperforms previous SOTA methods on univariate and multivariate forecasting tasks.

3. The network architecture of BasisFormer is carefully designed and well analyzed through ablation studies and model comparison. The paper is well written and easy to understand.

WEAKNESSES:
In 4.3, the author analyzed the interpretability of the learned bases by visualizing the time series and the corresponding learned basis. An additional visualization of the attention distribution in BCAB module can be helpful for checking different weights assigned to each basis in different attention heads, and thus understanding the network behavior.

QUESTIONS:
At this point, I don't have specific questions to ask. The paper is clearly written.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-25

comments: SUMMARY:
The authors of the paper propose a time series forecasting architecture with a self-supervised method for basis learning, called BasisFormer. Their assumption is that the selection of basis for a time series is consistent across both historical and future sections of the time series. They introduce a Coef module that measures the similarity between the time series and the basis in the historical view via bidirectional cross-attention, and a Forecast module that consolidates vectors from the basis in the future view according to the similarity yielded by the Coef module. In their evaluation they demonstrate improvements in forecasting tasks.

STRENGTHS:
1. The paper is well-written with clear explanations of the proposed architecture.

2. The empirical results, including the ones that were presented in the supplementary material, support the original claims across the manuscript.

WEAKNESSES:
1. In order to further validate the claims presented in this work, I would expect seeing another comparison to methods involve discretization of time-series, such as [Moskovitch, R. and Shahar, Y., 2015. Classification of multivariate time series via temporal abstraction and time intervals mining. Knowledge and Information Systems].

QUESTIONS:
1. What happens if bases are not learnable directly from the time-series, but require a domain expert's KB?

LIMITATIONS:
The novelty of this work is limited to cases where bases do exist and are detectable automatically in the input time-series, where in many cases the provided input is shorter than the length required for identifying cases of seasonality, for example.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-27