SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zelin Ni*
nz15116190@sjtu.edu.cn
Hang Yu*
hyu.hugo@antgroup.com
Shizhan Liu
Shanghai Jiao Tong University Shanghai, China
shanluzuode@sjtu.edu.cn
Jianguo Li
lijg.zero@antgroup.com
Weiyao Lin
wylin@sjtu.edu.cn
Bases have become an integral part of modern deep learning-based models for time series forecasting due to their ability to act as feature extractors or future references. To be effective, a basis must be tailored to the specific set of time series data and exhibit distinct correlation with each time series within the set. However, current state-of-the-art methods are limited in their ability to satisfy both of these requirements simultaneously. To address this challenge, we propose BasisFormer, an end-to-end time series forecasting architecture that leverages learnable and interpretable bases. This architecture comprises three components: First, we acquire bases through adaptive self-supervised learning, which treats the historical and future sections of the time series as two distinct views and employs contrastive learning. Next, we design a Coef module that calculates the similarity coefficients between the time series and bases in the historical view via bidirectional cross-attention. Finally, we present a Forecast module that selects and consolidates the bases in the future view based on the similarity coefficients, resulting in accurate future predictions. Through extensive experiments on six datasets, we demonstrate that BasisFormer outperforms previous state-of-the-art methods by  $11.04\%$  and  $15.78\%$  respectively for univariate and multivariate forecasting tasks. Code is available at: https://github.com/nzl5116190/Basisformer.
Bases, such as trends and seasonalities, are indispensable for time series modeling and forecasting, since they capture the underlying temporal patterns and serve as the key factors driving changes in the data over time. For example, seasonality can capture the regular fluctuations in demand for a product or a service, while trends can reflect the long-term growth or decline of a market or industry. Incorporating these bases into time series models can improve the understanding and prediction of future behaviors. Indeed, all commonly used deep learning models for time series forecasting can be reimagined as basis-driving models. N-BEATS [1], N-HiTS [2], and FiLM [3] all resort to explicit bases, such as Fourier and Legendre basis. More generally, the linear (or convolution) layers in MLPs [4] (or CNNs [5]) can be regarded as implicit bases, as they act as filter banks to decompose the time series. In addition, the covariate embedding (a.k.a. global timestamp embedding) in RNNs [6]
*Equal contribution. This work was done when Zelin Ni was a research intern at Ant Group.
† Corresponding authors.
and Transformers [7-13] is another form of basis, since they provide reference points for predicting future sequences.
To apply bases for time series forecasting, three steps are required. First and foremost, an appropriate basis should be chosen or learned for the set of time series at hand. In practice, the full space of the basis can often be very large, whereas the patterns of the time series in a set are often similar. It is therefore desirable to learn a basis that is tailored to the specific characteristics (e.g., periods or frequencies) of the time series data. This can help reduce the complexity of the forecasting model, as well as make it more accurate and interpretable. Second, each time series in the set is decomposed according to the basis. This involves computing coefficients or weights that determine the similarity or projection energy of the time series w.r.t. (with respect to) each vector (i.e., filter) in the basis. Note that these coefficients are supposed to vary across different time series, since each time series also exhibits unique patterns and characteristics. For instance, the Fourier coefficients corresponding to different time series will be dissimilar. Finally, the prediction is determined by the weighted aggregation of the future part of the basis.
Unfortunately, the aforementioned state-of-the-art methods fall short in satisfying the requirements in the first two steps simultaneously. On one hand, methods relying on classical bases, such as N-BEATS [1], N-HiTS [2], and FiLM [3], often assume that the basis is not learnable and instead learn the coefficients of each time series in a flexible manner when projecting to the basis. However, such a basis may not effectively account for temporal patterns since there is no guarantee that the given basis includes all periods or frequencies corresponding to the set of time series. On the other hand, approaches that aim to adaptively learn the basis from data, such as MLP [4], CNN [5], RNN [6], Transformer and their variants [7-13], often overlook the need for flexible associations between the basis and each individual time series. Specifically, although Transformer and its variants learn the covariate embeddings, they add or concatenate the same embeddings to different time series embedding in a restricted manner. For MLP and CNN, they adopt the same linear and convolution layers for all time series.
To effectively tackle the aforementioned quandary, it is imperative to obtain a basis that can accurately reflect the distinct traits of the dataset, and to devise a predictive network that can selectively utilize relevant vectors in the basis for forecasting purposes. To move forward to this goal, we propose BasisFormer, a time series forecasting architecture with learnable and interpretable basis. As a first step, we acquire the basis through adaptive self-supervised learning from the data. This involves treating the historical and future sections of a time series as two distinct views and employing contrastive learning to learn the basis, assuming that the selection of basis for a time series should be consistent across both views. Subsequently, we design the Coef module that measures the similarity between the time series and the basis in the historical view via bidirectional cross-attention, facilitating the flexible association between individual time series and the basis. Finally, we develop a Forecast module that consolidates vectors from the basis in the future view according to the similarity yielded by the Coef module, leading to accurate future predictions. We emphasize that the above three parts are trained in an end-to-end fashion. In summary, the key contributions of our work comprise:
- We propose a self-supervised method for basis learning by treating the historical and future sections of the time series as two distinct views and employing contrastive learning, which ensures that the selection of basis for a time series is consistent across both views.
- We design the Coef and Forecast module that chooses and merges relevant basis in the future view based on the coefficients that measure the similarity between the time series and the basis in the historical view.
- We conduct extensive experiments on six datasets, and find that our model outperforms previous SOTA methods by  $11.04\%$  for univariate forecasting tasks and  $15.78\%$  for multivariate forecasting tasks.
Time series forecasting models In recent years, deep learning methods have emerged as the predominant technique for time series forecasting. As illustrated in the introduction, these deep learning methods typically resort to bases to facilitate the prediction of the future. Depending on the types of bases used in the network, forecasting models fall into two categories: those using classical orthogonal bases and those using learnable bases. The first group involves N-BEATS [1], N-HiTS [2], and FiLM [3]. N-BEATS and N-HiTs typically utilize the Fourier basis and then learn the coefficients
for this basis in a recursive network such that the basis helps decompose the historical part of the time series into different components and these components can be further aggregated to predict the future. FiLM approximates the historical part via the Legendre polynomial basis and further removes the noise with the help of the Fourier basis. The major drawback with the group of methods is that the basis is predefined, thus giving rise to the problem of which type of basis to choose (e.g., Fourier or Legendre polynomial) and further which vectors in the basis to choose (e.g., which frequency components we choose from the Fourier basis). On the other hand, the models based on learnable bases, such as Dlinear [4], TCN [5], Deeper [6], LogTrans [11], Informer [7], AutoFormer [8], FedFormer [9], etc, use a learnable linear or convolution layer, or covariate embeddings as the basis. Although these bases are adaptable to the time series, the relationship between the basis and the time series is fixed for all time series. For example, the covariate embedding is added or concatenated to the embedding of different time series in the same way, without considering the unique frequencies and periodic patterns of each series. In our paper, we propose a method that allows for both a learnable basis and a flexible correlation between the basis and each time series for more accurate predictions.
Basis learning for time series analysis Apart from time series forecasting, learnable bases have also been explored for other time series-related tasks, such as time series classification. Note that basis learning is distinct from the representation learning of time series, as the former aims to capture the pattern of a set of time series using a common basis, while the latter aims to extract features from individual time series. Moreover, the basis can help extract features from time series as in Dlinear [4] and TCN [5]. Traditionally, a non-learnable basis is exploited, such as Fourier and wavelet basis. However, there exist a handful of works that overcome this limitation and enable the use learnable basis. The learnable group transform [14] generalizes the filter banks in the wavelet transformer and allows nonlinear transformation from the mother wavelet to obtain flexible filter banks that can better extract features from time series. Along this direction, Balestriero et al. [15] propose a learnable Gaussian filter over the Wigner-Ville transform with a few interpretable parameters, and prove that the resulting filter banks can interpolate between classical ones, including Fourier, wavelet, and chirplet basis. Similar works have also been proposed for audio signal processing [16, 17], suggesting that a learnable basis is a more effective feature extractor than a fixed basis. Thus, we utilize a learnable basis in our work and demonstrate its usefulness for time series forecasting. It should be noted that DEPTS [18] tackles the challenges posed by intricate dependencies and multiple periodicities in periodic time series through the implementation of a deep expansion learning framework. However, the complex initialization and optimization strategies employed by DEPTS, as well as its limitation of only being applicable to periodic sequences, have motivated us to develop a simpler and more universally applicable basis learning framework. Concretely, we propose a novel method for basis learning based on self-supervised contrastive learning.
Self-supervised time series representation learning Since we employ self-supervised representation learning techniques to learn the basis, it is necessary to examine related works in this domain. One prevalent method in this field is the contrastive predictive coding (CPC) [19] which implements contrastive learning to obtain time series representations by treating subsequent future time series as positive samples and random non-subsequent future time series as negative samples. Another method, TS-TCC [20], augments the data with two types of perturbations to obtain two views and performs a cross-view prediction task contrastively, similar to CPC. As an alternative, TS2VEC [21] generates positive samples via timestamp masking or random cropping without leveraging the future information. Note that all these methods seek to establish a common representation for a time series from various views. Unlike these approaches, our goal is to preserve the consistency of the relationship between the time series and the basis. In other words, while the representation of the time series in the historical and future view may differ, their relationships with the corresponding basis should remain consistent.
3 BasisFormer
Suppose we have a collection of time series with a dimension of  $C$ , implying that  $C$  correlated time series require simultaneous prediction. Each time series in this set is characterized by its history  $\boldsymbol{x} = (x_{1},\dots ,x_{I})$  and future  $\boldsymbol{y} = (y_{1},\dots ,y_{O})$ , where  $I$  and  $O$  correspond to the input and output sequence lengths, respectively. Our primary objective is to learn a basis  $\boldsymbol{z}$  that can account for the behavior of all time series in the group, and further exploit it for predicting  $\boldsymbol{y}$  given  $\boldsymbol{x}$ . Correspondingly,  $\boldsymbol{z}$  can also be split into the historical component  $\boldsymbol{z}_{x}$  and the future component  $\boldsymbol{z}_{y}$ .
Figure 1: The architecture of BasisFormer, consisting of  $①$  the Coef module,  $②$  the Forcast module, and  $③$  the Basis module. The green and blue lines denote the data flow of the set of time series and basis vector repespectively. The cyan diamond denotes tensor dot product. Note that the dot-dash line, which denotes the data flow of the future part of the time series, is only included during training but removed during inference.
As mentioned in the introduction, three steps are necessary to employ bases for time series forecasting: learning an appropriate basis, computing the coefficient of the time series with respect to each basis vector, and predicting based on a weighted aggregation of the future portion of the basis. The proposed BasisFormer facilitates these three steps in a highly versatile manner. As demonstrated in Figure 1, the overall architecture of BasisFormer also contains three parts:  $①$  the Coef module, which compares a time series to each basis vector to determine the corresponding coefficient;  $②$  the Forecast module, which predicts the future based on the coefficients and the future section of the basis; and  $③$  the Basis module, which learns the basis by aligning the relationships between the basis and the time series from the historical and future perspectives. We will now elaborate on each of the BasisFormer components.
3.1 Coef module for similarity comparison between time series and basis
The Coef Module is designed to measure the similarity between a set of time series and a set of basis vectors. Since our focus is on the relations between the two sets, rather than within each set, we exploit a bipartite graph to represent such relations, where the nodes in one set represent the time series (see green nodes in Figure 1) and the nodes in the other set represent the basis vectors (see blue nodes in Figure 1). As a result, the strength of the edge connecting two nodes in the graph is equivalent to the similarity coefficient between them (see red edges in Figure 1). To obtain the edge strength, we require representations of the nodes in the graph. We accomplish this by developing a bidirectional cross-attention block (BCAB) to learn the node representations via cross-attention, in a similar vein to the graph attention network [22].
As a first step, given two sets of inputs  $\pmb{a}^{(i)}$  and  $\pmb{b}^{(i)}$ , let us define the cross attention block (CAB) as:
$$
\hat {\boldsymbol {a}} = \operatorname {L a y e r N o r m} \left(\mathrm {M A} _ {H} \left(\boldsymbol {a} ^ {(i)}, \boldsymbol {b} ^ {(i)}, \boldsymbol {b} ^ {(i)}\right) + \boldsymbol {a} ^ {(i)}\right), \tag {1}
$$
$$
\boldsymbol {a} ^ {(i + 1)} = \operatorname {C A B} _ {H} \left(\boldsymbol {a} ^ {(i)}, \boldsymbol {b} ^ {(i)}\right) = \text {L a y e r N o r m} (\operatorname {F F N} (\hat {\boldsymbol {a}}) + \hat {\boldsymbol {a}}), \tag {2}
$$
where  $\mathrm{MA}_H$  denotes multihead attention with  $H$  heads whose query is given by  $\pmb{Q} = \pmb{W}_{q}\pmb{a}$ , key by  $\pmb{K} = \pmb{W}_{k}\pmb{b}$ , and value by  $\pmb{V} = \pmb{W}_{v}\pmb{b}$ . By exchanging information mutually between  $\pmb{a}^{(i)}$  and  $\pmb{b}^{(i)}$ , we can construct the BCAB as:
$$
\left(\boldsymbol {a} ^ {(i + 1)}, \boldsymbol {b} ^ {(i + 1)}\right) = \operatorname {B C A B} _ {H} \left(\boldsymbol {a} ^ {(i)}, \boldsymbol {b} ^ {(i)}\right), \tag {3}
$$
$$
\boldsymbol {a} ^ {(i + 1)} = \operatorname {C A B} _ {H} \left(\boldsymbol {a} ^ {(i)}, \boldsymbol {b} ^ {(i)}\right), \quad \boldsymbol {b} ^ {(i + 1)} = \operatorname {C A B} _ {H} \left(\boldsymbol {b} ^ {(i)}, \boldsymbol {a} ^ {(i)}\right). \tag {4}
$$
Note that the parameters in  $\mathrm{CAB}_H$  for computing  $a^{(i + 1)}$  and  $b^{(i + 1)}$  may differ, so as to capture the heterogeneity in the relations from  $a^{(i)}$  to  $b^{(i)}$  and from  $b^{(i)}$  to  $a^{(i)}$ .
Correspondingly, given  $C$  time series  $\pmb{x} \in \mathbb{R}^{C \times I}$  and the basis  $\pmb{z}_x \in \mathbb{R}^{N \times I}$  of size  $N$ , we can get their representations by stacking  $M$  layers of  $\mathrm{BCAB}_H$ , namely,  $\pmb{x}^{(M)} \in \mathbb{R}^{C \times D_c \times H}$  and  $\pmb{z}_x^{(M)} \in \mathbb{R}^{N \times D_c \times H}$ , where  $D_c$  represents the hidden dimension for each head in  $\mathrm{BCAB}_H$ . Note that the cross attention is computed between the time series and the basis, instead of across time which is frequently found in Transformer based models [7, 13]. Additionally, the attention mechanism is used to allow for flexible associations between the time series and basis vectors. This approach ensures that each time series can selectively attend to the most relevant basis vectors, and likewise, each basis vector can selectively attend to the most relevant time series.
Finally, the Coef module calculates the "coefficient" of each time series w.r.t. each basis vector as the inner product of their representations  $x^{(M)}$  and  $z_{x}^{(M)}$  for each of the  $H$  heads, resulting in the coefficient tensor  $\mathbf{c} \in R^{C \times N \times H}$ .
3.2 Forecast module for aggregation and future prediction
After obtaining the coefficients, we take advantage of them for the sake of forecasting. We begin by projecting the future part of the basis vectors  $\boldsymbol{z}_y$  into a space where they can be linearly aggregated using the coefficients. As the Coef module computes the coefficients for  $H$  heads, the projection of  $\boldsymbol{z}_y$  should also have  $H$  heads to maintain consistency. To this end, we employ a four-layer Multi-Layer Perceptron (MLP) with a bottleneck to map  $\boldsymbol{z}_y \in \mathbb{R}^{N \times O}$  to  $\hat{\boldsymbol{z}}_y \in \mathbb{R}^{N \times O}$ , which is then split into  $H$  heads, each with size  $N \times (O / H)$ , and is denoted as  $\tilde{\boldsymbol{z}}_y \in \mathbb{R}^{N \times H \times (O / H)}$ .
For each head, we aggregate the  $N$  basis vectors by computing the coefficients weighted sum of  $\tilde{z}_y$  over the dimension of  $N$ , that is,
$$
\tilde {\mathbf {y}} [ i, h ] = \sum_ {j = 1} ^ {N} \mathbf {c} [ i, j, h ] \tilde {\mathbf {z}} _ {y} [ j, h,: ], \tag {5}
$$
where  $h\in \{1,\dots ,H\}$  denotes the head index, and the size of  $\tilde{\pmb{y}}$  is  $C\times H\times (O / H)$
Next, we concatenate the  $H$  heads together and pass them through another four-layer MLP with a bottleneck, in order to exchange information between different heads. This is because different heads may have captured different aspects of the input sequence and the fusion MLP can help combine the information and improve the overall prediction performance.
It is noteworthy that the bottleneck layers in the above module are used to reduce the dimensionality of the input features before projecting them to a higher-dimensional space. This helps to reduce the computational complexity of the projection operation and prevent overfitting. Additionally, using a bottleneck layer can also assist in extracting more informative features by forcing the model to learn a compressed representation of the input, thus improving the prediction accuracy.
Finally, we compare the predicted values  $\hat{\pmb{y}}$  with the true values  $\pmb{y}$  via a Mean Squared Error (MSE) loss function, that is,  $L_{\mathrm{pred}} = \mathrm{MSE}(\hat{\pmb{y}},\pmb {y})$
3.3 Basis module for basis learning
In this subsection, we present our approach for learning a data-driven basis in a self-supervised manner. The goal is to obtain a basis that satisfies three essential properties.
First, the relation between the basis vectors and the time series should be consistent across time, such that we can predict the future by combining the future part of the basis using the coefficients obtained from the historical part of the basis and the time series. Specifically, given a time series  $(\pmb{x},\pmb{y})$  and the corresponding basis  $(z_{x},z_{y})$ , the coefficients (i.e., edge strength) between the time series and the basis should be consistent across the historical view  $(\pmb{x},\pmb{z}_{x})$  and the future view  $(\pmb{y},\pmb{z}_{y})$ . In other words, the relevance of a given time series to a particular basis vector in the historical view should be retained in the future view. To achieve this, we pass both  $(\pmb{x},\pmb{z}_{x})$  and  $(\pmb{y},\pmb{z}_{y})$  through the Coef module to get the coefficient tensor respectively for the two views,  $c_{x}$  and  $c_{y}$ , both with size  $C\times N\times H$ . For each time series, we then perform contrastive learning by regarding the coefficient w.r.t. each basis vector in  $c_{x}$  as the anchor point, the coefficient w.r.t. the corresponding basis vector in  $c_{y}$  as the positive sample, and the coefficient w.r.t. the remaining basis vectors in  $c_{y}$  as the negative sample. We optimize the InfoNCE loss to maximize the mutual information between  $c_{x}$  and  $c_{y}$ ,
which is given by
$$
L _ {\text {a l i g n}} = - \frac {1}{C N} \sum_ {i = 1} ^ {C} \sum_ {j = 1} ^ {N} \log \frac {\exp \left(\boldsymbol {c} _ {x} [ i , j , : ] \cdot \boldsymbol {c} _ {y} [ i , j , : ] / \epsilon\right)}{\sum_ {k = 1} ^ {N} \exp \left(\boldsymbol {c} _ {x} [ i , j , : ] \cdot \boldsymbol {c} _ {y} [ i , k , : ] / \epsilon\right)}. \tag {6}
$$
where  $\epsilon$  represents the temperature used to adjust the smoothness of alignment distribution.
In addition, we require the basis to be interpretable, which means that we can gain insights into the underlying patterns captured by the basis. To achieve interpretability, we promote smoothness across time with a regularization term, that is,
$$
L _ {\text {s m o o t h}} = \left\| z S \right\| _ {2} ^ {2}, \tag {7}
$$
where we concatenate the basis vectors for the historical and future views  $(z_{x},z_{y})$  along the last dimension to form  $z$ , and the smoothness matrix  $S\in \mathbb{R}^{(I + O)\times (I + O - 2)}$  can be expressed as:
$$
\boldsymbol {S} = \left[ \begin{array}{c c c c c} 1 & - 2 & 1 & & \\ & \ddots & \ddots & \ddots & \\ & & 1 & - 2 & 1 \end{array} \right]. \tag {8}
$$
It is apparent by multiplying  $z$  with  $S$ , we compute  $\| z[(:,t - 1] - 2z[(:,t] + z[(:,t + 1]\| _2^2$ , which is the curvature over time [23]. One advantage of using  $S$  is that the addition of a constant and a linear function of time makes the loss invariant. Therefore, the above smoothness loss can accommodate the change of the overall mean level as well as the linear trend.
Finally, the basis should be a function of the timestamp. As such, we develop a four-layer MLP with a skip connection between the input and the output of the second layer. The input to the network is the normalized timestamp associated with the first time point in the historical window. Suppose that the overall length of a time series in the dataset is  $T$ , then the normalized timestamp is defined as  $\tau = t / T$ , where  $t \in \{0, \dots, T - 1\}$ . The output of the networks is an  $N \times (I + O)$  tensor, which is the basis for the current time window.
Overall, the loss we optimize can be expressed as: