SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The experimental results yield several notable findings. Firstly, those methods that benefit from longer inputs, namely Dlinear, FiLM, and N-HiTS, exhibit a significant performance decline when the input length is reduced from longer settings to an input length of 96. Concretely, Dlinear, FiLM, and N-HiTS show performance declines of  $25.82\%$ ,  $19.48\%$ , and  $330.42\%$ , respectively. Conversely, our approach maintains most of its performance with a slight deterioration of  $6.23\%$ , as evident in Table 1 and Table 9. Secondly, concerning longer inputs, our method surpasses recent approaches such as Dlinear, FiLM, and N-HiTS, with an average MSE performance improvement of  $1.35\%$ ,  $0.63\%$ , and  $7.75\%$ , respectively, and a corresponding evaluation MAE performance improvement of  $3.15\%$ ,  $2.33\%$ , and  $4.06\%$ , respectively. It is noteworthy that our approach requires an input length
4All the six datasets can be downloaded from https://drive.google.com/drive/folders/1Z0YpTUa82_jCcxIdTmyrOLXQfvaM9vIy?usp  $\equiv$  sharing
Table 9: Multivariate results for six datasets using a longer input length. Lower MSE indicate superior model performance, and the best results are presented in boldface.
Models	Fedformer	Autoformer	N-HiTS	FiLM	Dlinear	TCN	Basisformer
Metric	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
ETT	96	0.203	0.287	0.255	0.339	0.176	0.255	0.165	0.256	0.167	0.260	3.041	1.330	0.185	0.270
192	0.269	0.328	0.281	0.340	0.245	0.305	0.222	0.296	0.224	0.303	3.072	1.339	0.247	0.307
336	0.325	0.366	0.339	0.372	0.295	0.346	0.277	0.333	0.281	0.342	3.105	1.348	0.298	0.341
720	0.421	0.415	0.422	0.419	0.401	0.416	0.371	0.389	0.397	0.421	3.135	1.354	0.381	0.393
electricity	96	0.193	0.308	0.201	0.317	0.147	0.249	0.154	0.267	0.140	0.237	0.985	0.813	0.145	0.245
192	0.201	0.315	0.222	0.334	0.167	0.269	0.164	0.258	0.153	0.249	0.996	0.821	0.165	0.263
336	0.214	0.329	0.231	0.338	0.186	0.290	0.188	0.283	0.169	0.267	1.000	0.824	0.178	0.276
720	0.246	0.355	0.254	0.361	0.243	0.340	0.236	0.332	0.203	0.301	1.438	0.784	0.219	0.310
exchange	96	0.148	0.278	0.197	0.323	0.092	0.211	0.079	0.204	0.081	0.203	3.004	1.432	0.084	0.205
192	0.271	0.380	0.300	0.369	0.208	0.322	0.159	0.292	0.157	0.293	3.048	1.444	0.172	0.298
336	0.460	0.500	0.509	0.524	0.341	0.422	0.270	0.398	0.305	0.414	3.113	1.459	0.303	0.403
720	1.195	0.841	1.447	0.941	0.888	0.723	0.536	0.574	0.643	0.601	3.150	1.458	0.781	0.668
traffic	96	0.587	0.366	0.613	0.388	0.402	0.282	0.416	0.294	0.410	0.282	1.438	0.784	0.403	0.293
192	0.604	0.373	0.616	0.382	0.420	0.297	0.408	0.288	0.423	0.287	1.463	0.794	0.421	0.301
336	0.621	0.383	0.622	0.387	0.448	0.313	0.425	0.298	0.436	0.296	1.479	0.799	0.418	0.298
720	0.626	0.382	0.660	0.408	0.539	0.353	0.520	0.353	0.466	0.315	1.499	0.804	0.464	0.312
weather	96	0.217	0.296	0.266	0.336	0.158	0.195	0.199	0.262	0.176	0.237	0.615	0.589	0.168	0.215
192	0.276	0.336	0.307	0.367	0.211	0.247	0.228	0.288	0.220	0.282	0.629	0.600	0.213	0.257
336	0.339	0.380	0.359	0.395	0.274	0.300	0.267	0.323	0.265	0.319	0.639	0.608	0.263	0.292
720	0.403	0.428	0.419	0.428	0.351	0.353	0.319	0.361	0.323	0.362	0.639	0.610	0.343	0.346
illness	24	3.228	1.260	3.486	1.287	1.862	0.869	1.970	0.875	2.215	1.081	6.624	1.830	1.427	0.778
36	2.679	1.080	3.103	1.148	2.071	0.969	1.982	0.859	1.936	0.963	6.858	1.879	1.464	0.813
48	2.622	1.078	2.669	1.085	2.184	0.999	1.868	0.896	2.130	1.024	6.968	1.892	1.660	0.862
60	2.857	1.157	2.770	1.125	2.507	1.060	2.057	0.929	2.368	1.096	7.127	1.918	1.853	0.917
of 192 (72 for the illness dataset), which is at least  $40\%$  lower than the input length of the other three methods. Furthermore, for even longer input lengths, our model's performance can be further enhanced, signifying that our approach can leverage limited data more efficiently.
We simultaneously study the inference time of our model for various input lengths, denoted as  $I \in \{96, 192, 336, 720\}$ , and output lengths, denoted as  $O \in \{96, 192, 336, 720\}$ . The results are shown in Table 10. These measurements are based on the "exchange" dataset.
Table 10: Inference time for various input/output settings
O=96	O=192	O=336	O=720
I=96	0.000833	0.001211	0.001419	0.00211
I=192	0.000884	0.001285	0.001437	0.002139
I=336	0.000893	0.001338	0.001469	0.002194
I=720	0.000941	0.001364	0.001547	0.002246
The table above illustrates the average inference time per instance of our algorithm under different configurations, measured in seconds. As observed, our algorithm exhibits notable speed, averaging at the millisecond level. Furthermore, when comparing the increase in output length to the extension of the input length, the additional inference time incurred by augmenting the input length is minimal. This is attributed to the preprocessing step where the input sequence, regardless of its length, is projected into a fixed-length (usually 100) sequence using a linear layer. As a result, extending the input length does not significantly amplify the time consumption in our method.
A.3 Additional abalation study
Impact of the number of the BCAB layers: The ablation study on the number of BCABs is shown in Table 11. The findings indicate that stacking a certain number of BCAB modules can enhance the performance of the model. However, exceeding a certain threshold can lead to overfitting, resulting in a decline in performance. Hence, we recommend the use of two layers of BCABs in practical experiments to achieve optimal performance without overfitting the model.
Impact of the bottleneck in the forecast module: The performance of the proposed model under varying bottleneck settings is presented in Table 12. The results demonstrate that employing a
Table 11: The impact of the number of stacked BCAB on the performance of the model. The electricity dataset is employed in this experiment. We present the best results in boldface.
BCAB number	0	1	2	3	4
Metric	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
96	0.186	0.273	0.166	0.260	0.166	0.259	0.168	0.263	0.171	0.266
192	0.187	0.274	0.176	0.270	0.176	0.268	0.176	0.269	0.179	0.272
336	0.208	0.297	0.187	0.280	0.190	0.283	0.190	0.283	0.191	0.284
720	0.244	0.325	0.228	0.313	0.218	0.306	0.234	0.319	0.237	0.319
avg	0.206	0.292	0.189	0.281	0.187	0.279	0.192	0.283	0.194	0.285
bottleneck architecture with a width of 48 can significantly reduce the number of model parameters without degrading the performance significantly, as opposed to not using a bottleneck architecture.
Table 12: The impact of the MLP bottleneck in the forecast module. The electricity dataset is employed in this experiment. Setting the bottleneck dimension to 96 is equivalent to not using a bottleneck since the input length is 96. The best results are highlighted in bold. The second best is underlined.
bottleneck	96	48	32	24
Metric	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
96	0.163	0.257	0.166	0.259	0.172	0.267	0.172	0.269
192	0.172	0.265	0.176	0.268	0.182	0.273	0.186	0.279
336	0.186	0.279	0.190	0.283	0.194	0.286	0.197	0.289
720	0.217	0.305	0.218	0.306	0.230	0.316	0.233	0.317
avg	0.184	0.276	0.187	0.279	0.195	0.286	0.197	0.288
A.4 Sensitivity analysis of the weights for the losses in Eq.(9)
Our model utilizes three distinct loss functions: the supervised MSE loss for prediction  $L_{pred}$ , the self-supervised InfoNCE loss for basis learning  $L_{align}$ , and the smoothness loss for smoothing the basis over time  $L_{align}$ . During training, we directly combine these loss functions as the model's performance is not significantly impacted by the relative weights of the individual losses within a certain range. This assertion is supported by the performance evaluation presented in Figure 4, which investigates the impact of different weight combinations of the three loss functions. In our setting, we fix the weight of the predicted loss function to be 1, and then fix either the weight of the contrast loss function or the smoothness loss function to be 1, while the other one varies within the range of \{0.2, 0.4, 0.6, 0.8, 1.0, 1.2, 1.4, 1.6\}. To explore the inflection point of the effect, we take the middle point of two points and calculate a finer range again. Our results indicate that the contrast loss function is essentially stable between the weight range of 0.6-1.2, while the smoothness loss function is similarly stable between the weight range of 0.9-1.5.
Figure 4: MSE for the testing data as a function of the weight for the smoothness (the red line) and the infoNCE loss(the blue line).
A.5 Comparison with self-supervised methods
Our proposed method adopts contrastive learning, here we provide comparative experiments with other self-supervised learning method namely Cost [25]. For the experiments, we fix the input length at 96 and vary the output length from 96 to 720. Table 13 and Table 14 illustrate that Basisformer outperforms Cost in terms of both MSE and MAE across all cases. It is important to note that Cost is trained using a two-step process: self-supervised training followed by supervised ridge regression. In contrast, Basisformer trains the basis, coef, and forecast modules end-to-end. This enables Basisformer to learn a basis that is specifically tailored for forecasting, potentially leading to improved performance.
Table 13: Multidimensional prediction comparative experiments on three datasets.
Models	Metric	ETT	Electricity	exchange
96	192	336	720	96	192	336	720	96	192	336	720
Basisformer	MSE	0.184	0.248	0.321	0.410	0.165	0.178	0.189	0.223	0.085	0.177	0.336	0.854
MAE	0.266	0.307	0.355	0.404	0.259	0.272	0.282	0.311	0.205	0.299	0.421	0.670
Cost	MSE	0.280	0.480	0.805	1.562	0.199	0.199	0.212	0.246	0.263	0.464	0.833	1.192
MAE	0.375	0.506	0.676	0.955	0.290	0.292	0.307	0.338	0.393	0.521	0.691	0.871
Table 14: Multidimensional prediction comparative experiments on another three datasets.
Models	Metric	traffic	weather	illness
96	192	336	720	96	192	336	720	96	192	336	720
Basisformer	MSE	0.444	0.460	0.471	0.486	0.173	0.223	0.278	0.355	1.550	1.516	1.877	1.878
MAE	0.315	0.316	0.317	0.318	0.214	0.257	0.298	0.347	0.814	0.819	0.907	0.902
Cost	MSE	0.576	0.546	0.555	0.591	0.372	0.528	0.835	1.394	2.330	2.497	2.650	2.829
MAE	0.377	0.359	0.363	0.379	0.415	0.517	0.666	0.894	0.923	0.962	1.032	1.062
A.6 Comparison with time series discretization
To compare with time series discretization, we have chosen an method, namely Boss [26], which utilizes the Bag-Of-SFA-symbols method for feature extraction.
In order to evaluate the effectiveness of our approach, we have conducted a classification task using several UCR datasets. The description of the datasets is concluded in Table 15:
Table 15: The description of selected UCR datasets
Dataset	Train Size	Test Size	Length	Classes	Is_prediction	Description
Mallat	55	2345	1024	8	Y	a simulated dataset
Rock	20	50	2844	4	Y	rock examples from the ASTER spectral library
Phoneme	214	1896	1024	39	N	Each series is extracted from the segmented audio
FaceUCR	200	2050	131	14	N	rotationally aligned version of facial outline
To adapt our method for classification, we followed these steps:
We partitioned the sequence into past and future parts, uniformly dividing them in a 6:4 ratio for all datasets. Different partitioning methods can be explored in future research to improve the model's performance.
We used a self-supervised approach for training, reserving  $10\%$  of the original training data for validating self-supervised performance. The remaining data was used for training, and the self-supervised loss function included prediction, alignment, and smoothness losses. Early termination based on validation set performance was done with a patience of 3.
From the well-trained self-supervised model, we extracted the aggregation coefficient matrix, specifically from the past perspective. This matrix was flattened to create sequence features, which were then fed into a random forest classifier for final classification. Notably, during self-supervised training, both past and future sequences were used for consistency, but only the past coefficient matrix was utilized in the classifier.
We conducted a fair comparison by extracting features using both Boss and our model, ensuring that our feature parameter count did not exceed Boss's. We employed a random forest classifier with 100 features and a maximum depth of 30 for classification, and the results are summarized in the Table 16.
Table 16: The comparison results of BOSS and Basisformer. We present the best results in boldface.
Model	Boss	Basisformer
Metric	acc	acc	valid_loss
Mallat	0.83	0.87	0.12
Rock	0.56	0.72	0.18
Phoneme	0.20	0.07	1.27
FaceUCR	0.68	0.41	1.68
The applicability of our method to self-supervision relies on predictability and consistency between past and future data. The validation set loss in the table indicates that datasets lacking predictability have high validation losses, posing challenges for loss function optimization. The Phoneme and FaceUCR datasets lack predictability. The Phoneme dataset includes speech segments from different individuals with random content before and after, while the FaceUCR dataset consists of flattened one-dimensional vectors of rotated face images, both lacking inherent predictability. These datasets require a holistic understanding of the entire sequence for meaningful interpretation, and as a result, our proposed Basisformer struggles to extract useful features, leading to lower performance compared to Boss.
On the other hand, datasets like Mallat and Rock, exhibiting predictability and low validation losses, allow our approach to achieve superior performance over Boss. Surprisingly, we achieve this performance using only the representation of the past sequence as input for the classifier.
A.7 Uncertainty of the results
To assess the stability of our proposed method, we performed 5 repeated experiments and calculated the standard deviations for all methods, as presented in Table 17. Notably, our method exhibits a relatively small variance within the table, indicating its high degree of stability.
The training and testing of BasisFormer are conducted on an NVIDIA GeForce RTX 3090 graphics card with 24268MB of VRAM. During the trainin process, we adopt the Adbelief optimizer [27] for optimization. We train the model for 30 epochs with the patience of 3 epochs. All experiments are averaged over 5 trials.
To implement the multi-head mechanism, we calculate the multi-head attention for each CAB separately, and then restore it to the original dimension through multiplication, concatenation, and a linear layer. In the last layer of the network, a mapping layer was utilized to map it to  $H$  heads, and the dot product outputs the final coefficients.
To promote the learning of bases and ensure consistency of time series across different dimensions, we normalized the time series during training and performed inverse normalization when outputting the results.
For the other models compared in the table, we utilized their original code and conducted experiments by only varying the input length.
C Analysis of the Limitations of BasisFormer
BasisFormer demonstrates proficiency in learning effective representations and capturing the relationship between bases and time series. However, this proficiency is contingent upon the multidimensional time series being on the same feature scale, which necessitates normalization of the time series during training and inverse normalization when outputting results. Despite this, the normalization and inverse normalization operations introduce changes to the original distribution of
the time series, making it arduous to fit certain distributions. As such, future work could explore alternative approaches to training on datasets with considerably different feature scales, eliminating the need for normalization and inverse normalization. Possible avenues for investigation include identifying appropriate mathematical methods or neural network transformations to map data to a suitable and universal feature space.
D Relation to Meta-learning
From a meta-learning standpoint, the learnable basis in our model is tantamount to meta-knowledge for all time series within the same window. The coefficients, which are derived from the similarity between each time series and the foundation, represent distinctive knowledge for each time series. Consequently, our model can be perceived as a manifestation of meta-learning. Notwithstanding, we departed from conventional meta-learning approaches by forgoing a two-stage inner-outer loop optimization method, instead opting for an end-to-end training method.
E Additional Visualization
We have incorporated the visualization of the attention map of the BCAB module on the traffic dataset, as depicted in Figure 5. This visualization demonstrates that different time sequences have
distinct attention scores for different the same set of basis vectors. It can also be seen that the attention mechanisms exhibit significant similarities between the past and future sequences. This demonstrates the consistency in the base elements between the two.
(a) attention map in the past
(b) attention map in the future
Figure 5: corresponding set of attention maps from past and future perspectives
Additionally, we have provided a visualization of a specific time sequence alongside the features corresponding to the highest and lowest attention scores, as shown in Figure 6. The highest attention score is 0.2316, while the lowest attention score is 0.03371. Figure R2 highlights that the representation with a total of 8 sets of main peaks (Figure 6(c)) more comprehensively captures the patterns of the data compared to the configuration with only 2-3 main peaks (Figure 6(b)). This indicates a correlation between the attention scores and the relationship between time sequences and features.
It is important to note two key points. Firstly, since bases represent condensed patterns of time sequences, it is unlikely for a base to be identical to any single time sequence, especially when  $N$  is small. Secondly, after the bases are processed through multiple linear and nonlinear layers in the network, they correspond to predicted sequences. Therefore, the numerical values of the bases serve as reference points only. The focus should be on the patterns exhibited by the bases.
(a) series
Figure 6: Time series and bases with higher and lower attention scores
(b) basis with low attention score
c) basis with high attention score
F Analysis of the Model Complexity
Suppose that the input and output length in BasisFormer is  $I$  and  $O$  respectively when forecasting a single time series. Note that the time and space complexity of BasisFormer are of the same order. Therefore, we refer to both of them as complexity in the sequel.
With regards to the coef module, the complexity is primarily determined by the cross-attention mechanism. Within our approach, BCAB utilizes attention on the channel dimension, and we
encode the time sequence dimension to a specified hidden length  $D_{c} \ll O$  via a linear layer during computation. Consequently, the complexity of this module is  $\mathcal{O}(N)$ , where  $N$  is the number of bases - a fixed hyperparameter which is usually not large. In this step, we omit the number of BCAB stacks  $M$ , since  $M$  is also a fixed hyperparameter. As previously mentioned in Appendix A.3, to limit overfitting,  $M$  is typically set to 2.
The prediction module incorporates two Multilayer Perceptron (MLP) networks, which are employed for separating and concatenating different heads. Both MLP networks have bottlenecks with constant values, and they carry a complexity of  $\mathcal{O}(O)$ . In terms of the aggregation of different base vectors, the complexity also is  $\mathcal{O}(O)$ . Therefore, the cumulative complexity of this module is  $\mathcal{O}(O)$ .
In summary, the total complexity of our model is  $\mathcal{O}(O)$ . Table 18 provides a comparison of the computational complexity among different models, and BasisFormer achieves the lowest complexity among them.
Table 18: Comparison of computational complexity for different models.
Methods	TIME	MEMORY
Fedformer	O(O)	O(O)
Autoformer	O(O log O)	O(O log O)
N-HiTS	O(O(1-rB)/(1-r)	O(O(1-rB)/(1-r)
FiLM	O(O)	O(O)
Dlinear	O(O)	O(O)
TCN	O(O)	O(O)
LogTrans	O(O log O)	O(O2)
Reformer	O(O log O)	O(O log O)
Informer	O(O log O)	O(O log O)
Basisformer	O(O)	O(O)