Dateformer: Transformer Extends Look-back Horizon to Predict Longer-term Time Series
Authors: Julong Young
Junhui Chen
Feihu Huang
Jian Peng
Venue: Submitted to ICLR 2023 (2023)
Keywords: Time-series forecasting
Long-term forecasting
Transformer
Time-modeling method

Abstract: Transformers have demonstrated impressive strength in long-term series forecasting. Existing prediction research mostly focused on mapping past short sub-series (lookback window) to future series (forecast window). The longer training dataset time series will be discarded, once training is completed. Models can merely rely on lookback window information for inference, which impedes models from analyzing time series from a global perspective.  And these windows used by Transformers are quite narrow because they must model each time-step therein. Under this point-wise processing style, broadening windows will rapidly exhaust their model capacity. This, for fine-grained time series, leads to a bottleneck in information input and prediction output, which is mortal to long-term series forecasting. To overcome the barrier, we propose a brand-new methodology to use Transformer for time series prediction. Specifically, we split time series into patches by day and reform point-wise to patch-wise processing, which considerably enhances the information input and output of Transformers. To further help models leverage the whole training set's global information during inference, we distill the information, store it in time representations, and replace series with time representations as the main modeling entities. Our designed time-modeling Transformer---Dateformer yields state-of-the-art accuracy on 7 real-world datasets with a 33.6% relative improvement and extends the maximum forecast range to half-year.

Under review as a conference paper at ICLR 2023
DATEFORMER: TRANSFORMER EXTENDS LOOK-BACK HORIZON TO PREDICT LENGER-TERM TIME SERIES
Anonymous authors Paper under double-blind review
ABSTRACT
Transformers have demonstrated impressive strength in long-term series forecast- ing. Existing prediction research mostly focused on mapping past short sub-series (lookback window) to future series (forecast window). The longer training dataset time series will be discarded, once training is completed. Models can merely rely on lookback window information for inference, which impedes models from ana- lyzing time series from a global perspective. And these windows used by Trans- formers are quite narrow because they must model each time-step therein. Under this point-wise processing style, broadening windows will rapidly exhaust their model capacity. This, for fine-grained time series, leads a bottleneck in informa- tion input and prediction output, which is mortal to long-term series forecasting. To overcome the barrier, we propose brand-new methodology to utilize Trans- former for time series forecasting. Specifically, we split time series into patches by day and reform point- wise to patch-wise processing, which considerably en- hances the information input and output of Transformers. To further help models leverage the whole training set's global information during inference, we distill the information, store it in time representations, and replace series with time represen- tations as the main modeling entities. Our designed time-modeling Transformer- Dateformer yields state-of oftt----thhtrr-eeeeeee art accuracy on 7 weal-world datasets with a 33.6% relative improvement and extends the maximum forecast range to half-year
INTRODUCTION
Time series forecasting is critical demand across many application domains, such as energy con- sumption, economics planning. traffic and weather prediction. This task can be roughly summed up as predicting future time series by observing their past values. In this paper, we study long-term forecasting that involves longer-range forecast horizon than regular time series forecasting.
Logically, historical observations are always available. But most models (including various Trans- formers) infer the future by analyzing the part of past sub-series closest to the present. Longer historical series is merely used to train model. For short-term forecasting that more concerns series local (or call short-term) pattern, the closest sub-series carried information is enough. But not for long-term forecasting that requires models to grasp time series' global pattern: overall trend, long- term seasonality, etc. Methods that only observe the recent sub-series can't accurately distinguish the 2 patterns and hence produce sub-optimal predictions (see Figure 1a models observe an obvious upward trend in the zoom-in window. But zoom out, we know that's ' yearly seasonality. And we can see a slightly overall upward trend between the 2 years power load series). However, it's imprac- ticable to thoughtlessly input entire training set series as lookback window. Not only is no model yet can tackle such a lengthy series, but learning dependence from therein is also tough. Thus, we ask: how to enable models to inexpensively use the global information in training set during inference?
In addition, the throughput of Transformers (Zhou et al. 2022 Liu al.) 2021 Wu et al. 2021 Zhou et al. 2021] Kitaev et al., 2020 Vaswani et al., 2017), which show the best performance in long-term forecasting, is relatively limited, especially for fine-grained time series (e.g., recorded per 15 min, half-hour, hour). Given common time series recorded every 15 minutes (96 time-steps per day). with 24GB memory, they mostly fail to predict next month from 3 past months of series, even
"oode will he released

---

Under review as a conference paper at ICLR 2023
they have struggled to reduce self-attention's 's computational complexity. They still can't 't afford such a length of series and thus cut down lookback window to trade off a flimsy prediction. If they are requested to predict 3 months. how do respond? These demands are quite frequent and important in many application fields. For fine-grained series, we argue, it has reached the bottleneck to extend the forecast horizon through improving self-attention to be more efficient. So, in addition to modifying self-attention, how to enhance the time series information imput and output of Transformers?
We study the second question first. Prior works process time series in a point-wise style: each time-step in time series will be modeled in- dividually. For Transformers, each time-step value is mapped to a token and then calcu- lated. This style wears out the models that en- deavor to predict fine-grained time series over longer-term, yet has never been questioned. In fact, similar to images (He et al. 2022). many time series are natural signals with temporal information redundancy- e.g., a missing time- step value can be interpolated from neighbor- ing time-steps. The finer time series' granular- ity, the higher their redundancy and the more accurate interpolations. Therefore, the point- wise style is information-iinflicceett and waste- ful. In order to improve the information density of fine-gre grained time series, we split them into patches and reform the point-wise to patch-wise processing, which considerably reduces tokens and enhances information efficiency. To maintain fix the patch size as
33303
(a) Two Year Load
(b) Full Day Load
Figure 1: (a) depicts two-year power load of an
area. (b) illustrates the area's full day load on Jan 2, 2012, a week ago, a week later, and a year ago: compared to week ago or later, the power load series of Jan 2, 2012, is closer to a year ago, which indicates the day's 's time semantics is altered to closer to a year ago but further away from a week ago or later.
equivalent token information across time series with different granularity, day. We choose the "day" as patch size because it's moderate, close to people's 's habits, and conve- nient for modeling. Other patch sizes are also practicable, we discuss the detail in Appendi
Nevertheless, splitting time series into patches is not a silver bullet for the first question. Even if do so, the whole training set patches series is still too lengthy. And we just want the global information therein, for this purpose to model the whole series is not a good deal. Time is one of the most important properties of time series and plenty of series characteristics are determined by or affected by it. Can time be used as a container to store time series' global information? For the whole historical series, time is a general feature that's very appropriate to model therein persistent temporal patterns. Driven by this, we try to distill training set's global information into time representations and further substitute time series with these time representations as the main modeling entities. But how to represent time? In Section 2 we also provide a superior method to represent time.
In this work, we challenge using merely vanilla Transformers to predict long-term series. We base vanilla Transformers to design brand-new forecast framework named Dateformer, it regards day as time series atomic unit, which remarkable reduces Transformer tokens and hence improves series information input and output, particularly for fine-grained time series. This also benefits Autore- gressive prediction: less error accumulation and faster reasoning. Besides, to better tap training set's s global information, we distill it, store it in the container of time representations, and take these time representations as main modeling entities for Transformers. Dateformer achieves the state-offthe art performance on benchmark datasets. Our main contributions are summarized as follows:
We analyze information characteristics of time series and propose splitting time series into patches. This considerably reduces tokens and improves series information input and out- put thereby enabling vanilla Transformers to tackle long-term series forecasting problems.
To better tap training set's global information. we use time representations as containers to distill it and take time as main modeling object. Accordingly, we design the first time- modeling time series forecast framework exploiting vanilla Transformers Dateformer.
As the preliminary work, we also provide a superior rene-represnnting method to support the time-modeling strategy, please see section 2 for details
"Related works in Appendix B
2

---

Under review as a conference paper at ICLR 2023
2 GE
To facilitate distilling training set's global information, we should establish appropriate time repre- sentations as the container to distill and store day, so we mostly study a special case of it-how represent a date? Informer (Zhou et al. 2021 provides a time-encoding that embeds time by stacking several time features into a vector. These time features contain rich time semanties and hence can represent time appropriately. We follow that and collect more date fcatures by a set simple algorithms to generate our date-cmbeddingg
this paper, we split time series into patches by
But this date-embedding is static. In practice, people pay different attention to various date features and this attention will be dynamically adjusted as the date context change. At ordinary times, people more concerned about what day of week month is today. When approaching important fes- tivals, the attention may be shifted to other date features. For example, few people care that Jan 2, 2012, a Monday because the day before it is New Year's Day (see Figure for an example). This semantics shifting is similar to the polysemy in NLP field, we call it date polysemy. It reminds us: superior reteesepprnnnnnioonn should consider the contextual information of date. However, the date axis is extended infinitely. The contextual boundaries of date are open, which is distinct from words that are usually located sentences and hence have comparatively closed context. Intuitively, distant dates exert a weak influence on today, so it makes sense to select nearby dates as context. In- spired by this, to encode dynamic date-representatioons, we introduce Date Encoder Representations from Transformers (DERT), a convolution-style BERT (Devlin et al. 2019).
Mean correlation) predict
ropresenaaton
Transformer Encoder
N
Linear Layer
DERT
FFN
Norm
Transformer Encoder
slide
Multi-Head Attention
Embed through Linear Project
Norm
25
Dec 26
Dec 27
Dec 28
Dec 29
Dec 30
bec 31
Jan 2
Jan 3
Jen 5
Jan 6
Jan 8
Jan 9
10
Jan 11
Positional
Jon
Jon
Jon
Encoding
Embedded
Figure 2: Date Encoder Representations from Transformers. To encoding the Jan 2. DERT selects date-embeddinggs of some days before and after it as the contextual padding (the green block).
Concretely, DERT slides a fixed window Transformer encoder on the date axis to select date- embeddings input. The input window contains date-emheddings of: some days before the target day (pre-days padding), the target day to be represented, and some days after the target day (post- days padding). After encoding, tokens of the pre-days and post-days paddings will be discarded, only the target day's 's token is picked out as the corresponding dynamic date-representation We design tasks that implemented by linear layers pre-train DERT.
Mean value prediction Predicting mean values of time series patches is direct and effective pre-training task, we require DERT to predict the target day's time series mean value by the corre- sponding date-representation. This task will lose time series patches' intraday information, design the latter task to mitigate the problem.
Auto-correlation prediction We observed that many time series display varied intraday trend and periodicity on different kinds of days. Like more steady holidays, or on the contrary. To leverage it, we utilize the circular auto-corree cuto-coorrlation in stochastic process theory (Chatfield 2003 Papoulis & Wiener-Khinchin Khinchin theorem (Wiener 1930).
Pillai 2002) to assess sequences' stability. According Wiener-
see Appendix C for details
3

---

Under review as a conference paper at ICLR 2023
series auto-correlation be calculated by Fast Fourier Transforms. Thus, Score the stability of the target day's 's time series patch A' with G-length by following equations:
Xie Xee -2mmm dt { iee- dt
Sxx(f) F(X)F (2))=
F (Sxx(()
--00 Sxx( ( ds
Rxx(T)=
(I)
NgNorm
V2i + +22 .... +.6
G-1 Σ Rxx(()
Score = Score=Score(X)) = G
7=0
where F denotes Fast Fourier Transforms, and F- denote the conjugate and inverse Transforms respectively. We ask DERT predict Score of the target day's time series patch by the day's date-representation.
We use the whole training set's time series patches to pre-train DERT, each day's time series patch is training sample and the loss is calculated by:
Pre-training
Pretraining Loss = MSF(linearLayer (d), Mean) + MISE(iineaLLayergcC
where denotes the target day's (patch's) s) dynamic date-representation that produced by DERT, Mean denotes true time series mean value of the day, and Score is calculated by equation 1 Our DERT relies on supervised pre-training tasks, so it's series-specified -iffffeeddiffereett different time series datasets
need to re-train respective DERT. Note that the 2 tasks are merely employed for pre-training DERT, so they') Il be removed after pre- pre-training and not participate in the downstream forecasting task.
DATEFORMER
We believe that time series fuse 2 patterns of components: global and local pattern. The global pattern derived from time series' inherent characteristics and hence doesn't fluctuate violently, while the local pattern is caused by some accidental factors such as sudden changes in the weather, so it's not long-lasting. As aforementioned, the global pattern should be grasped from entire training set series, and the local pattern can be captured from lookback window series. Based on this, we design Dateformer, it distills training set's global information into bute-representations, and then leverages the information produce global prediction. The lookback window used provide local pattern information, and then contributes local prediction. Dateformer fuses the 2 predictions into a final prediction. To enhance Transformer's throughput, we split time series into patches by day, and apply Dateformer in patch-wise processing style. Therefore, we deal with time series forecasting problems at the preedictnn next H days time series from historical series.
3.1 GLOBAL PREDICTION
We distill the whole training set's global information and store it in date-repressntttiions. Then, during inference, we can draw a global prediction from therein to represent learned global pattern.
Date Representation {d} Positional Encodings 2,, G} b2, ... ,l,) = Duplicatee( (d) + Positional Encodings
(d, .... 11,2,3 ... ,G}
copies
Global Prediction t2, .t)
Predictive Network As equations 2 given a day's (patch's) dynamic date-representation d, we duplicate it to G copies (G denotes series time-steps number every day), and add sequential posi- tional encodings to these copies, thereby obtaining finer time-representations {t), t2, t.....,......................... to) to represent each series time-step of the day. Then, to get the day's lay's global prediction, we employ position-win wise feed-forward network to map these time-representations into time series.
We use the canonical Positional Encoding proposed by Vaswani et al. 2017) other PFs also practicable.
4

---

Under review as a conference paper at ICLR 2023
Distilling Initially, the containers of these rinerrepresentations empty and can't represent time series' global patter.. So, before formally training the entire Dateformer, we separately train the global predictive network training set, to distill therein global information thereby preserving the whole training set time series' global pattern. Each day's s time series patch in training set is sam- ple. Time features sufficiently general for the whole historical series, so they won't distill local pattern's s ad-hoc information. For some datasets, end-to-end training Dateformer is also workable
3.2 LOCAL PREDICTION
The local pattern information volatile and hence can merely be available from recent observa- tions. To better capture the local pattern from lookback window, we eliminate the learned global pattern from lookback window series. As described in the previous section, for a day in look- back window, we can get the day's global prediction from its date-representation. Then, the day's Series Residual r carried only local pattern information will be produced by:
Series Residual = Series - Global Prediction
(3)
where Series denote the day's ground-truth time series. We apply the operation all days look- back window, to produce the Series Residuals (21. of the whole lookback window. Sub- sequently, we utilize a vanilla Transformer to learn local prediction from these Series Residuals.
W1 WwWaWa
Residuals Aggregation
Residuals Aggregation
FFN
SoftMax
26
Dac 27
Dec 28
29
Dac 30
Dec 31
Jan 2
Duc 26
Series
W1
Dec 27
Serien
Transformer Encoder
Transformer Decoder
W2
Local Prediction
Dec 28
Serien
Jegr
Linear Project
Embed through Linear Project
W3
Dec. 28
DBc 29
Dec 30
Decc 31
26
Dec 27
Dec 28
Doc 29
Dec 30
Dec St
Jan 2
Dec
Dec
saries
Jan
W7
Series Residuals
Date-representtaioos
Figure 3: Take Jan 2 as an example: to get the day's local prediction, we utilize Transformer to learn series residuals similarities W; between Jan and the lookback window (left), and use the similarities to aggregate Series Residuals of lookback window into the local prediction (right).
Concretely, as shown in Figure we feed the lookback window Series Residuals into Trans- former encoder. For multivariate time series, pre-flattening series is required. The encoder output will be sent into Transformer decoder as a cross information to help decoder refine input date- representations. To learn pair-wise similarities of the forecast day and each day in lookback window, the decoder eats date-reppessntatiinn of lookback window and forecast day. to exchange information between them. Given lookback window of P days, the similarities are calculated by:
Series Residuals = {rı, r2, rP} Date Representations {di de.... dp, dr+1)
[dr, Jz.... ... dp dp41} = Decoder Date Representations, Encoder(Seriee, Residuals))
(4)
Similarilies (W,, .2,... Wr} = SoINMax J., ,dr))
then, we can get a local prediction corresponding d by Aggregating Series Residuals:
Local Prediction Aggrogate(Series Residuals)
(5)
W, x ri + W2 x r2 +.. Wp TP
Wee discuss it in Appendix
5

---

Under review as a conference paper at ICLR 2023
3.3 FINAL PREDICTION
Now, we can obtain a final prediction by adding the global prediction and local prediction:
Final Prediction = Global Prediction + Local Prediction
(6)
For multi-day forecast, we need to encode multi-day date-representatioos and produce their corre- sponding global predictions and local predictions. Thus, we design Dateformer automatically conduct these procedures and fuse them into the final predictions, just like scheduler.
20
Dec 50
Dec S
input time-series patches
2
5
Dec
J Jon
Jon
series residuals
Global Predictions
Local
Global Predictions
Prodictions
Dec 28
30
31
2
Dec
Tor
J
20n
Jon'
Jon Jan
Final Predictions
Date Represents
bec 30
31
duplicate and PE
Des
Jon
Dec
Jan
Jan
Jan
Transformer
3
Transformer
4
DERT
Slide
Doc Dec
Jon
Encoder
Encoder
Date Represents
Prediction
Local
Earlier Residual
Local Prediction
pre-days
padding
lookback
window
forecast
window
past-days padding
DWC
Jan
Jan
Transformer
Decoder
2
Jan
29
Dec
DEL
Ton
Tanz
Transformer
Decoder
input static date-embeddding
Figure 4:
Work flow of Dateformer (4-predict-2 days case), 2 pre(post)-days padding- ① sliding DERT to encode multi-day date-represennatiions and building their global predictions by a single
feed-forward propagation; 2: making multi-day series residuals of lookback window; 3 Au-
toregressive local predictions; a): fusing the global and local predictions into the final predictions.
Referring to Figure for multi-day forecast, we input into Dateformer: i) static date-embeddings pre-days padding, lookback window. forecast window, and post-days padding to encode multi-day date-representationn;; ii) lookback window series to provide local information supporting the local prediction. Dateformer slides DERT encode dynamic cetteereressntttoonns of lookback and fore- cast window. Then, the global predictions can be produced by a single feed-forward propagation.
For the local predictions, Dateformer recursively calls the Transformer day-wise predict. Autore- gression is applied in the procedure-the previous prediction is used for subsequent predictions:
Series Residuals {r,, ,rp
Local Prediction Tp-1 Sngregate(Serice Residuals)
Series Residuals (ro,Ti, r2, .TP, TP-1
Local Prediction P--2 = Aegregate(Series Residuals)
To retard error accumulation, we adopt 3 tricks: 1) Dateformer employs Autoregression for only local predictions, the global predictions with most numerical scale of time series are generated in single feed -orwwrd propagation. which restricts the error's scale; 2) Dateformer regards day time series basic unit. For fine grained time series, the day-wise Autoregression remarkable reduces errors propagation times and accelerates prediction; 3) During local prediction Autoregression, when day predicted local prediction patch (such as TP41 in equation 7 is appended to the Series Residuals tail, Dateformer will insert earlier past a day ground-truth residual (such as no in equation 7 into their head to balance the proportion between true local information and predicted residuals.
We didn't tt adopt the one-step forward generative style prediction that is proposed by Zhou et al. (2021) and followed by other Transformer-basee based forecast models because the style lacks scalability. For different length forecast demands, they have to re-train models. We are committed to providing scalable forecast style tackle forecast demands of various lengths. Although has defects of slow

---

Under review as a conference paper at ICLR 2023
prediction speed and error accumulation, Autoregression achieve the idea. So, still adopt Autoregression and fix its defects. Dateformer has excellent scalability. It can be trained on a short- term forecasting setup, while robustly generalized to long-term forecasting tasks. For each dataset, we only train Dateformer once to deal with forecast demands for any number of days.
EXPERIMENTS
Datasets We extensively perform experiments on real-world datasets, including energy, traffic, economics, and weather: (1) ETT (Zhou et al. [021] 202T) dataset collects 2 years (July 2016 to July 2018) electricity data from transformers located in Chima, including oil temperature and load recorded every 15 minutes hourly; (2) ECA dataset collects the hourly electricity consumption data of 321 Portugal clients from 2012 to 2015; P1 dataset contains power load series areas in China from 2009 to 2015. It's recorded every 15 minutes and carries incomplete weather data. We eliminate the climate information and stack the 2 series into a multivariate time-series; (4) Traffic contains hourly record value of road occupancy rate in San Francisco from 2015 to 2016; (5) Weather is collection of 21 meteorological indicators series recorded every 10 minutes by a German station for the 2020 whole year; (6) ER collects the daily exchange rate of countries from March 2001 to March 2022. We split all datasets into training, validation, and test set by the ratio of 6:2:2 for ETT and PI. datasets and 7:1:2 for others, just following the standard protocol.
Baselines We select 6 strong baseline models for multivariate forecast comparisons, including 4 ofate-of-the-art Transformer-based, RNN-based, and 1 CNN-based models: FEDformer(Zhor 2022) (ICML 2022), Autoformer (Wu et al.] 2021) (NeurIPS 2021), Informer (Zhou et al. 2021 (AAAI 2021 Best Paper Award), Pyraformer (Liu al. 2021 (ICLR 2022 Oral), LSTM
iet al.
Hochreiter & Schmidhuber 1997). and TCN (Bai et al. 2018).
Implementation details The details of our models and Transformer- based baseline models are in AppendixI Our code will be released after the paper's acceptance.
4.1
MAIN RESULTS
We show Dateformer's performance here. Due to splitting time series by day, we evaluate models with wide range of forecast days: 1d, 3d, 7d, 30d, 90d (or 60d), and 180d covering short-, medium- and long-term forecast demands. For coarse- graiined ER series, we still follow the forecast horizon setups proposed by Wu et al. (202T). Our idea is to break the information bottleneck for models, so we didn't 't. restrict models' input. Any input length is allowed as long as the model can afford it. For these baselines, if existing, we'll choose the input length recommended in their paper. Otherwise, an estimated input length will be given by We empirically select 7d, 9d, 12d, 26d, 46d (or 39d), and 60d as corresponding lookback days for Dateformer, and we only train it once on the 7d-predict-1d task to test all setups for each dataset. For fairness, all these sequence sequence-modeling baselines are trained time-step-wiss but tested day-wise. Due to the space limitation, only multivariate comparisons is shown here, see Appendix for univariate comparisons and Table 12 for standard deviations results.
Multivariate results Our proposed Dateformer achieves consistent state-of-the-art performance under all forecast setups on all 7 benchmark datasets. The longer forecast horizon, the more sig- nificant improvement. For the long-term forecasting setting (>60 days), Dateformer gives MSE reductions: 82.5% (0.585 - 0.144, 1.672 > 0.176) in Pl., 42% % (1.056 + 0.690) in E7Tml, 38.6% (0.702 + 0.431) in ETTh2, 59.8% (0.316 > 0.220, 2.231 + 0.240) in ECL. 79.4% (2.557 + 0.526) in Traffic, and 31.5% % in ER. Overall, Dateformer yields a 33.6% % averaged curacy improvement among all setups. Compared with other models, its errors rise mostly steadily as forecast horizon grows. It means that Dateformer gives the most credible and robust long-range
"hutps//archive.icc http://awccive.....i....../////e
12014
"This dataset is provided by the 9th China Society of Electrical Engineering cup competition.
http::/ppess.do..cag..
"https://www. http:://www.hhcjjoe... wtt.:jjnnammmd///weeeeee.
"hhps://fred.stoouu wtt.eeed....liseeeeooooeeeeeeeee
7

---

Under review as a conference paper at ICLR 2023
Table 1: Multivariate time series forecasting results. OOM: Out Of Memory. "-" means failing to train because validation set is too short to provide even a sample. (Number) in parentheses denotes each dataset's time-steps number every day. A lower MSE or MAE indicates a better prediction. The best results are highlighted in bold and the second best results are highlighted with a underline.
Models
Dateformer
FEDformer
Autoformer
Informer
Pyraformer
LSTM
TCN
Metric
MSE
MAE
MSF
MAF
MSF
MAF
MSF
MAE
MSE
MAE
MSE
MAE
MSE
MAR
0.042
0.141
0.105
0.231
0.098
0.201
0.067
0.157
0.052
0.139
0.796
0.577
0.111
0.211
0.076
0.187
0.149
0.264
0.405
0.479
0.251
0.331
0.134
0.230
0.672
0.553
0.197
0.296
P1,966
0.093
0.211
0.196
0.303
0.398
0.462
0.370
0.424
0.211
0.313
0.970
0.698
0,216
0.304
30
0.115
0.249
0.423
0.471
0.902
0.727
0.373
0.458
0.268
0.381
2.025
1.115
0.698
0.649
90
0.144
0.291
00M
0OM
00M
00M
OOM
00M
0.585
0.590
3.394
1.359
1.415
0.956
180
0.176
0.320
0OM
00M
0OM
00M
00M
0OM
00M
0OM
3.312
1.462
1.672
1.004
FCL124) FTTh2(224 F (96)
0.322
0.355
0.341
0.390
0.491
0.476
0.640
0.570
0.515
0.505
1.058
0.709
0.734
0.667
0.368
0,381
0.419
0.439
0.598
0.522
0,963
0.750
0.849
0.713
1.505
0.890
0.833
0.722
0.417
0.410
0.473
0
0.646
0.547
1.129
0.800
1.053
0.818
1.897
1.028
0.881
0.729
30
0.438
0.446
0.547
0.525
0.681
0,581
1.132
0.831
1.020
0.806
2.970
1.240
1.033
0.811
90
0.690
0.600
OOM
OOM
0OM
00M
0OM
0OM
1,056
0.845
1.855
1.010
1.081
0.821
0.234
0.306
0.246
0.327
0.289
0.364
1.606
0.997
0.412
0.498
1.074
0.723
1.126
0.874
0.311
0.363
0.334
0.381
0.347
0.395
1.928
1.111
1.140
0.832
1.583
0.873
1,900
1.138
0,383
0.413
0.412
0.426
0.451
0.451
6.200
2.024
4.877
1.747
2.429
1.111
4,410
1.741
30
0.437
0.472
0.466
0.483
0.510
0.511
4.091
1.717
4.674
1.869
2,014
1.052
2.919
1.503
90
0.431
0.486
0.719
0.618
0,702
0.631
2.571
1.217
3.330
1.511
3.754
1.436
3.356
1,503
0,113
0.218
0.169
0.288
0.174
0.293
0.328
0.412
0.268
0.372
0.368
0.425
0,365
0.436
0.148
0.251
0. 0.186
0.302
0.215
0.329
0.358
0,430
0.308
0.388
0.475
0.474
0.434
0.471
0,163
0,266
0.201
0.316
0,208
0.320
0,387
0.459
0
0.381
0,786
0.634
0.362
0.432
30
0.187
0.291
0.242
0.351
0.259
0.364
0.400
0.460
0.288
0.382
1.304
0.834
0,442
0.484
90
0.220
0.322
0.316
0.404
0.382
0.439
0.550
0.552
0OM
OOM
1,815
1.048
0.501
0.513
180
0.240
0.339
2,231
1.1.9
0.343
0.252
0.547
0.357
0.554
0.363
0.678
0.383
0.600
0.337
1.220
0.622
0.814
0.480
(R( Weather[ (44) Traile(244
0.430
0.284
0,581
0.367
0.625
0.391
0.721
0.404
0.635
0.358
1.902
0.878
1,582
0.743
0.434
0.289
0.613
0.382
0.705
0.439
0.768
0.425
0.639
0.356
2.509
1.114
0,803
0.444
0.469
0.308
0.652
0.395
0.686
0.420
0.959
0.534
OOM
OOM
2.413
1.089
1.120
0.615
90
0.526
0,339
2.557
1.128
0.220
0.288
0.234
0.304
0.327
0.377
0.370
0.424
0.231
0.310
5.868
L.478
0,267
0.328
0.281
0.329
0.338
0.375
0.370
0.402
0.628
0.560
0.388
4.232
1.417
0.468
0.452
0.320
0,360
0.495
0.472
0.474
0.461
1.093
0.768
0.442
0.460
5.727
1.469
0.523
0.510
30
0.414
0.428
0.688
0.580
0.724
0.604
2.789
1.303
0.823
0.676
7.141
1.804
0.878
0.711
60
0.539
0,531
8.168
1.766
96
0.022
0.107
0.041
0.154
0.040
0.154
0.248
0.361
0.194
0.340
0.452
0.520
0.482
0,541
192
0.043
0.152
0.062
0.194
0.061
0.193
0.430
0.474
0.337
0.440
0.536
0.558
0.496
0.558
336
0.070
0.195
0.087
0.233
0.096
0.246
0.756
0.642
0.607
0.600
0.825
0.705
0.557
0.586
720
0.112
0.255
0.5
0.317
0.389
0.422
1.073
0.773
0.963
0.772
0.866
0.729
0.677
0.652
forecast. Note that our Dateformer still contributes the best predictions on the ER series which is coarse-grained time series without obvious periodicity.
4.2 ANALYSIS
We try analyze the 2 forecast components' contributions to the final prediction and explain why Dateformer's predictive errors rise so slow as forecast days extend. We use separate global predic- tion local prediction component to predict and compare their results, as shown below.
Table 2:
Multivariate time series forecasting comparison of different forecast components results. The results highlighted in bold indicate which component contributes the best prediction.
Datasets
PL
ECL
Truffic
Forecast Days
1
30
180
30
180
90
Global
MSE
0.129
0.132
0.156
0.310
0.303
0.302
0.634
0.640
0.656
Prediction
MAE
0.264
0.268
0.297
0.395
0.392
0.392
0.355
0.356
0.362
Local
MSE
0.029
0.406
1.386
0.101
0.249
0.378
0.330
0.545
0.725
Prediction
MAB
0.115
0.457
0.908
0.201
0.327
0.412
0.239
0.341
0.414
Final
MSE
0,042
0.115
0.176
0.113
0.187
0.240
0.343
0.434
0.526
Prediction
MAB
0.141
0.249
0.320
0.218
0.291
0.339
0.252
0.289
0.339
8

---

Under review as a conference paper at ICLR 2023
It can be seen that errors of the separate global prediction hardly rise forecast horizon grows. The separate local prediction, however, is rapidly deteriorating. This may prove our hypothesis that time series have 2 patterns of components: global and local pattern. As stated in section Bl the global pattern will not fluctuate violently, while the local pattern is not long-lasting. Therefore, as forecast horizon grows, global prediction that still keep stable errors is especially important for long-term forecasting, Although do best in short-term forecasting, errors of local prediction increase rapidly as forecast days extend because local pattern is not long-lasting. As time goes on, the local pattern shifts gradually. For distant future days, current local pattern even degenerates into a noise that encumbers predictions: comparing the 180 days prediction cases on PL, the best prediction contributed by the separate globul prediction instead of the final prediction that disturbed by the noise. Not only local pattern, but Dateformer also grasps global pattern, so errors rise mostly steadily. To intuitively understand the 2 patterns' contributions, we use Auto-correlation that presents the same periodicity with source series to analyze the seasonality and trend of ETT oil temperature series.
1000000000
(a) Ground-Truth
(b) Global Prediction
(c) Scries Residuals
(d) Remainder
Figure 5: Auto-correlation of four time series from ETT oil temperature series: (a) Ground truth;
(b) Global Prediction; (c) Ground-trute Global Prediction; (d) Ground Final Prediction.
As shown in Figure Sa the ETT oil temperature series mixes complex multi-scale seasonality, and the global prediction almost captures it perfectly in 5b But see Figure [c] we can find global prediction fail to accurately estimates series mean: although a sharp descent occurs in the left end after eliminate the global prediction, it didn't immediately drop to near O. Because local pattern will effect series trends. The local pattern is not long-lasting, so we can only approximate it from lookback window. After subtracting the final prediction, we get an Auto-correlation series random oscillations around 0 in 5d which indicates source series is as unpredictable white noise series.
4.3 ABLATION STUDIES
We conduct ablation studies about input lookback window length and proposed time-representing method, related results and discussions are Appendix E Due the space limitation, we only list major finds here. (1) Our Dateformer can benefit from longer lookback windows, but not for other Transformers; (2) The proposed dynamic time-representing method can effectively enhance Dateformer's ' performance; (3) But sequence- eequence-modeling Transformers, which underestimate the sig- nificance of time and let it just play an auxiliary role, may not benefit from better time-encodings.
In addition, we also conduct ablation studies about 2 pre-training tasks that are presented in section 2 Beyond all doubt, the pre-training task of mean prediction is naturally effective. Removing this task to pre-train, Dateformer's predictions remarkable deteriorates. But for some datasets, pre-training DERT on only this task also damages Dateformer's performance. Auto-correlation prediction can in- duce DERT to concern time series intraday information, thereby encoding finer date-represeentation..
CONCLUSIONS
In this paper, we challenge employing merely vanilla Transformers to predict long-term series. We analyze information characteristic of time series and propose splitting time series into patches to enhance Transformer's 's throughout. Besides, and store it in time representations, to help the model grasp time series' global pattern. The proposed Dateformer yields consistent state-of-the-ar performance on extensive 7 real-world datasets.
distill the whole training set's global information

---

Under review as a conference paper at ICLR 2023
REFERENCES
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv: 1e03:012321111 2018.
George Box and Gwilym M Jenkins. Some recent advances forecasting and control. Journal of the Royal Statistical Society. Series C (Applied Statistics), 1968.
Chris Chatfield. The analysis of time series: an introduction. Chapman and hall/CRC, 2003.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555. 2014.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:/810.04803 [cs], 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Kaiming Hc, Xinlci Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Nikita Kitaev, tukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.0445// 2020.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The s1 International ACM SIGIR Conference on Research Development Information Retrieval, pp. 95-104, 2018.
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in Neural Information Processing Systems, 32, 2019.
Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and fore- casting. In International Conference on Learning Representations, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
1r110:1711.05101, 2017.
Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint 2019.
Athanasios Papoulis and S Unnikrishna Pillai. Probability, random variables, and stochastic pro- cesses. Tata McGraw-Hill Education, 2002.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 2019.
Yao Qin, Dongjin Song, Haifeng Chen, Wei Cheng, Guofei Jiang, and Garrison Cottrell. A dual-stage attention-based recurrent neural network for time series prediction. arXiv preprint arXiv: 2017.
David Salinas, Valentin Flunkert, Jan Gasthaus. and Tim Januschowski. Deepar: Probabilistic fore- casting with autoregressive recurrent networks. International Journal of Forecasting, 36(3): 1613):1181- 1181- 1191, 2020.
10

---

Under review as a conference paper at ICLR 2023
Rajat Sen, Hsiang-Fu Fu Yu, and Inderjit S Dhillon. Think globally, act locally: A deep neural network approach to high-dimensional time series forecasting. Advances in neural information processing systems, 32, 2019.
Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial intelligence and machine learning for multi-domain operations applications, volume 10006, pp. 1100612. International Society for Optics and Photonics, 2019.
Sean J Taylor and Benjamin Letham. Forecasting at scale. The American Statistician, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Norbert Wiener. Generalized harmonic analysis. Acta mathematica, 55(1):117-258, 1930.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition trans- formers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34, 2021.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang- Informer: Beyond efficient transformer for long sequence time-series forecasting, In Proceedings of AAAI, 2021.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting, In Proc. 39th International Conference on Machine Learning (ICML 2022), 2022.

---

Under review as a conference paper at ICLR 2023
A FUTURE WORKS
In this paper, as the preliminary work, we briefly introduced the dynamic date-representation and DERT, but didn't talk about their transferability. Actually, under certain conditions, a DERT that pre-trained on a time series dataset can transfer to other similar datasets to encode date- representations. For example, PL and ETT are similar electrical series datasets recorded in different cities of China, PL range from 2009 to 2015 while ETT range from 2016 to 2018, but we found the DERT that is pre-trained with the transferred DERT, Dateformer demonstrates a fairly good few sample learning potential. On the ETT series, even though only one-third of training samples are provided. Dateformer still can converge to the same performance level as the full training samples provided (see Table The details are not completely clear yet, we will further study in future works.
PL can directly transfer to forecast tasks on ETT. More interestingly,
Table 3: Few sample learning multivariate results on ETTm/, only MSE is reported.
Models
Dateformer-ETTnie ppre-pee-tttt prainer
Dateformer-PLe Peeeformer-P---r--rainee
Samples
353
233
113
353
233
113
1
0.326
0.331
0.333
0.422
0.322
0.324
0.332
0.373
3
0.370
0.374
0.383
0.445
0.368
0.365
0.381
0.399
7
0.418
0.422
0.427
0.478
0.417
0.414
0.426
0.443
30
0.443
0.442
0.435
0,476
0.438
0.437
0.435
0.453
90
0.702
0.711
0.680
0.698
0.690
0.680
0.678
0.696
Referring to Table 3 train 2 Dateformers on ETT series ranging full 12 months (353 samples), 8 months (233 samples), 4 months (113 samples) and 8 days (1 sample), to test all forecast horizon setups on the same test set. The Dateformer whose DERT pre-trained on PL series shows the same performance level, even slightly better because PL is bigger dataset with longer time span. And fewer samples are required: months of ETT series is enough for Dateformer to convergence.
B RELATED WORKS
Time series forecasting is an enduring research topic, and numerous works have been developed to deal with the task. They can roughly divided into two categories: statistical methods and deep learning models. ARIMA (Box & Jenkins 1968). Prophet (Taylor & Letham 2018), and the filtering methods are representative methods of the former. The deep learning models mainly include RNN- based (Recurrent Neural Network based), CNN-based (Convolutional Neural Network based), and Transformer- based structures.
LSTM (Hochreiter & Schmidhuber 1997) and GRU (Chung et al. 2014) employ gating mecha- nisms to extend series dependence learning distance and relieve the gradient vanishing or explosion of RNN. DeepAR (Salinas et al.] 2020) further combines LSTM and Autoregression for time ries probabilistic distribution forecasting, LSTNet (Lai et al. 2018) adopts CNN and recurrent-skip connections modeling short- and long-term patterns of time series. Some RNN-based works introduced the attention mechanism to capture long-range temporal dependence, so as to improve predictions (Qin al. 2017). However, RNN's intrinsic flaws (difficulty capturing long-range de- pendence, slow reasoning, and accumulating error) prevent them from predicting long-term series.
Temporal convolutional network (TCN) (Bai et al. 2018 is CNN-based representative work. which models temporal dependence with the causal convolution. DeepGL. (Sen et al. 2019) also men- tioned the concept of global and local, and employs TCN to model them. Nevertheless, the global concept in their paper refers to relationships between related other time series, which is dis- tinet from this paper refers to the global historical observation on the series itself.
Transformer Vaswani et al. 2017 recently becomes popular in long-term series forecasting, owing to its excellent long-range dependence capturing capability. But directly applying Transformer to long-term series forecasting is computationally prohibitive due to inside self-attention's quadratic complexity about sequence length in both memory and time. Many studies are proposed to re- solve the problem. LogTrrans (Li et al., 2019) presented LogSparse attention that reduces the
12

---

Under review as a conference paper at ICLR 2023
computational complexity to ( and Reformer (Kitavv et al.
2020) presented local-
sensitive hashing (LSH) attention with (L complexity. Informer (Zhou et al. 2021) pro-
posed probSparse attention of (Llog)) complexity, and further renovated the vanilla Transformer architecture. Autoformer (Wu et al. 2021) and FEDformer (Zhou et al.) 2022) built decomposed blocks in Transformer and introduced low complexity enhanced attention in frequency domain or Auto-correlation to replace self-attention
Pyraformer (Liu et al., 2021) adopted hierarchical multi- resolution PAM attention achieve O()) complexity and learn multi-scale temporal dependence. To extend forecast horizon, all these variant Transformers designed various modified self-attentions or substitutes reduce computational complexity. so as tackle longer time series.
In CV community, ViT (Dosovitskiy et al., 2020) split images into patches, thereby enabling vanilla Transformers to tackle abundant pixels. Compared to highly abstract human language, we argue that information characteristic of time series is more similar to it of images. They both are natural signals and exist numerical continuity between adjacent signals. Inspired by this, we follow ViT to split time series into patches, thereby enhancing time series forecasting Transformers' throughput, and enabling vanilla Transformers to predict long-term series.
In addition to these structures, there are also other neural network methods such as N-BEATS Ore- shkin et al. 2019) that uses decomposition. Above most forecast methods can be summarized as learning mapping from lookback window to forecast window, and they some leverage time feature assist prediction. However, in the end, they are confined by information because they can only rely on the local information in lookback window. To best knowledge, forecasting work that distills the whole training set's global information into time-representations and takes time instead of series the primary modeling
the first time series
modelniiinng strategy.
C DATE EMBEDDING
Observing the movement of celestial planets, changes in temperature, or the rise and fall of plants, our forefathers distilled a set of laws, that is calendars. A wealth of wisdom is encapsulated in the calendar. People's activities are guided by the calendar and believe that's the fundamental source of seasonality in many time series. We try to introduce the wisdom in our models as priori knowledge. We use the Gregorian calendar as solar calendar and the traditional Chinese calendar as lunar calendar to deduce our datc-embedding, Besides, the vacation and weekday information is also taken into count. our code, we provide the date-embeddings range from 2001 to 2023 for 12 countries or regions: Australia, British, Canada, China, Germany, Japan, New Zealand, Portugal, Singapore, Switzerland, USA, and San Francisco.
Our static date-embedding stacks the following date features: abs day, year, day (month day), year_day, weekofyear, lunar lunar lunar_month, lunar_day, lunar_yeardlee dayofyear, dayof- month, monthofyear, dayofweek, dayoftunaryear, dayoflunarmonth, monthoftunaryeat jieqiofyear, jieqi day, dayoffieqi, holidays, workdays, residual holiday, residual workday. As an example, we embed today by following equations:
abs_day- days that have passed from December 31, 2000
365.25 *
year this year 1998.5
25
day- days that have passed in this month
days that have passed in this year
year day-
366
weeks that have passed in this year
weekof year year
54
this lunar year 1998.5
lunar year
25
Yunor_montl this lunar month
12
Junar_day days that have passed this lunar month
30
13

---

Under review as a conference paper at ICLR 2023
days that have passed in this lunar year
lunar yearreyarraa _day
384
dayofyean days that have passed in this year - 0.5 total days in this year-
dayof days that have passed this month
0.5
total days in this month -
montho months that have passed in this year 1
0.5
11
dayofweekk days that have passed in this week - 0.5
Seayoflunargerr days that have passed in this lunar year - 1
0.5
total days in this lunar year -
dayo lunarmonth days that have passed in this lunar month
0.5
total days in this lunar month
montho funaryear lunar months that have passed in this lunar year -
0.5
11
jieqio yeerr solar terms that have passed in this year - 0.5
23
fieqi_day days that have passed this solar term
15
days that have passed this solar term 0.5
total days in this solar term
We standardize these date features by respective corresponding maximum. For unbounded features (e.g., abs_day and year), to standardize year, then need not worry about it in our lifetime. With DERT encoding, our model shows excellent generalization when facing date never before (See Table 2 Because too complicated, some date features' equations are omitted. Email authors if interested.
large number to standardize them. For example,
100
D UNIVARIATE RESULTS
Baselines We also select 6 strong baseline models for univariate forecast comparisons, covering state-of-the--rt deep learning models and classic statistical methods: FEDformer Zhou et al. 2022]
Autoformer (Wu et al. 2021). Informer (Zhou et al., 2021), N-BEATS (Oreshkin al. 2019
DeepAR (Salinas et al. 2020 and ARIMA (Box & Jenkins 1968).
Table 4: Univariate series forecast results. A lower MSE or MAE indicates a better prediction. The best results are highlighted in bold and the second best results are highlighted with a underline.
Models
Dateformer
FEDformer
Autoforner
Informer
N-BEATS
DeepAR
ARIMA
Metric
MSE
MAE
MSE
MAE
MAE
MSE
MSE
MAE
MSE
0.040
0.133
0.132
0.249
0.145
0.240
0.058
0.144
0.092
0.170
0.211
0.272
0.996
0.801
Pl (96)
0.087
0.190
0.183
0.290
0.350
0.416
0.197
0.277
0.209
0.272
0.543
0.506
1.163
0.878
0.504
0.512
30
0.113
0,239
0.516
0.507
0.841
0.696
0.418
0.470
0.430
0.448
1.758
1.063
5.594
1.292
90
0.135
0.276
OOM
0OM
0OM
0OM
0OM
0OM
0.442
0.517
1.311
0.863
7,801
1.739
180
0.305
OOM
0OM
00M
0OM
00M
00M
00M
OOM
1.932
1.115
9.824
2.039
0.097
0,167
0.181
0.288
0.257
0.361
0.211
0.304
0.139
0.229
0.326
0.399
0.461
0.499
FR(() TTiffee24)
0.118
0.199
0.231
0.342
0.293
0.390
0.232
0.327
0.144
0.236
0.441
0.457
0.796
0.703
0.122
0.204
0.228
0.329
0.425
0.250
0.342
0.162
0.260
0.839
0.683
1.240
0.915
30
0.134
0.221
0.252
0.349
0.294
0.389
0.302
0.393
0.200
0.306
0.518
0.503
1.627
1.086
90
0.157
0.236
0.695
0.617
1.739
1.130
96
0.042
0.157
0.069
0.206
0.097
0.238
0.341
0.508
0.235
0.378
0.179
0.342
0.086
0,190
192
0.079
0.223
0.108
0.266
0.124
0
0.453
0.581
1.697
1.112
0.427
0.584
0,213
0.282
0.118
0.267
0.149
0.312
0.305
0.695
0.738
5.616
2.119
0.535
0.659
0.191
0.320
720
0.187
0.344
0.209
0.373
0.297
0.439
2.540
1.522
5.154
1.767
1.221
1.012
0.267
0.364
14

---

Under review as a conference paper at ICLR 2023
Univariate results In the univariate time series forecasting setting, Dateformer still achieves sistent ofate-of-the-re performance under all forecast setups. For the long-term forecasting set- ting. Dateformer gives MSE reduction: 82.3% (0.442 + 0.135, 1.932 - 0.162) in PL, 77.4% (0.695 + 0.157) in Traffi, 23.7% % ER. Overall, Dateformer yields a 41.6 % averaged accuracy improvement among all univariate forecast setups the given representative datasets.
E ABLATION STUDIES
E.1 IMPACT OF INPUT LENGTH
In this section, we study the impact several Transformer-based models of different input lengths. On representative forecast tasks, which cover short-, middle-, and long-term forecasting on various time series datasets, gradually extend their lookback window and record their predictive errors of different input window sizes. The results are shown in the form of line charts as follows.
(a) ETTH 7-day
(b) E7Th2 1 1-day
(c) ETTml 3-day
(d) PL. 7-day
(e) ECL 30-day
(1) Truffic 3-day
(g) Weather 1-day
(h) ER 720-day
Figure 6: The errors lines of several Transformers with different input lookback days of: predicting next (a) 7-day (b) 1-day (c) 3-day ETTmI,(d) 7-day PL.(e) 30-day
(CL,(f) 3-day on (gaffic,g) 1-day on Weather,()) 720-day on ER time series.
As shown in Figure 6 with lookback window extends, Dateformer's predictive errors gradually reduce until stable, which indicates that Dateformer can leverage more information continuously improve prediction. Although not the best under narrow lookback windows. Dateformer finally outperforms other all baseline Transformers may not benefit from larger input windows. Their performances are unstable, and even deteriorate as input length extends. This is against common sense: more information intake should lead to better predictions. Compared to them, the information utilization upper limit of Dateformer is higher. We can always expect better predictions by feeding longer lookback window series, at least not worse.
lookback window grows. But other Transformers
E.2 IMPACT TIME ENCODING
In this paper, we contribute time-encen canoeenncodnng: the static date- embeeddnbnnngg that is generated by stack- ing more date features straightforwardly. and the dynamic date- representation that further considers date contextual information. So, we try to clarify which time-encoding is better, and how well other Transformers perform when better time-encodings provided.
15

---

Under review as a conference paper at ICLR 2023
E.2.1 WHICH TIME ENCODING BETTER?
The better time-encoding should be able represent time more accurately and accommodate more time series global information. Thus, we employ the 2 time- uiee-nncdingg to distill time series' global information through global prediction components with roughly the same number of parameters, and then check their global prediction quality. The results are shown below.
Table 5: Global prediction multivariate comparison using 2 time-encoding.
Datasets
PL
ECL
Truffic
Forecast Days
30
180
30
180
90
Dynamic
MSE
0.129
0.132
0.156
0.310
0.303
0.302
0.634
0.640
0.656
Representation
MAE
0.264
0.268
0.297
0.395
0.392
0.392
0.355
0.356
0.362
Static
MSE
0.177
0.181
0.193
0.316
0.306
0.305
0.655
0,659
0.669
Embedding
MAE
0.306
0.310
0.325
0.398
0.393
0.393
0.363
0.362
0.363
As shown in Table | the global prediction component that employs dynamic date-representation always contributes better global prediction. This is enough prove the effectiveness of our pro- posed dynamic time-representing method. In order to more intuitively show the importance of date contextual information, and how DERT pays attention date context, visualize some attention weight distributions of DERT encoder. The figures are shown as follows.
Dic 27
28
Dic 29
Dec 30
Doc 51
Jant Janz
Jan
jana
Jan
jan
pana
jan1l
Jauel
Jen
Jan 18
lanta
Jan 20
Jan
Jan2a
jan24
Jan26
Jan
Jan
Feb
Rab
Feb
Ra
RR
RR
RO
headl
head2
head2
head
head3
heads
head4
head5
head5
head6
head6
head7
head7
head8
headl
Jan
(b) Jan
Mar 30
Mar
Apr
Apr
Apr
Apr
Apr
Apr
Apr
Apr
Apr 12
Apr 13
Apr 14
Apr 15
Aer 16
Apr
Aer 18
Sep 25
Sap 26
Sop 27
Sap 28
Sep 29
Sop 30
0c:1
Oct Oct
Oct
oct
oct
ox
oo 10
on 12
oce
ooct 14
Loc
oct 36
headl
headl
head2
head2
head3
head4
head4
head5
head8
headh
head7
head7
Fead8
head8
(c) 5pr
Oct
Figure 7: DERT encoder attention weight distributions from PL dataset during 2012, the darker the cell, the greater the attention weight. (a) is encoding
Jan is New Year's day: (b) encoding Jan 24, in traditional Chinese lunar calendar, Jan 22 is New Year's Eve and Jan 23 is the Spring Festival; (c) encoding Apr 5, Apr the Qingming Festival. (d) encoding 2, Sep 30 the Mid-Autumn Festival in Chinese lunar calendar, and Oct 1 is National Day in Gregorian calendar.
In Figure 7 when approaching important festivals, DERT pays more attention to these festivals. The more important festival, the more concentrated attention. In China, the Spring Festival the most important festival of a year. So, we can see the most intensive attention on the day in Figure 77b and other days barely receive little attention. But referring to Figure although very intensive attention on Apr 4, some attention is still distributed to other days. Because the Qingming Festival is not as important as the Spring Festival. DERT will not pay excessive attention to it. This is completely in line with Chinese habits. Besides, DERT also tackle the changeable date semantics in different calendars. In the traditional Chinese lunar calendar, Sep 30, 2012, is the Aut-Au Festival while Oct is Chinese National Day in the Gregorian calendar. The interval between the festivals change
16

---

Under review as a conference paper at ICLR 2023
every year, and in some years they even the same day. In Figure 7d they both exert influence on Oct 2. We can not sure how their influence change when they are on the same day or on adjacent days, let DERT learn from data. We also call the similar problems date polysemy, Note that we didn't t tell DERT which day is festival, all knowledge about festivals is learned by itself.
E.2.2 How WELL OTHER TRANSFORMERS PERFORM USING BETTER TIME ENCODINGS?
These Transformer-based baselines for comparison also leverage time features to assistant predic- tions. They adopt the time embedding that proposed by Zhou et al. (2021). We are inspired by it too, so follow it and stack more date features into our static sute-eemeddingg In this section, we try to clarify how well they perform when better encodings are provided. We modify these Trans- formers employ the timeenccoiing mentioned in the previous section, to compare the impact of different time-encoding on them. To better understand the role of time for these sequence-modeling Transformers, we eliminate time embedding comparison. The results are as follows.
Table 6: Ablation of time-encoding on ETTh/ multivariate forecast tasks with MSE metric. Without eliminates time-encoding. Origin adopts time-encoding proposed by Zhou et al. (021). Static and Dynamic denote the static date-embedding and dynamic date-representation respectively. Note that Origin and Static essentially the same, Static just simply collects more time features.
Forecat Days
3
90
Models
Without
Origin
Static
Dynamic
Without
Origin
Static
Dynamic
FEDformer
0.359
0.360
0.374
0.367
0.853
1.027
1.250
1.142
Autoformer
0,415
0.436
0.441
0.422
1.029
0.929
1.327
1.257
Informer
0.695
0.788
0.747
0.872
1.395
1.291
1.168
1.672
Pyraformer
0.606
0.605
0.600
0.591
1.064
1.100
1.079
0.987
Reformer 0.723
0.776
0.840
0.699
1.179
1.210
1.245
1.085
Dateformer
0.386
0.413
0.369
0.721
0.711
0.691
Table 7: Ablation of time-encoding on Traffic multivariate forecast tasks with MSE metric. means
can't train, because time-encoding necessary for Dateformer. Best results are highlighted in bold.
Forecat Days
7
30
Models
Without
Origin
Static
Dynamic
Without
Origin
Static
Dynamic
FEDformer
0.626
0.613
0.601
0.592
0.676
0.652
0.630
0.624
Autoformer
0.807
0.705
0.653
0.641
1.285
0.686
0.673
0.667
Informer
0.751
0.768
0.787
0.774
0.966
0.959
1.196
1.072
Pyraformer 0.646
0.639
0.644
0.671
OOM
OOM
00M
OOM
Reformer
0.978
0.720
0.703
0.696
1.470
0.683
0.721
0.885
Dateformer
0.483
0.451
0.434
0.510
0.487
0.475
Referring to and 7 as time-modeling method that mainly models time, Dateformer consis- tently benefits from better time-encodings. However, better time-encodings may not enhance other Transformers that mainly model series, and they only leverage time-encoding in auxiliary style just like positional encoding. In some forecast cases, better time-encodings can improve their pre- dictions. But in some other cases, eliminating time-encoding results in the best prediction. Their utilization of time is unstable. Besides, simply collecting more time features does not necessarily work, sometimes encumber predictions (see days prediction case in Table
17

---

Under review as a conference paper at ICLR 2023
E.3 ABLATION STUDY ABOUT PRE-TRAINING TASKS
In section we design 2 pre-training tasks for DERT: mean prediction and Auto-correlation predic- tion. Here, check their effectiveness. We use the separate pre-training task to train DERT, and compare their prediction results. The results are shown as follows.
Table 8: Multivariate forecast comparison of Dateformer using different pre-training tasks. Mean adopts only pre-training task of mean prediction. Auto employs separate Auto-correlation prediction to pre-train DERT. Both combines the 2 pre-training tasks together, and it's adopted by us. The best results are highlighted in bold. "_" denotes lacking test samples report result.
Time Series Datasets
PL
ECL
ETTml
Forecast Days
Metric
Mean
Auto
Both
Mean
Auto
Both
Mean
Auto
Both
MSE
0.044
0.041
0.042
0.123
0.128
0.113
0.328
0.336
0.326
MAE
0.146
0.138
0.141
0.232
0.240
0.218
0.355
0,362
0.356
MSE MAE
0.096
0.094
0.076
0.159
0.165
0.148
0.372
0.381
0.370
3
0.217
0.207
0.187
0,265
0.275
0.251
0,383
0,389
0.382
MSE
0.115
0.115
0.093
0.173
0.181
0.163
0.425
0.434
0.418
7
MAE
0.243
0,236
0.211
0,280
0.290
0.266
0.415
0,420
0.411
MSE
0.143
0.142
0.115
0.195
0.207
0.187
0.452
0.447
0.443
30
MAE
0.284
0,277
0.249
0.301
0.311
0.291
0.453
0,451
0.448
MSE
0.165
0.171
0.144
0.229
0.234
0.220
0.735
0.687
0.702
90
MAE
0.311
0.314
0.291
0.332
0.337
0.322
0.622
0.600
0.600
MSE
0.189
0.204
0.176
0.251
0.257
0.240
180
MAE
0.333
0.344
0.320
0.349
0.353
0.339
As shown in Table 8 for pre- pre-training DERT, the separate mean prediction or Auto-correlation pre- diction task is effective. But combining them can obtain better date-representations. The mean prediction will lose some intraday information of time series, so we design the Auto-correlation pre- diction to mitigate the problem, it can induce DERT to tap some time series intraday information.
F TRAINING DETAILS
There are 3 relatively independent components in Dateformer: DERT encoder, global prediction, and local prediction. They all can be trained separately. We found that Dateformers with different training stages strategies have different predictive performances. For some datasets, pre-training DERT global prediction component is necessary. For other datasets, end-to-end training the entire Dateformer is also feasible. Thus, we design 3-stage training methodology to tap the full capacity of Dateformer, and do the best various forecast horizon demands of different datasets.
Pre-training In section 2 we introduce tasks to pre- pre-train train DERT, and the first step is to pre- tre-train the DERT encoder. Time series are split into patches by day supervise the learning of DERT. Fach task contributes half of the pre-training loss. The pre-training stage aims to provide superior time representations as container to distill and store global information from training set.
Warm-up The second step is using the separately global prediction component distill time series global information from training set, we call warm-up phase. In warm-up, the pre-traines DERT encoder is loaded, then train the separately global prediction component training set. We insert this stage to force the global prediction component to remember the global characteristics help Dateformer produce
of time series, so
robust long-range predictions. Furthermore, it also serves as the adapter when transferring DERT from other datasets. This phase is optional, and we observed that better short-term forecast is usually provided by the Dateformer skipping warm-up. More credible long-term predictions, however, always draw from the preheated one.
18

---

Under review as a conference paper at ICLR 2023
Formal training Dateformer loads pre-trained DERT global prediction component then start training. We would use a small learning rate to fine-tune the pre-trained parameters. With enough memory, Dateformer can extrapolate to any number of days in the future, once trained.
Table 9: Multivariate time series forecasting comparisons of different training strategies, where Dateformer" goes through all stages. Dateformer skips the warm-up, and Dateformer" skips the pre-training stage. Dateformer is end-to-end trained. The * marked group of result is selected to compare with baseline models in the main text. The best results highlighted in bold.
Models
Dateformer"
Dateformer
Dateformer"
Dateformer
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
0..42" 0.141*
0.032
0.119
0.044
0,142 0.027 0.112
0.076" 0.187"
0.107
0.220
0.078
0.192
0.077 0.176
PL (L66))
0.093*
0.211'
0.147
0.271
0.097
0.219
0.122 0.227
0.115" 0.249"
0.275
0.375
0.133
0.269 0.324 0.384
90
0.144"
0.291"
0.600
0.587
0.166
0.311
0.798 0.679
180
0.176"
0.320"
0.911
0.733
0.218
0.358
1.238 0.843
ETTh2(24) ETTml (966)
0,344
0.368
0.322"
0.355*
0.345
0.363
0.325 0.356
0.383
0.396
0.368
0.381
0.389
0.390
0.369 0.382
0.441
0.431
0.417
0.410"
0,439
0.420
0.417 0.411
0.494
0.487
0.438
0.446'
0.477
0.472
0.440 0.447
90
0,789
0.646
0.690
0.600"
0.723
0.614
0.693 0.598
0.234
0.303
0.226
0.292
10.234"
0.306'
00.228 0.295
0,315
0.362
0.296
0.340
0.311
0.363' 0.299 0.345
0,406
0.422
0,369
0.389
0.383
0.413* 0.372 0.394
30
0.458
0.471
0.407
0.438
0.437"
0.472"
0.404 0.436
90
0.521 0.537
0.590
0.553
0.431
0.486"
0.582 0.542
00.113
0.218
0.101
0.198
0.121
0.231
0.100
0.198
ECL (24)
0.148 0.251
0.146
0.237
0.158
0.265
0.156 0.241
0.163* 0.266*
0.166
0.257
0.175
0.284
0.175 0.261
30
0.187 0.291*
0,219
0.306
0,202
0.311
0.226 0.309
90
0.220
0.322*
0.334
0.391
0.239
0.346
0.354 0.399
180
0.240
0.339*
0.338
0.392
0.255
0.362
0.353 0.398
ER()) Weather (144) Traffie(24))
0.341
0.248
0.340
0.240
00,33"
0.252'
0.339 0.239
0.459
0.293
0.557
0.321
0.430
0.284'
0.484 0.293
0.468
0.301
0.578
0.337
0.434 0.289'
0.494 0.306
0.503
0.319
0.621
0.362
0.469
0.308
0.543 0.332
90
0.559
0.344
0.685
0.389
0.526
0.339"
0.669 0,358
0.224
0.294
0.223
0.295
0..200
0.288*
0..23 0.286
0.289
0.338
0.285
0.338
0.281
0.329*
0.285 0.329
7
0.332
0.370
0.326
0.369
0.320
0.360'
0.329
0.364
30
0.427
0.435
0.419
0.438
0.414
0.428*
0.413
0.422
0.545
0.540
0.537
0.536
0.539*
0.531*
0.524
0.524
96
0.045
0.160
0.022
0.107*
0.036
0.135
10.022
0.107
192
0.076
0.206
0.043"
0.152'
0.065
0.181
0.043 0.152
336
0.137
0.274
0.070
0.195'
0.093
0.221
0.070
0.195
720
0.336
0.426
0.112
0.255
0.138
0.282
0.113
0.256
We use different training stages to train Dateformer, and results are shown in Table 9 It can be seen that different training stage setups can lead to different performances of Dateformers. There no versatile training strategy does best on all forecast horizons of all datasets. Generally, the pre-training and warm-up can induce Dateformers to be more concerned with the global pattern of time series, and produce robust long-term predictions. For some datasets, before distilling global information from training set, pre-training DERT encoder can enhance the distilling effect. But for other datasets, the stages can be combined into one. Actually, the global prediction is also a fairly good pre- meetrrining task for DERT. The end-to-end trained Dateformer always contributes the best short-term predictions. Because we train Dateformer in the short-term forecasting task of 7 predict-ld, 1d, the direct end-to-end training makes Dateformer more care local pattern of time series.
19

---

Under review as a conference paper at ICLR 2023
G PATCH SIZE CHOICE
In the main text, we choose the "day" as patch size, because it's moderate, close to people's habits, and convenient for modeling. Here, we try some other patch sizes. We select day, half-day, and 3-day as comparative patch sizes. The results are shown below.
Table 10: Multivariate forecast comparison of Dateformer using different patch sizes. denotes that the patch size is too coarse to align with the forecast horizon.
Time Series Datasets
ETTh2
Traffic
Forecast Days
Metrie
1 day
half-day
day
3-day
day
half-day
day
3-day
MSE
0.224
0.226
0.234
0.421
0.417
0.343
MAE
0.315
0.315
0.306
0.285
0.283
0.252
MSE
0.325
0.318
0.311
0.366
0.558
0.526
0.430
0.472
MAE
0.382
0.374
0.363
0.422
0.379
0.326
0.284
0.311
MSE
0.428
0.407
0.383
0.564
0.524
0.434
MAE
0.452
0.429
0.413
0.384
0.328
0.289
MSE
0,513
0,458
0.437
0.690
0.627
0.548
0.469
0.527
MAE
0.541
0.483
0.472
0.722
0.455
0.340
0.308
0.330
MSE
0.540
0.485
0.431
0.676
0.655
0.582
0.526
0.564
MAE
0.577
0.505
0.486
0.715
0.447
0.360
0.339
0.361
As shown in Table 10 for some time series, finer patch sizes can lead to better short-term predictions, because the closer time series carries more accurate local pattern information. Compared to a day ago, the local pattern of the present is more similar to it of half day ago. But their mid-long-tes long-term predictions deteriorate, because finer patch sizes will result in more local predictions recursions for the same size forecast horizons, which means more series whose daily periodicity is dominant (like Traffic), the "day" is the best patch size. These time series are usually closely related to human activities and hence more concerned with daily patterns. The coarse patch size beyond "day" is not the most important activity period of people, so it's difficult to construct sufficiently accurate ammereepreentttionn.. If there holiday in the 3-day patch, how to embed which day it is in the time-embedding by the method mentioned at the beginning of Section That leads over-coarse patch sizes inapplicable. most time series related to human activity, we recommend the "day" as the patch size, it's moderate, close to people's 's habits.
accumulation. In addition, for these time
H HYPER PARAMETER SENSITIVITY
We check the robustness with respect to the hyper-parameters: pre-days and post-days paddings. We
select groups of paddings: (1, 1), (3, 3), (7, 7), (7, 14), (14, 14) and (30, 30). To be more intuitive, test the global prediction component with above several paddings on PL dataset.
Table 11: Dateformer's multivariate global prediction results on PL. dataset using several paddings.
Paddings
1,
3,3
7.7
7, 14
14, 14
30, 30
Metric
MSE
MAE
MSR
MAE
MSE
MAE
MSE
MAE
MSR
MAE
MSE
MAE
1
0.155
0.290
0.151
0.285
0.127
0.264
0.134
0.270
0.139
0.278
0.156
0.291
3
0.155
0.290
0.152
0.286
0.128
0.264
0.134
0.271
0.139
0.278
0.157
0.292
7
0.156
0.291
0.153
0.287
0.128
0.265
0.134
0.271
0..39
0.277
0.157
0.299
30
0.161
0.296
0.158
0.292
0.132
0.269
0.137
0.274
0.142
0.281
0.161
0.299
90
0.179
0.315
0.176
0.311
0.145
0.284
0.151
0.291
0.156
0.297
0.177
0.314
180
0.190
0.326
0.187
0.324
0.155
0.295
0.165
0.305
0.173
0.314
0.197
0.335
As shown in Table 11 too large or too small padding sizes will make the prediction performance worse. The moderate padding size (7, 7) leads to the best global prediction. This also consistent with how people generally think-wy don't simply consider the here and now, but also don't 't think
20

---

Under review as a conference paper at ICLR 2023
too far ahead. In the main text, to balance predictive performances between various time series datasets, we use the padding size of (7, 14) instead of the best (7, 7).
I IMPLEMENTATION DETAILS
Our proposed models are trained with L2 loss, and using Adam (Loshchilov & Hutter, 2017 optimizer with weight decay of 7e We adjust learning rate by OneCycleLR (Smith & Topin 2019) which use 3 phase scheduling with the percentage of the cycle spent increasing the learning rate is 0.4, and the max learning rate is 1e-3 Batch size for pre-training is 64, and 32 for others. The total epochs are 100, but the models normally super converges very quickly. All experiments are repeated 3 times, implemented by PyTorch(Paszke et al. 2019). and trained on a NVIDIA RTX3090 24GB GPUs. Numbers of the pre-days and past-days padding are 7 and 14 respectively. There is layer in DERT encoder, and the inside Transformer contains layers both in encoder and decoder.
The implementations of Autoformer (Wu et al. 2021), Informer (Zhou et al. 2021). and Reformer (Kitaey et al. 2020) are from the Autoformer's s repository [1] And the implementations of FED- former (Zhou et al. 2022) and Pyraforme
13 (Liu et al. 202T) are from their respective repository. setting that recommended in their repositories but unify the token's dimension dmodet as 512. We fix the input series length as 96 time-steps for FEDformer, Auto- former, Informer, and Reformer. This recommended by them or empirical results, and we found extending their input length will result in unstable performances of these baselines (see section [..I) For Pyraformer. facing longer forecast horizons, we extend its input length as their paper recom- mended. For detailed hyper-parameters setting please refer their code repositories. For other baseline models, we also use grid-search method to select their hyper-parameters.
We adopt the hyper-paramente
J COMPLEXITY ANALYSIS
We also provide the complexity analysis of the global prediction and local prediction components. For an input lookback window with length L; and output forecast window with length Lo, they're all divisible by G in the setting of splitting time series into patches by day, where G denotes time- steps number every day of the time series datasets. The complexity of global prediction component O(2, + T.,). For local prediction component, it recurses Le times and the time complexity of each time shall not exceed OO( (( )²). So, the time complexity of local prediction component is ofty fot ( ) at most. Its maximum memory usage is O(L, + L. + ( )²), where (اا )2 happens in the last day's prediction. In practice, G² will be a big number for fine-grained time series and the Le + Lo takes up the most memory, becomes the main factor that restricts us to predict further away. Under 24GB memory, Dateformer's 's maximum forecast horizon exceeds all baseline models. And its inference speed is just slightly slower than Transformers that adopts one- step forward generative style inference (Zhou et al. 2021) because the local prediction component requires a few recursions.
12 "https//itthbbccm///mm////uuoour Paane://gghna..cccm//aayyyyyyyer
21

---

Under review as a conference paper at ICLR 2023
K
MAIN RESULTS WITH STANDARD DEVIATIONS
We repeat all experiments 3 times, and the results with standard deviations are shown in Table 12
Table 12: Quantitative results with fluctuations of different forecast days for multivariate forecast.
Models
Dateformer
FEDiormer
Autoformer
Informer
Pyraforuer
Metric
MSE
MAE
MSE
MAE
MSB
MAE
MSE
MAE
MSE
MAE
0.042
0,141
0.105
=.231=00000
0.098-0003
0.201=0017
0.067=9002
=.157
0.052=0080
0.139-n002
ECL/24) ETT12/2 (77) PL(96)
0.0993c000
0.0761cum1
0.21
0.187:ax01
0.149
0.196st
0.264
0.303
0.398
0.405
0.462
0.479
0.570
0:251:0045
0:421
=0331:
0.211=0005
0.
0.313:
0.230
30
0.115i
0.249.c3te
0.423acamn
L.47m
0:902u4371
0.727.am
0.373-am3
0.458.am03
0 0.268.9.00
0.381
90
0.144:com
0.291-0000 -0.
00M
0OM
00M
00M
ODM
0DM
0.585-0004
0.590-44044
180
0.176:t000
0.320
OOM
0OM
00M
00M
OOM
00M
ODM
ODM
0.322
0.355
0.341 se
0. ce
0.491 ii
0.476.0m
0.640.ares
0.570
0.515
0.50 0.505
0.368-0000
00
0.439-4001 0411
00
0.598-4000
0.522-0014
0.93-0002
0.750-002
0 0.849-0006
0.713
0.417:c006
0.410
0.473
0.
0.6-16
1.199
0 0,800
10053
0.818
30
0.438
0.446:03m1
0.547acx09
0 0.525aam
0.681
0.581.ame
L132.ame
L.83
L020-0003
0.80 0.806.90ce
0.690.c0m9 90
0.600.0.00
0OM
0OM
00M
00M
00M
00M
1.056-0018
0.845-0808
6.234:co01
0.246-2001
0. 0.327-4000 oio
0.289:...10 DID
0.364 oon
1666-- iss
0.997-000
0.412-0000
0.498-00
0.311
0,363:2uc
0.334
0.395 0,347=a0
004
1.92-=0
1111=:003
1. 1,140=0003
0.63220004
0.383
0,413:cms
0.426.a 0.412scxee
6.200
2.024:20es
4.877
30
4.037.cms
0.472:cm
0.483-2mm 0.466-2.m
mm
0.
0.511-am0
4.091-4 1o
1.717-am
4.674--779
1689 -.05
90
+,031
0.486:010
0.719
0.618st.017
0.702=s0066
06631
2.571=002
1.217
3.330
0.288
u.293.u 0.
0.328
0.412.n0ee
0.372
6.148:co0c
0.251:c.m
0.186-2001
0. 1.302--400 no1
0.215 004
0.329 ese
D. 0.558-a 0c6
0.430-0006
0. 00308-0000
0.388-00044
0.1634000c
0.266:2100
0,201
0.316
0 0:208=0000
0.320
0.387=a10
0:459:2
0 0,281
=.381
soasc
0,242
u.259 L.
0.364 tee
0.400.a
0.460..1044
0.288
0.382-u104
90
0.220
0.322:5x00
0.3
0.4
0.382
0.550=aa 0.4
0.552
00M
OOM
180
0.240 :cin
0.339 cono
(R(() Weadher344) Traffie/24)
0.343
0.554 0.357=t.04 0.547st.00
0.363 ice
0.678-0005
0.383-0007
0.600=0082
0.337
0.430.coMm
0.284:c0m6
0.58 Lacuee
0.625 0.367
0.721
0.404:3103
0.63
0.358
1.434ccmm
0.289 mcme
0.613 ie
0. 1.382 mn
0.705 1705--ame moe
0.439 19
D. 0.768-400
0.425-am
0.639
0. 0.386-0004 0000
0.469:co0z
0.652ss.00
0.395.0.00
0.686=00.00 020
0.420s0 011
0.959.am0
0.534=0017
ODM
0.526:coxc 90
0.339
0.220:cu01
0,234
0.304: sec
0.377 0.327
017
0.370ua
0.424:3
0:231
0.310
0.281 :c0m
0_329
0.338
0.375
0.370
0.402
0.628
0.560.
0. 0.388
0.427 .os
0.320:c006
0.360:0.00
0.495=8.12 112
0,472=4 oe8
0,474=0 o44
0.461 nes
1.033-0 067
0.768-0004
0 0.442-0607
0.460-0002
30
0.414:cuxe
0.428
0.688
0.580
0.724xxuxce
0.604
2 289
1.303=200
0.823
0.676208
60
0.539
0.531
96
0.022+t001
0.107:5000
0.0
0.15 0.0410=0001 0.15
0.248
0,361=000
0.19
0.340=0005
192
0.043
0.152:cam
0.062sexm
0.194:2
0.193 0.06
0.430uates
0.474:2105
0.357
0.440
336
0.070
0.195
1.087
0.233
0.096..mm
0.246-4.0
0.756-4406.
0.642..00s
0:607-0119
0.600
720
0.112:0001
0.255
0.165 0o6
0.422 0.389 0.317=4006
131
1.073-0001
0.773-0011
0.963-0089
0.772-00.1
L SHOWCASES
In order to more intuitively show Dateformer's prediction results, we visualized the time series ground-truth and predictions of several forecast tasks. The charts are shown as follows.
-10
10
(a) Dateformer
(b) FEDformer
(c) Autoformer
(d) Informer
Figure 8: 3 days prediction cases from ETTm/ oil temperature series
can be seen that Dateformer's predictions are closest to the ground-truth. Compared to other models, Dateformer accurately grasps time series' global pattern: e.g., overall trend and fangerange
seasonality, that is what other models are not good at. Because they can only analyze lookback window series to predict. But the global pattern should be captured from whole training set series.
22

---

Under review as a conference paper at ICLR 2023
334
690
-2.-
30-
-10
400
430
(a) Dateformer
(b) FEDformer
(c) Autoformer
(d) Informer
Figure 9: 3 days prediction cases from ETTh2 oil temperature series
Coartaal
(a) Dateformer
(b) FEDformer
(c) Autoformer
(d) Informer
Figure 10: days prediction cases from Traffic series
Crunthen
14
-2.2
422
-1.00
(a) Dateformer
(b) FEDformer
(c) Autoformer
(d) Informer
Figure 11: days prediction cases from Power Load series
410
196
200
600
WAY 1990 Vii 1000
(a) 30 days
(b) 90 days
(c) 90 days
(d) 180 days
Figure 12: Mid-long-term prediction cases of Dateformer from Power Load series
23

---

Under review as a conference paper at ICLR 2023
M LOCAL PREDICTIONS WITH GENERATIVE STYLE
In Section 3.2 we introduce Dateformer's local prediction component that implemented by the vanilla Transformer. But that the vanilla Transformer is just produce similarities to aggregate series residuals, it does not directly generate local prediction. If we employ local prediction component with generative style, how does Dateformer perform? As basic comparison, we modify Dateformer's s local prediction component to be generative style.
Dateformer- GD:: removing Equation 4 FEN and SoftMax that calculate similarities and attaching FFN to dp+1 to directly generate the local prediction corresponding dp+1.
Dateformer-GR: inputting Date Representations to the Transformer's encoder and inputting Series Residuals to the Transformer's decoder. For the decoder's output {Fi, (2,". ,PP). we use:
F AveragePPoolngg(f::33 f2, ,T))
Local Prediction FFN
directly generate the local prediction corresponding dr-1-
Table 13: Multivariate forecast comparison several Dateformer variants. denotes lacking test samples to report result.
Time Series Datasets
PL
Traffic
Forecast Days
Metric
Ours
GD
GR
Ours
GD
GR
MSE
0,042
0.047
0.047
0.343
0.639
0.642
MAE
0.141
0.153
0.149
0.252
0.384
0.387
MSE MAE
0.076
0.093
0.087
0.430
0.653
0.653
3
0.187
0.211
0.206
0.284
0.393
0.393
MSE
0.093
0.127
0.109
0.434
0.660
0.657
7
MAE
0.211
0.252
0,237
0.289
0.396
0.394
MSE
0.115
0.218
0.144
0.469
0.675
0.675
30
MAE
0.249
0.350
0.282
0.308
0.399
0.400
MSE
0.144
0,297
0.170
0.526
0.680
0.685
90
MAE
0.291
0.433
0.314
0.339
0.404
0.409
MSE
0.176
0.464
0.177
180
MAE
0.320
0.540
0.322
As shown in Table 13 the local prediction component with aggregating style always outperforms with generative style. Due to splitting time series into patches, training samples considerably reduce. That makes the local prediction component with generative style easy to overfit. To mitigate the problem, we introduce an inductive bias: the local patterns of adjacent time series are similar and hence design the local prediction component with aggregated style to aggregate similar local pattern information.
24

---

Under review as a conference paper at ICLR 2023
N
OTHER MODEL'S REMAINDERS
11A
(a) Informer (b) Reformer
(c) Autoformer
(d) FEDformer
Figure 13: Auto-correlation of 4 time series remainders from E7T oil temperature series.
We show 4 auto-correlation series in Figure 5 to provide intuitive view to the characteristic of global and local pattern. That's not for comparison with other models, just to intuitively explain each component's 's function.
But as additional interests, we also provide auto-correlation series of other Transformers' remainders here. As shown in Figure T3a and 13b the auto-correlation series of the remainders produced by Informer and Reformer did not drop to immediately at the left end, which indicates they fail to ac- curately predict time series' mean. And the remainders of Informer, Reformer, and Autoformer still exhibit obvious seasonality that they didn't I capture. Referring to Figure [3d though not obvious, there is also weak seasonality in FEDformer's remainder.
25


Reviews:
[{"review_id": "mt0iZU-kIun", "reviewer": "Reviewer_VVpa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a novel framework that tackles the time series forecasting task by distilling the temporal pattern in the date presentations.  ", "summary_of_the_review": "In this paper, the authors proposed an interesting framework that uses date representation as a container to carry the global pattern contained in the time series. Detailed comments are listed below:\n1. The connection between different model components and each component's functionality is not clearly addressed. \n2. What data is used to calculate equation 1 and the pretrained loss? Do the authors use time series in the training dataset? If so, is the time series for the target day used? Or time series of all time stamps involved in the date-embedding sequence? Is the purpose of DERT to acquire a date representation by building a connection between the training dataset time series and the static date embedding? Can the authors clarify that?\n3. The global prediction component is difficult to follow. First, global prediction is drawn to present a learned global pattern. However, this component feeds on the G copies of one day's presentation. So why these G copies can present the global pattern of the entire training set series? Second, what ground truth is used to train this global prediction component separately? Third, can the authors provide a clear definition of the \"global pattern\" or some examples?", "strength_and_weaknesses": "This paper proposed a novel framework that tackles the time series forecasting task by distilling the temporal pattern in the date presentations.  \n\nStrengths\n1. SOTA methods are compared in the paper. \n2. Comprehensive experiments are conducted.\n\nWeaknesses:\n1. The presentation of the model part is difficult to follow. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hcs_nqBQ_D", "reviewer": "Reviewer_Xz1K", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces the Dateformer – a new transformer-based architecture for time series forecasting which combines embeddings for calendar seasonality along with autoregressive inputs using specialised components for each.", "summary_of_the_review": "Given the difficulties in evaluating the paper, I would suggest that the authors improve clarity of presentation prior to submission. ", "strength_and_weaknesses": "Strengths\n---\nSpecialised components to encode seasonal relationships are relatively underexplored, make the idea behind fusing both temporal as well as seasonal relationships an interesting proposition.\n\nWeaknesses\n---\n\nHowever, there are several key issues associated with the paper that make it difficult to recommend for acceptance in its current form:\n\n1.\tThe paper is presented in a very unclear manner – with many ideas presented only in high-level terms and without explanation or citation, which makes detailed evaluation difficult. See section below for more.\n\n2.\tLack of hyperparameter optimisation – all benchmarks and the dateformer are calibrated using default settings, which makes it difficult to determine whether underperformance is due to improper hyperparams. This is particularly the case for time series data, where optimal hyperparams can vary wildly between datasets?\n\n3.\tThe two-step training process for Dateformer greatly increases the training time versus other benchmarks – can this be combined to be made end-to-end?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YYABwhsguAI", "reviewer": "Reviewer_atRd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work introduces Dateformer for time-series forecasting. The time-series is split into day based patches and the processing is shifted from time-based towards patch based. This work uses time representations as the modeling entity and empirically demonstrates the strength of the proposed approach on multiple datasets. ", "summary_of_the_review": "Overall, this work seems to be in the correct direction for utilizing local and long term global information for time-series forecasting. A structured approach and empirical explorations are provided to strengthen the overall pitch. However, there are some questions that come up such as the use of a fixed size (day based) patch , variations in the type of data between the datasets, and some basic baselines. \n", "strength_and_weaknesses": "Strength : \n\nThis work proposes a structured approach and uses several concepts to create a framework that can be utilized with a very long history to forecast time-series. Furthermore, the performance over several datasets demonstrates the strength of the proposed approach. Overall, this work does seem in the right direction for more robust time-series forecasting. However, there are some questions that come up.\n\n\nWeakness : \n\nThere seem to be some ideas that were not very clear. Such as :\n\n--- As a basic comparison, what would be the performance if the aggregate value (or a domain/data based learned representation) of each patch (a day in this case) is directly used as input to a transformer based forecasting model? The length of the input would be the same as what is proposed in this work so any additional resource issue would be mitigated. \n\n--- Why is it that broadening the window will exhaust model capacity. A sampling approach can be used to pick samples from a broad window and these samples can then behave like a narrow window.\n\n\n--- I am not sure I fully understand Figure 1. Both 1a and 1b seem to have similar trends. Therefore, I am not sure about the exact pitch of Figure 1.\n\n--- Why should equivalent token information be maintained across time-series. It might be beneficial to have variable patch sizes based on the information. Such an approach is useful especially to take into account unexpected events (such as a storm that causes power failure).\n\n--- What is the ground truth and loss used for training the global prediction model ? Is it a time-series value? Does this mean that a time-series value (e.g., power load) is being predicted using generic date representations. I think that I might have misunderstood and perhaps a clarification would be useful.  \n\n--- Most of the datasets are slow moving (except perhaps the economics data). It would be beneficial to gauge the performance of the proposed approach on fast moving data that still has global and local characteristics. This would perhaps again bring into question if using a day as the patch size is beneficial.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ae3qygNKF3", "reviewer": "Reviewer_Q1u4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": " This paper claimed that the global pattern in time series is not well captured in recent time series forecasting methods because they infer the future by analyzing the part of past sub-series closest to the present. To better leverage the global pattern and deal with long term time series forecasting, this paper proposed the following ideas:\n  1. Split time series into patches to reduce token number and tackle long-term series forecasting.\n  2. Distill time representation from training set, propose DERT, a Transformer based date encoder to generate the representation specific to a certain date in the dataset.\n  3. Propose Dateformer, a time-modeling time series forecast framework based on Transformers, to combine the prediction from global time representations and local lookback window.\n \nThe paper's experiments show this method can achieve state-of-the-art results on multiple representative datasets. Especially for a long prediction window, this method can generate results far better than previous methods.\n\nThe contributions are two-fold:\n1. Proposed to leverage a date representation to capture the global trend of the dataset, and combine with prediction from local look back window.\n2. Created the `DERT` and `DateFormer` architecture to utilize the global information and achieved excellent empirical results, especially in long term forecasting.\n", "summary_of_the_review": "Overall, this is a paper that proposed a date representation to better capture global trends in the training dataset. The idea is very interesting and empirical results show promising impact, especially in long term series forecasting. The paper is OK in writing and the idea is presented clearly. Ablation studies show that the global prediction is very critical in the final results and local residuals are helpful in short term prediction. Even though, there're a few incoherent parts in writing and some limitations due to the nature of date representation, this paper is meaningful in understanding and leveraging the global trend of time series. I recommend this paper to be accepted.\n  \n", "strength_and_weaknesses": "This paper proposed an interesting point of view that the current methods cannot well capture the global pattern in the training dataset, like the overall trend and long-term seasonality, etc. \nStrengths:\n1. Interesting idea of capturing the global trend of the dataset using time representation and pretrain over the training set.\n2. The example given is very informative -- while local trends of the time series are very similar, the actual value of the time series is increasing each year. The figures are also well generated, very helpful in understanding the overall architecture.\n3. Strong empirical results, showing improvements in various lengths of prediction, especially in long-term forecasting.\n4. Interesting ablation study, demonstrating the date representation could even be transferable.\n\nThe paper could be better with the following questions resolved:\n1. While `day` could be a very useful patch size, have you explored other patch sizes and how does the performance change with patch size?\n2. It's interesting that global trend of time series can be captured via pretraining the `DERT` encoder and generate time representation for date. I'm wondering if the distance between the training dataset and the test dataset can impact the final accuracy. For example, if the training dataset includes data in 2013, will the testset sample in 2014 generally perform better than 2015?\n3. How does the inference time / memory occupation of the method compare to previous baselines?\n4. It appears not clear to me what other baselines' remainder of ground truth minus prediction looks like, compared to Figure 6(d). The claim of getting \"white noise\" seems not supported well for me. Can you please add figures for baselines?\n5. This method seems to rely on date representation, in that case, does it work for different scales of time series, for example millisecond level / month or year level time series forcasting? Does changing to a different patch size help?\n\nSome minor issues:\n1. Section 1, paragraph 2, change `how do` to `how to`.\n2. Page 2, first line, ditto, change `how do` to `how to`.\n3. Page 3, first paragraph, this is talking about covariates, seems a bit unnecessary, can be mentioned in the appendix.\n4. Section 3, paragraph 1, line 2, change `time series inherent characteristics` to `time series' inherent characteristics`\n5. Figure 3 is taking a huge space in page 4 but it seems simple to describe or demonstrate in a smaller figure.\n6. Figure 5 is a bit chaotic in describing how the autoregressive results are combined with global prediction -- the figure can be better organized to show the work flow. \n7. Table 2, better explain what the bolded results mean.\n8. Second 4, implementation details, change `There are 1 layer` to `There is 1 layer`.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject