DAM: Towards a Foundation Model for Forecasting
Authors: Luke Nicholas Darlow
Qiwen Deng
Ahmed Hassan
Martin Asenov
Rajkarn Singh
Artjom Joosen
Adam Barker
Amos Storkey
Venue: ICLR 2024 poster (2024)
Keywords: Forecasting
time series
foundation model
transfer learning
imputation
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: A neural model for time series forecasting that can ingest randomly sampled long-tail histories from a long-tail distribution, and forecasts by way of adjustable basis function composition.

Abstract: It is challenging to scale time series forecasting models such that they forecast accurately for multiple distinct domains and datasets, all with potentially different underlying collection procedures (e.g., sample resolution), patterns (e.g., periodicity), and prediction requirements (e.g., reconstruction vs. forecasting). We call this general task universal forecasting. Existing methods usually assume that input data is regularly sampled, and they forecast to pre-determined horizons, resulting in failure to generalise outside of the scope of their training. We propose the DAM -- a neural model that takes randomly sampled histories and outputs an adjustable basis composition as a continuous function of time for forecasting to non-fixed horizons. It involves three key components: (1) a flexible approach for using randomly sampled histories from a long-tail distribution, that enables an efficient global perspective of the underlying temporal dynamics while retaining focus on the recent history; (2) a transformer backbone that is trained on these actively sampled histories to produce, as representational output, (3) the basis coefficients of a continuous function of time. We show that a single univariate DAM, trained on 25 time series datasets, either outperformed or closely matched existing SoTA models at multivariate long-term forecasting across 18 datasets, including 8 held-out for zero-shot transfer, even though these models were trained to specialise for each dataset-horizon combination. This single DAM excels at zero-shot transfer and very-long-term forecasting, performs well at imputation, is interpretable via basis function composition and attention, can be tuned for different inference-cost requirements, is robust to missing and irregularly sampled data by design.

Published as a conference paper at ICLR 2024
DAM: TOWARDS A FOUNDATION MODEL FOR TIME SERIES FORECASTING
Luke Darlow, Qiwen Deng, Ahmed Hassan, Martin Asenov, Rajkarn Singh, Artjom Joosen, Adam Barker"
Amos Storkey
School of Informatics
University of Edinburgh Edinburgh, UK a. s..torkeyeed.c ac. uk
Systems Infrastructure Research
Edinburgh Research Centre
Central Software Institute
Huawei
Edinburgh, UK
sirlab@huawei.com com
ABSTRACT
It is challenging to scale time series forecasting models such that they forecast accurately for multiple distinct domains and datasets, all with potentially different underlying collection procedures (e.g., sample resolution), patterns (e.g., period- icity), and prediction requirements (c.g., reconstruction vs. forecasting). We call this general task universal forecasting. Existing methods usually assume that input data is regularly sampled, and they forecast to pre-determined horizons, resulting in failure generalise outside of the scope of their training. We propose the DAM - a neural model that takes randomly sampled histories and outputs an adjustable basis composition as a continuous function of time for forecasting to non-fixed horizons. It involves three key components: (1) a flexible approach for using randomly sampled histories from long-tail distribution, that enables an efficient global perspective of the underlying temporal dynamics while retaining focus on the recent history; (2) a transformer backbone that is trained on these actively sampled histories produce, as representational output, (3) the basis coefficients of continuous function of time. We show that a single univariate DAM, trained on 25 time series datasets, either outperformed closely matched existing SoTA models at multivariate long-term forecasting across 18 datasets, including held- out for zero-shot transfer, though these models were trained to specialise for each dataset-horizze combination. This single DAM excels at zero-shot transfer and very-long- forecasting, performs well imputation, interpretable via basis function composition and attention, can be tuned for different inference-cost requirements, robust missing and irregularly sampled data by design.
INTRODUCTION
Time series forecasting can have a positive impact in a number of domains, including weather, traffic, finance, electricity, and cloud resource management (Wu et al.) 2021 Lai al. (0188). Most state- of-the-art (SoTA) forecasting methods assume fixed-length common-interval sequences (Nie et al.]
2022 Zeng et 2023 Lim & Zohren| (02T). otherwise known 'regular time series' (Rubanoval et al. 2019). However, this does not scale well for many practical applications, particularly where the data generating mechanism complex and varies over time. One example application is work- load forecasting for cloud computing, where single cloud provider can have tens or hundreds of thousands of time series workloads with diverse characteristics, differing length, and discontinuous data due to monitoring failures and outages (Sloss et al. Taylor & Letham| [0188 Darlow et al.) 2023 Joosen et al., 2023). Predicting at this scale requires more generalised forecasting methods as it is infeasible to train or tune model for each time series.
Existing methods fail to generalise outside the scope of their training. We argue that some of the key reasons for the ubiquitous poor generalisation in time series forecasting are that existing methods
Asso working the School of Computer Science, University of St Andrews, UK.

---

Published as a conference paper at ICLR 2024
sume (I) that input data is fixed-length and regularly sampled (i.e., evenly spaced and ordered), and (2) a pre-determined forecast horizon. It is common for existing methods to model future values di- rectly as vector output of fixed-length. Relaxing these assumptions enables universal forecasting an approach to scaling forecasting methods such that they are applicable across domains and gener- alise new datasets. Universal forecasting must be robust the underlying collection processes of data (e.g., resolution or continuity) and cross-domain differences (e.g., seasonality or stationarity). In this paper, we aim solve the challenge of designing single model that can forecast accurately for variety of time series datasets.
We present the deep data-dependant approximate analytical model (DAM) as significant step towards foundation model (Bommasani et al., (2021) for universal forecasting. The DAM is a neural model that takes in randomly sampled histories (Section 3.2) and outputs time-function fore- casts through an adjustable basis decomposition (Section 3.3). It uses a transformer backbone (Sec- tion 3.1 to ingest time series that are irregularly sampled, and forecasts via a continuous function of time, meaning that it can be applied across diverse domains with differing forecast requirements.
To the best of our knowledge, the DAM the first model for universal time series forecasting that can be trained simultaneously across many diverse datasets, of different resolutions, and for various horizons, such that it generalises well both within and outwith the training set. We trained it on 25 publicly available datasets, totalling 2280 univariate time series (over 44 mil. samples). A single DAM outperformed most specialised SoTA methods at long-term forecasting, was superior at very long-term forecasting and at imputation, and outperformed SoTA methods held-out datasets even when those methods were trained on those datasets. Our contributions can be summarised as:
1. The design and implementation of the DAM (Section 3
2. A new and flexible approach for using actively sampled histories (Section 3.2 a long-tail sampling method that enables efficient access to the distant past for a global perspective of the underlying signal, while maintaining focus on the recent past.
3. Forecasting via continuous basis functions, where the coefficients are the output of the DAM (Section 3.3). Such function is not constrained by pre-determined horizon, thus enabling longer term forecasts (Section 4.3) and past reconstruction (Section 4.4).
4. Demonstrations of: stable and performant forecasting in very long-term settings, flexible inference cost, transfer to held-out data, and interpretability (Section
2 RELATED WORK
PatchTST (Nie et al. 2022) is a modern and performant transformer method that operates by encod- ing regularly sampled patches of ceanellindeeenden (i.e., univariate) time series data into tokens for attention. DLinear (Zeng et al. 2023) decomposes the time series signal into trend and seasonal components, applies linear layers to these, and sums the resultant forecast. N-HiTS (Challu et al. 2023) is an extension of NBEATS (Oreshkin et al. 2020), both of which effectively forecast by way of multi-scale neural basis composition. The neural basis that NBEATS and N-HiTS use different from the basis functions that the DAM uses; neural basis are learned weighted connections between past and future as opposed to continuous function of time.
Multi-scale modelling. TimesNet (Wu et al. 2023) and MICN (Wang et al., 2022) both use plicit multi-scale mechanisms for effective forecasting, breaking the task up into multiple scales over which convolutional neural networks can operate. Pyraformer (Liu et al. 2021 and Crossformer (Zhag & Yan, 2022) use custom hierarchical attention mechanisms for mu -i-le modelling. In- former (Zhou et al. (202T) uses sparse attention improve efficiency. LighTTS (Zhang et al.) (022) applies careful down sampling strategies and MLP layers to model at multiple scales. The per- formance of explicit multi-scale methods evidence that multi-scale modelling paramount for accurate forecasting. The DAM also operates at multiple scales via its basis function composition, but can also access the distant past to model at longer scales because of the history sampling regime.
Frequency-domatee modelling. Autoformer (Wu et al. 2021) uses autocorrelation based atten- tion mechanism for forecasting. Zhou et al. (202bb) argued that frequency domain information en- ables more global perspective of time series, proposing Fedformer that uses 'frequency enhanced
2

---

Published as a conference paper at ICLR 2024
block' and Fourier projections to act in the frequency domain. Zhou et al. (2022a) used Legrende polynomials and Fourier projections to model and denoise historical data for FiLM. ETSFormer (Woo et al. 2022 uses two attention mechanisms that utilise (1) exponential decay to explicitly bias toward recent history and (2) high amplitude fourier components. The DAM operates both frequency and time domains simultaneously and utilise distant history (Fourier projections not suited to irregular data (Lomb 1976)). More related work can be found in Appendix
THE DAM, EXPLAINED
The DAM is model for universal forecasting. We designed it such that single model can be used for many time series datasets and across domains. It uses a transformer to ingest context data sampled from long-tail distribution, called the history sampling regime (HSR), and returns the coefficients of basis functions. These define the shape of a continuous function of time, f(t). The DAM trained estimate this function from actively sampled past data for any past future t.
3.1 BACKBONE
PAST FUTURE Mvlly bo Moliii
Forecast
Sample
using HSR
f(t)
Backcast
deeigron sum Kanctiona
Fit iniiial
Embod
Attertion
MHCAA
Compose
coefficiants
TV-tokans
LN
Updaled
B-lokerre
Merged
FVAtioen
O
Project
0u
Embed
Linear
LN
Trarnosnnn
lncated confficerts
12)
Uoditnd
LIN
TVlokken
1 Context time-value samples from the HSR (Section 3.2 are sent to linear solver to initialise the basis coefficients, 9g. These are 3 embedded into ④ B-tokens. Context data is also embedded into 6 TV-tokens and processed through 7 4 layers of MHSA, ToME, and forward blocks, with layer-norm, and used as 8 keys and values for cross attention, where the queries are the B-tokens. Both TV- and B-tokens are 10 passed to proceeding layers. The ①1) B-tokens from the final layer are projected into 12 basis coefficients for 11 forecasting and backcasting.
Figure 1:
Figure D shows the DAM architecture. Dmodel is the latent dimension of the model. The DAM takes input: (1) univariate time-value tuples sampled from the HSR (Section 3.2), with time units of days (i.c., St = is a day interval). and (2) initialised basis coefficients with corresponding fre- quency information (Section 3.3). Time-Vale pairs are embedded into what we call 'TV-tokens' (using transformer nomenclature) and initialised Basis coefficients are embedded into 'B-tokens'. Another TV-token initialised with 50 evenly-spacee percentiles of the values for affine adjust- ments. Appendix B gives code listing for the full model architecture.
Model structure. After embedding, the DAM uses 4 layers of processing, each consisting heads of multi-head self-attention (MHSA) (Vaswani et al. 2017) for TV-tokens; 4 heads of cross- attention for B-tokens where keys and values are T'-tooken;; 3 separate feed-forward blocks (lin- ear > GeL.U (Hendrycks & Gimpell 2016) -> linear, across D for TV-tokens, the affine token, and for B-tokens; an additional ferw-forward block acting across B-tokens (FFB B.coss in Figure []: multiple layer normalization (LN) layers; and token merging (ToME) (Bolya et al. 2022). ToME used to reduce the number of TV-tokens during each layer of processing. This backbone is simple compared to earlier methods: it uses standard MHSA, not time series-specific variant (Wu al.
2021
Zhou et al.) 202I); data need not be regularly sampled to yield continuous 'blocks' (Nie et al.
2022
explicit multi-scale mechanisms are required (Wu al. 2021 2023 Wang al. 2022
3

---

Published as a conference paper at ICLR 2024
3.2
HISTORY SAMPLING REGIME: NEW TREATMENT OF TIME
willigg
PAST FUTURE
Now; t-0
Regular targets
HSR distribution Regular context
HSR context samples
HSR target samples
Figure 2: The HSR employed by the DAM, with the distribution in Equation I shown in yellow. Regularly sampled context and targets of the same size as those from the HSR are shown demon- strate how the HSR enables more global perspective while retaining focus close to 'now" (t = 0).
Most existing forecasting models were designed to process regularly sampled data of fixed length. We argue that this restriction is one of the main reasons reason for poor generalisation in time series forecasting. The DAM can ingest irregular and variable-length data, making it suitable to a broader variety of time series datasets and domains. Consequently, this means that the DAM can be trained easily on large collection of time series datasets, thus mitigating early overfitting common in time series forecasting. The DAM uses a long-tut tail distribution over time steps, x t/ R., where the sample resolution (e.g., hourly). We call this the history sampling regime (HSR):
1
Phat (2)
(1)
1+
2
where (2) has the normalisation constant e e= ExEx(1+ =+)= and the sample support. a is the HSR 'width', where smaller biases sampling recent past more than distant past. The HSR is used to sample æ, from which (t, v) tuples are built, where v is the value at time t. The HSR is used for both context data from the past (x < 0) and target data (any 2). The number of points is variable and need not be the same for training and inference. Figure @ gives an intuitive perspective of the HSR and demonstrates how the same-sized HSR-based context gives access more global perspective of the signal than the strictly regular context. This distribution was chosen because it increases the likelihood of sampling the distant past, enabling global perspective of the signa while ensuring the majority of samples are recent. We list additional advantages in Appendix C
3.3 FORECASTING MECHANISM: BASIS FUNCTION COMPOSITION
The DAM forecasts using basis functions. We selected 437 frequencies from I minute (Ti days) to 10 years (+55- - 7 10 days) by concatenating even samples in the minute, hour, day, week, and year ranges. These ranges were selected for wide basis function coverage (see Appendix D). The basis function composition that the DAM uses as a forecasting mechanism is:
52-7-10
IQQ a
Σ 1440
O2,1 sin (2xvl) Ou,2 cos (2mvt)
- + MED,
where 0 is the output vector from the DAM containing basis function coefficients, E v is the frequency. 6,, and 0,,2. are coefficients for sine and cosine functions at frequency v,, which allows the DAM to smoothly capture all sinusoids between and 2v (Lomb] 1976). Median (M FD) and inter-quartile range (TQR) are computed per-datum for online robust standardisation. and b are affine adjustments also output from the DAM (Kim et al., (021). Other methods also leverage basis functions (Oreshkin et al., 2020 Challu et al.) 2023 2022; Triebe et al., 202I), but these typically use some form of implicit basis (i.e., within the model structure) instead of an explicit composition. Our approach has two major advantages: (1) Equation 2 has no fixed horizon and can be assessed for any t (see Section 4.3). and (2) basis functions naturally interpretable (see Section 5.1)
Basis function initialisation. We found empirically that was advantageous to initialise B-tokens with basis coefficients fit to the context. To this end, we use a linear differential equation solver to find initialisation coefficients, Og- Appendix D gives a code listing for this initialisation. Figure shows how O, is capable of representing the past sufficiently when coupled with the HSR, but fails at future extrapolation, meaning this initialisation strategy alone is insufficient for forecasting.
We determined empirically that this distribution performed better than Gaussian and Uniform distributions.
4

---

Published as a conference paper at ICLR 2024
(ruth (past]
Ground truth (future)
past
Basis inttialication, extrapolated to future backcast
The DAM Forecast
Figure 3: Basis function initialisation versus the DAM, showing past fit and future extrapolation.
3.4 TRAINING
We used the Huber loss for training (Huber 1992). computed over targets sampled from the HSR that include both past and future. Thus, the DAM trained reconstruct and forecast. The number of context points sampled from the HSR for context and targets was set to 540. was set to 720 during training. Before aggregation, the loss was re-scaled element wise using an exponential decay of 2 2-33300 (by 0.5 time step 360 empirically determined). We trained the single DAM used in this paper on 25 time series datasets, 10 of which are common benchmark datasets used for evaluation. Training commenced over 1, 050, 000 iterations. Appendices and give more details.
3.5 INFERENCE PROCESS
During inference, the DAM needs only time-value pairs from given variable, n, order to predict for given time, f, where r are the time steps (effectively indices into the past). Inference involves: (I) using the HSR probability defined over z, sample indices using a weighted random choice with- out replacement, and extract time-value pairs from the sampled indices; (2) compute O, from context, where O, is input to the model and not one of the model parameters; (3) forward pass to produce 0; and (4) compute basis composition at any other query times. Note that the DAM always operates in a univariate fashion (called 'channel independence' by recent works (Nie et al. 2022 all multivariate forecasts were generated by forecasting each variable separately.
HSR tuning. A significant advantage of using the HSR is that its settings (context size and σ)) can be set after training for better forecasting. To this end, we estimated the mean squared error (MSE) per-dataset (on the validation split) for range of context sizes and values. Section 4. shows results with and without this tuning while Appendix G has more details on the tuning.
4 EXPERIMENTS
We used a total of 33 datasets for training and evaluation. We augment 10 commonly used datasets (following e.g. Wu et al. 2021] Zhou et al. [202]] Liu et al. 2021) that we split into train/valid/test, with another 13 datasets that are additionally used to enhance training (details in Appendix HH The 10 datasets are used to test within-dataset generalisation (Section 4.1p and they are: ETThi. h2, ml, and m2; ECL.; Traffic; Weather; USWeather; Exchange; and Wind. In Section 4.2 test outwith generalisation on 8 held-out datasets, namely: Illness, Weekdays, UCIPower, Azure, MTemp, MWeb, MWeather, and MMeters. In Section [.3 we test the DAM on very-long-term forecasting, and in Section 4.4 we demonstrate how it can be used for imputation.
4.1 LONG-TERM TIME SERIES FORECASTING
Results and discussion. Table D gives the multivariate long-term forecasting results for the DAM against SoTA methods (average of 3 seeds). DAMHSR-nnel denotes when we used optimal HSR values based on validation set performance (Section 3.5 Appendix G). Baselines were trained to specialise on dataset-horizon combinations as designed, meaning that each baseline required 40 unique variants, while only trained one DAM. This setup provides best-case scenario for these models' performance and represents the current SoTA gamut in forecasting. The DAM achieves SoTA performance (1st) across 39 of 80 metries and is comparable to SoTA others. The
5

---

Published as a conference paper at ICLR 2024
closest competitor was PatchTST with 28 1st Normalised MSE and MAE results and results on an additional 8 SoTA methods can be found in Appendix I1 with visualisations in A We only trained one DAM for all the results in Table | The DAM was designed for generality, and while it has more data to train on compared to other methods because of this, forecasting across many dataset-horizon combinations is a more challenging task than specialisation.
Long-term multivariate forecasting. Mean squared error (MSE) and mean absolute error (MAE) are shown. Context size set to 720 for the DAM (ToME reduction to 333), and varied per dataset for DAMasR--und- Gold, silver, and bronze are 1 2nd and 3rd place per -ortii-- dataset combination. The DAM's placing is tallied simultaneously for both standard and HSR-tuund.
Table 1:
DAM
BAM ound
PuchTSTO1
DLinear
N-IGTS
Crossoomer
Pyralormer
MKCK
Horizon
MAE
MSE
MSE
MAE
MAE
MSE
MAE
FTTM FTTh FTTmm2 FT7ml
192
96
0,309
0.3335
0.358
0,377
0.308
0.543
0.356
0.377
0.334
0.363
0.370
0.353
0.536
0.301
0.344
0.305
0.402
0.352
0.381
0.418
0.373
0.452
0.475
0.416
0.17
0.620
0,589
0.529
0.361
0.319
0.370
0.397
336
0.384
0.351
0.364
0.390
0.370
0.387
0.443
0.447
0.685
0.628
0.749
0.630
0.405
0.431
720
0.403
0.418
0.407
0.418
0.416
0.27
0.492
0.455
0.991
0.729
0.975
0.736
0.506
0.497
96
0.175
0.252
0.170
0.251
0.166
0.256
0.169
0.263
0.319
0.295
0.342
0.412
0.354
0.441
0.185
0.282
192
0.2222
0.288
0.220
0.287
0.222
0.295
0.228
0.304
0.322
0.362
0.670
0.600
0.640
0.634
0.275
0.349
0.297
0.50
0.115
0.424
1,050
0.715
0.385
720
0.331
0.364
0.325
0.362
0.361
0.383
0.309
0.430
0.619
0.562
3.727
1.361
3,01
1,528
0.500
0.531
96
0.375
014014
0.367
0.401
0.372
0.401
0.579
0.20
0.42
0.411
0.397
0.42
0.721
0.641
0,408
0.426
192
0,422
0
0.431
0.433
0.458
0.479
0.422
336
0,400
0.427
0.396
0.419
0.422
0.444
0.455
0.447
0.555
0.506
0.671
0.608
1.019
0.796
0.570
0.530
720
0.438
0,462
0.411
0.443
0.458
0.474
11407
0.508
0.592
0.546
0.812
0.688
0.999
0.795
0.807
0.671
96
0,300
0.356
0.250
0.330
0.276
0.339
0.29
0.356
0.425
0.438
0,746
0.608
1.521
0,955
0.374
0.415
192
0.570
0.238
0.339
0.575
0.533
0.493
1,569
0.939
5.107
1.817
0.496
336
0.369
0.375
0.346
0.376
0.364
0.403
0.451
0.463
0.572
0.523
2.360
1,240
4,652
1.824
0.620
0.552
720
01:99]
0.404
0.392
0.420
0.391
0.430
0..90
0.592
0382
0.665
3.379
1.569
4,505
1.789
0.810
0.648
0.324
0.183
0.166
ESSWorteel Wind Etchange Weother Tarfic RCT.
192
0.176
0.250
0.171
0.148
0.242
0.156
0.189
0.286
0.162
0.262
0.295
0.395
0.177
0.257
336
0.181
0.285
0.176
0.279
166
0.260
0.171
0.271
0.208
0.302
0,205
302
0.306
0.400
0.186
0.207
730
0.242
0.337
0.237
0.331
0.200
0.202
0.206
0.304
0.342
0.336
0,253
0.338
0,305
0.393
01500
0.316
0,408
0.335
0,400
0,360
0.249
0.412
0.544
0.226
0.18
0.267
0.600
0.395
0.513
0.310
192
0,481
0.362
0.474
0.399
0,800
0.257
0.425
0.291
0.550
0.328
0.555
0.285
0.675
0.380
0.536
0.317
336
0.486
0.344
0.479
0.341
0.392
0.264
0.438
0.500
0.571
0.334
0.579
0.299
0,673
0.377
0.548
0.322
720
01.47
0.381
0.538
0.305
0.408
0.611
0.351
0.663
0.358
0.397
0.572
96
0.16
0.203
0.154
0.203
0.148
0.174
0.235
0.157
0.115
0.172
0.340
0.196
0.284
0.193
0.250
192
0.193
0,238
0.191
0.238
01193
0.241
0.216
0.274
0.189
0.252
0.223
0.294
0,231
0.315
0.234
0.291
336
0,205
0.248
0.201
0.347
0344
0.282
0.261
0.311
0:237
0.269
0.277
0.341
0.295
0.364
0.287
0.356
720
1305
0315
0.333
0.325
0.355
0,388
96
0,087
0.211
0.000
0.216
01993
02266
B1089
0.214
0.145
0.273
0.271
0.377
0.673
0.661
0.093
0.227
192
0,173
0.29%
0.178
0.302
0.231
0,348
0.163
0.300
0.299
0.455
0.489
0.520
0,878
0.756
0.186
0.333
336
0.2044
0.435
0.317
0.560
0.558
0.761
0.330
0.443
720
0.724
0.893
0.922
0.756
0.974
0.949
0.743
1,585
1.654
1,022
0.794
96
0.202
0.257
0.203
0.252
0.177
0.208
0.196
0.227
0.185
0.278
0.170
0.215
0.167
0.206
0.175
0.219
192
0,213
0.261
0.217
0.261
0.191
0.220
0.215
0.24]
0.198
0.240
0.188
0.227
0.185
0.220
0.192
0,230
336
0.220
0.229
0.309
0.202
720
0,230
0,282
0.247
0.287
0.206
0,253
0,252
0,274
0.232
0.263
0,210
0.238
0.214
0.245
0.220
0.246
96
0,477
0.466
0.478
0.462
0.469
0.476
0,472
0,450
0.508
0.501
0.453
0.466
17.47
0.475
01.76
014933
192
0,520
0.520
0.518
0.541
0.502
0.524
0.522
336
0.503
0.537
0.537
0.540
0.5%6
0.576
0.553
0.526
0.527
0.541
0.540
0.546
720
0.500
0,554
0.601
0.549
0.629
0.576
0.615
0.579
0.613
0.579
0.571
0.556
0.570
0.559
(154)
01540
Average
0.330
0.354
0.330
0.551
0.331
0.350
0.352
0.420
0,406
0.521
0.992
0,640
0.396
0.398
2""55"""
MSEx:16,17.1 MAEs:
23,20
14,599
145.13
2,116
3,7.7
100.2
0,0,
2,533
1,7,6
4,2.1
3,022
2/03
1,2.6
Limitations. It is common for other methods to use a low epoch count to mitigate overfitting (Wul et al. 2021 Zhou et al. 2022b 2021), which is a sub-optimal strategy when training large neural networks. While the training cost is relatively high for the DAM when compared to other forecasting models, this may indicate that it can learn more complex and useful patterns from the data. Nev- ertheless, the DAM requires more training than specialised models. Datasets with sharp changes (spikes), such as traffic, electricity, and wind, are challenging for the DAM because representing them with basis functions is difficult. The DAM is univariate, and while modern methods ad vooccate for 'channel independence" mitigate overfitting (Nie et al., 2022), the DAM does not evidence early overfitting. This suggest cross-variabl information could be useful but remains inaccessible.
4.2 FORECASTING ON HELD OUT DATASETS
A foundation model should transfer well within scope but outside of its training datasets. Model transfer is typically conducted using either fine-tuning- -va short training on target datasets- or in zero-shot mode, without any additional training. We tested both protocols on 8 datasets held out during training, from the Monash time series repository (Godahewa et al. (02T), the UCI dataset repository (Frank 2010), and Azure (Shahrad et al. 2020) (Appendix H.3 Table lists transfer
6

---

Published as a conference paper at ICLR 2024
performance of the DAM vs. PatchTST, Dlinear, and NHiTS baselines, including these baselines trained from scratch. These 3 methods are effectively univariate and can therefore be tested on zero- shot transfer We report zero-shot test performance of baseline model variants trained for Table I after selecting the best-performing models on validation sets. Thus, the results for these three baselines are those which are optimal across the 10 training datasets listed in Table
Table 2: Results on held-out datasets averaged over 4 horizons. Zero-shot performance is shown for the DAM vs. 3 SoTA baselines, alongside fine-tuned (the DAM) and standard training (baselines) for comprehensive comparison. Overall best performance is shown in bold. indicates average error (not validation-base selection) because the dataset was too short for the required context.
Zero-shot
Fine-tuned
Trained from scratch
DAM
PatcTTST
DLinerr
NIETS
DAM
PatchTSTed
DLinear
NIRTS
MSE
MAE
MSE
MAE
MSB
MAG
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Filress
1.964
0.920
4.356
4.733
1.400
2019
0.925
1,884
2.357
2.338
1.041
Weekdays
0.942
0.573
1,502
1.135
0.672
0.47
0.578
0.898
9.736
0.582
0.418
UCIPower
1.395
0.514
1,590
0.345
1,615
0.370
1,897
0.625
1.623
0.522
1.501
0.524
1.528
0.552
1.520
0,538
Azure
1.531
0.640
1.630
0.867
1.444
0.649
1353
0.607
1.312
0.510
1,489
0.558
1618
0.649
1.441
0.604
MTemp
0.950
0.975
0.637
1.029
0.804
0.595
MWab
12,633
12.656
13.321
0.732
0.678
12.512
0.603
12.831
0.632
14.
0.927
13,920
0.726
MWeather
0.479
0.500
0.300
0.792
0.862
0.769
1.177
0.895
0143
0.456
0.423
0.463
0.428
0.452
0.456
0.475
MMeters
0.836
0.499
0,805
0.546
0.940
0.566
1,057
0.657
03349
0.422
0.794
0.498
0.807
0
0.852
0.536
The DAM achieves SoTA zero-shot transfer on 14/16 metrics, and even outperforms baselines that were trained from scratch on some datasets. The DAM generalises well outside of its train- ing set. In some cases with very short datasets (e.g.. MTemp) even baseline methods outperform their variants trained from scratch, showing how brittle performance can be when training on small datasets. Details of the fine-tuning process and standard training in Appendix K
4.3 VERY LONG-TERM FORECASTING
Some scenarios demand very-long-term forecasting, owing to dataset resolution or user require- ments. For example, forecasting to one week for the Weather dataset (with a resolution of 10 min- utes) requires horizon of 1008 steps, which is beyond what is typically considered 'long-term' in existing works (i.e.. 720 time steps). Since the DAM is agnostic to the horizon, producing very-long- term forecasts requires no additional training: we simply evaluated the same DAM used throughout this paper. Figure 4 shows forecasts over 5000 steps (+:35 days) for the weather dataset, comparing against PStchTST and DLinear trained on this horizon. Evidently, these baseline methods capture the strong daily periodicity and some attempt at a long-term trajectory but fail to produce meaningful or performant very-long-tent forecasts. The results in the accompanying table show how the DAM performs well compared to SoTA methods, even when the latter are trained for these horizons.
MSE
Datavrlaras:
DAM PatchTST DI iincar
Ground truth
PatchTST64
0.755
0.835
DLinear
The DAM
0.6
ETTh221s0
0.529
0.379
0
ETTmlgss
0.747
0.858
0.770
ETTm22550
0.586
0.4
1904
0.375
0.296
0.295
Traffiessz
0 0.604
0.510
0.524
L. LSWeathergan
0..77
1,251
0.766
0.2
Weathersaco
0.524
Windsass
0.515
0.446
Average
0.564
0.650
0.746
2n.k geds
5,13
1,5,3
3,2,4
10 15
20
25
10
20
30
Time/horizon (in days)
Very long-term forecasting. The "OT' variable of Weather on the left and MSE versus horizon in the centre. The DAM produces better performing forecasts that also contain interesting multi-scale patterns, compared baselines. To produce these figures the DAM context was set to 512, matching PStchTST.. The inset shows from -512 to 720 steps. MSE vs. very-long horizon on 9 datasets given the table. Horizons were set according to 3/4 the length of the validation set.
Figure 4:
Figure 4 also demonstrates how a regular context is limited when compared to a context built using the HSR, which enables a more global perspective at the same cost (512 samples in this case).
Thhe are also the top 3 baseline models in Table strengthening the findings of Nie et al. 2022 regarding univariate models heing preferred over multivariate models, possibly owing to overfitting.
7

---

Published as a conference paper at ICLR 2024
4.4 HELD OUT TASK: IMPUTATION
Time series imputation important in cases where regu- larly sampled data is necessary. Upon release, TimesNet (Wu et al. 2023) evidenced SoTA performance on the im- putation task. 3 results using only basis func- tions with 8g coefficients. No training of the backbone even required in this case because the initialisation efficients are optimal for past data. Unlike Oo. the basis coefficients the DAM outputs, 0, are better suited fore- casting and reconstruction than imputation.
Table 3: Average imputation MSEs for
12.5, 25, 37.5, and 50% missing data.
Meric
MSE
MAE
MSE
MAR
ETThi
0.1324
0.115
0.2336
ET1ml
0,256
0.1000
0.0369
0.1207
ETim2
0.1135
0,0700
0/0274
0,0+C
BCL
0.0854
0.1283
0.0945
0.1934
Weather
0.0222
0.0463
0.0300
0.0829
In 3 we masked entire columns (i.e., all variables) time step as this is reasonable reflection of what may cause missing data and thus require imputation or reconstruction 3
5 ANALYSES AND DEMONSTRATIONS
5.1
INTERPRETABILITY
Partt data
data Samples
Bar ant
Forecast
Cross attention
Self attention
Layer head
Time in days
Coomerent
0.50
0.25
41.0 mg
20hr
4.
hr
6 hr
80
200
320m
140
200H br
22.0z hr
1.0 dy
1.1 dy
dy
dv
dy
oy
dy
09
1.0 dy
2.0 dy
by
C. 5y
0.0
507
Period basis weight
Figure 5: Attention analysis showing HSR samples, backeast, forecast, past and future data (ETTh) test), and cumulative attentions per TV-token. The degree of attention paid is colour-coded and normalised for each attention head. Basis coefficients per period are also shown.
55
78
100
by
Past data
Future iptta
Backcost
Figure 6: Low-to-high frequency basis composition on the 'OT" variable of ETTm1. From left to right, the percentage labels denote the percentiles of frequencies used to compose the forecast.
Figure [ demonstrates the DAM's interpretability, showing self- and cross-attention weights, and basis coefficients for ETTh1. The cumulative attention for each layer is shown behind the data and forecast. Cumulative attention is the sum of the attention weights paid to individual TV-tokens. Each attention head is performing a different task in order to build the prediction. High coefficients correspond to the periodicity in this dataset (e.g.. daily and weekly). The high coefficients for low frequencies are worth noting since this dataset spans less than 2 years. This evidences that the DAM
Thhe results for TimesNet are worse than those published by the authors. This is because their multivariate masking procedure applied randomly over 2D tensor instead of for all variables sampled times.
8

---

Published as a conference paper at ICLR 2024
is able to capture longer term trends (thanks to the HSR) even when only small part of that trend presents itself the data. Figure shows how the DAM composes its prediction from low-to-high frequency components. Appendices M and contain demonstrations other datasets.
5.2 ARCHITECTURE ABLATION STUDY
ggive the results when ablating components of the DAM architecture (see Figure namely: self-attention, cross-attention, feed-forward (TV), feed-forward (B), feed-forward in the coefficient dimension (B, cross), and ToME reduction. We ablated these components by skipping each one during the forward pass. The results suggest that each component necessary. but that the feed- forward block acting across basis coefficients FF. crucial. B. is the primary mechanism for updating B-tokens, with the attention components acting to share information and enhance predictive performance, hence the strong reliance this component.
Table 4: Ablation study showing average results over 4 horizons for the ETT datasets. The first row denotes which component is removed. Performance drops are highlighted red gradient.
Nothing
Self-Amn
Cross-Attn
FFTv
FFB
PFB.crox
ToME
MSE
MAF
MSF
MAF
MSE
MAF
MSE
MAF
MSF
MAF
MSE
MAE
MSR
MAE
ETThi
0,405
0.404
0.439
0.469
0.606
0.539
0,483
0.493
0,399
0.561
31.099
2.797
0.423
0,478
ETTh2
0.355
0.371
0.393
0.437
0.469
0.349
0.416
0.434
0.432
13.411
1.688
0.359
0.409
ETTml
0.352
0.385
0.411
0.439
0.642
0.528
1,354
0.498
0,706
0.531
51.323
3.519
0.364
0.425
ETTm2
0.240
0.300
0.335
0.392
0.574
0.473
0.258
0.380
0.316
0.387
18.481
2,090
0.241
0.366
5.3 FLEXIBLE INFERENCE COST
Figure 7 shows the relationship between performance (measured MSE) and inference cost (forecast time in milliseconds): MSE reduces approximately exponen- tially, while cost increases linearly. The DAM's flexi- bility to context size and forecast horizon make it casier to deploy at scale because can be tuned according to inference cost-requirements. For instance, now-resource environments or edge devices could use a smaller con- text size. The same model be scaled up down smoothly without loss of generality.
Forecast time (mo)
200
0.46
100
400
600
1000
2000
Context
Figure 7: Inference cost performance
on ETThl (minibatch size of 100).
CONCLUSION
We presented the DAM significant step towards foundation model for universal forecasting, It uses transformer backbone to ingest randomly sampled history data (from a long-tail distribution), and outputs continuous function of time via basis function coefficients. The DAM overcomes the practical challenges associated with training single model to forecast across datasets and domains, where differences in collection processes and data-generating mechanisms result in irregularly sam- pled data with significantly different patterns, seasonality, stationarity, and continuity. The DAM is flexible regarding data structure, thus enabling the use long-tail history sampling regime for an efficient global perspective of the time series signal. All results in this paper were computed us- ing single univariate DAM. This single DAM outperformed existing specialised SoTA methods on 39 of 80 dataset-horizon combinations, with the nearest competitor winning 28 of 80. Results on held-out datasets show that the DAM performs well at zero-shot transfer, even when com- pared against baseline models have been trained on the target datasets. The DAM also excels at very-long-tere -erm forecasting, as demonstrated a practical scenario for weather forecasting, and performs well imputation. We also demonstrated how the DAM is interpretable via basis compo- sition and attention and fost-flexible during inference. Future work will entail scaling up the model architecture and the training set in order to fully leverage the advantages of this foundation model.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
We included a simplified version of the PyTorch code for the DAM in Appendix [ and for initial- ising basis coefficients in Appendix [ We also provided this supplementary material for ease of use. Details on model and training hyper-parameters are given in Appendix F A full working code repository will be released in the near future.
REFERENCES
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your ViT but faster. arXiv preprint arXiv:2210.09461 2022.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv: Ttt::2008.02.588 2021.
Cristian Challu, Peihong Jiang, Ying Nian Wu, and Laurent Callot. SpectraNet: Multivari- ate forecasting and imputation under distribution shifts and missing data. arXiv preprint MiXiv:2210.22115. 2022.
Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza Ramirez, Max Mergenthaler Canseco, and Artur Dubrawski. NHiTS: Neural hierarchical interpolation for time series forecast- ing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 1989-69977 2023.
Song Chen. Beijing PM2.5 Data. htpps:/dbb.org/112244/////
UCI Machine
Learning Repository, 2017a. DOI:
Song Chen. PM2.5 Data of Five Chinese Cities. UCI Machine Learning Repository, 2017b. DOI: http://doo.org/1.22443//////2///
Luke Nicholas Darlow, Artjom Joosen, Martin Asenov, Qiwen Deng, Jianfeng Wang, and Adam Barker. FoldFormer: Sequence folding and seasonal attention for fine- ane-grained long-term FaaS forecasting. In Proceedings of the 3rd Workshop on Machine Learning and Systems, pp. 71-77, 2023.
Edward Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. GRU-ODE-Bayes: Continu- ous modeling of sporadically-obsevee time series. Advances in neural information processing systems, 32, 2019.
Andrew Frank. Uci machine learning repository. http:/arccive. ics. uci. edu/ml, 2010.
Elias Frantar, Carlos Riquelme, Neil Houlsby, Dan Alistarh, and Utku Evci. Scaling laws for sparsely-conneeted foundation models. arXiv preprint arXiv:2309.08520, 2023.
Rakshitha Godahewa, Christoph Bergmeir, Geoffrey I. Webb, Rob J. Hyndman, and Pablo Montero- Manso. Monash time series forecasting archive. In Neural Information Processing Systems Track on Dalasets and Benchmarks, 2021. forthcoming.
Google. Web traflic time
series forecasting.
https: www. www.wa.glle..m com/c c/ 2017. Accessed: 2023-11-15.
web web-t
Australian Governmen.
Historical data
rainfall and weather forecasting
temperature forecast
and observations hourly
verification
data
(2015-05 to 2016-04).
httpp::/daat gow. au/ data/ dataset/
weather- forecastingggggoee verification iww.ecssttinnveeifiatiiiio data 2015-0011200 00015-05-ta0 05- 2016-04 04 Accessed: 2023-11-15.
2017.
Georges Hebrail and Alice Berard. Individual household electric power consumption. UCI Machine Learning Repository, 2012. DOL: htpp://doi..rg/1122445////
Dan Hendrycks and Kevin Gimpel.
Gaussian error linear units (GeLUs).
arXiv preprint
arXiv: 1St::1606.084// 2016.
10

---

Published as a conference paper at ICLR 2024
John Hogue. Metro Interstate Traffic Volume. UCI Machine Learning Repository, 2019. DOL: https://bi.org/l1.2444//////
Peter Huber. Robust estimation of a location parameter. In Breakthroughs in statistics: Methodol-
ogy and distribution, pp. 492-518. Springer, 1992.
Artjom Joosen, Ahmed Hassan, Martin Asenov, Rajkam Singh, Luke Darlow, Jianfeng Wang. and Adam Barker. How does it function? characterizing long-term trends in production server- less workloads. In Proceedings of the 2023 ACM Symposium on Cloud Computing, SoCC '23, pp. 443-458, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400703874. doi: 10.1145/3620078.322477.8 URL 0820678.362477
Kaggle. Smart meters in london.
com/ httm:/daaasss///eee http://wwwwoogeeeeeaaaaatttttaaa 2023. Accessed: 2023-11-15.
jeanmidd
maart--nteer-inn-oonee
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08366 2020.
Markel Kelly, Rachel Longjohn, and Kolby Nottingham. The UCI Machine Learning Repository. online, 2007. http://arcciieeeees..c....l
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equa- tions for irregular time series. Advances in Neural Information Processing Systems, 33:6696- 6707, 2020.
Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jacgul Choo. Re- versible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations, 2021.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In 8th International Conference on Learning Representations, ICLR, 2020.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. The 4/st international ACM SIGIR conference research & development in information retrieval, pp. 95-104, 2018.
Bryan Lim and Stefan Zohren. Time-series forecasting with deep learning: survey. Philosophical Transactions of the Royal Society A, 379(219)):202002) 2021.
Minhao Liu, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma. and Qiang Xu. Scinet: Time series modeling and forecasting with sample convolution and interaction. Advances in Neural Information Processing Systems, 35:5816-15 16-5888,, 2022.
Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and fore- casting. In International conference on learning representations, 2021.
Nicholas R Lomb. Least-squares frequency analysis of unequally spaced data. Astrophysics and space science, 39:447-462, 1976.
James Morrill, Cristopher Salvi, Patrick Kidger, and James Foster. Neural rough differential equa- tions for long time series. In International Conference on Machine Learning, pp. 7829-7838.
PMLR. 2021.
Etienne Le Naour, Louis Serrano, Léon Migus, Yuan Yin, Ghislain Agoua, Nicolas Baskiotis, Vin- cent Guigue, et Time series continuous modeling for imputation and forecasting with implicit neural representations. arXiv preprint arXiv:2306.0558 2023.
Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference on Learning Representations, 2022.

---

Published as a conference paper at ICLR 2024
Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-BEATS: Neural ba- sis expansion analysis for interpretable time series forecasting. In International Conference on Learning Representations, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances neural information processing systems, 32, 2019.
Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for ireeularly-sampled time series. Advances in neural information processing systems, 32, 2019.
Abdulwahed Salam and Abdelaaziz El Hibaoui. Power consumption of Tetouan city. UCI Machine Learning Repository, 2023. DOL: htpps//bbi.rr//0223333/6B3BB
Mohammad Shahrad, Rodrigo Fonseca, Inigo Goiri, Gohar Chaudhry, Paul Batum, Jason Cooke, Eduardo Laureano, Colby Tresness, Mark Russinovich, and Ricardo Bianchini. Serverless in the wild: Characterizing and optimizing the serverless workload at a large cloud provider. In 2020 USENIX annual technical conference (USENIX ATC 20), pp. 205-218. 2020.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in neural information processing systems, 33:7462-7473, 2020.
Benjamin Treynor Sloss, Shylaja Nukala, and Vivek Rau. Metrics that matter. Communications of the ACM, 62(4):88-88) 88, 2019.
Adam H Sparks, Mark Padgham, Hugh Parsonage, and Keith Pembleton. bomrang: fetch australian government bureau of meteorology data in r. Journal of Open Source Sofwwaee 2(17):411, 2017.
Sean J Taylor and Benjamin Letham. Forecasting at scale. The American Statistician, 72(1):37-45, 2018
Time. The A to Z of Artificial Intelligence, to-
https://time.c 6271 cot. 2023. Accessed: 2023-11-16.
Oskar Triebe, Hansika Hewamalage, Polina Pilyugina, Nikolay Laptev, Christoph Bergmeir, and Ram Rajagopal. Neuralprophet: Explainable forecasting at scale. arXiv preprint arXiv:2111.15397 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Saverio Vito. Air Quality.
UCI Machine Learning Repository, 2016.
DOI:
Huiqiang Wang, Jian Peng, Feihu Huang, Jince Wang, Junhui Chen, and Yifei Xiao. MICN: Multi- scale local and global context modeling for long-term series forecasting. In The Eleventh Inter- national Conference Learning Representations, 2022.
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Etsformer: Exponential smoothing transformers for time-series forecasting. arXiv preprint arXiv:2202.01388. 2022.
Haixu Wu. Jichui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition trans- formers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34:22419-22430, 2021.
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. TimesNet: Temporal 2D-variation modeling for general time series analysis. In International Conference on Learning Representations, 2023.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 11121-11128, 2023.
12

---

Published as a conference paper at ICLR 2024
Tianping Zhang, Yizhuo Zhang, Wei Cao, Jiang Bian, Xiaohan Yi, Shun Zheng, and Jian Li. Less is more: Fast multivariate time series forecasting with light sampling-oriee oriented MLP structures. arXiv preprint arXiv:2207.01110 2022.
Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The Eleventh International Conference on Learning Representations, 2022.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting, In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 11106-11115, 2021.
Tian Zhou, Ziqing Ma. Qingsong Wen, Liang Sun, Tao Yao, Wotao Yin, Rong Jin, et al. Film: Frequency improved legendre memory model for long-term time series forecasting. Advances in Neural Information Processing Systems, 35: 15:77771-, 2022a.
Tian Zhou, Ziqing Ma. Qingsong Wen, Xue Wang. Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International Conference on Machine Learning. pp. 27268-27286. PMLR, 2022b.
A ADDITIONAL LITERATURE
Transformers and attention. Transformer-based architectures (Vaswani et al.) 2017) have be- come the standard backbone for building state-of-the-art methods for time series forecasting. PatchTST (Nie et al.) 2022) operates by encoding sequential patches of univariate time series data into tokens for the attention mechanism. This design choice leveraged the assumption that tokens require some local semantic information for efficient and performant modelling, resulting in some of the best forecasting performance to date. PatchTST also argued that univariate forecasting was pre- ferred over multivariate forecasting, owing to rapid overfitting in the multivariate regime. Patching enabled a forward-par pass speedup (fewer tokens) while also leveraging the advantages of processing sequences via parameterised projections (e.g., similar what DLinear (Zeng et 2023) uses). In terms of forecasting performance, PatchTST is the strongest competitor to the DAM, particularly on datasets with sharp changes. However, PatchTST is limited regarding the structure of input data (e.g., continuity within patches) and forecast horizon (pre-determined), making it less well-suited than the DAM as foundation model for universal forecasting. For future work, we would like to explore the use of patching for the DAM as this has clear performance benefits; we chose not to do so for this paper because it reduces the overall flexibility of a model to missing or discontinuous data.
Several recent works have made efforts to improve either efficiency or performance of transformers for time series, by way of modifying extending the standard attention mechanism (Zhou et al.) 2021 Wu et al. 2021 Liu et al. 2021 Zhou et al., (022b)) Sparse attention mechanisms, for ex- ample, reduce complexity for improved efficiency (Zhou al., 2021; Kitaev et al.) 2020). Informer ProbSparse self-attention mechanism to effectively reduce the number of comparisons made by the internal matrix multiplication behind attention. The ProbSparse structure biases recent history over distant history, much like the HSR of the DAM, but does in a pre-determined and constrained fashion.
(Zhou et al. [202] used a KL--divergence-bass
Zeng et al. (203)) questioned the use of transformers for time series forecasting. They proposed a simple, yet performant, linear method - DLinear. DLinear parameterised the connection between trend-seasonal decomposed context sequences and a vector over the forecast horizon, by way of a simple linear projection from input to output (and summation at all scales of the trend-seasonal decomposition.. Their work brought into question whether complex and computationally-cost/ transformer architectures were at all necessary, given that the performance improvements they offer over the DLinear baseline are often marginal. Evidently, non-linear models (e.g., transformers) offer additional benefits over a linear projection because they can access non-linear temporal dynamics, particularly when reversible instance normalization (Kim et al. 2021) is employed. We argue in this paper that the full capability of transformers is not available in the small data regime followed in earlier works, and that universal forecasting is necessary to unlock the potential of transformers.
13

---

Published as a conference paper at ICLR 2024
Frequency domain Modelling directly in the frequency domain affords global perspective be- cause low-frequency components describe long-term trends (Wu et al. 2021; Zhou et al. 2022b a). Fedformer (Zhou et al.] 2022b) used 'frequency-enae eenannceeeebbeddddeeeeee block and corresponding attention mechanism. This enhanced block uses a low-rank approximation to encourage learning long-range seasonality. Autoformer (Wu al. 202T) used an nutocorrelationn-assee attention mechanism to model multi-scale trends. The autocorrelation mechanism works by computing attention scores via autocorrelation and aggregates similar sub-series weighted according these scores. Any method that uses decomposition process requiring continuity (e.g., FFT) cannot work with irregular time series data (Rubanova al. 2019). Furthermore, FFT and autocorrelation are both sensitive noise because they are exact solutions. The frequency decomposition procedure for the DAM uses a least- squares approximation (Lomb) 1976), meaning that it can work with irregular randomly sampled histories (i.e., the HSR) and is robust to noisy data.
Explicit multi-scale methods. TimesNet (Wu et al. 2023 was designed to operate at multiple resolutions by way of reshaping ID time series into 2D tensors such that the convolutional kernels employs are able to span a multitude of periods in the data. This approach enables learning intra- and inter-period variations over a given input sequence. The importance of pross-periodic relationships is highlighted by the strong performance of TimesNet. Internally, TimesNet selects the top-k (with k=5 for long term forecasting) periods within the data using an FFT decomposition. MICN (Wang et al., 2022) uses multi-scale convolutions for short-term patterns, and isometric convolutions to capture longer-term pattners.. Internally, MICN uses a Seasonal Prediction Block to predict seasonal information and Trend-cyclical Prediction Block to predict trend-cyclical information.
NBEATS OOrehkkin et al., 2020) used multiple blocks dense connections between inputs and out- puts to directly forecast over pre-determined horizon. Following the 'classical' forecasting regime, three separate blocks were used to model trend, seasonality and residuals. Each block predicts ba- sis expansion coefficients, both forward and backwards in time, model the trend and seasonality, while residuals are modeled by a regular weighted connection. NBEATS produces explainable fore- casts but the model scales poorly to long-sequences. N-HiTS (Challu al., 2023) uses multi-rate data sampling and hierarchical interpolation to extend NBEATS, improving both performance and efficiency. This explicit use of multi-scale processing evidently improves performance because forces a model to find patterns that extrapolate. The DAM models explicitly at multiple scales, but does not require a complex architecture or pre-processing of input data.
Continuous functions. Applying neural ordinary differential equations (ODEs) to irregularly sampled time series (Rubanova et al. 2019 Kidger et al.) 2020) Morrill et al. 2021 De Brouwer al. 2019) is a promising approach that may, in the future, be applicable universal forecasting. Although the Neural ODE approach also models a continuous function of time. applying it to the universal forecasting task still be accomplished. Recent work has also shown how neural im- plicit representations (NIRs) (Sitzmann et al. (220)) can be learned for time series (Naour et al.) 2023). The NIR approach similar to the DAM basis composition but involves a non-linear neu- ral network as output and is challenging to learn. We tested using small neural network as an additional predictor (projecting its weights from its own B-token) for residuals, but found that the performance gain over basis functions was negligible. This is likely because (1) the basis functions are sufficient to capture the underlying temporal dynamics (see Figure 3) and (2) are easier to learn.
B DAM MODEL CODE
import torch
import torch.. nn as
class D3A2M (nn.Module):
The DAM wrapper to build the embedders, backbone, and collapsors.
The forward method estimates the basis coefficients and the forecast methed assess those for any given times.
Args:
d_model: latent width of model; default=256
d_ff: width of hidden layer for each feed-forward block;
default-256
14

---

Published as a conference paper at ICLR 2024
layers: number of DAM layers; default=4
notom: number TV-tokens after iterative TOME; default=250 h_hoads: number of MBSA and cross attention heads; default=4
dropout: default=0
def
"n__iinitelll_l d_model, d_ff, n_layers, n_heads, n_tome, dropout-0, base_ (reeeffrruunny-8100011 super (D3A2M, The DAM's base frequency is day (86400 seconds)
init_())
self. hatpstterrbufeerreee srrime_scaalng's
tonshr [B86400/baee_freeueennee
Basis frequencies, shape [1, 1, 437]
self. torch.concatenater
((440) (149///t arange 61, 10),
24/ (torch. (1, 48, 0.5)),
//torcch.aaanee/22 28, 0.5),
arangeh (28,, 52x7, 7)
1/torch. (52+7, 52+7+10+1, 26+7))
) u.nsqueczz unsqueeze
Embeddings for TV-tokens, B-tokens, and affine
self. www.tteeeaaaaeeeeeaeee
In.nin (self, d_model)
frequencies.sizet iri.eerrre.....rrreeiieee......
self. value gallue_mmbedding
Lineirea (1, d mmddel]
Linear (2, m self.btoken walf.bbooee_ccoffee coeffs embedder nn. Linear (2, d model]
wtt.:bbocee_neeeeeeeeee perieee_eeeddddeeedddeee embedder
wwt.affneeembedddng mn.iinear (50, d_model)
Backbone model
self.backbone
TransformerBBakkonne [d_model=d_modell
d 437,
d d_ff=d_ff,
m_layerg=n_iayess
n_tomo-n_tome,
n_heads=n_heads
dropout-dropout)
Collapse B-tokens into 2 coeffs each sel f. f.bbassscclllapsssssssss = nn.linea (d_model, 2) Affine col into offset and scale sel self.affiieecollapsse
In. (d_model,
def forecast (self, times):
The actual forecasting method (after the DAM forward pass)
args:
times: forecast times (past or future) in seconds
pred = # Set this up for the forecast
times times (-1)ueeee (ttae/eeeet///ess..ttteeeessaaa
cos_coeff self.basis self.basisscoeeff:::: onssppeeeee..............eeeeeee
unsqueeze (1)
sin_coeff = sels.bbaii_.coefll:.:: .usguuuee
unsqueeze (1)
sins = sin_coeff * torch (2+torch,
wwwterrch.iisse.... pi+self. wtw.eei....eueeeeesses.......ee. prequencies=timeet
coss cos_cceff
torch.cos (2+terch.pl) pissel.. freeeeeeeeee...e.iiiieieeeeee peeguencies+timeet
pred sins. (-1)
pred +- coss. sum coss.sum (-1)
Reverse the robust standardisation and apply affine params:
med self www.mnnimmaaiiiio.omom.om...i.ae
igr self. www.aliiaaiiini.nommmmm.m params[::;;]]]
affine_offsct
self so
affine_scale = nalai i;
pred = ięr (ppeed -
+ med
return pred
def forward (self, times, values)
The DAM wrapper expects times in seconds, where 0 is 'now', the future is positive time and the past is negative time.
15

---

Published as a conference paper at ICLR 2024
Values always univariate. args:
times: time in seconds; shape= (M, C] values: values at times; shapo=[M, [M, C)
M: minibatch size
C: context size (e.g., 540)
times = times/sels tttee//see..ttmmmeallee
# Convert time to
with berch.no_gracee
Robust standardisation, uniike RevInv
med terch. median (values, axis=l, keepdin-True) [0]
igr torch, quantila (values, q=0.75, axis-l, keepdim-True)-
torch.quant quantile (values, q=0.25, axis=l, keepdim-True igr [crg<<11e--]] 1e-6
values (values med) /iqr
sin_coeffs, cos_coeffs thit_thete (times,
values, self. frequencies)
Embed times for IV-tokens at the scales of the froqueneics
te
sel tetperaalepprpeeebeaeaao
torch (2*torch. (2rthrsh va gellu_embedding - golf.value_embeee (vel w unsqueeze (-1))
(imeee.nsqqueeze (-1)))
tv_tekens = sallue_embedding + semperral_embedddngg
Embed frequencies as periods similarly
perioas embed self.btoken torch. concatenate((
www.eetteeeriooeddeeeoeoooo period embedder
torch. sin (2+torcch. pi/self..ppeeeeeep.p frequencies). trepetooes transpose (1, 2),
terch. cos (2+torch. pi/ /wt.//l........ ) transpose (1,2)), -1))
arcsinh damps absolutely high coeffs
b_tokens = wal.fbbooee_coeffee _
forch.aresinlee (0.1+)
torch. stacks (cos_coeffs, sin_coeffs), -1))/0.11 periods_ember
Quantiles for affine token
quantiles = terch.quantile (values,
q-torsh. linspace (0., 1, 50+2) [1:-11, dim-l)
offine_token aalf.affine_embedd (quanti unsqueeze
Backbone
b tokenss_saous out, affine token_ooeooooooco out backbbo (tv ttv_tokens,
b tokens, affine_token)
Collapse into usable function space affine_collapsed = holf..afinc_collapsc (affine_ttken_oout)
Set coeffs and normalisation params as attributes for forecasting step (hereafterr)
self www.amaiatiinnnoonooooo
torch. concatenate
(med. unsqueeze (-1),
iqr unsqueqqee
affine_ (aaie___ollapsed) -1)
self. half.basis_coee self.basis sosssssco caspos (b (b_tooes__ott)
class TransformerBackbone (nn. ( Module)
r =un
Backbone for the DAM, taking in embedded TV-tokens, B-tokens, and affine token. Args:
dmoddel: width of model; default=256
d_freq: number of frequencies and B-tokens; dofault=437
d.ff: width of hidden layer for each feed-forward block;
defauit=256
n_ m_layers: rumber of DAM layers; default-4 or
a. tome: number of TV-tokens after iterative ToME; default=250
n. heads: number of MHSA and cross attention heads; default=4 dropout: defauit=0
16

---

Published as a conference paper at ICLR 2024
def
init feelf, modeel, d_freq, d_ff,
n_layer,, n n_tcme, n hhaads, dropout=0,
super (fraassfrrreer//Bkkebbe)) self) _ init_i
self.n_layers n_layers
self.n_tome n_tome
self hetteeenddr__aaerrr
layers = nn ModuleeLiitt
DAMLayer model, d free, d_ff, n b_heads, dropout) \
for in range (n_lyers)))
})
def forward (sclf, tv_tokens, p_tokens, affine_token):
Logic in this wrapper includes making sure TOME reduces by the right amount with each layer. Args:
t_tv_tokens wt..tokens.sizeeee
Reduce TV-tokens iteratively reduce at each layer using TOME htt.__ooooeeeee...aaaeee... for 1i in range (sng
r int
if 1i self.n_layers 1:
self.n_tome rounding tv wv_tokens, P p_tokens, offfne_tckenn =
(p_tokens.size
self encoder (1
P_tokens,
affine_token,
return P_tckens, affine_token
class DAMLayer (nn. Module): (mn.uulee))
def wiiiii (self,
d model,
d freq, ff,
n_heads, dropout=0,
):
super (DAMLayer, self. = M..Mllttheedktteeiiin
(d_model, n_heads, dropout, hatch_first-Truu add_
self.cross_ = an.Milttheadkteetioon [dl n_heads,
dropout, fitsh_first-True add_zero_attn-True)
self. felf.feed_forrardd on.equentiall ( d_ff),
nn.GELU
In.Linear (d ff, d model), Dr.poot (dropout))
self. folfffeddffoowrddd. nn. Sequential (nn. Linear Liner (d_ d_ff),
Gn.GELU
nn Linear ff, d_ndel), nn. an.Dropout (dropout))
self. feed_ fored__fowward_bcross nn.Sequential(
Linear (d_freq, d_freq=2), nn. nn.GRLD nn Linear (d. freq*2, d d_ nn. Dropout (dropout))
self. feec, forward aff Sequenttt Iinear (d model, ff),,,
17

---

Published as a conference paper at ICLR 2024
Gn.GELU()
nn. Lineir ffff, d_model),
n0. n0. Dr.propout (dropout))
self. salf.layernerml = Lm. (d_mmdcc)
self.1ayernerm2 (d_ayye))ormm__mmde]])
self.1ayernerm3 nn. (d_mcdell)
self.layernerm! nn. (d_medel)
self.1ayeznorms no..LayerNommdd. md_model)
saye.nay = Lay.ayerNerm (d_mmdell
sele.Layernorm on.LayerSorm mede)l
def forward (self, tv_tokens, b_tokens, affine_token, r):
r"*"
Concatenate TV-tokens and affine token (in additional p_tokens) compute attention, but split this up again for ToME
Args:
tv_tokens: Timo-value tokens; shape= [M, NEv, D] b_tokens: B-tokens for basis coeffs; shapo=[M, Nb, D]
affine_toke token: for affine adjustment; shape= 1, D] r: TOME reduction amount; scalar.
M: minibatch size
D: model dimension
tokens torch. concatenaee (affine token, tv tv_tokens), attn_ wwwpttutppttttt.o self.mhsa (tckeess,,
tokens,
tokens)
attn_output_afffne a
(1)]
attn aatn__outpil oott_ooutuu_tr
w Bipartite soft matching from ToME
[1):]
merge_method, bipartiit__oftt_aachhinn oattn_ou r
tv_tokens (ttoeeeeettodtttttttttteee # Merged
attn_output_tv (attee_
Merged
tv_tokens self. 1ayeroormm (tv_tckens attn_output_tv)
tv_tokens - 1ayeraayro
self. ttt__ffowwrddttt_____ere
#tv_tokens) # PP
affine_token self. 1ayerorym (affine_token attn_output_afffne) affine_token self layernommd
self.feed forward (fffine +\
affine_tcke
Transfer information from TV-toekns into B-tokens
kv_tokens torch. concatenate ((affine_tokke.n tv_tokens), 1)
cross_atta, tbt..rroooottttttoooottttooooe
kv_tokens,
kv_tokens)
b_tokens self. layernorm5 (b_tckens
cross_attn)
b_tokens self. Layernoró
self foreardd__daadbbbbbbbbbee
b_tokens) # FF
Frocess ACROSS B-tokens
b_tokens_T b_tokens.ttansposs (1,2)
b_tokens self. self.
self. selfffeee__orrvrrdb__ooooo (b_ (b_tckens_1) (tackens_)).transspoooo .transpose (1, 2
+ bb.tkkes)
return tv_tckens, b b_tokens, affine_token
name main_'
dam (252, 256, 4, 4, 250)
Dummy data and times would be sampled from
A leng tail distribution and are not necessarily sequenttall.
dummy_valuse terch. zeros ( (32, 540)) float
shape: [M, C]
-10 until before 'now':
torch. zeros (32, 540)) random (oaddo__--644000 (-864000, -1) .foat ()
Estimate the basis coeffs
18

---

Published as a conference paper at ICLR 2024
dam (dummy times, dumny_ values)
Times can be past or future
caamyy_forecsttiiie
torch.2cros ( (32, 540)) ) randdnn
864000) float () $ anywhon
Forecast = dam. forecast (dumy_forecast_timme)
assert forecast... shape
commy__orreast_tiime shape
Listing Working PyTorch (Paszke al. 2019) code for the DAM architecture.
C ADDITIONAL HSR ADVANTAGES
1. Data from far in the past is available with no additional cost because intermediate data can ignored.
2. A HSR can be chosen such that the recent history has a higher likelihood of being sam- pled, thus weighing the importance of the recent past higher than the distant past. This is intuitively similar to the way humans perceive time.
3. Longer periods and trends
made available at additional cost earlier research Wu at. (2023); Challu et al. (2023) capitalise on multi-periodicity in time series data, but these approaches limited by the short-range perspective offered by fixed-length sequence.
4. The HSR is not fixed during inference and can be use-case specific altering the shape or type of distribution does not change the model structure and does not necessarily require retraining.
5. Missing data can be dealt with elegantly because those regions can simply be ignored in the HSR.
6. Taking multiple samples for single forecast enables notion of sample uncertainty.
7. The non-i.dd, nature time series datasets is alleviated by the randomness of the HSR.
C.1 THE IMPACT OF THE HSR DURING INFERENCE
Varying the HSR involves changing either the number of context points the width of the distribu- tion from which they are sampled (see Section 3.2 Figure & shows the effect that context size and have on the shape of the distribution defined by the HSR and the consequential forecasts. small (e.g.. 100) results in samples most similar to fixed-length contexts because we sample without replacement.
D BASIS CHOICE
We pre-selected the basis function frequencies span range of resolutions that common in time series datasets. The periods (for readability) we selected are:
Minutes: [1, 6, 11, 16, 21, 25, 31, 36, 41, 45, 50, 55]
Hours: [1.0, 1.2, 1.5, 1,7, 2.0, 2.2, 2.5, 2.7, 3.0, 3.2, 3.5, 3.7, 4.0, 4.2, 4.5, 4.8, 5.2,
5.8, 6.0, 6.3, 6.5, 6.7, 7.0, 7.3, 7.5, 7.8, 8.0, 8.2, 8.5, 8.7, 9.0, 9.2, 9.5, 10.0, 10.3, 10.5,
10.7, 11.0, 11.2, 11.5, 11.8. 12.0, 12.2, 12.5, 12.7, 13.0, 13.2. 13.5, 13.7, 14.0. 14.2, 14.5,
14.8, 15.0. 15.3, 15.5, 15.7, 16.0, 16.2, 16.5, 16.8, 17.0, 17.2. 17.5, 17.8, 18.0. 18.3, 18.5,
18.8, 19.0, 19.3, 19.5, 19.7, 20.0, 20.2, 20.5, 20.8. 21.0, 21.2, 21.5, 22.0, 22.3, 22.5,
22.7, 23.0, 23.5,
Days: [1.00, 1.01, 1.02, 1.03, 1.04, 1.05, 1.06, 1.07, 1.08, 1.09, 1.10, 1.12, 1.14, 1.15,
1.16, 1.17, 1.18, 1.19, 1.20, 1,21, 1.22, 1.23, 1,24, 1.25, 1.26, 1,27, 1.28, 1.29, 1.30, 1.31,
1.32, 1.33, 1.34, 1.35, 1.36, 1.37, 1.39, 1.40, 1.41, 1.42, 1.43, 1.44, 1.45, 1.46, 1.47, 1.48,
1.49, 1.50, 1.51, 1.52, 1.53, 1.54, 1.55, 1.56, 1.57, 1.58, 1.59, 1.60, 1,61, 1.64, 1.65,
1.66, 1.67, 1.68, 1.69, 1.70. 1.71, 1.72, 1.73, 1.74, 1.75, 1.76, 1.77, 1.78, 1.79, 1.80, 1.81,
1.82, 1.83, 1.84, 1.85, 1.86. 1.87, 1.89, 1.90. 1.91, 1.92, 1.93. 1.94, 1.95, 1.96, 1.97, 1.98,
1.99, 2,00, 2,25, 2.50, 2.75. 3.00, 3.25, 3.50, 3.75, 4.00, 4.25, 4.50, 4.75, 5.00,
5.75, 6.00, 6.25, 6.50, 6.75, 7.00]
19

---

Published as a conference paper at ICLR 2024
SensRivity to context
150
Contaxt siacs (50ma-720)
hot
Forccasttcokaz 500 Fonccastuckkaa 200 Foncaastaaciaaz
1000
200
Ruure dato
300
Resksshackkst 1000
200
440
-36
28
10
History sample time (Ir days)
Time (n days!
Senskivky to sigma
200
Sigmas (contest=T20)
1440
336
Future data
foncautbactcak 236
foncastbucteaa, 540
250
560
100
herscssbaacicst 1440
Forscastbecccak 100
Frequency
100
40
-3
-32
-0
-2
-23
-36
-12
-4 0 -10
-0
History
plap Lime 10 days)
Time Gn days!
Figure 8: The effect of varying the history sampling regime (HSR). The top row demonstrates vari- ation on the context size (i.e., the number of samples) on the forecast. The bottom row demonstrates variation sigma (i.e., the width of the distribution from which samples are drawn). Note that sampling is done without replacement. The forecasts are shown for 720 sequence points, and 1440 points for the backcast.
Weeks: [1.04, 1.07, 1.11, 1.14, 1,18, 1,21, 1,25, 1.29, 1.32, 1.36, 1.39, 1.43, 1.46, 1.50,
1.54, 1.57, 1.61, 1.64, 1.68. 1.71, 1.75, 1.79, 1.82, 1.86, 1.89, 1.93, 1.96, 2.00, 2.04, 2.07,
2.11, 2.14, 2.18, 2.21, 2.25, 2.29, 2.32, 2.36, 2.39, 2.43, 2.46, 2.50, 2.54, 2.57, 2.61, 2.64,
2.68, 2.71, 2.75, 2.79, 2.82, 2.86, 2.89, 2.93, 2.96, 3.00, 3.04, 3.07, 3.11, 3.14, 3.18, 3.21,
3,25, 3,29, 3.32, 3.36, 3.39. 3,43, 3,46, 3.50. 3.54, 3.57, 3.61. 3.64, 3.68, 3.71, 3.75, 3.79,
3,82, 3.86, 3.89, 3,93, 3.96. 4.00, 4.50, 5,00, 5.50, 6.00, 6.50. 7.00, 7.50, 8.00, 8.50, 9.00,
9.50, 10.00, 10.50, 11.00, 11.50, 12.00, 12.50, 13.00, 13.50, 14.00, 14.50, 15.00, 15.50,
16.00, 16.50, 17.00, 17.50, 18.00, 18.50, 19.00, 19.50, 20.00, 20.50, 21.00, 21.50, 22.00,
22.50, 23.00, 23.50, 24.00, 24.50, 25.00, 25.50, 26.00, 26.50, 27.00, 27.50, 28.00, 28.50,
29.00, 29.50, 30.00, 30.50, 31.00, 31.50, 32.00, 32.50, 33.00, 33.50, 34.00, 34.50, 35.00,
35.50, 36.00, 36.50, 37.00, 37.50, 38.00, 38.50, 39.00, 39.50, 40.00, 40.50, 41.00, 41.50,
42.00, 42.50, 43.00, 43.50, 44.00, 44.50, 45.00, 45,50, 46.00, 46.50, 47.00, 47.50, 48.00,
48.50, 49.00, 49.50. 50.00, 50.50, 51.00, 51.50, 52.00]
Years: [1.25, 1.50, 1.75, 2.00, 2.25, 2.50, 2.75, 3.00, 3.25, 3.50, 3.75, 4.00, 4.25, 4.50, 4.75, 5.00, 5.25, 5.50, 5.75, 6.00, 6.25, 6.50, 6.75, 7.00, 7.25, 7.50, 7.75, 8.00, 8.25, 8.50, 8.75, 9.00, 9.25, 9.50, 9.75, 10.00]
We found during initial experimentation that the DAM was relatively robust the choice of fre- quencies and the number thereof. We choose the above 437 frequencies because: (1) fewer B-tokens enables faster learning. (2) inspection of the initial basis weight fits, Og for all 25 training datasets revealed sufficient reconstruction of the signal (see Figure 3 and Appendix D and (3) these fre- quencies cover ranges common time series datasets because their sources (e.g., weather traffic) are aligned with human time elements (minutes, hours,, etc.).
E BASIS COEFFICIENT INITIALISATION
Listing is a PyTorch Paszke et (2099) listing the code used to initialise the basis coefficients, Op. for processing by the DAM.
F HYPER-PARAMETERS
Section B.ltionn.]]mm details the DA. architecture and Appendix Bls is full Py Torch code listing of the model architecture. We included both for completeness and refer to nomenclature from these below.
20

---

Published as a conference paper at ICLR 2024
import torch
def init (t, v, nu, lambda=1)
t has shape (B, L, 137 times
v has shape (B, L, 1] values
nu has shape [1, 1, 437]; pre-solected frequencies
lambda is for rogularisation
x = torch. concatcnate((
# broadcast to shape (B, by 4373
torch.cos (2wtarch.pi.muse.....
-1) shape [8, 437]
torch. (x. I10,01 0
shape 1437, 4371
reg
repeat. seseat__intereeave
Lambda=I. (0),
size dim=0)
theta
torch. Norch.1inalg.solvee.
torch.bmm (ommmX... transposs (1,2), X) + reg,
torch. (X. transtosnsseeeee (1,2), values). sum sum(11) (-1))
return theta
shape (9, 437+2]
Listing 2: Pytorch
(Paszke et 2019) code for 0g initialisation.
Model hyper parameters. paper:
The following model hyper-paramenn were used for the DAM in this
Model width, dmodel. of 256.
Feed-forware internal width, dyy. of 256.
4 layers.
4 MHSA and cross-attention heads.
A ToME reduction target of 250 TV-tokens.
Dropout of 0.1.
Time units of I day, such that öt = 1 denotes one day from 'now' and öt = -1 is one day into the past.
Training hyper parameters. this paper:
The following training hyper-parameters were used for the DAM in
Minibatch size of 32.
Data sampling with replacement, where the probability of sampling a given univariate is proportional to how "useful" it is. We define proxy for utility here be combination of the standard deviation over an average day (if the sample resolution is sufficient, oth- erwise an average week) and the average standard deviation of all daily points, taking the average day the vector mean. The former ensures that we do not over-sample very noisy data, while the latter captures intrinsic variation from the commonly periodic data. This choice was determined experimentally to be optimal when compared to sampling regimes balanced according to dataset length or the number of variables. Within any chosen uni- variate, samples are taken uniformly.
Two phase learning: (a) 1,000,000 iterations with 10,000 warmup steps followed by cosine annealing from 1 - to 1 14, and (b) an additional 50,000 iterations with 2,000 warmup steps followed by cosine annealing from 1 to O.
Gradient clipping to the th percentile of the latest 1000 gradients of all model weights.
HSR context size and of 540 and 720, respectively.
540 target points (also sampled from the HSR) over which compute the loss.
Random seed of 42 (the answer).
21

---

Published as a conference paper at ICLR 2024
G
HSR TUNING
Figure Shows 4 heatmaps for MSE estimates on the validation sets of the 10 datasets tested on in Section 4.1 The differences in optimal HSR parameters across datasets both insightful regarding the datasets themselves, and the differences amongst them, and how the DAM is adaptable to these differences. Note that no training was undertaken to optimise these values and the same model can be used with any combination of context size and ø. In each case retained ToME reduction ratio as per training (see Appendix F
The optimal HSR settings for each dataset are:
ETTml: context size of 900, 100, ToME reduction to 417.
mTTm2: context size of 1200, 100, ToME reduction 556.
ETThl: context size 1300, σ = 100, ToME reduction to 602.
ETTh2: context size of 400, σ = 100, ToME reduction to 185.
ECL: context size of 900, 100, ToME reduction to 417.
Traffic: context size of 1300, a 300, ToME reduction to 602.
Weather: context size of 1400, a = 200, ToME reduction to 648.
Exchange: context size of 100, σ 100, ToME reduction to 509.
Wind: context size of 900, 100, ToME reduction to 417.
USWeather: context size of 1200, σ 100, ToME reduction to 556.
H DATASETS' DETAILS
H.1 DESCRIPTIONS AND SUMMARIES
Dataset
Res.
Horizons
Train/valid/test
Num
Domain
ET ETTm2
15 mins
19,,192,336,77 720)
[34465, 1152T, 11521]
Flectricity
ETThI, ETTh2
I hour 196,192,336,220
[8545, 2881, 2881]
Electricity
ECL
hour [96,192,336,7]] 720)
[18317, 2633, 5261]
321
Electricity
Traffic
hour 196.192,336.72)
[12185, 1757, 3509]
862
Traffic
Weather
10 mins 19,,192,336,72)
[36792, 5271, 10540]
21
Weather
Exchange
day 196,192,336,722
[5T20, 665, 1422]
8
Finance
Wind
hour 16,393233667200
[183982, 26299, 52594]
Weather
USWeather
hour
19,,192,336,700)
[24544, 23229, 46455]
12
Weather
Table 5: Dataset details. 'Res.' is the dataset resolution, or sampling rate. The listed horizons those that are commonly tested on. 'Num' lists the number of variables (i.c., columns) in each dataset.
Table T contains information about the public datasets used for both training and evaluation. Ap- pendix H.2 details an additional 15 datasets used only for training. The ETT dataset:"] electrical transformer statistics (e.g., load capacity and oil temperature). ECL tracks electricity consumption for 321 clients, from 2012 to 2014, and is recorded in kilowatts. Traffic traffic tracks road occumpancy rates for 862 sensors on San Francisco Bay area freeways. It is a collection of 48 months (2015- 2016) worth of hourly data from the California Department of Transportation. Exchange [ tracks daily exchange rates of currencies (Australian Dollar, Pound Sterling, Canadian Dollar, Swiss Franc, Chinese Yuan, Japanese Yen, New Zealand Dollar, and Singapore Dollar) versus the US Dollar. It spans 26 years, from 1990 to 2016.
http://pems.dot.co https://github.com http://ttto..m..kkkk///iiiieette
22

---

Published as a conference paper at ICLR 2024
-.
0.75
0.70
1009
1000
1000
1100
1100
1100
1300
1200
1300
1200
1100
1200
0.0
3400
1400
1400
100
200
300
600
710
800
800
305
2001
300
400
710
800
000
1000
200
350
400
700
030
400
Sigma
Sigma
Sigma
(a) ETTh1
(b) ETTh2
(c) ETTml
-.3
0.35
0.70
0.30
500
02
0.30
0.65
Contoxt-
600
600
700
100
800
000
800
100
200
100
0.0
1000
-020
1009
1800
3300
1100
11.00
1200
-018
1200
1200
1300
3400
018
1300
1400
1400
1300
0.2
100
220
420
550
600
250
500
3000
120
200
520
X20
800
200
320
420
100
520
900
1000
Sigma
Sagmi
Ságma
(d) ETTm2
ECL
Traffic
100
200
100
0.500
0.9
400
500
0415
0410
00425
1800
1100
-03
0.400
100
250
S20
200
820
820
300
250
300
8:0
500
1000
300
300
300
1000
gg)
Weather
(h) Exchange
(i) Wind
100
-0725
00675
Cortest
0.000
00625
-0575
0050
1400
100
200
600
100
900
100
(i) USWeather
Figure 9: HSR parameter search for commonly used benchmark datasets. The heatmaps show the MSE values for the corresponding context size and combination
23

---

Published as a conference paper at ICLR 2024
Weather the widely used Germany weather dataset, while USWeather is rarely test on, most notably for Informer (Zhou et al., 2021). Wind 10) contains hourly power generation estimates 28 area's energy potential from 1986 2015 as percentage of the power plants maximum output;
is also rarely tested on, most notably used for Pyraformet|.iu et al. (2021).
These datasets were all split according prior work (Wu et al. 2021 Zhou et al. 2021 Liu al.
2021
H.2 ADDITIONAL DATASETS FOR TRAINING
We sourced an additional 15 datasets for training, aside from the 10 commonly used benchmarking datasets discussed above. Since the DAM is universal forecaster that is not constrained to single dataset or characteristic (e.g. resolution), it should benefit from additional training data. Many of the datasets listed below contain missing data, making it challenging for models that expect regularly sampled data. For instance, stock prices are not recorded on the weekend, resulting in persistent missing data. We did not evaluate the DAM these additional datasets because there typically no existing comparison points in the literature.
We used four sources for additional data, namely the Monash time series repository (Godahewal et al. 2021 11 the UCI machine learning repository (Kelly et al. 2007 12 the Huawei public and private cloud data release (Joosen al. 2023 13 and Kaggle for stock market data. Table6 con-
tains information about the additional 15 publicly datasets used to augment training in this paper.
Dataset
Res.
Train/valid/test
Num
Missing
Domain
Monash Australian electricity
30 mins
[139363, 46454, 46454]
6
Yes
Electricity
Monash Oikolab Weather
hour
[6034, 2011, 2011]
8
No
Weather
Monash Solar
10 mins
[31356, 10512, T0512]
137
No
Weather
Monash Sunspot
I day
[44354, 14784, 14784]
1
Yes
Weather
UCT Beijing air quality
hour
[21038, 7012, 70121
132
Yes
Air quality
UCI PM data
hour
[31550, 10517, 10517]
41
Yes
Air quality
UCI Air Quality
hour
[5612, 1872, 1872]
13
Yes
Air quality
UCI Tetouan City power
mins
[31449, 10483, 10483]
8
Yes
Electricity
UCI Metro Traffic Volume
hour
[28922, 9641, 9641]
1
Yes
Traffic
StocksForbes2000
day
18013, 2671 26711
146
Yes
Finance
StocksSP500
1 day
18013, 2671 26711]
154
Yes
Finance
StocksNASDAQ
1 day
[8013, 2671 .6711]
111
Yes
Finance
StocksNYSE
day
18013, 2671 26711
206
Yes
Finance
HuaweiPubM
min
[22463, 7489, 7489]
14
No
Cloud
HuaweiPvtM
I min
[121823, 40608, 40609]
35
Yes
Cloud
Table 6: Additional training dataset details. 'Res.' is the dataset resolution, or sampling rate. The 'Missing" column denotes whether the dataset contains any missing data.
The Monash time series repository (Godahewa et al., 2021) contains 30 datasets in total, from which we selected the following 4 based on their lengths: Australian electricity, Oikolab Weather, Solar, and Sunspot. Australian electricity demand is dataset curated by the Monash team. The Oikolat weather dataset is a public weather dataset extracted by the Monash team from a domain-specitec platform. The Solar dataset contains simulated power output of photovoltaic power plants [6 it is also used by other works (Lai et al., [2018] Liu et al. 2022). Sunspot was sourced from the sunspot index and long-term solar observations platform
https://oikolab.co
https:/
24

---

Published as a conference paper at ICLR 2024
We included additional 5 datasets from the UCI dataset repository: PM Data, Air Quality, Beijing Air Quality, Tetouan City power, Metro Traffic Volume. PM Data Chen 2017b) records hourly data for PM2.5 (particles that are 2.5 microns less in diameter) and meteorological variables for 5 cities, namely Beijing, Shanghai, Guangzhou, Chengdu and Shenyang. Air Quality (Vito) 2016) contains the hourly responses of gas multisensor device deployed on the field in Italian city. Beijing Air Quality (Chen) 2017a) contains PM2.5 data of the US Embassy in Beijing along with meteorological data from Beijing Capital International Airport. Tetouan City power (Salam & El Hibaoui 2023) includes the power consumption of three different distribution networks of Tetouan city in Morocco. Metro Traffic Volume (Hogue 2019) includes the hourly traffic volume westbound on the 1-94 in Minneapolis-St Paul, USA.
The stocks data on Kaggle contains data for Forbes2000, NASDAQ, NYSE, and the SP500. We filtered the stocks, keeping only those that had at least 10000 days of data (ignoring missing weekend data, however).
Huawei recently released 2 cloud resource datasets for their private and public cloud services (Joosen et al. 2023), spanning over months at per-minute and per-second sample resolutions. The HuaweiPVtM dataset is derived from Huawei's internal workloads and contains detailed per-minute (aggregated from per-second) statistics for 200 functions running across multiple Huawei cloud data centers. We selected the function request data, keeping only those functions whose 10th percentile was greater than 30 requests (35 functions). We applied this filtering because infrequently requested functions do not present interesting data for training. The HuaweiPubMi dataset is from Huawei's public amnction-as-a-eervie platform, containing per-minute data. Following a similar justification, employed the same filtering procedure, but kept only those functions whose 10th percentile was greater than 100 requests (14 functions).
H.3 HULD-OUT DATASETS
Dataset
Train/valid/test
Num
Missing
Time unit scaling for the DAM held-out inference
1931
year
Monash
30 mins
[68800 2840, 8680)
100
No
Electricity
24,36,48.60
day
London Smart
Meter
(MMeters)
Monush
day
[7579, 1083, 2166]
192,336777
year
Weather
(MWesther)
Monash Web
day
1562, 81, 160]
500
Web
24,36,48,60
week
traftic (MWeb)
Monash
day
[507, 73, 146]
1266
Yes
Weather
24,36,48,60
hours
Temperature
Rain (MTemp)
Azure
minute
[12096, 4032]
Cloud
19,,3623,
10 day
UCI Power
I hour
[24211, 3459, 6918]
Yes
Electricity
16.192,336.720
Wikipedia
day [1789, 597, [194]
No
Web
16,136,336,340
weok
weekdays
Table 7: Additional training dataset details. 'Res.' is the dataset resolution, or sampling rate. The "Missing' column denotes whether the dataset contains any missing data. For comparison to baseline methods we filled all missing values with zeros prior to dataset normalisation. Time unit scaling (i.c., multiplier on time for better frequency coverage) was determined empirically by inspection on the respective validation sets.
Table 7 lists details of the 8 datasets used for held-out evaluations in Section 4.2 IIIness
captures weekly patient data from Centers for Disease Control and Prevention in the United States, between 2002 and 2021. The values are the ratio of patients in certain age groups versus the total number of patients. Columns were filtered and splits determined according to carlier work (Wu
al. 2021 Zhou et al. 2021 Liu al. 2021).
http:///////.o.oooo///////////// daashcaashhhaace htm]
hittps://gis.cde.
25

---

Published as a conference paper at ICLR 2024
We selected four datasets from the Monash time series forecasting repository, namely: (1) London Smart Meters (MMeters) (Kagle 2023). , (2) Weather (MWeather) (Sparks al. 2017). Web traffic
(MWeb) (Google 2017). and Temperature Rain (MTemp) (Governmen 2017). Note that MWeather distinet from Oikolab Weather.
For MMeters selected the top 100 longest variables (of 5560) and, owing to slight differences in start and end collection times, truncated each variable to the latest and earliest start and end times, respectively. This yielded half-hourly dataset spanning from 10 October 2011 I to 16 February 2014.
For MWeather we selected 679 of 3010 variables of the same length (10828). For MTemp we extracted the minimum, mean, and maximum temperatures from the 422 weather stations in the original time dataset. For MWeb we selected the 500 longest (considering missing data) of 145063 available time series, and filled remaining missing data with zeros (for training of baseline methods).
Similarly to the Huawei datasets, we processed the Azure 2019 function trace (Shahrad et al. 2020) by selecting the top 159 function requests by removing any functions whose 10 percentile was more than 70 requests. For the UCI household power consumption dataset (UCIPower) (Hebrail & Berard 2012), we aggregated the minutely samples to per-hour and filled missing values with zeros. We constructed Wikipedia weekdays using the Wikipedia pageviews 19 tool, extracting page views for all 7 weekdays (Monday to Sunday) for 2983 days.
H.4 MULTIVARIATE CORRELATIONS
(a) Electricity
(b) ETTh1
(c) ETTh2
(d) ETTml
405
05
15
(e) ETTm2
(1) Exchange rate
(g) Traffic
(h) US weather
455
(i) Weather
(j) Wind
Figure 10: Spearman correlations across variables for all datasets (the train split only), showing how some datasets have more correlation across variables than others. The DAM is univariate model, future work may incorporate multivariate component.
We were interested in the correlation across variables for the commonly used time series datasets. To this end we computed the Spearman correlations between variables in each benchmark dataset, as shown in Figure 10 Coefficient values are only colored when < 0.05, with the cell remaining
httpss:/pagget
wmo1 .ud.org/
26

---

Published as a conference paper at ICLR 2024
blank (null) otherwise. We undertook this analysis to understand what potential impact a multivari- ate component may have on the DAM. Future work will entail making full use of cross- variibbe information.
Even though there is significant correlation across variables for some datasets, this does not nec- essarily mean that a multivariate perspective necessarily improves forecasting. In other words, correlation between variables at a shared time does not directly translate into additional informa- tion about the future state of any of those variables. Nie al. (2022) argued for a univariate approach and showed how multivariate models overfit rapidly. but also suggested that a simplified inclusion of 'some' cross variable information may be beneficial.
ADDITIONAL LONG-TERM FORECASTING RESULTS AND DETAILS
For all the comparison methods in this paper we adopted hyper- papem--arameter settings as described in the corresponding papers. Where relevant we set input/context size windows to be as large as possible (constrained to settings described by the authors), but always ensured that we selected the best performing settings such that the results in this paper represent the best case scenario for comparison methods. We used the official implementation for PatchTST Dlinea 2 and TimesNet. For other methods we used the TimesNet repository as it contains many implementations We give normalised MSE and MAE in Table ① in and results on additional SoTA methods in Section
and
22
.22
1.1 NORMALISED MSE AND MAE
Table 8: Normalised metries corresponding to Table 1 For NMSE we normalised by the 12-norm of the ground truth variable, |r/2, and for NMAE we normalised by the 11-norm thereof, |m/1-
Hongon
NMSE
NMAE
AMSE
NMAE
NMSE
NMAE
NMBS
NMAE
NMSE
NMAE
NMSE
NMAB
NMSE
NMAA
NMSE
NMAE
E17m2 BT hm!
192
0.310
0.153
0.479
0.310
0.150
0477
0.244
0.302
0.468
0.222
0.304
0.463
0.315
0.363
0.528
0.337
0.409
0.600
0.560
0.558
0.694
0.288
0.326
0.168
0.502
336
0.317
0:485
0.317
04483
0.329
0.493
0.334
0.489
0:400
0.565
0.619
0.794
0.677
0.796
0.366
0.545
720
0.361
0.528
0.368
0.528
0.376
0.533
0.566
0.537
0.445
0.600
0.896
0.921
0.881
0,930
0.457
0.628
0.186
0.185
0103
0.189
0.044
0.193
0.000
0.109
0.113
0.325
00059
0.20N
192
0.071
0.212
0.070
0.211
0.011
0.217
0.03
0.224
0.103
0.267
0.214
0.442
0.205
0.445
0,888
0.257
336
0,075
0.219
0.074
0.219
0L088
0.243
0.092
0 0.266
0.133
0.312
0,336
0.527
0.410
0.658
0.125
0,312
0.268
0.267
0.116
0.128
0.19%
1.195
0.964
0,186
0,391
96
0.336
0.309
0.330
0.503
0.35
0.505
0.341
0.505
0.383
0.543
0.358
0.531
0.689
0.801
0.367
0.336
E17h2 BTThi
192
0.361
0.531
0.352
0.532
0.375
0.543
0.390
0.553
0.428
0.577
0:467
0.651
0.792
0.921
0.431
0.594
336
0.368
0.537
0.357
0.527
0.289
0.559
0.401
0.563
0.500
0.604
0.765
0.918
1,002
0,513
0.678
0.582
0.558
0.412
0,597
0.448
0.533
0,73]
0.966
0.900
1,00)
0.727
0,845
96
0.006
0.247
0.089
0.245
0L08
0.249
0L0R3
0.262
0.135
0.521
0.258
0.447
D.485
0.700
0.119
0.3015
192
0.115
0.272
0.108
0.271
0.108
0.280
0.130
0.305
0.170
0.362
0,500
0.600
1.627
1.335
0.158
0.354
0.276
0.276
0.116
0.144
0.340
0.182
0.384
0.752
1.485
1,340
0.198
0.406
720
0.125
0.797
0.125
0.309
0.125
0.316
0.233
0.455
0.281
0
1,077
1,151
1.373
1,115
0.588
0.476
96
0.151
0.320
0.152
0.312
0.127
0.270
0.340
0.291
0.151
0.315
0.145
0.259
0.281
0.356
0116
0.334
UUS Weathel Wind Exchunge Weather Thamle RC1.
192
0.174
0.338
0.169
0.330
0.146
0.292
0.154
0.306
0.186
0.345
0.160
0.316
0.24
0,474
0.175
0,346
0.344
01336
0.162
0.169
0.327
0.202
0.202
0,304
0,202
0,482
0.184
0.358
720
0.239
0.406
0.23
0.399
0.197
0.352
0.203
0.367
0.239
0.405
0,250
0,408
0.301
0,474
0.205
0.381
96
0.323
0.420
0.317
0.166
0.248
0.312
0.254
0.358
0.374
0.409
0.357
0.335
03475
0.603
0.357
0,389
0.420
0.455
0.362
0.291
0.579
0.382
0.465
0.369
0.398
0.432
0.22
0.270
0.331
0.301
0.376
0.393
0.419
0.398
0.375
0.46
0,475
0.377
0.404
720
0.376
0.478
0,370
0.422
0.308
0.383
0.502
0.309
0.421
0.440
0.456
0.449
0.489
0.496
0.394
0.416
0.215
0.31
0.241
0334
0.333
0.326
0.273
0.387
0.247
0.351
0.270
0.395
0.308
0,467
0,303
0,411
192
0.392
0.392
0.303
0.397
0.339
0.297
0.415
0.350
0,484
0.365
0.588
0.368
0.479
3356
0.322
0.408
0.319
0.406
0.383
0.364
0.410
0.112
01357
0.476
0.435
0.561
0.465
0,599
0.451
0.553
720
0.445
0.504
0.440
0.502
0.195
0,548
0.511
0.601
0.45]
0.553
0,582
0.873
0.676
0.717
0.558
0.638
0.136
0.136
01066
07025
0.01)
0,076
0.189
0.026
0,146
192
0.049
0.192
0.050
0.65
0.224
0.066
193
0.112
0.203
0.137
0.335
0.246
0.487
0.052
0.214
336
0.057
0.209
0.058
0.211
0.099
0,280
0.089
0.274
0.157
0.359
0.269
0.590
0.311
0.556
0.090
0,285
720
0.258
0.466
0.251
0:457
0.278
0:487
0.273
0.475
0.266
0.479
0.445
0.439
0.464
0,658
0.223
0,450
0.233
0.211
0.178
0.265
0.197
0.150
0.171
0.168
0.203
0.176
0.279
192
0.214
0.336
0.218
0.333
0.192
0.2800
0.216
0.307
(1.000
0.306
0.189
0.289
0.186
0.280
0.195
0.295
336
0.216
0.338
0.221
0.335
0.200
0.288
(1230
0.321
(.210
0.314
0.199
0,300
0.196
0.288
0.203
0.298
720
0.231
0.359
0.248
0.366
0.207
0.297
0.253
0.349
0.233
0.335
0.211
0:215
0.221
0,314
0.503
0.603
0.505
0.388
0.195
0.616
0.499
0,621
0.537
0.618
0.478
0.80
0.472
0.612
0.503
0.638
192
0.549
0.643
0.557
0.640
0.561
0.673
0.547
0.666
0.589
0.700
0.526
0.649
0.541
0.669
0.553
0.675
336
0.539
0.651
0.619
0.589
0.685
0.500
0.693
0.608
0.715
0.555
0.682
0,571
0.698
0,577
0,691
0.704
0.710
0.664
0.650
0.617
0.603
Seciiime--eree--e Seriea-liybrary
27

---

Published as a conference paper at ICLR 2024
1.2 RESULTS ON ADDITIONAL SOTA METHODS
Table 9: Additional long term forecasting results. Results are always computed as the average of seeded runs. Owing to time and resource constraints, methods with asterisk" are from single runs (not an average of 3 seeded runs). These will be updated in due course.
FiLM
TetesNet
Autoformer MSE
FEDformer
Informer
LightTS
ETSformer
MSE
MAE
MSE
MAE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.309
0.351
0.341
0.376
0.371
0.413
0,474
0.465
0.624
0,560
0.395
0.409
0.528
0.496
192
0.341 0.369
0.398
0.410
0.429
0.444
0.531
0.493
0.781
0.643
0.423
0.433
0.565
0.537
336
0.370
0.386
0.418
0.422
0.453
0,461
0.608
0.527
0.769
0.805
0.458
0.460
0.657
0.602
720
0.419
0.413
0.481
0,457
0.538
0.501
0.604
0.529
1.032
0.773
0.546
0.517
0.800 0.695
96
0.177 0.265
0.185
0.266
0.193
0.282
0.249
0.330
0.391
0.479
0.232
0.328
0.350 0.409
192
0.224 0.296
0.255
0.308
0.264
0.325
0.281
0.338
0.705
0.644
0.401
0.446
0.421 0.468
336
0.275
0.331
0.320
0.348
0.325 0,364
0.339
0,373
1.454
0.922
0.481
0.492
1.421 0.993
720
0.359
0.388
0.431
0,410
0.431
0.427
0.456
0,440
4.291
1.529
0.739
0.621
4.563
1.821
96
0.402
0.416
0.394
0.414
0.376
0.418
0.433
0.447
0,914
0.744
0,442
0.447
0.555
0.536
192
0.441
0.441
0.455
0.452
0.420
0.444
0.503
0.488
1.007
0.785
0.496
0.479
0.686
0.619
336
0.473
0.462
0.484
0,463
0.464
0,469
0,533
0,505
1.038
0.791
0.552
0.511
0.870
0.730
720
0,496 0.495
0.514
0,493
0,508
0.503
0.537
0.522
1.196
0.876
0.615
0.570
1.088
0.851
96
0.315
0.360
0.339
0.376
0.348
0.391
0.384
0.415
2.949
1.361
0.410
0.447
0,382
0.429
192
0.384
0.406
0.402
0.412
0.432
0.441
0.448
0.452
6.521
2.115
0.558
0.526
0.543
0.523
336
0.409
0.431
0.460
0,452
0.472
0,476
0.486
0,488
5.249
1.920
0.665
0.577
0.624
0.563
720
0.428
0,448
0,456
0,463
0.476
0,488
0.590
0.540
3,936
1.683
0.992
0.712
0.632 0.573
96
0.154
0.246
0.168
0.271
0.196
0.310
0.207
0.320
0.336
0.417
0,214
0.317 0.250 0.353
ECL
192
0.167 0.259
0.185
0.286
0,204
0.318
0.229
0.335
0.359
0.439
0,224
0.327 0,270 0.365
336
0.189 0.284
0.202
0.303
0.227
0.340
0.244
0,348
0.358
0.438
0.244
0.347
0.285 0.377
720
0.249 0.341
0,223
0.318
0.275
0.375
0.296
0.382
0.388
0.452
0,281
0.375
0.311 0.395
96
0,411 0.284
0.593
0.319
0.582
0.365
0.664
0.413
0.730
0.414
0.667
0.422
0,970
0.583
Traffic
192
0.407 0.284
0.617
0.327
0.608
0.380
0.634
0.397
0.759
0.439
0.661
0.426
1.020 0.593
336
0.426 0.298
0.631
0.336
0.620
0.387
0.630
0,394
0.848
0.479
0.682
0.436
1.043 0.599
720
0.526 0.372
0.665
0.352
0.639
0.393
0.645
0.399
0.978
0.547
0.738
0.457
1.088 0.614
SWeather Wind Exchange Weather
96
0.185 0.235
0.171
0.220
0.223
0.306
0.308
0.361
0.385
0.437
0.173
0.233
0.222 0.307
192
0.225 0.265
0.230
0.270
0.275
0.341
0.324
0.377
0.458
0.468
0.216
0.276 0.280 0.365
336
0.268 0.297
0.283
0.304
0.338
0.383
0.370
0.399
0.560
0.522
0.266
0.315 0.359
0.425
720
0.330 0.341
0.358
0.354
0.410
0.418
0.429
0.435
1.052
0.756
0.330
0.363
0.494
0.513
96
0.213
0.345
0.112
0.240
0.162
0.292
0.152
0.282
0.913
0.765
0.132
0.217
0.096
0.220
192
0.320
0.424
0.220
0.341
0,280
0.384
0.280
0.388
1.153
0.859
0.440
0.391
0.195
0.319
336
0,512
0.539
0.385
0.456
0.492 0.446
0.494
0.524
1.593
1.017
0.506
0.541
0.373
0.449
720
0.801 1.144
0.978
0.752
0.834 1.178
0.822 1.119
2.668
1.341
0.967
0.754
0.779
0.668
96
0.199
0.222
0.201
0.230
0.176
0.241
0.201
0.262
0.181
0.224
0.172
0.217
0.198
0.262
192
0.219
0.236
0.221
0.242
0.3 0.253
0,214
0.269
0.196
0.236
0.189
0.229 0.208 0.269
336
0.234
0.251
0.231
0,246
0.206 0.266
0.246
0,292
0.204
0,244
0.200
0.237 0.215 0.277
720
0.264 0.289
0.244
0.253
0.228
0.292
0.256
0.306
0.206
0.246
0.220
0.253 0.228 0.297
96
0,480 0.482
0,493
0.483
0.504
0.502
0.556
0.540
0.498
0.506
0.487
0.492
0.487 0.501
192
0.528 0.516
0.552
0.520
0.568
0.542
0.630
0.581
0.562
0.550
0.533
0.526 0.529 0.534
336
0.561
0.537
0.586
0.543
0.602
0.560
0.612
0,571
0.606
0.577
0.550
0.541
0.552 0.549
720
0.647
0.589
0.611
0.559
0.632
0.581
0.655
0.595
0.596
0.575
0.587
0.566
0.589 0.573
28

---

Published as a conference paper at ICLR 2024
J FORECASTING VISUALISATIONS
PoothFST
OuJneer
-10
gutur
-10
Unuved Centas:
Presidion
Fature pound inth
-14
-1A
-03
-400
-200
200
400
600
-200
200
400
600
Tine
Tin
(a) PatchTST64
(b) DLinear
VICH
-02
-1.0
Unased Coltec
Unuset Centont
grann
Fature gound tuth
-20
-400
-200
200
400
800
200
600
600
Thme
(c) Nhits
(d) MICN
Cammao
Pynuforma
-1.2
Unused Contec:
Unured Contast
Fulure grand
Fsture grund irthh
-20
216
200
410
200
400
600
Tima
(e) Crossformer
(1) Pyraformer
Figure 11: Forecasting demonstrations on ETTml (test set; forecasts commence exactly halfway through the entire set) for competitor methods.
Mochis!
ouárear
Unuved Centast
Futur yound muth
20
2.8
-000
-200
400
-200
200
400
Time
Tim
(a) PatchTST64
(b) DLinear
Uraited Conteer
Unised Conteit
greane
Futuro graund mrth
$1.0
-23
-100
-200
200
400
650
-200
200
400
600
Tim
(c) Nhits
(d) MICN
Croudform
Pynaoomar
Nodel Contec:
La
Centect Unuied Cantess
Unssed Contect
future pround
Future grund tith
03
450
-205
200
400
600
(e) Crossformer
(f) Pyraformer
Figure 12: Forecasting demonstrations on ETTm2 (test set; forecasts commence exactly halfway through the entire set) for competitor methods.
29

---

Published as a conference paper at ICLR 2024
PatenTST
Meant Connutt
Podiatio
10
grens
-1.2
-1.75
-225
430
PatchTST64
(b) Diiinear
Mrkt
MICN
Model Centeet
Contoot
Funayy glond mum
Futuro
Free olan
-15
Tine
Time
(c) Nhits
(d) MICN
Pynalomer
-0.5
Medin Ceatrot
Vedul Contact
Unaad Content
-1.
rutus gound
-1.2
Phiditttn
-20
-225
200
410
600
(e) Crossformer
(I) Pyraformer
Figure 13: Forecasting demonstrations ETTh1 (test set; forecasts commence exactly halfway through the entire set) for competitor methods.
Medtl Conied
Medki Conted
Futurs ghore vuth
Unoved Centas:
-000
-200
200
400
620
-200
200
400
600
(a) PatchTST64
(b) DLinear
Mesel Centoot
Meddil Centect
Eraited Contect
Contect
-000
-200
200
400
K20
-200
200
4.00
600
(c) Nhits
(d) MICN
Macel Contec
Micda Cented:
Unaaed Contact
0.5
Unuaed Centess
Prediaiion
-0.5
.026
-200
400
K20
-200
620
(e) Crossformer
(l) Pyraformer
Figure 14: Forecasting demonstrations ETTh2 (test set; forecasts commence exactly halfway through the entire set) for competitor methods.
30

---

Published as a conference paper at ICLR 2024
PatchTat
DLinear
Made: Coite
Canteal
Prodictio:
PStchTST64
Diiinar
MICN
Content
grobii
(c) Nhits
MICN
Crosfooma
Pytafomme
Nodel Contact
Pullen grejed
Puttee greiind
Timme
Time
Crossformer
Pyraformer
Figure 15: Forecasting demonstrations on ECL (test set; forecasts commence exactly halfway through the entire set) for competitor methods.
30
Maiid Cantrel
Prodkttion
PatchTST64
DLinear
30 Musi
Madel Cantoot groan Proditiinn
Trudittio
1imz
(c) Nhits
MICN
Pysstomer
10
Modd Contect
Modd Cenmost Dn
Preeitie
Prodkttinn
18nz
(c) Crossformer
Pyraformer
Figure 16: Forecasting demonstrations on Traffic (test set; forecasts commence exactly halfway through the entire set) for competitor methods.
31

---

Published as a conference paper at ICLR 2024
IST
DLincer
Model Cantod
Modal Centes
3.5
015
Pradicton
hutuas truih
Prod cthnn
3.20
203
420
200
(a) PatchTST64
(b) DLinear
knite
MEdee Cented
Mooel Cuntod
0.15
0.22
Pres dioe
0.16
0.55
0.50
0.83
-0.25
420
-466
-230
(c) Nhits
(d) MICN
Creestomer
Pyrafome
Meoe Centrnl
0.2
Moont Centeat
Frod slion
Fred dlan
Time
(c) Crossformer
Pyraformer
Figure 17: Forecasting demonstrations on Weather (test set; forecasts commence exactly halfway through the entire set) for competitor methods.
3.2
Mocal Coruok
Mocia Coried Unase Comrod Prediczen
Faors goond
0.5
200
402
812
200
(a) PatchTST64
(b) DLinear
Hodi Contist
Unsidd canrot
Mood Ceripd
future ground ltght
Predcctin
200
412
200
Time
(c) Nhits
(d) MICN
Creechrmer
Pynaformen
Mceel Deciot
Mond Ceeted Pelaee Prodiciin
My
05
Time
(c) Crossformer
Pyraformer
Figure 18: Forecasting demonstrations on Exchange (test set; forecasts commence exactly halfway through the entire set) for competitor methods.
32

---

Published as a conference paper at ICLR 2024
PanchTST
SLineer
Cantrnl
2.
Mocei Centrel
Fatuin
Predition
-200
Time
Time
PatchTST64
DLinear
MICI
Midii Cantest
Midel Centee
Futuls
Proditionn
Predithon
-0.5
Time
Tim
(c) Nhits
MICN
Dnoalarmer
Pynaloma
Model Carted
Model Centec
Futuee greie Yuth
Putulc yoans buth
Proditbei
Presithee
Time
Time
Crossformer
Pyraformer
Figure 19: Forecasting demonstrations on Wind (test set; forecasts commence exactly halfway through the entire set) for competitor methods.
PasthTST
SLlnear
Propiction
0.5
0.5
PatchTST64
DLinear
Wr
MICN
Miciel
ghoand
Fus Predecion
Prog dlann
Timc
(c) Nhits
MICN
Pytetoree
Modill Contad
Progiclan
(c) Crossformer
( Pyraformer
Figure 20: Forecasting demonstrations on USWeather (test set; forecasts commence exactly halfway through the entire set) for competitor methods.
33

---

Published as a conference paper at ICLR 2024
K ADDITIONAL DETAILS FOR HELD-OUT EXPERIMENTS
This appendix details the experimental setup for results in Section 4.2 Table contains three main sections: 'erro-shot', 'fine-tuned", and 'standard training'.
Baselines For standard training of baseline methods we used settings from their respective pa- pers, paying heed to which settings (for short versus long-term forecasting; e.g., Illness versus UCIPower). For baseline methods in the zero-shot setting, we reported the average performance across all model variants trained for Table
DAM zero-shot For the DAM in zero-shot mode first scaled time units by inspection the validation set for some datasets in order to enable the DAM to a better basis function coverage details of these settings in Table Such visual inspection process is realistic compared full validation search over time units when, during deployment, very little data exists to determine what scales are reasonable. We used the same HSR settings as in Table for long-term datasets and reduced context size to 250, ToME reduction to 200, and a to 40 for short-term datasets (Illness, MWeb, and MTemp).
DAM fine-tuned For fine-tuning the DAM implemented short training runs for each target dataset. Owing to difference dataset sizes and complexity, we and conducted a small hyper- parameter search, using validation, over training iterations and initial learning rates (decayed with cosine annealing scheduler to 1-14) to yield the following settings:
1. Illness: starting learning rate of - and 10000 iterations.
2. Weekdays: starting learning rate of 15 and 10000 iterations.
3. UCIPower: starting learning rate of 1 and 10000 iterations.
4. Azure: a starting learning rate of 155 and 10000 iterations.
5. MTemp:: starting learning rate of - and 10000 iterations.
6. MWeb: a starting learning rate of 1 -5 and 10000 iterations.
7. MWeather: starting learning rate of -3 and 30000 iterations.
8. MMeters: starting learning rate of - and 30000 iterations.
Note that these training runs are 2 orders of magnitude shorter than training the DAM from scratch. On a NVIDIA A40 GPU they take between 20 minutes and 1.5 hours.
34

---

Published as a conference paper at ICLR 2024
L
FURTHER BASIS COMPOSITION DEMONSTRATIONS
Figure 21 is an expanded version of Figure 6 and Figure 22 is another version on ETTh2.
of frequencies
13
%
gro Uym
Past ditte
Futuré
Bacecast) Forecast
M
Uyn
63
66
68
%
74 %
76% %
79
81
%
87
89 %
92
95 36
%
100 %
Figure 21: Dataset: ETTml; basis function composition from low to high frequency components. From upper left to lower right, the percentage labels denote the percentiles of frequencies used to compose the forecast (e.g., 10% denotes using all frequencies that constitute the first 10 percentile).
35

---

Published as a conference paper at ICLR 2024
frequencies
13
15
Prest cate
Fulture dase
Blacccaitt
Forerastt
36
50 %
52
55
%
63 %
66
6B
74 %
76 %
79
81 %
%
87 %
89 %
92
%6
1.00
Figure 22: Dataset: ETTh2; basis function composition from low to high frequency components. From upper left lower right, the percentage labels denote the percentiles of frequencies used to compose the forecast (e.g., 10% denotes using all frequencies that constitute the first 10 percentile).
36

---

Published as a conference paper at ICLR 2024
M ADDITIONAL INTERPRETABILITY DEMONSTRATIONS
Figures and are extensions Figure 5 but on different datasets. They show the DAM's attention and resultant basis coefficients for variety of datasets. The supplementary video entitled 'DAM M.AMiineprrciabiity..dem _demol' demonstrates the same interpretability of Figure but as a video as time progresses, and shows how some heads in the attention mechanism focus on certain patterns, while other heads are dedicated to certain points in time. This is not a property that is built into the DAM, but rather one that emerges from learning.
37

---

Published as a conference paper at ICLR 2024
Future data
Samples
4
Backcast
Forecast
Cross attention
Self attention
Layer_head
21
days
Coefficion Cetceent
1.0
hr
S0M
100 hr
12.00
140hi
180
200 ht
22.0
1,0
0v
dv
dv
0y
dy
dy
05
2.9 mk
32.0
56.0 ak
40.0
MMO
52
R VT
70
9.0 yr
Period vs basis weight
(a) ETTh2
Past
Samples
Backcast
Forecast
Cress attention
Time days
Confiicient
0.00
1.0 mm
6.0 hr
9.0 hr
100 hr
16.
18.0 hr
20.0 hr
22.0 hr
1.0 OY
dy
1.2 07
0V
1. 07
07
OY
0Y
WK
wk
wk
2.9 wk
3-7 wk
wk
8.0 wk
12.0 wk.
10.0%00 wis
wik
28.0 wk
22.0 wk
36.0 wic
wis
wk
12.0
1.4
16.0
Peried vs basis weight
(b) ETTml
Past data
Samples
Backcast
Cross attention
Self attention
Layer head
Time
Coefficient
0.50
0.25
0.00
1.0 mn
41.0
60hr
s0M
200
22.0
:8.01
20.0 hr
22.0
1:081
1:100 81
1.2 BV
113 aV
1.4
1.5
1:7 by
1.8 dv
dy
dy
mk
mk
2.9 mk
mk
mk
0.0
12.0
28.0 mk
22.0
36.0
0
12.0 mk
vs basis weight
ETTm2
Figure 23: Extensions to Figure 5 with other datasets.
38

---

Published as a conference paper at ICLR 2024
Future data
Samples
Bajckcast
Forecasst
Cross attention
attention
Layer_head
21
days 14
Coefficion Cnncieett
1.0
41.0
6.hr
5.
100
12.000
140m
160 hr
180
200 hr
22.0
1.000
11000
1.2
dy
1,38
1.4
dy
dy
05
Wk.
mk
2-3 mk
2.6 ak
2.9 mk
wk
ak
mk
12.0 ak
wk
28.0 ak
32 ak
36.0 ak
40.0
68 ak
52 30
ak
S0
10 y
9.0 yr
Period vs basis weight
(a) Traffic
Past
Samples
Backcast
Forecast
Cress attention
Layer_heae
-20
Time days -14
Confiicient
0.00
1.0 mm
6.0 hr
8.0 hr
18. hr
20.0 hr
22.0 hr
1.0 OY
1.2 dy
07
1.3 0V
1.6 OY
1. 07
0V
oy
ay
oy
Wk
wk
2-3 wk
2.5 w/
2.9 wk
3-7 Wk
wk
8.0 wk
12.4 wk
wk
28.0 wi
22.0 wk
36.0 wi
wk
$2.0 wi
90 ST
100
12.0
16.0
10.0%M0
Peried vs basis weight
(b) ECL
Past data
42
Future
41
Saniples
Backcast
Forecast
Cross attention
Self attention
Layer hhea
Time
Coefficient
0.50
0.25
0.00
1.0 mn
41.0
60hr
200
12.0
18.01
20.0 hr
22.0
1:0
1:
1.2 0V
1.2 Av
1.3.8 BV
1.4 dy
1.5
17880 dV
1:7 bV
1.8 dv
dy
dy
6.000 dy
mk
mk
2.9 mk
mk
mk
0.0 mk
12.0 mk
28.0 ak
22.0
36.0
12.0 mk
20
Perted vs basis weight
(c) Weather
Figure 24: Extensions to Figure 5 with other datasets.
39

---

Published as a conference paper at ICLR 2024
N SCALING LAWS OF FOUNDATION MODELS
The term foundation model was coined by researchers in the field as "any model that is trained on broad data (generally using self-supervision scale) that can be adapted (e.g., fine-tuned) wide range of downstream tasks." (Time 2023) Scaling laws (Kaplan et al. 2020) Frantar et al. (023) are property (rather than requirement) of almost all larger models, not particularly just foundation indeed do not necessarily imply the required generalisation iitiaiisaaiontthuughh they are helpful understanding how the future development might unfold.
Owing to time and resource constraints, we trained several variants of the DAM on ETTh1 (using shorter training run of 100000 iterations) in order build a perspective on how the DAM scales with increased model size. Figure 25 shows the test performance of ETTh1 for different model sizes and also indicates the DAM trained for this paper. While not conclusive, these results show that the DAM scales well with increased model size. Further, training across multiple datasets and for longer has a strong positive impact, as evidenced by the superior performance of the DAM used throughout this paper.
0.52
0.50
MSE on ETTh1 test set
0.48
0.46
0.44
0.42
Model trained across 25 datasets
Line fit
0.40
2.00M
4.00M
6.00M
8.00M
10.00M
No. parameters
Figure 25: The DAM scaling laws. We trained several variations of the DAM. by scaling D and the number layers, on ETThl to develop this plot. Also shown is the performance the single DAM used throughout this paper (i.e., across 25 datasets), evidencing that it benefits from training across diverse datasets.
What defines foundation model not that is trained across multiple datasets, but that it performs well when tested across diverse tasks (and hence datasets) that not of the exact form it is trained on something that we have done extensively in Section All foundation models have a scope (language, vision tasks, etc.) and the scope of the DAM is univariate time series forecasting.
40


Reviews:
[{"review_id": "QgjhJPTGii", "reviewer": "Reviewer_yXbk", "summary": "This is an interesting paper proposing to solve a very general time series problem. This paper aims at flexible (long or short horizon) time series forecasting with long or short context input for any general time series. Moreover the context could also be irregularly sampled. The paper additionally proposes a strategy for efficient long context inputs.\n\nThe model is therefore a general time series forecaster and trained on many time series for generalization to any held out datasets. As such it could be considered a foundational model for time series forecasting.\n\nExperimental results are promising, especially on held out datasets. One of the main advantages of a foundational model is to be general enough to be able to predict on held out datasets. While the results are most positive on held out datasets, the model performs at par with the baselines on the training datasets. To summarize, I believe this to be an interesting contribution to the time series literature, and I am willing to increase my score if the concerns below are addressed.\n\nUpdate: Increased the score", "questions": "- When using a trained model for forecasting a single time series, how does inference look like in such a simple setting? Do attention models work well in such scenarios? In other words, do attention models produce better forecasts when a larger context in provided? The larger context could be in the form of multiple time series or a larger history.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The problem being solved in the paper is one of the most interesting problems in the time series community i.e. general time series forecasting for flexibly sampled time series.\n- The proposed solution (described as a foundational model) involving attention is a befitting solution to this problem by virtue of its ability to adapt to flexible input sizes.\n- Experimental results are promising on both training and held out datasets.", "weaknesses": "- The model architecture is quite complex and the rationale behind such a choice is not fully explained. One important question that is raised is the usefulness of each of the components in the model. An ablation study may be performed to measure the importances of each of the components proposed in the paper.\n- Simple linear forecasting models are not considered as baselines. For comparison between different models a normalized score is preferred rather than a absolute score. Normalization helps interpret the improvements easier since they are scaled. As such, the experimental section need improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1YuSmwTF8H", "reviewer": "Reviewer_wavr", "summary": "The paper proposes deep data-dependant approximate analytical model (DAM) as a \"foundational model\" for time series forecasting. DAM uses a long tail distribution to sample from the history of the time series. These irregularly-sampled time-value pairs are fed into a transformer-based model which outputs basis coefficients. The basis coefficients are then used in a basis function composition to generate forecasts. The authors trained a single DAM model across multiple datasets and show that this model is competitive against dataset-specific baselines on long-term forecasting benchmarks. Small-scale analyses have also been conducted on very long-term forecasting and imputation tasks.", "questions": "- How exactly is inference performed on a dataset? Is basis function initialization also required during inference?\n- How costly is context selection during inference, in general?\n- Can you clarify what is meant by \"No training of the backbone is even required in this case because the initialisation coefficients are optimal for past data\"? Is model training not needed for imputation?\n- You mention \"The DAM produces relatively high basis coefficients for ETTh2 in the year range, suggesting that it captures the long-term trend.\" but the ETTh2 dataset only has data over two years (from 2016 to 2018) while relatively high positive values are seen for 3, 5, 7, 9 yr basis components. How do you explain this phenomenon?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The combinations of ideas presented in this work involving history regime sampling, basis composition-based forecasting, and training a single model across multiple datasets are original and interesting.\n- The model provides the flexibility to forecast arbitrarily far into the future which is an attractive property. While autoregressive models can already do that, they are generally slow.\n- The paper is well-written in general. Some discussion and visualizations can be improved (see weaknesses).", "weaknesses": "- The main weakness of this paper is that it overclaims and underdelivers. In its current state, the study is not strong enough to claim the title of a \"foundational model\".\n    - The authors mention that they use datasets from diverse domains. However, out of the 12 datasets studied, 6 come from a single domain. The distribution of sampling frequencies of these datasets are also not diverse with 6 hourly datasets and a limited representation of other frequencies (and some popular frequencies completely missing).\n    - Another aspect that could have justified the term \"foundational\" is a diversity of tasks. However, the paper mostly focuses on the long-term forecasting tasks with limited discussion of other tasks. Importantly, the practically relevant task of short-term forecasting (e.g., Monash time series forecasting archive) gets very less attention.\n    - The claim _Most existing forecasting models were designed to process regularly sampled data of fixed length. We argue that this restriction is the central reason for poor generalisation in time series forecasting_ has not been justified convincingly.\n- The visualizations are poorly done and confusing for a serious academic paper. Please consider using cleaner figures. It is unclear how exactly inference on a new dataset is performed. It would improve the clarity of the paper if a specific paragraph on inference is added. Please see specific questions in the questions section.\n- The results on the long-term forecasting benchmarks, while reasonable, are not impressive for a \"foundational model\" that has been trained on a larger corpus of datasets.\n- The very-long-term forecasting task is of limited practical significance. Despite that, the discussion requires improvement, e.g., by conducting experiments on more datasets and training the baseline models with the \"correct\" forecast horizon to put the results in a proper context.\n- The zero-shot analysis (Sec. 4.3) has only been conducted on two datasets. Moreover, since prior works such as PatchTST and NHITS do not claim to be foundational models, a proper comparison would be with baselines trained specifically on these held-out datasets. DAM would most likely be worse in that case but it would be a better gauge for the zero-shot performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VFKPKX1C1U", "reviewer": "Reviewer_sH5F", "summary": "The paper proposes a foundation model for universal time-series forecasting. Building such a model is challenging because the sample resolution, periodicity, and prediction task are different. To address these challenges, the paper proposes to take randomly sampled histories and output coefficients of basis functions. The designed basis function enables forecasting to non-fixed horizons.\nExperiments show that the single model trained on 10 datasets can outperform or match existing SOTA dataset-specific models. It also performs well on held-out datasets and has good interpretability.", "questions": "How does the training cost of the proposed method compare with the baselines?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Building a foundation model for time series forecasting is challenging because different datasets have different patterns and resolutions. This paper proposes reasonable solutions to address these challenges. The experimental results also show that the proposed method is promising. Additionally, it has good interpretability and is robust to missing and irregularly sampled data.", "weaknesses": "Training such a foundational model requires high computational cost. The biggest advantage of the foundation model is its ability for zero-shot forecasting. However, the paper only reports the performance on two held-out datasets which is not sufficient. This restricts the practical use of the model in the real-world application.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hnKVZkz7W6", "reviewer": "Reviewer_PhNp", "summary": "The author proposes a foundation model for time series forecasting called DAM (deep data-dependent approximate analytical model). \nDAM takes input time series data and samples (time, value) pairs using HSR (history sampling regime) technique. HSR is not a regular and fixed-length sampling method unlike other time series forecasting techniques. Instead, it defines a probability distribution that is closer to the top-head section as the sampling points are closer to the prediction target point, and closer to the long-tail section as they are farther away. The author claims that this method is a major difference from other time series forecasting models and prevents overfitting and improves generalization performance. The sampled (time, value) pairs and the coefficient values initialized by the linear solver are used in the transformer to determine the coefficient values of 219 basis functions. The final predicted time-series is a combination of the basis functions. \nThe author evaluated 12 benchmark datasets used at training and 2 datasets not used at training (for validation of generalization performance), and compared them with 6 existing sota models trained specifically for each dataset, showing better or similar performance. In particular, DAM showed excellent performance in long-term forecasting. Additionally, DAM can adjust the trade-off between inference cost and performance by changing the context size, and also provides interpretabilty features based on basis function composition and attention map analysis.", "questions": "1. According to the description of Table 1, DAM performed multivariate forecasting. I guess that when creating (time, value) pairs, multiple variables were included in the value part. Is that right? If so, I think it should be more clarified in the paper.\n\n2. At Figure 6, it seems that the inference time can be reduced from 200ms to below 100ms depending on the context size. However, the inference time and cost seem to be very small compared to LLM (~seconds), so it seems possible to reduce costs by increasing the computing resources not expensively at around 200ms. I don’t fully understand what the authors want to deliver in 5.1 FLEXIBLE INFERENCE COST. Could you explain more about it? (sorry)", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. Addressing foundation model for time-series forecasting model is a significant and timely task (and it is a very difficult problem). \n\n2. The irregular and non-fixed size sampling technique is interesting.\n\n3. Various and numerous experiments were conducted to verify the ability of DAM.", "weaknesses": "1. To be a foundation model, it is necessary to verify that the computational amount and performance follow the scaling law, but there is no related content. Even if the computational amount and performance do not follow the scaling law in time series forecasting, the authors should have explained why they had to use 10 datasets to train DAM (not single dataset or many more), and what was the bottleneck for improving the foundation model performance.\n\n2. The paper too much depends on appendix. The content provided in the main paper should be a complete argument, but it is difficult to follow without looking at the appendix. For example, in 4.1 Results and discussion, the authors need to summarize the content of Table 1 and argue the authors’ claim, but it is omitted. Instead, the content of the appendix is summarized. To be a regular paper, it seems that the structure of the paper needs to be more organized.\n\n3. One of the most important abilities of a time series forecasting model is peak time prediction, but as the authors pointed out in Limitations, DAM does not seem to predict well. This limits the role of DAM in real-world problems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)