Scaleformer: Iterative Multi-scale Refining Transformers for Time Series Forecasting
Authors: Mohammad Amin Shabani
Amir H. Abdi
Lili Meng
Tristan Sylvain
Venue: ICLR 2023 poster (2023)
Keywords: Time-series forecasting
Transformers

Abstract: The performance of time series forecasting has recently been greatly improved by the introduction of transformers. In this paper, we propose a general multi-scale framework that can be applied to state-of-the-art transformer-based time series forecasting models
(FEDformer, Autoformer, etc.). Using iteratively refining a forecasted time series at multiple scales with shared weights, architecture adaptations and a specially-designed normalization scheme, we are able to achieve significant performance improvements with minimal additional computational overhead. Via detailed ablation studies, we demonstrate the effectiveness of our proposed architectural and methodological innovations. Furthermore, our experiments on various public datasets demonstrate that the proposed method outperforms the corresponding baselines. Depending on the choice of transformer architecture, our mutli-scale framework results in mean squared error reductions ranging from 5.5% to 38.5%. Our code is publicly available in https://github.com/BorealisAI/scaleformer.

Published as a conference paper at ICLR 2023
SCALEFORMER: ITERATIVE MULTI-SCALE REFINING TRANSFORMERS FOR TIME SERIES FORECASTING
Mohammad Amin Shabani, Simon Fraser University, Canada & Borealis AI, Canada mshaban @ @sfu.ca
Amir Abdi, Lili Meng, Tristan Sylvain
Borealis AI, Canada (fi (firstname lastnamm))@oo aabbreallsat s boai.co com
ABSTRACT
The performance of time series forecasting has recently been greatly improved by the introduction of transformers. In this paper, propose general multi-scale framework that can be applied to the state-of-the-art transformer-based time series forecasting models (FEDformer, Autoformer, etc.). By iteratively refining fore- casted time series at multiple scales with shared weights, introducing architecture adaptations, and a specially-designee normalization scheme, we able achieve significant performance improvements, from 5.5% to 38.5% across datasets and transformer architectures, with minimal additional computational overhead. Via detailed ablation studies, we demonstrate the effectiveness of each of our contri- butions across the architecture and methodology. Furthermore, our experiments on various public datasets demonstrate that the proposed improvements outper- form their corresponding baseline counterparts. Our code publicly available https://github. hwtptthhhu..cooeeeeeee Aww.aoorraaaasaalllleaeeeeee
INTRODUCTION
Course Forecasting
Integrating information at different time scales is es- sential to accurately model and forecast time series (Mozer, 1991; Ferreira et al., 2006). From weather patterns that fluctuate both locally and globally, as well as throughout the day and across seasons and years, to radio earrier waves which contain relevant signals at different frequencies, time series forecast- ing models need to encourage scale awareness in learnt representations. While transformer-based ar- chitectures have become the mainstream and state- of-the-art art for time series forecasting in recent years,
Fine Forecustng
advances have focused mainly on mitigating the stan- dard quadratic complexity in time and space, e.g.,
attention (Li et al., 2019; Zhou et al., 2021) or struc- tural changes (Xu et al., 2021; Zhou et al., 2022b), rather than explicit scale-awareness. The essential cross-scale feature relationships are often learnt im- plicitly, and are not encouraged by architectural priors of any kind beyond the stacked attention blocks that characterize the transformer models. Autoformer (Xu al., 2021) and Fedformer (Zhou et al., 2022b) introduced some emphasis on scale-aware scale-awareness by enforcing different computational paths for the trend and seasonal components of the input time series; however, this structural prior only focused on two scales: low- and high- figh-freeuency components. Given their importance to forecasting, can make transformers
Figure I: Intermediate forecasts by our model at different time scales, Iterative refinement of a time series forecast is a strong structural prior that benefits time series forecasting.
scale-aware?
We enable this scale-awareness with Scaleformer. In our proposed approach, showcased in Figure 1, time series forecasts iteratively refined at successive time-steps, allowing the model to better capture the ineededeps and specificities each scale. However, scale itself is not sufficient. Iterative refinement different scales can cause significant distribution shifts between intermediate

---

Published as a conference paper at ICLR 2023
forecasts which can lead to runaway error propagation. To mitigate this issue, introduce cross- scale normalization at each step.
Our approach re-orders model capacity to shift the focus scale awareness, but does not fundamen- tally alter the aatention-drivee paradigm of transformers. As a result, it can be readily adapted to work jointly with multiple recent time series transformer architectures, acting broadly orthogonally to their own contributions. Leveraging this, we chose to operate with various transformer- based backbones (e.g. Fedformer, Autoformer, Informer, Reformer, Performer) further probe the effect of our multi-scale method on a variety of experimental setups.
novel iterative scale-refinement paradigm that can be readily adapted to a variety of transformer- time series forecasting architectures, (2) To minimize distribution shifts between scales and windows, we introduce cross-scale normalization on outputs of the Transformer. (3) Using Informer and AutoFormer, two state-of-the -frte--r-- architectures, as backbones, we demonstrate empirically the effectiveness of our approach on a variety of datasets. Depending the choice of transformer architecture, our mutli-scale framework results in mean squared error reductions ranging from 5.5% to 38.5%. (4) Via a detailed ablation study of our findings, we demonstrate the validity of our architectural and methodological choices.
Our contributions
follows: (1) we introduce
2 RELATED WORKS
Time-series forecasting: Time-series forecasting plays important role in many domains, including: weather forecasting (Murphy, 1993), inventory planning (Syntetos et al., 2009), astronomy (Scargle, 1981), economic and financial forecasting (Krollner al., 2010). One of the specificities of time series data is the need to capture seasonal trends (Brockwell & Davis, 2009). There exits a vast variety of time-series forecasting models (Box & Jenkins, 1968; Hyndman al., 2008; Salinas al., 2020; Rangapuram et al., 2018; Bai et al., 2018; Wu et al., 2020). Early approaches such as ARIMA (Box & Jenkins, 1968) and exponential smoothing models (Hyndman et al., 2008) were followed by the introduction of neural network based approaches involving either Recurrent Neural Netowkrs (RNNs) and their variants (Salinas et al., 2020; Rangapuram al., 2018; Salinas et al., 2020) or Temporal Convolutional Networks (TCNs) (Bai et al., 2018).
More recently, time-series Transformers (Wu et al., 2020; Zerveas et al., 2021; Tang & Matteson, 2021) were introduced for the forecasting task by leveraging self-attention mechanisms to learn complex patterns and dynamics from time series data. Informer (Zhou et al., 2021) reduced quadratic complexity in time and memory to OL log L) by enforcing sparsity in the attention mechanism with the ProbSparse attention. Yformer (Madhusudhanan et al., 2021) proposed a Y-shaped encoder- decoder architecture to take advantage of the multi-resolution embeddings. Autoformer (Xu et al., 2021) used a ceessscorrelatio-beee FEDformer (Zhou et al., 2022b) employs frequency transform decompose the sequence into multiple frequency domain modes to extract the feature, further improving the performance of Autoformer.
attention mechanism to operate at the level of subsequences.
Multi-scale neural architectures: Multi-scale and hierarchical processing is useful in many domains, such as computer vision (Fan et al., 2021; Zhang et al., 2021; Liu et al., 2018), natural language processing (Nawrot et al., 2021; Subramanian et al., 2020; Zhao et al., 2021) and time series forecasting (Chen et al., 2022; Ding al., 2020). Multiscale Vision Transformers (Fan al., 2021) is proposed for video and image recognition, by connecting the seminal idea of multiscale feature hierarchies with transformer models, however, focuses on the spatial domain, specially designed for computer vision tasks. Cui et al. (2016) proposed to use different transformations of a time series such as downsampling and smoothing in parallel to the original signal to better capture temporal patterns and reduce the effect of random noise. Many different architectures have been proposed recently (Chung et al., 2016; Che et al., 2018; Shen et al., 2020; Chen et al., 2021) to improve RNNs in tasks such as language processing, computer vision, time-series analysis, and speech recognition. However, these methods are mainly focused on proposing a new RNN-based module which is not applicable to transformers directly. The same direction has been also investigated in Transformers, TCN, and MLP models. Recent work Du et al. (2022) proposed multi-scale segment- wise correlations multi-scale version of the self-attentio mechanism. Our work is orthogonal to the above methods
2

---

Published as a conference paper at ICLR 2023
Xpe
Step
X
Forecasting
X
Xi
Cross-scale Normalization
Skp
Mall
Lass function
k
X
Sepo
Upsammle
X
Figure 2: Overview of the proposed Scaleformer framework. (Left) Representation of a single scale block. In each step, pass the normalized upsampled output from previous step along with the normalized downsampled encoder as the input. (Right) Representation of the full architecture. We process the input in multi-scale manner iteratively from the smallest scale the original scale.
as model-agnostic framework to utilize multi-scale time-series in transformers while keeping the number parameters and time complexity roughly same.
METHOD
In this section, first introduce the problem setting in Sec. 3.1, then describe the proposed framework in Sec. 3.2, and the normalization scheme in Sec. 3.3. We provide details on the input's representation in Sec. 3.4, and the loss function Sec. 3.5.
3.1 PROBLEM SETTING
We denote (L) and (H) the look-back and horizon windows for the forecast, respectively, of corresponding lengths fr, ET. Given starting time to we can express these time-series of dimension dz. follows: (I.) {x||x; e Rd= [to. to + fg]} and X {x|Xx, Rd E [to lg
+ l, + LH]. The goal of the forecasting task predict the horizon window X (H) given the book-back window (4)
3.2 MULTI-SCALE FRAMEWORK
Our proposed framework applies successive transformer modules to iteratively refine time-series forecast, different temporal scales. The proposed framework is shown Figure 2.
Given input time-series (L) iteratively apply the same neural module mutliple times
different temporal scales. Concretely, we consider a set of scales S = {sTm ".m...... s2 2.8" 1} (i.e. for the default scale of
[log, fr. and is downscaling factor. The input to the encoder at the i-th step (0 < i < m)) is the original look-back window (1), downsampled by scale factor of s; = s m- via an average pooling operation. The input to the decoder, on the other hand, is X upsampled by factor of via linear interpolation.
S
of consecutive powers of 2), where

---

Published as a conference paper at ICLR 2023
Finally, xge is initialized to array of Os. The model performs the following operations:
X,,
Xy,
==
(1)
8i
7+1
x!
Xt.à
lo
<
l0 lt.
(2)
5i
2 to
S1S to + ll lH
x!
%t,b
81
8i
where x!4) and x(#) are the look-back and horizon windows at the ith step at time with the scale factor of sm and with the lengths of f1 and HH respectively. Assuming x' 2ii-1 is the output of the forecasting module at step - and time I, we can define x and X as the inputs to the normalization:
(H)
X x(() (L)
(4)
-- [t/8]
x" 1, = x Lt/ajii11 + (x' 11/0]/01 -X x (1/a)/-1)
(5)
x = fx" to + l +1
ht to + E1 + LH
(6)
8i
8;
Finally, calculate the error between X
(H)) and X."It as the loss function to train the model. Please
refer to Algorithm for details on the sequence of operations performed during the forward pass.
Algorithm Scaleformer: Iterative Multi-scale Refining Transformer
Require: input lookback window (L) E R& xda, scale factor s,
set of scales S {s" ..... s2 s², 1), Horizon length εH, and Transformer module F. for i + 0 to m do
x.nc AvgPool (x.),, window size=s"--1)
Equation (1) and (2) of the paper
then
X dec
else
X +- Upsample (X scale=s)
Equation (5) and (6) of the paper
end if
f E2, Exxerxx xene Extrxx xdee
ixx
d +- ; fxx;
X.n +- X.ne Dxx,
X +- F (x X. + Pxx.
end for
Ensure: X
return the prediction all scales
3.3 CROSS-SCALE NORMALIZATION
Given a of input series (X.Th X with dimensions l, d2 and lH, x de, respectively for the encoder and the decoder of the transformer in ith step, we normalize each series based on the temporal average of e and d More formally:
Σ Ex
dec
ΛX.
lLeb + EHH
x
X = x.
Tix..
Fene
X
PX.
(8)
where ,x, E Rd is the average over the temporal dimension of the concatenation of both look-back window and the horizon. Here, xçae and iddce the inputs of the ith step to the forecasting module.
4

---

Published as a conference paper at ICLR 2023
Without Normalization
With Normalization
Original Ditstribution
Downsampled Distribution
0.8
0.4
0.4
0.2
1.0
0.50
0.5
0.25
Figure 3: The figure shows the output results of two series using the same trained multi-scale model with and without shifting the data (left) which demonstrates the importance of normalization. On the right, we can see the distribution changes due to the downsampling of two series compared to the original scales from the Electricity dataset.
Distribution shift is when the distribution of input model its sub-components changes across training to deployment (Shimodaira, 2000; loffe & Szegedy, 2015). our context, two distinct distribution shifts are common. First, there natural distribution shift between the look-back window and the forecast window (the covariate shift). Additionally, there is a distribution shift between the predicted forecast windows at two consecutive scales which is result of the upsampling operation alongside the еттог accumulation during the intermediate computations. As result, normalizing the output given step by either the look-back window statistics the previously predicted forecast window statistics result in an accumulation of errors across steps. We mitigate this by considering moving average of forecast and look-back statistics the basis for the output normalization. While this change might appear relatively minor, it has significant impact on the resulting distribution of outputs. The improvement is more evident when compared to alternative approaches, namely, normalizing by either look-back or previous forecast window statistics.
3.4 INPUT EMBEDDING
Following the previous works, we embed our input to have the same number of features as the hidden dimension of the model. The embedding consists of three parts: (1) Value embedding which uses linear layer to map the input observations of each step X₃ to the same dimension as the model. We further concatenate an additional value 0, 0.5, or 1 respectively showing each observation is coming from the look-back back window, zero initialization, or the prediction of the previous steps. (2) Temporal Embedding which again uses linear layer to embed the time stamp related each observation the hidden dimension of the model. Here concatenate an additional value 1/s; 0.5 as the current scale for the network before passing to the linear layer. (3) We also use a fixed positional embedding which is adapted to the different scales s; follows:
pos X
pos X
PE PE(pos, 2k, si) = sin (100002k/dmazz /ammia
=cos PE(pos, 2h + 11, = cos
(9)
(100002k/ daniel
3.5
LOSS FUNCTION
Using the standard MSE objective to train time-series forecasting models leaves them sensitive to outliers. One possible solution is use objectives more robust outliers, such as the Huber loss (Huber, 1964). However, when there are no major outliers, such objectives tend to underperform. Given the heterogeneous nature of the data, we instead utilize the adaptive loss (Barron, 2019):
a 2| (( ((5102
a/2 -1)
a(E,a,c)
+1
(10)
5

---

Published as a conference paper at ICLR 2023
Table 1: Comparison of the MSH and MAE results for our proposed multi-scale framework version of different methods (-MSA) with respective baselines, Results are given in the multi- variate setting, for different lenghts of the horizon window. The best results are shown in Bold. Our method outperforms vanilla version of the baselines over almost all datasets and settings. The average improvement (error reduction) shown in Green numbers at the bottom with respect the base models.
Method/ Dataset
FEDXanmer
PED MSE MAF
Autofarmer MSE MAF
Auto-MSA MSF
Informer MSE
Info-MSA MAF
Kefonmer MSB
Ref-MSA MSE MAB
Performer MSE
Per-MSA
MSE MAE
MAF
MAF
MSE
MAF
MAR
MSE
MAB
96
0.285
0.109
0.240
0.285
0.126
0.966
0.168
0.298
0.311
10667
0.305
Exchange
192
336
0.452
0.38-
0.498
0.241
0.471
0.383
0.508
0.441
0.28
0.495
0.519
0.253
0:538
1.598
1.888
1.016
0.427
0.500
0.484
0.535
1.712
1.597
1.070
1.029
0.375
0.605
0.446
0.591
1.081
1.339
0.344
0.439
0.563
0,486
0.577
220
1.172
0.839
1,259
0.965
1.118
0.819
0.928
0.751
2,079
1.340
1,017
0.790
1.918
1.100
1,089
0.857
0.867
0.766
1.219
0.882
96
10.288 0.365
00.22
0.289
0.267
0.334
0.163
0.226
10,388
0.435
0.210
0.279
0.347
0.388
0.199
0.263
00.441
0.479
0.228
0.291
Weather
336
192
00,447 (:469
0:425
0,463
0.455
0.376
0.397
0.282
0:433
0.610
0.289
0.418
0.427
0.734
0.463
0.463
0.294
0.464
0:475
0:478
0.302
0.441
0.456
720 [.6400 0.574
0.682
0.565
0.425
0.434
0.369
0.396
0.977
0.723
0.595
0.532
0.815
0.674
0.493
0.471
0.563
0.552
0.817
0.655
96 [0.201 0.317
00.12
0.297
0.197
0.312
0.188
0.303
0..344
0.421
0.203
0.315
0.294
0.382
0,183
0.291
0.2294
0.387
0.190
0.300
Electricity
192 0.200 0.314
0.188
0.300
0.219
0.329
0.197
0.310
0.344
0.426
0.219
0.331
0.331
0.409
0.194
0.304
0,305
0.400
0.200
0.310
336 0.330
0.324
0.359
0.234
0.358
0.253
0.361
0.321
0.331
720 (0.239 0.350
0.233
0.339
0.290
0.380
0.249
0.358
0.386
0.452
0.293
0.390
0.316
0.393
0.234
0.340
0.304
0.386
0.228
0.335
96 [0.601 0.376
(0.664
0.351
0.628
01393
0.567
0.350
10.748
0.426
0.597
0.369
00.698
0.386
0.615
0.377
[0:730
0.405
0.612
0.371
Trallic
336
192 0.6633 0.379
0.602 0.375
0.57
0.570
0,349
0.349
0.634
0.619
0.385
(.40)
0.609
0,589
0.360
0.383
0,868
0.772
0.493
0.436
0,761
0.655
0.455
0.399
0.694
0.695
0.379
0.377
0.617
0.613
0.360
0.367
0.698
0.678
0.287
0.370
0.604
0.608
0.356
0.368
720
0.378
0.602
0.360
0.403
0.642
1.074
0.606
0.924
0.521
0.692
0.376
0.638
0.360
0.672
0.361
0.634
24
33,025
1.189
2.745
1,075
3,862
1.370
3,370
1.213
5:402
3,742
1.252
3.961
1.288]
3,534
1.212
1,806
1.471
3.437
1.148
[L]
48
32
2,444
3.034
1.041
1.201
2.748
2.793
1.072
1.059
2
5:871
1.138
1.379
3
3.207
1.164
1.153
5.225
5:296
1.569
1.587
3.807
3,940
1.272
1.272
4.022
4,269
1.340
1,311
3.652
3,506
1.168
1.235
4.669
4.488
1.371
1.455
4.055
4.055
1.248
1.248
64
2,686
1.112
2.678
1,071
3.164
1.223
2.954
5.504
1.578
3,670
1,234
1,370
1.385
3,487
1.177
4.607
1.401
3.828
1.224
Va Ours
55.6%
5.9%
13.55%
9.15% 1
38.5%
26.7%
38.3% 25.25
23.35
16.9%
with E = (Xqui X
in step i. The parameters a and c, which modulate the loss sensitivity to outliers, are learnt in an end-to-end fashion during training. To the best of our knowledge, this is the first time this objective has been adapted to the context of time-series forecasting.
4 EXPERIMENTS
In this section, we first showcase the main results of our proposed approach on a variety of forecasting dataset in Sec, 4.1. Then, we provide an ablation study of the different components of our model in Sec. 4.2, and also present qualitative results in Sec. 4.3. Moreover, we discuss series of additional extensions of our method in Sec. 4.4, shedding light on promising future directions.
4.1
MAIN RESULTS
Baselines: To measure the effectiveness of the proposed framework, we mainly use state-of-the-a art transformer-based models FedFormer (Zhou et al., 2022h) Reformer (Kitaev et al., 2020), Per- former (Choromanski et al., 2020), Informer (Zhou et al., 2021) and Autoformer (Xu et al., 2021) which are proven to have beaten other transformer-bass former (Kitaev et al., 2020)). RNN-based (e.g. LSTMNet (Lai et al., 2018), LSTM) and TCN (Bai et al., 2018) models. For brevity, we only keep comparisons with the transformer models in the tables.
(e.g. LogTrans (Li et al., 2019), Re-
Datasets: We consider four public datasets with different characteristics to evaluate our proposed framework. Electricity Consuming Load (ECL)' corresponds to the electricity consumption (Kwh) of 321 clients. Traffic' aggregates the hourly occupancy rate of 963 car lanes of San Francisco bay area freeways. Weather contains 21 meteorological indicators, such as air temperature, humidity, etc, recorded every 10 minutes for the entirety of 2020. Exchange-Rate Rate (Lai et al., 2018) collects the daily exchange rates of 8 countries (Australia, British, Canada, Switzerland, China, Japan, New Zealand and Singapore) from 1990 to 2016. National Illness (ILI) corresponds to the weekly
hhtpss//archive.. ics http::/wwwiiicocc........naannne www.ea/hwwlllaaeeeeeeeeiiaiaaaae 10014
hhtpss//ppmmsde https///pem..do.....ov
'https://www.. http://wwwwwww/.....ee www.bhgga.mpg.ddeeeee wtw.w///tteeee////////m/m///
4 "hups://gis.cde www.g/aawwhniiooohhhaasssaaaaa.m
6

---

Published as a conference paper at ICLR 2023
Table 2: Multi-scale framework without cross-scale normalization. Correctly normalizing across different scales (as per our cross-mean normalization) is essential to obtain good performance when using the multi-scale framework.
PEDiomer
FED-MS (wlo
Autoformer
MS
N)
Informer
Info- MS-
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Electricity Weather
96
0.288
0.365
0.300
0.342
0.267
0.334
0.191
0.277
0.388
0.435
0.402
0.438
0.425
0.376
0.393
336
0.447
0.469
0.531
0.493
0.364
0.397
0.376
0.420
0.610
0.551
0.566
0.528
96
0.201
0.317
0.258
U.356
0.197
0.312
0.221
0.337
0.344
0.421
0.407
0.465
192
0.200
0.314
0.259
0.357
0.219
0.329
0.251
0.357
0.344
0.426
0.407
0.469
336
0.214
0.330
0.268
0.364
0.263
0.359
0.288
0.380
0.358
0.440
0.392
0.461
recorded influenza-like illness patients from the US Center for Disease Control and Prevention. We consider horizon lengths of 24, 32, 48, and 64 with an input length of 32.
Implementation details: Following previous work (Xu et al., 2021; Zhou et al., 2021), we pass eenc_ ( as the input to the encoder. While an array of zero-values would be the default to pass the decoder, the decoder instead takes as input the second half of the look-back window padded with zeros X = (Xto /2,..... .... .t,, . 0, .....) 0} with length L + εH. The hidden dimension of models 512 with batch size of 32. We use the Adam optimizer with a learning rate of le-4. The look-back window size is fixed to 96, and the horizon is varied from 96 to 720. We repeat each
experiment times and report average values to reduce randomness. For additional implementation details on our model and baselines please refer to Appendix A.
Main results and comparison with baselines: Table shows the results of the proposed framework compared to the baselines. Our proposed multi-scale framework with the adaptive loss outperforms the baselines almost all of the experiments with an average improvement of 5.6% over FEDFormer, 13% over Autoformer and 38% over Informer which are the three most recent transformer-base architectures on MSE. We also achieved significant error reduction on MAE. The improvement is statistically significant in all cases, and in certain cases quite substantial. In particular for the exchange-rate dataset, with Informer and Reformer base models, our approach improves upon the respective baselines by over 50% averaged over the different horizon lengths.
Time and memory complexity: The proposed framework uses the same number parameters for the model as the baselines (except two parameters and of the Adaptive loss). Our framework sacri- fices a small amount of computation efficiency for the sake of significant performance improvement. We expand our analysis in Appendix C. As shown in Table 4 in Appendix, if we replace the operation at the final scale by an interpolation of the prior output, we can achieve improved performance over the baselines, at no computational overhead.
Informer
Autoformer
FEDformer
MSB
2.4
Electricity
Exchango
Traffic
Weather
Electricity Exchange
Traffic
Weather
Electricity
Exchange
Traffic
Weather
Figure 4: Comparison of training with Adaptive loss "-A", multi-scale framework with MSE loss "-MS". Multi-scale framework and Adaptive loss "-MSA". It shows the combination of all our proposed contributions is essential, and results in significantly improved performance.
7

---

Published as a conference paper at ICLR 2023
Table 3: Single-scale framework with cross scale normalization "-N". The cross-scale normalization (which in the single-scale case corresponds to mean normmalzaaiiininnn of the output) does not improve the performance of the Autoformer, as already has an internal trend-cycle normalization component. However, it does improve the results of the Informer and FEDformer.
Daraset
FEDfomer
FEDformer-N
Autoformer
Autoformer-N
Informer
Informer-N
Metric
MSB
MAB
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Blectricity Wealher
0.288
0.365
0.234
0.292
0.267
0.334
0.323
0.401
0.388
0.435
0.253
0.333
192
0.368
0.425
0.287
0.337
0.323
0.376
0.531
0.543
0.433
0.453
0.357
0.408
0.447
0.436
0.443
0.364
0.859
0.708
0.610
0.551
0.459
0.461
720
0.640
0.574
0,545
0.504
0.425
0.434
1.682
1.028
0.978
0.723
0.870
0.676
96
0.201
0.317
0.194
0.307
0.197
0.312
0.251
0.364
0.344
0.421
0.247
0.356
192
0.200
0.314
0.195
0.304
0.219
0.329
0.263
0.372
0.344
0.426
0.291
0.394
0.200
0.310
0.359
0.440
0.321
0.416
720
0.239
0.350
0.225
0.332
0.290
0.380
0.280
0.385
0.386
0.452
0.362
0.434
4.2 ABLATION STUDY
We present main ablation studies in this section, more ablation results are shown in Appendix G.
Impact of each component: Two important components of our approach are the multi-scale frame- work and the use of the adaptive loss. We conduct multiple experiments (1) removing the multi-scale framework and/or (2) replacing the Adaptive loss by the MSE for training, in order to demonstrate the benefits of these two components. Figure 4 shows the effect of multi-scale and the loss function with different base models. Considering the impact of ablating the adaptive loss, we can see that for both the multi-scale and base models, training with the adaptive loss improves performance. Similarly, adding the multi-scale framework improves performance, both with and without the adaptive loss. Overall, combining the adaptive loss with the multi-scale framework results in the best performance.
Cross-scale normalization: As we discussed in Section 3.3, having the cross-scale normalization is crucial to avoid the distribution shifts. To confirm that, conduct two experiments. Firstly, the multi-scale framework without the cross-scale normalization to argue that the error accumulation and covariate shift between the scales leads higher error compared to only a single scale. As shown in Table 2, while the multi-scale framework can get better results in a few cases, it mostly have higher errors than baselines.
On the other hand, adding the normalization with only single scale can still help to achieve better performance by reducing the effect of covariate shift between the training and the test series. As shown in Table 3, the normalization improves the results of Informer and FEDformer consistently. The decomposition layer of the Autoformer solves a similar problem and replacing that with our normalization harms the capacity of the model.
4.3 QUALITATIVE RESULTS
We have also shown the qualitative comparisons between the vanilla Informer and FEDformer versus the results of our framework in Figure 5. Most notably, in both cases our approach appears significantly better at forecasting the statistical properties (such as local variance) of the signal. Our ccaleformer-basee models capture trends (and other Auman-relevann) information better than their basclines. Despite these interesting findings, we would like to emphasize that these are randomly selected qualitative examples that may have their own limitations. For more qualitative results, please refer to Section in the Appendix.
4.4 EXTENSIONS AND DISCUSSION
The Scaleformer structural prior has been shown to be beneficial when applied to transformer-based. deterministic time series forecasting. is not however limited to those settings. In this section, we show it can be extended to probabilisite forecasting forecaat and non transformer-butt based encoders, both of which are closely coupled with our primary application. We also aim to highlight potential promising future directions.
8

---

Published as a conference paper at ICLR 2023
Informer
Informe-MSAA
F.Dformer
FEDDforme--SA
Gecundtiurh
Predittion
Electricity Va
M Mantyaa
Weather
Exchange-rat VAtee
Figure 5: Qualitative comparison of base models with framework. Our multi-scale models can better capture the global trend and local variations of the signal than their baseline equivalents. To keep the figure concise and readable, we only show Informer and FEDformer, please refer to Figure in Appendix for more results.
We show that our Scaleformer can improve performance in a probabilistic forecasting setting (please refer Table 9 in Appendix for more details). We adopt the probabilistic output of DeepAR (Salinas et al., 2020), which is the most common probabilistic forecasting treatment. In this setting, instead of a point estimate, we have two prediction heads, predicting the mean l and standard deviation trained with negative log likelihood loss (NLL). NLL and continuous ranked probability score (CRPS) are used as evaluation metrics. All other hyperparameters remain unchanged. Here, again, scaleformers continue to outperform the probabilistic Informer.
While we have mainly focused on improving transformer- -bssed models, they are not the only encoders. Recent models such NHits (Challu et al., 2022) and FiLM (Zhou al., 2022a) attain competitive performance, while assuming fixed length univariate input/output. They are less flexible compared with variable length of multi-variate input/output, but result in strong performance and faster inference than transformers, making them interesting to consider. The Scaleformer prior demonstrates a statistically significant improvement, on average, when adapted by NHits and FiLM to iteratively refine predictions. For more details please refer to Appendix K.
The results mentioned above demonstrate that ScaleFormer can adapt settings distinct from point- wise time-series forecasts with transformers (the primary scope of our paper), such as probabilistic forecasts and non- transfffffferr models. We consider such directions therefore be promising for future work.
CONCLUSION
Noting that introducing structural priors that account for multi-scale information is essential for accurate time-series forecastings, this paper proposes novel multi-scale framework on top of recent state-the-art methods for time-series forecasting using Transformers. Our framework iteratively refines a forecasted time-series at increasingly fine-grained scales, and introduces a normalization scheme that minimizes distribution shifts between scales. These contributions result in vastly improved performance over baseline transformer architectures, across a variety of settings, and qualitatively result in forecasts that better capture the trend and local variations of the target signal. Additionally, our detailed ablation study shows that the different components synergetically work together to deliver this outcome.
For future work, is promising to extend the preliminary work that has been done applying Scale- Former architectures to both probabilistic forecasting and con-transformer models.

---

Published as a conference paper at ICLR 2023
REFERENCES
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv:1803.0//27/ 2018.
Jonathan T Barron. A general and adaptive robust loss function. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 4331-4339, 2019.
George EP Box and Gwilym M Jenkins. Some recent advances in forecasting and control. Journal of the Royal Statistical Society. Series C (Applied Statistics), 17(2):91-109 1968.
Peter J Brockwell and Richard Davis. Time series: theory and methods. Springer Science & Business Media, 2009.
Cristian Challu, Kin G Olivares, Boris Oreshkin, Federico Garza, Max Mergenthaler, and Artur Dubrawski. N-hits: Neural hierarchical interpolation for time series forecasting. arXiv preprint arXiv:2201.12288 2022.
Zhengping Che, Sanjay Purushotham, Guangyu Li, Bo Jiang, and Yan Liu. Hierarchical deep generative models for multi-rate multivariate time series. International Conference on Machine Learning, pp. 784-793. PMLR, 2018.
Ling Chen, Donghui Chen, Zongjiang Shang, Youdong Zhang, Bo Wen, and Chenghu Yang. Multi- scale adaptive graph neural network for multivariate time series forecasting. arXiv preprint arXiv:2201.04828, 2022.
Zipeng Chen, Qianli Ma, and Zhenxi Lin. Time-awart aware multi-scale mns for time series modeling. In IJCAI, 2021.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. Rethinking attention with performers, 2020.
Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hierarchical multiscale recurrent neural networks. arXiv preprint arXiv: arXiv:1609.00 1609.0/000000 0000000/044 704, 2016.
Zhicheng Cui, Wenlin Chen, and Yixin Chen. Multi-scale convolutional neural networks for time 2016.
series classification. arXiv preprint
Qianggang Ding, Sifan Wu, Hao Sun, Jiadong Guo, and Jian Guo. Hierarchical multi-scale gaussian transformer for stock movement prediction. In IJCAL, pp. 4640-4646, 2020.
Dazhao Du, Bing Su, and Zhewei Wei. Preformer: Predictive transformer with multi-scale segment- wise correlations for long-term time series forecasting. arXiv preprint arXiv:2202.1 1356, 2022.
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li. Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 6824-6835, 2021.
J.Doyne Farmer, Edward Ott, and James Yorke. The dimension of chaotic attractors. Phys- ica D: Nonlinear Phenomena, 7(1):153-180, 1983. ISSN 0167-2789, doi: https///oi.org// 10.101600166-27888339991553333 URL https: http///wwiieceireretttttt. wwtpeeeettcccceeeeoooeeeeeeeeeee scinnceeee
Marco AR Ferreira, David M Higdon, Herbert KH Lee, and Mike West. Multi-scale and hidden resolution time series models. Bayesian Analysis, 1(4):947-967, 2006.
Peter J. Huber. Robust Estimation of Location Parameter. The Annals Mathematical Statistics, 35(1):73 101, 1964. doi: 10. 10.12144aomm/11777/ 177703732. URL /tttps::/doi..org10..12222. aoms/1177703732.
Rob Hyndman, Anne Kochler, J Keith Ord, and Ralph D Snyder. Forecasting with exponential smoothing: the state space approach. Springer Science & Business Media, 2008
10

---

Published as a conference paper at ICLR 2023
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448 456. PMLR. 2015.
M. G. Kendall. Studies in the history of probability and statistics. where shall the history of statistics begin? Biometrika, 47(3/4):4474499 1960. ISSN 00063444. URL http: //ww://wwwww...... jstor. org/ htte/staabee2333330
Nikita Kitaev, Lukasz. Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference Learning Representations, 2020, URL net / Eorum? -i
Bjoern Krollner, Bruce J Vanstone, Gavin R Finnie, al. Financial time series forecasting with machine learning techniques: a survey. In ESANN, 2010.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st International ACM SIGIR Conference on Research Development Information Retrieval, pp. 95-104, 2018
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting, Advances in Neural Information Processing Systems, 32:5243-5253 2019.
Pengju Liu, Hongzhi Zhang, Kai Zhang. Liang Lin, and Wangmeng Zuo. Multi-level wavelet-cn for image restoration. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 773-782, 2018.
CH López-Caraballo, I Salfate, Lazzús, P Rojas, M Rivera, and Palma-Chilla. Mackey-glast noisy chaotic time series prediction by swarm- -ararm-optimized neural network. In Journal of Physics: Conference Series, volume 720, pp. 012002. IOP Publishing. 2016.
Michael C Mackey and Leon Glass. Oseillation and chaos in physiological control systems. Science, 197(4300):28-289,, 1977.
Kiran Madhusudhanan, Johannes Burchert, Nghia Duong-Trung. Stefan Born, and Lars Schmidt- Thieme. Yformer: U-net inspired transformer architecture for far horizon time series forecasting. arXiv preprint arXiv:21 2021.
Michael C Mozer.
Induction of multiscale temporal structure. Advances in neural information processing systems, 4, 1991.
Allan H Murphy. What is good forecast? an essay on the nature of goodness in weather forecasting. Weather and forecasting, 8(2):281-22) 1993.
Piotr Nawrot, Szymon Tworkowski. Michat Tyrolski, Lukasz Kaiser, Yuhuai Wu, Christian Szegedy, and Henryk Michalewski. Hierarchical transformers are more efficient language models. arXiv preprint arXiv:21 13710, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, An- dreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjic Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox. and R. Garnett (eds.), Advances in Neural In- formation Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019. URL an imperative-styye pdf..
http::/ neurips cc/ -high-performancee -e
Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. Advances neural information processing systems, 31, 2018.
11

---

Published as a conference paper at ICLR 2023
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3): 1181-1191, 2020.
Jeffery D Scargle. Studies in astronomical time series analysis. i-modeling random processes in the time domain. The Astrophysical Journal Supplement Series, 45:1-71, 1981.
Lifeng Shen, Zhuocong Li, and James Kwok. Timeseries anomaly detection using temporal hierar- chical one-class network. Advances Neural Information Processing Systems, 33:13016-115 3016-13026, 2020.
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log- likelihood function. Journal of statistical planning and inference, 90(2):227-244, 2000.
Sandeep Subramanian, Ronan Collobert, Marc' Aurelio Ranzato, and Y-Lan Boureau. Multi-scale transformer language models. arXiv preprint arXiv:2005.c 00581, 2020.
Aris Syntetos, John E Boylan, and Stephen M Disney. Forecasting for inventory planning: 50-year review. Journal of the Operational Research Society, (0(1):5149-S16) 2009.
Binh Tang and David S Matteson. Probabilistic transformer for time series analysis. Advances in Neural Informaionn Processing Systems, 11-23592--2608. 2021.
Neo Wu, Bradley Green, Xue Ben, and Shawn O' Banion.. Deep transformer models for time series forecasting: The influenza prevalence case. arXiv preprini arXiv:2001.08317, 2020.
Jiehui Xu, Jianmin Wang, Mingsheng Long, al. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34, 2021.
George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A transformer-based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 2114-2124, 2021.
Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi-scale vision longformer: A new vision transformer for high-resolution image encoding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2021.
Yucheng Zhao, Chong Luo, Zheng- Jun Zha, and Wenjun Zeng. Multi-scale group transformer for long sequence modeling in speech separation. In Proceedings of the Twenty-Nie International Conference International Joint Conferences Artificial Intelligence, pp. 3251-3257, 2021.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of AAAI, 2021.
Tian Zhou, Ziqing Ma, Qingsong Wen, Liang Sun, Tao Yao, Rong Jin, el. Film: Frequency improved legendre memory model for long-term time series forecasting. arXiv preprint arXiv: 2et.20282.7777 2022a.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Proc. 39th International Conference on Machine Learning (ICMI. 2022), 2022b.
12

---

Published as a conference paper at ICLR 2023
A IMPLEMENTATION DETAILS
Our implementation is based on the Pytorch (Paszke et al., 2019) implementation? of the Autoformer and Informer (Xu et al., 2021). The hidden dimension of models are fixed to 512 with a batch size of 32, and we train each model for for 10 epochs with early stop enabled. To optimize the models, an Adam optimizer has been used with a learning rate of le-4 for the forecasting model and le-3 for optimizing the adaptive loss. The forecasting module is fixed to 2 encoder layers and decoder layer. The book-back window size is fixed to 96, and the horizon is varied from 96 to 720. We repeat each experiment 5 times to reduce the effect of randomness in the reported values. In all experiments, the temporal scale factor s fixed to 2.
For Informer, we train the model without any changes as the core of our framework. However, Autoformer uses decomposition layer at the input of the decoder and does not pass the trend series to the network, which makes the model unaware of the previous predictions. To resolve this, we pass zeros as the trend and the series without decomposition as the input to the decoder. For Reformer, used the available implementation" of the model from Xu et al. (2021). In addition, we used the pytorch library' of Performer (Choromanski et al., 2020) for our performer baseline using the same parameters as Reformer model, and finally, use the official implementation" of FEDformer (Zhou et al., 2022b) for the FEDformer model. We fixed the number of modes to 64 following the original paper and Wavelet Enhanced Structure for the core modules. To make consistent with our other experiments, we use the moving average with the kernel size 25 as in Xu al. (2021), however, FEDformer (Zhou et al., 2022b) is using moving average with the kernel size of 24. Our random seed is fixed to 2022 in all experiments.
B
MORE MOTIVATIONS
This section aims to provide more motivations for the use of a multi-scale architecture. Let us first consider the following classical example, highlighted in section 2 of Ferreira al. (2006), corresponding to the monthly flows of the Fraser River from January of 1913 to December of 1990. As shown in the their corresponding plot, the annual averages are strongly inter-related, pointing to the fact that seasonality alone will not suffice to model the variations. In the context of the paper, this showcases a failure mode of an ARMA model, but this failing is more general: models that do not explicitly account for inter-scale dependencies will perform poorly on similar datasets.
Different approaches have attempted to introduce multi-scale processing (Ferreira al., 2006; Mozer, 1991) in ways that differ from our own approach. The multi-scale temporal structure for music composition is introduced in (Mozer, 1991). Ferreira ct al. (2006) proposed a time series model with rich autocorrelation structures by coupling processes evolving at different levels of resolution through time. However, their base models are constrained to simple statistical models, e.g. Autoregressive models.
To conclude, we note the following: (1) the approaches mentioned above have applied multi-scale modeling with success, and (2) we the first work explicitly consider multi-scale prior by construction for transformers.
C REDUCING COMPUTATIONAL COST
To obtain an estimation of the total running time of Scaleformer, the running time of each scale as the terms of a geometric progression based on the scale factor s which results the total time of L-s" T-s multiplied by the running time of the baseline method. In this regard, Table shows the running times of different experiments where 8 = 2 with the batch size is 32 GeForce GTX 1080 Ti GPU with 64 cores. While our current code is not optimised, still the scaleformer of each method takes roughly twice of the baselines which is consistent with the mentioned formula. Note that this is the smallest scale which means that our method is bounded to twice of the baseline and considering larger scale
httpps//githuuuccm// http:/ihhhb.ooo///////uoee/e Auutoommmm//tttoomrrrr..
"https///ithhbb... htaps://gihuuh.om//uudrrriii com//ciiianisssprrooooppppcocccc
"htpp://github..c http:://ggguuaocccm//A/aa////nnn
13

---

Published as a conference paper at ICLR 2023
Table 4: Quantitative comparison of training and the test time of different experiments. Using scale factor 2 the Scaleformer version of the baselines (showed by -MSA) should takes roughly twice of the original baseline time which is almost the same in the quantitative comparisons.
Method
Training time (6)
Test time (s)
Output Length
96
192
336
720
96
192
336
720
Informer
11.38
13.83
17.38
25.31
0.79
1.12
1.21
1.23
Informer-MSA
28.73
31.88
36.37
51.15
2.42
2.63
2.87
2.96
Autoformer
17.34
23.34
31.35
52.34
1.94
2.29
2.88
3.38
Autoforer-MMA
40.57
51.17
63.53
111.84
5.22
6.06
7.04
8.10
factor can reduce the time overhead. Table 5 shows the impact of replacing the Scaleformer operation the real scale by an interpolation of values of the previous scale, i.e., at scale s" we do not apply the transformer but rather compute the results by linear interpolation of X thereby reducing the compute cost. This lower cost alternative results better performance than the baselines, yet worse results than the full Scaleformer. This shows that adding scale is indeed more efficient, and that there is the possibility of a trade-off between further improved performance (full Scaleformer) or improved computational efficiency (interpolated Scaleformer).
Table 5: Comparison of the MSE and MAE results for our proposed multi-scale framework version of Informer and Autoformer by removing the last step and using an interpolation instead (-MSA,) with the corresponding original models as the baseline. Results are given in the multi-variate setting, for different lenghts of the horizon window. The look-back window size is fixed to 96 for all experiments. The best results are shown in Bold. Our method outperforms vanilla versions of both Informer and Autoformer over almost all datasets and settings.
Dataset
Autoformer
Autoformer-MSS
Informer
Informer-MS..
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Exchange
96
0.154
0.28510.00
0.132.00.2
0.265
0.966.10.10
0.792.10.04
0.248 LO.07
0.366
192
0.356:0009
0.428
0,418:10.22
0.466=0.12
1.088
0,842.0.01
0.727
0.637
336
0,441:0002
0.49510001
0.736110:25
0.629-0010
1.598
1.016:1000
0.643::000
0.620.1002
720
1.1182004
0.819=0002
0.773=0026
0.709=0.13
2.679+#0222
1.340+0.06
1.0036=0.08
0.803+0003
Weather
0.267=0003
0.334-1..02
0.168 =:001
0.239
0.388
0.435
0.211
0.278
192
0.323
0.376
0.22610001
0.296_0001
0.433.10.05
0.453-10.03
0,288 10.03
0.336
336
0.364=0022
0.
0,298:2002
0.351 =0002
0.6110+0:04
0.551
0.459:0002
0.445
0.425
00
0.412:1004
0.4
0.978:10:05
0.723:1002
0.593
0.528:1004
Traffic Electricity
96
0.197=001
0.312+0001
0..89=0000
0.305=0000
0,344++000
0.421
0.194=0.00
0.308
192
0.219
0.329
0.207::0.00
0.323=0000
0.344:1.001
0.426:10.01
0.212.:0.00
0.327 1.000
336
0.263.0004
0.35910003
0,225 10001
0.339_0000
0.358100:01
0. 0.44011.001
0.246
0.357
720
0.290=0005
0.
0.250=001
0.361=001
0,386++0:00
0.452+00.0
0.283=0001
0.386#001
0.628=ooz2
0.393
0.585
0.365=0001
0.748:40.01
0.426#0.01
0.595
0.362
192
0.63411001
0.401
0.60610001
0,375
0.772.10.02
0.436.10.01
0.629 1.0.00
0.38
336
0.61910001
0.385
0.601
0.400
0.868 1..06
0.493310.00
0.692.1000
0.410
720
0.656:2001
0.403::0001
0.660::0.00
0.418:0001
1.07410:02
0.606.10.01
0.803:0001
0.461 1:000
D JUSTIFICATION FOR THE ADAPTIVE LOSS
Mathematically, the justification for the adaptive loss is as follows. Considering the E term in equation 10, the function is asymptotically close (but not equivalent due to the denominator) ga. As result, for outliers (for which & will be large), the loss term will function as L penalty on Ç. which will penalize outliers more for large a. The converse of this is that we would expect a model trained with such a loss to learn lower values of a for settings with fewer outliers.
E INTERPLAY BETWEEN ADAPTIVE LOSS AND MULTI-SCALE ARCHITECTURE
The main reason for combining the adaptive loss and multi-scale architecture under a unified frame- work is: they are synergetic. How do we explain this synergy? Compared to other transformer
14

---

Published as a conference paper at ICLR 2023
Table 6: Comparison of the MSE and MAE results for proposed multi-scale framework version of Informer and Autoformer (-MSA) with the iterative refinemenet bascline of keeping the original scale in each iteration (-IA)
Dataset
Autoformer- MSA
Autoformer-IA
Informer-MSS MSA
Informer-Ia IA
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.126:1001
0.259
0.410
0.485
0.168-0005
0.298-0008
0.649-00.8
0.632-1004
Exchange
192
0.253+0.03
0.373
0.809
0.698+om.
0.427
0.484
0.938-008
0006
0.163+6m
0.226+0.01
0.233
0.293+0m00
0.210=0002
0.222-0001
0.286=2002
Weather
192
0.001
0.290
0.401 :0.02
0.429:000
0.289
0.333=001
0.357=000
0.393:0000
Electricity
0.18810.00
0.30310001
0.248.00.0
0.36010.01
0.203
0.315
0,237_0000
0,344.10.00
192
0.197.10.00
0.310
0.265
0.373
0.21910000
0.331
0.271
0.374.1001
Traffic
96
0.567:10.00
0.350
0.681 10.03
0.43220001
0.547=001
0.369_000
0.641=0011
0.367:0001
192
0.589+0m1
0.360=0001
0.699
0.446=0.11
0.655=001
0.399=001
0.695-000
architectures (notably the baselines used), ScaleFormer has more iterative steps: the sequential multi-scale operations. Iterative computation tends accumulate more errors, which will behave like outliers for the purpose of this loss. As a result, the process that leads to the need for the two components be expressed as: (1) The multi-scale architecture is beneficial for performance as a useful structural prior for time series data. (2) The multi-scale architecture however relies on sequential computation that increases the likelihood of explosive error accumulation. (3) The adaptive loss serves to mitigate this issue, leading to more stable learning and better performance.
F SINGLE-SCALE MODEL WITH MEAN-NORMALIZATTON
A single-scale model with mean normalization performs better than the multi-scale version without normalization. The reason for this is that normalization as our proposed scheme targeting at all forms of internal distribution shift, not only those induced by the multi-scale architecture. In our submission, we make the case for the multi-scale prior as a natural prior to add to transformers. From empirical observations, we found that such prior requires adapting normalization. When investigating means of normalizing, we observed additional benefits to non-multiscale architectures well. This means that they also suffer from other forms of distribution shift: attribute it in the paper to c.g. shifts between lookback and forecast distributions.
G MORE ABLATION STUDIES AND PARAMETER ANALYSIS
G.1 ITERATIVE REFINEMENT USING THE SAME SCALE
To further confirm the effect of the multi-scale framework, we also compare proposed framework with another baseline by keeping the original scale in each iteration. As Table 6 shows, using multi- scale framework outperforms keeping the original scale while having significantly lower memory and time complexity overhead. Following the main experiments, we use the scale factor of 2 which results S {16, 1} for Scaleformer (showed by -MSA) and a scale factor of I with similarly 5 iterations for iterative refinement (showed by -IA).
15

---

Published as a conference paper at ICLR 2023
G.2 MORE RESULTS ON DIFFERENT COMPONENTS
Table 7 extends the results of Figure 4 of the paper to all four datasets for both Autoformer and Informer, in the multivariate setting and for the two backbones, Autoformer and Informer.
Table 7: Comparison of training with either only an Adaptive loss "-A", only the multi-seale framework with MSE loss "-MS", or the whole Multi-scale framework and Adaptive loss "-MSA". The results confirm that the combination of all our proposed contributions is essential, and results in significantly improved performance in both Informer and Autoformer. Experiments are done in the multi-variate setting.
Dataset
Informer
Informer-A
Informer-MB
Informer-Ao MSA
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Exchange
96
0.966=0.10
0.792=004
0.962=00m
0.796=004
0.201++0.05
0.325
0.168
0.298=0003
1.088=004
0.842#0001
1.106#003
0.853=000
0,446+0.14
0.502+4.07
0,427
0.484
336
1.598
1.016
1. 1.60210007
10199.001
0,520.10.5
0.540.10.2
0,500
0.538100.2
720
2.679
1.340
2.719
1.35
0.997.10.08
0.783
1.017.0005
0.790
Weather
96
0.388
0,435
0.342=a.s
0.382=002
0.249
0.324
0.210
0.279
192
0.433=000
0.453=000
0.385=0.2
0.40==001
0.315++0.02
0.380
0.289+0.01
0.333+0001
336
0.610=0004
0.05
0.656:10.10
0.550=004
0.473:400004
0.478:+0.02
0.418
0.427 +.003
720
0.978.1005
0.72310002
0.9531104
0.680
0.664:10.04
0.585
0.59510004
0.532.10.02
Electricity
96
0.344=000
0:
0:02
0.402=0000
0.2111+0.00
0.32640000
0.203=0001
0.315
0.344=001
0.426=0000
0.341
0.422=000
0.233++0:01
0.348+00.0
0.219
0.331
336
0.358
0.440
0.344100.00
0.422__0.00
0.279.10.0
0.388110.1
0.253 10.01
0.36010001
720
0.386::0000
0.452
0.363
0.432:0000
0,315
0.411
0.293
0.390
0.74820001
0.4261:001
0.744:1002
0.413=0001
0.602:10:0
0.37510.01
0.597:1001
0.369
Trafiic
192
0.772=0.22
0.436=20.1
0.765=0.01
0.416=001
0.669+0..1
0.411++0.00
0.655=0.01
0.399
336
0.868=0004
0.493
0,8524:0.5
0.461=0003
0.815:+0.00
0.501
0.761
0.405
720
1.07410.02
0.606100001
1.03011.00
0.539_0.08
0.94910.01
0.55610.02
0,92410.02
0.52110001
Dataset
Autoformer
Autofomer- A
Autoformer MS
Autoformer MSA
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Exchange
96
0.154=001
0.285=0.00
0.152#001
0.282=a00
0,145++0.z
0,276*0.01
0.126=0.01
0.259
192
0.356:10099
0.428
0.342:10.10
0.421=0006
0.247.:0.02
0,366:10.01
0.253
0.373
336
0,441::000
0.49510001
0.56611022
0.554_0.10
0.456.10.08
0.51410005
0.51910.16
0.538
720
1.1811004
0.81910022
1,1201024
0.811
0.729.10.16
0.67911008
0,928
0.701
Wenthr
0.267=0003
0.334-10.02
0.229
0.288
0.174:10:01
0.254
0.163
0.226
192
0.323
0.376=0000
0.293
0.340=o02
0,250++0.02
0.333+0002
0.221
0.290
336
0.364.10.2
0.39710001
0.357
0.387=0001
0.314:10.02
0.380.1.02
0.282.0.00
0,340
0.425:2001
0:
0.419:0011
0.422:0001
0.414:10:03
0.457.10.02
0.369::0004
0.396:1003
Electricity
96
0.197=001
0.
0.201 =0:01
0.312=001
0,196++0:00
0.312
0.188=0.00
0.303+0m1
192
0.219
0.32
0:021
0.308
0.208
0.323-10.00
0.197
0.310
336
0.263.1004
0.35910003
0.232.1001
0.339_0001
0.220:10:00
0.336:10.00
0.22411002
0.03310001
720
0.290...05
0.380
0.249.10.1
0,351
0,252.10:00
0.364-10.00
0.24910.01
0.358
Triffic
0.628-1022
0.393:002
0.610:002
0.381=001
0.580:10:01
0.358:0.00
0.567:00.0
0.350:0000
192
0.634=0011
0.401
0.633
0.396=0002
0.601
0.369:0.00
0.589:0001
0.360
336
0.61910011
0.38510001
0.618
0.381=0000
0.639:10.02
0.397:10.01
0.61910.01
0.383
0.65620011
01
0.654:1:002
0.397=0001
0.68010.02
0.427-10.01
0.642:0001
0.397.1001
16

---

Published as a conference paper at ICLR 2023
G.3 RESULTS ON DIFFERENT SCALES
Table 8 showcases the results of our model using different values of the scale parameter 8. It shows that scale of 2 results in better performance.
Table 8: Comparison of the baselines with our method using different scales. Reducing the scale factor from 16 to increases the number of steps but achieves lower error on average.
Dataset
Informer
Informer MSA(s=16)
Informer MS
Informer MS MSA(s=2)
Metrie
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Exchange
96
0.96610.10
0.792::004
0,376
0.480=004
0.246-0006
0.374:1004
0.168-0005
0.298
1.088=044
0.842=001
0.878=0008
0.721=003
0.661=0220
0.617=0009
0.427=#0.12
0,484=0006
336
1.598
1.016
0.899
0.733=00s
0.697=0.10
0
0.500
0,535
720
2.679.0.22
1.340
1.633.10.14
1,03 1.005
1,457_0027
0.958
1.017 10.00
0.790.0..02
Weather
0.
0.435=003
0.208
0.275=0002
0.189=001
0.259
0.210::000
0.279=0002
192
0.433=0005
0.453=0.03
0.302=00u2
0.354=002
0.287 0002
0.333 =:000
0.289:
0,333=H001
336
0.610=004
0.551
0
0.456=0004
0.420=000
=.433=0002
0.418=0004
0.427=0008
0.978=0005
0.723
0.639
0.544=0004
0.627=007
0.543.10004
0.594
0.532:0082
Electricity
96
0.344:10:00
0:001
0.215:10.00
0.009
0,203-0000
0.315 1.000
0.315
0.203-10.01
0.315
0.344=001
0.426=0001
0.257=0011
0.370=0001
0.235=001
0.347=0001
336
0.358
0.4400=0.00
0.300=00:44
0,400=0003
0.264=000
0.373
0.253-10.0
0.360
720
0.386
0.002
0.334
0.41810002
0.306_.0.01
0.401.1000
0.293.10.01
0.390.0.01
0.748=001
0.426=0011
0.648::002
0.386-001
0.616=000
0.374=0001
0.597:1000
0.369=0000
Trftic
192
0.772=0.22
0.436=0000
0.679=000
0.391
0.686=010
0.404=
0.655=1101
0.399=u001
336
0.868=0004
0.493=0.0
0.811=a022
0.455=0ot
0.782-003
0.451
=
0.455=000
1.0740002
0.60610.00
1.020=0007
0.542=003
0.965
0.521
0.924
0.521:0001
Autoformer
Autoformer MSA(16)
Autoformer MSA(4)
Autoformer MSA(2)
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Exchange
0.154
0,285=000
0.182
0.316=003
0.170=003
0.304
0.126:0001
0.259=000
192
0,356=00.9
0.428
0.514
0.537=a10
0.359=0.14
0.443
H.253
0.373=:0.z
336
0.441 L0002
0.495
0.527
0.570.004
0.606_0018
0.58510.10
0.51910.16
0.538.00.9
720
1.118
0.81910002
1.019
0.819
0.973-022
0.809
0.928
0.751
Weather
0.334=0002
0.169
0.239=0ot
0.164=000
0.234
0.163-2000
0 0.226=u001
0.323-001
0.376=0000
0.240=000
0.310=0003
0.229=001
0.299-0002
0.2 0.290=0000
336
0.364=00022
0.397
0.304
0.354=008
0.303=001
0.350
0.282
0.340
720
0.425
0.434110.0
0.375.10001
0.403
0,382.00022
0.414.10.01
0.369100.4
0.396
Electricity
0.197
0.312=001
0.190=000
0.306=0000
0,188=000
0.3033=000
0.188
0. 0.303-0001
192
0.219=001
0.329
0.206=-0ur
0.320=001
0.207=000
0.320=#u.00
0.197 =900000000
0.310=0000
336
0.263=0004
0.359
0.236
0.344=00t
0.237=000
0.344
0.224=00.0
0.333=0001
720
0.290
0.380
0.260
0.368
0.261=001
0.369
0.249
0.358
96
0. 0.628:0000
0.
0.
0.380=001
0,594_0002
0.367:10.0 1:000
0.567-10:00
0.350
Traffic
0.634-001
0.401=0.01
0.6262001
0.393=001
0.600=001
0.369-0000
0.360==0.01
336
0.619=0001
0.385
0.635::0001
0,400=0000
0.625=000
0.386
0.619:10.01
0,383
720
0.656.10.11
0,403
0.697.10.03
0.43910001
0.678_00022
0.422.10.02
0.642.10.01
0.097
17

---

Published as a conference paper at ICLR 2023
G.4 EFFECT OF ON THE LOSS FUNCTION
Figure 6 shows the impact of the shape of the loss function on the left, and example ground truth time-series corresponding to the horizon window on the right. As noted in the main paper, lower values of tend to result in better robustness to outliers. This is indeed confirmed empirically in the case of the weather dataset, which corresponds to the lowest learnt a value, and has the outliers with the largest relative scales. For simplicity, have excluded the analysis as it does not impact the robustness with regards to the outliers, as shown in Barron (2019).
25
(0=2.1))
Exchange rate
(to- 196)
Flectricity
Weether (12003367)
00
Weather
Electricity
10
Echange
(a)
(b)
Figure 6: (a) The loss value based as a function of the & (absolute difference between prediction and target) and the learned a for each dataset. Different colors correspond to different datasets (each with specific value of at). (b) Samples taken from the ground-truth horizon window of each dataset. The Weather dataset sample has the outliers with the largest scales compared to the input series. As consequence, the learnt value of a is the lowest. Different colors correspond to différent variables in the variate time-series we are considering.
Scale Peeetbee
Scale
Scale
Scale (Original series)
PED MSA
Exchtang
Autofomns- MSA Flearisit
Wo Mo
wht Mn Mi
Infomer-AAte
17
Figure 7: Qualitative results of each scale. The model can correct its previous mistakes on higher scales, showcasing increased robustness to forecasting artifacts that occur individually at each scale.
18

---

Published as a conference paper at ICLR 2023
H DISCUSSION ON NORMALIZATION FOR AUTOFORMER
There is limited performance gain for Autoformer from the normalization alone. The reason that Autoformer benefits from inner series decomposition module which acts the normalization by nature. Indeed one benefit of our proposed framework a simple solution to bring the benefits of these specified designs to other baselines which significantly reduces the gap between for example Informer and Autoformer. AutoFormer already benefits from an internal component that reduces internal distribution shift: the series decomposition module.
I ADDITIONAL QUALITATIVE RESULTS
Figure 7, Figure 8, and Figure provide additional qualitative results respectively showing the intermediate results of each scale and the comparison between the baselines and our approach with the horizon length of 720 and 192. As we can see, Scaleformer is able to better learn local and global seress variations.
Informer
InformerMSAA
Autoformer
Autoformer- MSA
GroundTruth
Electricity
Moka AMMAPA
MAN MM0 MW
Weather
Exchunge-rate
Figure 8: Additional qualitative comparison of the baselines against our framework.
19

---

Published as a conference paper at ICLR 2023
Informer
Informer-MSA
Autoformer
Autoformer-MSA.
Mr
GroundTiuth Prediction
My
MWW
Electritty
MONtA
Wealbes par
Tratio
Excehange- raiinnng--rie Noher
M
Time
Time
Timo
Time
Figure 9: Additional qualitative comparison of the baselines against our framework.
J PROBABILISTIC FORECASTING
Table 9 shows the comparison of probabilistic methods for Informer by following the probabilistic output DeepAR (Salinas et al., 2020), which the most common probabilistic forecasting treatment. On implementation details, instead of point estimate, we have two prediction heads (one predict and one predict or) with negative log likelihood loss. The other hyper parameters is the same as before. We use NLL loss and continuous ranked probability score (CRPS) which commonly used in probabilistic forecasting as evaluation metrics.
Table 9: Scaleformer improving probabilistic metrics in Probabilistic forecasting for Informer. By adapting Informer to make probabilistic predictions, we are able to show the Scaleformer prior again brings benefits. While such an analysis excludes dedicated probabilistic approaches for conciseness, it nevertheless shows the generality of our proposed approach.
Dataset
96
192
336
720
Metrie
CRPS
NLL
CRPS
NLL
CRPS
NLL
CRPS
NLL
Exchange
Informer
0,548+a022
2,360=0.20
0.702=005
4.350+145
0,826=0.22
4.302=00.0
1.268+0006
13. 140
Informer-MSA
0.202:1001
0.452:009
0.284
0.81810.11
0.414:0006
1.724:0.43
0.570:003
2.862.1021
Weather
Informer
0.3764000
1.180#021
0.502+0.03
1.752+n.3
0.564
1.928=00.2
0.684:2009
2.210#044
Informe-MSA
0.250:002
0,392:0.14
0.294
0.610:n04
0.308-20u02
0.728
00.18
0.438:0004
1.270
Electricity
Informer
0.3301001
1.1061005
0.338
1.254.004
0.348
1.24441.00
0.528
1.856.1000
Infomer-MSA
0.238
0,578=ou1
0.290
0.776an81
0.324
0.904=0.10
0.358
1.022
Trafic
Informer
0.3721004
1.376.005
0.3401..01
1.404.1004
0.372
1.51610.00
0.568
1,6581001
Iaformer-MSA
0.288
1009400u3
0.312+0.01
1.102+004
0.368:=0.02
1.194:00.05
0.4422:002
1.378+0005
20

---

Published as a conference paper at ICLR 2023
K EXTENSION TO OTHER METHODS
Table 10 shows the comparison results of NHiTs (Challu et al., 2022) and FiLM (Zhou et al., 2022a) as two baselines. For each method, we copy original model to have model for different scales and we concatenate the input with the output of previous scale for the new scale. The training hyperparameters such as optimizer and learning rate the same as the previous baselines.
Table 10: The effect of applying our proposed framework to NHits and FiLM as two non non-transformer
based models. Best results are shown Bold.
Dataset
NHITS
NHETS MSA
FiLM
FiLM MSA
Metrie
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Exchange
96
0.091
0.21810001
0,087 100000
0.206_.0.00
0.083.10.00
0,201 .10,00
0.081
0.197.10.00
0:200=002
0.332:0001
0.306=0000
0.179::0:00
0.301 10000
0.156:00.00
336
0.347=000
0.442=0002
0.38
0,445=000
0.341+00.00
0.421 +0000
0.253
0
720
0.721
0.662.0008
1.124:10007
0,808
0.896.10001
0.714:10.00
0.728 10.01
0.689
Weather
0.16920000
0.228:0000
0.:67::0000
0.211
0.194:0000
0. 0.235
0.1
0.232
192
0.210=0.00
0.268=0.00
0.208=0000
0.253=000
0,238++0.0
0,270+0o.0
0.235
0.269
336
00261=000
0.
0,261::0000
0.294=0000
0.288:+0.00
0.305:+0.00
0.275
0.
0.333=001
0.372#0000
0.331=0000
0.348
0.359:0.00
0.350:#0.0
0.337
0.356
L CROSS-SCALE NORMALIZATION
The choice of normalization approach is essential our method, due the aforementioned distribu- tion shifts. In this section, we showcase multiple experiments that were done to study the impact of different ways of normalizing inputs. In Table 11, we compare using two forms of normalization with Informer-Me MSA as the baseline: mean-only versus normalization using both mean and standard devia- tion. We consider that adding normalization to our method is trade-off between two issues faced during training. One one hand, (internal) distribution shifts hinder performance, demonstrated in the experiments (See Table 2). On the other hand, normalizing the internal representations results in form of information loss (during the processing of the input): we lose information about mean and standard deviation of the input data. We believe that the reason mean mean-normalization outperforms mean and standard deviation normalization is because it results in better trade-off. losing less information while still managing to address the internal distribution shift.
Fig. 10 shows the different dataset distributions. As for datasets such as electricity and traffic with almost unimodal distributions. For datasets such exchange-rate and some extent weather, which have multi-modal distributions, find that mean and standandard deviation works better. Our hypothesis that mean neean-oormalizatinn is too simple an approach for more complex input distributions. We note that, in all cases, either form of normalization results in strong improvements compared to the absence of normalization.
see, normalization works better
electiiity
raffcc
exchange.rot rate
weather
Dataset distribution with histogram and kernel density estimation. Note that different datasets have either mostly uni-modal or multi-modal distributions.
Figure 10:
M SYNTHETIC DATASET
To further evaluate our proposed framework and adaptive loss function, we generate a new dataset based the Mackey-Glas equations (Mackey & Glass, 1977). Concretely, we use the following
21

---

Published as a conference paper at ICLR 2023
Table 11: Comparison of using standard normalization and zero-mean shifting in our scale normalization. Zero-mean shifting gets better results in almost all of the Traffic, Electricity, and Illness dataset. While standard normalization gets better results in Weather and Exchange datasets. We use zero-mean shifting in our experiments as it removes less information from the input time-series.
Prediction Length
96 (24)
192 (32)
336 (48)
720 (64)
Metric
MSE
MAE
MSE
MAF
MSF
MAF
MSF
MAE
Electricity
Mcan
2.199
0.312:1000
0.215
0.327:10.00
0.250
0,358
0.297 200r
0.3911001
Mean+STDEV
2:248:001
0.345g00:
0.271
0.363-19.0
0,265
0.3592000
0.365
0.4252001
Exchinge
0.10120002
0.326:802
0.374=0.0
0.453=0.00
0.555
0.567=0001
1001
0.782
Mca
0.165
0.300.0m
0.280
0.387:19.0
1.487
0.518
1,406
0.931
Mean
20095
1.242
3.798=0007
1.266:0.01
3.759:0009
1.238=00.0
3.006:0000
1.222+000
II.I
Mean+STDEV
4.582
14435
2.011
1.470.9.00
3.611
1.243
4.156
1.3761100
Traffic
0.611:2001
0.377.1001
0.642:001
0.389
0.760::001
0.453-0001
0,940::000
0.514:000
Mean+STDEV
0.712
0.421
0.777=001
0.474
0.852#0.1
0.501 =0.00
0 #0,02
0.530+0001
Weather
MTDEV
Mean
0.197:0011
0.208:00001
0.274::0m
0.242:0000
0.224:001
0.303=0022
0.276=0.01
0.347:0.00
0.443
0.301
0.43520002
0.331=001
0.392::002
0.625
0.544g001
0.384:001
equation:
dæ
0.2 x x - T) + x0(1 T1 (0.1 x x()),,
(11)
dt
where follow Caapzz--aaabllooeeooo et al. (2016); Farmer et al. (1983) and we also consider z(0) 1.2. We create three series of length 10k. First we create a series by only using the above equation and considering 18 where the series has chaotic behaviour when additional series with r = 12 and r = 9, and also with added seasonal and trend components. Our synthetic dataset is combination of these 3 series (See Figure 11 for example). We compare our method using the new dataset on both Informer and Autoformer in Table 12, showing our proposed framework significantly improves the performance over the baselines.
17. We also provide two
Table 12: Comparison of our proposed framework and the baselines using synthetic data with 3 series generated based on Mackey Glass formulation by r = {18, 12, 9}.
Dalasel
96
192
336
720
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Autoformerbase
0.435
0.46410.0
0.399
0.4561100
0,361 2003
0,4491002
0.301
0.452
AntofomerMS
0.0752001
0.19220.00
0.113
0,237
0.164=003
0,298:003
0.46822022
0.501
Informerhase
0.265=0006
0.367=0.03
0.287=2001
0.396=000
0.243=002
0.372#u01
0.246=0001
0.375=am
InformerMS
0.078:00.00
0.192:10.00
0.109
0.233
0.164=000
0.309
0.227 20:00
0.356
7M
Infomser
Relative Improvenent (%)
ISS
20
1.5
1.0
30
0.5
20
0.0
-05
0
300
200
302
400
500
602
700
65
25
Time
Outliers Percentage (%)
Figure 11: Left: an example input of the synthetic dataset. Right: The relative improvement of adaptive loss increases with increasing the percentage of outliers in the dataset.
22

---

Published as a conference paper at ICLR 2023
In addition, to further analyze the effect of the adaptive loss noisy datasets with outliers, randomly replace a percentage of the training data with outliers by defining an outlier as a sample with distance of larger than 50 times of standard deviation of the series from the median. Figure 11 shows the improvement of using adaptive loss on both Informer baseline and our multi-scale version of Informer. Adaptive loss increases the performance up roughly 70% when there is noisy input with extreme outliers (5% of data) while it gets comparable results for a clean dataset, demonstrating it strong candidate to replace MSE loss in the time series forecasting tasks.
N STATISTICAL TESTS
To further validate the strengths of our empirical results, we have conducted statistical tests. Each test is a Student's 's t-test Kendall (1960) to determine whether the two sets of results can be distinguished. The first test, Table 13, shows that for most settings, the adaptive loss provides gains that are statistically significant (p < 0.05). The second test, in Table 14 shows that for most settings, the multi-scale architectures provides gains that are statistically significant (p < 0.5).
Table 13: Student's t-test p-values for the test corresponding to whether the Adaptive loss provides an improvement. The base model is FedFormer. While we cannot conclude the adaptive loss provides an improvement for exchange rate and certain window sizes for traffic, for all other datasets the improvement is notable.
Window size
192
336
720
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Exchange rate
0.7157
0,77286
0.46741
0,57392
0.61317
0.68405
0.97102
0.56755
Electricity
0.05844
0.01267
0.00124
0.00187
0.01294
0.00576
0.00374
0.00597
Weather
0.25509
0,11246
0.00014
8c-05
0.0648
0.00826
0,05881
0.02213
Traffic
0,21029
0,02572
0.78931
0.19641
0.40819
0.248
0,3491
0.17579
Table 14: Student's ' t-test p-values for the test corresponding to whether the multi- prior provides an improvement. The base model is FedFormer. For certain window sizes of weather and exchange rate the improvement is not statistically provable. For most other settings it is almost always significant.
Window size
192
336
720
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Exchange rate
0.01 109
0.01657
0.02263
0.01674
0.14988
0.26543
0.11071
0.13141
Electricity
0.00029
0.00088
05
0.00017
0.28221
0.3316
0.00593
0.07219
Weather
0.02525
0,01987
0.2201
0.05433
0,14593
0.12861
0.00362
0.02396
Traffic
0.01024
0.00241
0.017144
0.00749
0.00058
0.00033
0.00019
0.00012
23


Reviews:
[{"review_id": "h0TEf8zCPKL", "reviewer": "Reviewer_UakB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a Transformer-based framework that iteratively forecast time series at different scales with shared weights. In particular, it proposes normalizing the downsampled time series to avoid distribution shift and using adaptive loss to deal with outliers. Experiments show the multi-scale framework can significantly improve the performance of various transformer-based forecasting models.", "summary_of_the_review": "Overall I believe this paper make incremental contribution to forecasting community with a multi-scale framework. The empirical results are inspiring, however, for future research.", "strength_and_weaknesses": "Strengths:\n+ The paper is well-organized and the essential ideas are easy to follow.\n+ Extensive experiments are conducted with impressive improvement in main results.\n\nWeakness\n- The introduction of multi-scale structure is not well motivated. It would be nicer if a good motivating example of using multi-scale forecasting is provided other than the ablation study.\n- The reason of using average only in cross-scale normalization is not well justified. Why would it be able to address distribution shift without standardization as well?\n- The adaptive loss is introduced for outliers but there is no such demonstration. And the improvement of adaptive loss seems marginal in the ablation study (figure 4)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "N2O33ZTqgc", "reviewer": "Reviewer_VvHJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a transformer-based framework for time-series forecasting for multi-scale frequency components. The solution is inspired by two recent state-of-the-art transformer architectures used as backbones by iteratively refining a forecasted time series at increasingly fine-grained scales and introducing a normalization scheme that minimizes distribution shifts between scales.", "summary_of_the_review": "Overall, the authors propose a novel framework Transformer-based model for time series forecasting for multiple scales. However, the paper requires much discussion on why one model is superior to the other and interprets the layer-wise representations.", "strength_and_weaknesses": "Strengths:\n* The novelty of the paper is interesting,\n* The idea of time-series forecasting for multiple scales, by minimizing the distribution shifts between scales and windows using cross-scale normalization, is an interesting contribution to the different domains: Neuroscience, Stock Market, Weather Forecasting, etc.\n* The paper is well written and is very thorough with the problem formulations (Both Architecture and its components are well explained with mathematical proofs).\n\nWeaknesses:\n* The Informer-MSA results are much closer to the actual time series compared to other methods in Figure 5; however, Fedformer and Autoformer display lower MSE and MAE in Tables 2 and 3. It is not clear to me why these models are performing poorly in the qualitative analysis compared to Informer-MSA.\n* Figure 4 requires standard error bars for better comparison across the baseline models. Also, the authors could perform a statistical significance test to compare the results. \n* While the authors focused more on comparing the model performances, whether authors have tried if we have frozen the layer of the Transformer and only train at the final layer? How do the frozen model effects approximate dynamic time-series forecasting?\n* It will be interesting if authors can present the layer-wise representations across the models and how these models learn the temporal aspect while forecasting the time series.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BJjlQ72p-og", "reviewer": "Reviewer_N6mC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a general multi-scale framework for time series forecasting.", "summary_of_the_review": "In this paper, the authors proposed a  multi-scale framework that can apply directly to the existing time series forecasting framework. Detailed comments are listed below:\n1. The technical contribution of this paper is a little bit incremental. The idea of leveraging multiple time resolutions of time series is not something new. The major contribution comes from cross-scale normalization.\n2. How is the model trained with learnable parameters in the loss function? Also, how are alpha and c learned during the training?\n3. Based on Figure 4, it seems that the adaptive loss doesn't have significant benefits until the multi-scale framework is used. Can the authors provide a move analysis and discussion?\n4. It's observed that, in many cases (Tables 2 and 3), a single-scale model with mean normalization performs significantly better than the multi-scale version without normalization. Does that mean a simple mean normalization is more important than the multi-scale information in the time series? Can the authors provide more insight analysis?", "strength_and_weaknesses": "Strengths\n1. SOTA methods are compared in the paper. \n2. Comprehensive experiments are conducted.\n\nWeaknesses:\n1. Technical contribution is incremental.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0ICOgu-hgj", "reviewer": "Reviewer_V4CM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduced a multi-scale forecasting training module to improve the forecasting performance of existing transformer models. The basic idea is laying on training scaling-shift aware fine-grind forecasting model. To achieve this goal, the authors introduce an across scale normalization module and a multi-resolution forecasting training structure. The result shows that the proposed module could improve the forecasting performance with various transformer deep learning models without adding additional burdens.\n", "summary_of_the_review": "Overall, this is an interesting paper, and the proposed module can be widely used in existing time series forecasting framework. Therefore, I believe this is an accept paper.", "strength_and_weaknesses": "Strength:\n1. the paper is well written and the technique is simple yet intuitive\n2. the contribution can contribute any transformer based forecasting model\n3. Extensive experiments are conducted to justify the claim\n\nWeakness\n1. Introducing Adaptive loss seems unrelated to the major contribution of this paper\n2. The multiscale forecasting is so-what similar to U-Net structure", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster