Scaling Law for Time Series Forecasting
Authors: Jingzhe Shi
Qinwei Ma
Huan Ma
Lei Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Time series forecasting
Scaling law
Theory
Primary area: deep_learning_architectures
TL;DR: Our research proposes a novel theory for scaling laws in time series forecasting, addressing anomalies observed in previous studies and emphasizing dataset size, model complexity, and forecast horizon in deep learning methodologies.

Abstract: Scaling law that rewards large datasets, complex models and enhanced data granularity has been observed in various fields of deep learning. Yet, studies on time series forecasting have cast doubt on scaling behaviors of deep learning methods for time series forecasting: while more training data improves performance, more capable models do not always outperform less capable models, and longer input horizon may hurt performance for some models. We propose a theory for scaling law for time series forecasting that can explain these seemingly abnormal behaviors. We take into account the impact of dataset size and model complexity, as well as time series data granularity, particularly focusing on the look-back horizon, an aspect that has been unexplored in previous theories. Furthermore, we empirically evaluate various models using a diverse set of time series forecasting datasets, which (1) verifies the validity of scaling law on dataset size and model complexity within the realm of time series forecasting, and (2) validates our theoretical framework, particularly regarding the influence of look back horizon. We hope our findings may inspire new models targeting time series forecasting datasets of limited size, as well as large foundational datasets and models for time series forecasting in future works.

Scaling Law for Time Series Forecasting
Jingzhe Shi', Qinwei Ma"; Huan Ma², Lei Li Institute for Interdisciplinary Information Sciences, Tsinghua University
Zhill College, Tsinghua University 3 University of Copenhagen 4 University of Washington
1.2 (shi-jz21,mqul) mahh1)@maalll ts tsinghua. edu. cn, 3,4 1ilei@di. wwteddiikuc.e dk
Abstract
Scaling law that rewards large datasets, complex models and enhanced data gran- ularity has been observed in various fields of deep learning. Yet, studies time series forecasting have cast doubt on scaling behaviors of deep learning methods for time series forecasting: while more training data improves performance, more capable models do not always outperform less capable models, and longer input horizons may hurt performance for some models. We propose theory for scaling law for time series forecasting that can explain these seemingly abnormal behaviors. We take into account the impact of dataset size and model complexity, as well as time series data granularity, particularly focusing on the look-back horizon, an aspect that been unexplored previous theories. Furthermore, we empirically evaluate various models using a diverse set of time series forecasting datasets, which (1) verifies the validity of scaling law on dataset size and model complex- ity within the realm of time series forecasting, and (2) validates our theoretical framework, particularly regarding the influence of look back horizon. We hope our findings may inspire new models targeting time series forecasting datasets limited size, as well as large foundational datasets and models for time series forecasting in future work
1 Introduction
Because of the practical value of time series forecasting, past years have seen rapid development for time series forecasting methods using the paradigm neural network training. Neural Nets utilize different model architectures, including FFN-based [1] 3. Transformer-baat based 4 5
g and Convolution-ban based 10 neural nets have been proposed. Starting from around 2022, some previous work M [0] proposed that powerful models could be enhanced by extending the look-back horizon because more historical information can be utilized. However, our experiments (Figure 3) show that this claim may not hold for datasets in practice with a certain amount of training data: optimal horizon does exist, and will increase if amount of available training data increases. This calls for more thorough understanding of the impact of horizon and dataset size forecasting loss.
In Natural Language Processing (NLP)
Computer Vision (CV) 14 and other fields in deep learning, the impact of dataset size, model size and data granularity performance is sometimes summarized as the Scaling Law: larger dataset, larger models and more detailed data granularity improves performance in these cases, and theories [5 16 have been proposed to explain these behaviors. However, these theories do not lay emphasis on the horizon of time series, hence cannot be used directly to explain the impact of the horizon.
'Equal Contribution.
"Code for our experiments has heen made public at: Caa.ingg.aaForrriimeeeeeeeerroo
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

In this work, we introduce a comprehensive scaling law theory for time series forecasting, with particular emphasis on the impact of the horizon. This theory integrates dataset size and model size to optimize predictive performance hased on the look-back horizon. We further conduct experiments to (1) verify that scaling behaviors for dataset size and model size do exist time series forecasting and (2) validate our theory, especially about the influence of different horizons.
Our main contribution regarding time series forecasting includes:
1. We introduce novel theoretical framework that elucidates scaling behaviors from an intrinsic space perspective, highlighting the critical influence the look-back horizon on model performance. Our theory identifies optimal horizon, demonstrating that beyond this point, performance degrades due to the inherent limitations of dataset size.
2. We conduct a comprehensive empirical investigation into the scaling behaviors of dataset size, model size, and look-back horizon across various models and datasets. Our research es- tablishes robust scaling law for time series forecasting, providing a foundational framework that adapts to diverse modeling contexts.
As corollary to our conclusions, point out that different models might have different optimal look-back horizon for the same dataset (Figure IJ therefore, we call for future work to compare different models using the optimal look-back horizon for each model correspondingly rather than using fixed look-back horizon.
As further result of our findings, though widely used in previous work, to show a model benefits from longer horizon compared to baseline models is unnecessary for proving its superiority over these baseline models.
We hope our work may inspire future work when designing forecasting models for specific datasets of limited size, as well as future work proposing large foundational datasets and models in the field of time series.
2 Related Work
2.1 Time Series Forecasting
The task of time series forecasting is predict time series with N variables in its next frames (denoted as Y = {11. V2. ................................ yN E R Rxx) given its previous observations with H frames (denoted [11 F2, ... 2N E R Nx"". ). H called look back horizon in some scenarios.
as X
Channel Indeee--nddeenneett model means to predict y; by ý = fx::). Linear models and MLPs have been proven be effective learners for time series forecasting. A series of work (1 3 17 utilizes linear layers and methods like low-pass-filte and Ordinary Least Squares regression to learn linear relationships in the time series. Reversible MLP (18] proposes to use linear layers and MLPs with reversible Instance Norm [19] and obtains satisfying results. PatchTST 7 proposes to use patch embedding for time series.
Channel- -Deeeddeeenndttt model means to predict y by d. = [f [2, T2...., ZN NN series of works based Transformers and its variants have been proposed 4 5 G1. as well series of convolution- based methods based on TCN (T). More recently, iTransformer g proposes to use attention to capture the relationship between different variables. ModernTCN 10 proposes convolutional architectures enabling a wide Effective Receptive Field.
There have been many works analyzing mathematical properties of time series before machine learning exists. 222 23 give good summaries the early works from different perspectives.
Recently, there have been works proposing large foundational datasets and models for time series. Some propose foundational models that can do zero-shot forecasting Some propose open-source foundational datasets and verify the ability for transfer learning of foundation models trained on the datasets 25 26 There are also works utilizing LLMs do zero-shot prediction or use LLM backbones pretrained on text contexts to perform time series forecasting (2 30
'Released in close timing with ours, 27 verifies scaling law experimentally for the classic transformer architecture large mixed dataset for time series forecasting.

---

2.2 Scaling Law and related Theory
plethora of research has been conducted to investigate the scaling law in various domains of deep learning, encompassing Natural Language Processing, Computer Vision tasks, and Graph-based Neural Networks [12 13 14, [2]. These studies have not only observed the existence of scaling law but also proposed theories elucidate them. For instance, certain work has interpreted the scaling behavior from the standpoint of intrinsic space or data manifold, providing a deeper understanding of the underlying mechanisms [13 [6]. In parallel, the properties of time series have been the subject of extensive theoretical and empirical studies. Notably, some research has established bounds for the quantization error of time series, contributing to the body of knowledge on time series analysis (33
3 Theory for Scaling Law in Time Series Forecasting
3.1 Forecasting in Intrinsic Space
To represent the amount of information carried by time series slice of length L, we consult the concept of intrinsic dimension and intrinsic space. Consider all time series slices of length L in particular scenario, these slices are hypothesized to reside within an intrinsic space that encapsulates the fundamental characteristics of the data, effectively representing its inherent features. Denote this intrinsic space as (L) and its intrinsic dimension as dy (L).
It immediately follows that time series forecasting is equivalent to predicting a vector in M(S) given its previous H frames, which can be represented by vector in the space M(H)..
3.2 Formulation
Before studying the impact of horizon, dataset size and model on loss, we first formulate the intrinsic space, the data distribution and the form of the loss.
3.2.1 Intrinsic Space
For time series so.1.....I of length L (each s; is unit element in the sequence that may contain single or multiple variables, dependent or independent), we represent it using a vector x MLL).
We make these assumptions on the spaces M(2),, and the distribution of data in the spaces:
1. information-preserving Intuitively speaking, we should be able recover the real se- quence (which might be a multivariable sequence or a singlevariable sequence) from its corresponding intrinsic vector with the error bounded by a small constant value. Formally we can state this as follows:
Exists a mapping d from the original length-L sequence space ((L) to M(L), an inverse mapping o MM(L) + (LL) and a constant e < 1 related to L so that for any Rewo(L),
| 1 ( eL)).
2. Inverse Lipschitz: o should be K,-Lipschitz under L2 norm. That is:
Va, E M(L), (w)) S Kr|læ- - vll2
3. Bounded: For simplicity, we assume the values in all dimensions of the intrinsic space bounded, and thus we can scale the intrinsic space to fit it into M(H) = [0, (1).)
4. Isomorphism: M(L,) is isomorphic to subspace of M(L2) for L1 < L2- Moreover, the isometry should also preserve the data distribution in the space. Formally, we can state it as:
Let P(H,, Hzl denote the linear projection from M(H,) to the subspace of it isomorphic to M(H2), and let Coer, denote the covariance matrix of data distribution in M(L), then CoUL, should be congruent to P[L2, L [))(Co for any Lı < Lzz
5. Linear Truncation: Truncation time series space is close to linear projection mapping in M(L). Formally, state it as:
3

---

Define truncating function ty, ty|H1, H2](Zo.H, ). so that for any sequence let Shizhz denote the intrinsic vector for the subsequence from s[hı] to s(h₂ - 11. then H (3. 8o then 1p[H1, H2] R 1(H,, H2).
6. Causality: There should be an optimal model to predict the next S frames given the previous h + 0o frames, so that the error only originates from the intrinsic noise. That is:
]F[S]: M + M(S), s.t. lim P(y x= h-+00
Es);
where 7) stands for the noise ratio in the system, 8(-) stands for the Dirichlet function and [p, Σ) stands for normal distribution with mean and covariance Σ. (Notice that the noise distribution not necessarily normal distribution and our result holds for any noise distribution with mean and covariance a. However we only consider the normal distribution case here for simplicity.)
Moreover, we assume F(S) is first-order K Ki-Lipschitz and second-order K2- Kz-Lipschitz in L2 metric (as assume S is fixed don't diseuss how the Lipschitz coefficients vary with S and simply take them as constant).
7. Uniform Sampling Noise:
When drawing length-L sample, we assume the sampling noise is uniform in each direction in M(L).
8. Zip-f Distribution:
We assume the data distribution in the intrinsic space follows a Zip-f law on different dimensions of the intrinsic space. That is, the eigenvalue spectrum of Coel satisfies X Ag2 oz where X represents the i-th largest eigenvalue. This is shown by other work 34 35 and also verified in our experiments.
This result is asymptotic and does not suggest uniform intrinsic dimensions across sequence ele- ments. Specifically, for a sequence element with variables, dr(Ht) approximately in channel- independent scenarios. In contrast, for channel-dependee intrinsic dimension is typically less than the of dimensions for individual variables.
cases, which are more common, the total
In the deduction part we do not assume the specific relationship between dy (H) and H. Some previous work shows that in some cases (H( x @(H). In Appendix A.1 we discuss more about their relationship.
We formulate the intrinsic space and these assumptions more strictly and provide brief construction in Appendix A.2 Also, we discuss cases where these conditions are not strictly satisfied in Appendix A.3
of M())..
Moreover, under these assumptions made, the loss in the original space Lori can be linearly bounded by the loss in the intrinsic space Lines details of which can be found at Appendix A.4
In the following deduction, we use L to denote Lins-
3.2.2 Loss: Overall
In the following sections, we consider the task to predict a vector in M(S) given the vector cor- responding to its previous H frames in M(H) with model m. For simplicity, we represent the operation of obtaining vector in M(tz t1] by truncating the sequence from time f to t2 as zit t2]. where x is representation in M([s]).
= E H m==(0 H]))
Let m" denote the optimal Bayesian model, then it should satisfy:
m (x H)) = Eg- H 511010 H]
Thus:
== + - +
HJ model(x H)))
=En-M(H+s][([ + 11+55mm""a0
H|) - 5]-m"(x((0
+ -x)-)
+ E (0 : [++)-m""xpl) (x H]) * -m(x))
HH

---

Since z|H
and E--M((++) (2 H and m² (z)- (z) are independent with respect to s H H+S)
H + m (a(0 H]] 0, the loss is sum of the previous two terms: one is decided by the capability of the optimal Bayesian model (or the Bayesian Error), and the other
the model's ability to approximate the Bayes Estimation: I. Bogassian Tapproz-
We then calculate each of the two terms corresponding to the horizon, dataset size and model size.
3.2.3 Bayesian Loss
We consider the Bayesian error for MM(oo), when the loss for predicting S frames originates from the inherent uncertainty of the system. That is, we evaluate the amount of information carried by M(H) compared to ((o)). By assumption 5 and 6 in section 3.2.1 It can be verified that:
I Bayesion I (1 - 17) Bear [xo, ()))) + tr(Es)
According to assumption 7, the noise in the i-th predicted frame would be proportional to vi, and the total noise in the predicted S frames would be proportional S. Let oi denote the variance of the noise for a single frame in a single dimension, then tr(Es) should be equal to of ())
According to assumptions 4 and 8, we can express the inverse projection term into:
Ao
E[var(P [[)))
Σ
X
(az - (H)
d((H)sied
which indicates that:
Ao (az [[d,(H)az- + 5
Bogesian 20 KT 11)
3.2.4 Approximation Loss: Two Cases
The training data is sampled from the distribution of M [s|[ : H++S|==00: H]). Following previous works [6 37 we utilize the piece-wise linear assumption for deep learning models. That is, assume the model partitions the intrinsic space into N subregions, and does linear prediction for each subregion independently. Here we discuss two cases: the large-dataset limit and the small-dataset
limit.
If the dataset is large and contains many samples in the piece of the model, the model tends to learn the averaged behavior of the region. Intuitively, a larger dataset size brings smaller noise averages, and a larger model brings smaller diece-wise linear pieces, reducing the error caused by deviation of the data distribution to the linear model in the small blocks. Roughly speaking, the loss consists of two terms: one caused by the uncertainly within the subregions partitioned by the model L,, and one caused by the noise in the data that makes the model fail to learn the optimal model Lon If assume that the model partitions the space uniformly, then the loss should satisfy:
(H)
2 3
Ndj(H) (H) (o (d + (az - (H
KiD0
Lopprow K22
K2
47²2
The uniform partitioning should be the most naive case for the model to learn, and we also analyze learned in Appendix[D] Also, it is worth noticing
on the cases where more advanced partitionings that the noise consists of two term, one is the systematic uncertainly as shown section [32.3] and the other is caused by the horizon limitation so that the effect of unseen dimension seems exactly like noise for the model. Please refer to Appendix B.2 for a more detailed derivation.
Otherwise if there are few data samples, or the model is sufticiently large and the model cannot learn to average the noises of closed samples, but rather learns to remember each sample in certain pieces. This would give data-scaling loss determined by nearest-neighbor distance. In this case:
K2 (H श,
Layproz RJ 4TH
Please refer Appendix B.3 for detailed derivation of this scenario.
Boundary of the two phases decided by Dataset size and Model size, as well as horizon. is worth noticing that in time series forecasting tasks, the sliding window method for constructing
5

---

data samples is usually used. Therefore, the closest data points are always dependent on each other and are strongly correlated. The effective number of mutually independent data fragments is approximately proportional to D H rather than D, and = D/NH would be very approximate order parameter separating the previous two scenarios. Again, please refer to Appendix for more detailed derivation.
3.3
Optimal Horizon under Each Circumstance
As stated in section 3.2.2 the total loss would be simply the sum of the components deduced above. Analyzing the total loss form, we may achieve an optimal horizon H (or a corresponding optimal intrinsic dimension d7 that minimizes the loss for different cases.
3.3.1 Optimal Horizon for large amount of data
For the case with sufficient data, the total loss is:
= Bayesion L Approz
Rk2 di (H)2N aro0
+ K7(1 - Ao
1
4x²
0% 1 di (H
(H)(H)
K7X0
( + (az 1)d9 1)dpz-1(HH (H)
D
Kk d, (H) मरे
(d)(H)
+ K7 11) az (H
o sy
4T²
D
(since always assume N < D)
We consider two cases. The detailed derivation can found Appendix C.I
4,(M3 If model size too small compared dataset size such that N o (DarttTT)) ) then the effect of dataset size on picking optimal horizon could be neglected, thus:
di = W(
ln
OZ N)
In ug N.
C
az
0
where W(.) is the Lambert W function (W(x)= x) and C
K2
If is not that small: w(D Then the noise effect would be dominant in picking optimal H, and the optimal d, (H) would be:
di (R(1-1))02) )
Sdd
In this case, dj grows noticeably with the increment of D and the decrement of N.
3.3.2 Optimal Horizon for a relatively small amount of data
data scarce, the loss could be written as:
loss ~ KP - 11 az (H
Ao
K? 4n
( +,,,
It is worth noticing that in this casc, we assume the model is always large enough to find the nearest neighbor for a test sample, hence the loss irrelevant with N. We can estimate the optimal d' as:
In
di = C. In
Where C, constant irrelevant with D, N or H. The exact form provided in Appendix C.2 Compared to the first scenario, the optimal d, changes much less in this scenario.
'Data points between time [t- [t+H:t+S] is considered one sample, and [t- [++H+I:+++1]] 1] is considered another sample the dataset. These two samples strongly correlated with each other.

---

Experiment Results
4.1 Scaling Law for dataset size and model width do exist for Time Series Forecasting
Traffic
ETTh1
ETTh2
Made Miad
1.
0:4
Weather
E
ETTm2
Figure I: Data Scaling. The proposed formula loss(D) = A+ B/Dº fits well. More comparison with other formulas can be found at Appendix !
Traffic
Weather
ETTh2
ETTm1 deveeii Teade
Figure 2: Width Scaling. When the model is not powerful enough, (Ws)(W)= = A + B/W fits well for these situations. When data is scarce, large model may lead overfitting, observed with ModernTCN on ETTml.
As depicted in Figure and Figure and model-size scaling across a diverse range of datasets and models. This validation underscores the robustness and versatility of our proposed theoretical framework in the context of time series forecasting.
corroborate the scaling behaviors pertaining to data scaling
Here we mainly include NLinear [DMLP, ModemnTCN[ and iTrrnsformer [ our models, covering a scenario of Channel- Inneeeeenddeenndent and Channel- Deaunel-Dependet,, FaN-based, Transformer- based and Convolution-basee Traffic and Weather [4] 5 as our datasets. Detailed experiment settings can be found at Appendix
methods. For datasets, mainly use ETTh1, ETTh2, ETTm1, ETTm2.
It can be seen that for all these models on all these datasets, for the dataset-scaling case where loss 1D++//l/m D and the model width-scaling with large amount of data: loss = Cw +1/ The results fit well, thus verifying the existence of the original understanding of the scaling law.
However, in some special model size would not gain performance, and may hurt performance (if regularization that is not strong enough is added) in some cases, like ModernTCN for ETTml.
when the model is large enough approximate the data, increasing
4.2
The Impact of Horizon to Final Loss
4.2.1
Optimal Horizon and Training Data Amount
Optimal Horizon grows with more available training data. We conduct experiments, fixing the available training data amount and model size while adjusting the horizon. As shown in Figure 3 an
7

---

optimal horizon is observed at each data amount, and this optimal value grows with more available training data.
Traffic, MLP
Traffic, ITransformer
Traffic, ModernTCN
ECL, MLP
Mitin Weather, iTransformer
Weather, ModernTCN
5555332
337
20:07
Figure 3: Loss v.s. Horizon for certain amount of training data, for different datasets and different models.
A dataset with a larger feature degradation has smaller optimal horizon. As our theory predicts, larger as leads to a smaller optimal horizon. In the experiment in Figure 4 we use the same Linear model on two different datasets with comparable amount of training data: the Exchange Dataset has 70% available training data compared to the ETThl Dataset. We use eigenvalues obtained by doing Principal component analysis on sampled series as an approximation the feature variance in the intrinsic space. Detailed procedure and more PCA results can be found at Appendix G
Figual Index
Exchange, NLinear
ETTh1, NLinear
0.50
8.0
CTT)
Hobel
Figure 4: PCA results under Indeeenden and Instance Normalization setting(left). Loss v.s. Horizon for certaim amount of training data on Exchange(middle) and ETThI Exchange dataset has 70% data points compared ETTh for training. However, since its feature degradation is stronger, the optimal horizon (< 30) using 100% of Exchange dataset is much smaller than the optimal horizon of the ETTh1 dataset (> 300) with only 11% of available training data.
Channel-Dependeee and Channel-Independent Models sometimes are in different states of the two cases. For CD models, d, (H) is larger and less training data is available, hence it tends to be in the few-data limit. For CI models, d, (H) is smaller and is larger, hence it may reach the data-dense limit (where the scaling exponent for D -)).
In the following Figure iTransformer on Traffic dataset in the data-dense limit. For MLP on Transformer, when the horizon is small it is in the few-data limit. For the Linear model, since is simply linear (rather than piece-wise wise linear), we expect to be within the data-dense limit even for long horizons and when the dataset is relatively small (like ETTh1).
The advantage of channel-independent and deaeneet models can be explained from the perspective of our theory. For channel-dependent models, the horizon limitation is smaller hence it performs better with plenty of training data. For snanpee-indepeedent models, more training data available; moreover, d is smaller, making the scaling exponent larger.

---

Traffic, (Transformer
Traffic, MLP
ETTh1. NLinear
810200 418
1c4
Figure 5: Data scaling behavior for iTransformer PCChannel-Indeeendeee model, middle) and NLinear (CI, right).
model, left) and Norm-
4.2.2 Optimal Horizon v.s. Model Size
As predicted by our theory there are two cases. (1) When N is small, the optimal horizon does not change much with N. (2) When N is large, the model size scaling term no longer dominates, the dominants thus larger N leads to smaller optimal H.
coefficient of the noise term D
Ndj
SingleayyniL
Medem
Whather ModerntCN
Figure 6: Loss v.s. Horizon for models of different widths. For MLP on ETThl (left), ModernTCN on (TThlmiiddle) and ModemnTC on Weather (right).
From the observations in Figure can be discerned that in the initial scenario, where an enhance- ment in N results in performance augmentation, the optimal horizon remains relatively invariant with respect to N. This is exemplified in the case of the SingleLayerMLPP model. However, in the second scenario, where N attains its optimal value (i.e., for certain horizon, smaller N surpasses larger N in performance), larger N will correspond to a reduced optimal horizon. This phenomenon evident in the instances involving the Modem'TCN model.
As predicted by our theory, we see that dataset size has an impact on optimal horizon, while model size has a less significant influence on it.
5 Discussion and Conclusion
5.1 Limitation and Discussion
Our theory mainly covers the part of time series forecasting, and experiments verify our proposed theory on some of the well-used datasets of various sizes. However, these datasets are still small compared some of the recently proposed large datasets 25 The scaling behavior predicted in our work of the horizon on these large datasets remains to be experimentally verified and studied. Moreover, we mainly use popular time series forecasting models recent work and these models might be over-designed for a few datasets for time series.
Moreover, work mainly focus the in-domain setting, rather for wwerraindd-then-fiieee models or foundation models trained on mixed datasets. We discuss more about whether the assumptions made and theory deduction work for mixed datasets at Appendix H
Since we focus on the impact the horizon which is an adjustable hyper-parameter for forecasting tasks but a fixed parem--prreeter for other time series tasks, this work does not involve scaling behaviors for other tasks related to time series analysis. The theory for scaling law in other areas of time series forecasting as well as for foundation time series forecasting models remains to be studied.

---

Also, although our theory is compatible with analyzing the effect of the predict length S and current experiment for fixed S verifies the impact of H, is still worth studying the impact of S on optimal H. Also, our theory only holds with assumptions made in Section 3 We leave these to future work.
We call for future work to compare different models by 'performance at optimal look back horizon' rather than 'performance at certain look back horizon' to improve robustness. This work further elucidates that, though used in a lot of previous work, to show model benefits from longer horizon compared to baseline models does not necessarily prove its superiority over these baseline models.
5.2 Conclusion
In this work, propose theoretical framework that contemplates the influence of the horizon on scaling behaviors and the performance of time series forecasting models. We take into account the size of the dataset, the complexity of the model, and the impact of the horizon the ultimate performance. An expanded horizon results in a diminished Bayesian Error, but it simultancously complicates the task for a limited dataset to fully encompass the entire space and for a model of restricted size to learn effectively. Furthermore, our empirical investigations corroborate the existence of scaling behaviors in relation to dataset size and model size, and validate our proposed theory concerning the impact of the horizon.
Our theory provides some insight into the area of time series forecasting. For certain dataset, it would be beneficial to design the models and hyperparameters according to the dataset size and feature degradation property of the particular dataset. Moreover, we think further experiments larger foundational time series dataset about the optimal horizon with respect to pretraining loss and the loss for transferring certain datasets may provide further insight well.
In conclusion, we aim to provide insights to future work on time series forecasting, emphasizing the importance of the horizon and its potential impact on scaling behaviors in time series forecasting tasks.
Acknowledgments and Disclosure of Funding
We would like express our great gratitude Professor Robert M. Haralick for significantly insightful and invaluable help and discussion. We would also like to extend our deep gratitude to Jiaye Teng, whose invaluable help and advice have been instrumental to our progress. In addition, we are profoundly grateful to Professor Yang Yuan for the enlightening discussion we had, which provided significant inspiration and insight into our work.
We would also like to thank Professor Xiaolong Wang and Jiarui Xu at UCSD for inspiring discussions about scaling context length for NLP, and Yu Sun at Stanford for discussions about properties of time series.
We would also like to express our gratitude to CPHOS (https://cphor) en), an academic non- profit organization dedicated to providing Physics Olympiad Simulations (targeting Physics Olympiads like IPhO) for high school students for free. At CPHOS, we communicate innova- tive ideas and get to know great academic collaborators, without which this work could not have been carried out.
We would also like to thank reviewers, ACs and PCs from NeurIPS 2024 who have provided valuable suggestions which have helped to improve this work.
'Meanwhile, NLP and CV tasks feature long-range dependency (low degradation with horizon), large datasets, hence more detailed data granularity (e.g., context length, image resolution) produces better results in nowadays" work. However, since the community is using larger and larger models and data granularity (context length, etc), but the amount of training data is limited, it is nature to raise such question: will we see similar patterns in time series forecasting that more detailed data granularity may hurt performance in NLP or CV tasks in the future? This is also an interesting question for future work.
10

---

References
[1] A. Zeng, M. Chen, L. Zhang, and Q Xu, "Are transformers effective for time series forecasting?" 2022.
[2] S.-A. Chen, C.-L. Li, N. Yoder, S. O. Arik, and T. Pfister, "Tsmixer: An all-mlp architecture for time series forecasting," 2023.
[3] Z. Xu, A. Zeng, and Q Xu, "Fits: Modeling time series with 10k parameters," 2024.
[4] H. Zhou, S. Zhang, J. Peng, S. Zhang. J. Li, H. Xiong, and W. Zhang, "Informer: Beyond efficient transformer for long sequence time-series forecasting." 2021.
[5] H. Wu, J. Xu, Wang, and M. Long. "Autoformer: Decomposition transformers with auto- correlation for long-term series forecasting," 2022.
[6] T. Zhou, Z. Ma, Q. Wen, X. Wang, L. Sun, and R. Jin, "Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting," 2022.
[7] Y. Nie, N. H. Nguyen, P. Sinthong, and J. Kalagnanam, "A time series is worth 64 words: Long- term forecasting with transformers," " in International Conference on Learning Representations, 2023.
[8] Y. Liu, T. Hu, H. Zhang, H. Wu, S. Wang. L. Ma. and M. Long, "itransformer: Inverted transformers are effective for time series forecasting," 2024.
[9] H. Wu, T. Hu, Y. Liu, H. Zhou, J. Wang, and M. Long, "Timesnet: Temporal 2d-variation modeling for general time series analysis," 2023.
[10] L. donghao and wang xue, "ModemTCN: A modern pure convolution structure for general time series analysis," in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available:
[11] H. Wang. J. Peng. F. Huang, J. Wang. Chen, and Y. Xiao, "Mien: Multi-scale local and global context modeling for long-term series forecasting," 2023.
[12] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, Wu, and Amodei, "Scaling laws for neural language models," 2020.
[13] Hoffmann, S. Borgeaud, Mensch, E. Buchatskaya. T. Cai, Rutherford, D. de Las Casas,
L. A. Hendricks, J. Welbl, A. Clark. T. Hemnigan, E. Noland, K. Millican, G. van den Driessche,
B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, W. Rae, O. Vinyals, and L. Sifre, "Training compute-optimal large language models," 2022.
[14] X. Zhai, A. Kolesnikov, N. Houlsby, and I.. Beyer, "Scaling vision transformers," " 2022.
[15] Y. Bahri, E. Dyer, J. Kaplan, J. Lee, and U. Sharma, "Explaining neural scaling laws," 2024.
[16] U. Sharma and J. Kaplan, "A neural scaling law from the dimension of the data manifold," 2020.
[17] W. Toner and L. Darlow, "An analysis linear time series forecasting models," 2024.
[18] Z. Li, S. Qi, Y. Li, and Z. Xu, "Revisiting long-term time series forecasting: An investigation on linear mapping." 2023.
[19] Kim, J. Kim, Y. Tae, C. Park, J.-H. Choi, and Choo, "Reversible instance normalization for accurate time-series forecasting against distribution shift," in International Conference on Learning Representations, 2021. [Online]. Available: https:/oppanevvew.eee/foummm cGDAKQolCOp
[20] S. Bai, J. 7. Kolter, and V. Koltun, "An empirical evaluation of generic convolutional and recurrent networks for sequence modeling." 2018.
[21] R. Sen, H.-F Yu, and Dhillon, "Think globally, act locally: deep neural network approach to high ligh-dimensional time series forecasting." 2019.
[22] E. Parzen, "An Approach to Time Series Analysis." The Annals of Mathematical Statistics, vol. 32,
4, pp. 951 989, 1961. [Online]. Available: httpst
"nneaatinngIItttenatiioon [23] J. G. De Gooijer and R. J. Hyndman, "25 years of time series forecasting." International journal of forecasting, vol. 22, no. 3, pp. 443 473, 2006.
[24] A. Garza, C. Challu, and M. Mergenthaler-Canscoe "Timegpt-1," 2024.

---

[25] Y. Liu, H. Zhang, C. Li. X. Huang, J. Wang, and M. Long, "Timer: Transformers for time series analysis at scale," 2024.
[26] A. Das, W. Kong, R. Sen, and Zhou, "A decoder-only foundation model for time-series forecasting," 2024.
[27] T. D. P. Edwards, J. Alvey, J. Alsing, N. H. Nguyen, and B. D. Wandelt, "Scaling-lawe laws for large time-series models," 2024. [Online]. Available:
[28] N. Gruver, M. Finzi, S. Qiu, and A. G. Wilson, "Large language models are zero-shot time series forecasters," 2023.
[29] C. Chang, W.-Y. Wang, W.-C. Peng, and T.-F. Chen, "Lim4ts: Aligning pre-trained Ilms as data-efficient time-series forecasters," 2024.
[30] Zhou, P. Niu, X. Wang, L. Sun, and R. Jin, "One fits all:power general time series analysis by pretrained Im," 2023.
[31] M. Jin, S. Wang, L. Ma, Z. Chu, J. Y. Zhang, X. Shi, P.-Y. Chen, Y. Liang, Y.-F. Li. S. Pan, and Q. Wen, "Time-Ilm: Time series forecasting by reprogramming large language models," 2024.
[32] J. Liu, H. Mao, Z. Chen, T. Zhao, N. Shah, and J. Tang, "Neural scaling laws on graphs," 2024.
[33] P. Zador, "Asymptotic quantization error of continuous signals and the quantization dimension," IEEE Transactions on Information Theory, vol. 28, no. 2, pp. 139-149, 1982.
[34] N. Levi and Y. Oz, "The underlying scaling laws and universal statistical structure of complex datasets," 2024.
[35] S. Petrini, A. Casas-i Muñoz, J. Cluet-i Martinell, M. Wang, C. Bentz, and R. Ferrer-i Cancho, "Direct and indirect evidence of compression of word lengths. zipf's law of abbreviation revisited," Glottometrics, vol. 54, p. 58-87, 2023. [Online]. Available: 54_407
[36] M. Buzug, J. von Stamm, and G. Pfister, "Characterising experimental time series using local intrinsic dimension." " Physics Letters A, vol. 202, no. 2-3, pp. 183-190, 1995.
[37] N. Chan, Li, L. U, and R. Cheng, "Plame: Piecewise-lineal approximate measure for additive kernel svm," IEEE Transactions on Knowledge and Data Engineering, vol. 35, no. 10, pp. 9185-99977 2023.
[38] K. D. Ba, P. Indyk, E. Price, and D. Woodruff, "Lower bounds for sparse recovery," in Proceedings of the twenty-first anmual ACM-SIAM sympositm on Discrete Algorithms. SIAM,
2010, pp. 1190-1197.
[39] X. Qiu, J. Hu, L. Zhou, X. Wu, J. Du, B. Zhang, C. Guo, A. Zhou, C. S. Jensen, Z. Sheng, and B. Yang. "Tfb: Towards comprehensive and fair benchmarking of time series forecasting methods," 2024.
[40] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang. Z. Vetioo, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang. J. Bai, and S. Chintala, "Pytorch: An imperative style, high-pen gegh-performance deep learning library," 2019.
[41] H. Liu, Z. Dai, D. So, and Q. V. Le, "Pay attention to mips," 2021.
12

---

A More about the intrinsic space
A.1 Relationship between intrinsic dimension and horizon
We claimed in section 3.2. that (H) should be asymptotically linear to H. Here we give rough explanation for this. Since the reconstruction error is bounded by a constant value irrelevant to L, the relative error should be O That is, the error reconstructing y E (L)) from z E .M(L) should be O
On the hand, this error can be viewed as error caused by compressing a do dimeensssiio vector into a ddimensiill space. From a sparse recovery aspect. if the covariance matrix in (T)) has rank r, then the do [.) -immennsiol vector would be equivalent with r-sparse vector. 33 shows that, in this case, the minimal dimension that could recover it with relative error bounded by C. is (()) Since in our case I. due to intrinsic uncertainty, hence dN (L. (2( (L). And since
dy (L) S do(L) = O(L), we proved that (L) = O(L).
A.2 Construction Method
We informally provide specified method to construct M(.) theoretically in recursion method. If (()),, MML 1) has been defined, then we define M(L) as follows: given a time series Fo. L-1 of length L, we represent by concatenating the representation of 20. L-2 in M(L - 1) and - in space with dimension dim(M(L- 1) + 1). Then we find a manifold that these points lie in in this space to represent (L)..
A.3 Non-linear truncation
We assumed that the truncating function is close to linear projection in section 3.2.1 But in fact the two functions might have some subtle differences. This because for a certain feature, estimating the feature with sequence of finite length L might result in error for the feature even if each element in the sequence is accurate enough. (For example, if the feature comes directly from DFT (Discrete Fourier Transform), there might be "leakage error' if the base frequency is not an exact multiple of the feature frequency.) This a systematic error caused by the "precession', where the feature's characteristic frequency mismatches the measuring frequency.
For any sequence S and let 2t E (M222 t1) denote the vector for the sequence S(t1 +2] we can define a 'truncating function' to H, H2] .MHH) + .MH2) for any Hi 2 H₂ > 0 with typ [H,, H2l = To.H,, Let [H,, H2] denote the inverse image function, which maps x E M(H2) to {y y E t, (3)) = x. Notice that this truncating function is unrelated to the specific sequence. Now, assume that bp H,, H2) is smooth, then from implicit function theorem we know that for any vector & E M(H,) if (H,, H2]
is locally differentiable in a neighborhood of x then [H,, (tp(tp H2](z)) is a submanifold of M(Hi) with dimension (H, - d (H2), and from Sard's theorem we know that the targets of these locally undifferentiable points have measure zero. That is:
Let S = {x x E t, (2) & Md((H))-d(HH))) where Md denotes the set of all d-dimensional manifolds. Let μ. denote a measuring function defined in M(H1), then μ(S) = O.
Hence the impact of such 'critical points' to the total loss is negligible. For simplicity, we can assume = 0. Then ip could be written as:
(H,, H2)(2) P(H, H2]g|H,, H2)
where P(H,, H2] is the projection mapping from M(H,) M(H2), and 9H,, H2) is an invertible mapping in (MH,) that maps 172 (y) to a (d,(H, - (H2)))-dmmmesiooaa subspace of M(H,) for each y E M(H2)..
We can naturally assume that the 'precession' effect on each subsequence with certain length ( only related to I and intrinsic properties of the task itself, then consider length-21 sequence, could be viewed two independent measuring of length-/ subsequences, hence this error is always bounded by O( 1 We can formalize this condition into:
VI
H,, Ha](x) x112
K
Vx E M(HH),
19] H2](x)
WHI
13

---

where A. is constant, and || 1-12 represents L2-norm. Also we should have Elg(z) - x| = 0. This
results in extra term in the Bayesian loss. We deduce it again formally from (=)))
E[var(t,"(H((())) = Elvar(g ( (x))
R
b + B[uar P (x:)
H
k²
X + Elvar
Osi<dr
Same what we have done in section 3.2.3
E Eluar P
(x))1 =
Σ
X
d((H)smich
Hence:
h²²
Biuar(t, [(vr(t,>(HH[(()) [H](t)]
S
Σ A
Σ
X
H
0gi<d
Ag
k²az
R
07
H
dj (H)
which indicates that:
Aa
x2azz
loss < K2
11)
+ no 52 5 dy
oz H
dz(H)oz-1)
Since should be small, and typically could be neglected. Hence, it is reasonable that we assume tp [H, Hy) == P(H, Hz).
(it holds for all experiment results), the extra term
A.4 Reduction of Loss into Intrinsic Space
We prove that under assumptions made in Section 3.2. the loss in the orginal space can be linearly bounded by the loss in the intrinsic space.
Consider we are predicting =[0 S] from x1-H : 0J. let yi E M(H) be the intrinsic vector of
x[-H and yo E MSS) be the intrinsic vector of x00 8] (the true intrinsic vector). If have model m so that:
Ez BA(- - yol2l S Line
where Lins represents the expected enror (or loss) in the intrinsic space. Then, from assumption 2 we have:
Ex1 (= (56)ll < KiLins
and from assumption we know that Esjo:steorrsss z S]12 e(S). Therefore:
 (mstsoooorrrssss==0m0m0m)))))))) z(0: S112]
Lori =E,
20 - - 22 2E, 20. (yo) - x(0: 2]
L2K, + 2e(S).
Therefore, the loss in the original space is linearly bounded by the loss in the intrinsic space. W.L.o.g we may focus on studying the loss the intrinsic space.
B Details about the loss deduction
B.1 Boundary
Recall that in section 3.2.4 we mentioned that there are two scenarios which the model should work in totally different ways. The boundary between the two scenarios depends not only the
14

---

dataset size D but also on the model size and prediction horizon. A larger model subdivides the intrinsic space into smaller subregions, each containing fewer training samples, with the number of samples per subregion inversely related to model size, expressed as N-1.
In time series forecasting, unique characteristic is that single sequence, containing multiple length-/f subsequences, counts as multiple samples. However, these subsequences are not fully independent due their proximity in the intrinsic space. To quantity independence, we define separation function f(H), assumed to be proportional to H, meaning that sequence of total length I. independent samples. Thus, the effective number of independent samples in the represents І(H)
dataset scales with D π
D Combining the above analysis, we can define a hyperparameter as 5 = N If E is large enough, then we can assume data is sufficient, and the model tends to learn linear approximation for each subregion; & is small, we can assume data is scarce and the model tends to find the nearest neighbour for each test sample.
B.2 Sufficient Data
We formally derive the total loss in the scenario where we have large amount of data. First recall that the Bayesian loss calculated in section 3.2.3 is:
A0
L Bayesian = K7 11 (az - 1)d, (H) + 5
As we assume S is fixed, no ( is constant and hence could be ignored. Now let's consider the approximation loss, should come from two sources: the limited model size would result in uncertainty in the subregions, and the limited dataset size would bring noise to the data. Let's consider these two causes separately.
B.2.1 Model Size Limitation
We assume that a model is separating the intrinsic space into N regions, and does a linear prediction for each region. It is easy to verify that the optimal prediction from M(H) to M(S) should be first-order Li-ishhhiitz and second-order Ky-Lipschitz, too.
For any region with volume V, the loss could be estimated as:
L = / (H) x: x(f(x) 1xx)2
where 7 the predicted linear function for the region. Expanding with Taylor's series, we have f(x) R ((x) (f'(x)) ff'vvv)) - vv). From the second-order Lipschitz condition, we can conclude that:
[f(x) - 1(x)) < K3(x uv)*
The total loss could be bounded as follows:
loss K3 f(=)(x-
where Qxz) represents the center the region that is in and f(x) represents the probability density. We can find that the integral in the formula is exactly the 4th power distortion of the quantization. For an arbitrary quantization with code density g(x),
33 gives an exact value of distortion equal to:
/ f())(z - (d)) = -2 2e r di (H) +4 (T
dH( +
/
f(x) dr
d, (H)
2
g(z) 3ztRy
Assume dy(H) is sufficiently large, then I ( z 1. So the expression can be transformed into:
f ((()(=-Q<(())))) d apdn
f
f(x) dr
2
g(x) arm
15

---

For uniform distribution, g(z) 1 (as we assume M(H) = [0, 1]4,(1)
Hence:
f(a) da 1
g(x) ((tt))r(m3
And the total loss could be bounded as:
dj ( 4ptr
loss
K2
4
From this bound, we can see that a larger horizon implies larger burden for the model, which might lead to worse performance for the model.
B.2.2 Dataset Limitation
Following the previous chapter, as we assume the mapping represented by the model is piecewise linear function, then regionally it can be written the usual form of the goussian-markon linear model, assuming there are homoscedastic uncorrelated noise e for every single sample in the dataset. Now we show that this noise will cause another term of loss even compared to the optimal performance of the model. We first analyze a single region containing D; samples. Consider X,Y from the training set:
= +
Where X is of shape D. x dH( and Y, is of shape x (S). 8 is the variable to learn of shape di (H) x (S). The BLUE estimator should be = xX
Now consider X', Y' from the test set:
y = X'B+ +E'
Lirecon E'(('' - X'B)
EI(E' - X'(x"xx-XXX"
=E (x s)
The first term goes to 0 if the test set is sufficiently large, and it is a constant given a fixed test set, so we can simply ignore it our analysis. Consider the second term, we can show that:
[[(X'(x"x)" X E) D, (1))
Recall the noise consisting of two sources:
Y X + F[S)(X P2(H))+(()) + +
As we've done in section 3.2.3 we can derive that:
var - - PP A0
Σ A
di di(H)
Ao
(az - 1)d (H)
As F[S] is Ky-Lipschitz, this indicates that:
K7A0
Eh (az 1)d9 (H)
And as mentioned section 3.2.3 var(EB) = 0 S2 (S), hence we can bound the total loss related
to D, into:
KfA0
loss; S (H)
(az 1)d =
It is worth noticing that, although different regions have their own distributions and hence may have different cigenvalues, the loss expression is irrelevant with the eigenvalues, so we can ignore this issue. Considering the entire dataset with all regions, the total loss should be:
loss D2
loss ~
D (d) D
K7\0 (az 1)d
(0 S ds)
(H)
16

---

This indicates that, using a larger model would be more prone to being affected by noise in the dataset, so especially when the data noisy and the dataset size is limited, our theory suggests that it would perhaps be better to use smaller model.
B.2.3 Total loss
Combining the above approximation loss and the Bayesian loss, the total loss would
loss = K2 N
+ K?(1 11) A 1 az 1 (H) o2 D 0 S
(H)
4
B.3
Scarce Data
As the model finds the nearest neighbor, the approximation loss in this scenario should consist of two sources: the distance from the test sample to its nearest neighbor and the noise in its nearest neighbor. the distribution is smooth, then locally around any point x, the distribution of other points can be viewed as uniform when D is large.
Consider sphere of radius around test sample, the probability that no point the training lies in the ball is approximately:
P(r) s - = e D7,,4
I 4,00)
where Cd =
(1)2(1))+1) denotes the volume of a unit sphere.
From this, we can derive that:
[r (T()) + 1)) कगीय T( 2
dz(H) D गसंक, f(x) f(x)
Td (H)
Considering (H) is large, can get that:
dr dr(H), D उा:ा
B/ra]
4T
which means:
Ktd fD
loss R
Combining the Bayesian loss similar to the previous scenario, the total loss should be:
loss KP(1 1)) az (H)
Ao
KP
d JnयN
4T
C
Optimal Horizon deduction
Continuing from section B we study the optimal horizon in two scenarios.
C.1 Sufficient Data
Let's first study the case where the dataset is large, which is the case in section B.2
C.1.1 Small Model
(
the model size is too small compared to the dataset size, i.c. N () 2y03)44 (however we still assume N is sufficiently large compared to other variables), then the effect of dataset size picking the optimal horizon could be neglected. In this case, the optimal value of d (ff) is unrelated with the dataset size D. and should satisfy:
KZ N
d' =
)az
K3 In N
17

---

Solving the formula we get the following result:
di = W(
N) ~
4
1+
1
O
(1-)(1-
where W(.) is the Lambert W function and Co
K2
We can see that in this case, dj is not only irrelevant with D, but also changes very little with N.
C.1.2
Large Model
-ا ) then the
On the other hand, if assume the model is large enough, such that N w(D (a)()>))) noise effect would be dominant in picking the optimal H. In this case, the optimal value of d1 (H) should be:
k²(1 >) d; = No )
We can see that this case the optimal d, changes rapidly to and N. Moreover, the optimal horizon increases with larger dataset size D and decreases with larger model size N (this is quite counter-intuitive but it surprisingly matches our experiment result).
C.2 Scarce Data
This is the case in section B.3
We can directly solve
Bloss Jdy(H)
O. which equivalent to:
4m
n)1a0
2 D
D
of
d7 zz
di
We can estimate that d7 = 30
In In D
2 In D
d'
=
az 1 In
where C' = 4T (1 - Ao Substituting di into the formula, we can get:
2
(az - 1).32 + 3
Hence we have:
8(ag
C
B
2(11 - 1)
and di should be:
8(0g-1 11)
Cas
In
In In
di
Ozz-I
2(az
In
Compared to the first scenario, the optimal dj almost doesn't change in this scenario.
D Details about model's partitioning
As we mentioned in section 3.2.4| we assume the model partitions the intrinsic space uniformly, but there should be better partitions. As 33 states the optimal partition should satisfy g(æ) x dy(H)
(z) (,00+ and if the distribution is Gaussian then the loss (only for this term) should satisfy:
N माीक
Lp K3
ed (H)2-2
18

---

Replacing the corresponding term with this, we see that in the case in section C.1.2 since the noise
makes the most impact, this term does not matter. However, in the case in section C.I.J if assume the dataset size is infinite, then the loss is asymptotically monotonically decreasing with dy (H). Thus, the larger the horizon we use, the better performance the model will potentially reach.
However, experiments show that the optimal horizon always exists in this case. This might suggest that the model structure limits itself from finding the best partitioning. Hence, we expect that when we design good enough model structure (maybe only exists theoretically) and when we have enough data, we should be able to make full use of our horizon regardless of the model size or computational resources.
E
Experiment settings
E.1
Dataset
Dataset
Dim
Pred Len
Dataset Size
Frequency
Information
ETTh
7
192
(8545, 2881, 2881)
Hourly
Electricity
ETTh2
7
192
(8545, 2881, 2881)
Hourly
Electricity
ETTml
7
192
(34465, 11521, 11521)
15 min
Electricity
ETTm2
7
192
(34465, 11521, 11521)
15 min
Electricity
Exchange
8
192
(5120, 665, 1422)
Daily
Economy
Weather
21
192
(36792, 5271, 10540)
10 min
Weather
ECL
321
192
(18317, 2633, 5261)
Hourly
Electricity
Traffic
862
192
(12185, 1757, 3509)
Hourly
Transportation
Table 1: Dataset used for experiments
We conduct experiments on 8 datasets listed here. ETTh1, ETTh2, ETTml, ETTm2 4 contains 7 factors of electricity transformer from July 2016 July 2018 at certain frequency. Exchange [5]
includes panel data for daily exchange rates from 8 countries across 27 years. Weather [5] includes
21 meteorological factors from Weather Station. ECL [5] records electricity consumption of 321 clients. Traffic [5] records hourly road occupancy rates for 2 years in San Francisco Bay area freeways. For all experiments, the pred-len is set to 192.
In our experiments, we do iterations for ETTh1, ETTh2, ETTm1, ETTm2, Weather and Exchange, and draw graphs with error bar equaling the standard enror of these iterations. For Weather dataset, actually the error is minor. (see Appendix [E.3), and for larger datasets including Traffic and Electricity, the error is even smaller. Therefore, conduct 1 iters for Traffic and ECL since they are large and will introduce minor std error.
E.2 Drop-last issue
The drop-last issue is reported by several researchers B 39
That is, in previous work evaluating the model on test set with drop_ lapt_aassteeeeee setting may cause additional emrors related to test batch size. In our experiments, use drop-last set to False avoid this issue.
E.3 Time-related Domain Difference in Datasets: can be neglected to a certain extent
We use the first p percent of our training dataset as our dataset as available training data. This may introduce an error caused by time-index-relater domain differences in these datasets. To avoid this we utilize the Instance Normalization [9]. Our further experiments on the Weather dataset show that this error is actually minor compared to the randomness introduced in the training procedure. We use an NLinear model to forecast 192 frames based on a took-back window size 336. We use 10% of training data, from different slices in the training set. For each slice, we repeat the experiment for times. The result is:
It can be seen that the variance introduced by different slicing points is very small.
19

---

MSE
starting point=20%
40%
60%
80%
Expl
0.223
0.220
0.222
0.223
Exp2
0.221
0.221
0.223
0.224
Exp3
0.221
0.221
0.223
0.221
Avg
0.222 + 0.001
0.221 + 0.001
0.223 = 0.001
0.223 = 0.001
Table 2: MSE measured for weather dataset
E,4 Models and Experiment settings
For all models we utilize instance normalization that normalizes an input sequence based on its mean and std. All the deep learning networks implemented in Torch[40) and conducted on cluster with NVIDIA RTX 3080. RTX 3090, RTX 4090D and A100 40GB GPUs. Although we use multiple types of GPUs, it is possible to earry all experiments single GPU with greater equal to 24GB of memory. Experiments conducted in this work cost about O(103) gpu hours for RTX 3090 in total.
E.4.1 Linear Models
For linear models, we use the linear layer after instance normalization, and perform de-nommalization after the linear layer, equivalent to the NLinear model for all datasets and all experiments. Batch size are chosen from (1024, 2048, 4096, 8192, 16384), ,, learning rate chosen from {0.003, 0.001) and weight decay is chosen from {0.0005, 0.005, 0.001, 0.0001 ). and we decay the learning rate with factor of (0.96, 0.97, 0.98) for at most 100 epochs with patience most 30. For larger datasets including Electricity, Traffic and weather the learning rate and weight decay barely make differences to the result. For relatively small datasets includgin ETTh//ETTh2, CTTm//ETTm2 and Exchange, larger weight decay may improve performance.
E.4.2 MLP
We use types of MIPs.. For the data scaling (including the HorizonXDatasee] experiments, the gated :1]: hidden(x) = Wi (I)) @ (ggmoidW2(2z)) where @ is the element-wise product improve the capability of our model. For the width scaling (and the HorizonXWidth) experiments, we use plain MLP with leaky-relu as activation function.
Gated MLP is used on Traffic, Weather and Electricity dataset. On these datasets use 3 - layer mlp with model dimension 768 and hidden dimension 2 768. Learning rate is set to 0.001 and weight decay is set to {0.00001, 0.0005).
For single layer mlp that we use to experiment the effect of model size scaling, we us the relu activation function and the model width denotes the hidden dimension of the single hidden layer. For different datasets, the learning rate is set (0.003} and the weight decay is set (0.0005).
Batchsize is set to 4096 or 8192 for these cases.
E.4.3 iTransformer
We follow the original codes provided by iTransformer[10) 10
For dataset size scaling and (Horizon X Dataset Size) we use iTransformer encoder blocks of dimension 512 for Traffic, Weather and Electricity. For other smaller datasets, we use blocks of dimension 192.
For Model size scaling case we use 2 iTransformer encoder blocks for large datasets including Traffic, Weather and ECL, and 1 block for smaller datasets. The width indicates the model dimension as well as the feedforward dimension.
Learning rate is set to 0.003 for small datasets and 0.0003 for large datasets. We decay the learning rate with a factor of 0.97 or 0.98 every epoch, and run the training procedure for at most 100 epochs with patience 10 for small datasets, and 40 epochs with patience 10 for larger datasets including Traffic, Weather and Electricity. Batch size is chosen from {12, 20, 32), mainly to fit into the gpu memory.
20

---

E.4.4 ModernTCN
We follow the original codes provided by ModernTCC[[OO, We use the recommended settings given by the authors of ModemnTCN except for the horizon, the channel dimension and the training procedure. Since MCdernTCN has many hyperparameters that can be flexibly tuned, we do include other same hyperparameter here, and please refer to the original paper and cods for ModernTCN for a more detailed description. Here we only include the different hyper-parameters.
For Dataset Size Scaling and (Horizon X Dataset Size) experiments, we only modify the look back horizon. For training procedure, we use learning rate in {0.00005, 0.0001 We train for 100 epochs with patience 20 for small datasets and 40 epochs with patience 10 for larger datasets.
For Model Size Scaling and (Horizon X Model Size) experiments, we only modify the look back horizon as well as the chanenl dimension and the depth-wise dimensions in the code.Again the learning rate is chosen from {0.00005, 0.0001). and we train for 100 epochs with patience 20 for small datasets and 40 epochs with patience 10 for larger datasets. Batchsize is chosen from {16, 64) following the original hyperparameter settings given by the author of ModernTCN.
F Downsampling May Improve Performance: power of patch, low-pass-filter, and downsampling
Previous work has proven the success of patches 7 and low-pass-filcs B for time series prediction. These can be explained with our theory by making the assumption that high frequency features are the most unimportant ones: thus by filtering up high frequency features we make the model visible to the most important dimensions of the intrinsic space.
From more detailed theoretical perspective, downsampling can be viewed as projection subspace in the intrinsic space and thus has a similar effect as decreasing the horizon. Experimental results (as shown in experiments in previous works like FITS PatchTST[Z] and in Figure 7 show that the projected subspace of higher frequency tends to fall on the large-eigenvalue directions, or the 'invisible" dimensions masked by the projection tends to the more unimportant ones. Although the precise effect of down sampling is unknown or may need further assumptions and methods for precise consideration, it is acceptable that we may approximate the effect of down sampling to be similar to a projection to the first deff < dy (H) dimensions in the intrinsic space. After making this assumption, the overall loss could expressed as:
losSnew L(dery)) dels < dH
Hence, if the original intrinsic dimension is larger than (local) optimal d. indicating ôloss/Od > O. reducing d from (H) to defs would help reduce loss. Otherwise if dy d7(H) is already smaller than (local) optimal d', meaning öloss/Od < O, we would expect no performance improvement from reducing d from (H) to deff-
To validate this idea, We do similar experiment here, that to do down-sampling and to use the downsampled sequence as input sequence. We conduct experiments Traffic dataset and Weather dataset with a lookback horizon set be 336 and a prediction length set to be 192. We use a 4-layer 512-dim gated mlp for traffic dataset and 2-layer 192-dim gated mip for weather dataset. We can see that doing downsampling would improve the performance of the weather dataset and the optimal downsample ratio also changes with different training dataset size. The explanation for this is similar to how we explain the optimal horizon growth with the growth of the training dataset. More data in the training dataset would lead to less noise on higher dimensions, thus seeing more of these dimensions (or filtering out fewer dimensions with downsampling with longer interpolate length) can improve performance.
Meanwhile, for the traffic dataset, downsampling may hurt performance. This is because Traffic has large amount of training data, and 336 is smaller than the optimal horizon. At this all dimensions in the intrinsic space are not dominated by noise, hence filtering them out would hurt performance, and would expect the performance to be improved with the increment of interpolate length.
21

---

Traffic
Weather
160
0.2
32
037
0.
Interpo hoilen
Figure 7: MSE loss v.s. interpolate-to- to-len (length after downsampling) for Traffic and Weather dataset for a certain amount of training data.
G PCA settings
We use PCA obtain eigen values that we use approximate the importance of features in the intrinsic space. We conduct PCA at (all or part of the) training samples (after instance normalization), mainly for the Chanpee-Indepeendntt case. For the PCA used in Figure and Figure &he the horizon set to 2000 for both datasets and we perform PCA on all the training samples. We provide more PCA results in Figure if we approximately view the PCA eig-val as the feature importance, then they do follow Zip-f law approximately.
traffic, 2000
weather, L 2000
Exchange. 2000
Exprraat sdrt P" 203
16-3
363
163
1e4
ECL, 2000
ETThL, 2000
ETTml 2000
FOger ponskke Seee 1-2-
222755eeeen 1330
1i 1222 172 77--886
Figure 8: PCA results for I. 2000 under Channel-Independeenn setting some datasets.
We further conduct PCA on intermediate vectors of the iTransformer(B S model and obtain results, further validating the Zip-f assumption our paper for intrinsic space for Non-Linear Channel- Dependent Multivariable cases, as shown in Figure
H Foundation Models for Mixed Datasets
Our theory holds without the need for any modification long as the dataset itself follows a Zip-f distribution the intrinsic space. This assumption sort of natural given that Zip-f law a natural distribution, and previous datasets we examined (like Traffic, ETT, etc) to follow the Zip-f law are composed of smaller datasets. Moreover, we provide further analysis from Theoretical and Experimental perspective why it holds for mixed datasets.
22

---

Tertinas S0INEES Zart The CE
Figure 9: PCA obtained with iTransformer features. Zip-f law fitting receives p 0.05 in all cases, Note that it is hard for deep-learning methods (like iTransformer) to learn feature with small covariance very well, may be hard to learn features in exchange dataset well (in which feature degrades very fast), hence the seemingly not-fitting figure for exchange is more likely caused by under-training of deep learning models.
H.1 Zip-f distribution for Mixed Datasets: a toy model
Theoretically, if large dataset is composed of sub-datasets of similar size each following the Zip-f law with degradation coefficient O1 < 02 <.. < Os, each with size S; and follows Zip-f law: Auj- A:/ja, where Äı represents the j-th largest eigenvalue of the i-th dataset. Suppose the intrinsic dimensions are orthogonal with each other (hence PCA components are orthogonal). A simple assumption is that the new intrinsic space is direct product of the old intrinsic spaces, hence eigenvalues are the union of old eigenvalues. For which, an eigenvalue of value S should be the Malttota--argge eigen value, which:
Σ idx; E(A./S)
idvtotal
When S is small (or correspondingly, when iddrotal is large) this sum is dominated by small as, and in limitation cases the sum is dominated by an term: idx so (Ai/S) +C, which is approximately a Zip-f distribution.
H.2 Experiment results for Zip-f distribution for Mixed Datasets
Experimentally, we use the Mixed Dataset of Traffic, Weather, ETTh1, ETTh2, ETTm1, ETTm2, exchange and ECL to train diate vector before the decoder layer as a feature vector to do PCA analysis. We found that the result follows Zip-f distribution for higher-order components, as shown in Figure I0
2-layer 512-dimension MLP and use the Interme-
I More comparison with other formulas
We compare AiC and BiC value of our proposed formula f(x) = A + B/x" with other possible formulas, on MCN, iTrransformer and MLP or Linear models:
(x) A/a
92(x) A + B log(z)
ga(x)= = A + Ba + Ca.
23

---

Mixed Dataset. MLP iCharielindegenddett 512->-844
Z.p-f fit: rela coof rela_coef
dividend rank"alpha
Original dita
hited lis: povalue 0.521.
-12
legatanicng index)
Figure 10: PCA obtained with MLP features trained on Mixed Dataset. The 2-layer 512-dimension MLP is trained under Channel Independent setting on mixed dataset of Traffic, Weather, Exchange, ETThi, ETTh2, ETTm1, ETTm2 and ECL. The mixed feature shows a well-aligned Zip-f law for features of higher rankings.
AIC, BIC
Traffic
Weather
ETThi
ETTh2
-103.9,-103.4
-87.2,-87.0
-69.5,-69.6
-64.7,-65.3
gl
-95.3,-94.9
-79.1.-79.0
-60.6.-60.7
-45.6,44.0
g2
-94.6,-63.3
-71.1,-71.0
-59.7,559.8
45,4,-45.8
g3
-93.3,-94.9
-81.3,-83.1
-56.5,-66.7
43.11,43.8
Table 3: regression results for ModemnTCN
AiC, BiC
Traffic
Weather
ETTh1
ETTh2
-71.6,-70.7
-74.8,-74.6
-50.7,-50.9
-56.9,-56.7
gl
-66.7,-66.1
-73.1,-72.9
45.7,,45.8
-57.9,57.88
g2
-63.3,662.6
-71.1,-71.0
41.7,-41.8
-57.9,-57.7
g3
-54.9,-56.3
-70.1,-72.0
-38.9,-39.7
-56.4,-6.22
Table regression results for i'Transformer
AiC, BiC
Traffic
Weather
ETTh1
ETTh2
f
-91.9,991.3
-91.5,991.7
-89.1,-8.88
-62.8,-62.6
gl
-67.1,-66.7
-83.1,-83.2
-67.5,-67.4
-61.1,-60,9
g2
-66.0,-55.6
-82.1,-82.3
-65.6,-65.5
-59.9,-59.8
g3
-60.1,-61.2
-81.3,-83.4
-89.1,-88.8
-60.1,-59.9
Table 5: regression results for MLP or Linear models
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification; We made claims the abstract and introduction part that our main contribution includes: (1) proposes a theory (please refer to Section 3 for scaling law for time series forecasting, taking into account impact of dataset size, model size and look back horizon and (2) conduct experiments (please refer to Section 4 to verify the validity of scaling law on dataset size and model complexity, as well as to validate our theoretical framework on time series forecasting models. This aligns with claims and our paper's contribution.

---

Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss our limitation in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We gave thorough assumptions and clear proofs. More specifically, formulate the intrinsic space in section 3.2. and calculate the loss in the rest of section
3.2
Then analyze the optimal horizon section 3.3 The detailed proof is shown from
Appendix C
Guidelines:
The answer NA means that the paper does include theoretical results. the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
25

---

All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We give details for our experiments at Appendix E logs in the supplementary materials.
We also provide code and
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: we describe the hyperparameters used in Appendix F. and we provide code in our supplementary material. We will make our code publicly available after refactoring.
Guidelines:
26

---

The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips.cc.
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https Cww.deuumisssiooooooooooooooo
7/nips. //wpps.ccppub/////// wttp//pwwlcggcccccceeeeoooo
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if' applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We give different dataset configs and corresponding hyperparameters
Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report and analyze the error in our experiments. Please refer to Appendix E for more details about errors.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
27

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative emror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide information about implementation framework and compute re- sources at Appendix E.4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We follow NeurlPS Code of Bthics and give clear details.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work proposes a theory for scaling behaviors for time series forecasting tasks and experimentally validates the proposed theory. Though we hope our work may inspire work for time series forecasting, this work alone is foundational research that sees no social impact on its own.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
28

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not release new models or datasets, hence the paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use previously proposed models and datasets, and include license for existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
29

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We use publicly available models and datasets and not release new assets including models datasets. We mainly conduct experiments based code bases of previous work.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowd sourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowd sourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
30

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "D5exUMcybp", "reviewer": "Reviewer_urDc", "summary": "The paper introduces a theoretical framework for scaling laws in time series forecasting, focusing on the impact of dataset size, model complexity, and look-back horizon on model performance. \n\nThe authors make two main contributions: \nThey propose a novel theory that explains scaling behaviors in time series forecasting from an intrinsic space perspective, emphasizing the critical role of the look-back horizon. The theory identifies an optimal horizon and demonstrates that beyond this point, performance degrades due to inherent limitations of dataset size. Conducting comprehensive empirical investigations to verify scaling behaviors related to dataset size, model size, and look-back horizon across various models and datasets. These experiments establish a robust scaling law for time series forecasting and validate the proposed theoretical framework.\n\nThe paper examines how the optimal look-back horizon changes with different amounts of training data and model sizes. It also explores the relationship between channel-dependent and channel-independent models in the context of the proposed theory.\nThe authors test their theoretical predictions using a range of popular time series forecasting models (including linear models, MLPs, iTransformer, and ModernTCN) on multiple datasets (such as ETTh1/h2, ETTm1/m2, Traffic, Weather, and Exchange). Their experiments demonstrate the existence of scaling laws in time series forecasting and provide evidence supporting their theoretical framework, particularly regarding the influence of the look-back horizon.", "questions": "It would be interesting to understand how the proposed theory extends to multi-variate time series forecasting and if there are any specific considerations for such cases. \n\nThe paper mentions that downsampling can sometimes improve performance, and it would be valuable for the authors to elaborate on how this relates to their theory of optimal horizon. \n\nLastly, given the changing nature of many real-world time series, it would be beneficial to know how sensitive the optimal horizon is to changes in the data distribution over time (concept drift) and whether the theory accounts for non-stationary time series.", "limitations": "The authors demonstrate transparency by addressing limitations. They acknowledge that their theory primarily covers time series forecasting and may not generalize to other time series tasks. They also note that their experiments are conducted on datasets smaller than some recently proposed large datasets, and that the theory doesn't consider self-supervised or pretrained models. Additionally, they mention that the impact of prediction length on optimal horizon is not thoroughly explored. \n\nWhile the authors are commendably upfront about these limitations, they could potentially enhance this section by discussing implications for computational resources and energy consumption related to finding optimal horizons in practice. Given the theoretical nature of the work, the absence of a discussion on potential negative societal impacts may be appropriate, but a brief mention of this could further strengthen the paper's consideration of broader impacts.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper demonstrates several notable strengths. Its originality lies in introducing a novel theoretical framework for scaling laws in time series forecasting, particularly addressing the impact of look-back horizon, an aspect previously unexplored in scaling law theories. The quality of the work is evident in its combination of thorough theoretical analysis with extensive empirical validation across multiple models and datasets, including Linear, MLP, iTransformer, and ModernTCN models tested on various datasets such as ETTh1/h2, ETTm1/m2, Traffic, Weather, and Exchange. \n\nThe paper maintains clarity through clear mathematical derivations, well-defined assumptions for the theoretical framework, and presentation of experimental results with error bars and statistical analyses. The significance of this work is apparent in its potential to inform future model design and hyperparameter selection in time series forecasting, which could lead to performance improvements across various applications.", "weaknesses": "Despite its strengths, the paper has a few areas that could be improved. The authors acknowledge that their experiments are conducted on datasets smaller than some recently proposed large datasets, which may limit the generalizability of their findings. The theoretical framework relies on several simplifying assumptions about the intrinsic space properties, which may not always hold in practice. A more in-depth discussion on the implications of these assumptions could enhance the paper's robustness. Additionally, while the theory provides valuable insights, the paper could benefit from more concrete guidelines on how practitioners can apply these findings to improve their models, which would increase the practical impact of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer urDc,\n\nThank you for a detailed and inspiring review! We truly appreciate your feedback and concerns! Here are our responses:\n\n**1.How the proposed theory extends to multi-variate?**\n\nWhile our deduction is primarily written in a single-variable manner, it can be easily adapted to the multi-variable case. As mentioned in Section 3.2.1, we made the Information-preserving assumption:\n\nThere exists a mapping $\\phi$ from the original length-$L$ sequence space $\\mathcal{O}(L)$ to $\\mathcal{M}(L)$, along with an inverse mapping $\\phi^{-1}:\\mathcal{M}(L)\\to \\mathcal{O}(L)$ and a constant $e \\ll 1$ that is independent of $L$, such that for any $x \\in \\mathcal{O}(L)$, $\\|x - \\phi^{-1}(\\phi(x))\\|_2^2 \\leq e$.\n\nFor a certain time-series dataset, we assume it has a fixed multi-variable dimension $d_{\\text{multi}}$. By substituting $\\mathcal{O}(L)$ with $\\mathcal{O}(L \\times d_{\\text{multi}})$, we observe the multi-variable format:\n\nThere exists a mapping $\\phi$ from the original length-$L$ dimension $d_{\\text{multi}}$ sequence space $\\mathcal{O}(L \\times d_{\\text{multi}})$ to $\\mathcal{M}(L)$, along with an inverse mapping $\\phi^{-1}:\\mathcal{M}(L) \\to \\mathcal{O}(L \\times d_{\\text{multi}})$ and a constant $e \\ll 1$ that is independent of $L$, such that for any $x \\in \\mathcal{O}(L \\times d_{\\text{multi}})$, $\\|x - \\phi^{-1}(\\phi(x))\\|_2^2 \\leq e$.\n\nAnother assumption to be checked is the Zip-f assumption on different dimensions of the intrinsic space for the multi-variable case. In **Figure 8 in Appendix G in our original submission**, the single-variable case is validated by conducting PCA on Channel-Independent time series. We further use the intermediate vector of iTransformer as the feature vector in the intrinsic space of multivariable time series and conduct PCA on them. This can be found in **Figure 1 in our additional page of PDF uploaded to our “global” rebuttal response**, which validates the Zip-f assumption of the ** Non-Linear Multivariable** case.\n\nOther deductions still hold for multivariate cases (up to a constant factor). Hence our assumptions and deduction can be extended to the multivariate cases.\n\n**2. Down-sampling and theory of optimal horizon**\n\nDownsampling can be viewed as a projection to a subspace in the intrinsic space and thus has a similar effect as decreasing the horizon. Experimental results (as shown in experiments in previous works like FITS, PatchTST and in our work) show that the projected subspace of higher frequency tends to fall on the large-eigenvalue directions, or the `invisible' dimensions masked by the projection tends to be the more unimportant ones. Although the precise effect of downsampling is unknown or may need further assumptions and methods for precise consideration, it is acceptable that we may approximate the effect of downsampling to be similar to a projection to the first $d_{eff}<d_I(H)$ dimensions in the intrinsic space. The overall loss could be then expressed as:\n\n$$\nloss_{new} = L(d_{eff})\\text{ where $d_{eff} < d_I(H)$.}\n$$\n\nHence, if the original intrinsic dimension is larger than (local) optimal $d$, $\\partial loss/\\partial d>0$, hence reducing $d$ from $d_I(H)$ to $d_{eff}$ would help reduce loss. Otherwise if $d_I(H)$ is already smaller than (local) optimal $d^*$, meaning $\\partial loss/\\partial d<0$, we would expect no performance improvement from reducing $d$ from $d_I(H)$ to $d_{eff}$.\n\n\n**3. How sensitive the optimal horizon is to changes in data distribution over time? Does the theory accounts for non-stationary time series?**\n\nThis is indeed a very important question in time series forecasting. In our work we mainly consider the case where training and testing distributions approximately follow the same distribution, because this is a basic case and we can make the gap between training set and test set smaller with normalization methods (like RevIN) or online-training methods. The former methods are included in our theory; however, it is challenging for our theory to model the online-training or test-time-training methods. This is partially because these methods may employ different training and optimization strategies that are significantly different in modeling.\n\nFor traditional deep-learning methods (i.e. training on training set, test on test set, no hyper-parameter tuning, etc on test set, do not use timestamp as a parameter of model input, which is the usual case for time series forecasting models like Linear, iTransformer, ModernTCN, etc), if we assume the test set is fixed and we sample training set each time for training. If we assume sampling training set has a distribution difference compared to the test set that has expectation $0$ but a certain variance, then this shift can be modeled into a constant loss term in our theory model, thus making no effect on hyper-parameter choosing (on expectation of training-set sampling).\n\nHowever, when considering more complex scenarios where distribution differences may be correlated with certain observable parameters (e.g. timestamps), or when online training methods are employed, analyzing the effect of the look-back horizon becomes more intricate. In such cases, both the exact look-back horizon (the length of past data used as input to the current model) and the implied look-back horizon (the length of past data that continues to influence the model during online training) may impact the model's performance. This likely involves a distribution shift in the intrinsic space over time, necessitating distinct considerations for these two horizons. We believe addressing this complexity would be a valuable direction for future research.\n\n\n**4. Others**\n\nThank you for the constructive advice! We would add more detailed description about computational resources and energy consumption in our experiments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lWN0ydDWyv", "reviewer": "Reviewer_Vwbb", "summary": "This paper proposes a theory that explains why complex models do not necessarily outperform simpler models even under the presence of larger amount of data, and why longer inputs hurt performance of some models. The authors consider the data size, model complexity, data granularity and the look-back horizon. Together with the theory here proposed the authors present empirical evidence verifying the theoretical insights.", "questions": "How do the presented result hold for the case of pretrained models, or any other setting where models are trained on multiple datasets? Since the recent developments of large models for time series this kind of results would be of high value.\n\nIt would be nice if the authors could provide more details on the derivations presented in 3.2.2. Although the main sketch is there is rather unclear how this is done.", "limitations": "The authors clearly state that the analysis on pretrained models remains for future work. This is a valid statement from the authors as one can assume that novel challenges arrive for this setting, and arguably the notion of scaling law for single-dataset models is not fully answered yet, being this work an interest step towards this topic.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The authors present an analysis of scaling laws for time series forecasting by adding components that are characteristic of time series forecasting, i.e. the fact that a limited amount of information of the past is provided to do inference/forecasts. Whereas in other domains it is usually assumed that the larger the look-back window then the better the estimations, in time series forecasting this is not always the case. Based on this the authors proceed to present a framework based on intrinsic spaces where slides of time series can be represented as vectors - they consider a suitable space for the look-back window and the forecasted values.\n\nPerhaps the most relevant contribution of the authors is the theoretical framework considered, together with numerical verifications.", "weaknesses": "- The authors present a theoretical framework that is interesting but lacks of precision. The authors constantly derive equations but at the end it is unclear what are the hypothesis required to be sure that the corresponding results hold. \n\n- The authors present results in different models, but these results seem to be done on models trained on individual datasets, and hence it is unclear if the results here presented hold for the case of pretrained models, which are pretrained in multiple time series datasets.\n\n- In line 193 the authors claim that their results are well approximated by the laws here derived. But this is completely unclear a the derived laws are not plotted together with the empirical ones per model. This leaves the verification task completely to the readers. This further makes it more challenging to check the correctness of the presented results.\n\n- Appendix A.1: The proof presented is not a formal proof but a sketch of a proof.\n\n- Typos:\n- - line 113: “ and thus We can scale” [there is a wrong capital letter]\n- - line 122 “s[h_1] to s_[h_2-1]” there is a wrong positioning of a bracket", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Vwbb,\n\nThank you for a detailed and inspiring review! We truly appreciate your concerns and adivce! Here are our responses:\n\n\n**1. Fitting experimental results with our derived formula**\n\nAs displayed in **Figure 1,2 and 5** in our original submission, the solid lines are the fitted lines of the theoretical formulas. We recognize that the presentation of these graphs may seem that the lines are obtained by connecting data points, hence we will add further captions to these figures to show that these lines are fitted lines in our next version of paper.\n\n\n**2. With respect to Pretrained Models or Large Mixed Datasets**\n\nThis is a very insightful question given that there have been works focusing on pretrained time series forecasting models with zero-shot forecasting abilities. However, the theory for scaling laws for pretraining and finetuning still remains a relatively unexplored area compared to the theory for scaling laws for single-dataset models, which is still a topic to study these days. The case of training on Mixed Datasets may be closer to our case which considers single-dataset.\n\nOur theory holds without the need for any modification as long as the dataset itself follows a Zip-f distribution in the intrinsic space. This assumption is sort of natural given that Zip-f law is a natural distribution, whereas we give two analyses from Theoretical and Experimental correspondingly on why it holds for mixed, large datasets.\n\n**Theoretically**, if a large dataset is composed of $s$l subdatasets of similar size each following the Zip-f law with degradation coefficient $\\alpha_1 < \\alpha_2 < \\ldots < \\alpha_s$, each with size $S_i$ and follows Zip-f law: $\\lambda_{ij}= A_i/j^{\\alpha_i}$, where $\\lambda_{ij}$ represents the $j$-th eigenvalue of the $i$-th dataset. Suppose the intrinsic dimensions are orthogonal with each other (hence the PCA components are orthogonal). A simple assumption could be the new intrinsic space is a direct product of the old intrinsic spaces, hence eigenvalues are the union of all the old eigenvalues. After which, an eigenvalue of value $S$ should be the $idx_{total}$-largest eigen value, in which:\n$$\nidx_{total} = \\sum_i(idx_{i}) = \\sum_i (A_i/S) ^{1/\\alpha_i}.\n$$\n\nWhen $S$ is small (or correspondingly, when $idx$ is relatively large) this sum is dominated by the small $\\alpha$s, and in limitation cases this sum is dominated by the $\\alpha_1$ term: $idx \\approx (A_1/S)^{1/\\alpha_1} + C$, which is approximately a Zip-f distribution.\n\n**Experimentally**, we use the Mixed Dataset of Traffic, Weather, ETTh1, ETTh2, ETTm1, ETTm2, exchange and ECL to train a Channel-Independent 2-layer 512-dimension MLP and use the Intermediate vector before the decoder layer as a feature vector to do PCA analysis. We found that the result follows a Zip-f distribution for higher-order components (**Please refer to Figure 2 in our Extra Page PDF**).\n\n(Actually **the individual datasets themselves (like Traffic, Weather, etc) are composed of data from different times, areas, etc, so they are actually composed of sub-datasets themselves** and they are showing the Zip-f distribution. This also indicates that larger datasets may have similar Zip-f distributions.)\n\nHence it would not be a bad assumption that Zip-f law still holds for datasets with mixed sub-datasets, and our theory, without major modification, could be applied to these cases.\n\n**3.More detailed proof for Section 3.2.2**\n\nYes, we would like to provide a more detailed proof here, which will be updated in the next version of our paper.\n\n$$\n\\begin{aligned}\nL&=E_{x\\sim\\mathcal M(H+S)}[(x[H:H+S]-m(x[0:H]))^2]\\\\\n\\end{aligned}\n$$\nLet $m^*$ denote the optimal Bayesian model, then it should satisfy:\n$$\nm^*(x[0:H])=E_{x\\sim\\mathcal M(H+S)}[x[H:H+S]|x[0:H]]\n$$\nThus:\n$$\n    L=E_{x\\\\sim\\\\mathcal M(H+S)}[(x[H:H+S]-m^*(x[0:H])+m^*(x[0:H]-m(x[0:H]))^2]\\\\\\\\\n$$\n$$\n    L=L_{sum}+L_{cross}, \\text{in which}:\n$$\n\n$$\nL_{sum} = E_{x\\\\sim\\\\mathcal M(H+S)}[(x[H:H+S]-m^*(x[0:H]))^2]+E_{x\\\\sim\\\\mathcal M(H)}[(m^*(x[0:H])-m(x[0:H]))^2]\n$$\nis the sum of squares\n\nand\n\n$$\nL_{cross} = 2\\*E_{x\\\\sim\\\\mathcal M\\(H+S)} [\\(x\\[H:H+S]-m^\\*\\(x[0:H]))\\*\\(m^*\\(x\\[0:H])-m\\(x\\[0:H]))]\n$$\nis the cross term.\n\nThe optimal model (Bayesian model) should satisfy:\n\n$$\nm^\\*(x[0:H]) = \\sum\\limits_{x[H:H+S]} P(x[H:H+S]\\mid x[0:H]) x[H:H+S].\n$$\n\nTherefore, it follows that: (note the conditional expectation of $x[H:H+S]-m^*(x[0:H])$ given $x[0:H]$ is $0$, which is from the definition of the Bayesian Model)\n\n$$\nL_{cross}= \\\\sum\\\\limits_x P(x[H:H+S]\\\\mid x[0:H])P(x[0:H])[(x[H:H+S]-m^\\*(x[0:H]))*(m^\\*(x[0:H])-m(x[0:H]))] =0.\n$$\n\nHence, the cross term should be zero, and the loss is a sum of the square terms: one is determined by the capability of the optimal Bayesian model (the Bayesian Error), and the other is about how well the model approximates the Bayesian model (the Approximation Error): $L=L_{cross}+L_{sum}=L_{sum}=L_{Bayesian}+L_{Approx}$.\n\n**4.Typos**\n\nThank you for pointing out these typos! We would check typos carefully and further polish our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eOvPcJqUNo", "reviewer": "Reviewer_SJaN", "summary": "This paper proposes a theory for scaling law in time series forecasting that accounts for dataset size, model complexity, and data granularity, with a particular focus on the look-back horizon. This paper empirically evaluates various models across diverse time series forecasting datasets to verify the validity of the scaling law and validate the theoretical framework, especially regarding the influence of the look-back horizon.", "questions": "See weakness", "limitations": "See weakness", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. this paper is complete with good organization.\n2. several theatrical analyses are provided for the justification.", "weaknesses": "1. the main argument is not novel. I think it is well-known that a larger horizon may not bring up a better performance in time series forecasting. In some cases, it can but the larger horizon would also bring up more noise that can hinder the forecasting,\n2. the contribution is kind of weak. I don't find some quite useful conclusions or contributions for time series forecasting. With the scaling laws the author verified, there is no further improvement of new models on state-of-the-art benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer SJaN:\n\nThank you for the review! We truly appreciate your ideas and questions. Here are our responses:\n\n1.**Novelty and contribution related to the 'Main Argument'**\n\n1.1. ‘Longer horizon gives worse performance’ **is an important fact that could bring insight to the community, and the impact of the look-back horizon is not yet well understood**. As presented in the Introduction part in our original submission, **in the past two years it has been a common practice for the time series community**[1, 2, 3, 4]  to try to prove the advantage of the proposed methods by claiming that these methods could benefit from improving look-back horizon; which **we show is not necessarily an accurate way to show the superiority of a certain model.**\n\n1.2. **Our main argument is not only that a longer horizon gives worse performance**. Actually, we have provided a theory to explain Scaling Laws in Time Series Forecasting, which pays special attention to the impact of input horizon. This is, actually, very different from proposing only that a long horizon would harm performance, as our theory analyzes the complex impact of different combinations of model width, dataset size and lookback horizon. We also summarize our contribution in the next point we would like to make.\n\n\n**2. Our Theoretical and Experimental Contribution**\n\nAs stated in our paper, we summarize our contribution as follows:\n\n2.1. We introduce a novel theoretical framework that elucidates scaling behaviors from an intrinsic space perspective, highlighting the critical influence of the look-back horizon on model performance.\n\n2.2. We conduct a comprehensive empirical investigation into the scaling behaviors of dataset size, model size, and look-back horizon across various models and datasets to validate the effectiveness of our claim.\n\nWe do agree that proposing novel architectures and approaches achieving state-of-the-art performance is very important for the development of Machine Learning, but we also think that providing insights into important questions, validating hypotheses and proposing related theories could also contribute to the community, **which may lead to potential improvement in performance of future models.** For example, in our original submission, we have demonstrated in **Appendix F** why down-sampling can improve performance.\n\nWe further clarify our contribution, novelty as well as papers cited in this rebuttal section in the global rebuttal section, which may also act as a reference for your constructive concerns. Please refer to the global rebuttal section for more details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nyef21MQeW", "reviewer": "Reviewer_oD7J", "summary": "This paper introduces scaling laws for DNN analysing MTS data, involving dataset sample size, model size, look-back horizon, dataset covariate size (dimension), and noise/uncertainty in the dataset. It relies on an axiomatization of the intrinsic information space (linearity, isomorphism) and bayesian optimal model (lipschitz-continuity) to derives an approximation of the Loss in different scenarios, and tests aspects of this laws in experiments.", "questions": "Since the paper relies on an intrinsic space formulation, did authors study the representation learned by the neural networks? For instance, using PCA and/or tSNE on intermediary vectors. Could we verify that the assumptions made in 3.2.1 are plausible in such a way?", "limitations": "Several limitations are identified in the paper: need for larger datasets, impact of prediction horizon (fixed in this work), supervised forecasting task specific analysis.\n\nI would also add limitation/concern of mine: the evaluation datasets come from a same MTS benchmarking effort, and current DNN structures where built to optimize their performance on these datasets (among others). It is unknow if this induces bias compared to datasets generally not tested during standard benchmarking. It could happen that our current TS DNN models are overoptimized for the common benchmarking datasets. Optimally, the power scaling law should involve a few datasets not typically evaluated in benchmarking models.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper enters the difficult discussion about power-laws in DNN to the time series forecasting field, with a clear approach motivated by mathematical assumptions on what time series intrinsic information should respect. Linearity with respect to horizon especially. The identification of the influence of model overparametrization is explicited. While I am not entirely clear on which approaches are completely new and which other are inspired by scaling law studies of other data types, I believe that the resulting theoretical laws are novel and take into account the particular of TS data.\n\nExperiments cover several important parameters in the laws, and show matching results with the derived scaling laws.", "weaknesses": "It is hard to understand where the assumptions on the intrinsic space (3.2.1) come from. Especially lipschitz-continuity and space quasi-isomorphism. Are those properties 1) required for results to hold? 2) observed in practice in meaningful representations of (multivariate) time series data? 3) Are all the described properties completely new, or inspired from other works on Deep Neural Networks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer oD7J,\n\n\nThank you for the detailed and constructive review! We truly appreciate your reviews and questions. Here are our responses:\n\n**1. Theory assumptions**\n\nLet us give a sketch of our proof, where our assumptions are marked as **bold text**:\n\nSuppose the true sample is $x[-H:S]$, i.e. we are going to predict $x[0:S]$ from $x[-H:0]$. Now we have a model $m$ that predicts the corresponding intrinsic vector $\\phi(x[0:S]) \\in \\mathcal{M}(S)$ with error bounded by $e_i$, then by **assumptions 1, 2** we can verify that the error in the original space is bounded by:\n$$\n\\alpha e_i - e - e' \\leq L_{ori}\\leq \\alpha e_i + e + e'\n$$\nHence, analyzing the loss in the intrinsic space would derive the same result, which justifies our task formulation.\n\nBy **assumption 3**, we may partition the intrinsic space uniformly to get a lower bound of the loss.\n\nWe can first decompose the total loss into two components, i.e. Bayesian loss and approximation loss (**section 3.2.2**). Then the Bayesian loss could be written as:\n\n$$\nL_{Bayesian} \\leq (1-\\eta)K_1^2E[var(P^{-1}[\\infty,H]\\(x))] + \\eta \\cdot tr\\(\\Sigma_S)\n$$\n\nWhere the first term is a projection term, and the second is a noise term which almost makes no effect on our analysis. **By assumptions 4 and 5**, the relationship of a sequence with its subsequences is similar to linear projections to the subspace with the minimal eigenvalues if the horizon is large, thus the projection term could be bounded with the eigenvalues of the distribution, ensuring the Bayesian loss is bounded. (**section 3.2.3**)\n\nConsidering the approximation loss. If we have sufficient data, the approximation loss should come from two sources, the intrinsic noise in the data and the effect of unseen dimensions in the intrinsic space. The former is assumed to be uniform and thus can be easily calculated, while the latter should be calculated with help from **assumptions 4 and 5**, similar to the deduction in Bayesian loss. For the scenario where data is scarce, we don't need the assumptions and the approximation loss basically depends on the distance of a test sample to its nearest training sample in the intrinsic space. (**section 3.2.4**)\n\nThen we can combine the two loss components and analyze the optimal horizon or other properties. (**section 3.3**)\n\n**2.PCA on intermediate vectors**\n\nThank you for the constructive advice! We obtain intermediate vector representations for multi-variate time series datasets and conduct PCA experiments on them, further validating that these time series datasets do follow a Zip-f distribution with respect to their features. Please refer to **Figure 1 in our additional page of PDF uploaded to our “global” rebuttal response**, in which we show the results. We observe that such a deep-learning method may face under-training issues, thus causing possible uncertainty in evaluating high-rank features with a limited amount of training data, hence this PCA result could complement each other with the PCA result obtained by directly applying PCA on raw input sequence (under Channel-Independent and RevIN settings) provided in **Figure 8 in Appendix G in our original submission**.\n\n**3.TS DNN models are overoptimized for the common benchmarking datasets.**\n\nIt is indeed possible that some DNNs have structures overfitting several datasets which may not generalize beyond them. Here we mainly choose models that are designed with simplicity (e.g. linear models, iTransformer models which are basically transformers, ModernTCN models which are composed of time-series-optimized convolutional layers) and the datasets are actually quite different from each other in size (e.g. 1000x difference for data sample number) and in feature degradation (e.g. please refer to PCA results for exchange and other datasets), so we think the power law we observe here should be general, at least to a certain extent. However, we do appreciate the important point you have mentioned. This concern may extend beyond the field of time series forecasting, and we will include a description of it in the next version of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WjXIqj0qKH", "reviewer": "Reviewer_7rY6", "summary": "The authors propose a set of scaling laws on time series based relating model size, data size, and the \"history dependence\" of the time series. They end with a power law which they then try on different datasets.", "questions": "Check weaknesses. Also, how would the authors' results relate to the double-descent curve? This would put the two cases in context.\nAlso, their scaling law seems to be tied to ever-decreasing loses", "limitations": "They should really address the limitations of their assumptions and discuss when are they valid.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "The topic is interesting and it would be a good thing to study.", "weaknesses": "The presentation is really bad and it makes the understanding difficult. The theoretical results have assumptions that are sometimes not properly stated as assumptions, and sometimes hard to justify.\n\nSome of the theoretical preliminaries are either questionable, or badly explained. If this is solved I might look into the actual derivations more in detail:\n- Quasi-isometric assumption: it seems to imply that the mapping is either the identity or close to it. Imagine you have a language where there are two synonyms; then the only way in which the equation holds is when alpha is very small. Which is fine, but then the value of alpha can be arbitrarly small! They should justify when this is valid.\n- A similar limitation appears in assumptions 4 and 5, which seem to indicate that the data must follow some markovian structure. And I can definitely find cases where they would not apply: imagine a language with X letters forming Y words of length Z. If Y<<X^Z (which almost always happens) when L<Z you could easily end up with the rank of the covariance matrix for sequences (of length less than Z) being higher than the rank of words. In more practical settings, there are languages (ex: german) where the sense of a word is tied to a word that happens much later in the sentence (in german, verbs with preposition when there is a nebensatz), which would relate to the toy example I mention before, because the rank of the covariance decreases as the length increases.\n- In line 132 they assume that the data follows Zipf's law. This is a big assumption, and while it could apply to languages, it is hard to argue that it applies to every time series (and see Clauset et al. \"Power-law distributions in empirical data\"). They should at least mention why they think it applies to the datasets they are using.\n- In sec 3.2.4 they assume that the model partitions the space uniformly, and claim that this is the worst case. It is not clear why would that be the case, and don't seem to actually analyze other scenarios later on (if they do, refer to it on the same sentence). \n\nI haven't checked this in detail, but it seems to me that with assumptions 1 to 3 the model is either linear or very similar to it, and by 4 and 5 the data might need to be implicitly autoregressive. Taking those two together and assuming that the data has a power law distribution, it seems almost necessary that the loss or model size would follow the laws that they propose. \n\nFor the experiments:\n- From looking at Fig 1, what I could conclude is that different datasets/models follow different distributions.  \n- I think that from the fit is very hard to make any proper justification. MOst of the trends could be fit by some exponential or a simple polynomial, so they should compare to some sensible alternatives, and use some valid statistic criterion (akaike, for example) to compare.\n- In the same line of thought, fitting on a logscale is often hard, and I don't see how they did the fit. The fit should be more properly explained (and likely done, check http://bactra.org/weblog/491.html for example)\n- Fig. 2, from looking at it I would expect the double-descent to relate to those results. \n- I did not see in the theory any indication that the losses could scale negatively (as in some cases of Fig. 2). THis seems to be important, as some data has such trend. If their model is not suposed to overfit, this is a crucial assumption?\n- In line 85, a less relevant point: \"[33] shows that the average intrinsic dimension for a time series sequence should converge to a finite value, which indicates that the intrinsic dimension of a sequence should be proportional to its length.\" This sentence is wrong as far as I can tell: if the average intrinsic dimension converges to a finite value, it is not proportional to its length. For example, imagine the sequence 01010101010.... irrespective of the length, the intrinsic dimension is fixed. \n\nThe notation is also problematic. They use L for both length (line 100) and loss (Eq. in line 144), and sometimes they just write \"loss\" instead of L.\n\n I pointed out a few problems with the writing. There are more, but at this point it's on the authors to fix them:\n- Lines 21-22 Neural Nets utilize different model architectures, including FFN-based, Transformer-based\nand Convolution-based neural nets have been proposed. \n- Sometimes they refer to \"scaling laws\" in plural, and sometimes as \"the Scaling Law\" in singular as if there was a single one.\n- Line 67: \"There have been many works analyzing on different mathematical properties \"\n- Line 1667: if the data samples are \"sparse\". I assume they mean that there are few data points.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 7rY6,\n\nThank you for a detailed review! We truly appreciate your reviews and questions, especially for our theories, which aim our theory towards an omnipotent theory. We would like to briefly summarize your concerns, and our rebuttal and then expand upon this summary.\n\n**1.Theory Assumptions**\n\n1.1. Your view: Current Assumptions lead to close-to-linear relationships.\n\nResponse:\n\n1.1.1.Our current assumptions is not equivalent to linear, and **we can further weaken our assumptions while deriving similar results**, (which would be more complicated a process hence we did not choose to do so in our original submission, **which is presented later in this rebuttal**).\n\n1.1.2. We chose stronger assumptions for simplicity of derivation, which may lead to closer to linear models. We would like to mention that although counter-intuitively, **linear models are indeed important models in time series forecasting**, represented by a series of important experimental and theoretical works for time series[1,5,6].\n\n**We make these assumptions, not to include every possible scenario, but to focus on the most relevant properties related to the research problem.**\n\n1.2. Your view: Current assumptions lead to autoregressive behavior.\n\nResponse:\n\n1.2.1. Our theory utilizes Conditioned Expectation, which **actually still holds even in cases you refer to as Nebensatz**. We will use the exact example of Germany to show how it works.\n\n1.2.2. **The task of Time Series Forecasting itself has assumed time series follows at least partially autoregressive behaviors**, and the success of data-driven methods has validated this assumption.\n\n1.3. Your view: Zip-f is too strong an assumption.\n\nResponse:\n\n1.3.1. In our paper we have proven it under CI case experimentally using PCA, presented in **Figure 4 and Figure 8 in the original submission**.\n\n1.3.2. We further conduct PCA on intermediate vectors with iTransformer, which represents the CD case. We appreciate your advice, hence we use p-value to show fitness. Please refer to **Figure 1 in Extra Page PDF**.\n\n1.4. Your view: The claim in line 85 (the intrinsic dimension of a sequence is proportional to its length) is wrong.\n\nOur response:\n\n1.4.1. This is an **empirical result** of previous work[7] for chaotic systems, which can not be denied by manually constructed counter-examples. (E.g. It does not hold for one to claim Zipf-law for vocabularies in NLP being wrong by constructing counter-examples like '\\n'.join(['I like {}'.format(n) for n in ALL_NOUNS]).)\n\n1.5. Notions and Typos:\n\nThank you for pointing these out! We would check them and further polish our paper.\n\n**2. Double-descent curve and ever-decreasing loss.**\n\nWe have never claimed that our theory covers all aspects and all phenomena in time series forecasting tasks. What we are focusing on theoretically is to explain the impact of look-back horizon on scaling behaviors (as well as dataset size and model size), which is not yet well-studied. In Figure 2 of our original submission, we posited that the observed increase in loss could be attributed to overfitting. We further conduct experiments to validate the impact of dataset size, model size and look-back horizon on scaling behaviors for time series forecasting, which was not validated before. (Please refer to global rebuttal).\n\n**More details.**\n\n**1.1.1 Weaker Assumptions**\n\nThe **Quasi-isometric** assumption can be replaced with this **Inverse Lipschitz** assumption: $\\\\phi^{-1}$ would be $K_I$-Lipschitz under L2 norm. That is:\n$$\n\\\\forall x,y \\\\in \\\\mathcal{M}(L), \\\\| \\\\phi^{-1}(x)-\\\\phi^{-1}(y)\\\\|_2 \\\\leq K_I\\\\| x-y\\\\|_2.\n$$\n\nWe further make two assumptions:\n\n**Causality**. \nWe assume existing an optimal model $F[S]$ to predict the next $S$ frames given previous $h\\rightarrow\\infty$ frames, so that the error only originates from the intrinsic noise. \n\n$$\n\\\\exists F[S]: \\\\mathcal{M}\\rightarrow \\\\mathcal{M}(S),  s.t. \\\\lim\\\\limits_{h\\\\rightarrow \\\\infty} \\\\mathbb{P}(y\\\\mid x_{-h:0}) = (1-\\\\eta)\\\\delta(F[S]\\(x_{-h:0})) + \\\\eta \\\\mathcal{N}(F[S]\\(x_{-h:0}), \\\\Sigma_S)\n$$\n\nWhere $\\\\eta$ stands for the noise ratio in the system and we use $\\\\mathcal{N}(\\\\mu, \\\\Sigma)$ to represent a normal distribution with mean $\\\\mu$ and covariance $\\\\Sigma$ here.\n\n**Uniform Sampling noise**: When drawing a single length-$L$ sample, we assume that the sampling noise is uniform in each direction in $\\\\mathcal{L}$,\n\n$$\n        \\mathbb{E}_{x[-H:S]\\in \\mathcal{O}(H+S)}[\\|\\phi^{-1}(m(y_i)) - \\phi^{-1}(y_o)\\|_2] \\leq K_I e_m\n$$\n\nAnd from assumption 1 the second term is bounded by $e(S)$. We can derive that:\n\n$$\n        L_{\\text{ori}} \\leq K_I e_m + e(S)\n$$\n\nwhich shows loss in original space is linearly bounded by loss in the intrinsic space. W.l.o.g we may only study loss in the intrinsic space. \n\n\nBecause this rebuttal section has character limitation, **we would like to present further deduction in detail later in the discussion period**.\n\n**1.2.1. How it works for future-dependency like Nebensatz.**\n\nTake two phrases as an example:\n\na.Ich wusste nicht, dass er Hamburger essen wollte.\n\nb.Ich wusste nicht, dass er Bier trinken wollte.\n\nIf P(essen | Ich wusste nicht, dass er) = P(trinken |Ich wusste nicht, dass er ) = 1/2, and P(Hamburger | Ich wusste nicht, dass er __ essen wollte) = P(Bier | Ich wusste nicht, dass er __ trinken wollte) = 1.\n\nIt is possible in practice that we do not know what comes next had we not seeing 'essen' or 'trinken' no matter how many words before that we see, but this uncertainty has already been modeled into Bayesian Loss for $M(\\infty)$ and would be absorbed in the term $\\\\eta* tr(\\\\Sigma_S)$ in our bound for $L_{Bayesian}$, which will be further absorbed into the term $\\\\eta \\\\sigma_M^2 S^2 d_I(S)$ **in the last formula on Page4 of our original Submission**, only adding a constant to the final loss.\n\nWe have listed papers cited above in the global rebuttal. They could possibly act as supplement to our reply for your detailed concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers,

Thank you for the detailed reviews! In this global Rebuttal section, we would like to further clarify **(1) Contribution and Novelty of our work**, **(2) contents in the Extra Page Pdf** (which mainly contains more experimental results validating Zip-f law for more complicated cases) and **(3) papers that are cited in our rebuttal** in more detail.

**1.Contribution and Novelty**

**1.1. We summarize our contribution as follows.**

1.1.1 Novel theoretical framework elucidates Scaling Laws from the perspective of look-back horizon, dataset size and model size, with a specific emphasis on the task of Time Series Forecasting. By focusing on Time Series, our theory provides an innovative approach for understanding Scaling Laws for Time Series Forecasting, advancing theoretical comprehension and providing inspiration for further  theoretical investigations in this area.

1.1.2 Comprehensive empirical investigation into the scaling behaviors of Time Series Forecasting. To our knowledge, we are **among the first** to carry out experiments to validate Scaling Laws for Time Series Forecasting. Notably, **no previous work** has validated scaling law on Time Series Forecasting that **examines the effect of different look-back horizon in detail**. Previous works for Scaling Laws discussing the impact of look-back horizon (or equivalently, context length) draws mainly focus on LLMs in NLP[8] which **exhibits differently compared to Time Series Forecasting tasks**. This work promises to bridge this gap and provide valuable insights into the scaling dynamics unique to Time Series Forecasting.

**1.2. Our theoretical and experimental findings have a novel and positive impact on the time series community.**

Besides the theory and experimental validation for Scaling Law for Time Series Forecasting considering look-back horizon, dataset size and model size, there are more potential positive impacts, two of which are listed as follows and there could be more:

1.2.1. **The time series community has been using 'benefit from increasing horizon' as a metric for 'better models' for at least 2 years[1,2,3,4], while our work shows that it may not be necessary for good models to always benefit from increasing horizon**. As shown in Section 4.2. in our original submission, longer look-back horizon may give worse test results for all models, it is not the case that better model should benefit from further extending look back horizon. Moreover, this behavior is actually very different from some behaviors observed in long-context large language models[8]. **Our work could provide insight about understanding look-back horizon for the Time Series community both theoretically and experimentally.**

1.2.2. Our theory could potentially explain the benefits and disadvantages of commonly used down-sampling blocks (including low-pass-filter[5],  Patching[9], etc.), as shown in Appendix F in our original submission.

**2.Extra Page Pdf**

We further conduct PCA on intermediate vectors of the iTransformer model and obtain results, **further validating the Zip-f assumption** in our paper for intrinsic space for **Non-Linear Channel-Dependent Multivariable cases**. Please refer to Figure 1 in our Extra Page Pdf for more details.

Meanwhile, to study the property of Mixed Datasets, we further conduct PCA on the features obtained by a simple MLP **trained on a Mixed Dataset of various time-series datasets** under Channel-Independent settings (note that different datasets have different number of multivariables) and **validate the Zipf-law**. Please refer to Figure 2 in our Extra Page Pdf for more details.

Please refer to our attached PDF for more details.

**3.Papers cited in our rebuttals (for this global rebuttal and reviewer-specific rebuttals):**

These papers are cited in our rebuttals and could possibly act as reference or supplement to our rebuttals:

1. A. Zeng, M. Chen, L. Zhang, and Q. Xu, “Are transformers effective for time series forecasting?” in AAAI 2022.

2. H. Wang, J. Peng, F. Huang, J. Wang, J. Chen, and Y. Xiao, “Micn: Multi-scale local and global context modeling for long-term series forecasting,” in ICLR 2023.

3. L. donghao and wang xue, “ModernTCN: A modern pure convolution structure for general time series analysis,” in ICLR 2024.

4. Y. Liu, T. Hu, H. Zhang, H. Wu, S. Wang, L. Ma, and M. Long, “itransformer: Inverted transformers are effective for time series forecasting,” in ICLR 2024.

5. Z. Xu, A. Zeng, and Q. Xu, “Fits: Modeling time series with 10k parameters,” in ICLR 2024.

6. William Toner and Luke Darlow, "An Analysis of Linear Time Series Forecasting Models", in ICML 2024.

7. T. M. Buzug, J. von Stamm, and G. Pfister, “Characterising experimental time series using local intrinsic dimension,” Physics Letters A, vol. 202, no. 2-3, pp. 183–190, 1995.

8. Wenhan Xiong and Jingyu Liu and Igor Molybog and Others, "Effective Long-Context Scaling of Foundation Models".

9. Y. Nie, N. H. Nguyen, P. Sinthong, and J. Kalagnanam, “A time series is worth 64 words: Long-term forecasting with transformers,” in ICLR, 2023.

Thank you again for your reviews, and we are looking forward to possible further discussions.

Meta-review: The paper introduces a novel theoretical framework for scaling laws in time series forecasting, addressing the relationship between model size, dataset size, and look-back horizon. The authors empirically validate their theory across a variety of datasets, demonstrating the practical applicability and potential of their approach to enhance forecasting accuracy in diverse scenarios.

**Strengths**
- The integration of scaling laws into the time series forecasting domain is an innovative contribution that fills a critical gap in the existing literature (Reviewers urDc, Vwbb).
- The empirical validation across multiple datasets and models further strengthens the credibility and generalizability of the proposed framework (Reviewer oD7J). 
- The theoretical underpinnings are well-grounded and offer a new perspective on optimizing model performance by adjusting the look-back horizon and model complexity (Reviewer 7rY6).

**Weaknesses**
- The theoretical framework, while innovative, relies on several assumptions that are not fully justified or clearly presented. This lack of clarity could limit the framework’s generalizability (Reviewer 7rY6, Reviewer Vwbb).
- The presentation of the paper is somewhat lacking, with certain mathematical derivations and assumptions not being sufficiently detailed or explained (Reviewers 7rY6, SJaN).
- The experiments, although thorough, are conducted on relatively small datasets, which may raise questions about the generalizability of the findings to larger, more complex datasets (Reviewer SJaN, Reviewer Vwbb).

Based on the comprehensive reviews and the authors’ rebuttal, the paper presents a novel contribution to time series forecasting.

The authors must improve the clarity of their theoretical framework and provide further justifications for their assumptions in the camera-ready version of this work if it is accepted. Regardless of the decision, these actions would strengthen the paper, and I hope to see them done.
Decision: Accept (poster)