Towards Neural Scaling Laws for Time Series Foundation Models
Authors: Qingren Yao
Chao-Han Huck Yang
Renhe Jiang
Yuxuan Liang
Ming Jin
Shirui Pan
Venue: ICLR 2025 Poster (2025)
Keywords: Time series
scaling law
foundation model
transformer
forecasting
Primary area: foundation or frontier models, including LLMs
TL;DR: This paper studies scaling laws for time series foundation models on both in-distribution (ID) and out-of-distribution (OOD) data.

Abstract: Scaling laws offer valuable insights into the design of time series foundation models (TSFMs). However, previous research has largely focused on the scaling laws of TSFMs for in-distribution (ID) data, leaving their out-of-distribution (OOD) scaling behavior and the influence of model architectures less explored. In this work, we examine two common TSFM architectures—encoder-only and decoder-only Transformers—and investigate their scaling behavior on both ID and OOD data. These models are trained and evaluated across varying parameter counts, compute budgets, and dataset sizes. Our experiments reveal that the log-likelihood loss of TSFMs exhibits similar scaling behavior in both OOD and ID settings. We further compare the scaling properties across different architectures, incorporating two state-of-the-art TSFMs as case studies, showing that model architecture plays a significant role in scaling. The encoder-only Transformers demonstrate better scalability than the decoder-only Transformers, while the architectural enhancements in the two advanced TSFMs primarily improve ID performance but reduce OOD scalability. While scaling up TSFMs is expected to drive performance breakthroughs, the lack of a comprehensive understanding of TSFM scaling laws has hindered the development of a robust framework to guide model scaling. We fill this gap in this work by synthesizing our findings and providing practical guidelines for designing and scaling larger TSFMs with enhanced model capabilities.

Published as a conference paper at ICLR 2025
TOWARDS NEURAL SCALING LAWS FOR TIME SERIES FOUNDATION MODELS
Qingren Yao Chao-Han Huck Yang", Renhe Jiang', Yuxuan Liang"; Ming Jin'; Shirui Pan'
Griffith University The Hong Kong University of Science and Technology (Guangzhou) NVVIDI Research "The University of Tokyo
ABSTRACT
Scaling laws offer valuable insights into the design of time series foundation mod- els (TSFMs). However. previous research has largely focused on the scaling laws TSFMs for in-distribution (ID) data, leaving their out-of-distribution (OOD) scaling behavior and the influence of model architectures less explored. In this work, we examine two common TSFM architectures encoder-only and decoder- only Transformers-andd investigate their scaling behavior both ID and OOD data. These models are trained and evaluated across varying parameter counts, compute budgets, and dataset sizes. Our experiments reveal that the negative log- likelihood of TSFMs exhibits similar scaling behavior in both OOD and ID set- tings. We further compare the scaling properties across different architectures, in- corporating two state-of-the-ard TSFMs as case studies, showing that model archi- tecture plays significant role in scaling. The encoder-only Transformers demon- strate better scalability than the decoder-only Transformers in ID data, while the architectural enhancements in the two advanced TSFMs primarily improve ID performance but reduce OOD scalability. While scaling up TSFMs is expected to drive performance breakthroughs, the lack of a comprehensive understanding of TSFM scaling laws has hindered the development of a robust framework to guide model scaling. We fill this gap this work by synthesizing our findings and pro- viding practical guidelines for designing and scaling larger TSFMs with enhanced model capabilities.
INTRODUCTION
Time series analysis is important piece of data mining, facilitating decision- and scientific inference across various domains (Zhang et al.] 2023). As an important analysis task, time series forecasting has long been studied and drives wide range of practical applications, from energy, climate and quantitative finance to urban computing and system management (Jin et al., 2023 Nie
al.) 2024 Wen al. 2024). Various methods have been proposed for this task, ranging from classical statistic models (Hyndman & Athanasopoulos/ 2013). bespoke dynamical models (Prado) 2020), to the more recent deep-learning based approaches (Wen al. 2022). Despite their compet- itive performance, the methods are typically designed for specific tasks. poor to generalize to other domains (Fan et al., [203]] Rasul al., (2023). Concurrently, we are witnessing paradigm shift in time series forecasting from task-specific models to universal models, with the emergence of time series foundation models (TSFMs). Timer (Liu al. 2024), Moirai (Woo et al., 2024), and more recently proposed Time-MoE (Shi et al., 2024b) show trends of scaling in both data volume and model size, aiming to achieve performance breakthroughs through more resource investment.
The neural scaling law (Kaplan al. 2020) quantitatively describes how model performance grows with the scaling of three basic training factors: model parameters, computational resources and training dataset size. Establishing such scaling laws is crucial for developing TSFMs, it pro- vides a framework for predicting expected performance gains, enabling the community to rationally allocate efforts toward key designs. The exploration on scaling laws for TSFMs is still in ini- tial stage; recent research has primarily focused on studying ID) scaling behavior (Edwards et al.
"Correspondence Y. Liang wyu.liang@outlook.coo and M. Jin <minginedu gmail.com.>

---

Published as a conference paper at ICLR 2025
2024 Shi al. 2024a) In practical applications. TSFMs primarily face challenges from unseen scenarios (Wang et al. 2024). where OOD forecasting capability is most critical. This raises an unresolved question: do neural scaling laws also apply predict onttro--istribbution forecasting performance? Moreover, various architectures of TSFMs have been arising, but they typically focus on performance improvement at specifie scales. No studies have investigated the scaling behaviors across different architectures, leaving a key question unanswered: how do model architectures affect scalability? ? Although we are seeing an increasing investment in training resources for TSFMs, the bottlenecks and potential driving factors for developing larger TSFMs remain unclear. This raises another practical question: how to design TSFMs from the perspective of scalability?
In this paper, we aim to provide empirical answers to the above research questions. To investigate the scaling laws in OOD scenarios, we trained a family of encoder-only Transformer-based TSFMs, varying three basic training factors: model sizes, compute budgets, and training set sizes. We evalu- ated their performance on both II) and OOD test sets and established scaling laws for three training factors in each scenario. To examine the impact of model architecture scaling behavior, we trained decoder-only Transformer based TSFMs and compared them with the encoder-only versions. Ad- ditionally, we included two sfate-of-the-ar TSFMs, Moirai (Woo et al. 2024) and Chronos (Ansari eet al., (202)), as case studies for detailed analysis. Our experiment results suggest that the nega- tive log-likelihood loss of TSFMs exhibits similar scaling behavior in both OOD and ID scenarios; encoder-only Transformers have a similar scalability with decoder-only Transformers, with a slight advantage in ID data; the architectural modifications introduced by two advanced TSFMs mainly improve ID performance but compromise OOD scalability. Based on the findings and comparative analysis, we finally provided design principles for TSFMs from a scaling perspective.
Our contributions are summarized as follows:
Scaling laws across data distributions. We extend the scaling laws for TSFMs from ID scenarios OOD scenarios across three core training factors: model size, computational resources, and dataset size, establishing a foundation for predicting expected OOD performance gains of TSFMs.
Scaling laws across model architectures. We investigate the scaling patterns of different TSFM architectures, showing the scalability varies depending on the model architecture and the design of specific modules.
Scaling laws-guided design principles. We provide practical design principles for TSFMs from the perspective of data, model and compute scaling, via analyzing the commonalities and differ- ences in scaling behaviors across data distributions and model architectures
PRELIMINARY
To investigate the scaling laws of TSFMs, we curated a large, diverse, and balanced dataset for pre-training. Leveraging this dataset, we trained both encoder-only and decoder-only transformers within two state-of-the-art TSFMs: Moirai and Chronos. For comparative analysis, we evaluated these models (i) in-distribution and (ii) diitt--iiitributionn test sets, focusing key performance metrics to examine the scaling behavior across architectures.
2.1 DATASETS
A large scale, diverse, balanced and high quality pre-training dataset is the foundation build FMs. To this end, we constructed our time series corpus for TSFM pre- vre-training from the large-scale open time series archive. LOTSA (Woo et al. 2024). The corpus comprises approximately 17B time points from 39 datasets spanning seven distinct domains. To ensure that the model performs fairly across all domains, we maintained a balanced ratio of data from different domains. Furthermore, we performed quality filtering on the corpus by constraining the signal-to- geenal-to-noise no-noise ratio of time series to be greater than 20 dB, ensuring that the pre-training corpus exhibits strong predictability. A detailed breakdown of the data sources is provided in Appendix A with summary in Table
To assess the impact of pre- pre-trainn training data scale on model performance, we partitioned the corpus into subsets containing 10M, 100M, and 1B time points, ensuring that each subset maintained similar diversity. For each subset, 95% of the data was allocated for model training, with the remaining 5% reserved validation set to evaluate in-distribution forecasting performance. Additionally,
2

---

Published as a conference paper at ICLR 2025
Table 1: Dataset summary. M indicates million and indicates billion.
Domain
Transport
Climate
Energy
CloudOps
Health
Sales
Web
Total
Datasets
8
2
14
3
9
39
Time points
4.82B
4.73B
4.76B
2.15B
232M
140M
40M
16.8B
Proportion
28.52%
28.06%
28.21 %
12.76%
1.38%
0.83%
0.24 %
100%
Mixture Distribution
Output Embedding
1'
2'
3'
4'
5'
2'
3'
5'
Encoder-only Transformer
Decoder-only Trans former
Patch Embedding
2
1
3
mask
2
Input
Time Series
Figure 1: Architectures of Baseline Time Series Foundation Models. As the most widely used two Transformer architectures, encoder-only Transformer and decoder-only Transformer are se- lected to our baseline. A time series is divided into multiple patches, each treated token and fed into the Transformer model. The shaded patches represent the future horizon to be predicted.
used subset from widely recognized long-sequent prediction benchmark (Wu et al. 2023) to test the model's vut-of-distribution forecasting capabilities. To further enhance the reliability, we also incorporated subset of the Monash dataset (Godahewa al.) (021) additional OOD test data. The details of the dataset composition and properties are provided in Appendix A Table
2.2 MODELS
TSFMs are predominantly built upon the Transformer architecture (Wen et al. 2022). ). For our base- line models, selected widely adopted architectures: the encoder-only Transformers (Woo jtt al., 2024) (Moirai) and the decoder-only Transformers (Ansari et al., 2024) (Chronos). The pri- mary distinction between them lies in the attention mechanisms applied the inputs, illustrated To better adapt them for time series forecasting, we introduce three key modifications input layer, positional encoding and prediction head. More details given in Appendix B
in Figure |
Patch Embedding. There are several approaches for generating inputs for transformer-based TSFMs, including point embedding, patch embedding, and lagged feature embedding. Due the high computational cost of point embedding for long sequences and the limited robustness of lagged feature embedding, we adopt patch embedding in our models. This method, initially introduced by Vision Transformers (Dosovitskiy et al. (2020) and later adapted by PatchTST (Nie et al. (023) time series forecasting, divides the time series into non-overlapping segments, which are then projected into feature space.
Rotary Position Embedding. This technique (RoPE) has rapidly gained popularity as a positional encoding method in recent large language models (Su et al. 2024). Given the improved pre-training efficiency observed with RoPE (Woo et al. 2023), we adopt RoPE as a replacement for the original Transformer's s positional encoding. RoPE encodes absolute positions using rotation matrix while embedding relative position dependencies directly into the self-attention mechanism.
Mixture of Distributions. Our models designed to predict the probability distribution of future time series. However, real-world time series often exhibit complex distributions, including outliers, heavy tails, and extreme skew, which pose significant challenges for accurate modeling. To address

---

Published as a conference paper at ICLR 2025
these complexities, incorporate more flexible output likelihood by utilizing Student-T mixture models (Flunkert et al. 2017). Compared to the commonly used Gaussian mixture models, Student- T mixture models offer greater robustness in handling outliers and heavy-tailed distributions. An empirical comparison between the two mixture distributions is shown in Appendidx B Figure
Our models are characterized by several key hyper-pan phper-parameters: the number of layers (Mayer)) the in- put/output dimensions of the residual stream (dm), the dimensions of the intermediate feed-forward layers (d#), the number of attention heads per layer (Theads), and the dimension of the attention output (dhead). The overall model size can be expressed as:
N 2 Mayer (4dm) Chead + 2d dg) 2dm Mayer (2Rheadsedhead + dg)
= 12Muyer with the standard Thead Chead = dm = 4//4,
where the embedding layer, prediction head, biases and other sub-leading terms are excluded for cleaner scaling laws. The embedding layer mixture distribution prediction head comprises multiple independent linear layers that predict each Student-t mixture distribution parameter for a patch separately, with 512dm parameters in total. In the study, we explore models with ~ 103 to ~ 10 trainable parameters.
a patch size of 32 with 32dm parameters. The
2.3 TRAINING AND EVALUATION DETAILS
In this study, we focus exclusively on uni-variate time series forecasting avoid the confound- ing effects introduced by multivariate time series, such as variable interactions, correlations, and the complexities of modeling multivariate relationships. Future research will address these factors, aiming to establish more comprehensive scaling laws for multivariate time series models.
Training Details. Our training objective is to optimize the mixture distribution log- likeliiiooddd... We utilize the AdamW optimizer with batch size of 128, and maximum learning rate of 10-3 with linear warm-up of 1 training steps, followed by cosine decay for the remaining To facilitate learning data representations across diverse domains with varying series lengths and sample sizes, we visited each sample with probability p: = tz/L,, where t, is the series' time points follow the approach used in Moirai (Woo
10a steps.
and T is the corpus' total time points. In addition, et al. 2024) and Timer (Liu et al.) 2024) by capping the sampling probability at 0.05 to ensure a more balanced contribution from each dataset. We then randomly selected segment from each chosen sample.
Evaluation Details. We evaluate the model randomly selected 10% subset of the test data ery 10sh steps to reduce computational costs. For performance measurement, we observed that non- normalized metries like MAE and MSE are highly sensitive to the amplitude of time series data, often causing the overall average to be disproportionately influenced by high-amplitude datasets. To mitigate this issue, we primarily use the normalized metric, mean absolute percentage error (MAPE), along with the negative log-likelihood (NLL), to assess forecasting performance. For more comprehensive understanding TSFM scaling laws, we also include additional results using symmetric mean absolute percentage error (SMAPE), mean absolute scaled error (MASE), and con- tinuous ranked probability score (CRPS) in Appendix D.4 Detailed descriptions of these metrics provided in the Appendix C33
3 SCALING LAWS FOR TIME SERIES FOUNDATION MODELS
In this section, we first present experimental results using the encoder-only Transformer to explore scaling laws across different data distributions. Following this, conduct a comparative study the scaling behavior of encoder-only and decoder-only TSFMs, Chronos and Moirai, to investigate how various scaling factors influence the characteristics of time series models.
3.1 SCALING LAWS ACROSS DATA DISTRIBUTIONS
Parameter Scaling. In Figure 2 we display the ID and OOD performance of wide variety of encoder-only Transformers, ranging from small models with IK parameters through large models
4

---

Published as a conference paper at ICLR 2025
1.7
118:101
1.2
MAPE
0.7
2.6
0:5
LIN) 122-10''000s
OOD: (5.2 puya
0.2
OOD: LINI (2.8 10
1.6
10
10
10
10
10
10
10'
10"
10"
10"
Model Size
Model
Figure 2: Parameter Scaling, The scaling effect of total trainable model parameters on the in- distribution (ID) and out-of-distribution (OOD) forecasting performance, which is evaluated using NLL and MAPE metrics. When evaluated with NLL, both ID) and OOD results follow an approx- imate power law scaling with parameter count, exhibiting consistent trends across different data distributions. The blue and red horizontal dashed lines represent the baselines of the exponential smoothing (ETS) forecasting method.
with 100M parameters. We trained models the full pre-training corpus convergence and report the minimum NLL and MAPE. We can see that both ID and OOD performance roughly follow power-law behavior over five orders of magnitude in model sizes. Formally, the power law can be expressed as:
N N
L(N) ~
where the performance metric function (i.e., MAPE, or NLL), is a given parameter count, Ne is the normalization coefficient, and o. is the exponent value that indicates the degree of per- formance improvement expected as we scale up N.
Observing the NLL metric, the lines fitting the scaling laws for both ID and OOD data exhibits roughly constant shift and close slopes. This implies that while models incur a consistent perfor- mance bias when transferred OOD data, their scaling patterns correlate well with their perfor- mance on the ID data. When evaluated using MAPE, the power-law for OOD scenario shows bigger exponent value than ID scenario. This indicates that increasing model size yields greater improvements in OOD performance than ID performance. In other words, for models with weak OOD generalization capabilities, increasing model size may enable them to perform equally well on both ID and OOD data.
To evaluate whether the benefits of large-scale pre-training are warranted, we compare the pre- trained models with the classical exponential smoothing (ETS) forecasting method. The results indicate that the pre-trained models consistently outperform ETS on ID data and progressively excel on OOD data the model size increases. This suggests that pre-trained models must reach certain scale, at least 3M parameters in this case, to demonstrate a level of superiority on OOD data that justifies their high pre-training cost.
Compute Scaling, Following the similar method in (Kaplan et al. 2020 we estimate the compute budget using the formula C 6N BS, where B is the batch size, the number of parameter updates, i.e. the input sequence length, and 6 is the factor to account for the forward and backward passes. The ID and OOD test loss for compute budget varying over six orders of magnitude shown in Figure We see that the optimal results for each compute budget are achieved by different model sizes N. but the lowest loss decreases according approximate power law with respect to the amount of training compute. The lowest losses appear as the heavy lines, which can be fit with
C.
L(C) ~
(3)
C
5

---

Published as a conference paper at ICLR 2025
4.0
5.5
3.0
2.0
4.5
MAPF
1.0
2,5
ID: ((C) (6.5 10
O00: ((C) - (4.8 10
000: LIC)
-10
1.5
0.1
10
10
10T
10
10
10
10
10
10
10
Compute day-
Compute [PF-days]
Figure 3: Compute Scaling, The computation scaling results indicate that model performance scales approximately according power law with increasing compute, consistent across both ID and OOD scenarios. The ID and OOD results illustrate that there is an lower bound for loss and MAPE on both test data under given computational budget.
1.7
5.0
1.2
4.0
0.7
MAPE
3.0
2.0
10
000:100 (0D) (8.4 100
0.2
000: 100) (2.9-10-1)
10
10
10
10
10
10
10
10'
Figure 4: Data Scaling, The blue and red plots illustrate how data volume affects the ID and OOD forecasting performance of encoder-only Transformers, evaluated using NLL and MAPE metrics. The results indicate that in both scenarios, model performance scales approximately power law with data volume.
We observed significant noise NLL and MAPE during training, which may be caused by the learning rate scheduler and the random sampling evaluation strategy. Similar to parameter scaling, the ID and OOD NLL show similar scaling patterns; however, when evaluated using MAPE, the OOD power law shows larger exponent values than the ID power law.
Data Scaling. We display empirical trends for the performance as function of dataset size in Figure For the trend, trained multiple IB encoder-only Transformers series of subsets of the pre-training dataset and report the averaged evaluation results during training. We see that the NLL and MAPE can be fit well with simple power-law
De D
ap
L(D)* ~
(4)
Different from the parameter scaling and computational scaling, we found that when evaluated using NLL, ID and OOD performance do not exhibit the same scaling behavior. Instead, ID performance more sensitive to the scaling of dataset size compared to OOD performance. However, similar to the observations greater than its impact on MAPE in ID data. This suggests that the scaling of various factors yields greater improvements in OOD performance than ID performance.
the other two factors. the impact of data scaling MAPE in OOD data is
Cross-distribution Scaling Effects. We summarize the key findings on how model ID and OOD performance scales with the model parameters, data volume. and compute. (1) In both ID and OOD uni-variate time series forecasting, model performance follows a simple power law as a function of model parameters, data volume, and compute. (2) For NLII, the model exhibits similar scaling

---

Published as a conference paper at ICLR 2025
73
63
(O0) -32---32--00
(Gl) 199
End dit 18.4 10
53
Dec 118) 10"
Dec 137 10
Dec 1402 (1.3 10
49
4.3
39
3,3
23
2.9
Boci 1181 CL tomtggoo
Erer 264) m0.3 sohicpiin 11r 10
Zect UID 10thoas
Deet 200 182 1omh-oy
Dcc L1B) 1otggr
10"
In
10"
10
10
10
10
10
10'
10
Model Size
Data Size
Figure 5: Scaling Laws of Encoder-only vs. Decoder-only Transformer. This figure presents comparison of scaling behaviors on NLL between encoder-only and decoder-only Transformer across three different axes: number of parameters, compute, and dataset size. Overall, both models exhibit similar scalability patterns with respect to model parameters, computation and dataset sizes across ID and OOD data, but differ in ID performance.
Enc: L(V) 1n) oo
L.7
Enc: L(V) 10' 854
Moirai: L(M) (2.3 /
Moirai: LIM) (6.5 1005-100550 11) Pay--000
1.2
4.5
3.5
MAPE
0.7
2.5
1.5
Enc: LIN) (4.1 (4.1-10"*MNpe 1Njass
Enc: ((N) - 12.8 10
Moirat: ((M) = (1.0 m0) ois
0.2
Moirai: L100 = (7.2-100 (7.2-10"""/"paaaa
10
10
10
10
10
10"
10
10
10"
10"
10
10
Model Size
Model Size
Figure 6: Scaling Laws of Encoder-only Transformer vs. Moirai. This figure compares parame- ter scaling between the encoder-only Transformer and Moirai. While Moirai demonstrates notable improvements on ID test data compared to the baseline, certain design choices may hinder its scal- ability on OOD data.
patterns in both ID and OOD scenarios, in terms of model size compute resources. (3) When using MAPE as the metric, scaling of all three factors results greater improvements in OOD performance compared to ID performance.
3.2 SCALING LAWS ACROSS MODEL ARCHITECTURES
The above results suggest that the power-law captures the scaling behavior of encoder-only Trans- formers in both ID and OOD scenarios. Similarly, we analyze the scaling properties of decoder-only Transformers, along with two other state-of-the-a TSFMs, Chronos and Moirai, to assess the im- pact of model architectures on scaling behavior. These models are trained on the dataset we built with the same training setup.
Encoder-only ys. Decoder-only Transformer. Figure presents comparison of the scaling pat- terns between encoder-only and decoder-only Transformers. In parameter scaling, the encoder-onl Transformer shows slight advantage, with marginally higher power-law exponent on ID data and lower NLLL values. For compute and data scaling, the two architectures demonstrate nearly iden- tical scalability across ID and OOD settings, with some performance differences observed on ID data. Overall, the two architectures exhibit similar scalability, aside from performance difference observed in ID data.
Encoder-only Transformer vs. Moirai. Moirai is TSFM based encoder-only Transformer architecture. introduces "ani-tariate" attention to capture relationships between multiple vari- ables. Additionally, Moirai incorporates multi-scale patch embedding to handle different fre- quency patterns, and a more diverse mixture distribution to model real-world probability distribu- tions. Figure / show comparison between the scaling behavior of encoder-only Transformer and

---

Published as a conference paper at ICLR 2025
12.0
10.0
1/M (2.5 10"
0.8
(1.3 Sayjo
Chronos: LIN) (2.9 10 coe
0.7
Chronos: L(M) (1.1 "Njuue
8.0
6.0
0.5
0.4
4.0
Dec: L(M) - (1.4 10
Dec: L(M) 12.4- 10/ 10
2.0
Chronos: ((1) = (1.2 10
Chronos: L0W) = (3.3 1000M
10
10
10
10"
10
10
10"
10'
10"
Model Size
Model Size
Figure 7: Scaling Laws of Decoder-only Transformer vs. Chronos. This figure presents compar- ison of parameter scaling between decoder- -ondeer-onlyyyooo Transformer and Chronos. The design introduced by Chronos enhances time series forecasting, but fails to improve the generalization of OOD.
Moirai. On ID data, Moirai demonstrates better performance. However, for OOD data, as the num- ber of model parameters increases, Moirai is gradually surpassed by the encoder-only Transformer. Comparing the power-law lines the two models, we see that Moirai shows a smaller slope, indi- cating relatively weaker scalability. Collectively, Moirai shows significant improvements on ID time series forecast than our baseline, but some designs may limit its scalability on OOD data.
Decoder-only Transformer vs. Chronos. Chronos-T5 an encoder-decoder Transformer-based TSFM that, like decoder-only Transformers, follows an auto-regressive prediction approach but uses separate encoder extract contextual information. It adopts point-wise prediction and transform numerical regression into discrete probability prediction. Figure 7 compares the scaling patterns of decoder-only Transformers and Chronos-T5. When evaluated with NLL,, Chronos-T5 exhibits power laws with very small exponents, indicating limited scalability. We attribute this to the dis- crete probability prediction, as NLL on discrete distribution is not distance-sensstive,, meaning the loss remains high unless the predicted value exactly matches the label. Compared to NLL, sym- metric mean absolute percentage error (SMAPE, definition is shown in Appendix C.3) is a more appropriate metric for evaluating the two models. Chronos-T5 shows slight advantage in perfor- mance and scalability on ID data. However, on OOD data, the decoder-only Transformer performs better. Despite some performance differences, both models exhibit similar scaling patterns on OOI) data. Overall, the design improvements in Chronos-TT enhance ID time series forecasting, but they do not effectively improve OOD generalization.
Scaling Effects Summary. Our analysis of the scaling properties across architectures reveals sev- eral key findings: (1) Model architecture and design play a crucial role in determining scalability. (2) Encoder-onj, models and decoder-only models show a similar scalability on OOD data; while encoder-only only models have a slightly higher parameter scalability and certain performance advan- tage on ID data. (3) While Chronos and Moirai improve ID forecasting, these gains do not extend effectively to OOD scenarios.
DESIGN PRINCIPLES FOR TIME SERIES FOUNDATION MODELS
Building on our findings regarding the scaling laws in TSFMs, we elaborate design principles to guide the development of scalable models. These principles are framed around three key dimensions: training data, model parameters and architecture, well computational resources.
Training Data. Our experiments show that increasing the size of the training dataset leads to a greater performance improvement OOD data compared to ID data. Enlarging the pre-training dataset is crucial for achieving better generalization. Based on Equation 4 we can expect that doubling the size of the pre-training dataset will result in OOD MAPE reduction to approximately 90% of its previous value, and an ID MAPE reduction to approximately 97%. But maintaining diversity within the dataset is equally important while increasing the data volume.
8

---

Published as a conference paper at ICLR 2025
births
mm5 daily
australian electricity
0.14
0.87
0.20
0.12
0.18
MAPU
0.10
0.77
0.16
0.08
0.67
0.06
10' 0.12
10'
10
10
10
10'
10
10 10'
10'
Model Sizee
Model Size
Figure 8: Case studies on the "Emergent Abilities" of scaling time series models. We present three examples of zero-shot (unseen) outtof-distribution (OOD) time series prediction. In the us births, nn5 daily, and australian electricity datasets, we observed that model behavior deviates from expected power law patterns, instead exhibiting characteristics more akin emergent phenomena.
Model Parameters and Architecture. Our study highlights that model size is the most critical factor for improving TSFM performance. Compared to collecting more data, increasing model size yields the greater benefit for both ID and OOD forecast. However, as model size scales up, data
bottlenecks inevitably emerge. From the ratio of exponents in Equations 2 and we infer that
scaling up model size should be accompanied by a sublinear increase in dataset size. Specifically,
this relationship can be approximated as с N a N meaning that doubling the model size requires roughly 1.7 times more data. In terms of model architecture, we found some modifications facilitate ID performance but can not generalize well OOD scenarios. Model design can impact scalability, and good design should promote performance, generalization, and scalability.
Compute. The power law we established indicates that there an unbreakable lower bound for NLL loss and MAPE under given computational budget. This means that, with other factors remaining constant, as the model size increases, more compute resources must be invested to achieve better performance. Our findings suggest that compute resources should prioritize scaling up model size rather than extending training time for fixed models as larger models are more sample-efficient than smaller models. Moreover, different training objectives model architectures can significantly affect this performance bound, exploring a compute-efficiem training paradigm will be promising.
RELATED WORKS
Neural Scaling Laws. Neural scaling laws seek to provide predictive framework for optimizing the allocation of computational resources to maximize model performance. In language domains, Kaplan et al. (2020) demonstrated that performance follows a power-law relationship, improving as more computational resources, parameters, and data utilized. Subsequent research has expanded this to predict other factors, such as downstream task performance (Isik et al. 2024) and inference time (Sardana et al. 2024). In vision domains, scaling laws have been explored in criminative modeling (Hestness et al. 2017) and visual auto-regressive modeling (Henighan et al.] 2020). Recently, [Edwards et al. (2024) introduced scaling laws for large time series models, show- ing that performance scales according to a power law with model size, compute, and dataset size. Shi et al. (2024a) examined the effect of time-series forecasting horizon on model scaling behav- 1or, offering theoretical framework to explain its influence. However, both studies have focused on in-distribution scenarios, leaving the investigation of scaling laws in -iisttiiittittto contexts largely unexplored.
like dis-
Time Series Foundation Models. Foundation models (Das et al. 2024 Goswami et al. 2024) represent a new paradigm aimed at generalizing across diverse domains and tasks by leveraging knowledge from large-scale data in the pre-training phase. They have significantly advanced time series forecasting, particularly in zero-shot scenarios, where predictions made data from pre- viously unseen domains. For instance, |Woo et al. (2024) introduced Moirai, an encoder-only trans- former architecture that employs "any-variate attention mechanism to capture dependencies in multivariate time series. Ansari et al.] (2024) proposed a method that tokenizes time series val- through scaling and quantization into fixed vocabulary, training series of transformer-based

---

Published as a conference paper at ICLR 2025
models known Chronos. Liu et al. (2024) developed Timer, simple decoder-only transformer architecture designed for univariate time series forecasting, while Rasul et al. (2023) introduced Lag-Llama, decoder-only transformer that integrates lags covariates to improve forecasting curacy. These models incorporate various modifications to the standard Transformer architecture for time series data. However, the impact of these changes model scaling properties has not been systematically studied. As model size increases, it remains an open question whether these modifications will continue enhance performance.
DISCUSSION
6.1 ADDITIONAL STUDIES
We also conduct the following analysis to better understand the scaling behaviors of TSFMs. Due to the space limit, see their details in Appendix D.2 to D.4.
Emergent Behaviors. Figure 8 shows some examples of zero-shot OOD time series prediction, where the model's s performance remains low until the model size reaches a critical threshold, af- ter which performance improves significantly. This scaling behavior deviates from the previously observed power law and more akin to emergent phenomena (Wei et al. 2022).
Sample Efficiency. Appendix D.2 Figure 14 shows the evaluation results during training. Large models are more sample-efficient than small models, reaching better performance with the same optimization steps and using fewer time points.
Scaling Pattern Depends on Data Distributions. Appendix D.3 Figure 15 19 show the scaling behavior of TSFMs on the Monash subset, suggesting that OOD scaling behaviors varies depending on the relationship between the unseen data distribution and the pre-training data distribution.
Scaling Pattern Depends Performance Metrics. Appendix D.4, Figure 15 19 provides
comprehensive analysis of scaling behavior across five performance metries: NLL, MAPE, SMAPE, MASE, and CRPS. All metrics exhibit consistent decreasing trend, approximately following power law.
6.2 CONCLUSION
We have observed consistent scaling of encoder-only Transformer NLL with parameter count, train- ing computation and dataset size both ID and OOD test data, as encapsulated in power law. The experimental results show that as the number of parameters, computational resources, and training data increase, both models' ID and OOD performance will continue to improve. Furthermore, established laws suggest that for ID and OOD forecasting performance, larger model may be more important than more data.
We have also study the scaling properties of two common types of TSFMs, encoder-only Trans- former and decoder-only Transformer. The two architectures show similar scalability in OOD data across model size, computational resources, and training data. While encoder-only models have a slightly better parameter scalability and performance in ID data based our evaluation setting. More- over, we further include two advanced TSFMs, Chronos and Moirai, as specific cases to study the impact of module design on scaling behavior. We find that both Chronos and Moirai improve ID forecasting performance, but these gains can not translate effectively to OOD scenarios. Overall, model architecture and module design important influence factor for scalability.
Our experiments independently investigated the scaling behavior of TSFMs in terms of parameter count, computational resources, and training data size, assuming other factors were unlimited. In future work, it would be valuable to develop unified model to explore the relationships between these factors. This could help guide the community in optimizing resource allocation when one resource is limited. Additionally, multiple studies have shown that model performance varies across different context window lengths and forecast horizons. In future research, we plan to incorporate the impact of context length and forecast horizon scalability.
10

---

Published as a conference paper at ICLR 2025
Acknowledgments and Disclosure of Funding
This research was partly supported by Australian Research Council (ARC) under grants FT210100097 and DP240101547 and the CSIRO - National Science Foundation (US) Al Research Collaboration Program. This work has been also supported by NVIDIA Academic Grant from the
NVIDIA Developer Program 2025.
Ethics Statement
Our research is dedicated exclusively to addressing scientific challenges and does not involve human participants, animals, or materials that pose environmental concerns. We anticipate no ethical risks conflicts of interest.
Reproducibility Statement
We provide the implementation details in Appendices B and C including model configurations and training & evaluation details. The source code and related source of this work are available at Sca Laws for reproducibility.
https:
REFERENCES
Alexander Alexandrov, Konstantinos Benidis. Michael Bohlke-Sehneider. Valentin Flunkert, Jan Gasthaus. Tim Januschowski. Danielle C. Maddix, Syama Sundar Rangapuram, David Salinas, Jasper Schulz, Lorenzo Stella, Ali Caner Türkmen, and Bernie Wang. Gluonts: Probabilistic and neural time series modeling in python. J. Mach. Learn. Res., 21:116:1-11666, 2020. URL https: semant intemmntisccii icscholar
Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, Jasper Zschiegner, Danielle C. Maddix, Michael W. Mahoney, Kari Torkkola, Andrew Gordon Wilson, Michael Bohlke-Schneider, and Yuyang Wang. Chronos: Learning the language of time series. ArXiv, abs/2403.078155 2024. URL http:s///ap.seemttt.ccchoooooooo org/ Corpus1D:26836353.
Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. dynamical model of neural scal- ing laws. ArXiv, abs/2402.01092, 2024. URL https: api www.nticschoolar org/
CorpusID:267400
Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. forty-first International Conference Machine Learning, (CML20244 Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https::/ /tttps:///opnnreveew
fet//oormmid=ie jndiidnjniiiia
Hoang Anh Dau, A. Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn J. Keogh. The time series archive. IEEE/CAA Journal of Automatica Sinica, 6:1293-1305, 2018. URL http::htt//apti semanticschol arr org/ otw.:/cccc.aaaaooooooooooooaaaaa
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ArXiv, abs/2010.1 19929, 1929, 2020. URL https: /ttp//aapp.pp www.antitsccooo.oooooooo org/ CorpusID:22503320
Thomas D. P. Edwards, James Alvey, Justin Alsing, Nam H. Nguyen, and Benjamin D. Wandelt. Scaling-laws for large time-series models. ArXiv, abs/2405. 13867, 2024. URL httpps//s://
www.laarrrrg/oooooooeooe ortp:cchooo....oooooooouaaaaaaaa :ttp:///ooo111::222233333 20:399823322
Patrick Emami, Abhijeet Sahu, and Peter Graf. Buildingsbench: A large-scale dataset of 900k buildings and benchmark for short-term load forecasting. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Anmual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.

---

Published as a conference paper at ICLR 2025
Wei Fan, Pengyang Wang, Dongkun Wang, Dongjie Wang, Yuanchun Zhou, and Yanjie Fu. Dish-ts: a general paradigm for alleviating distribution shift in time series forecasting. In Proceedings of the AAAl conference artificial intelligence, volume 37, pp. 7522-7529, 2023.
Valentin Flunkert, David Salinas, and Jan Gasthaus. Deepar: Probabilistic forecasting with autoregressive recurrent networks. ArXiv, abs/1704.04410,, 2017. URL https://api.
wwtp://aaar.../oooooooooo 12199225
B. Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Gareía, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation. ArXiv, abs/2109.07740, 2021. [emnaa/.eessto...rrrrrooo/ooooop
Rakshitha Godahewa, C. Bergmeir, Geoffrey I. Webb, Rob Hyndman, and Pablo Montero-Manso. Monash time series forecasting archive ArXiv, abs/2105.066 066643,, 2021. URL https://api.
Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. Moment: A family of open time-series foundation models. ArXiv, abs/2402.00385, 2024. URL
ar. wwtaaar../o/ooooaaa//aaaa :26500022267000
Albert Gu, Karan Goel, and Christopher R'e. Efficiently modeling long sequences with structured org/
state spaces. ArXiv, abs///11..00001101 2021.
CorpusTD:22033544
Tatsunori Hashimoto. Model performance scaling with multiple data sources. In International
Conference on Machine Learning, pp. 4107-4116. PMLR, 2021.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Rad- ford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling laws for autoregressive generative modeling. ArXiv, abs/2010.14701, 2020. URL
stw.apppsseaaaacnccccccoeae hwtp://o.......o//ooooos.ssss 22:004441
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Frederick Diamos, Heewoo Jun, Has- san Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling predictable, empirically. ArXiv, abs/1712.00409, 2017. URL hhtps:://pp. semant www.tticschhll.co htwp:aaiiisssaaar.eoooooooooseee 2222077
Rob Hyndman and George Athanasopoulos. Forecasting: principles and practice. 2013. URL http:///appseeennncccccc.....aaa wtt//CorpssD:::112222222
Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii. and Sanmi Koyejo. Scaling laws for downstream task performance of large language mod- cls. ArXiv, abs/2402.004177.
2024. URL https: http://ppp..mmmttccccccc.aa.....
CoxpusID:257499803
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen- cralization in neural networks. Advances in neural information processing systems, 31, 2018.
Jiawei Jiang, Chengkai Han, Wenjun Jiang, Wayne Xin Zhao, and Jingyuan Wang. Libeity: A unified library towards efficient and comprehensive urban spatial-temporal prediction. 2023. URL htt.:/oaar.rrrrooorssi2222255-
Iuttps://age
Ming Jin, Qingsong Wen, Yuxuan Liang, Chaoli Zhang, Siqiao Xue, Xue Wang, James Zhang, Yi Wang, Haifeng Chen, Xiaoli Li, et Large models for time series and spatio-temporal data: A survey and outlook. arXiv preprint arXiv:23100 101996. 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. Scaling laws for neural language models. ArXiv. abs/2001.08361, 2020. URL https://////..... //apti//pii.. swwant .ttpoolrrrrrr/g//oooooooo CorpusTD:21096100:
12

---

Published as a conference paper at ICLR 2025
Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Haim Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avshalom Manevich, Nir Ratner, Noam Rozen, Frez Shwartz, Mor Zusman, and Yoav Shoham. Jamba: A hybrid transformer-mamba language model. ArXiv, Abs/2403.19887 2024. URL https://api. semanticscholes Crw.CorppoC/ooDC..o...... 268793598
Xu Liu, Yurong Xia, Yuxuan Liang, Junfeng Hu, Yiwei Wang, Lei Bai, Chao Huang. Zhenguang Liu, Bryan Hooi, and Roger Zimmermann. Largest: A benchmark dataset for large-scale traffic forecasting. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference Neural Information Processing Systems 2023, NewrlPS 2023, New Orleans, LA, USA, De- cember 10 16, 2023, 2023. URL httt:::/paperst nips http:/apppr_lll/ppppperer Abst otg.://cceccaaaaaaaaaaaaaaaaaaar
/3a (bdd27ffcaadaaadcd3aaaaaa and_Benchmarrd [ttnchmarrss.tmm]]]
Yong Liu, Haoran Zhang, Chenyu Li, Xiangdong Huang, Jianmin Wang, and Mingsheng Long.
Timer: Generative pre-trained transformers are large time series models. International
Conference on Machine Learning, 2024. URL https://apii stt.pppp.semmttccsccocoleeeeeeee /ttp org/
(TrpusID:267112211
James E Matheson and Robert L Winkler. Scoring rules for continuous probability distributions. Management science, 22(10):0087-1096 1976.
Tung Nguyen, Jason Jewik, Hritik Bansal, Prakhar Sharma, and Aditya Grover. Climate- learn: Benchmarking machine learning for weather and climate modeling. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference formation Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http: http:/aapppa.aeeeaaeeeeeeee ni pss wwtpppppeaflleeecceee file_f htt.ii/s//pppp//222222 dec 4- Absttacttttt...................t ppatasets_anc,
Neural In-
Benchmarks.ht htmll
Yuqi Nie, Nam Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference on Learming Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenRevvew.net, 2023. Forum id Jbdc0v10col
URL
net /wt.:ertttttte///eeao
Yuqi Nie, Yaxuan Kong, Xiaowen Dong, John M Mulvey, H Vincent Poor, Qingsong Wen, and Stefan Zohren. A survey of large language models for financial applications: Progress, prospects and challenges. arXiv preprint arXiv:2406.1190 11903, 2024.
Youngsuk Park, Danielle Maddix, François- Xavier Aubet, Kelvin Kan, Jan Gasthaus, and Yuyang Wang. Learning quantile functions without quantile crossing for distribution-fiee time series forecasting. In International Conference on Artificial Intelligence and Statistics, pp. 8127-8150.
PMLR, 2022.
Raquel Prado. Bayesian forecasting and dynamic models.
2020.
URL https:
Xiangfei Qiu, Jilin Hu, Lekui Zhou, Xingjian Wu, Junyang Du, Buang Zhang, Chenjuan Guo, Aoy- ing Zhou, Christian Jensen, Zhenli Sheng, and Bin Yang. Tib: Towards comprehensive and fair benchmarking of time series forecasting methods. Proc. VLDB Endow., 17:2363-23777 2024. URL httpthttt////pptt http:::://ppl.seeemccccccoooooo
semant wascooaa...cccaaa. htt.r//arrrrrrroooosssts:: 6939335
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou. Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-61 2020.
Kashif Rasul, Arjum Ashok, Andrew Robert Williams, Arian Khorasani, George Adamopoulos, Rishika Bhagwatkar, Marin Bilos, Hena Ghonia, Nadhir Vincent Hassen, Anderson Schneider, Sahil Garg, Alexandre Drouin, Nicolas Chapados, Yuriy Nevmyvaka, and Irina Rish. Lag-Ilama:
13

---

Published as a conference paper at ICLR 2025
Towards foundation models for time series forecasting. ArXiv, abs/2310.08278,, 2023. URL www..acccc...................... www.maaniicccooo.......ooooooaaa 2tt.:pppps:::::7755500
https://api.
Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws. In Forty-first International Conference
on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
URL https: pestps://openee /ttp:////ppeeeeeeeeeeeee net forum? fttp:mmmmiddmmmmemmmmmmm-- TDUu
Jingzhe Shi, Qinwei Ma. Huan Ma. and Lei Li. Scaling law for time series forecasting. ArXiv, abs/2405.15124, 2024a. URL https::/ www.ttiisccottooooooooei wwt.cchoola..oooooouuoooeeee
2 270045141
Xiaoming Shi, Shiyu Wang, Yuqi Nie, Dianqi Li, Zhou Ye, Qingsong Wen, and Ming Jin. Time- moe: Billion-scale time series foundation models with mixture of experts. arXiv preprini arXiv:2409. 16040, 2024b.
Jianlin Su, Murtadha H. M. Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu, Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024. doi: 00.016//NEUCOM.2223.226666
URL /tttps::/abi.rgg///
neucom.
20233 127063
Chang Wei Tan, Christoph Bergmeir, François Petitjean, and Geoffrey I Webb. Time series extrin- sie regression: Predicting numeric values from time series data. Data Mining and Knowledge Discovery, 35(3):0032-1060, 2021.
Yihe Wang, Yu Han, Haishuai Wang, and Xiang Zhang. Contrast everything: A hi- erarchical contrastive framework for medical time-series. In Alice Oh. Tristan Nau- mann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Informa- tion Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023a. URL http::/ www.::s.issss.c.ppeeeee httpppppee...leeepppprr///331112 Abt.::rttttatttttt--a-aatt Centrrract-oofffeencee html
ae
Yuxuan Wang, Haixu Wu, Jiaxiang Dong, Yong Liu, Mingsheng Long, and Jianmin Wang. Deep time series models: A comprehensive survey and benchmark. ArXiv, abs/2407. 2024. URL
https://api. www.nt org. wttp:cccoo.....ooooorooo..... ID: 7002876633
Zhixian Wang, Qingsong Wen, Chaoli Zhang, Liang Sun, Leandro Von Krannichfeldt, and Yi Wang.
Benchmarks and custom package for electrical load forecasting. ArXiv, abs/2307.0713 2023b.
URL httth:t/p/:ppp
i cscho ar ort..r//////ss10005550000
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed Huai hsin Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. ArXiv, abs/2206.07682, 2022. URL https::/aapppp www.nteesactooooooooo .rrg/ org/
CorpusTD:22967755
Haomin Wen, Youfang Lin, Lixia Wu, Xiaowei Mao, Tianyue Cai, Yunfeng Hou, Shengnan Guo, Yuxuan Liang, Guangyin Jin, Yiji Zhao, et al. A survey service route and time prediction in instant delivery: Taxonomy, progress, and prospects. IEEE Transactions on Knowledge and Data Engineering, 2024.
Qingsong Wen, Tian Zhou, Chao Zhang. Weiqiu Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Transformers in time series: survey. International Joint Conference on Artificial Intelligence, 2022. URLpttts: / api semant.enntttt..mmmmmmm www.ttisscchh......oooo...o....e wwtp/r/ggooouuooo ID:
Gerald Woo, Chenghao Liu, Akshat Kumar, and Doyen Sahoo. Pushing the limits of pre-training for time series forecasting in the cloudops domain. ArXiv, abs/2310.050633, 2023. URL https:
Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers. In Forty-first International Con- ference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. htt.:ttss:
14

---

Published as a conference paper at ICLR 2025
Haixu Wu, Teng Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Tem- poral 2d-variation modeling for general time series analysis. ArXiv, abs/2210.02186, 2022. URL
https://apls www.ttiisncnnoooooe.ee
ott/ ID: 252715492
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In The Eleventh International Conference on Learning Representattions, ICLR 2023, Kigali. Rwanda, May 1-5, 2023. OpenRe- net net/forumidd--ee 2il_uuq33400
view.net, 2023. URL httpt::ttppppeeeeeeee
Chao-Han Huck Yang, Yun-Yu Yun Tsai, and Pin-Yu Chen. Voice2series: Reprogramming acoustic models time series classification. In International conference on machine learning, pp. 11808- 11819. PMLR, 2021.
Jiasheng Ye, Peiju Liu, Tianxiang Sun, Yunhua Zhou, Jun Zhan, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. arXiv preprint Stti::2403.6955:5 2024.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 12104-12113, 2022.
Kexin Zhang, Qingsong Wen, Chaoli Zhang, Rongyao Cai. Ming Jin, Yong Liu, James Zhang, Y. Liang, Guansong Pang. Dongjin Song, and Shirui Pan. Self-supervised learning for time series analysis:
Taxonomy, progress, and prospects. IEEE Transactions on Pattern Analysis and Ma- URL https: / api semanticscholar..rgg
chine Intelligence, 46:6775-6794, 2023.
CorpusID:255203655:
15

---

Published as a conference paper at ICLR 2025
A PRE-TRAINING DATASETS
Previous work has proposed several public large-scale time series datasets, e.g. LOTSA (Woo et al.) 2024) and UTSD (Liu al. 2024). Instead of using these datasets directly, we built pre-training corpus through data filtering. There are three main reasons for this: First, these datasets contain numerous periodic time series with substantial pattern redundancy. Second, these datasets show a heavily long-tailed distribution, with multiple domains accounting for less than 5% of the total. Third, data quality variable, with some time series missing many observations or having low signal-to-noise ratios. To address these issues, we developed a data filtering pipeline:
Deduplication. In these public datasets, we observed significant redundancy across many subsets. This redundancy typically manifests as numerous repeated patterns within individual samples or high similarity between multiple samples. Such redundancy can negatively impact training effi- ciency. To address this issue, we applied a down-sampling strategy for datasets with redundant samples based on the data sampling period.
Quality Filtering. We focused selecting time series data that have missing values and signal-to-noise ratio (SNR) greater than 20 dB to ensure better predictability. To calculate the SNR, first used Fourier transform to identify the low-frequency components of the time series. We then applied a low-pass Butterworth filter to extract these low-frequency components as the signal, with the remaining residuals considered as noise.
Domain Balancing. In the LOTSA dataset, the climate domain accounts for nearly 90% of the data, while the remaining seven domains collectively represent only 10%. To create more balanced pre-training dataset, we selected the transport domain, which contains 4.8 billion time points, as the reference. We then scaled the data from the other domains similar magnitude.
Following the aforementioned processing steps, we compiled high-quality time series dataset con- taining a total of 16.8 billion time points across 7 domains. The data volume is sufficient support training models with parameters ranging from 10 to 10 parameters. Furthermore, most domains include at least 100 million time points, ensuring that even when the dataset is divided into subsets of various sizes, domain diversity is preserved. Below, we introduce the datasets for each domain and outline the key properties of the datasets after processing, including domain type, sampling frequency, number of time series, total number of observations, and data source.
The climate data are sourced primarily from the ERA5 and CMIP6 datasets, which pro- vide time series for various climate-related variables, including temperature, humidity, and pressure levels. During data curation, we observed a high degree of similarity across yearly data. To reduce redundancy, only two years of data from ERA5 and CMIP6 were included.
Climate.
The energy data primarily come from the BuildingsBench dataset, which provides time series data residential and commercial building energy consumption. After applying quality filtering based on SNR, we selected samples from the Buildings-900k BDG-2, and Sceaux datasets. Additionally, high-quality data from the ProEnFo library, including the GEF, ELF, PDB, Spanish, and Energy datasets, were incorporated.
Energy.
Transport. The primary source for transport data is the LargeST traffic dataset, covering traffic flow in California from 2017 to 2021. We also integrated datasets from LibCity, PEMS, Loop Seattle, and Q-Traffic into our corpus.
Cloud Operations. The cloud operations data is from the large-scale CloudOps time series datasets, which measures various variables such CPU and memory utilization.
Healthcare. We include diverse sets of healthcare data from the uer time series archive.
Web. The web data comprises the Kaggle Web Traffic Weekly dataset and the Wiki-Rolling dataset.
Sales. Here we use the Favorite Sales dataset.
16

---

Published as a conference paper at ICLR 2025
Table 2: Pre-training datasets and key properties.
Dataset
Domain
Frequency
Time Series
# Obs.
Source
CMIP6
Climate
196,608
2.870,476.800
Nguyen et
2023
ERA5
Climate
212,992
1.860,688.1
Nguyen
2023
Buildings900K
Energy
538,577
4,718,473,097
Emmami 2023
Australian Electricity
Energy
30T
1,153,584
Godahewa al
2021
BDG-2 Bear
Energy
H
1,482,312
Emami et al.
2023
BDG-2 Fox
Energy
135
2,324,568
Emami et
2023
BDG-2 Punther
Energy
919,800
Emami et
Sccaux
Energy
34,223
Emami
2023
Solar Power
Energy
48
7,397,222
Godahewa al.
2021
Covid19 9 Energy
Energy
H
31,912
Wang al.
2023b
Spanish
Energy
35,064
Wang al.
2023b
Eleedemand
Energy
17,520
Godahewa al
2021
Energy
Wang al.
2023b
GEF17
Energy
140,352
Wang al.
2023b
GEF14
Energy
H
17,520
Wang al.
2023b
ELF
Energy
H
21,792
Wang al.
2023b
Traffic Weekly
Transport
862
82,752
Godahewa al
2021
Q-Traffic
Transport
45,148
264,386,688
Jiang et
PEMS04
Transport
Jiang
PEMS07
Transport
ST
883
24,921,792
Jiang
202.
PEMS08
Transport
ST
510
9,106,560
Jiang
2023
PEMS Bay
Transport
ST
325
1,690
Jiang et al.
2023
Loop Scattle
Transport
ST
1,809
33,953,760
Jiang et al
2023
LurgeST
Trunsport
42,333
4,452,510,528
Liu et
2023
Azure Traces 2017
CloudOTS
159,472
885,522,908
Woo al.
Borg Cluster Data 2011
CloudOTS
ST
286,772
1,075,105.708
Woo al.
2023
Alibaba Cluster Trace 2018
CloudOTS
ST
116,818
190,385,060
Woo al.
2023
Wiki-Rolling
Web
D
47,675
40,619,100
Alexandrov et al.
2020
Kaggle Weh Traflic Weekly
Web
145,063
16,537, 182
Godahewa al.
21121
111,840
33,,111,660
PigArtrressrre
Hcalth
312
624,000
Dau
2018
SeliRegulationSCPI
Health
0.004 SBC
3,366
3,015,936
Dau
2018
SelfRegulatioSCF2
Health
0.004 SBC
2,660
3,064,320
Duu et ul.
2018
TDBrain
Health
0.002 SEC
28644
73,299.996
Wang al.
2023a
MotorImagery
Health
0.001 SEC
24,192
72,576.000
Dau et al.
2018
PigCVP
Health
312
624,000
Dau al.
2018
AtriaFibrillation
Hcalth
0.008 SEC
38,400
Dau
2018
IEEEPPG
Health
0.008 SEC
15,480
15,480,000
Tan
2021
BIDMC52HR
Health
15.898
63,592,000
Tin et al.
2021
17

---

Published as a conference paper at ICLR 2025
B TIME SERIES MODELS
We define time series forecasting as the following problem: given collection of multivariate time series samples with look back window (x1,...; xL) where each x; at time step t vector of dimension of M, our goal is to forecast 7" future values (XL+1).... (x XI-T).
Patch Embedding. We split the input (X1,-..,xx, xt.) into M uni-variate time series x(0) E IRIx, independently forecasting future time series for each variate. Each uni-variate time series x(0 is first divided into non-overlapped patches. Specifically, given the patch length P. the patching process will generate the a sequence of patches x5" E PRN where = # is the number of patches. Then the patches are mapped to the latent space of dmn via learnable linear projection Wp E Rd xP, In our baseline models, the patch size P is set to 32.
Rotary Position Embedding. Rotary Positional Embedding (RoPE) is type of position encod- ing that encodes absolute positional information with a rotation matrix and naturally incorporates explicit relative position dependency in self-attention formulation. In detail, RoPÉ incorporates absolute position information to the embedding and transform them into queries, keys through func- tion:
fg. (Xms m) Ro W v.
where
M
M2
cos m0,
sin m0;
R
M,
sin m0,
cos m0,
Ma/2_
is the rotary matrix with pre-defined parameters O {0, 10000 E [1, d/2),,
Wg. is the learned query or key projection weights, and X.m E Rd is the embedding of the mn token. Next, applying RoPE to the dot product of query and key, can obtain:
9m kn = (R Wa "R Wax.) = xm W. R m WAX.
R g In this way, RoPE naturally incorporates relative position information through rotation matrix product. In general, the self-attention enhanced with RoPE can be written
where R R.n-m (R
EN (Rg m p(qm)) ΣN p(qm)) (kkk))))
T (kkn)) Vn
Attention Q, V)m
where o(-), sp()) are usually non-negative functions, e.g. elu(-)+1, or exp(-). Overall, unlike the traditional approach of adding positional information before the query and key projection, RoPE utilizes a rotary matrix to transform query and key embeddings, effectively leveraging the geometric properties of vectors.
Mixture of Distributions. As described in (Flunkert et al. 2017). ), our model predicts the parameters of probability distribution, specifically mixture of Student-t distributions. The probability density function for a random variable x following a Student-t distribution is given by:
11
(.;UppT)
r (T)
TUT
with parameters 0, E R. 0 represent the degrees of freedom (df), location, and scale coefficient, respectively, and I' is the gamma function. We modeled the predicted distribution using mixture of four Student-t distributions. Our baseline model employs independent linear layers
18

---

Published as a conference paper at ICLR 2025
ID
OOD
Loss Convergencer GM vs. STM
Regiiie Loghkstrod
GH
GM
STN
SIN
Training Loss
10K
3M
10M
100M
1K
10K
3M
10M
100M
Model Size
Model Size
0.0
CM
1.0
5TM
STM
STM
200
400
600
900
1000
1K
10k
TM
10M
100M
1K
10k
1M
10M
100M
Epochs
Model Sime
Model Size
Figure 9: Gaussian mixture (GM) distribution vs. Student-t mixture (STM) distribution. Left: GM generally pnder-performs compared to STM in both and OOD test data, particularly as model size increases. Right: GM exhibits unstable loss convergence during training, while STM achieves smoother convergence. These findings underscore the prevalence of long-tail distributions in real- world time series data and highlights the crucial role of Student-t mixtures in effectively modeling these distributions.
predict the distribution parameters: degrees of freedom, location, scale, and mixture weights. To ensure positive values for the scale and df, we apply soft-plus transformation. The mixture weights are constrained within the probability simplex using soft-max function. To avoid undefined variance for low df values, we impose a lower bound of on the df parameter.
Additionally, we conducted experiment compare the performance of Gaussian mixture distri- bution and Student-t mixture distribution when used as prior distributions to approximate real-world time series. Figure shows that Student-t mixtures generally outperform Gaussian mixtures, partic- ularly as model size increases. Furthermore, we observe unstable convergence when using Gaussian mixture distribution as the prediction head. These findings highlight the prevalence of long-tail distributions in real-world time series data and emphasize the critical role of Student-t mixtures in effectively modeling such distributions.
Encoder-only Transformer. The encoder-only Transformer has proven effective for time series tasks (Nie et al. 2023). In input time series, the future horizon replaced by learnable mask tokens combined with position information. After passing through bidirectional attention blocks, the future representations are derived from these mask tokens and mapped to the parameters a mixture distribution. Through sampling from the mixture distribution, can get predictions for future patches.
Decoder-only Transformer. The key difference in the decoder-only Transformer is its unidirec- tional attention. In the input sequence, no masking is needed. The future horizon is predicted based on the preceding token. In other words, given sequence of input patches, the model optimized to predict the next patch as function of all past patches. Similar to LLMs this can be done in parallel over the entire context window, and automatically enables the model predict the future after having seen varying number of input patches.
C IMPLEMENTATION DETAILS
C.I TRAINING SETUP
Based the constructed datasets of different sizes, we trained a series of encoder-only Transformers and decoder-only Transformers with increasing data sizes and model parameters. To improve batch processing efficiency and handle varying sequence lengths, we employ sequence packing (Raffel et al., 2020), which reduces padding requirements. Moreover, train models with forecasting capabilities over varying context and prediction lengths. Rather than defining a fixed context and prediction length, we sample 15% 50% lengths forecast horizon and the remaining context horizon, for a given time series. As most probabilistic forecasting models, we minimize the neg- ative likeeioll of the predicted patch with respect the ground truth. The loss convergence
19

---

Published as a conference paper at ICLR 2025
Encoder
Decoder
4.0
4.0
100M
100M
3.5-
30M
30M
Training Loss
Peining LOsS
10M
3.0
10M
3.0-
2.5
1M
1M
2.0
100K
100K
2.0
1.5
200
400
600
800
1000
200
400
600
800
1000
Epochs
Epochs
Figure 10: Convergence Process of Training Loss. This figure show the training loss convergence for encoder-only (left) and decoder-only (right) models of various sizes, ranging from 1K to 1B parameters. Larger models demonstrate faster convergence and lower final training loss, highlighting their improved capacity for optimization.
process of encoder-only models and decoder-only models is shown in Figure 10 As the number of parameters increases, both encoder-only and decoder-only models tend converge faster to lower training loss.
C.2 EVALUATION SETUP
We primarily used the LSF (Wu et al. 2023) and Monash (Godahewa al. (02T) datasets to eval- uate out-of-distribution generalization ability. For the LSF dataset, only used the test set and performed evaluations in a non-overlapping rolling window fashion, with the stride equal to the prediction length. For the Monash dataset, considering its large size, set only window per sample for evaluation. details the evaluation setup per dataset.
Following the statistical method in (Qiu al. 2024), compute several statistical characteristics of the datasets used in this study. Table presents the mean statistical features of the pre- pre-training, LSF, and Monash datasets. SNR refers to the signal-to-noise ratio, which quantifies the proportion of meaningful signal relative to random noise in a time series. Shifting captures changes in the probability distribution of a time series over time, with values closer indicating more severe distributional shifts. Stationarity describes a property of a time series where the mean remains constant over time, and the variance finite and stable across observations. Transition reflects the regular and identifiable fixed features within a time series, such trends, periodicity, or other predictable patterns. Overall, our pre-training data and OOD test data both show a high quality and stationarity.
Table 3:
Evaluation Datasets.
Dataset
Domain
Frequency # Prediction Length
Samples.
ETThl
Energy
175
ETTh2
Energy
H
192
819
ETTml
Energy
H
192
819
ETTm2
Energy
H
192
819
Electricity
Energy
H
192
7062
Weather
Climate
1029
M4 Hourly
Finance
414
M4 Daily
Finance
D
14
4227
M4 Weekly
Finance
W
13
359
20

---

Published as a conference paper at ICLR 2025
Table continued from previous page
Dataset
Domain
Frequency
# Prediction Length
Samples
M4 Monthly
Finance
M
48000
M4 Quarterly
Finance
Q
24000
MI Monthly
Finance
M
18
617
M3 Monthly
Finance
M
18
1428
M3 Other
Finance
Q
174
NN5 Daily
Finance
NN5 Weekly
Finance
W
111
Tourism Monthly
Finance
M
24
366
Tourism Quarterly
Finance
Q
8
427
CIF 2016
Finance
M
15
Traffic Hourly
Transport
H
168
862
Rideshare
Transport
2304
Saugeen
Climate
D
30
Sunspot
Climate
D
30
Temperature Rain
Climate
D
30
32072
Vchicle Trips
Transport
D
30
329
Weather
Climate
3010
Parts
Sales
2674
FRED MD
Finance
M
12
107
Pedestrian Counts
Transport
H
12
66
Hospital
Health
M
12
767
Covid Deaths
Health
D
30
266
KDD Cup 2018
Energy
H
168
270
Bitcoin
Finance
Us Births
Health
D
30
Table 4: Mean Statistic Characteristics.
Dataset
SNR
Shifting
Stationarity
Transition
Pre- Pre-training
25.47
0.33
0.47
0.03
LSF
19.78
0.15
0.50
0.01
Monash
19.84
0.34
0.38
0.09
C.3 EVALUATION METRICS
Mean Absolute Percentage Error (MAPE). MAPE is a commonly used metric to evaluate the accuracy of forecasting model by expressing the error in percentage terms, making it unit-free and easily interpretable. measures the average magnitude of the absolute percentage error between the predicted and actual values, relative to the actual values. For uni-variate time series, the error is defined as:
(6
(8)
(8)
yj
13
21

---

Published as a conference paper at ICLR 2025
( where yj and yj are the target and predicted values of the i-th time series and j-th time step, respectively. The MAPE of the i th time series is then calculated as:
100 I+H H
MAPE
This formula calculates the percentage error at each time step, averaging it over the forecast horizon H. MAPE is intuitive and interpretable in percentage terms. However, MAPE can be highly sensi- (6) leading extremely large errors when actual tive small actual values because it divides by 1/ values approach zero.
Symmetric Mean Absolute Percentage Error (SMAPE). SMAPE addresses the shortcoming of MAPE by using symmetric formula that normalizes both the actual and predicted values. The SMAPE of the i-th time series defined to be
( CJ
SMAPE 200 H
Σ
6 + M9)
The SMAPE ensures that errors are balanced between over- and inder-prediction by dividing by the sum of the absolute actual and predicted values. This normalization helps avoid extreme errors when actual values are small, but it can also produce undefined results when both actual and predicted values are zero.
Mean Absolute Scaled Error (MASE). MASE is scale-independent metric used to evaluate fore- casting accuracy by comparing the absolute error of forecast the average error of naive fore- casting method, such as a one-step-and step-ahead random walk. MASE provides robust way to assess model performance across different datasets with varying scales, addressing the limitations met- rics like MAPE that are sensitive to the magnitude of actual values. For a uni-variate time series, the MASE the i-th time series calculated as:
1 1
I+H
(8)
MASE
H
1+1
7-1 ΣΣ T (2)
yk (
2
where 10/ 8k-1 represents the difference between consecutive values in the naive forecast. MASE particularly useful because accounts for the variability and scale of the time series, making it comparable across datasets. Unlike MAPE, MASE does not produce extreme values for small zero actual values, making more stable and reliable for time series evaluation.
Continuous Ranked Probability Score (CRPS). Before we can introduce the CRPS, we need to introduce the weighted quantile loss Park et al. (2022), which is a metric normalized over the test set. We first define the a-quantile loss, also known as the pinball loss at quantile level a, to be:
Aa(g. v) (a 1y< -4)
The weighted quantile loss is then the normalized sum of quantile losses,
WQL[a] = Eniyen (a)
EaSer (0,
where Ω - (i.5) E :: 1 T.J.
The CRPS is proper scoring rule Matheson & Winkler 1976). meaning that it is minimized when the predictive distribution is equal to the distribution from which the data is drawn.
CRPS = 1'' (F-1 (a),v) do
22

---

Published as a conference paper at ICLR 2025
4008
13009
INE
13003
12008
19008
9008
Figure 11: Emergent Behaviors us birth Dataset. The sub-figures, arranged from left to right and top bottom, show prediction results for models of sizes 10K, 1M, 10M, and 100M. Models with sizes 10K and 1M are unable to capture the periodic pattern, whereas models with sizes 10M and 100M accurately reflect it.
However, we are unable to evaluate this quantity since we generally are not able to compute the integral in closed form and only have access finite number of quantile predictions. The approx- imation of the CRPS is an average of the weighted quantile loss over K quantiles, and thus is also known as the mean weighted quantile loss.
CRPS
wQL.(ar)
D ADDITIONAL RESULTS
D. EMERGENT BEHAVIORS
Figure &iguee&preeeetts presents studies "emergent abilities" TfTSMMs for OOD predictions. As model size increases, one would expect the MAPE to follow a smooth power-law decline, indicating continuous improvement. However, the plots reveal distinct phenomena, where performance changes abruptly rather than gradually. Figures IT 1 visually show the transition from the small model prediction results to the large model prediction results. We see that the 10K and 1M models are unable to capture the historical seasonal patterns to make accurate predictions; however, when the model size reaches 10M, it can make accurate predictions, far better than the small model. This behavior akin to the "emergent behaviors" described in (Wei et al. (202)). where certain capabilities absent in smaller models only appear in larger ones. The emergence improved performance certain model sizes could indicate the development of higher-order patterns or representations within the model. These emergent abilities might stem from the model's capacity to recognize and generalize complex temporal patterns that were previously inaccessible at smaller scales. Such abilities is evident particularly in zero-shot OOD prediction, where the model must extrapolate from training data to entirely unseen scenarios.
23

---

Published as a conference paper at ICLR 2025
20
Figure 12: Emergent Behaviors in nn5 daily Dataset. From left to right and top to bottom, the figures illustrate the prediction outcomes of models with sizes 10K, 1M, 10M, and 100M, spectively. While the smaller models (10K and IM) fail to recognize the periodic pattern in the data, the larger models (10M and 100M) effectively capture and reflect the periodic structure.
1200
13300
0000
8003
4000
E300
Figure 13: Emergent Behaviors in australian electricity Dataset. From left to right and top to
bottom, the sub-figures display the prediction results of models with sizes 10K, 1M, 10M, and 100M.
The models with sizes 10K and 1M fail to accurately capture the periodic pattern, while those with sizes 10M and 100M successfully do so.
24

---

Published as a conference paper at ICLR 2025
100M
2.8
30M
10M
2.6
2.4
300K
2.2
100K
10K
2.0
1K
10
10'
Time Points Processed
Figure 14: The figure illustrates the validation NLL during training for encoder-only models of different sizes. Larger models tend to achieve lower NLL with processing the same time points.
6.3
7.2
6.2
Nognive Log Weellhood
2,3
3.2
2000 Lolloppee
Maraeh: UG 1L4 1019ep000
Horzat: (p.--1pfoopm
ID: WLL 10theyy
190120 12D H.B. Woppr
LSF: u (54 Lohoppe
LSR 12W 15.1 10
uci 12
1D LIDt (12- -101-122-10 Yap
10
10
10"
10
10
10
10
10"
Model Sizee
Compate
Damset Size
Figure 15: Scaling Pattern of NLL. This figure illustrates scaling laws for NLL in relation to model size, compute, and dataset size. The blue lines represent ID performance, while the red and green lines show OOD performance on LSF subset and Monash subset.
D.2 SAMPLE EFFICIENCY
Figure 14 illlustrates the evaluation results the ID data for encoder-only models of different sizes during training. The key observation is that larger models achieve better performance with fewer training steps compared smaller models, demonstrating higher sample efficiency. This is impor- tant in scenarios where data are limited, as larger models can achieve superior performance without needing process much data. An explanatory intuition is that, when the number of parameters large, even a small change in parameters can cause a large change in outputs. From a theoretical perspective, the Neural Tangent Kernel (lacot al. 2018) framework provides insights into why this occurs. As neural networks become infinitely wide, their behavior approximates that of a kernel method, enabling them effectively minimize the data distribution's 's generalization error. This al- lows large models to generalize well with fewer samples, further supporting their observed sample efficiency in empirical settings. Moreover, NLP (Kaplan et al.] (020)) and CV (Zhai et al. 2022) fields, larger models have demonstrated higher sample efficiency than smaller models.
D.3 SCALING PATTERN DEPENDS ON DATA DISTRIBUTIONS
In Figures 15 19 we present the scaling laws evaluated on the Monash subset, along with results from the LSF subset and the ID test data. By comparing the green line, representing the Monash dataset, and the red line, representing the LSF dataset, we observe relatively consistent offset and similar slopes across various metrics. This suggests that, when transferring the model from the training data distribution to other OOD distributions, there a predictable decrease in performance that varies by the target dataset. However, the model's gains from scaling -whether by increasing model size, compute, or dataset size follow a fixed proportional relationship. This indicates that while OOD performance degrades, scaling the model still yields consistent improvement ratio across different distributions, albeit with varying levels absolute performance. Moreover. in the future, further analyze the reasons why transfer cost arises and establish law predict the
25

---

Published as a conference paper at ICLR 2025
3.1
21
1.1
3.0
2.0
1.0
Momasts 4/00 (8.2 20flyy sotvegsane
Monset: (OC) In (1.D sereer
Mamant: 15.8 10%
LSF: -18 20r2/ggyla
LID1 (72 10:22
0.1
0.1
10
10
10
Compute days]
Untnoet Size
Figure 16: Scaling Pattern of MAPE. This figure presents scaling laws for MAPE functions of model size, compute, and dataset size. ID performance is illustrated by the blue lines, while OOD performance depicted red for the LSF subset and green for the Monash subset.
0.8
1.0
0.6
0.6
0.4
SMAPE
0.4
0.3
0.4
0.2
Monast: 100 Sapher
UD) (000:w62:::
0.2
Lortgge
UO (8.1
Dr cldh -1.0--12--0 30-1100000
0.1
10
10
10
10
10
10
10
10
10
10 10
10"
10
Model Siree
Computc [PF-any)
Datsous Size
Figure 17: Scaling Pattern of SMAPE. This figure depicts the scaling laws for SMAPE with re- spect to model size, computational resources, and dataset size. ID performance is marked by blue lines, while red and green lines correspond to OOD performance on the LSF and Monash subsets, respectively.
transfer performance loss in population-risk based theoretical analysis Wasserstein measurement.
YYng et al. 2021) through
D.4 SCALING PATTERN DEPENDS ON PERFORMANCE METRICS.
We investigate the scaling behaviors of five common performance metrics: NLL, MAPE, SMAPE, MASE, and CRPS, (as shown in Figures 15 19 All metrics exhibit decreasing trend following an approximate power-law; however, each metric demonstrates distinct scaling characteristics, re- flected in their varying power-law exponents. Previous work by (Ghorbani al. 2021 establishes relationship between large language models' log- ligeliied loss and the BLEU score in translation tasks. Similarly, future research could explore transformation between log- lilkkiikkoood loss and time series forecasting metrics, offering means to predict forecasting performance from training loss.
E FURTHER DISCUSSION
Scaling Laws for Multivariate Time Series Forecasting, Extending our findings multivariate time series forecasting is promising but presents some challenges. First, large and diverse multivari- datasets are limited, unlike the more standardized uni-variate time series datasets. This scarcity makes it difficult to conduct experiments that scale both model size and data volume in controlled manner. Second, establishing scaling laws for multivariate time series require analyzing the impact of variable count and correlation strength, increasing the experimental complexity. Third, multivari- ate foundation models lag behind uni-variate ones, facing challenges in designing architectures that accommodate variable counts and capture complex inter-variable dependencies.
Pre-training Data Mixing Strategy. It's significant to investigate the impact of data mixing strate- gies on model performance, as different strategies introduce performance bias or degrade model performance. Time series pre-training datasets, often comprising data from dozens to hun- dreds of sources, create exponentially large possibilities for combinations. Instead of exhaustive
26

---

Published as a conference paper at ICLR 2025
2.0
1.0
0.4
0.2
Monash: LIM (2.5 101m7311
Munsolt 6ld) - 20
Monash 0006 (3.5 (3.5-19-07
12: LIM) 12.5 10 2onttggeas
LSF (IC) (5,8 -1---ycc
LSP: 2000 1 122-13 "oooe
17.-10 "ypphen
10 L101-123 10
0.0
10
10
10"
Stze
Compute (PF-days)
Dotnsct Size
Figure 18: Scaling Pattern of MASE. The figure shows how MASE scales with model size, com- pute, and dataset size. ID performance is indicated by blue lines, whereas OOD performance is highlighted in red and green for the LSF and Monash subsets.
2.1
2.0
1.1
CRPS
10
(D: (2.8 Sottgyeae
LSR: (OC) i 10-1NAAA
LBB) (112 1200py14
LSP LDBE w 10h0y
1DD 12.4 1244-100-00
ID: (82)) 11.5 10/12/440
10
10
10
10
10
10
10
10
10
10 10
10
10
10
10
Modal Size
Compuute [PF-doys]
Dattee: Size
Figure 19: Scaling Pattern of CRPS. The figure illustrates the scaling laws for CRPS in relation to model size, compute, and dataset size. Blue lines represent ID performance, while red and green lines depict OOD performance on the LSF and Monash subsets, respectively.
empirical research, a more practical approach is establish an extrapolable analytical framework that balances experimental complexity (Hashimoto) 2021] Ye et al.) 2024). Additionally, deeper analysis of time series dataset characteristics is essential. This would provide insights into the na- ture of time series data and facilitate the development of better data organization formats, moving beyond simple combination strategies.
Impact of Model Architecture on Scalability. Our research comparing the scalability of encoder- only and decoder-only Transformers can serve as the groundwork for future exploration of other architectures. CNN-based models (Wu et al. (022). RNN-based models (Gu al., (021), and hybrid models (Lieber et al. 2024) each demonstrate unique strengths in sequence modeling. In- vestigating their scalability within the context of time series modeling is promising direction for future research, and we hope our findings will inspire further work in this area.
Impact of Specific Module Scalability. Our studies the scalability of Chronos and Moirai demonstrate that certain module designs in these models may hinder their scalability in OOD settings. Key factors, such as the time series embedding approach, attention mechanism. positional encoding, and forecasting strategy, all impact scalability to varying degrees. Among these factors, the embedding approach particularly crucial, as it directly determines the input patterns fed into the model backbone and influences the operations the model can learn.
Theoretical Framework for Scaling Laws. Our scaling laws, similar most scaling laws, are empirical findings. We believe theoretical understanding of the learning dynamics that form the laws can provide more solid justification. A potential perspective is understanding the optimization process through dynamics modeling (Bordelon et al., 2024). Specifically, based the dynamical mean field theory, we can derive the infinite limit statistical description of the scaling behavior of training factors. By analyzing the response function in the infinite limit, which gauges the correlation between training factors and test error, we can obtain insights into how the scaling laws manifest.
Correlation Estimation for Metrics. Time series forecasting involves diverse set of metrics, each measuring prediction error from a unique perspective. However, due to the lack of clear and definitive correlations between NLL loss and these metrics, it's difficult understand model predic-
27

---

Published as a conference paper at ICLR 2025
tion behavior directly from NLL loss. A potential perspective is to develop a prediction framework that bridges these metrics by empirically analyzing their relationships and dependencies (Isik et al.) )024). Such framework would provide insights into how different metrics reflect model perfor- mance under varying conditions, enabling a deeper understanding of prediction behavior.
28


Reviews:
[{"review_id": "dDxv0xN62P", "reviewer": "Reviewer_oPfC", "summary": "The paper analyzes two common TSFM architectures -- encoder-only and decoder-only transformers -- in terms of both in-distribution and out-of-distribution data. The results show that encoder-only can better scale to both ID and OOD data. The authors also show that specific architecture choices in existing TSFMs could negatively affect OOD generalization.", "questions": "1. For the training details on line 183, I am not sure if I understood correctly. If one dataset is much larger than the other datasets, are we much more likely to sample from this dataset? Is there a cap to this likelihood? In existing works such as MOIRAI and Timer, they apply some threshold that prevents a large dataset from dominating from the training distribution. In addition to \"Domain Balancing\" as mentioned in Appendix A in your paper, I think balancing the datasets within each domain should also be helpful. Could you compare the different data mixing strategies?\n2. Why does OOD have a larger exponent for MAPE than ID in Figure 2 and 3, but not for log-likelihood? It feels to me that it might be due to the type of data distribution you have.\n3. In Timer, the findings are decoder-only scales better than encoder-only. Could you explain why your finding is different? In Timer, the batch size is chosen to be 8192, but you only used a batch size of 128. Is it possible that using a larger batch size would change your observations and make the decoder-only model more favorable?\n4. Did you try retraining MOIRAI/Timer/Chronos using your filtered dataset? The dataset mixture is often a very important factor. Not using the same pretraining datasets can be an unfair comparison in Section 3.2.\n5. \"Scaling-laws for Large Time-series Models\" suggests that model size has a big impact on the optimal learning rate. What learning rate did you use and how did you change it for different model sizes? Are the observations consistent for encoder-only and decoder-only models?\n\nMinor:\n- \"Lotsa\" should be capitalized on line 94.\n- \"Sum\" in Table 1 is a bit confusing. Consider replacing it with \"Total\" and use a vertical line to separate it from previous columns.\n- In Figure 2 and 3, consider adding ID and OOD to the legend, in addition to using the colors.\n- Line 794: \"and Covid19_Energy datasets\", the underscore here seems to be a typo?\n- \"Scaling-laws for Large Time-series Models\" suggests that using a separate head for each distribution parameter should stabilize training. \n- Could you add an experiment to show the benefit of using student-T mixture versus gaussian mixture?\n\nOverall, I believe the findings in this paper will be very helpful to the community. I will raise my score if my concerns regarding the training and evaluation setups are addressed.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The paper flows very well and is easy to follow.\n2. While most existing TSFMs for forecasting choose a decoder-only structure, this paper presents new findings on the encoder-only versus decoder-only that contradicts prior findings, and analyzed it from the scope of OOD data.", "weaknesses": "1. Evaluation setup could be improved. On line 130, \"This subset includes test data from the ETTh1-2, ETTm1-2, electricity, and weather datasets.\" Are these the only datasets that are used as OOD test data throughout all the experiments? To strengthen your findings, I suggest varying the OOD datasets (maybe include a confidence interval), and see if your findings still hold.\n2. The paper could benefit from explaining why its findings are different from those of the existing works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jW3PP7LIKu", "reviewer": "Reviewer_sVAF", "summary": "8648_Towards_Neural_Scaling_La\npdf\nLT\nHere is a new  paper, please read it in its entirety, and help me to write the review, first, you should:\nSummary*\nBriefly summarize the paper and its contributions. This is not the place to critique the paper; the authors should generally agree with a well-written summary.\n\n\n2 / 2\n\nSummary\n\nThis paper investigates the scaling laws of time series foundation models (TSFMs) across different data distributions and model architectures. The authors train encoder-only and decoder-only Transformer-based TSFMs with varying model sizes, compute budgets, and dataset sizes. They evaluate the models' performance on both in-distribution (ID) and out-of-distribution (OOD) data.\n\nThe key findings and contributions of the paper are as follows:\n\nThe log-likelihood loss of TSFMs exhibits similar scaling behavior in both OOD and ID scenarios, following a power law with respect to model size, compute resources, and dataset size.\nWhen evaluated using MAPE, the scaling of all three factors (model size, compute, and data) results in greater improvements in OOD performance compared to ID performance.\nEncoder-only Transformers demonstrate better scalability than decoder-only Transformers, although both architectures show similar scaling trends.\nThe architectural modifications introduced in two state-of-the-art TSFMs, Moirai and Chronos, primarily improve ID performance but compromise OOD scalability compared to the baseline models.\nThe paper provides practical design principles for TSFMs from the perspective of data, model, and compute scaling, based on the findings and comparative analysis.\nOverall, this work extends the understanding of scaling laws for TSFMs from ID to OOD scenarios and across different model architectures. The authors establish a foundation for predicting the expected performance gains of TSFMs and offer insights for designing scalable TSFMs.", "questions": "1. Justification for the chosen model architectures: Can the authors provide a more detailed explanation for why they selected encoder-only and decoder-only Transformers as the main focus of their study? While these architectures are indeed popular in TSFMs, it would be helpful to understand the rationale behind this choice and whether the authors believe their findings would generalize to other architectures, such as encoder-decoder models or hybrid models.\n2. Potential for theoretical analysis: The paper primarily relies on empirical observations to establish the scaling laws for TSFMs. Have the authors considered complementing their empirical findings with a theoretical analysis? If so, what challenges do they anticipate in developing a theoretical framework for the observed scaling behaviors, and how might such a framework enhance our understanding of the underlying mechanisms governing the relationship between model size, compute, data, and performance?\n3. Factors contributing to the ID-OOD performance trade-off: The paper highlights that architectural modifications in Moirai and Chronos improve ID performance but compromise OOD scalability. Can the authors provide more insights into the factors that might contribute to this trade-off? For example, are there specific architectural choices (e.g., attention mechanisms, embedding strategies) that could be more prone to overfitting on ID data, leading to reduced OOD scalability? Understanding these factors could help guide the design of future TSFMs.\n4. Computational efficiency considerations: While the paper considers compute budgets as a scaling factor, it does not provide a detailed analysis of the computational efficiency of the investigated models. Can the authors comment on the practical trade-offs between performance and computational cost for the different model architectures and sizes? Including metrics such as training time, inference time, and memory consumption could provide valuable insights for practitioners looking to deploy TSFMs in resource-constrained environments.\n5. Generalizability to multivariate time series: The paper focuses exclusively on univariate time series forecasting to avoid the confounding effects introduced by multivariate time series. Can the authors discuss the potential challenges and opportunities in extending their findings to multivariate TSFMs? What additional factors (e.g., variable interactions, correlations) might need to be considered when investigating scaling laws in the multivariate setting?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The study extends the existing research on scaling laws for TSFMs in a novel direction by investigating their behavior across different data distributions (ID and OOD) and model architectures. While previous works have primarily focused on ID scenarios, this paper breaks new ground by systematically examining the scaling properties of TSFMs in OOD contexts. Furthermore, the comparative analysis of encoder-only and decoder-only Transformers, as well as the inclusion of state-of-the-art TSFMs (Moirai and Chronos), provides original insights into the impact of model architecture on scalability.\n2. The methodology employed in this study is sound and comprehensive. The authors train a wide range of models with varying sizes (from 10^3 to 10^8 parameters), compute budgets, and dataset sizes. They evaluate the models' performance using appropriate metrics such as log-likelihood loss and MAPE. The experiments are well-designed, covering both ID and OOD scenarios, and the results are presented clearly with informative visualizations. The analysis of the scaling behavior is thorough, considering the effects of model size, compute resources, and dataset size independently.\n3. The findings of this paper have significant implications for the development of TSFMs. By establishing scaling laws that hold across both ID and OOD scenarios, the authors provide a framework for predicting the expected performance gains of TSFMs as they scale up in size, compute, and data. This is crucial for guiding resource allocation and model design decisions. Moreover, the comparative analysis of different model architectures reveals important trade-offs between ID and OOD performance, highlighting the need for careful consideration when designing TSFMs for specific applications. The practical design principles derived from the study offer valuable guidance for researchers and practitioners working on TSFMs.", "weaknesses": "1. Limited scope of model architectures: The study focuses on two main model architectures: encoder-only and decoder-only Transformers. While these are indeed widely used in TSFMs, the paper could benefit from including a broader range of architectures, such as encoder-decoder models (e.g., Seq2Seq) or hybrid models that combine Transformers with other neural network components (e.g., CNN, RNN). Expanding the scope of the investigated architectures would provide a more comprehensive understanding of how architectural choices affect scalability.\n2. Lack of theoretical analysis: The paper primarily relies on empirical observations to establish the scaling laws for TSFMs. While the experimental results are valuable, the work could be enhanced by providing a theoretical foundation for the observed scaling behaviors. Theoretical analysis could help explain why certain scaling patterns emerge and offer insights into the underlying mechanisms that govern the relationship between model size, compute, data, and performance. For example, the authors could explore the connection between the power-law scaling and the model's capacity to learn and generalize from the training data.\n3. Limited discussion on the trade-offs between ID and OOD performance: The paper highlights that architectural modifications in Moirai and Chronos improve ID performance but compromise OOD scalability. However, the discussion on this trade-off is relatively brief. A more in-depth analysis of the factors contributing to this trade-off and the potential implications for TSFM design would be valuable. For instance, the authors could investigate whether certain architectural choices (e.g., attention mechanisms, embedding strategies) are more prone to overfitting on ID data, leading to reduced OOD scalability.\n4. Absence of computational efficiency analysis: While the paper considers compute budgets as a scaling factor, it does not provide a detailed analysis of the computational efficiency of the investigated models. As TSFMs scale up in size, computational efficiency becomes increasingly important. The paper could be strengthened by including metrics such as training time, inference time, and memory consumption for the different model architectures and sizes. This would provide valuable insights into the practical trade-offs between performance and computational cost.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZSinUJ6ONc", "reviewer": "Reviewer_CUNM", "summary": "The authors examine the scaling laws of Time Series Foundation Models (TSFMs), both encoder-only and decoder-only Transformers, on in-distribution (ID) and out-of-distribution (OOD) data. They find that the NLL (and to some extent the MAPE) of TSFMs shows similar scaling behavior in both ID and OOD settings, scaling as a power law with model parameters, compute, and dataset size. Results suggest that encoder-only Transformers exhibit better scalability than their decoder-only. The authors propose design principles for building scalable TSFMs, focusing on data, model architecture, and compute, recommending encoder-only Transformer architectures for better scalability. Authors also study \"emergent behaviors\" in TSFMs, where performance on certain OOD datasets improves abruptly after reaching a critical model size.", "questions": "- Section 3: The MAPE is consistently used as a metric in the main results, although Appendix C.3 outlines a well-known limitation, which is that it is unreliable when the time series takes values close to zero (and suggests using the sMAPE instead). It seems that for time series foundation models, which should be quite agnostic to shift and scale variations of the series, the MAPE should be the last metric to be considered. Why has not the sMAPE been used, or more robust scale-invariant methods like the MASE? (https://en.wikipedia.org/wiki/Mean_absolute_scaled_error )\n- Figures 2-4: Although the NLL difference between ID and OOD is of the expected sign (i.e. with OOD having greater NLL than ID), it is truly mystifying that the opposite is consistently true for the MAPE. This is not properly addressed by the authors. In my view, this is the main issue that is causing my \"Soundness\" assessment not to be higher.\n- Figure 5: Although encoder-only clear beat decoder-only in distribution, this difference is almost insignificant out of distribution, in contrast to the claim in the caption. If the authors intend to make this claim, they should conduct a formal statistical test of the fit difference between encoder-only and decoder-only models.\n- Suggestion related to Figure 8: across these examples, it'd be instructive to plot (in appendix) what the actual forecasts look like as we transition from a regime of \"relatively higher MAPE\" to \"relatively lower MAPE\".", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- Originality: The paper's focus on the scaling laws of TSFMs on OOD data represents a novel and timely contribution.\n- Quality: The research is well-executed, with a comprehensive methodology that involves training and evaluating a wide range of models across various parameter counts, compute budgets, and dataset sizes. The use of a large and diverse dataset further strengthens the quality of the empirical analysis.\n- Clarity: The paper is well-written and organized.\n- Significance: Author's findings could have significant implications for the development and deployment of TSFMs, subject to comments and suggestions in the remainder of this review.", "weaknesses": "- Line 046: provide citations for neural scaling laws that provide ground for believing that they exist.\n- Table 1: review the \"proportion\" row against the \"Time points\" row. Sales is 0.96% of dataset with 140M points, whereas Web is 0.40% of dataset with 600M points.\n- Line 149: \"Given its proven effectiveness in improving time series forecasting performance (Woo et al., 2023), we adopt RoPE as a replacement for the original Transformer’s positional encoding.\" ==> my reading of Woo (2023) is that the evidence in favor of RoPE for time series forecasting is very weak (results presented without confidence bands, and do not appear statistically significant), and likely that better evidence should be presented than the statement of fact that RoPE is \"proven\" — I don't see that it is, especially in the context of a foundation model.\n- Line 190: The paper consistently talks about \"log likelihood\", for which higher values indicate a better fit. However, all results seem to be obtained under the \"negative log-likelihood\" (NLL, which is the loss counterpart), for which lower values indicate a better fit. This elementary terminology confusion should be clarified corrected.\n- Section 3: The paper only studies Transformer-based architectures, without any baseline results with simple time series models (e.g. ETS-type exponential smoothing or ARIMA) — which could be shown as horizontal lines in the scaling law plots. In particular, the MAPE is around or below 1% for most of the studied scaling regime, which suggest that the datasets are very highly predictable: it is likely that simpler models would do well on these datasets too, bringing into question whether TSFMs' additional complexity is warranted.\n- Appendix C.2: Since the training and test sets are taken from long-available public sources, is there a contamination analysis of the test set against the training set to ensure there's no overlap?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OTIta0jGu8", "reviewer": "Reviewer_5SSN", "summary": "This paper examines how time series foundation models scale, focusing on the performance of both encoder-decoder and decoder-only transformer architectures in in-distribution (ID) and out-of-distribution (OOD) settings. Through a series of experiments, the authors analyze how changes in model size, computational power, and dataset volume impact these models' effectiveness. One major contribution of this study is the adaptation of existing scaling laws to OOD scenarios, providing a framework to anticipate performance improvements in OOD forecasting based on key training variables. The paper also reveals that encoder-only transformers generally perform better and scale more effectively than decoder-only models. Additionally, the authors offer practical guidelines for building robust, scalable time series foundation models, emphasizing the value of training data diversity, model size, and architectural choices to boost both performance and generalization.", "questions": "Please check the Weaknesses section for more questions!", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "As to the originality, the authors explore a relatively new area in time series forecasting by expanding the concept of scaling laws from in-distribution (ID) settings to out-of-distribution (OOD) contexts. This fresh perspective enhances the relevance of scaling laws and addresses an important gap in understanding how model architectures perform across various data distributions. By comparing encoder-only and decoder-only transformer architectures, the paper merges ideas from neural scaling laws with practical applications in time series analysis, offering new insights into model design! \n\nThe work quality is underscored by the rigorous experimental methods the authors use. They systematically evaluate various time series foundation models across parameters such as model size, computational resources, and dataset dimensions. This comprehensive approach ensures the findings are reliable, providing a strong basis for the paper's conclusions. Using clear performance metrics, like log-likelihood loss and MAPE, adds credibility to the results! \n\nThe paper's clarity is another strength. Concepts are explained in a structured and accessible way, enabling readers from diverse backgrounds to follow the research. The logical flow and transparent presentation of methods and findings make the implications of this work easy to understand. Detailed discussions on scaling behaviours among different architectures further clarify the complex relationships between model design and performance! \n\nThis work is significant for both theoretical and practical reasons. It advances the understanding of time series foundation models and provides empirical insight and design principles for scaling, benefiting researchers and practitioners in developing more effective forecasting models. The insights offered could influence future studies and propel advancements in time series forecasting and neural network design!", "weaknesses": "(1) One notable limitation in this study is its minimal exploration of how different model architectures impact scalability. Although the paper briefly acknowledges architecture's role in scaling, it falls short of offering a thorough analysis on how specific architectural choices, like attention mechanism, layer normalization, or residual connections, might influence scaling behaviour. A deeper, comparative study covering a broader set of architectures could reveal which design choices are most beneficial for performance across diverse data distributions! \n\n(2) The study's experiments center on log-likelihood loss and MAPE as key performance metrics. While these are certainly relevant, they do not fully capture the complexity of model performance in time series forecasting. A wider range of evaluation metrics, like the Continuous Ranked Probability Score (CRPS) or quantile loss, could offer a more comprehensive view of performance and its practical impact. Although the paper briefly suggests future research might investigate transformations between log-likelihood and forecasting metrics, including preliminary results or further discussion on this topic within the study would enhance the depth and clarity of the findings! \n\n(3) The paper would benefit from a deeper exploration of why performance tends to degrade on OOD data. Although the authors highlight that scaling the model improves OOD performance, they stop short of examining the mechanisms driving this effect. An in-depth look at how the training data distribution relates to OOD performance could offer valuable insights for practitioners applying these models in a real-world context. For example, investigating domain adaptation techniques or transfer learning strategies might help reduce the performance drop seen in OOD situations! \n\n(4) The discussion on sample efficiency is insightful but would benefit from concrete examples or case studies that demonstrate how larger models achieve improved performance with fewer training steps. Including specific examples or empirical data to back up this claim would strengthen the credibility of the findings and provide practical guidance for researchers and practitioners looking to optimize their training processes! \n\n(5) The dataset used for training and evaluation seems carefully chosen but lacks a detailed account of its diversity and representativeness. While the paper highlights the role of dataset size and variety in enhancing OOD performance, it provides limited insights into the dataset's specific characteristics, such as the types of time series it includes, their distributions, or any noise or anomalies present. A deeper examination of the dataset's attributes would offer a clearer picture of the models' performance and their relevance to real-world applications! \n\n(6) Another important limitation is the lack of attention to how scaling laws impact practical applications. The authors offer theoretical insights into scaling behaviour but do not go far enough in showing how these findings can be turned into useful strategies for practitioners. For example, guidance on balancing model size, computational resources, and dataset size in real-world settings would be valuable for those aiming to apply time series foundation models effectively. Including case studies or examples of successful implementations could also help demonstrate the research's relevance to industry professionals!", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)