Towards Neural Scaling Laws for Time Series Foundation Models

## Abstract

Abstract Scaling laws offer valuable insights into the design of time series foundation models (TSFMs). However, previous research has largely focused on the scaling laws of TSFMs for in-distribution (ID) data, leaving their out-of-distribution (OOD) scaling behavior and the influence of model architectures less explored. In this work, we examine two common TSFM architectures—encoder-only and decoder-only Transformers—and investigate their scaling behavior on both ID and OOD data. These models are trained and evaluated across varying parameter counts, compute budgets, and dataset sizes. Our experiments reveal that the log-likelihood loss of TSFMs exhibits similar scaling behavior in both OOD and ID settings. We further compare the scaling properties across different architectures, incorporating two state-of-the-art TSFMs as case studies, showing that model architecture plays a significant role in scaling. The encoder-only Transformers demonstrate better scalability than the decoder-only Transformers, while the architectural enhancements in the two advanced TSFMs primarily improve ID performance but reduce OOD scalability. While scaling up TSFMs is expected to drive performance breakthroughs, the lack of a comprehensive understanding of TSFM scaling laws has hindered the development of a robust framework to guide model scaling. We fill this gap in this work by synthesizing our findings and providing practical guidelines for designing and scaling larger TSFMs with enhanced model capabilities.

## 1 Introduction

Time series analysis is an important piece of data mining, facilitating decision-making and scientific inference across various domains (Zhang et al., ). As an important analysis task, time series forecasting has long been studied and drives a wide range of practical applications, from energy, climate and quantitative finance to urban computing and system management (Jin et al., ; Nie et al., ; Wen et al., ). Various methods have been proposed for this task, ranging from classical statistic models (Hyndman & Athanasopoulos, ), bespoke dynamical models (Prado, ), to the more recent deep-learning based approaches (Wen et al., ). Despite their competitive performance, the methods are typically designed for specific tasks, poor to generalize to other domains (Fan et al., ; Rasul et al., ). Concurrently, we are witnessing a paradigm shift in time series forecasting from task-specific models to universal models, with the emergence of time series foundation models (TSFMs). Timer (Liu et al., ), Moirai (Woo et al., ), and more recently proposed Time-MoE (Shi et al., ) show trends of scaling in both data volume and model size, aiming to achieve performance breakthroughs through more resource investment.

The neural scaling law quantitatively describes how model performance grows with the scaling of three basic training factors: model parameters, computational resources and training dataset size. Establishing such scaling laws is crucial for developing TSFMs, as it provides a framework for predicting expected performance gains, enabling the community to rationally allocate efforts toward key designs. The exploration on scaling laws for TSFMs is still in an initial stage; recent research has primarily focused on studying ID scaling behavior (Edwards et al., ; Shi et al., ). In practical applications, TSFMs primarily face challenges from unseen scenarios(Wang et al., ), where OOD forecasting capability is most critical. This raises an unresolved question: do neural scaling laws also apply to predict out-of-distribution forecasting performance? Moreover, various architectures of TSFMs have been arising, but they typically focus on performance improvement at specific scales. No studies have investigated the scaling behaviors across different architectures, leaving a key question unanswered: how do model architectures affect scalability? Although we are seeing an increasing investment in training resources for TSFMs, the bottlenecks and potential driving factors for developing larger TSFMs remain unclear. This raises another practical question: how to design TSFMs from the perspective of scalability?

In this paper, we aim to provide empirical answers to the above research questions. To investigate the scaling laws in OOD scenarios, we trained a family of encoder-only Transformer-based TSFMs, varying three basic training factors: model sizes, compute budgets, and training set sizes. We evaluated their performance on both ID and OOD test sets and established scaling laws for three training factors in each scenario. To examine the impact of model architecture on scaling behavior, we trained decoder-only Transformer based TSFMs and compared them with the encoder-only versions. Additionally, we included two state-of-the-art TSFMs, Moirai (Woo et al., ) and Chronos (Ansari et al., ), as case studies for detailed analysis. Our experiment results suggest that the log-likelihood loss of TSFMs exhibits similar scaling behavior in both OOD and ID scenarios; encoder-only Transformer has better scalability than decoder-only Transformer; the architectural modifications introduced by two advanced TSFMs mainly improve ID performance but compromise OOD scalability. Based on the findings and comparative analysis, we finally provided design principles for TSFMs from a scaling perspective.

Our contributions are summarized as follows:

- •
Scaling laws across data distributions. We extend the scaling laws for TSFMs from ID scenarios to OOD scenarios across three core training factors: model size, computational resources, and dataset size, establishing a foundation for predicting expected OOD performance gains of TSFMs.
- •
Scaling laws across model architectures. We investigate the scaling patterns of different architectures of TSFMs, offering some insights for scalable TSFM design.
- •
Scaling laws-guided design principles. We provide practical design principles for TSFMs from the perspective of data, model and compute scaling, via analyzing the commonalities and differences in scaling behaviors across data distributions and model architectures

## 2 Preliminary

To investigate the scaling laws of TSFMs, we curated a large, diverse, and balanced dataset for pre-training. Leveraging this dataset, we trained both encoder-only and decoder-only transformers and two state-of-the-art TSFMs: Chronos and Moirai. For comparative analysis, we evaluated these models on (i) in-distribution and (ii) out-of-distribution test sets, focusing on key performance metrics to examine the scaling behavior across architectures.

### 2.1 Datasets

A large scale, diverse, balanced and high quality pre-training dataset is the foundation to build FMs. To this end, we constructed our time series corpus for TSFM pre-training from the large-scale open time series archive, Lotsa (Woo et al., ). The corpus comprises approximately $15$B time points from $39$ datasets spanning seven distinct domains. To ensure that the model performs fairly across all domains, we maintained a balanced ratio of data from different domains. Furthermore, we performed quality filtering on the corpus by constraining the signal-to-noise ratio of a time series to be greater than $20$ dB, ensuring that the pre-training corpus exhibits strong predictability. A detailed breakdown of the data sources is provided in Appendix A, with a summary in Table 1.

**Table 1: Dataset summary. M indicates million and B indicates billion.**
| Domain | Transport | Climate | Energy | CloudOps | Health | Sales | Web | Sum |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Datasets | 8 | 2 | 14 | 3 | 9 | 1 | 2 | 39 |
| Time points | 4.82B | 4.73B | 2.34B | 2.15B | 240M | 140M | 600M | 14.46B |
| Proportion | 33.31% | 32.71% | 16.15% | 14.86% | 1.61% | 0.96% | 0.40% | 100% |

To assess the impact of pre-training data scale on model performance, we partitioned the corpus into subsets containing $10$M, $100$M, and $1$B time points, ensuring that each subset maintained similar diversity. For each subset, $95$% of the data was allocated for model training, with the remaining $5$% reserved as a validation set to evaluate in-distribution forecasting performance. Additionally, we used a subset from a widely recognized long-sequence prediction benchmark (Wu et al., ) to test the model’s out-of-distribution forecasting capabilities. This subset includes test data from the ETTh1-2, ETTm1-2, electricity, and weather datasets.

Figure: Figure 1: Architectures of Baseline Time Series Foundation Models. The the most widely used two Transformer architectures, encoder-only Transformer and decoder-only Transformer, are selected to be our baselines. The time series is divided into patches, each treated as a token input to the Transformer model. Shaded patches represent the future horizon to be predicted.
Refer to caption: x1.png

### 2.2 Models

TSFMs are predominantly built upon the Transformer architecture (Wen et al., ). For our baseline models, we selected two widely adopted architectures: the encoder-only Transformers and the decoder-only Transformers. We also include the state-of-the-art models from both architectures, Moirai (Woo et al., ) (encoder-only) and Chronos (Ansari et al., ) (decoder-only), in our study for comparison. Although Chronos-T5 is technically an encoder-decoder Transformer, we categorize it as analogous to decoder-only Transformers because it primarily relies on the decoder’s auto-regressive method for prediction. The primary distinction between encoder-only and decoder-only architectures lies in the attention mechanisms applied to the inputs, as illustrated in Figure 1. To better adapt them for time series forecasting, we introduce three key modifications in input layer, positional encoding and prediction head. More details are given in appendix B.

Patch Embedding. There are several approaches for generating inputs for transformer-based TSFMs, including point embedding, patch embedding, and lagged feature embedding. Due to the high computational cost of point embedding for long sequences and the limited robustness of lagged feature embedding, we adopt patch embedding in our models. This method, initially introduced by Vision Transformers (Dosovitskiy et al., ) and later adapted by PatchTST (Nie et al., ) for time series forecasting, divides the time series into non-overlapping segments, which are then projected into a feature space.

Rotary Position Embedding. This technique (RoPE) has rapidly gained popularity as a positional encoding method in recent large language models (Su et al., ). Given its proven effectiveness in improving time series forecasting performance (Woo et al., ), we adopt RoPE as a replacement for the original Transformer’s positional encoding. RoPE encodes absolute positions using a rotation matrix while embedding relative position dependencies directly into the self-attention mechanism.

Mixture of Distributions. Our models are designed to predict the probability distribution of future time series. However, real-world time series often exhibit complex distributions, including outliers, heavy tails, and extreme skew, which pose significant challenges for accurate modeling. To address these complexities, we incorporate a more flexible output likelihood by utilizing Student-t mixture models (Flunkert et al., ). Compared to the commonly used Gaussian mixture models, Student-t mixture models offer greater robustness in handling outliers and heavy-tailed distributions.

Our models are characterized by several key hyperparameters: the number of layers ($n_{\rm layer}$), the input/output dimensions of the residual stream ($d_{\rm m}$), the dimensions of the intermediate feed-forward layers ($d_{\rm ff}$), the number of attention heads per layer ($n_{\rm heads}$), and the dimension of the attention output ($d_{\rm head}$). The overall model size can be expressed as:

$$ $\displaystyle N$ $\displaystyle\approx n_{\rm layer}\left(4{d_{\rm m}}n_{\rm heads}d_{\rm head}+ 2{d_{\rm m}}*d_{\rm ff}\right)$ (1) $\displaystyle=2{d_{\rm m}}{n_{\rm layer}}\left(2{n_{\rm heads}}d_{\rm head}+d_ {\rm ff}\right)$ $\displaystyle=12{n_{\rm layer}}d_{\rm m}^{2}\quad\textup{with the standard} \quad n_{\rm head}\cdot d_{\rm head}=d_{\rm m}=d_{\rm ff}/4,$ $$

where the embedding layer, prediction head, biases and other sub-leading terms are excluded for a cleaner scaling laws. The embedding layer uses a patch size of 32 with $32{d_{\rm m}}$ parameters. The mixture distribution prediction head comprises multiple independent linear layers that predict the Student-t mixture distribution parameters for each patch, with $512{d_{\rm m}}$ parameters in total. In the study, we explore models with $\sim 10^{3}$ to $\sim 10^{8}$ trainable parameters.

### 2.3 Training and Evaluation Details

In this study, we focus exclusively on univariate time series forecasting to avoid the confounding effects introduced by multivariate time series, such as variable interactions, correlations, and the complexities of modeling multivariate relationships. Future research will address these factors, aiming to establish more comprehensive scaling laws for multivariate time series models.

Training Details. Our training objective is to optimize the mixture distribution log-likelihood. We utilize the AdamW optimizer with a batch size of 128 and a cosine learning rate scheduler with a linear warm-up of $10^{4}$ training steps, training for a total of $10^{5}$ steps. To facilitate learning data representations across diverse domains with varying series lengths and sample sizes, we visited each sample with probability $p_{i}=t_{i}/T$, where $t_{i}$ is the series’ time points and $T$ is the corpus’ total time points. We then randomly selected a segment from each chosen sample.

Evaluation Details. We evaluate the model on a randomly selected $10$% subset of the test data every $10^{3}$ steps to reduce computational costs. For performance measurement, we observed that non-normalized metrics like MAE and MSE are highly sensitive to the amplitude of time series data, often causing the overall average to be disproportionately influenced by high-amplitude datasets. To mitigate this issue, we utilize the normalized metric, mean absolute percentage error (MAPE), along with the log-likelihood loss, to assess forecasting performance. Detailed descriptions of these metrics are provided in the Appendix C.3.

## 3 Scaling Laws for Time Series Foundation Models

In this section, we first present experimental results using the encoder-only Transformer to explore scaling laws across different data distributions. Following this, we conduct a comparative study on the scaling behavior of encoder-only and decoder-only TSFMs, Moirai and Chronos, to investigate how model architectures influence the scalability of time series models.

### 3.1 Scaling Laws Across Data Distributions

Figure: Figure 2: Parameter Scaling. The scaling effect of trainable model parameters on the in-distribution (ID) and out-of-distribution (OOD) forecasting performance, which is evaluated using log-likelihood and MAPE metrics. When evaluated with log-likelihood, both ID and OOD results follow an approximate power law scaling with parameter count, exhibiting consistent trends across different data distributions.
Refer to caption: x2.png

Parameter Scaling. In Figure 2, we display the ID and OOD performance of a wide variety of encoder-only Transformers, ranging from small models with 3K parameters to large models with 300M parameters. We trained models on the full 15B pre-training corpus to convergence and report the minimum log-likelihood loss and MAPE. We can see that both ID and OOD performance roughly follow power-law behavior over five orders of magnitude in model sizes. Formally, the power law can be expressed as:

$$ $L(N)\approx\left(\frac{N_{c}}{N}\right)^{\alpha_{N}},$ (2) $$

where $L$ is the performance metric function (i.e., MAPE, or log-likelihood), $N$ is a given parameter count, $N_{c}$ is the normalization coefficient, and $\alpha_{N}$ is the exponent value that indicates the degree of performance improvement expected as we scale up $N$.

Observing the log-likelihood metric, the lines fitting the scaling laws for both ID and OOD data exhibits a roughly constant shift and close slopes. This implies that while models incur a consistent performance bias when transferred to OOD data, their scaling patterns correlate well with their performance on the ID data. When evaluated using MAPE, the power-law for OOD scenario shows a bigger exponent value than ID scenario. This indicates that increasing model size yields greater improvements in OOD performance than ID performance. In other words, for models with weak OOD generalization capabilities, increasing model size may enable them to perform equally well on both ID and OOD data.

Figure: Figure 3: Compute Scaling. The computation scaling results show that model performance approximately follows a power law trend with increasing compute, consistent across both ID and OOD scenarios. The ID and OOD results demonstrate that a lower bound exists for the loss and MAPE under a given computational budget.
Refer to caption: x3.png

Compute Scaling. Following the similar method in (Kaplan et al., ), we estimate the compute budget using the formula $C=6NBS$, where $B$ is the batch size, $S$ is the number of parameter updates, i.e. the input sequence length, and 6 is the factor to account for the forward and backward passes. The ID and OOD test loss for compute budget varying over six orders of magnitude are shown in Fig 3. We see that the optimal results for each compute budget are achieved by different model sizes $N$, but the lowest loss decreases according to a approximate power law with respect to the amount of training compute. The lowest losses appear as the heavy lines, which can be fit with

$$ $L(C)\approx\left(\frac{C_{c}}{C}\right)^{\alpha_{C}}.$ (3) $$

We observed significant noise in log-likelihood and MAPE during training, which may be caused by the learning rate scheduler and the random sampling evaluation strategy. Similar to parameter scaling, the ID and OOD log-likelihood shows similar scaling patterns; however, when evaluated using MAPE, the OOD power law shows larger exponent values than the ID power law.

Figure: Figure 4: Data Scaling. The blue and red plots illustrate how data volume affects the ID and OOD forecasting performance of encoder-only Transformers, evaluated using log-likelihood and MAPE metrics. The results indicate that in both scenarios, model performance scales approximately as a power law with increasing data volume.
Refer to caption: x4.png

Data Scaling. We display empirical trends for the performance as function of dataset size $D$ in Fig 4. For the trend, we trained multiple 1B encoder-only Transformers on a series of subsets of the 15B pre-training dataset and report the averaged evaluation results during training. We see that the log-likelihood and MAPE can be fit well with simple power-law

$$ $L(D)\approx\left(\frac{D_{c}}{D}\right)^{\alpha_{D}}.$ (4) $$

Different from the parameter scaling and computational scaling, we found that when evaluated using log-likelihood, ID and OOD performance do not exhibit the same scaling behavior. Instead, ID performance is more sensitive to the scaling of dataset size compared to OOD performance. However, similar to the observations on the other two factors, the impact of data scaling on MAPE in OOD data is greater than its impact on MAPE in ID data. This suggests that the scaling of various factors yields greater improvements in OOD performance than ID performance.

Cross-Distribution Scaling Effects Summary. We summarize the key findings on how model ID and OOD performance scales with the model parameters, data volume, and compute. (1) In both ID and OOD univariate time series forecasting, model performance follows a simple power law as a function of model parameters, data volume, and compute. (2) For log-likelihood loss, the model exhibits similar scaling patterns in both ID and OOD scenarios, in terms of model size or compute resources. (3) When using MAPE as the metric, scaling of all three factors results in greater improvements in OOD performance compared to ID performance.

Figure: Figure 5: Scaling Laws of Encoder-only vs. Decoder-only Transformer. This figure presents a comparison of scaling behaviors on log-likelihood between encoder-only and decoder-only Transformer across three different axes: number of parameters, compute, and dataset size. Overall, encoder-only Transformer shows better scalability on model parameters, computation and dataset sizes across ID and OOD data.
Refer to caption: x5.png

### 3.2 Scaling Laws across Model Architectures

The above results suggest that the power-law captures the scaling behavior of the encoder-only Transformer in both ID and OOD scenarios. Similarly, we analyze the scaling properties of the decoder-only Transformer, along with two other state-of-the-art TSFMs, Chronos and Moirai, to assess the impact of model architectures on scaling behavior.

Encoder-only vs. Decoder-only Transformer. Fig. 5 presents a comparison of the scaling patterns between encoder-only and decoder-only Transformers. In terms of parameter scaling, the power law exponent of the encoder-only Transformer is consistently higher than that of the decoder-only Transformer across ID and OOD settings, indicating its superior scalability on parameters. Moreover, for a given number of parameters, the encoder-only Transformer generally achieves a lower log-likelihood loss, signifying better forecasting performance. Regarding compute scaling, both models demonstrate highly similar compute scalability on OOD data and slightly differ on ID data. When it comes to data, the scaling law fitting lines for encoder-only and decoder-only Transformers exhibit nearly constant offsets and similar slopes on data size scaling. This suggests that despite differences in performance, the scaling behavior of encoder-only and decoder-only Transformers is highly similar on pre-training dataset size.

Encoder-only Transformer vs. Moirai. Moirai is a TSFM based on an encoder-only Transformer architecture. It introduces any-variate attention to capture relationships between multiple variables. Additionally, Moirai incorporates a multi-scale patch embedding to handle different frequency patterns, and a more diverse mixture distribution to model real-world probability distributions. Fig. 6 shows a comparison between the scaling behavior of encoder-only Transformer and Moirai. On ID data, Moirai demonstrates better performance. However, for OOD data, as the number of model parameters increases, Moirai is gradually surpassed by the encoder-only Transformer. Comparing the power-law lines of the two models, we see that Moirai shows a smaller slope, indicating relatively weaker scalability. Collectively, Moirai shows significant improvements on ID time series forecast than our baseline, but some designs may limit its scalability on OOD data.

Decoder-only Transformer vs. Chronos. Chronos-T5 is an encoder-decoder Transformer-based TSFM that, like decoder-only Transformers, follows an auto-regressive prediction approach but uses a separate encoder to extract contextual information. It adopts point-wise prediction and transform numerical regression into discrete probability prediction. Fig. 7 compares the scaling patterns of decoder-only Transformers and Chronos-T5. When evaluated with log-likelihood, Chronos-T5 exhibits power laws with very small exponents, indicating limited scalability. We attribute this to the discrete probability prediction, as log-likelihood loss on a discrete distribution is not distance-sensitive, meaning the loss remains high unless the predicted value exactly matches the label. Compared to log-likelihood, symmetric mean absolute percentage error (SMAPE, definition is shown in Appendix C.3) is a more appropriate metric for evaluating the two models. Chronos-T5 shows a slight advantage in performance and scalability on ID data. However, on OOD data, the decoder-only Transformer performs much better. Despite some performance differences, both models exhibit similar scaling patterns on OOD data. Overall, the design improvements in Chronos-T5 enhance ID time series forecasting, but they do not effectively improve OOD generalization.

Figure: Figure 6: Scaling Laws of Encoder-only Transformer vs. Moirai. This figure compares parameter scaling between the encoder-only Transformer and Moirai. While Moirai demonstrates notable improvements on ID test data compared to the baseline, certain design choices may hinder its scalability on OOD data.
Refer to caption: x6.png

Figure: Figure 7: Scaling Laws of Decoder-only Transformer vs. Chronos. This figure presents a comparison of parameter scaling between decoder-only Transformer and Chronos. The design introduced by Chronos enhances ID time series forecasting, but fails to improve the generalization of OOD.
Refer to caption: x7.png

Cross-Architecture Scaling Effects Summary. Our analysis of the scaling properties across architectures reveals several key findings: (1) Model architecture and design play a crucial role in determining scalability. (2) Encoder-only models demonstrate superior scalability and performance compared to decoder-only models, although both architectures exhibit similar scaling trends. (3) While Chronos and Moirai improve in-distribution ID forecasting, these gains do not extend effectively to OOD scenarios.

## 4 Design Principles for Time Series Foundation Models

Building on our findings regarding the scaling laws in TSFMs, we elaborate design principles to guide the development of effective and scalable models. These principles are framed around three key dimensions: training data, model parameters and architecture, as well as computational budget.

Training Data. Our experiments show that increasing the size of the training dataset leads to a greater performance improvement on OOD data compared to ID data. Enlarging the pre-training dataset is crucial for achieving better generalization. However, maintaining diversity within the dataset is equally important while increasing the data volume. Additionally, we observed that while there is a performance bias between encoder-only and decoder-only Transformers, they exhibit a consistent scaling pattern as the data size increases. This indicates that the benefits of data expansion are not influenced by model architecture, meaning that collecting more data can proceed in parallel with model improvements.

Model Parameters and Architecture. Our study highlights that model size is the most critical factor for improving TSFM performance. Among the three factors we examined, increasing the model size yielded the greatest benefit for ID forecast. In terms of architecture, an encoder-only Transformer architecture generally performs better and is more scalable than a decoder-only Transformer. Additionally, although the advanced models Chronos and Moirai showed some improvements in ID time series forecast, they failed to enhance OOD prediction capabilities. We speculate this is due to inductive biases introduced by the model modifications that do not generalize well. Compared to our baseline models, the scalability of Moirai and Chronos decreased, suggesting that the introduced designs limit the scalability of TSFMs. Model architecture significantly impacts scalability, and a good design should promote improvements in performance, generalization, and scalability.

Computational Budget. The power law we established indicates that there is an unbreakable lower bound for loss and MAPE under a given computational budget. This means that, with other factors remaining constant, as the model size increases, more compute resources must be invested to achieve better performance. However, different training objectives or model architectures can significantly affect this bound. For example, using masked recovery as a training objective typically shows a lower bound compared to next-token prediction. Similar to the scaling behavior of model size and dataset size, the increase in compute scale has a greater impact on improving OOD performance than on ID performance. This suggests that achieving robust generalization across distributions requires larger models, more data, and greater compute resources.

## 5 Related Works

Neural Scaling Laws. Neural scaling laws seek to provide a predictive framework for optimizing the allocation of computational resources to maximize model performance. In language domains, Kaplan et al. () demonstrated that performance follows a power-law relationship, improving as more computational resources, parameters, and data are utilized. Subsequent research has expanded this to predict other factors, such as downstream task performance (Isik et al., ) and inference time (Sardana et al., ). In vision domains, scaling laws have been explored in areas like discriminative modeling (Hestness et al., ) and visual auto-regressive modeling (Henighan et al., ). Recently, Edwards et al. () introduced scaling laws for large time series models, showing that performance scales according to a power law with model size, compute, and dataset size. Shi et al. () examined the effect of time-series forecasting horizon on model scaling behavior, offering a theoretical framework to explain its influence. However, both studies have focused on in-distribution scenarios, leaving the investigation of scaling laws in out-of-distribution contexts largely unexplored.

Time Series Foundation Models. Foundation models (Das et al., ; Goswami et al., ) represent a new paradigm aimed at generalizing across diverse domains and tasks by leveraging knowledge from large-scale data in the pre-training phase. They have significantly advanced time series forecasting, particularly in zero-shot scenarios, where predictions are made on data from previously unseen domains. For instance, Woo et al. () introduced Moirai, an encoder-only transformer architecture that employs an “any-variate” attention mechanism to capture dependencies in multivariate time series. Ansari et al. () proposed a method that tokenizes time series values through scaling and quantization into a fixed vocabulary, training a series of transformer-based models known as Chronos. Liu et al. () developed Timer, a simple decoder-only transformer architecture designed for univariate time series forecasting, while Rasul et al. () introduced Lag-Llama, a decoder-only transformer that integrates lags as covariates to improve forecasting accuracy. These models incorporate various modifications to the standard Transformer architecture for time series data. However, the impact of these changes on model scaling properties has not been systematically studied. As model size increases, it remains an open question whether these modifications will continue to enhance performance.

Figure: Figure 8: Case studies on the “emergent abilities” of scaling time series model: We present three examples of zero-shot (unseen) out-of-distribution (OOD) time series prediction. In the US births, NN5 daily, and Australian electricity datasets, we observed that model behavior deviates from expected power law patterns, instead exhibiting characteristics more akin to emergent phenomena.
Refer to caption: x8.png

## 6 Discussion

### 6.1 Additional Studies

We also conduct the following analysis to better understand the scaling behaviors of TSFMs. Due to the space limit, see their details in Appendix D.2 to D.4 .

Emergent Behaviors. Fig. 8 shows some examples of zero-shot OOD time series forecasting, where the model’s performance remains low until the model size reaches a critical threshold, after which performance improves significantly. This scaling behavior deviates from the previously observed power law and is more akin to emergent phenomena (Wei et al., ). This suggests that achieving good forecasting performance for certain time series may require a model with sufficient parameters to capture the underlying dynamic systems. For example, if a system of multistep differential equations requires $l$ sequential computation steps, the model may requires a depth of at least $O(l)$ layers to handle the complexity.

Sample Efficiency. Appendix D.2, Fig 9 shows the evaluation results during training. Large models are more sample-efficient than small models, reaching the better performance with the same optimization steps and using fewer time points.

Scaling Pattern Depends on Data Distributions. Appendix D.3, Fig. 10 shows the scaling behavior of TSFMs on Monash dataset, suggesting that OOD scaling behaviors depend on the relationship between the unseen data distribution and the training data distribution.

Scaling Pattern Depends on Performance Metrics. Appendix C.3, Fig. 11; 12;13 show the scaling behavior of continuous ranked probability score (CRPS), symmetric mean absolute percentage error(sMAPE), and mean absolute error (MAE). Comparing the results with log-likelihood and MAPE, we see obvious difference in the scaling patterns.

### 6.2 Conclusion

We have observed consistent scaling of encoder-only Transformer log-likelihood loss with parameter count, training computation and dataset size on both ID and OOD test data, as encapsulated in a power law. The experimental results show that as the number of parameters, computational resources, and training data increase, both models’ ID and OOD performance will continue to improve. Furthermore, our established laws suggest that for ID forecasting performance, larger model may be more important than more data. While, larger training dataset size plays a more crucial role in improving OOD forecasting performance.

We have also study the scaling properties of two common types of TSFMs, encoder-only Transformer and decoder-only Transformer. The two architectures show different scaling patterns in model size, computational resources, and training data. Generally, encoder-only Transformers show better performance and scalability. Moreover, we further include two advanced TSFMs, Chronos and Moirai, as specific cases to study the impact of model design on scaling behavior.
We find that both Chronos and Moirai improve ID forecasting performance, but these gains can not translate effectively to OOD scenarios.

Our experiments independently investigated the scaling behavior of TSFMs in terms of parameter count, computational resources, and training data size, assuming other factors were unlimited. In future work, it would be valuable to develop a unified model to explore the relationships between these factors. This could help guide the community in optimizing resource allocation when one resource is limited. Additionally, multiple studies have shown that model performance varies across different context window lengths and forecast horizons. In future research, we plan to incorporate the impact of context length and forecast horizon on scalability.

## Appendix A Appendix: Datasets and Setup Information

Previous work has proposed several public large-scale time series datasets, e.g. LOTSA (Woo et al., ) and UTSD (Liu et al., ). Instead of using these datasets directly, we built a pre-training corpus through data filtering. There are three main reasons for this: First, these datasets contain numerous periodic time series with substantial pattern redundancy. Second, these datasets show a heavily long-tailed distribution, with multiple domains accounting for less than 5% of the total. Third, data quality is variable, with some time series missing many observations or having low signal-to-noise ratios. To address these issues, we developed a data filtering pipeline:

Deduplication. In these public datasets, we observed significant redundancy across many subsets. This redundancy typically manifests as numerous repeated patterns within individual samples or high similarity between multiple samples. Such redundancy can negatively impact training efficiency. To address this issue, we applied a down-sampling strategy for datasets with redundant samples based on the data sampling period.

Quality Filtering. We focused on selecting time series data that have no missing values and a signal-to-noise ratio (SNR) greater than 20 dB to ensure better predictability. To calculate the SNR, we first used Fourier transform to identify the low-frequency components of the time series. We then applied a low-pass Butterworth filter to extract these low-frequency components as the signal, with the remaining residuals considered as noise.

Domain Balancing. In the Lotsa dataset, the climate domain accounts for nearly 90% of the data, while the remaining seven domains collectively represent only 10%. To create a more balanced pre-training dataset, we selected the traffic domain, which contains 4.8 billion time points, as the reference. We then scaled the data from the other domains to a similar magnitude.

Following the aforementioned processing steps, we compiled a high-quality time series dataset containing a total of 14.46 billion time points across 7 domains. The data volume is sufficient to support training models with parameters ranging from $10^{3}$ to $10^{8}$ parameters. Furthermore, each domain includes at least 100 million time points, ensuring that even when the dataset is divided into subsets of various sizes, domain diversity is preserved. Below, we introduce the datasets for each domain and outline the key properties of the datasets after processing, including domain type, sampling frequency, number of time series, total number of observations, and data source.

Climate. The climate data are sourced primarily from the ERA5 and CMIP6 datasets, which provide time series for various climate-related variables, including temperature, humidity, and pressure levels. During data curation, we observed a high degree of similarity across yearly data. To reduce redundancy, only two years of data from ERA5 and CMIP6 were included.

Energy. The energy data primarily come from the BuildingsBench dataset, which provides time series data on residential and commercial building energy consumption. After applying quality filtering based on SNR, we selected samples from the Buildings-900K, BDG-2, and Sceaux datasets. Additionally, high-quality data from the ProEnFo library, including the GEF, ELF, PDB, Spanish, and Covid19_Energy datasets, were incorporated.

Transport. The primary source for transport data is the LargeST traffic dataset, covering traffic flow in California from 2017 to 2021. We also integrated datasets from LibCity, PEMS, Loop Seattle, and Q-Traffic into our corpus.

Cloud Operations. The cloud operations data is from the large-scale CloudOps time series datasets, which measures various variables such as CPU and memory utilization.

Healthcare. We include a diverse sets of healthacre data from the ucr time series archive.

Web. The web data comprises the Kaggle Web Traffic Weekly dataset and the Wiki-Rolling dataset.

Sales. Here we use the Favorita Sales dataset.

**Table 2: Pretraining datasets and key properties.**
| Dataset | Domain | Frequency | # Time Series | # Obs. | Source |
| --- | --- | --- | --- | --- | --- |
| CMIP6 | Climate | 6H | 8192 | 2,870,476,800 | Nguyen et al. () |
| ERA5 | Climate | H | 8192 | 1,860,698,112 | Nguyen et al. () |
| Buildings900K | Energy | H | 264980 | 2,321,489,780 | Emami et al. () |
| Australian Electricity | Energy | 30T | 5 | 1,153,584 | Godahewa et al. () |
| BDG-2 Bear | Energy | H | 91 | 1,482,312 | Emami et al. () |
| BDG-2 Fox | Energy | H | 135 | 2,324,568 | Emami et al. () |
| BDG-2 Panther | Energy | H | 136 | 893,840 | Emami et al. () |
| Sceaux | Energy | H | 1 | 34,223 | Emami et al. () |
| Solar Power | Energy | 4S | 26 | 5,248 | Godahewa et al. () |
| Covid19 Energy | Energy | H | 1 | 31,912 | Wang et al. () |
| Spanish | Energy | H | 1 | 35,064 | Wang et al. () |
| Elecdemand | Energy | 30T | 1 | 17,520 | Godahewa et al. () |
| PDB | Energy | H | 1 | 17,520 | Wang et al. () |
| GEF17 | Energy | H | 8 | 140352 | Wang et al. () |
| GEF14 | Energy | H | 1 | 17,520 | Wang et al. () |
| ELF | Energy | H | 1 | 21,792 | Wang et al. () |
| Traffic Weekly | Transport | W | 821 | 78,816 | Godahewa et al. () |
| Q-Traffic | Transport | 15T | 45,148 | 257,200,384 | Jiang et al. () |
| PEMS04 | Transport | 5T | 307 | 14,638,784 | Jiang et al. () |
| PEMS07 | Transport | 5T | 883 | 23,789,760 | Jiang et al. () |
| PEMS08 | Transport | 5T | 170 | 8,684,480 | Jiang et al. () |
| PEMS Bay | Transport | 5T | 325 | 15,975,920 | Jiang et al. () |
| Loop Seattle | Transport | 5T | 1,809 | 33,700,832 | Jiang et al. () |
| LargeST | Transport | 5T | 8521 | 4,452,510,528 | Liu et al. () |
| Azure VM Traces 2017 | CloudOTS | 5T | 159,472 | 880,648,165 | Woo et al. () |
| Borg Cluster Data 2011 | CloudOTS | 5T | 143,386 | 176,650,715 | Woo et al. () |
| Alibaba Cluster Trace 2018 | CloudOTS | 5T | 58409 | 191,740,824 | Woo et al. () |
| Wiki-Rolling | Web | D | 47,675 | 40,619,100 | Alexandrov et al. () |
| Kaggle Web Traffic Weekly | Web | W | 145,063 | 15,206,232 | Godahewa et al. () |
| Favorita Sales | Sales | D | 111,840 | 139,111,860 | Woo et al. () |
| PigArtPressure | Health | - | 312 | 624,000 | Dau et al. () |
| SelfRegulationSCP1 | Health | 0.004 SEC | 3,366 | 3,015,936 | Dau et al. () |
| SelfRegulationSCP2 | Health | 0.004 SEC | 2,660 | 3,064,320 | Dau et al. () |
| TDBrain | Health | 0.002 SEC | 28644 | 73,299,996 | Wang et al. () |
| MotorImagery | Health | 0.001 SEC | 24,192 | 72,576,000 | Dau et al. () |
| PigCVP | Health | - | 312 | 624,000 | Dau et al. () |
| AtrialFibrillation | Health | 0.008 SEC | 60 | 38,400 | Dau et al. () |
| IEEEPPG | Health | 0.008 SEC | 15,480 | 15,480,000 | Tan et al. () |
| BIDMC32HR | Health | - | 15,898 | 63,592,000 | Tan et al. () |

## Appendix B Time Series Models

We define time series forecasting as the following problem: given a collection of multivariate time series samples with look back window $L:(\mathbf{x}_{1},\dots,\mathbf{x}_{L})$ where each $\mathbf{x}_{t}$ at time step $t$ is a vector of dimension of $M$, our goal is to forecast $T$ future values $\left(\mathbf{x}_{L+1},\dots,\mathbf{x}_{L+T}\right)$.

Patch Embedding. We split the input $(\mathbf{x}_{1},\dots,\mathbf{x}_{L})$ into $M$ univariate time series $\mathbf{x}^{(i)}\in\mathbb{R}^{1\times L}$, independently forecasting future time series for each variate. Each univariate time series $\mathbf{x}^{(i)}$ is first divided into non-overlapped patches. Specifically, given the patch length as $P$, the patching process will generate the a sequence of patches $\mathbf{x}^{(i)}_{p}\in\mathbb{R}^{P\times N}$ where $N=\left[\frac{L}{P}\right]$ is the number of patches. Then the patches are mapped to the latent space of $d_{m}$ via a learnable linear projection $\mathbf{W}_{p}\in\mathbb{R}^{d_{m}\times P}$. In our baseline models, the patch size $P$ is set to 32.

Rotary Position Embedding. Rotary Positional Embedding (RoPE) is a type of position encoding that encodes absolute positional information with a rotation matrix and naturally incorporates explicit relative position dependency in self-attention formulation. In detail, RoPE incorporates absolute position information to the embedding and transform them into queries, keys through function:

$$ $f_{q,k}(\mathbf{x}_{m},m)=\mathbf{R}^{d}_{\Theta,m}\mathbf{W}_{q,k}\mathbf{x}_ {m}$ $$

where

$$ $\mathbf{R}^{d}_{\Theta,m}=\begin{bmatrix}\mathbf{M}_{1}&&&\\ &\mathbf{M}_{2}&&\\ &&\ddots&\\ &&&\mathbf{M}_{d/2}\end{bmatrix},\mathbf{M}_{j}=\begin{pmatrix}\cos m\theta_{j }&-\sin m\theta_{j}\\ \sin m\theta_{j}&\cos m\theta_{j}\end{pmatrix}$ $$

is the rotary matrix with pre-defined paramters $\Theta=\{\theta_{i}=10000^{-2(i-1)/d},i\in[1,2,\dots,d/2]\}$, $\mathbf{W}_{q,k}$ is the learned query or key projection weights, and $\mathbf{x}_{m}\in\mathbb{R}^{d}$ is the embedding of the $m$ token. Next, applying RoPE to the dot product of query and key, we can obtain:

$$ $\mathbf{q}_{m}^{\top}\mathbf{k}_{n}=\left(\mathbf{R}^{d}_{\Theta,m}\mathbf{W}_ {q}\mathbf{x}_{m}\right)^{\top}\left(\mathbf{R}^{d}_{\Theta,n}\mathbf{W}_{k} \mathbf{x}_{n}\right)=\mathbf{x}^{\top}_{m}\mathbf{W}_{q}\mathbf{R}^{d}_{ \Theta,n-m}\mathbf{W}_{k}\mathbf{x}_{n}$ $$

where $\mathbf{R}^{d}_{\Theta,n-m}=\left(\mathbf{R}^{d}_{\Theta,m}\right)^{\top}
\mathbf{R}^{d}_{\Theta,n}$. In this way, RoPE naturally incorporates relative position information through rotation matrix product. In general, the self-attention enhanced with RoPE can be written as

$$ $\text{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})_{m}=\frac{\sum_{n=1}^{N} \left(\mathbf{R}^{d}_{\Theta,m}\phi(\mathbf{q}_{m})\right)^{\top}\left(\mathbf {R}^{d}_{\Theta,n}\varphi(\mathbf{k}_{n})\right)\mathbf{v}_{n}}{\sum_{n=1}^{N} \phi(\mathbf{q}_{m})^{\top}\varphi(\mathbf{k}_{n})}.$ $$

where $\phi(\cdot),\varphi(\cdot)$ are usually non-negative functions, e.g. elu($\cdot$)+1, or exp($\cdot$). Overall, unlike the traditional approach of adding positional information before the query and key projection, RoPE utilizes a rotary matrix to transform query and key embeddings, effectively leveraging the geometric properties of vectors

Mixture of Distributions. As described in (Flunkert et al., ), our model predicts the parameters of a probability distribution, specifically a mixture of Student-t distributions. The probability density function for a random variable $x$ following a Student-t distribution is given by:

$$ $p(x;\nu,\mu,\tau)=\frac{\Gamma\left(\frac{\nu+1}{2}\right)}{\Gamma\left(\frac{ \nu}{2}\right)\sqrt{\pi\nu\tau}}\left(1+\frac{1}{\nu}\left(\frac{x-\mu}{\tau} \right)^{2}\right)^{-\frac{\nu+1}{2}}$ $$

with parameters $\nu>0,\mu\in\mathbb{R},\tau>0$ represent the degrees of freedom (df), location, and scale coefficient, respectively, and $\Gamma$ is the gamma function. We modeled the predicted distribution using a mixture of four Student-t distributions. Our baseline model employs independent linear layers to predict the distribution parameters: degrees of freedom, location, scale, and mixture weights. To ensure positive values for the scale and df, we apply a soft-plus transformation. The mixture weights are constrained within the probability simplex using a soft-max function. Additionally, to avoid undefined variance for low df values, we impose a lower bound of 2 on the df parameter

Encoder-only Transformer. The encoder-only Transformer has proven effective for time series tasks (Nie et al., ). In input time seires, the future horizon is replaced by learnable mask tokens combined with position information. After passing through bidirectional attention blocks, the future representations are derived from these mask tokens and mapped to the parameters of a mixture distribution. Through sampling from the mixture distribution, we can get predictions for future patches. The pseudo code of encoder-only Transformer-based TSFM is as follows:

[frame=single,framesep=5pt]python
class EncoderOnlyTransformer(nn.Module):
patch_tokenizer = nn.Linear(N, d_m)
encoder = TransformerEncoder()
mask_token_embedding = nn.Embedding(1, d_m)
mixture_distribution_head = MixtureDistributionHead()

def forecast(self, x, T):
B, L, N = x.shape
# Patch Embedding: (B, L, N)-¿(B, L, d_m)
x = patch_tokenizer(x)

# Add mask tokens: (B, L, d_m)-¿(B, L+T, d_m)
mask_tokens = self.mask_token_embedding.weight.expand(B, T, -1)
x = torch.cat((x, mask_tokens), dim=1)

# Pass through bidirectional attention blocks
encoded_output = self.encoder(x)

# Truncate the future tokens: (B, L+T, d_m)-¿(B, T, d_m)
future_tokens = encoded_output[:,-self.num_mask_tokens:]

# Predict the parameters of mixture distribution
future_dist = self.mixture_distribution_head(future_tokens)

return future_dist

Decoder-only Transformer. The key difference in the decoder-only Transformer is its unidirectional attention. In the input sequence, no masking is needed. The future horizon is predicted based on the preceding token. In other words, given a sequence of input patches, the model is optimized to predict the next patch as a function of all past patches. Similar to LLMs this can be done in parallel over the entire context window, and automatically enables the model to predict the future after having seen varying number of input patches. The pseudo code of decoder-only Transformer-based TSFM is as follows:

[frame=single,framesep=5pt]python
class DecoderOnlyTransformer(nn.Module):
patch_tokenizer = nn.Linear(N, d_m)
decoder = nn.TransformerDecoder()
mixture_distribution_head = MixtureDistributionHead()

def forecast(self, x, T):
# Patch Embedding: (B, L, N)-¿(B, L, d_m)
x = self.patch_tokenizer(x)

# Generate predictions for future time steps step-by-step
preds = []

for t in range(T):
# If it’s the first loop, use the original input
tgt = torch.cat([x,*preds],dim=1) if preds else x

# Pass through unidirectional attention blocks
decoded_output = self.decoder(tgt)

# Take the last token
next_token = decoded_output[:, -1:, :]

# Record the prediction of the current time step
predictions.append(next_token)

# Concatenate all predictions: (B, T, d_m)
future_tokens = torch.cat(predictions, dim=1)

# Predict the parameters of mixture distribution
future_dist = self.mixture_distribution_head(future_tokens)

return future_dist

## Appendix C Implementation Details

### C.1 Training Setup

Based on the constructed datasets of different sizes, we trained a series of encoder-only Transformers and decoder-only Transformers with increasing data sizes and model parameters. To improve batch processing efficiency and handle varying sequence lengths, we employ sequence packing (Raffel et al., ), which reduces padding requirements. Moreover, we train models with forecasting capabilities over varying context and prediction lengths. Rather than defining a fixed context and prediction length, we sample 15% - 50% lengths as forecast horizon and the remaining as context horizon, for a given time series. As most probabilistic forecasting models, we minimize the negative log-likelihood of the predicted patch with respect to the ground truth.

### C.2 Evaluation Setup

We primarily used the LSF (Wu et al., ) and Monash (Godahewa et al., ) datasets to evaluate the model’s out-of-distribution generalization ability. For the LSF dataset, we only used the test set and performed evaluations in a non-overlapping rolling window fashion, with the stride equal to the prediction length. For the Monash dataset, considering its large size, we set only one window per sample for evaluation. The evaluated results in Monash dataset are exhibited in Appendix D.3. The below table details the evaluation setting per dataset:

**Table 3: Out-of-distribution Evaluation Datasets.**
| Dataset | Domain | Frequency | # Prediction Length | # Rolling Evaluations. |
| --- | --- | --- | --- | --- |
| ETTh1 | Energy | H | 192 | 175 |
| ETTh2 | Energy | H | 192 | 819 |
| ETTm1 | Energy | H | 192 | 819 |
| ETTm2 | Energy | H | 192 | 819 |
| Electricity | Energy | H | 192 | 7062 |
| Weather | Climate | H | 192 | 1029 |
| M4 Hourly | Finance | H | 48 | 414 |
| M4 Daily | Finance | D | 14 | 4227 |
| M4 Weekly | Finance | W | 13 | 359 |
| M4 Monthly | Finance | M | 18 | 48000 |
| M4 Quarterly | Finance | Q | 8 | 24000 |
| M1 Monthly | Finance | M | 18 | 617 |
| M3 Monthly | Finance | M | 18 | 1428 |
| M3 Other | Finance | Q | 8 | 174 |
| NN5 Daily | Finance | D | 56 | 111 |
| NN5 Weekly | Finance | W | 8 | 111 |
| Tourism Monthly | Finance | M | 24 | 366 |
| Tourism Quarterly | Finance | Q | 8 | 427 |
| CIF 2016 | Finance | M | 6 | 15 |
| Traffic Hourly | Transport | H | 168 | 862 |
| Traffic Weekly | Transport | W | 8 | 862 |
| Australian Electricity Demand | Energy | 30T | 336 | 5 |
| Rideshare | Transport | H | 168 | 2304 |
| Saugeen | Climate | D | 30 | 1 |
| Sunspot | Climate | D | 30 | 1 |
| Temperature Rain | Climate | D | 30 | 32072 |
| Vehicle Trips | Transport | D | 30 | 329 |
| Weather | Climate | D | 30 | 3010 |
| Car Parts | Sales | M | 12 | 2674 |
| FRED MD | Finance | M | 12 | 107 |
| Pedestrian Counts | Transport | H | 12 | 66 |
| Hospital | Health | M | 12 | 767 |
| Covid Deaths | Health | D | 30 | 266 |
| KDD Cup 2018 | Energy | H | 168 | 270 |
| Bitcoin | Finance | D | 30 | 18 |
| Us Births | Health | D | 30 | 1 |

### C.3 Evaluation Metrics

Mean Absolute Percentage Error (MAPE). MAPE is a commonly used metric to evaluate the accuracy of a forecasting model by expressing the error in percentage terms, making it unit-free and easily interpretable. It measures the average magnitude of the absolute percentage error between the predicted and actual values, relative to the actual values. For a univariate time series, the error is defined as:

$$ $e_{j}^{(i)}=y_{j}^{(i)}-\hat{y}_{j}^{(i)}$ $$

where $y_{j}^{(i)}$ and $\hat{y}_{j}^{(i)}$ are the target and predicted values of the $i$-th time series and $j$-th time step, respectively. The MAPE of the $i-th$ time series is then calculated as:

$$ $\text{MAPE}=\frac{100}{H}\sum_{j=t+1}^{t+H}\frac{|e_{j}^{(i)}|}{|y_{j}^{(i)}|}$ $$

This formula calculates the percentage error at each time step, averaging it over the forecast horizon $H$. MAPE is intuitive and interpretable in percentage terms. However, MAPE can be highly sensitive to small actual values because it divides by $y_{j}^{(i)}$, leading to extremely large errors when actual values approach zero.

Symmetric Mean Absolute Percentage Error (sMAPE). sMAPE addresses the shortcoming of MAPE by using a symmetric formula that normalizes both the actual and predicted values. The sMAPE of the $i$-th time series is defined to be

$$ $\text{sMAPE}=\frac{200}{H}\sum_{j=t+1}^{t+H}\frac{|e_{j}^{(i)}|}{|y_{j}^{(i)}| +|\hat{y}_{j}^{(i)}|}$ $$

The sMAPE ensures that errors are balanced between over- and under-prediction by dividing by the sum of the absolute actual and predicted values. This normalization helps avoid extreme errors when actual values are small, but it can also produce undefined results when both actual and predicted values are zero.

Continuous Ranked Probability Score (CRPS). Before we can introduce the CRPS, we need to introduce the weighted quantile loss Park et al. (), which is a metric normalized over the test set. We first define the $\alpha$-quantile loss, also known as the pinball loss at quantile level $\alpha$, to be:

$$ $\Lambda_{\alpha}(q,y)=(\alpha-1_{y<q})(y-q)$ $$

The weighted quantile loss is then the normalized sum of quantile losses,

$$ $\text{wQL}[\alpha]=2\frac{\sum_{(i,j)\in\Omega}\Lambda_{\alpha}(\hat{q}_{j}^{( i)}(\alpha),y_{j}^{(i)})}{\sum_{(i,j)\in\Omega}|y_{j}^{(i)}|}$ $$

where $\Omega=\{(i,j)\in\mathbb{Z}^{2}:1\leq i\leq n,\tau_{i}+1\leq j\leq T_{i}\}$.

The CRPS is a proper scoring rule Matheson & Winkler (), meaning that it is minimized when the predictive distribution is equal to the distribution from which the data is drawn.

$$ $\text{CRPS}=\int_{0}^{1}2\Lambda_{\alpha}(F^{-1}(\alpha),y)\,d\alpha$ $$

However, we are unable to evaluate this quantity since we generally are not able to compute the integral in closed form and only have access to a finite number of quantile predictions. The approximation of the CRPS is an average of the weighted quantile loss over $K$ quantiles, and thus is also known as the mean weighted quantile loss.

$$ $\text{CRPS}\approx\frac{1}{K}\sum_{k=1}^{K}\text{wQL}[\alpha_{k}]$ $$

Figure: Figure 9: Training encoder-only models show that, as model scale increases, larger models require fewer samples to achieve the same level of performance.
Refer to caption: x9.png

Figure: Figure 10: Scaling Pattern Depends on Data Distributions. The blue, red, and green plots illustrate the scaling behavior of log-likelihood loss on in-distribution (ID) data, out-of-distribution (OOD) LSF data, and OOD Monash data, respectively. We observe that the model exhibits different scaling patterns on the two different OOD datasets, indicating that OOD scaling behaviors significantly depend on the relationship between the unseen data distribution and the training data distribution.
Refer to caption: x10.png

## Appendix D Additional Results

### D.1 Emergent behaviors

Fig. 8 presents case studies on ”emergent abilities” of TSFMs for out-of-distribution (OOD) predictions. As model size increases, one would expect the MAPE to follow a smooth power-law decline, indicating continuous improvement. However, the plots reveal distinct phenomena, where performance changes abruptly rather than gradually. This behavior is akin to the “emergent behaviors” described in (Wei et al., ), where certain capabilities absent in smaller models only appear in larger ones. The emergence of improved performance at certain model sizes could indicate the development of higher-order patterns or representations within the model. These emergent abilities might stem from the model’s capacity to recognize and generalize complex temporal patterns that were previously inaccessible at smaller scales. Such abilities is evident particularly in zero-shot OOD predictions, where the model must extrapolate from training data to entirely unseen scenarios.

### D.2 Sample Efficiency

Fig. 9 illustrates the evaluation results of different-sized encoder-only models on a validation set during training. The key observation is that larger models achieve better performance with fewer training steps compared to smaller models, demonstrating higher sample efficiency. This is important in scenarios where data are limited, as larger models can achieve superior performance without processing as much data.

### D.3 Scaling pattern depends on data distributions

Fig. 10 highlights the influence of different data distributions on the scaling behavior of the model. By comparing the green line, representing the Monash dataset, and the red line, representing the LSF dataset, we observe a relatively consistent offset and similar slopes across the scaling curves. This suggests that, when transferring the model from the training data distribution to other OOD distributions, there is a predictable decrease in performance that varies by the dataset. However, the model’s gains from scaling—whether by increasing model size, compute, or dataset size—follow a fixed proportional relationship. This indicates that while OOD performance degrades, scaling the model still yields a consistent improvement ratio across different distributions, albeit with varying levels of absolute performance. Moreover, in the future, we can further analyze the reasons why transfer cost arises and establish a law to predict the transfer performance loss in population-risk based theoretical analysis (Yang et al., ) through Wasserstein measurement.

### D.4 Scaling pattern depends on performance metrics

We investigate the scaling behaviors of five common performance metrics: log-likelihood, MAPE, CRPS, sMAPE and MAE(as shown in Figures 11 12 and 13). All metrics exhibit a decreasing trend following an approximate power-law; however, each metric demonstrates distinct scaling characteristics, reflected in their varying power-law exponents. Previous work by Ghorbani et al. () establishes a relationship between large language models’ log-likelihood loss and the BLEU score in translation tasks. Similarly, future research could explore a transformation between log-likelihood loss and time series forecasting metrics, offering a means to predict forecasting performance from training loss.

Figure: Figure 11: Scaling Pattern of CRPS. This figure illustrates scaling laws for CRPS across three training factors: model size, compute, and dataset size. The blue plots indicate the ID performance, while the red plots represent OOD performance. In both ID and OOD scenarios, CRPS decreases with the three factors scaling up.
Refer to caption: x11.png

Figure: Figure 12: Scaling Pattern of sMAPE. This figure presents the scaling laws for sMAPE in relation to three training factors: model size, compute, and dataset size. The blue plots show performance in ID scenarios, while the red plots depict OOD performance. In both cases, sMAPE decreases as all three factors increase.
Refer to caption: x12.png

Figure: Figure 13: Scaling Pattern of MAE. This figure illustrates scaling laws for MAE in relation to model size, compute, and dataset size. The blue lines represent ID performance, while the red lines show OOD performance. In all cases, increasing model size, compute, or dataset size reduces MAE, with out-of-distribution performance improving more significantly.
Refer to caption: x13.png

## Peer Reviews

confidence: 5
contribution: 4
initial_rating: 5
presentation: 3
questions: 1. For the training details on line 183, I am not sure if I understood correctly. If one dataset is much larger than the other datasets, are we much more likely to sample from this dataset? Is there a cap to this likelihood? In existing works such as MOIRAI and Timer, they apply some threshold that prevents a large dataset from dominating from the training distribution. In addition to "Domain Balancing" as mentioned in Appendix A in your paper, I think balancing the datasets within each domain should also be helpful. Could you compare the different data mixing strategies?
2. Why does OOD have a larger exponent for MAPE than ID in Figure 2 and 3, but not for log-likelihood? It feels to me that it might be due to the type of data distribution you have.
3. In Timer, the findings are decoder-only scales better than encoder-only. Could you explain why your finding is different? In Timer, the batch size is chosen to be 8192, but you only used a batch size of 128. Is it possible that using a larger batch size would change your observations and make the decoder-only model more favorable?
4. Did you try retraining MOIRAI/Timer/Chronos using your filtered dataset? The dataset mixture is often a very important factor. Not using the same pretraining datasets can be an unfair comparison in Section 3.2.
5. "Scaling-laws for Large Time-series Models" suggests that model size has a big impact on the optimal learning rate. What learning rate did you use and how did you change it for different model sizes? Are the observations consistent for encoder-only and decoder-only models?

Minor:
- "Lotsa" should be capitalized on line 94.
- "Sum" in Table 1 is a bit confusing. Consider replacing it with "Total" and use a vertical line to separate it from previous columns.
- In Figure 2 and 3, consider adding ID and OOD to the legend, in addition to using the colors.
- Line 794: "and Covid19_Energy datasets", the underscore here seems to be a typo?
- "Scaling-laws for Large Time-series Models" suggests that using a separate head for each distribution parameter should stabilize training. 
- Could you add an experiment to show the benefit of using student-T mixture versus gaussian mixture?

Overall, I believe the findings in this paper will be very helpful to the community. I will raise my score if my concerns regarding the training and evaluation setups are addressed.
soundness: 3
strengths: 1. The paper flows very well and is easy to follow.
2. While most existing TSFMs for forecasting choose a decoder-only structure, this paper presents new findings on the encoder-only versus decoder-only that contradicts prior findings, and analyzed it from the scope of OOD data.
summary: The paper analyzes two common TSFM architectures -- encoder-only and decoder-only transformers -- in terms of both in-distribution and out-of-distribution data. The results show that encoder-only can better scale to both ID and OOD data. The authors also show that specific architecture choices in existing TSFMs could negatively affect OOD generalization.
weaknesses: 1. Evaluation setup could be improved. On line 130, "This subset includes test data from the ETTh1-2, ETTm1-2, electricity, and weather datasets." Are these the only datasets that are used as OOD test data throughout all the experiments? To strengthen your findings, I suggest varying the OOD datasets (maybe include a confidence interval), and see if your findings still hold.
2. The paper could benefit from explaining why its findings are different from those of the existing works.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Justification for the chosen model architectures: Can the authors provide a more detailed explanation for why they selected encoder-only and decoder-only Transformers as the main focus of their study? While these architectures are indeed popular in TSFMs, it would be helpful to understand the rationale behind this choice and whether the authors believe their findings would generalize to other architectures, such as encoder-decoder models or hybrid models.
2. Potential for theoretical analysis: The paper primarily relies on empirical observations to establish the scaling laws for TSFMs. Have the authors considered complementing their empirical findings with a theoretical analysis? If so, what challenges do they anticipate in developing a theoretical framework for the observed scaling behaviors, and how might such a framework enhance our understanding of the underlying mechanisms governing the relationship between model size, compute, data, and performance?
3. Factors contributing to the ID-OOD performance trade-off: The paper highlights that architectural modifications in Moirai and Chronos improve ID performance but compromise OOD scalability. Can the authors provide more insights into the factors that might contribute to this trade-off? For example, are there specific architectural choices (e.g., attention mechanisms, embedding strategies) that could be more prone to overfitting on ID data, leading to reduced OOD scalability? Understanding these factors could help guide the design of future TSFMs.
4. Computational efficiency considerations: While the paper considers compute budgets as a scaling factor, it does not provide a detailed analysis of the computational efficiency of the investigated models. Can the authors comment on the practical trade-offs between performance and computational cost for the different model architectures and sizes? Including metrics such as training time, inference time, and memory consumption could provide valuable insights for practitioners looking to deploy TSFMs in resource-constrained environments.
5. Generalizability to multivariate time series: The paper focuses exclusively on univariate time series forecasting to avoid the confounding effects introduced by multivariate time series. Can the authors discuss the potential challenges and opportunities in extending their findings to multivariate TSFMs? What additional factors (e.g., variable interactions, correlations) might need to be considered when investigating scaling laws in the multivariate setting?
soundness: 3
strengths: 1. The study extends the existing research on scaling laws for TSFMs in a novel direction by investigating their behavior across different data distributions (ID and OOD) and model architectures. While previous works have primarily focused on ID scenarios, this paper breaks new ground by systematically examining the scaling properties of TSFMs in OOD contexts. Furthermore, the comparative analysis of encoder-only and decoder-only Transformers, as well as the inclusion of state-of-the-art TSFMs (Moirai and Chronos), provides original insights into the impact of model architecture on scalability.
2. The methodology employed in this study is sound and comprehensive. The authors train a wide range of models with varying sizes (from 10^3 to 10^8 parameters), compute budgets, and dataset sizes. They evaluate the models' performance using appropriate metrics such as log-likelihood loss and MAPE. The experiments are well-designed, covering both ID and OOD scenarios, and the results are presented clearly with informative visualizations. The analysis of the scaling behavior is thorough, considering the effects of model size, compute resources, and dataset size independently.
3. The findings of this paper have significant implications for the development of TSFMs. By establishing scaling laws that hold across both ID and OOD scenarios, the authors provide a framework for predicting the expected performance gains of TSFMs as they scale up in size, compute, and data. This is crucial for guiding resource allocation and model design decisions. Moreover, the comparative analysis of different model architectures reveals important trade-offs between ID and OOD performance, highlighting the need for careful consideration when designing TSFMs for specific applications. The practical design principles derived from the study offer valuable guidance for researchers and practitioners working on TSFMs.
summary: 8648_Towards_Neural_Scaling_La
pdf
LT
Here is a new  paper, please read it in its entirety, and help me to write the review, first, you should:
Summary*
Briefly summarize the paper and its contributions. This is not the place to critique the paper; the authors should generally agree with a well-written summary.


2 / 2

Summary

This paper investigates the scaling laws of time series foundation models (TSFMs) across different data distributions and model architectures. The authors train encoder-only and decoder-only Transformer-based TSFMs with varying model sizes, compute budgets, and dataset sizes. They evaluate the models' performance on both in-distribution (ID) and out-of-distribution (OOD) data.

The key findings and contributions of the paper are as follows:

The log-likelihood loss of TSFMs exhibits similar scaling behavior in both OOD and ID scenarios, following a power law with respect to model size, compute resources, and dataset size.
When evaluated using MAPE, the scaling of all three factors (model size, compute, and data) results in greater improvements in OOD performance compared to ID performance.
Encoder-only Transformers demonstrate better scalability than decoder-only Transformers, although both architectures show similar scaling trends.
The architectural modifications introduced in two state-of-the-art TSFMs, Moirai and Chronos, primarily improve ID performance but compromise OOD scalability compared to the baseline models.
The paper provides practical design principles for TSFMs from the perspective of data, model, and compute scaling, based on the findings and comparative analysis.
Overall, this work extends the understanding of scaling laws for TSFMs from ID to OOD scenarios and across different model architectures. The authors establish a foundation for predicting the expected performance gains of TSFMs and offer insights for designing scalable TSFMs.
weaknesses: 1. Limited scope of model architectures: The study focuses on two main model architectures: encoder-only and decoder-only Transformers. While these are indeed widely used in TSFMs, the paper could benefit from including a broader range of architectures, such as encoder-decoder models (e.g., Seq2Seq) or hybrid models that combine Transformers with other neural network components (e.g., CNN, RNN). Expanding the scope of the investigated architectures would provide a more comprehensive understanding of how architectural choices affect scalability.
2. Lack of theoretical analysis: The paper primarily relies on empirical observations to establish the scaling laws for TSFMs. While the experimental results are valuable, the work could be enhanced by providing a theoretical foundation for the observed scaling behaviors. Theoretical analysis could help explain why certain scaling patterns emerge and offer insights into the underlying mechanisms that govern the relationship between model size, compute, data, and performance. For example, the authors could explore the connection between the power-law scaling and the model's capacity to learn and generalize from the training data.
3. Limited discussion on the trade-offs between ID and OOD performance: The paper highlights that architectural modifications in Moirai and Chronos improve ID performance but compromise OOD scalability. However, the discussion on this trade-off is relatively brief. A more in-depth analysis of the factors contributing to this trade-off and the potential implications for TSFM design would be valuable. For instance, the authors could investigate whether certain architectural choices (e.g., attention mechanisms, embedding strategies) are more prone to overfitting on ID data, leading to reduced OOD scalability.
4. Absence of computational efficiency analysis: While the paper considers compute budgets as a scaling factor, it does not provide a detailed analysis of the computational efficiency of the investigated models. As TSFMs scale up in size, computational efficiency becomes increasingly important. The paper could be strengthened by including metrics such as training time, inference time, and memory consumption for the different model architectures and sizes. This would provide valuable insights into the practical trade-offs between performance and computational cost.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 4
questions: - Section 3: The MAPE is consistently used as a metric in the main results, although Appendix C.3 outlines a well-known limitation, which is that it is unreliable when the time series takes values close to zero (and suggests using the sMAPE instead). It seems that for time series foundation models, which should be quite agnostic to shift and scale variations of the series, the MAPE should be the last metric to be considered. Why has not the sMAPE been used, or more robust scale-invariant methods like the MASE? (https://en.wikipedia.org/wiki/Mean_absolute_scaled_error )
- Figures 2-4: Although the NLL difference between ID and OOD is of the expected sign (i.e. with OOD having greater NLL than ID), it is truly mystifying that the opposite is consistently true for the MAPE. This is not properly addressed by the authors. In my view, this is the main issue that is causing my "Soundness" assessment not to be higher.
- Figure 5: Although encoder-only clear beat decoder-only in distribution, this difference is almost insignificant out of distribution, in contrast to the claim in the caption. If the authors intend to make this claim, they should conduct a formal statistical test of the fit difference between encoder-only and decoder-only models.
- Suggestion related to Figure 8: across these examples, it'd be instructive to plot (in appendix) what the actual forecasts look like as we transition from a regime of "relatively higher MAPE" to "relatively lower MAPE".
soundness: 2
strengths: - Originality: The paper's focus on the scaling laws of TSFMs on OOD data represents a novel and timely contribution.
- Quality: The research is well-executed, with a comprehensive methodology that involves training and evaluating a wide range of models across various parameter counts, compute budgets, and dataset sizes. The use of a large and diverse dataset further strengthens the quality of the empirical analysis.
- Clarity: The paper is well-written and organized.
- Significance: Author's findings could have significant implications for the development and deployment of TSFMs, subject to comments and suggestions in the remainder of this review.
summary: The authors examine the scaling laws of Time Series Foundation Models (TSFMs), both encoder-only and decoder-only Transformers, on in-distribution (ID) and out-of-distribution (OOD) data. They find that the NLL (and to some extent the MAPE) of TSFMs shows similar scaling behavior in both ID and OOD settings, scaling as a power law with model parameters, compute, and dataset size. Results suggest that encoder-only Transformers exhibit better scalability than their decoder-only. The authors propose design principles for building scalable TSFMs, focusing on data, model architecture, and compute, recommending encoder-only Transformer architectures for better scalability. Authors also study "emergent behaviors" in TSFMs, where performance on certain OOD datasets improves abruptly after reaching a critical model size.
weaknesses: - Line 046: provide citations for neural scaling laws that provide ground for believing that they exist.
- Table 1: review the "proportion" row against the "Time points" row. Sales is 0.96% of dataset with 140M points, whereas Web is 0.40% of dataset with 600M points.
- Line 149: "Given its proven effectiveness in improving time series forecasting performance (Woo et al., 2023), we adopt RoPE as a replacement for the original Transformer’s positional encoding." ==> my reading of Woo (2023) is that the evidence in favor of RoPE for time series forecasting is very weak (results presented without confidence bands, and do not appear statistically significant), and likely that better evidence should be presented than the statement of fact that RoPE is "proven" — I don't see that it is, especially in the context of a foundation model.
- Line 190: The paper consistently talks about "log likelihood", for which higher values indicate a better fit. However, all results seem to be obtained under the "negative log-likelihood" (NLL, which is the loss counterpart), for which lower values indicate a better fit. This elementary terminology confusion should be clarified corrected.
- Section 3: The paper only studies Transformer-based architectures, without any baseline results with simple time series models (e.g. ETS-type exponential smoothing or ARIMA) — which could be shown as horizontal lines in the scaling law plots. In particular, the MAPE is around or below 1% for most of the studied scaling regime, which suggest that the datasets are very highly predictable: it is likely that simpler models would do well on these datasets too, bringing into question whether TSFMs' additional complexity is warranted.
- Appendix C.2: Since the training and test sets are taken from long-available public sources, is there a contamination analysis of the test set against the training set to ensure there's no overlap?

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: Please check the Weaknesses section for more questions!
soundness: 3
strengths: As to the originality, the authors explore a relatively new area in time series forecasting by expanding the concept of scaling laws from in-distribution (ID) settings to out-of-distribution (OOD) contexts. This fresh perspective enhances the relevance of scaling laws and addresses an important gap in understanding how model architectures perform across various data distributions. By comparing encoder-only and decoder-only transformer architectures, the paper merges ideas from neural scaling laws with practical applications in time series analysis, offering new insights into model design! 

The work quality is underscored by the rigorous experimental methods the authors use. They systematically evaluate various time series foundation models across parameters such as model size, computational resources, and dataset dimensions. This comprehensive approach ensures the findings are reliable, providing a strong basis for the paper's conclusions. Using clear performance metrics, like log-likelihood loss and MAPE, adds credibility to the results! 

The paper's clarity is another strength. Concepts are explained in a structured and accessible way, enabling readers from diverse backgrounds to follow the research. The logical flow and transparent presentation of methods and findings make the implications of this work easy to understand. Detailed discussions on scaling behaviours among different architectures further clarify the complex relationships between model design and performance! 

This work is significant for both theoretical and practical reasons. It advances the understanding of time series foundation models and provides empirical insight and design principles for scaling, benefiting researchers and practitioners in developing more effective forecasting models. The insights offered could influence future studies and propel advancements in time series forecasting and neural network design!
summary: This paper examines how time series foundation models scale, focusing on the performance of both encoder-decoder and decoder-only transformer architectures in in-distribution (ID) and out-of-distribution (OOD) settings. Through a series of experiments, the authors analyze how changes in model size, computational power, and dataset volume impact these models' effectiveness. One major contribution of this study is the adaptation of existing scaling laws to OOD scenarios, providing a framework to anticipate performance improvements in OOD forecasting based on key training variables. The paper also reveals that encoder-only transformers generally perform better and scale more effectively than decoder-only models. Additionally, the authors offer practical guidelines for building robust, scalable time series foundation models, emphasizing the value of training data diversity, model size, and architectural choices to boost both performance and generalization.
weaknesses: (1) One notable limitation in this study is its minimal exploration of how different model architectures impact scalability. Although the paper briefly acknowledges architecture's role in scaling, it falls short of offering a thorough analysis on how specific architectural choices, like attention mechanism, layer normalization, or residual connections, might influence scaling behaviour. A deeper, comparative study covering a broader set of architectures could reveal which design choices are most beneficial for performance across diverse data distributions! 

(2) The study's experiments center on log-likelihood loss and MAPE as key performance metrics. While these are certainly relevant, they do not fully capture the complexity of model performance in time series forecasting. A wider range of evaluation metrics, like the Continuous Ranked Probability Score (CRPS) or quantile loss, could offer a more comprehensive view of performance and its practical impact. Although the paper briefly suggests future research might investigate transformations between log-likelihood and forecasting metrics, including preliminary results or further discussion on this topic within the study would enhance the depth and clarity of the findings! 

(3) The paper would benefit from a deeper exploration of why performance tends to degrade on OOD data. Although the authors highlight that scaling the model improves OOD performance, they stop short of examining the mechanisms driving this effect. An in-depth look at how the training data distribution relates to OOD performance could offer valuable insights for practitioners applying these models in a real-world context. For example, investigating domain adaptation techniques or transfer learning strategies might help reduce the performance drop seen in OOD situations! 

(4) The discussion on sample efficiency is insightful but would benefit from concrete examples or case studies that demonstrate how larger models achieve improved performance with fewer training steps. Including specific examples or empirical data to back up this claim would strengthen the credibility of the findings and provide practical guidance for researchers and practitioners looking to optimize their training processes! 

(5) The dataset used for training and evaluation seems carefully chosen but lacks a detailed account of its diversity and representativeness. While the paper highlights the role of dataset size and variety in enhancing OOD performance, it provides limited insights into the dataset's specific characteristics, such as the types of time series it includes, their distributions, or any noise or anomalies present. A deeper examination of the dataset's attributes would offer a clearer picture of the models' performance and their relevance to real-world applications! 

(6) Another important limitation is the lack of attention to how scaling laws impact practical applications. The authors offer theoretical insights into scaling behaviour but do not go far enough in showing how these findings can be turned into useful strategies for practitioners. For example, guidance on balancing model size, computational resources, and dataset size in real-world settings would be valuable for those aiming to apply time series foundation models effectively. Including case studies or examples of successful implementations could also help demonstrate the research's relevance to industry professionals!

Meta-review: (a) Summary of Scientific Claims and Findings
This paper investigates scaling laws for Time Series Foundation Models (TSFMs), focusing on both in-distribution (ID) and out-of-distribution (OOD) settings. The key contributions include:
Empirical Findings on Scaling Laws: The study demonstrates that the log-likelihood loss of TSFMs scales predictably with model size, compute resources, and dataset size across both ID and OOD contexts, adhering to power-law relationships.
Architectural Comparisons: Encoder-only architectures exhibit superior scalability in ID tasks compared to decoder-only models, with similar OOD scaling trends. However, architectural enhancements in state-of-the-art TSFMs, such as Moirai and Chronos, improve ID performance at the cost of OOD scalability.
Emergent Behaviors: The authors identify critical model sizes at which TSFMs demonstrate abrupt improvements in OOD performance.
Practical Guidelines: Based on the findings, the authors propose design principles for scaling TSFMs, balancing architecture, data, and compute considerations.

(b) Strengths of the Paper
Novel Focus: Extends the study of scaling laws to time series data, addressing gaps in the understanding of OOD scaling behavior for TSFMs.
Comprehensive Analysis: Evaluates models across a wide range of sizes, datasets, and compute budgets, providing robust empirical insights.
Actionable Contributions: Offers practical guidelines for designing scalable TSFMs, which can inform resource allocation and architecture design.
Emergent Phenomena: Identifies novel scaling behaviors, such as critical model sizes for OOD performance, contributing to foundational knowledge.
Well-Executed Rebuttal: The authors thoroughly addressed reviewer concerns, incorporating additional experiments, metrics (e.g., sMAPE, MASE), and theoretical discussions to strengthen the paper.

(c) Weaknesses of the Paper
Limited Architectural Scope: Focuses on encoder-only and decoder-only Transformers, omitting other architectures such as encoder-decoder hybrids or non-Transformer-based models.
Trade-off Analysis: The discussion of trade-offs between ID and OOD performance, particularly for architectural modifications, could be expanded.
Efficiency Metrics: While compute budgets are considered, detailed analyses of training/inference time and memory consumption are missing.
Focus on Univariate Data: The findings are restricted to univariate time series, leaving open questions about scalability in multivariate contexts.

(d) Reasons for Acceptance
Significant Contribution to TSFMs: The paper provides a novel and comprehensive exploration of scaling laws for TSFMs, addressing both ID and OOD scenarios—a critical area of research for developing robust, large-scale time series models.
Practical Utility: The proposed design principles and findings have direct implications for researchers and practitioners working on scalable time series models.
Comprehensive Rebuttal: The authors have addressed reviewer concerns with additional experiments, clarifications, and revisions, demonstrating the soundness and robustness of their work.
Community Impact: By identifying emergent behaviors and offering actionable insights, the paper lays a foundation for future research in TSFMs and scaling laws.

Overall, the strengths and contributions of the paper significantly outweigh its weaknesses. The empirical findings, combined with the proposed design principles, represent a valuable advancement for the time series modeling community.

Decision: Accept (Poster)