Sundial: A Family of Highly Capable Time Series Foundation Models
Authors: Yong Liu
Guo Qin
Zhiyuan Shi
Zhi Chen
Caiyin Yang
Xiangdong Huang
Jianmin Wang
Mingsheng Long
Venue: ICML 2025 oral (2025)
Keywords: Time Series
Foundation Models
Primary area: deep_learning->foundation_models
TL;DR: We introduce Sundial, a family of native, flexible, and scalable time series foundation models pre-trained on a trillion time points.

Abstract: We introduce Sundial, a family of native, flexible, and scalable time series foundation models. To predict the next-patch's distribution, we propose a TimeFlow Loss based on flow-matching, which facilitates native pre-training of Transformers on continuous-valued time series without discrete tokenization. Conditioned on arbitrary-length time series, our models are pre-trained without specifying any prior distribution and can generate multiple probable predictions, achieving more flexibility in representation learning than using parametric densities. Towards time series foundation models, we leverage minimal but crucial adaptations of Transformers and curate TimeBench with one trillion time points, comprising mostly real-world datasets and synthetic data. By mitigating mode collapse via TimeFlow Loss, we pre-train a family of Sundial models on TimeBench, which achieve unprecedented model capacity and generalization performance. In addition to excellent scalability, Sundial achieves state-of-the-art results on both point and probabilistic forecasting benchmarks with a just-in-time inference speed, i.e., making zero-shot predictions within a few milliseconds. We believe that Sundial's pioneering generative forecasting capability can improve model reliability in real-world decision-making. Code is available at: https://github.com/thuml/Sundial.

Sundial: A Family of Highly Capable Time Series Foundation Models
Yong Liu
Guo Qin
Zhiyuan Shi
Zhi Chen
Caiyin Yang Mingsheng Long
Xiangdong Huang'
Jianmin Wang'
Abstract
Time Series Data
Continuous Token
Discrete Token
We introduce Sundial, a family of native, flexible, and scalable time series foundation models. To predict the next-patch's distribution, we propose TimeFlow Loss based how-matching, which facilitates native pre-training of Transformers on continuous-valued time series without discrete to- kenization. Conditioned on arbitrary-length time series, our models are pre-trained without specify- ing any prior distribution and can generate multi- ple probable predictions, achieving more flexibil- ity in representation learning than using paramet- ric densities. Towards time series foundation mod- els, we leverage minimal but crucial adaptations of Transformers and curate TincBench with one trillion time points, comprising mostly real-world datasets and synthetic data. By mitigating mode collapse via TimeFlow Loss, we pre-train a family of Sundial models TimeBench, which achieve umprecedented model capacity and generalization performance. In addition to excellent scalability, Sundial achieves state-of-the-art results on both point and probabilistic forecasting benchmarks with a just-in-time inference speed, i.c., making zero-shot predictions within few milliseconds. We believe that Sundial's pioneering generative forecasting capability can improve model reliabil- ity in real-world decision decision-making. Code is avail- able at:
8.12, 2.33, 2.47.3.40 3.4,6...
34
Continuous Tokenization (Native)
Discrete Tokenization (Foreign)
Time Series Foundation Models
Specific
Flexible
[1) TimesFM, Times, Moiral
[2] Chranas, LLMTine
(3) Sundial (Ours)
0000
0000
Emropy
0000
0000
Model
Prior Loss
Model
Model
Time
Parametric Densities
Language Modeling
Generative Modeling
Figure 1. A native time series model operates the original series of continuous values. A flexible foundation model is pre-trained without specifying prior distributions. Sundial is the first family of native and flexible time series foundation models.
2015). Therefore, generating variety of probable predic- tions is crucial for decision-making The growing demand has facilitated numerous statistical approaches over the past decades (Hyndman, 2018; Box, 2013), which provide high- profile theories and probabilistic tools for making reliable schedules. Recent advancements bring the boom of deftly designed models that automatically learn intricate dynamics and correlations from raw data (Oreshkin et al., 2019; Nie et al., 2022: Zhang & Yan, 2023; Liu et al., 2023a). Despite the impressive performance, deep models necessitate task- specific training on sufficient in-distribution data. Motivated by advances large models (Bommasani et al., 2021), pre- trained time series foundation models have shown promising capabilities in Liu et al., 2024b; Woo et al., 2024; Ansari et al., 2024).
tasks (Das et al., 2023b;
1. Introduction
Time series forecasting has fascinated people for thousands of years. Although people have been able to determine the time using instruments like sundials in 3000 BC, time se- ries forecasting is intrinsically non-deterministic (Box et al.,
Current research on time series foundation models has con- verged on building unified, scalable, and out-of-the-bor fore- casters, exhibiting zero-shot performance close to or some- times surpassing supervised methods (Aksu al.. 2024). Notably, Transformers (Radford et al., 2018) are currently the de facto architecture of these models. While pre- trained Transformers with an inherent generative ability have facil- itated great success in language, image, and video genera- tion (Ramesh et al., 2021; OpenAl, 2023; Liu et al., 2024c), most time series foundation models are not "generative" or, more specifically, probabilistic forecasters, thereby limiting
'Equal contribution 'School of Software, BNRist, Tsinghua University. Yong Liu <lluyynn21 @ maill. www.llsssiggua.eu...c. edu.cn>> Guo Qin <qinguo24 @ mails. taiiss.t www.aaadu.oo... >n>> Correspondence to: Mingsheng Long <mingsheng @wwsinguaa.du.cn....
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Sundial: A Family of Highly Capable Time Series Foundation Models
reliability in decision-making. Although parametric densi- ties specified with prior distributions (Wen et al., 2017; Woo et al., 2024) can be adopted to address uncertainty in time series forecasting, they can reduce the capacity of distribu- tions learned by pre-trained models, especially on time se- ries modality characterized by high heterogeneity. To learn arbitrarily intricate distributions without mode collapse, lan- guage modcling (Bengio al., 2000) that learns the categor- ical distribution via cross-entropy loss inspires subsequent works (Gruver et al., 2023; Ansari et al., 2024), which treat time series as a foreign language using discrete tokenization. Still, discrepancies between continuous-valle valued time series and discrete language tokens can lead to out-of- ofovobbullary issues and coarse-grained prediction intervals.
Our contributions lie in these aspects:
We propose TimeFlow Loss to predict next-patch's is dis- tribution, allowing Transformers to be trained without discrete tokenization and make probable predictions.
We present Sundial, family of scalable and efficient time series foundation models built upon our enhanced Transformer and pre-trained on a trillion time points.
Experimentally, Sundial achieves tate-of-thhe-ate zero- shot performance on point and probabilistic forecasting benchmarks, including TSLib, GIFT-Eval, and FEV, in- dicating a promising generative approach for the future improvement of time series foundation models.
As shown in Figure 1, Sundial is presented as the first family of generative models among time series foundation models. As foundation models intend to learn complicated distribu- tions from extensive datasets and facilitate transferability across agnostic downstream datasets, we do not specify any prior parametric densities, such as unimodal and multimodal Gaussian mixtures. Instead, we delve into generative mod- eling to tame Transformers as native, flexible, and scalable time series foundation models. By comparing denoising diffusion models (Li et al., 2024), we opt for simple yet effective fow-matching framework (Lipman et al., 2022), which provides notable efficiency and sample quality (Tong et al., 2023). We propose TimeFlow Loss, a parameterized training objective (Zhang et al., 2018) for autoregressive models to learn and sample from each token's predictive distribution. Optimizing models in the original continuous- valued domain, TimeFlow Loss facilitates patch-level gen- eration and enables fast inference, which is naturally com- patible with the time series modality.
2. Related Work
2.1. Time Series Forecasting
Forecasting is essential for decision -making.. Advancements in deep learning for time series include theory-inspired deep modules (Wu et al., 2021; Liu et al., 2023b; Wu et al., 2022), architecture-orientee adaptations (Bai et al., 2018; Salinas et al., 2020; Lim et al., 2021), and time series preprocess- ing (Kim et al., 2021; Nie et al., 2022). Deep models learn the dataset-level distribution and benefit from strong gener- alization and model capacity. Statistical methods conduct case-by-case fitting on input series, achieving notable per- formance small data (Ke et al., 2017; Hyndman, 2018).
One of the efforts towards more capable forecasters focuses on the foundation models (Bommasani et al., 2021), which address data-scarce scenarios by pre-training. More capable models support zero-shot forecasting, making inferences as fast as statistical methods and possessing large model capac- ity as deep models. Another aspect is to address uncertainty in time series forecasting. There is growing research em- phasis on probabilistic forecasting (Woo et al., 2024; Ansari et al., 2024). While parametric densities can be adopted as training objectives of probabilistic forecasting, they can be too specific to meet the heterogeneity of large-scale datasets, resulting in mode collapse in representation learning and over-smooth predictions (Figure 14-15). In this work, we introduce generative time series foundation models, which naturally address the uncertainty in forecasting.
In addition to TimeFlow, we enhance the Transformer with minimal but critical adaptations. We develop feasible patch tokenization for arbitrary-length input time series. We adopt RoPE (Su et al., 2024), Pre-LN (Xiong et al., 2020), FlashAt- tention (Dao et al., 2022), and KV Cache (Pope et al., 2023), which are crucial but generally neglected in the development of time series foundation models. Besides, we pre-train our models by multi- patth- prediction to reduce autoregression steps. We realize a rapid generation of multiple samples by reusing shared lookback representation. Beyond facilitat- ing scalable pre-training, these adaptations help real-time long-con long-context inference and long-tern term generation.
2.2. Time Series Foundation Models
To validate the scaling law of time series foundation models, we collect and curate TimeBench with an unprecedented vol- ume of a trillion time points. We present Sundial as a family of highly capable foundation models, which achieve of the-art on three large-scale and best-recognized benchmarks, including Time-Series-Librayy (TSLib) (Wu et al., 2022), GIFl-Eval (Aksu et al., 2024), and FEV (Ansari et al., 2024).
Recent research has concentrated on building versatile large time series models (Liang al., 2024). With the advances made in large language models, Transformer has become the dominant architecture. Several works adapt Transformers to address the unique 2D-dimensionality and heterogeneity of time series (Woo et al., 2024; Liu et al., 2024a). Specif- ically, our work delves into tokenization and optimization.
2

---

Sundial: A Family of Highly Capable Time Series Foundation Models
Models such TimesFM (Das al., 2023b), Timer (Liu al., 2024a;b), and Time-MoE (Shi al., 2024b) embed continuous values and fit unimodal distributions via MSE or quantile loss (Wen et al., 2017). However, prior loss may result in mode collapse because predictive distributions are highly divergent across different domains. Besides, these models cannot provide the confidence level of predictions, limiting reliability for decision makiing. Based on continu- ous tokenization, Moirai (Woo al., 2024) presents prob- abilistic model learning mixture of distributions, but this prior can still fail to accommodate complex distributions. In- spired by language modeling, Chronos (Ansari et al., 2024) discretizes series via bucket quantization, learning more flex- ible categorical distributions by cross-entropy. Still, discrete tokenization is applied at each time point, which can lead to long contexts. Also, the final performance can be sensitive to quantization techniques. Unlike before, we tame Trans- formers as native time series foundation models, learning flexible distributions without discrete tokenization.
from a source distribution into sample X1 P1 drawn from target distribution. The transformation is continuous in time. For d-dimensional distributions, it is defined by time- demeddeenddent velocity field u, [0, 1] x R - R which is the solution of the ordinary differential equation (ODE);
d
t/a (x) = uy (v)()xx)) and vx((x) = x.
dt
The velocity field un determines a flow th.. For all [o, 1]. th generates the probability path pt that interpolates pu and Pl, i.e., x, = the (Xo) ~ Pt for X ~ po- The implementa- tion of flow-matching is to train a network uy parametrized by 0 to fit the velocity field u, which is a regression- task formulated as the Fowt-attching objective:
LrM (0) = Ex,, [xx](x)) uu (xx)x]]l²
Furthermore, Lipman et al. (2022) proved the equivalence of optimizing the Conditional Flow-Matching objective:
CCFM (6) xx,x (xxe) uu (xx|><1)]]
2.3. Generative Modeling for Time Series
Leveraging the conditional optimal-transpor (linear) path and a source Gaussian, the objective can be formulated as:
By addressing complicated distributions during pre-training, generative modeling has become a focal point in the devel- opment of various foundation models (Zhao et al., 2023; Liu et al., 2024c). While this direction for time series mostly focused on time series generation (Tashiro et al., 2021) and task-specific forecasters (Rasul al., 2021; Shen & Kwok, 2023; Kollovieh et al., 2024), generative modeling for time series foundation models is hardly explored. With the com- parable flexibility in distribution learning as language mod- eling, diffusion denoising (Sohl-Dickstein et al., 2015) and flow-matching (Lipman et al., 2022) have gained increasing prevalence in continuous-valued modalities (Lipman et al., 2024). Compared with diffusion denoising models, flow- matching provides simple yet efficient framework. With fewer steps involved in the forward and reverse processes, large models based on low-matching have shown superior performance in image generation (Esser et al., 2024).
Clanss (0) = Er,,,x || (xx) - (x1 - -xo)]l (1) where ~~ u(0 1]. NN(0 1) and x, = |x1 + (1 - l)e.
Consequently, we can train a generative network on given samples from the target distribution, and generate new sam- ples by applying push-forward process on samples drawn from a simple source Gaussian distribution:
X++AI X2 = (x: Xo N(O, t [0.1]] (2)
3.2. Generative Models for Probabilistic Forecasting
Given a historical observation F1#t = {ar,, ...; Jt the tar- get of time series forecasting to predict future time series F (Iz+1, Ft+f). The task can be generally formulated p(æ [h), where h, fø(11a) is the learned representation from deep model fa. In probabilis- tic forecasting, explicit optimization objectives are utilized to predict the statistics of future series, e.g., MSR or quantile loss, which have specified p as a prior distribution. While using one parametric density generally fits well on a small amount of data, it can be the major bottleneck for scaling time series foundation models. Inspired by the success of large generative models (Rombach et al., 2022; OpenAl, 2023; Esser et al., 2024). we introduce generative modeling to realize probabilistic forecasting:
Despite the connection in value continuity, generating im- ages and future time series are fundamentally different tasks due to the autoregressive property of forecasting. Our pro- posed TimeFlow Loss is designed for autoregressive models to conduct conditional generation, which is parameterized loss function (Zhang et al., 2018) for arbitrary distributions and enhances representation learning of foundation models.
3. Preliminaries
3.1. Flow-Matching
The goal of generative modeling is to learn the underlying probability distribution that generates the data. The frame- work of flow-matching transforms sample xo Po drawn
p(2++104/ (h,) go (ff(11)))
(3)
go is a small trainable generative network conditioned on the learned representations of so, which is jointly optimized
3

---

Sundial: A Family of Highly Capable Time Series Foundation Models
Tokenization
Transformer
TimeFlow Loss
N
1
Inference
Step k
RoPE
PatchEmbed
2
2
(0)
(hA()
y
(1)
Re-Norm
Pre-LN
FlashAttentio
& KV Cache
Pre-LN
FFN
3
Noise
Patch
Velocity
Condition
Flow-Matching Network
h;
Training
Time & Noise
Truth
Figure 2. Overall architecture of Sundial. The input time series is divided into patch tokens, which are embedded from original continuous values. The patch embeddings fed into decoder-only Transformer, stable and speedup version that lears token representations via causal self-attention. The model is optimized using our TimeFlow Loss, a parameterized loss function that models per-token probability distribution conditioned on the learned representations, and generates multiple plausible predictions under the flow-matching framework.
with fa. While the generative model automatically fits the target distribution, it can sample raw predictions and calcu- lating their statistics for probabilistic forecasting. The aim conceptually related to conformal prediction (Vovk et al., 2005) but models uncertainty beyond prediction intervals.
dress temporal distribution shift and outlier ranges in input series, improving generalizability for zero-shot forecasting.
Patch Embedding Given a univariate time series X (T1,..., FT }, it divided into patches x; (1+((-1)P=SP with the length of P. To address non-divisible length, pad the input at the beginning and use binary mask m, R for each patch to indicate the padded position. It will lead to N = [T/P] such input tokens. Subsequently, we use a shared MLP g2r + RD to embed all patch tokens:
4. Approach
In this work, we conduct univariate pre-training paradigm, which adopts the S3 format proposed by Liu et al. (2024b) to address multivariate data. To mitigate value range discrep- ancy, we conduct normalization on time series individually per variable. Afterwards, we sample varying-lenglo training samples with the maximum context length of 2880. As foundation model, Sundial is required to predict on out-of- distribution series with varied lengths during inference.
h PatchEmbed ( (Conca((x; m.)),
(4)
where h, E R and D is the dimension of token embedding. Unlike point-level quantization (Ansari et al., 2024), we reserve original values without discrete quantization. also reduces the context length (in token) of the Transformer.
4.1. Sundial
4.1.2. TRANSFORMER BACKBONE
Given N token embeddings (h,), ). we adopt several crucial adaptations on a decoder-only Transformer to obtain per- token representations aggregated from all previous tokens. First, we adapt Pre-LN (Xiong et al., 2020) to improve pre- training stability. Second, we leverage a causal self-attention mechanism with RoPE (Su et al., 2024) that introduces the position information of patch tokens. It can be formulated as follows (the layer index is omitted for simplicity):
As shown in Figure 2, the Sundial models consist of three parts: (1) time series tokenization, including a context-level re-normalization and patch embedding that addresses any- length time series, (2) Transformer backbone that learns the per-token representation of time series, and (3) Time- Flow Loss, a parameterized loss function model the per- token distribution and generate series during inference. Intuitively, Sundial can be regarded as an ARMA (Auto- Regression and Moving-Average deep model, i.e., Trans- former learns token representations autoregressively. Con- ditioned on the lookback representations, TimeFlow trans- forms random noises into non-deterministic predictions.
Ay h R,R..,,W,T... hy:
MMsk((A)
(5)
Attention( (H) Softmax
HW
Vd
where W. Wk. W. Dxd project token embeddings
1.1.1. TIME SERIES TOKENIZATION
H {h:} into d-dimensional queries, keys, and values.
R E Rdxd is the rotary matrix with rotation degree (l-O). Lastly, we implement FlashAttention (Dao et al., 2022) and KV Cache (Pope et al., 2023), since these enhancements for deployment are increasingly emphasized in large foundation models (Shoeybi al., 2019; Rasley et al., 2020).
Re-Normalization We adopt stationarization (Liu al., 2022), non-parametric two-stage instance normalization conducted within each sample, which is initially proposed to mitigate non-stationarity of time series, Here, it helps to ad-

---

Sundial: A Family of Highly Capable Time Series Foundation Models
4.1.3. TIMEFLOW LOSS
Algorithm 1 Floe Loss: Sampling
Require: condition h, E RR, path steps K. 1: Sample initial noise y, ~ (0.
Given representations }h:} extracted by the last layer of the Transformer, we aim to generate length-F predictions F+PP at each position i via our autoregressive model. Motivated by the empirical observation that a larger patch size improves the performance in decoder-only Trans- formers (Das et al., 2023b) while a small patch size can be more flexible to accommodate data of different frequencies, we adopt multi-patch predictions (F > P) for pre- pre-training, which also reduces the steps of autoregressive inference.
2: At = =1/K
y. 11+iP, P++P
3: for k in 10.1.... K 1] do
4: y: 4 Fr FM-Nee (Si, Kd1 h;)
end for
6: Return: Чr
4.2. TimeBench
Based on Equations I and 3, we formulate new generative forecasting conditioned on sequential representation h;:
We collected and curated TimeBench, which comprises over trillion time points from various sources, as shown in Fig- ure 3. Several datasets originate from research teams (Woo et al., 2024; Ansari et al., 2024; Liu et al., 2024a:b). While most datasets are collected from real-world records, a small portion (0.05%) is generated synthetically to enhance pat- tern diversity, following KernelSynth proposed by Ansari et al. (2024). We also leverage substantial meteorological data (Hersbach et al., 2020) because of the predictability of weather systems. Data of different frequencies encompasses common and comprehensive temporal dynamics.
(+) lb.) (y: C(0, h.) = Er,,.y |uf (v.
- )1 (6)
where y, E ER is the groundtruth value and y; (0) is a d- dimensional Gaussian noise, is sampled from (0 1). and constructed by the conditional yi -t; + (1 - (1
optimal-transport path. It is important to note that the con- ditional representation h, differs from the conditional path and the conditional source distribution. Instead, h, is con- dition of position i, also a time-invariant condition of the whole flow-man fow-matching process é E [0, 1]. Technically, we implement the flow- mawching network by a small MLP:
1.31%
0. 0.44%
Chronas
ECG
ui (y: (1) [h,) = FM-Net ,t."t,,h)))
5.62%
9.11
4.65%
Finance
1.02%
lloT
The training process involves sampling the noised y; jointly imput it with t. The condition h, is integrated into the flow-matching network via AdaLN (Peebles & Xie, 2023). TimeFlow Loss for autoregressive models is formulated as:
(1) and
0.56%
LOTSA
Synthetic
39.35%
22.29%
ERAS 3h
ERRS 12h
N
ERAS Daily
FM-Net (y (y,),, h.) - -yy, - y: ||
LimmeFlow
12.50%
ERAS Weekly
3.10%
00.55%
ERAS Monthly
TimeBench (1 Trillion Time Points)
ERAS Quarterly
Figure 3. Ratios of data sources in TimeBench, the pre- traininin corpora of Sundial. Detailed statistics are provide in Table 4.
Inference Based on Equation 2, the forw-dorward process conditioned on a learned representation h; is formulated as
= y + (y (0)h) AV.
5. Experiments
(9)
Technically, we adopt K-step uniform trajectory, and set Al = 1/K. The sampling is done via starting from initial Gaussian noise and advancing with the velocity generated by the trained FM-Net. iteratively, as shown in Algorithm 1.
We evaluate Sundial on best-recognized zero-shot forecast- ing benchmarks (Section 5.1) and investigate the scaling behavior of Sundial (Section 5.2). We compare TimeFloww with other training objectives (Section 5.3). We delve into test-time calibration of generative forecasters (Section 5.4). We conduct model adaptation of Sundial, i.e., instruction tuning (Section 5.5) and provide in-depth ablation studies to evaluate our modular enhancement (Section 5.6).
This procedure generates a predicted sample y, at position To calibrate probabilistic forecasting results during infer- ence, we repeat this procedure using different initial noises and estimate statistics such as the median and quantiles from set of generated predictions. We implement an efficient sepeated-sampling in the TimeFlow module. The condition (representation) of lookback series is shared and reused for different initial noises, thereby reducing the overhead of repeated forwarding in the Transformer backbone.
5.1. Time Series Forecasting
In this section, we focus on zero-shot forecasting, we com- pare Sundial with advanced time series foundation models on various benchmarks, including (1) point forecasting: we
5

---

Sundial: A Family of Highly Capable Time Series Foundation Models
Table 1. Zero-shot forecasting results of time series foundation models on long-term forecasting datascts (Time- Semee-Srress-Jiiee Lebrers (Wu et al., 2022). Corresponding prediction lengths include {96, 192, 336, 720). A lower MSE or MAE indicates better prediction. Averaged results of four prediction lengths are reported here. 1st Count represents the number of wins achieved by a model under all prediction lengths and datasets. Results of baseline models are officially reported by Shi et al. (2024b). Datasets in pre-training are not evaluated on corresponding models, which are denoted by the dash (-). Full results under all prediction lengths are provided in Table 9.
Models
Sundialsowr Sundialau Sundialiagy Time-MoEss Time-MoE.agr Tine-MoEss. (Ours)
Timer-XL Moiraia
Moiraitagr (2024)
Chronosows Chronoscapy
TimesFM (2023b)
(Ours)
(Ours)
(2024b)
(2024b)
(2024b)
(2024a)
(2024)
(2024)
(2024)
Metric
MSE MAE] MSE MAE MSE MAE MSE MAE
MSE
MAE
|MSE
MAE
MSE MAE MSE MAB MSE MAE MSE MAE MSE
MAE
MSE MAE
ETTml
[0.354
0.388
0.388/0.336
103366
0.377/0.331
0.377
0.331
0.369
(0.394 0,415
|0.376
0.405
|0.356
0.391
[0.373 0.392 0.406 (0.52 0.391 I 0.645 0.500
(0.555
0.465
[0.433 0.418
ETTm2
|0.265
0.324|6
(.2558
0.320
0.254 0.315 (0.317 0.365
(0.316
0.361
|0.288
0.344
[0.273
0.336 0.311 0.337 0.343 0..10 0.350
(0.295
0.338
00.328 0.346
ETThl
[0.390
0.418/0.411
0.418
[0.411
0.3395
0.420 (0.400
0.424
|0.394
0.419
|o.412
0.426
[0.404
0.417 0.417 0.419/0.480
0.439 0.591 0.468
(0.588
0.466
[0.473 0.443
ETTh2
[0.340
0.387|
[0.333
0.387
0.384
0.334
0.387
(0.366
0.404
[0.405
0.415
[0.371
0.399
[0.347
0.388 0.362
0.382 0.377 00:45
0.410
(0:455
0:427
|0.392 0.406
ECL
[0.169
0.265
16.
0.265
0.166
0.262
00:174
0.278 0.1877
0.274|0.188 0.270
0.214
0.278
|0.204
0.273
Weather
[0.233
0.271
[0.234
0.270
0.238
0.275
(0.265
0.297
|0.270
0.300
|0.256
0.288
(0.256 0. 0.294
0.287 0.281 [0.264
0.273
0.292
0,315
(0.279
0.306
1 Count]
7
2
8
5
16
16
1
2
1
3
6
0
0
0
0
Table GIFT- Eval comprises 23 datasets characterized by a variety of frequencies, variate numbers, and prediction lengths. We evaluate
zero-shol performance using 100 generated series, being consistent with Woo et al. (2024). A lower MASE or CRPS indicates better performance. Rank assigns a numerical ranking of all 97 configurations. Baseline results are officially reported by Aksu et al. (2024).
Type
Statistical Methods
Task-Specific Models (Superwised)
Time Series Foundation Models (Zero-Shot)
Scusonal Naive
Auto ARIMA
Auto
DeepAR (2020)
TIDE
N-BEATS (2019)
PTST. (2022)
iTrans.
TimesFM
TabPFN
Chronos (2024)
Moirai (2024)
Sundial (Ours)
Model
Naive
Theta
(2023a)
(2023a)
(2023b)
(2025)
MASE
1,260
1.000
0.964
0.978
1,206
0.980
0.842
0.762
0,802
0.680
0.748
0.786
0.809
0.673
CRPS
1,383
1.000
0.770
1.051
0.721
0.652
0.689
0.496
0,524
0.465
0,480
0.551
0.515
0.472
Rank
28,072
26.175
21.515
24.031
18.938
18.557
21.381
10.052
11.320
8.237
8.268
14,309
10.175
9.062
adopt the long-term forecasting benchmark (Wu et al., 2022), which assesses the performance under different forecasting horizons using MSE and MAE; (2) probabilistic forecasting: we experiment on GIFT-Eval (Aksu et al., 2024) and FEV leaderboard (Ansari et al., 2024), following their official evaluation suite and assessing point (MASE) and probabilis- tic (CRPS and WQL) metrics. All evaluated datasets are excluded from the pre-training dataset. Model is available on HuggingFace' and configurations are detailed in Table 5.
lowing Ansari et al. (2024), we calculate the median and quantiles using a set of raw predictions of Sundial. While several baseline models have been pre-trained by the con- sistent objective function for probabilistic evaluation, e.g., quantile loss for WQL, Sundial calculates these statisties for evaluation without any prior knowledge.
GIFT-Eval Aggregated results are presented in Table 2. The benchmark evaluates performance from 23 datasets and 13 baseline models, encompassing statistical methods, task-specific models, and time series foundation models. Among supervised models and advanced foundation models, Sundial attains the first place in MASE and second place in CRPS on all unseen datasets. While the top PatchTST (Nie et al., 2022) is exhaustively trained and tweaked on each dataset, the zero-shot performance of Sundial highlights its simplicity and robustness on this comprehensive benchmark.
5.1.1. POINT FORECASTING
As shown in Table 1, Sundial consistently outperforms other advanced time series foundation models. Compared with the previous otate-of-the-art model Time-MoE (Shi et al., 2024b), the Sundial family using fewer parameters achieves the average MSE reduction of 7.57% and averaged MAE re- duction of 4.7%% Notably, continuous tokenization allows our model to conduct patch-level forecasting with fewer au- toregression steps, while Chronos using point-wise discrete tokenization may not be suitable in long-term forecasting,
FEV Leaderboard We evaluate our Sundial on the open leaderboard established by AutoGluon (Ansari et al., 2024). which includes 27 datasets for probabilistic forecasting. As shown in Figure 4, the zero-shot forecasting performance of Sundial exceeds 70% statistical methods and deep models that are superwisedly trained in distribution. While Sundial is ranked as the second zero-shot pre-trained models after Chronos, Sundial realizes 35x inference speedup as shown in Figure 5. Based on patch- wise tokenization and multi-
5.1.2. PROBABILISTIC FORECASTING
Beyond point forecasting, Sundial possesses a unique gen- erative capability for making probabilistic predictions. Fol-
http://ttngiiiiisssccoommmmmmmaa 128m.
6

---

Sundial: A Family of Highly Capable Time Series Foundation Models
Statistical Mathods
Task-Specific Models (Supervised)
Pre-trained Models (Zoro-Shot)
Pre-trained Models (Other)
PatchTST
0,810
Chronos (Base)
0.662
N-HITS
0.830
N-HTS
0.672
Chronos (Base)
N-BEATS
Sundial (Besa)
0,834
PatchTST
0.884
N-BEATS
Sundial (Base)
0.694
DeepAR-
0.843
Moiral (Basc)
TFT
0.847
AutoTheta
0.875
DLinee
0.894
DLinear
0.757
GPT4TS-
0.895
AuloARIMA
0.761
Moiral (Basc)
0.907
AutoARIMA
0,804
WaveNet
0.951
AutoETS
0.838
AutoETS
WaveNet
LLMTTme-
0.962
1.000
Seasonal Naive
1.000
LagaL
1.097
1.108
Naive
1.162
Lag- Laaa
1.291
0.2
1.2
0.6
0.8
Agg. Relative WQL
Agg. Relative MASE
by Sundial. Baseline models be categorized into statistical methods fitting on each time series, task-specific deep models trained on each dataset, and pre-trained foundation models. Pre-trained Models that have several datasets during pre-training denoted Pre-trained Models (Other). A lower MASE/WQL indicates a better result. Sundial makes probabilistic predictions using 20 generated series, being consistent with Ansari et al. (2024).
Figure 4. Model evaluation the FEV leaderboard, which includes datasets
patch prediction, our inference speed is near to N-BEATS.
sizes. Compared to Sundial (Small), the large version leads to 15.38% reduction in the converged training loss, exhibit- ing promising model capacity generative forecasters.
Besides, we provide qualitative showcases in Appendix D. TimeFlow can generate highly eventful and coherent tempo- ral patterns with input series. Beyond the mean or quantiles, model enables the estimation of arbitrary statisties sampling directly from the predictive distribution.
2.0
Sundial (Small)
Sundial (Base)
Sundial (Large)
Training Loss
Model Type
Naive
Seasonal Naive
Statistical Methods
TimesFM
Task-Specific Models
DLinear
Pre-trained Models
PatchTST
Compute Type
Sundial (Base)
CPU
GPU (1 A10G)
DeepAR
GPU V100)
WaveNet-
V100)
0K
20K
40K
60K
80K
100K
N-HITS
Training Step
N-BEATS
Moiral (Basc]
Figure 6. Training curves on TimeBench of different model sizes.
Liama
ForecastPFN
Chronoe (Base)
5.3. TimeFlow Loss
AutoETS
AutoTheta
Based the flow-matching framework, TimeFlow Loss allows autoregressive models learn and generate flexible distributions while enhancing representation learning. To validate the effectiveness of this design, we implement two alternatives: (1) an MI.P network and MSE Loss and (2) a parameterized training objective based on the denoising di diffusion procedure (Li et al., 2024). We adopt the same parameterized network and Transformer backbone and pre- train them TimeBench. Since the converged training loss is not comparable across different objective functions, we compare zero-shot performance in Table 3. Despite allowing for sampling predictions, performance using diffusion- objective is notably inferior to TimeFlow Loss.
AutoARIMA-
LLMTime-
Avg. Inference Time (ms)
Figure 5. Inference time evaluation following Ansari et al. (2024), which is averaged from the FEV leaderboard. Computing resources of different models are marked. We plot the logarithmic x-axis.
5.2. Scalability
From Table 1, the larger Sundial model consistently achieves better performance with the scaling of parameters. Beyond downstream performance, we delve into the utilization of model capacity. Figure shows training curves of different
In addition zero-shot performance, we provide showcases

---

Sundial: A Family of Highly Capable Time Series Foundation Models
0.851
00.718
Table Zero-shot performance using different training objectives. We use the same model configuration and pre-training scale. Aver- aged MSE four prediction lengths are reported here.
0.715
0.845
0.710
MASE
WOL
0.705
Objective
ETTml
ETTm2
ETThh
ETTh2 ECL Weather
Avg.
0.700
TimeFlow
0.336
0.258
0.411
0.333
0.169
0.234
0.290
0.835
0.695
0.684
0.689
Diffusion
0.362
0.265
0.444
0.360
0.202
0.252
0.314
0.852
030
0.690
0.687
MSE
0.360
0.264
0.404
0.341
0.175
0.231
0.296
0.830
10 20
100
10 20
50
100
Sampled Predictions
Sampled Predictions
for quality evaluations in Appendix D.2. Pre-trained models optimized by the specific MSE Loss can only output a single prediction. And the prediction is sometimes over-smooth due to mode collapse (refer to Appendix C.I). Instead, gen- erative modeling can accommodate significantly different future variations even if their lookback series are similar. We also provide a probablistic metric CRPS compare different objectives in Table 7, which validate that the pre- dictive distribution modeled by TimeFlow is more coherent and diverse than counterpart training objectives. It benefits downstream tasks by generating multiple plausible predic- tions, conveys various future possibilities and enhances the reliability of decision-making.
0.842
0,942
0.12
0.710
0.840
0.80
MASE
WOL
0.705
0.838
0.702
0.700
0.836
0.896
0.834
0.695
0.64
0.834
10
20
50
10
20
50
Sampling Steps
Sampling Steps
Figure 7. We show the MASE (left) and WQL (right) on FEV w.r.t the number of generated predictions (top) and the steps sample prediction (down). More predictions or more sampling steps generally achieve better prohahilistic metries.
5.4. Dest-Time Calibration
Generative modeling facilitates the ficxibility to calibrate the final prediction during inference. Based on the median- based forecasting strategy, i.e., starting from multiple noise of standard Gaussian and calculating the median of raw predictions, there are two configurations to calibrate final predictions: (1) the number of samples to calculate statistics and (2) sampling steps used for flow-matching. Figure shows the results using different configurations.
We fine-tune pre-trained Sundial (Base) on the FEV leader- board, including short-term tasks with different prediction lengths. Our model is tuned once on all aggregated datasets. We evaluate the performance on unseen test splits (Figure 8). We observe that the performance can be further improved compared to zero-shot forecasting. Furthermore, training from scratch on aggregated datasets results in inferior perfor- mance, implying knowledge transfer in pre-trained models.
The top two figures conform to the central limit theorem. Generating more samples leads more calibrated estima- tion of prediction and confidence interval. The bottom two figures indicate that using fine-grai fine-grained steps during the push- forward process can leads to more precise predictions.
0.910
Train from Scratch
Zero-Shot
WQL
0.694
Fine- Fine-Tuning
0.675
1.091
The trade-off between inference time and performance re- veals the potential of test-time calibration, which does not require retraining models. The generative capability of Sun- dial provides flexibility for various use cases requiring dif- ferent levels of uncertainty. In experiments, sampling 20 predictions with each generated by 50 steps consumes nearly one second on a CPU, which is notably more efficient than tuning deep models or statistical methods. Advanced strategies of sampling and post-processing of raw prediction leave interesting directions for future exploration.
MASE
0.834
0.821
0.7
0.9
1.1
Agg. Relative Score
Figure 8. Performance the FEV leaderboard, including (1) train- ing Sundial from scratch on all datasets from the FEV leaderboard, zero-shot forecasting using pre-trained Sundial, and (3) fine- tuning once on all datasets from the FEV leaderboard.
5.6. Ablation Study
We conducted several ablation studies that provide insights into the enhancement made to Sundial's architecture. We evaluate the overall zero-shot performance on TSLib, which covers six different datasets and four prediction lengths.
5.5. Model Adaptation
Inspired by the prevalence of instruction tuning (Wei et al., 2021) that adapts foundation models on collection of tasks.
8

---

Sundial: A Family of Highly Capable Time Series Foundation Models
(a) RoPE
(b) LayerNorm
(c) FlashAttention
(d) Cache
0.00
1.30
w/o
Pre-NN (15k (or)
FlachAtti
w/la
RoPR
Pas tter)
FlashMtt
KV Cache
0.35
Fost-LX (Jia the?
1.-
14.00
Powt-LX tbee)
0.30
TSLi Zaro- Shol (Avg.)
TSSi Zere- Pare-Shot (Avg)
Training Speed is (s/feee)
Menorr Footpeiut (CB)
Inference Speed (s/iter)
0.30
0.30
1.30
0.30
0:32
0/7
0.38
66-
0..0
MSE
MAE
MSE
MAE
Time
Memory
Time
Figure 9. Ablation studies with respect to architectural enhancements. We report the averaged results of TSLib datasets (Wu et al., 2022) from four prediction lengths (96, 192, 336, 720} and all six datasets. The context length is set to 2880 and the patch length is 16.
RoPE Prior research (Liu et al., 2024a) observed that the introduction of RoPE (Su et al., 2024) yields better results in supervised forecasting tasks. As shown in Figure 9 (a), RoPE can also improve zero-shot forecasting, presenting general enhancement for time series foundation models.
from non-categorical distribution. Our pre-trained Sundial models make substantial advances on best-recognized fore- casting leaderboards. We hope this work can inspire future paradigms for pre-training time series foundation models and enhance their applicability real-world applications.
Layer Normalization Pre-LN (Baevski & Auli, 2018) is widely adopted in large language models (Touvron et al., 2023) due to the training stability. As depicted in Figure 9 (b), training with Pre-LN for more iterations yield better performance. contrast, training with Post-LN, which is the predominant choice supervised models, may adversely affect downstream results.
A Acknowledgements
This work was supported by the National Natural Science Foundation of China (U2342217 and 62021002), the BNRist Innovation Fund (BNR2024RC01010, and the National Engineering Research Center for Big Data Software.
We extend our gratitude Xingzhuo Guo for his expertise in flow-matching and meticulous proofreading of the method section. We further thank Jialong Wu, Yuezhou Ma and Yu Zhang for insightful discussions about generative models. Their collective support significantly enhanced this work.
TlshAttention and KV Cache We make it to leverage FlashAttention (Dao et al., 2022) and KV Cache to reduce the computational costs. As shown in Figure 9 (c) and (d), they notably reduce 14.8% memory footprint and 43.6% inference time without affecting performance.
Impact Statement
6. Conclusion
This paper aims advance the development of time series foundation models. We curated pre-training dataset from publicly available resources. Our models employ an efficient tokenization and incorporate generative training objective. The proposed TimeFlow Loss provides insights for train- ing generative foundation models time series forecasting. We released our pre-trained models that demonstrate notable zero-shot forecasting performance and versatility across dif- ferent benchmarks. The generative forecasting paradigm enhances the model reliability for decision-makin madiiiionmmking... The mil- lisecond inference speed of our model can be beneficial to many practitioners. Our paper mainly focuses on scientific research and has no obvious negative social impact.
In this work, we collect and curate TimeBench, a trillion- scale time series dataset for building time series foundation models, which can benefit the research community. Towards time series foundation models, we delve into tokenization and optimization, presenting contributions in two aspects. First, we demonstrate that continuous tokenization, such patch tokens, be more effective and efficient for the time series modality, and generative modeling presents a na- tive approach for learning on amtinuous-valuse time series. Second, we propose a novel training objective to accom- modate heterogeneous time series distribution. It endows autoregressive models with an inherent capability to sample

---

Sundial: A Family of Highly Capable Time Series Foundation Models
References
Goldberger, A. L.., Amaral, L. A., Glass, L.,, Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E.,, Moody, G. B., Peng, C.-K., and Stanley, H. E. Physiobank, physiotoolkit, and physionet: components of new research resource for complex physiologic signals. circulation, (23); e215-e220, 2000.
Aksu, T., Woo, G., Liu, J., Liu, X., Liu, C., Savarese, S., Xiong, C., and Sahoo, D. Gift-eval: A benchmark for gen- eral time series forecasting model evaluation. In NeurlPS Workshop on Time Series in the Age of Large Models, 2024.
Ansari, A. F., Stella, L., Turkmen, C., Zhang, X., Mercado, P., Shen, H., Shchur, O., Rangapuram, S. S., Arango, S. P., Kapoor, S., et al. Chronos: Learning the language time series. arXiv preprint arXiv:2403.07815 2024.
Goswami, M., Szafer, K., Choudhry, A., Cai, Y., Li, S., and Dubrawski, A. Moment: family of open time-series foundation models. arXiv preprint arXiv:2402. 03885, 2024.
Baevski, A. and Auli, M. Adaptive input representa- tions for neural language modeling. arXiv preprint ar0.v:1809. 10853, 2018.
Gruver, N., Finzi, M., Qiu, S., and Wilson, A. G. Large language models are zero-shot time series forecasters. arXiv preprint arXiv:2310.07820, 2023.
Bai, S., Kolter, J. Z., and Koltun, V. An empirical evalua- tion of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01277 2018.
Gruver, N., Finzi, M., Qiu, S., and Wilson, A. G. Large language models are zero-shot time series forecasters. Advances in Neural Information Processing Systems, 36, 2024.
Bengio, Y., Ducharme, R., and Vincent, P. A neural proba- bilistic language model. Advances in neural information processing systems, 13, 2000.
Hersbach, H., Bell, B., Berrisford, P., Hirahara. S., Horányi, A., Muñoz-Sabater, J., Nicolas, J., Peubey, C., Radu, R., Schepers, D., et al. The era5 global reanalysis. Quarterly Journal of the Royal Meteorological Society, 146(730): 1999-2049, 2020.
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosse- lut, A., Brunskill, E., et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.0725. 2021.
Hoo, S. B., Müller, S., Salinas, D., and Hutter, F. The tabular foundation model tabpfn outperforms specialized time series forecasting models based on simple features. arXiv preprint arXiv:2501.02945. 2025.
Box, G. Box and jenkins: time series analysis, forecasting and control. In A Very British Affair: Six Britons and the Development of Time Series Analysis During the 20th Century, pp. 161-215. Springer, 2013.
Hyndman, R. Forecusting; principles and practice. OTexts, 2018.
Box. G. E., Jenkins, G. M., Reinsel, G. C., and Ljung, G. M. Time series analysis: forecasting and control. John Wiley & Sons, 2015.
Ke, G., Meng, Q., Finley, T., Wang. T., Chen, W., Ma, W., Ye, Q., and Liu, T.-Y. Lightghm: highly efficient gradient boosting decision tree. Advances in neural infor- mation processing systems, 30, 2017.
Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C. Flashat- tention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Process- ing Systems, 15:16344-16355 2022.
Kim, T., Kim, J., Tae, Y., Park, C., Choi, J.-H., and Choo, J. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations, 2021.
Das, A., Kong, W., Leach, A., Sen, R., and Yu, R. Long- term forecasting with tide: Time-series dense encoder. arXiv preprini arXiv:2304.08424 2023a.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Das, A., Kong, W., Sen, R., and Zhou, Y. A decoder- only foundation model for time- forecasting, arXiv preprint ar 100 10688, 2023b.
Kollovieh, M., Lienen, M., Lüdke, D., Schwinn, L., and Günnemann, S. Flow matching with gaussian process priors for probabilistic time series forecasting. arXiv preprint 44X1v:2410.03024, 2024.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F,, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-firs International Conference on Machine Learning, 2024.
Li, T., Tian, Y., Li, H., Deng, M., and He, K. Autoregres- sive image generation without vector quantization. arXiv preprint arXiv:2406. 1838, 2024.
10

---

Sundial: A Family of Highly Capable Time Series Foundation Models
Liang, Y., Wen, H., Nie, Y., Jiang, Y., Jin, M., Song, D., Pan, S., and Wen, Q. Foundation models for time series analysis: A tutorial and survey. In Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining, pp. 6555-6565, 2024.
OpenAI, R. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2:13, 2023.
Oreshkin, B. N., Carpov, D., Chapados, N., and Bengio, Y. N-beats: Neural basis expansion analysis for interpretable time series forecasting, arXiv preprint arXiv:1905. 10437, 2019.
Lim, B., Arik, S. Ô., Loeff, N., and Pfister, T. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting, 37(4):1748-1764 2021.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, I.,, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Lipman, Y., Chen, R. T., Ben- Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. arXiv preprint wrXiv:22/0.02744 2022.
Peebles, W. and Xie, Scalable diffusion models with transformers. In Proceedings the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 4195-4205, 2023.
Lipman, Y., Havasi, M., Holderrieth, P., Shaul, N., Le, M., Karrer, B., Chen, R. T., Lopez-Paz, D., Ben-Hamu, H., and Gat, I. Flow matching guide and code. arXiv preprint orXiv:2412.06264 2024.
PEMS. Traffic Dataset. http: /pems/eemsss./ dot ca. gov/
Liu, Y., Wu, H., Wang, J., and Long, M. Non-stationary transformers: Exploring the stationarity in time series forecasting. Advances in Neural Information Processing Systems, 35:9881-9899 2022.
Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury,
J., Heek, J., Xiao, K., Agrawal, S., and Dean, J. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5:606-624 624, 2023.
Liu, Y. Hu, T., Zhang, H., Wu, H., Wang, S., Ma, L.., and Long, M. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.0662. 2023a.
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al. Improving language understanding by generative pre-training. OpenAI, 2018
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Rad- ford, A., Chen, M., and Sutskever, I. Zero-shot text-to- image generation. In International conference on ma- chine learning, pp. 8821-8831. Pmlr, 2021.
Liu, Y., Li, C., Wang, J., and Long, M. Koopa: Learning non- stationary time series dynamics with koopman predictors. arXiv preprint 2305::2.00./800 2023b.
Liu, Y., Qin, G., Huang, X., Wang, J., and Long, M. Timer- xl: Long-context transformers for unified time series fore- casting. arXiv preprint arXiv:2410.0444 1..00440,,..... 2024a.
Rasley, J., Rajbhandari, S., Ruwase O., and He, Deep- speed: System optimizations enable training deep learn- ing models with over 100 billion parameters. In Proceed- ings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3505-3506, 2020.
Liu, Y., Zhang, H., Li, C., Huang, X., Wang, J., and Long, M. Timer: Generative pre-trained transformers are large time scries models. In Forty-first International Conference on Machine Learning, 2024b.
Rasul, K., Seward, C.. Schuster, I., and Vollgraf, R. Au- toregressive denoising diffusion models for multivariate probabilistic time series forecasting. In International Con- ference on Machine Learning, pp. 8857-8868. PMI.R, 2021.
Liu, Y., Zhang, K., Li, Y., Yan, Z., Gao, C., Chen, R., Yuan, Z., Huang, Y., Sun, H., Gao, J., et al. Sora: A review on background, technology, limitations, and opportumities of large vision models. arXiv preprint arXiv:2402.17177 2024c.
Rasul, K., Ashok, A., Williams, A. R., Khorasani, A., Adamopoulos, G., Bhagwatkar, R., Biloš, M., Ghonia, H., Hassen, N. V., Schneider, A., et al. Lag-llama: Towards foundation models for time series forecasting. arXiv preprint arXiv:2310.08278, 2023.
Muñoz-Sabater, J., Dutra, E., Agusti-Panareda A., Albergel, C., Arduini, G., Balsamo, G., Boussetta, S., Choulga, M., Harrigan, S., Hersbach, H., et al. Era5-land: A state- of-the-art global reanalysis dataset for land applications. Earth system science data, 43499433444 2021.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Nie, Y., Nguyen, N. H., Sinthong, P., and Kalagnanam, J. A time series is worth 64 words: Long-tern forecasting with transformers. arXiv preprint 2r.iv:2211..44 14730, 2022.

---

Sundial: A Family of Highly Capable Time Series Foundation Models
Salinas, D., Flunkert, V., Gasthaus, J., and Januschowski, T. Deepar: Probabilisticc forecasting with autoregressive recurrent networks. International journal of forecasting, 2020.
Woo, G., Liu, C., Kumar, A., Xiong, C., Savarese, S., and Sahoo, D. Unified training of universal time series fore- casting transformers, arXiv preprint arXiv:2402.02592. 2024.
Shen, L. and Kwok, J. Non-autoregrresive conditional diffusion models for time series prediction. In Inter- national Conference on Machine Learning, pp. 31016- 31029. PMLR, 2023.
Wu, H., Xu, J., Wang. J., and Long, M. Autoformer: Decom- position transformers with auto-correlation for long-term series forecasting, Advances in Neural Information Pro- cessing Systems, 34:22419 222400,, 2021.
Shi, J., Ma, Q. Ma, H., and Li, I.. Scaling law for time series forecasting, arXiv preprint 7xXi::2405.51144 2024a.
Wu, H., Hu, T., Liu, Y., Zhou, H., Wang, J., and Long, M. Timesnet: Temporal 2d-variation modeling for general time series analysis. arXiv preprint arXiv:2210.02.16, 2022.
Shi, X., Wang, S., Nie, Y., Li, D., Yc. Z., Wen, Q. and Jin, M. Tome-moe: Billion-scale time series founda- tion models with mixture of experts. arXiv preprint arXiv:2409.16040 2024b.
Xiong, R., Yang, Y., He, D., Zheng, K.. Zheng, S., Xing, C., Zhang, H., Lan, Y., Wang, L., and Liu, T. On layer normalization in the transformer architecture. In Inter- national Conference Machine Learning, pp. 10524- 10533. PMLR, 2020.
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-Im: Training multi- billion parameter language models using model paral- lelism. arXiv preprint arXiv:1909.0800 2019.
Zhang, R., Isola, P., Efros, A., Shechtman, E., and Wang. O. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. In International conference on machine learning, pp. 2256-2265, PMLR, 2015.
Zhang, Y. and Yan, J. Crossformer: cross-dimension dependency for multivariate time series forecasting. In The eleventh international conference on learning representations, 2023.
Transformer utilizing
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocompuling, 568:127063. 2024.
Tashiro, Y. Song. J., Song. Y., and Ermon, S. Csdi: Con- ditional score- cosre-based diffusion models for probabilistic time series imputation. Advances in Neural Information Processing Systems, 34:24804-24816, 2021.
Zhao, W. X., Zhou, K., Li, Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. A survey of large language models. arXiv preprint 20X.v:2303.18222 2023.
Tong, A., Fatras, K., Malkin, N., Huguet, G., Zhang, Y., Brotor, J., Wolf, G., and Bengio, Y. Improving and generalizing flow-based generative models with mini- batch optimal transport. arXiv preprint arXiv:2302.00482, 2023.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-.,, Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.13377, 2023.
Vovk, V., Gammerman, A., and Shafer, G. Algorithmic learning in random world, volume 29. Springer, 2005.
Wei. J., Bosma, M., Zhao, V. Y.. Guu, K., Yu, A. W., Lester, B., Du N., Dai, A. M., and Le, Q. V. Finetuned lan- guage models are zero-shot learners. arXiv preprint 109.02009 2021.
Wen, R., Torkkola, K., Narayanaswamy, B., and Madeka, D. A horlti- quantile recurrent forecaster. arXiv preprint arxiv: 1053, 2017.
12

---

Sundial: A Family of Highly Capable Time Series Foundation Models
A. Dataset Statistics
Large-scale datasets are of paramount importance for pre-training foundation models. Recent research has contributed significant time series datasets (Das et al., 2023b; Liu et al., 2024b: Shi et al., 2024b). While the scaling law of time series foundation models has been explored in the recent work (Shi et al., 2024a), the pre-training scale remains relatively limited, Given the heterogeneity of time series compared to other modalities, it raises the question of whether it is feasible to learn from enormous series. To address the question, we curated TimeBench with a trillion time points from various domains.
Unlike other modalities, most time series are unavailable on open websites or repositories. There are also limited domains that encompass typical and predictable time series, leading to slow progress on dataset construction. Therefore, we conducted tedious preprocessing, including missing values imputation, abnormalities exclusion, and normalization techniques. We conducted statistical analysis, examining time series through the lenses of intrinsic properties, e.g., non-stationarity. forecastability, and seasonality. This approach allows us to characterize the data quality inherent time series, which affects the training stability of next-token prediction. We also adopt synthetic techniques to improve pattern diversity. Further, we adopt ERA5 (Muñoz-Sabater et al., 2021), including systematic real-world temporal observations.
The statistical details of TimeBench are summarized in Table 4. In addition to open-source datasets from research teams on time series foundation models (Woo et al., 2024; Ansari et al., 2024; Liu et al., 2024b;a), we collected substantial real-world time series from various domains such as finance, loTT, meteorology, and healthcare (Goldberger et al., 2000). These resources enable us to construct large-scale time- merieeris corpora exceeding a trillion time points. The corpora include highly credible and predictable data with wide range of frequencies, lengths, and numbers of variates, providing comprehensive temporal dynamies and variation patterns to facilitate downstream applications. To prevent data leakage, exclude all datasets evaluated in Section 5.1 to make sure that Sundial conducts zero-shot forecasting.
Table 4. Key statistics of TimeBench, the pre-training dataset of Sundial,
Chronos (2024)
ECG
Finance
loT
LOSTA Synthetic ERA5 3h ERA 12h
ERA5 Daily ERAS Weekly ERAS Monthly ERAS Quarterly
Source
Total
(2000) (Ours)
(Ours)
(2024)
(2024)
(2021)
(2021)
(2021)
(2021)
(2021)
(2021)
# Pts.
94B
48B
10.5B
5.8B
230B
0,5B
129B
32B
406B
58B 5.62 %
13.5B
4.5B
1032B
%
9.11 %
4.65 %
1.02 % 0.56 %
22.29 %
0.05 %
12.50 %
3.10 %
39.35 %
1.31 %
0.44 %
100%
B. Implementation Details
All experiments are implemented using PyTorch (Paszke et al., 2019) and executed with 32 NVIDIA A100 GPUs We employ the Adam optimizer (Kingma & Ba, 2014) for model optimization. We adopt S3 format (Liu et al., 2024b) for univariate pre-training. During training, data from different domains is sampled according to a predefined ratio to balance the domain weightings and ensure diversity in the training data. We implement global shuffle strategy by loading time series into a standard parquet format. We use variable-wise normalization to unify the scope of values.
On the FEV leaderboard (Ansari et al., 2024), which consists of short-term forecasting datasets, we train Sundial models by TimeFlow Loss with the prediction length of F = 16. For the point forecasting (Wu al., 2022) and GIFT-Eval (Aksu et al., 2024), which consist of forecasting datasets with prediction length ranging from 6 to 900, we train Sundial models by TimeFlow Loss with the prediction length of F = 720. For the required prediction length less than the model prediction length, we truncate the output generated by Sundial. For the required length more than the prediction horizon, we conduct rolling forecasting. Following Chronos (Ansari et al., 2024), we sample 20 raw predictions to calculate MASE and WQL on FEV. Being consistent to Moirai (Woo et al., 2024), we sample 100 raw predictions to calculate MASE and CRPS for GIFT-Eval. The sampling step is fixed as K 50. Configurations of Sundial in different sizes are provided in Table 5. We provide model summary in Table 6, which summarizes several aspects of current time series foundation models.
C. Supplementary Results
C.1. Discussion of Mode Collapse
Mode collapse is a failure of representation learning, where a model generates a limited variety of outputs, ignoring the diversity in the training data. For time series foundation models, mode collapse stems from the heterogeneity of the time series distribution, e.g., a similar lookback time series goes into divergent trending. In other words, the semantics of time
13

---

Sundial: A Family of Highly Capable Time Series Foundation Models
Table 5. Model configurations of the Sundial family.
Patch Size
Context Length
Prediction Length Layers
Dimension
MHA Heads
TimeFlow
Total Parameters
Model
(P)
(T)
(F)
(L)
(D, Dn)
H
(Dus Lu)
#Count
Sundialsmelt
16
2880
720)
6
(512, 2048)
8
(512,3)
32M
Sundial Base
16
2880
{16, 720}
12
(768, 3072)
12
(768.3)
128M
Sundial
16
2880
{16, 720)
24
(1024, 4096)
16
(1024, 6)
444M
D is the embedding dimension of Transformer. Dr is the hidden dimension FFN, Da is the hidden dimension the flow- matching network. L is the layer number of Transformer. Le the layer number of the flow-matching network.
Table Comparison of time series foundation models. Architccuree denotes the Transformer category. Model Size presents parameter counts of different model sizes. Pre-training Scale measures pre-training datasets in time points. Token Level presents the graininess of time series tokens. Tokenization denotes what kind of values are embedded from time series. Context Length means the input length supported by the model. Probabilistic means generating multiple probable predictions, which the opposite of deterministic forecasters.
Sundial
Time-MoB Timer-XI
Moirai (2024)
MOMENT LLMTime Chronos Lag-Llama
TimesFM
Method
(Ours)
(2024b)
(2024a)
(2024)
(2024)
(2024)
(2023)
(2023b)
Architecture
Decoder
Decoder
Decoder
Encoder
Encoder
Decoder
EncDec
Decoder
Decoder
32M
113M
84M
14M
40M
46M
200M
17M
Model Size
128M
453M
91M
125M
200M
70M
444M
2.4B
311M
385M
710M
200M
Pre-training Scale
1032B
300B
260B
231B
1.13B
84B
0.36B
100B
Token Level
Patch
Point
Patch
Patch
Patch
Point
Point
Point
Patch
Tokenization
Continuous
Continuous
Continuous
Continuous
Continuous
Discrete
Discrete
Continuous
Continuous
Context Length
<8880
<4096
<8880
<5000
=122
512
<1024
<512
Probabilistic
True
False
False
True
False
True
True
True
False
series patterns highly unstable. It sometimes leads over-smooth predictions from models optimized by MSE becuase the results are global-optimal for this loss (See showcases the right of Figure 14-15). Such training objective pre-defines unimodal predictive distribution of data, which struggles to accommodate large-scale datasets like TimeBench.
Our work addresses this phenomenon through generative modeling. Generative forecasters learn flexible distributions without relying on probabilistic priors. We evaluate the distributional metric Continuous Ranked Probability Score (CRPS) to assess the quality of generated predictions across different training objectives. The results indicate that the predictive distribution modeled by TimeFlow more coherent and diverse compared alternative training objectives, particularly on the highly diverse GIFT-Eval (Aksu al., 2024). It validates the effectiveness of TimeFlow in mitigating mode collapse.
Table Zero-shot probabilistic forecasting performance using different training objectives. Averaged CRPS reported here.
Objective
ETTh1
ETTh2
ETTml
ETTm2
ECL
Weather
GIFT
TimeFlow Loss
0.0059
0.0037
0.0057
0.0029
0.0082
0.0021
0.5050
Diffusion Loss
0.0082
0.0053
0.0070
0.0039
0.0095
0.0032
0.5340
MSE Loss
0.0063
0.0040
0.0058
0.0032
0.0080
0.0023
0.6420
14

---

Sundial: A Family of Highly Capable Time Series Foundation Models
C.2. Scaling Behavior Using More Data
We compare Sundial with other time series foundation models that are pre-trained with smaller datasets: Chronos (Ansari al., 2024) is pre-trate pre-trained on 94 billion time points, and Moirai is pre-trained 230 billion time points. As their pre-training datasets part of the subset of TimeBench,
also conduct pre- training Sundial using these subsets. As shown in Table 8, these results highlight the scaling behavior of Sundial using larger datasets. Additionally, Sundial still achieves better zero-shot forecasting than its counterpart models with the same pre-training dataset.
Table 8. Zero-shot forecasting performance of models trained on different scales of datasets (measured in time points, pts, and IB means a billion). We report the averaged results from four prediction lengths {96, 192, 336, 720) Time-Series-Librre (Wu al., 2022).
Model (pts.)
Chronos (94B)
Moirai (230B)
Sundial (94B)
Sundial (230B)
Sundial (1032B)
Dataset
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
ETThl
0.591
0.468
0.417
0.419
0.402
0.429
0.403
0.419
0.411
0.434
ETTh2
0.405
0.410
0.362
0.382
0.377
0.414
0.364
0.398
0.333
0.387
ETTml
0.645
0.500
0.406
0.385
0.367
0.402
0.352
0.385
0.336
0.377
ETTm2
0.310
0.350
0.311
0.337
0.280
0,341
0.273
0.334
0.258
0.320
ECL
0.214
0.278
0.187
0.274
0.172
0.269
0.171
0.267
0.169
0.265
Weather
0.292
0.315
0.287
0.281
0.254
0.301
0.252
0.297
0.234
0.270
C.3. Performance with Varying Lookback Lengths
Time series foundation models operate independently of training, functioning similarly to statistical methods. Given specific forecasting tasks, one of the most important hyperparameters is the lookback length. Unlike fixed-context models, Sundial offers fiexibility for practitioners, allowing the context length to be dynamically adjusted during inference. In Figure 10, we present the performance of Sundial utilizing various lookback lengths. Based on our observations, we contend that performance is largely dependent on the forecasting task itself. Specifically, the size of the lookback window can be tuned to meet the forecasting horizon and data periodicity. Time series foundation models provide a training-fres approach for rapid adjustments; still, they should enhance their fundamental long-context capabilities to handle high-frequency data.
0.45
0.40
TST.h Zero-Shi Shot (MSE)
ETThl
0.35
ETT12
ETTml
ETTm2
ECL
0.25
Weather
0.20
0.35
1440
2400
Figure 10. Zero-shot forecasting performance using different lookback lengths in {480, 960, 1440, 1920, 2400, 2880). We report the averaged results from four prediction lengths (96, 192, 720) on Limee (Wu al., 2022).
15

---

Sundial: A Family of Highly Capable Time Series Foundation Models
C.4. Zero-Shot Results of Point Forecasting
Table 9 provides full zero-shot results on Lime-Series-Lbbrary forecasting benchmark (Wu al., 2022), including prediction horizons in (96, 192, 336, 720). We build Sundial with different model sizes with configurations in Table 5. The context length is fixed as 2880. We truncate the model's predictions for tasks requiring prediction length less than F = 720.
We compare the most advanced time series foundation models based on their official checkpoints, including Time-MoE (Shi et al., 2024b), Timer (Liu et al., 2024a;b), Moirai (Woo et al., 2024), TimesFM (Das et al., 2023b), and Chronos (Ansari et al., 2024). We conduct zero-shot evaluations on datasets that are not included during the pre-training of the corresponding models. For each of the evaluated model, we use their maximum input length during inference. Metries (MSE/MAB) are calculated from all predicted windows in the test split of each dataset following Liu et al. (2024a).
C.5. Zero-Shot Results on GIFT-Eval and FEV Leaderboard
We evaluate our models on GIFT-Eval, benchmark designed to comprehensively assess forecasting performance across diverse time series. GIFT-Eval includes 23 datasets covering 144, 000 time series and 177 million data points, which constitute total of 97 forecasting configurations. We use the official evaluation suite established by the research team of Salesforce and report aggregated results in Table 2. We evaluate the performance and inference time the FEV leaderboard, which was originally proposed by Ansari et al. (2024) and established by AutoGluon, which comprises 27 datasets for zero-shot evaluation. We report aggregated metrics in Figure 4 and assess the inference time in Figure 5, We released the detailed results by submitting Sundial to their open benchmark?.
D. Showcases
D.I. Showcases of Sundial
Figure 11-13 present zero-shot forecasting showcases on all the datasets from FEV (Ansari et al., 2024) and TSLib (Wu et al., 2022). By generating 20 predictions with different initial noise, we estimate the median and 80% prediction interval.
D.2. Showcases of Generative Forecasters and Deterministic Forecasters
As introduce generative modeling in time series foundation models, compare zero-shot forecasting showcases from two types of models, including (1) Sundial, a generative forecaster pre-trained by TimeFlow, which can predict multiple future possibilities based on a lookback series. (2) Using the same backbone and TimeBench, Transformer pre-trained by MSE Loss. As a deterministic forecaster, the model can only output the mean prediction. As depicted in Figure 14-15, the unimodal Gaussian prior specified by MSE can be infeasible to handle large-scale pre-training, manifested as sometimes over-smooth predictions in downstream forecasting tasks. Therefore, we hope this work can inspire future paradigms for pre-training time series foundation models and enhance their applicability real-world scenarios.
E. Limitations
Our models represent an initial effort to incorporate generative modeling into time series foundation models, which enables pre-training on heterogeneous time series without specifying any prior distribution. This approach mitigates mode collapse in representation learning and generates a diverse range of probable predictions compared to previous deterministic forecasters. Despite significant progress in enlarging model capacity, the Sundial family may still face hallucinations. The performance on very fighffrequency data is not guaranteed, since Time contains many middle- and wwwffrquency time series. Therefore, an important future direction is to generalize Sundial to multi-scale time series. This situation may also indicate new opportunities during inference. As we only adopt naïve sampling strategy that begins with random Gaussian noise, it leaves much room for future improvement in sampling strategy and post post-processing, such as frequency normalization.
Another aspect of future development lies in model adaptation. Sundial is pre- pre-trained in a univariate approach to address the discrepancy variate numbers, which prevents it from explicitly utilizing variate correlations or covariate information. As an inereasing number of studies address 2D dimensionality, multivariate pre-training likely be conducted for domain-specific time series foundation models. Lastly, while autoregressive models provide flexibility in the input context length, multiple steps of autoregression may still lead to over-smooth predictions and unreliable results.
16

---

Sundial: A Family of Highly Capable Time Series Foundation Models
Table Zero-shot forecasting results of time series foundation models on long-term forecasting datasets (Wu et al., 2022). A lower MSE or MAE indicates better prediction. Averaged results of four prediction lengths reported here. 1st Count represents the number of wins achieved by a model under all prediction lengths and datasets. Results of baseline models are ofticially reported by Shi et al. (2024b).
Datasets for pre-training not evaluated on corresponding models, which are denoted by the dash (-).
Models
Sundialssan Sundialaw Sundiakaayy Time-MoF....
Time-MoK...>
Time-MoEas.. Timer-XI
Moiraiaw
Moirau..y Chronosace Chronos.any (2024)
TimesFM (2023b)
(Ours)
(Ours)
(Ours)
(2024h)
(2024h)
(2024h)
(2024a)
(2024)
(2024)
(2024)
Metric
MSE MAE
MSE MAE MSE MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSB
MAE
MSE MAE
MSE MAE
MSE MAB
MSE
MAE
MSE MAB
96
0.292
0.342
0,280
0.334
0.273 0.329
0.338
0.368
0.309
0.357
10.281
0.341
0.317 0.356 0.363 0.356 0.361 [0.454 0.408
0.445
0.403
(0.36 0.370
192
0.337
0.376
0.321
0.366
0.312 0.357
0.353
0.388
0.346
0.381
0.305
0.358
0.358
0.381 0.388
0.375
0.412 0.383 0.566
0.477
0.530
0.450
0.414 0.405
336
0.401
0.389
0.343
0.378
01381
0.373
0.408
0.369
0.395
0.386
0.392
01.436
0.40000662.
0.525
00.577
0.481
00,445
0.429
720
0.418
0.394
0.418
0.397
0.413
0.504
0:445
0.477
0.446
0.430
0.431
0.460
0:418
0.446
0.420
0.900
0.591
0.666
0.526
00.512 0.471
Avg
0.354
0.388
0.336
0.377
0.331
0.369
00.394
0.415
(0.376
0,405
|0.356
0.391
0..373
0.392
0.406
(..85
0.391
[0.645
0.500
00.555
0.465
[0.433 0.418
96
0.178
0.260
0.170
0.256
0.172
0.255
(0.201
0.291
10.197
0.286
00.198
0.288
00.189
0,277 0.205
(0.21
0.274
0,274
(0.197
0.271
[0.202 0,270
192
0.235
0.304
0.229
0.300
0.227
0.296
0.258
0.334
0.250
0.322
0.235
0.312
0.241
0,315 0.275
0.316
0.288
0.318
0.261
0.322
0.254
0.314
0.289 0.321
336
0.287
0.342
0,281
0.337
0.275
0.331
0.324
0.373
0.337
0.375
0.293
0.348
0.286
0.348 0.329
0,344
0.355
00.326
0.366
0.313
0.353
0.366 0,366
720
0.360
0.390
0.351
0.387
0.343
0.378
0.488
0.464
(0:480
0.461
00.427
0.428
0.375
0,402 0.437
0,411 0.411
0.448
0.428
0.455
0.439
00.416
0.415
(0.462 0.430
Avg
00.265
0.324
0.255
0.520
0.254
0.315
(0.317
0.365
[0.316
0.361
|0.288
0.344
(0.273
0.336 0.311 0.337|0.329
0.343
0.350
(0:295
0.338
|0.328 0.346
96
0.341
0.381
10.348
0.385
0.346
0.383
30.357
0.381
(0.350
0.382
(0.349
0.379
0.369
0.391 0.376
0.392/0.381 0.388
00.440
0.393
[0.44]
0.390
(0.414 0.404
192
0.381
0.408
0.393
0.418
0.386
0.410
0.
0.404
0.388
0.412
0,395
0.413
0.405
0.413
0.412
0.413 0.434
0.415
0.492
0.499
0.426
0.502
0.524
0.466 0,434
336
0.440
0.410
0.426
0.411
0.434
00.411
0.447
0.453
0.418
0.423
0.433
0.428
01.485 D.445
0.550
0.462
00.576
0.467
00.503 0.456
720
0.481
0.493
0.438
0.459
0.444
0.477
0.427
0.:457
0.462
0.423
0.441
0.447
0.46400.611
0.464
0.611
0.510
0.591
0.835
0.583
(0.511 0.481
Avg
0.390
0.418
0.411
0.434
0.395
0.420
00000
0.424
[o.
[0.394
0.419
J0.412
0.426
00.404
0.417
0.417
0,419/0.480
0 0.439 (0.591
0.468
(0.588
0.466
|0.473 0.443
96
0.272
0.332
0.271
0.333
0.269
0.330
0.300
0.359
10.302
0.354
[0,292
0.352
002283
0.342 0.294
0.330
0.330
0.330/0.300
00.308
0.343
|0.320
0.345
|0.31.5 0,349
192
0.329
0.374
0.327
0.376
0.325 0.373
0.351
0.386
0.364
0,385
0.347
0.379
0.340
0,379 0.365 0.375 0.361
0.371 0.384
0.392
0.406
0.399
0.388
0.395
336
0.357
0.399
0.354
0.402
0.354
0.400
0.391
0.418
0..17
0.425
0.400
0.419
0.366
0.400 0.376
0.390...390
01399
0.390
0.429
0.430
0.492
0.453
(0,422 0.427
720
0.401
0.442
0.381
0.435
0.389
0.443
0.419
0.454
0.537
0.496
00.439
0.447
0.397
0.431 0.416
0,433
0,433
0.442
0.4180..501
0.418
0.501
0.477
0.603
0.511
0.443
0.454
Avg
00.340
0.387
0.333
0.387
0.334
0.387
00.66
0.404
0.445
0.415
|0.371
0.399
00.347
0,388 0.362 0.382 (03367 0.377
00.405
0.410
[0.455
0.427
|0.392 0.406
96
0.144
0.231
10.122
0.229
0.130
0.227
0.141 0.237 0. 160 0.250
153
0.241
0.231
00.152
0.229
192
0.154
0.251
0.152
0.250
0.150
0.247
0.159 0.254
0.5 0.263 069
0.255
0.179
0.254
0.172
0.250
ECL
336
0.174
0.173
0.271
0.170 0.268
0.177 0.272 087 0.277 187 0.273
0.214
0.284
00.203
0.276
720
0.215
0.307
0.218
0.311
0.214
0.307
0.219 0.308 0.228
0.309
0.309
0.237 0.313
0.311
0.346
0.228
0.337
Avg
0.169
0.265
0.169
0.265
0.166
0.262]
00.174
0.278 0.187
0.274/0.118
0.270/0.214
0.278
|0.204
0.273
96
0.1.58
0.206
0..15
0.205
0.157
0.208
060
0.214
0.159
0.213
0.157
0.211
0..77
0.225
0.220
0.217
0.211
|0.203
0.238
|0.194
0.235
Weather
192
0.205
0.253
0.205
0.251
0.207
0.256
0.210
0.260
0.215
0,266
0.208
0.256
0.221
0.271
0.271
0.259
0.246
0.251
0.256
0.290
0.249
0.285
336
0.254
0.290
0.253
0.289
0.259
0.295
0.274
0.309
0.291
0.322
0.255
0.290
0.274
0.311
0.286
0.297
0.274
0.291
0.314
0.336
0..002
0.327
720
0.315
0.336
0.320
0.333
0.327
0.342
0.418
0.405
0.415
0.400
0.405
0.397
0.356
0,370
0.373
0.354
0.337
0.340
0.340
0.397
0.396
0.372
0.378
Avg
0.233
0.271
|0.234
0.270
0.238
0.275
(0. (0.265
0.297
|0.270
0.300
00.256
0.288
00.256
0.294
0.287 0.281 |0. |0.264 0.273
|0.292
0.315
(0.279
0.306
1d Count
16
16
0
2
0
0
0
Traffic (PEMS) is not evaluated because it included in the pre- pre-training datasets of these time series foundation models.
17

---

Sundial: A Family of Highly Capable Time Series Foundation Models
F. Societal Impacts
F.I Real-World Applications
In this work, we present Sundial, a family of time series foundation models to facilitate out-of-the-box forecasting. Our models employ native tokenization for continuous-valuee time series and incorporate flexible training objective, proposed TimeFlow Loss, enable probabilistic forecasting. With unprecedented model capacity and trillion-scale dataset, models can be used directly adapted for various forecasting scenarios, such energy planning, weather forecasting, and financial risk prevention. With multiple predictions generation and just-in-time inference speed, our model enhances the reliability of decision-Di maiiing and streamlines the forecasting pipeline for practitioners. This paper primarily focuses on scientific research and does not present any evident negative social impact.
F.2 Academic Research
We curate TimeBench, a trillion-leve time series dataset for pre-training foundation models for time series analysis, which we believe will be beneficial to the research community. Technically, we propose a TimeFlow Loss to facilitate the learning of flexible next-patch distributions. Conditioned on the lookback representations acquired by autoregressive Transformers, our model is endowed with novel generative capability for probabilistic forecasting, enhancing representation learning of Transformers without the need for discrete tokenization. Through pre-training on an unprecedented scale, we identify subtle scalability bottlenecks that not solely attributable architectural design but predominantly influenced by the training objectives of foundation models. The proposed TimeFlow Loss applied to autoregressive and generative models may provide insights for the future development of time series foundation models.
18

---

Sundial: A Family of Highly Capable Time Series Foundation Models
Mosthly
Quartery
Corfiarc
4100
11.00
Brh
1873
Loctkok
2000
Tourien Quartarly
Touriam Yearly
Trefffic
Weather
2HD4P113
Figure 11. Showcases of zero-shot predictions from Sundial (Basc) on the FEV leaderboard (Ansari et al., 2024).
19

---

Sundial: A Family of Highly Capable Time Series Foundation Models
Domisick
2411-8427
2014
201463
Sachange Rate
Md Quarterly
0.13
Lobkbcck
0.7
5000
Bean
1.31
0.1
030
241444
2014
Yeady
MS
3.0
1008
my M
350
300m
400
Tirestary
1400
1800
3072
1473
1.70
2377
2000
300
3003 16.11
2000
NOS
2028 07:00
200300
Trad 3dd
HoupRta
2011
2813
2033
2004
2008
1242.00
00-2
10:0
3041
Figure 12. Showcases of zero-shot predictions from Sundial (Base) on the FEV leaderboard (Ansari et al., 2024).
20

---

Sundial: A Family of Highly Capable Time Series Foundation Models
ECL (Prod-36)
ECL (Pred-336)
1590
3500
Prodicionn
406
900
L208
2000
2000
ETTh1 (Prod-96)
ETTh1 (Pred-336)
LoctBack
(Pred-96)
(Pred-338)
Loctact
EEN Oerienc Wooee
E95 Confancermane
1000
2000
1900
ETTml (Pred-96)
ETTmi (Pred-338)
Mohil a t
vall
vill
30
Predictinn
1200
19.00
10:00
2008
1008
1259
1750
2000
3250
3000
3700
ETTm2 (Pred-96)
ETTm2 (Pred-336)
Greandruu
Coundra
Loniback
March
trieed
30
Weather (Pred-96)
Weather (Pred-336)
-55
Craur Stith
Intancc
80% Confilanoo roan
440
435
-32
-35
250
330
3000
3250
3300
1750
2000
Figure 13. Showcases of zero-shot predictions from Sundial (Basc) on long-term forecasting datasets (Wu et al., 2022).
21

---

Sundial: A Family of Highly Capable Time Series Foundation Models
Australtan Electricity
tlectricity
68:0
14.00
600
Creat
2147
1143
3442
2097
3440
2423 18.07
203 14
3031
2031
60-27
201463 24
2013
Ettm
(9242)
09:3
19:2
04:22
Eschange Rate
4.2
Hovgittal
M1 Monthhy
1972
180
Figure 14. Showcases of Sundial (Left) and the same Transformer backbone pre-trained MSE Loss (Right). MSE Loss optimizes deterministic forecaster: given a lookback series, the model can only produce one prediction as the estimation of mean values. This objective may fail to accommodate divergent future variations during large-scale pre-training, leading mode collapse and over-smooth results (as illustrated in the fourth row). TimeFlow optimizes generative forecaster: it generates various possibilities observed in the pre-training dataset. Based on these raw predictions, can estimate the underlying complicated distribution and different statisties. Besides, the greater concentration in generated predictions, the higher the model's confidence in its predictions.
22

---

Sundial: A Family of Highly Capable Time Series Foundation Models
NL Yearly
Yearty
3073
3006
180
180
Quarterly
Indian
1M8
1M3
188
Yearty
Lookles
Loadtnt
2000
2072
1843
2001
1899
2091
MS
3.30
2014
2010.00
2014
200 54
2004
2014 2112
Weekly
www Iww Jull
M
209647
Weather
Figure 15. Supplementary showcases of Sundial (Left) and the same Transformer backbone pre-trained by MSE Loss (Right).
23


Reviews:
[{"review_id": "UucHztrJQ4", "reviewer": "Reviewer_VkQC", "summary": "This paper introduces Sundial, a family of native, flexible, and scalable time series foundation models. It proposes a TimeFlow Loss based on flow-matching for model training, which can generate multiple probable predictions. It also proposes some crucial adaptations of Transformers and the TimeBench with 1 trillion time points to enhance the time series foundation model. Sundial shows good generalization performance on zero-shot forecasting and achieves new state-of-the-art on both point forecasting and probabilistic forecasting benchmarks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks to Reviewer VkQC for providing thorough insightful comments.\n\n> **Q1**: Explanations about different tokenization ways and modeling techniques.\n\nThanks for your suggestion, we provide a comparison to enhance the clarity:\n\n| Tokenization | Meanings| Advantages | Disadvantages |\n|-|-|-|-|\n| Discrete | quantize time series into a fixed vocabulary| compatible with language modeling| foreign (discrete precision), compute-intensive, OOV risk |\n| Continuous | embed time series into latent representations | native (operate on original values), efficient (patch token) | unconstrained output range|\n\n| Modeling | Meanings| Advantages | Disadvantages |\n|-|-|-|-|\n| Parametric Densities | specify data with prior distributions | competent with suitable prior, fit well on small-scale data | inflexible, risk of mode collapse |\n| Language Modeling | predict categorical distributions of word sequences | flexible, scalable | rely on discrete tokens |\n| Generative Modeling | learn the underlying distribution that generates data | flexible, scalable, compatible with continuous tokens| require sampling|\n\n> **Q2**: Effectiveness of architectural adaptations.\n\n* RoPE maintains the temporal causality, leading to better performance.\n\n| TSLib (Zero-Shot, Averaged) | w/o RoPE | with RoPE|\n|-|-|-|\n| MSE \\| MAE| 0.302 \\| 0.360 | **0.290** \\| **0.342** |\n\n* Pre-LN improves training stability, leading to stable convergence and better performance.\n\n| TSLib (Zero-Shot, Averaged) | Post-LN (15k iter) | Post-LN (30k iter) | Pre-LN (15k iter) | Pre-LN (30k iter)|\n|-|-|-|-|-|\n| MSE \\| MAE| 0.295 \\| 0.348 | 0.297 \\| 0.350 | 0.294 \\| 0.347| **0.290** \\| **0.342** |\n\n* FlashAttention reduces computational costs. KV Cache improves the inference speed of multi-step autoregression.\n\n| Context Length = 2880, Patch Size=16, Batch Size=96 | w/o FlashAttention | with FlashAttention |\n|-|-|-|\n| Training Speed (s/iter) | 1.2723 | **1.2245**|\n| Memory Footprint (GB) | 35.41| **30.18** |\n\n| Prediction Length = 160, Autoregression Steps = 10 | w/o KV Cache | with KV Cache |\n|-|-|-|\n| Inference Speed (s/iter) | 1.08 | **0.62**|\n\n> **Q3**: Discussion of mode collapse in time series foundation models.\n\nMode collapse is a failure of representation learning, where a model generates a limited variety of outputs, ignoring the diversity in the training data. For time series foundation models, mode collapse stems from the heterogeneity of time series distribution and sometimes leads to **oversmooth predictions** (See showcases in Figure 13-14). \n\nOur work addresses it by training objectives. We adopt generative modeling to learn flexible distribution **without probabilistic priors**. As an extension of Table 3, we evaluate the distributional metric CRPS to assess the quality of generated predictions using different training objectives.\n\n| Zero-Shot Benchmark (CRPS) | ETTh1| ETTh2| ETTm1| ETTm2| ECL| Weather| GIFT-Eval |\n|-|-|-|-|-|-|-|-|\n| TimeFlow | **0.0059** | **0.0037** | **0.0057** | **0.0029** | 0.0082 | **0.0021** | **0.505** |\n| Diffusion| 0.0082 | 0.0053 | 0.0070 | 0.0039 | 0.0095 | 0.0032 | 0.534 |\n| MSE| 0.0063 | 0.0040 | 0.0058 | 0.0032 | **0.0080** | 0.0023 | 0.642 |\n\nResults show that the predictive distribution modeled by TimeFlow is **more coherent and diverse** than counterpart training objectives, especially on the highly diverse GIFT-Eval, which validates TimeFlow's effectiveness in addressing mode collapse.\n\n> **Q4**: Principles in collecting datasets in TimeBench.\n\nThe curation of trillion-scale TimeBench includes:\n\n| Preprocessing| Quality Control| Composition |\n|-|-|-|\n| impute missing values via mean values or ARIMA | measure statistics like ADF-test, predictability, exclude datasets that deviate from normal ranges | collect datasets from typical domains that consist of predictable and applicable data, such as weather, traffic, marketing, energy, etc |\n| replace abnormal values (e.g., 3-sigma)| exclude less predictable series based on the performance of simple ML forecasters | adopt synthetic technique (e.g., KernelSynth), which improves the capability of seasonal/trend prediction |\n| conduct normalization to mitigate range discrepancies | use statistics to determine the sampling weight during pre-training | use systematic datasets with well-defined temporal dynamics, such as meteorological ERA5, which enhance the understanding of local variations |\n\n> **Q5**: About the smaller model size compared to Time-MoE and Chronos.\n\nThe difference mainly comes from the **architectural choice**: Time-MoE adopts MoE, which increases parameter counts greatly. Compared with the encoder-decoder Chronos, Sundial adopts a decoder-only Transformer. To effectively scale the parameter counts, it is crucial to enhance the domain diversity and dataset completeness in the pre-trained corpora, which leaves an important future work.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria make sense for the problem.", "theoretical_claims": "There is no proof or theoretical claims.", "experimental_designs_or_analyses": "Experimental designs and analyses are sound and valid.", "relation_to_broader_scientific_literature": "This paper contributes to the literature of time series foundation models. Existing models mainly use parametric densities or language modeling, while this paper introduces a new perspective of generative modeling. It also proposes a large TimeBench for model training and a family of foundation models with SOTA performance on point and probabilistic forecasting.", "essential_references_not_discussed": "There are no essential references not discussed.", "other_strengths_and_weaknesses": "Strengths:\n\n(1)\tThis paper is well presented. Most motivations, designs, and contributions are clearly described. The proposed method is easy to follow.\n\n(2)\tThe idea of introducing generative modeling into time series foundation models is interesting. The proposed TimeFlow loss allows Transformers to be trained without discrete tokenization and make probable predictions.\n\n(3)\tThe proposed Sundial is a family of scalable and efficient time series foundation models, which achieves state-of-the-art zero-shot performance on point forecasting benchmarks and probabilistic forecasting benchmarks, including GIFT-Eval and FEV.\n\nWeaknesses:\n\n(1)\tFigure 1 needs more detailed explanations, such as the explanations on these two tokenization ways and three modeling techniques (their meanings, advantages and disadvantages) and the differences between them.\n\n(2)\tThe proposed model makes some critical adaptations to the Transformer architecture, such as RoPE, Pree-LN, FlashAttention, and KV Cache. More explanations on why these adaptations help the model are needed. It would also be better to conduct some experimental analysis on the effects of these adaptations.\n\n(3)\tThis paper mentions mode collapse in foundation models. It needs more descriptions of this term and more explanations on how the proposed model overcomes mode collapse.", "other_comments_or_suggestions": "(1)\tAre there any principles in collecting datasets in TimeBench? For example, how can we decide which dataset should or should not be included during collection? How do these different datasets help the pre-training? \n\n(2)\tWith a larger scale of pre-training data, why does Sundial use a smaller model size than Time-MoE and Chronos?", "questions_for_authors": "Please see the ’Other Strengths And Weaknesses’ and ‘Other Comments Or Suggestions’ parts. Addressing the weaknesses or suggestions above may help improve the paper.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q5h3dBA9bZ", "reviewer": "Reviewer_gDT6", "summary": "The paper introduces Sundial, a novel family of time series foundation models that address fundamental challenges in time series forecasting through a native, flexible, and scalable approach. The work's primary innovation is the proposed TimeFlow Loss, an optimization objective based on flow-matching that enables Transformers to be pre-trained directly on continuous time series data without requiring discrete tokenization. This approach allows the model to generate multiple probable predictions when conditioned on arbitrary-length time series, achieving flexibility in representation learning beyond parametric densities that constrain distribution modelling capacity. The authors make several significant technical contributions to achieve highly capable time series foundation models. First, they employ continuous tokenization through patch embedding that accommodates variable-length inputs alongside minimal but crucial adaptations to the Transformer architecture, including Pre-LN for stability, RoPE for temporal causality, and optimizations like FlashAttention and KV Cache. Second, they curate TimeBench, an unprecedented corpus containing 1 trillion time points from diverse sources, including real-world datasets and synthetic data across various frequencies. This extensive pre-training dataset enables the model to learn comprehensive temporal dynamics and patterns. The paper presents a systematic comparison between generative and deterministic forecasting paradigms, revealing that models pre-trained with MSE loss often produce over-smoothed predictions due to mode collapse on heterogeneous data distributions. In contrast, Sundial generates diverse yet coherent temporal patterns that align well with input patterns. The authors validate their design choices through extensive ablation studies that compare TimeFlow Loss with alternative approaches, demonstrating trade-offs between inference speed and prediction quality.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks to Reviewer gDT6 for providing a detailed review and recognizing our contributions.\n\n> **Q1**: Quantitative evaluation of prediction diversity.\n\nThanks for your suggestion. We extend the ablation study of Table 3, where we provide probabilistic metrics CRPS to evaluate the diversity. Note that MSE-optimized models give deterministic predictions (a peak distribution), while generative forecasters can give non-deterministic results to estimate the distribution (we keep consistent with the original paper by using 20 raw predictions):\n\n|Zero-Shot (CRPS)|ETTh1|ETTh2|ETTm1|ETTm2|ECL|Weather|GIFT-Eval|\n|-|-|-|-|-|-|-|-|\n|TimeFlow|**0.0059**|**0.0037**|**0.0057**|**0.0029**|0.0082|**0.0021**|**0.505**|\n|Diffusion|0.0082|0.0053|0.0070|0.0039|0.0095|0.0032|0.534|\n|MSE|0.0063|0.0040|0.0058|0.0032|**0.0080**|0.0023|0.642|\n\nThe predictive distribution modeled by TimeFlow is **more coherent and diverse** than its counterparts, notably on the diverse GIFT-Eval, validating TimeFlow's advantages in generative modeling.\n\n> **Q2**: Computational resources requirements.\n\nPre-training: We use 32 A100-40G GPUs. Each GPU handles a batch size of 128 and a context length of 2880. Benefiting from the patch tokenization, our mode finishes pre-training (100k iterations) on TimeBench in around 24 hours, totally using 768 GPU hours.\n\nInference (Context length=2880; Sampling steps=50; Sampling numbers=20; A single GPU with memo > 2GB): It takes 7ms to generate one prediction (140ms for 20 raw samples per forwarding). This cost is approximately proportional to sampling steps and numbers.\n\n> **Q3**: Context length sensitivity.\n\nWe provide performance under different context lengths.\n\n|Zero-Shot (MSE, Averaged)|480|960|1440|1920|2400|2880|\n|-|-|-|-|-|-|-|\n|ETTh1|0.404|0.412|0.427|0.422|0.417|0.411|\n|ETTh2|0.343|0.346|0.337|0.336|0.340|0.333|\n|ETTm1|0.385|0.375|0.361|0.349|0.339|0.336|\n|ETTm2|0.288|0.287|0.264|0.252|0.255|0.258|\n|ECL |0.171|0.172|0.171|0.169|0.169|0.169|\n|Weather|0.248|0.239|0.237|0.235|0.236|0.234|\n\nNote that Sundial handles various context lengths on FEV and GIFT-Eval. Unlike fixed-context models, Sundial can be flexible for practitioners, where the context length can be dynamically adjusted during inference instead of re-training.\n\n> **Q4**: Discussion about the conservative prediction behavior.\n\nWe delved into this behavior after the initial submission, focusing on the pre-training distribution: Meteorological data (ERA5) taking a great proportion is less likely to encompass extreme trends. The conservative behavior is mitigated by reweighing training samples in TimeBench. \n\n|Zero-Shot (FEV)|MASE|WQL|\n|-|-|-|\n|Original|0.845|0.712|\n|Reweighting|**0.832**|**0.685**|\n\n> **Q5**: Controlled comparisons of baseline models.\n\nWe report official results from public leaderboards and papers. We regard the context length as an inherent capability, which a TSFM can make continuous self-improvement.\n\nDespite different context lengths in baseline models, our experiments still validate that performance improvement is not simply attributable to length variations: On TSLib, Sundial (context length=2880) surpasses previous state-of-the-art Time-MoE (context length=3072); On FEV and GIFT-Eval, there are more than half time series with less than 512 context length, less than the maximum context length of most TSFMs. We will also extend the baseline evaluation like **Q3** in the revised version.\n\n> **Q6**: Extension for multivariate forecasting.\n\nUnivariate pre-training addresses different variate numbers of datasets during pre-training. A multivariate extension is crucial for the future improvement of Sundial, including:\n\n* Architecture: Recent works proposed new **attention mechanisms** (e.g., Moirai, Timer-XL) for intra-/inter-variate modeling, which can be seamlessly incorporated into Sundial. Multivariate extension on the **flow-matching network** (e.g., from MLP to iTransformer) is applicable to make the post-merging on univariate representations.\n\n* Post-training: Another roadmap is univariate pre-training and multivariate fine-tuning. Similar to GPT-3, a univariate pre-trained TSFM is a **start-point model**. We will explore multivariate prompting (e.g., special variate token) to instruct TSFMs on downstream tasks. \n\n> **W1**: Experimental supplement to mentioned issues:\n\n* Quantify the relationship between parameters and inference time (Extension of Figure 6):\n\n|FEV Evaluation|Small|Base|Large|\n|-|-|-|-|\n|Parameters|32 M|128 M|444 M|\n|Inference Time|2.36ms|3.13ms|5.31ms|\n\n* Statistical significance testing (five runs):\n\n|Zero-Shot (MSE)|ETTh1|ETTh2|ETTm1|ETTm2|ECL|Weather|\n|-|-|-|-|-|-|-|\n|Mean$\\pm$Std.|0.411 $\\pm$ 0.001|0.333 $\\pm$ 0.000 |0.336 $\\pm$ 0.000|0.258 $\\pm$ 0.001|0.169 $\\pm$ 0.000|0.234 $\\pm$ 0.001|\n\nDifferent from the high variance typically observed in forecasting performance, generative forecasting aggregates multiple predictions for calibration, achieving a small variance.", "claims_and_evidence": "The authors make several significant claims regarding their proposed time series foundation models, and overall, most claims are backed by thorough evidence experimentation. The central contribution—the TimeFlow Loss based on flow-matching—is well-established through comprehensive mathematical formulations in Sections 3.1 and 4.1.3, providing a clear theoretical foundation for the approach. The claim of state-of-the-art performance is convincingly demonstrated through extensive benchmarking across multiple datasets. Table 1 shows Sundial consistently outperforming other advanced foundation models on point forecasting tasks, with quantitative improvements (7.57% MSE reduction and 4.71% MAE reduction compared to Time-MoE). For probabilistic forecasting, Table 2 demonstrates Sundial achieving first place in MASE and second place in CRPS on the GIFT-Eval benchmark across 23 datasets. These results are particularly impressive given the zero-shot nature of the evaluation. \n\n---\nThe authors' claims regarding model scalability are supported by both empirical performance gains across model sizes (as shown in Table 1) and convergence improvements (Figure 7 shows a 15.38% reduction in training objectives for larger models). However, the scaling analysis could have been strengthened with more model size variants to establish clearer scaling laws. The assertion that TimeFlow Loss mitigates mode collapse is partially supported through ablation studies in Table 3 comparing different training objectives and visualizations in Figures 13-14 that contrast Sundial's diverse predictions with the over-smoothed outputs from MSE-trained models. While these comparisons are informative, a more quantitative evaluation of prediction diversity would have strengthened this particular claim.\n\n---\nThe paper's ambitious dataset contribution (TimeBench with 1 trillion time points) is well-documented in Table 4, with clear attribution of sources and distributions, lending credibility to the large-scale pre-training claims. The inference speed claims require some nuance—while Figure 6 demonstrates that Sundial achieves an 11.34× speedup compared to Chronos, it's not necessarily faster than all baseline approaches. This represents a reasonable trade-off given the probabilistic capabilities of the model.", "methods_and_evaluation_criteria": "The methods and evaluation criteria proposed in the paper are fundamentally well-aligned with the challenges of time series foundation modelling. The authors recognize a critical limitation in existing approaches—specifically, the tension between discrete tokenization (which limits the representation of continuous values) and parametric densities (which restrict distribution modelling capacity). Their proposed TimeFlow Loss offers a theoretically sound solution by enabling Transformers to learn flexible predictive distributions without requiring tokenization or prior distribution specification, which is particularly appropriate for the heterogeneous nature of large-scale time series corpora. \n\n---\nThe patch-based continuous tokenization approach sensibly balances computational efficiency with representation quality, addressing the unique characteristics of time series data while maintaining compatibility with Transformer architectures. Technical adaptations like Pre-LN for stability and RoPE for temporal causality demonstrate thoughtful consideration of the domain-specific challenges in time series modelling. \n\n---\nRegarding evaluation criteria, the authors employ a comprehensive benchmarking strategy that convincingly validates their claims. The use of both point forecasting metrics (MSE, MAE) and probabilistic metrics (CRPS, MASE, WQL) across three established benchmarks—Time-Series-Library, GIFT-Eval, and FEV—ensures thorough performance assessment. The GIFT-Eval benchmark is particularly appropriate as it encompasses 23 datasets with diverse characteristics, providing a robust measure of generalization capability. The evaluation against both statistical methods and competing foundation models offers the necessary context for interpreting performance gains. \n\n---\nThe TimeBench dataset, with its unprecedented scale of 1 trillion time points and diverse sources spanning different frequencies and domains, constitutes an appropriate foundation for pre-training models intended for broad applicability. The careful exclusion of test datasets from pre-training data demonstrates methodological rigour in preventing data leakage that could invalidate zero-shot performance claims.\n\n---\nThe ablation studies comparing TimeFlow Loss with alternative approaches (Table 3) and the exploration of inference trade-offs (Figure 8) provide crucial scientific validation of design choices. However, the paper would benefit from more domain-specific evaluations to complement the general-purpose benchmarks and from deeper analysis of prediction diversity beyond visual showcases. Nevertheless, the overall approach and evaluation strategy are well-matched to the foundational goals of developing generalizable, probabilistic time series models capable of effective zero-shot performance.", "theoretical_claims": "The paper's main theoretical foundation rests on the flow-matching framework originally proposed by Lipman et al. (2022), which the authors adapt to time series forecasting. The primary theoretical contribution—TimeFlow Loss—is formulated in Section 4.1.3, where the authors extend the conditional flow-matching objective to handle sequential time series data. While the mathematical formulation is clearly presented (Equations 4-7), the authors don't provide formal proofs for the new theoretical properties of this adaptation. Instead, they demonstrate its effectiveness through empirical validation across multiple benchmarks. \n\n---\nThe theoretical formulation builds directly upon the preliminaries established in Section 3.1, where the standard flow-matching framework is introduced. The authors correctly present the existing theoretical elements including the velocity field ODE (Equation 1), the Flow-Matching objective (Equation 2), and the Conditional Flow-Matching objective with Gaussian formulation (Equation 3). These equations are properly cited to the original work. The adaptation to time series involves conditioning the flow-matching process on a learned representation h_i, which appears mathematically sound, but the paper doesn't provide a rigorous proof of its properties beyond empirical results. The inference procedure (Algorithm 1) is a straightforward application of numerical ODE solving, consistent with standard flow-matching literature.\n\n---\nThe claim that TimeFlow Loss mitigates mode collapse during pre-training is primarily substantiated through qualitative visual showcases (Figures 13-14) and comparative experiments (Table 3) rather than through theoretical guarantees. This empirical validation approach is reasonable for a systems paper but limits the theoretical depth of the contribution.\n\n---\nFrom my understanding, while the paper presents a coherent mathematical framework for applying flow-matching to time series forecasting, it does not contain novel theoretical proofs requiring verification. The mathematical soundness of the approach derives from its faithful extension of established flow-matching theory, with innovations focused on architecture and application rather than fundamental theoretical advances.", "experimental_designs_or_analyses": "The authors present a comprehensive evaluation framework for their proposed time series foundation models, with experiments that are generally well-designed but I think it contains several methodological limitations that merit consideration. I am providing some of them here. \n\n(1) The zero-shot forecasting evaluation methodology demonstrates strong validity through the use of established benchmarks (Time-Series-Library, GIFT-Eval, and FEV) and clear separation between pre-training and evaluation datasets. The authors implement appropriate metrics (MSE/MAE for point forecasting and MASE/CRPS/WQL for probabilistic forecasting) that align with community standards. The careful exclusion of evaluation datasets from the pre-training corpus (explicitly denoted by dashes in Table 1) strengthens the experimental integrity of zero-shot claims. However, the comparison methodology raises concerns regarding controlled evaluation conditions. While Table 6 documents architectural differences between models, the paper doesn't explicitly confirm whether all baseline models had access to identical context lengths during inference, which could significantly impact forecasting performance. Additionally, the reliance on officially reported results from other papers (noted in Table 7) introduces potential inconsistencies in evaluation protocols that could affect fair comparison.\n\n---\n(2) The ablation study for TimeFlow Loss (Table 3) is methodologically sound in maintaining architectural consistency while varying only the training objective. However, this analysis is limited to reporting point forecasting metrics (MSE) without evaluating probabilistic metrics, which undermines a complete assessment of the central claim regarding improved distribution modelling. The qualitative comparisons in Figures 13-14 partially address this gap but lack objective quantification of prediction diversity.\n\n---\n(3) The scaling behaviour analysis in Figure 7 presents valid training curves but would be strengthened by more intermediate model sizes to establish clearer scaling laws. Similarly, the inference speed versus performance trade-off analysis (Figure 8) systematically varies generation parameters but doesn't properly quantify the relationship between these parameters and actual inference time.\n\n---\n(4) Most concerning is the absence of statistical significance testing or confidence intervals across all experimental results, particularly important given the high variance typically observed in time series forecasting performance. This omission limits the robustness of performance comparisons, especially for closely-matched results.", "relation_to_broader_scientific_literature": "I think this work makes several notable contributions that both build upon and diverge from established research trajectories in time series modelling. The authors' central contribution—TimeFlow Loss—represents a significant advancement in how foundation models handle continuous-valued time series data. This approach extends the flow-matching framework of Lipman et al. (2022) to the autoregressive time series domain, creating a bridge between continuous generative modelling and sequential forecasting that has been largely unexplored in prior literature. The authors' decision to embrace generative modeling rather than discrete tokenization strategically positions their work as an alternative to the language-modeling inspired approach taken by Chronos (Ansari et al., 2024). While Chronos adapted techniques from NLP by discretizing continuous values, Sundial's native approach avoids the information loss inherent in quantization—addressing a fundamental limitation recognized but unresolved in previous work. Similarly, the TimeFlow Loss offers greater flexibility than the parametric mixture distributions employed by Moirai (Woo et al., 2024), which the authors correctly identify as potentially constraining when modelling heterogeneous time series distributions at scale.\n\n---\nThe architectural adaptations, while individually derived from existing techniques like RoPE (Su et al., 2024) and Pre-LN (Xiong et al., 2020), represent a thoughtful integration of components that specifically address the challenges of time series forecasting. This integration draws from disparate research streams that have not previously been united for time series foundation models. Particularly, the deliberate incorporation of FlashAttention and KV Cache reflects an understanding of the efficiency challenges faced by practitioners—a consideration often neglected in academic time series research but well-established in large language model literature. \n\n---\nTimeBench's trillion-point scale represents an order-of-magnitude increase over previous datasets like those used in Time-MoE (300B, Shi et al., 2024b) and Timer (231B, Liu et al., 2024a,b). This scaling aligns with the broader foundation model literature's emphasis on dataset size as a critical factor in model capability, while specifically addressing the unique challenges of time series heterogeneity. The authors' careful compilation of diverse frequency data connects to emerging work on scaling laws for time series (Shi et al., 2024a), extending these insights into previously unexplored data volumes.\n\n---\nI think the empirical results position Sundial at the intersection of deterministic and probabilistic forecasting research streams. The comparative analysis against both MSE-trained models and diffusion-based alternatives provides a valuable empirical bridge between these previously separate approaches. This integration of multiple modelling paradigms within a unified foundation model framework represents a meaningful synthesis of previously disparate research directions in time series forecasting.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "This work demonstrates considerable originality in its approach to time series foundation models. The authors identify a fundamental tension in prior work—the trade-off between discrete tokenization and parametric distributions—and present a creative solution through their TimeFlow Loss framework. This contribution is conceptually significant as it bridges flow-matching techniques with autoregressive time series modelling, enabling Transformers to learn flexible distributions directly from continuous values without imposing restrictive prior distributions. This represents a meaningful step forward in the relatively nascent field of time series foundation models, not merely an incremental improvement over existing methods.\n\n---\nThe paper's most significant contribution is its comprehensive approach to the generative modelling paradigm for time series. While individual components (flow-matching, Transformer architectures) exist in isolation, their integration into a cohesive framework specifically designed for time series forecasting demonstrates genuine innovation. The TimeBench dataset with 1 trillion time points represents a substantial resource contribution to the research community and enables proper exploration of scaling behaviour in time series foundation models, an important but previously under-explored area.\n\n---\nFrom a technical writing perspective, the paper generally maintains strong clarity. The mathematical formulations in Sections 3 and 4 are precise and well-structured, providing sufficient detail for implementation. The ablation studies effectively isolate the contribution of the TimeFlow Loss compared to alternatives, though they could benefit from more rigorous quantification of prediction diversity beyond the qualitative showcases in Figures 13-14.\n\n---\nHowever, several weaknesses merit attention. While the model shows impressive zero-shot performance, the paper inadequately addresses computational efficiency during training. Training on 1 trillion time points likely requires significant computational resources, but the paper provides a limited discussion of training time, hardware requirements, or environmental impact—considerations increasingly important in foundation model research. Additionally, the inference procedure introduces complexity with its sampling parameters, creating practical deployment challenges that aren't thoroughly addressed.\n\n---\nThe model's conservative prediction behaviour (noted in the limitations section) represents a substantive weakness that could limit practical utility. Since accurate trend forecasting is critical in many applications, this limitation deserves a more thorough analysis rather than a brief acknowledgment. Furthermore, the univariate pre-training approach, while pragmatic, sidesteps the important challenge of modelling inter-variate correlations in multivariate time series—a limitation that restricts the model's applicability to many real-world scenarios where complex interdependencies exist.\n\n---\nI think despite these limitations, the paper makes a compelling contribution to time series foundation model research by establishing a new paradigm for generative modelling in this domain, demonstrating strong empirical results, and providing a solid foundation for future work in this direction.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Here are my questions and I would love to hear back from the authors on them.\n\n---\n(1) Quantification of Prediction Diversity: While you demonstrate TimeFlow Loss mitigates mode collapse through visualizations in Figures 13-14, could you provide quantitative metrics to measure prediction diversity across your generated samples? I think this would strengthen your claims regarding the advantages of flow-matching over MSE-based training and help practitioners better understand when to choose each approach.\n\n---\n(2) Computational Resource Requirements: Your work demonstrates impressive results using a trillion-point dataset but lacks details on computational requirements. Could you provide specific information about training time, hardware configurations, and estimated computational costs? This information would be valuable for assessing scalability and environmental impact considerations that are increasingly important in foundation model research.\n\n---\n(3) Context Length Sensitivity: How sensitive is Sundial's performance to the availability of historical context? Your experiments maintain fixed context lengths, but real-world applications often face varying historical data availability. An analysis showing performance degradation curves as context length decreases would provide important insights into the model's robustness in practical deployment scenarios.\n\n---\n(4) Conservative Prediction Behaviour: You briefly mention conservative prediction behaviour as a limitation. Could you elaborate on the specific scenarios where this manifests most prominently and provide a quantitative analysis of this phenomenon? This information would help users understand when Sundial might underperform on trend forecasting tasks compared to alternative approaches.\n\n---\n(5) Controlled Baseline Comparisons: For comparing against baseline models (Table 1), did all models have access to identical context lengths and inference settings? The architectural differences noted in Table 6 raise questions about whether performance differences might be partially attributable to these variations rather than fundamental modelling approaches.\n\n---\n(6) Multivariate Extension Strategy: Given your univariate pre-training approach, what specific architectural or training methodology modifications would be required to effectively model complex inter-variate correlations in multivariate settings? Would this require fundamental changes to the TimeFlow Loss formulation or primarily architectural adaptations? I think a clearer roadmap for this extension would significantly enhance the paper's impact.\n\n---\nI am happy to change the score if the answers are convincing, and look forward to hearing back from the authors.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "bxQlqKXs2b", "reviewer": "Reviewer_4SCU", "summary": "The paper presents a collection of foundation models for time series. To this end, the authors proposed a loss called TimeFlow for predicting the distribution next-patch, enabling Transformers to be pre-trained without the need for discrete tokenization. It is argued that this loss function helps in preventing mode collapse. The pre-training is conducted on a dataset called TimeBench.\n\n## update after rebuttal\nFollowing the rebuttal and the authors' feedback, I have increased my scores accordingly.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks to Reviewer 4SCU for providing a valuable review.\n\n> **Q1**: Include the relevant citation (Li et al. 2024).\n\nWe referred to it in Section 5.3. In this Section, we provided the comparison using different training objectives: MSE Loss, Diffusion Loss (Li et al., 2024), and TimeFlow Loss (Ours). Performance using Diffusion Loss is notably inferior to TimeFlow Loss. The different generative frameworks and target modalities distinguish our work from the previous work.\n\n> **Q2**: Detailed description of the TimeBench datasets.\n\nThanks for your valuable suggestion for the dataset description. We consider TimeBench as an unprecedented large-scale dataset since the previous scale of time series datasets is still at the million level. **The largest pre-trained time series dataset before our work was 300B (Shi et al., 2024)**. TimeBench initially extends the scale to the trillion level, following a streamlined curation:\n\n* **Collection**: Different from other modalities, time series data are highly heterogeneous and confidential. Most of them are unavailable on open websites or repositories. There are also limited domains that encompass typical and predictable time series, such as weather, traffic, marketing, energy, etc, leading to slow progress on dataset construction.\n* **Preprocessing**: Due to common device faults, it is difficult to pre-train large models using raw time series data. We have conducted tedious preprocessing, including missing values imputation, abnormal exclusion, and normalization techniques.\n* **Quality control**: We conduct statistical analysis in our collection, examining time series through the lenses of intrinsic properties, e.g., non-stationarity, forecastability, and seasonality. This approach allows us to characterize the data quality inherent to time series, which affects the training stability of next-token prediction.\n* **Diversity and Generality**: We adopt synthetic techniques to improve pattern diversity and the capability of seasonal/trend forecasting. Further, we adopt ERA5, which includes well-defined and systematic real-world temporal observations. \n\n**We will provide a clear and detailed description of TimeBench (See also **Q4 of Reviewer VkQC**) in the final revision**.\n\n> **Q3**: Does TimeBench have a great influence on the results compared to other methods that consider smaller datasets?\n\nWe compare Sundial with other time series foundation models that are pre-trained with smaller datasets. We also conduct pre-training on Sundial using different scales of datasets (Chronos-94B, LoTSA-230B, TimeBench-1032B). These results highlight **the scaling behavior of using larger datasets**.\n\n| Zero-Shot (MSE \\| MAE) | Chronos (94B)  | Moirai (230B)      | Time-MoE (300B)    | Sundial (94B)  | Sundial (230B)     | Sundial (1032B)        |\n| ---------------------- | -------------- | ------------------ | ------------------ | -------------- | ------------------ | ---------------------- |\n| ETTh1                  | 0.591 \\| 0.468 | 0.417 \\| **0.419**     | **0.400** \\| 0.424 | 0.402 \\| 0.429 | 0.403 \\| **0.419** | 0.411\\| 0.434          |\n| ETTh2                  | 0.405 \\| 0.410 | 0.362 \\| **0.382** | 0.366 \\| 0.404     | 0.377 \\| 0.414 | 0.364 \\| 0.398     | **0.333** \\| 0.387     |\n| ETTm1                  | 0.645 \\| 0.500 | 0.406 \\| 0.385     | 0.394 \\| 0.415     | 0.367 \\| 0.402 | 0.352 \\| 0.385     | **0.336** \\| **0.377** |\n| ETTm2                  | 0.310 \\| 0.350 | 0.311 \\| 0.337     | 0.317 \\| 0.365     | 0.280 \\| 0.341 | 0.273 \\| 0.334     | **0.258** \\| **0.320** |\n| ECL                    | 0.214 \\| 0.278 | 0.187 \\| 0.274     | in distribution    | 0.172 \\| 0.269 | 0.171 \\| 0.267     | **0.169** \\| **0.265** |\n| Weather                | 0.292 \\| 0.315 | 0.287 \\| 0.281     | 0.265 \\| 0.297     | 0.254 \\| 0.301 | 0.252 \\| 0.297     | **0.234** \\| **0.270** |", "claims_and_evidence": "Yes, they are clear and convincing.", "methods_and_evaluation_criteria": "YES", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "The experimental designs and the conducted analysis seem to be acceptable.", "relation_to_broader_scientific_literature": "I find the idea interesting and the results pertinents.", "essential_references_not_discussed": "The following paper is not referenced in the submitted paper, even though it achieves high-quality image generation without the need for discrete tokenization, thus having some connections to the proposed method:\nTianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. arXiv preprint arXiv:2406.11838, 2024.", "other_strengths_and_weaknesses": "It is not clear why the authors considered as “unprecedented” the large-scale TimeBench dataset. Moreover, it would have been better to clearly describe this dataset, and how it was curated. The description in the paper and appendix is only limited to half a column. The paper would have benefited from a clear in-depth description of the dataset.\nMoreover, it is not clear the influence of the dataset to the obtained results, namely does this dataset has a great influence on the obtained results compared to other methods that considered smaller datasets.\n\nFollowing the rebuttal and the authors' feedback, I have increased my scores accordingly.", "other_comments_or_suggestions": "none", "questions_for_authors": "none", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: 1. It proposes a TimeFlow Loss based on flow-matching for model training, which can generate multiple probable predictions, demonstrated on both point forecasting and probabilistic forecasting benchmarks. [Reviewer VkQC]
2. The idea of introducing generative modeling into time series foundation models is interesting. [Reviewer VkQC]
3. The proposed TimeFlow loss allows Transformers to be trained without discrete tokenization and make probable predictions. [Reviewer VkQC]
4. This work demonstrates considerable originality in its approach to time series foundation models. [Reviewer gDT6]
5. Most significant claims made by the authors are backed by thorough evidence experimentation. [Reviewer gDT6]
6. The central contribution—the TimeFlow Loss based on flow-matching—is well-established through comprehensive mathematical formulations in Sections 3.1 and 4.1.3, providing a clear theoretical foundation for the approach. [Reviewer gDT6]
7. The paper's ambitious dataset contribution (TimeBench with 1 trillion time points) is well-documented in Table 4, with clear attribution of sources and distributions, lending credibility to the large-scale pre-training claims. [Reviewer gDT6]
8. The patch-based continuous tokenization approach sensibly balances computational efficiency with representation quality, addressing the unique characteristics of time series data while maintaining compatibility with Transformer architectures. [Reviewer gDT6]
9.Technical adaptations like Pre-LN for stability and RoPE for temporal causality demonstrate thoughtful consideration of the domain-specific challenges in time series modelling. [Reviewer gDT6]
10. The paper presents a collection of foundation models for time series. To this end, the authors proposed a loss called TimeFlow for predicting the distribution next-patch, enabling Transformers to be pre-trained without the need for discrete tokenization. [Reviewer 4SCU]

All reviewers acknowledge the author responses.
Decision: Accept (oral)