Chronos: Learning the Language of Time Series
Authors: Abdul Fatir Ansari
Lorenzo Stella
Ali Caner Turkmen
Xiyuan Zhang
Pedro Mercado
Huibin Shen
Oleksandr Shchur
Syama Sundar Rangapuram
Sebastian Pineda Arango
Shubham Kapoor
Jasper Zschiegner
Danielle C. Maddix
Hao Wang
Michael W. Mahoney
Kari Torkkola
Andrew Gordon Wilson
Michael Bohlke-Schneider
Bernie Wang
Venue: Accepted by TMLR ()

Abstract: We introduce Chronos, a simple yet effective framework for pretrained probabilistic time series models. Chronos tokenizes time series values using scaling and quantization into a fixed vocabulary and trains existing transformer-based language model architectures on these tokenized time series via the cross-entropy loss. We pretrained Chronos models based on the T5 family (ranging from 20M to 710M parameters) on a large collection of publicly available datasets, complemented by a synthetic dataset that we generated via Gaussian processes to improve generalization. In a comprehensive benchmark consisting of 42 datasets, and comprising both classical local models and deep learning methods, we show that Chronos models: (a) significantly outperform other methods on datasets that were part of the training corpus; and (b) have comparable and occasionally superior zero-shot performance on new datasets, relative to methods that were trained specifically on them. Our results demonstrate that Chronos models can leverage time series data from diverse domains to improve zero-shot accuracy on unseen forecasting tasks, positioning pretrained models as a viable tool to greatly simplify forecasting pipelines.

Published in Transactions on Machine Learning Research (10/2024)
Chronos: Learning the Language of Time Series
Abdul Fatir Ansari"; Lorenzo Stella"; Caner Turkmen', Xiyuan Zhang Pedro Mercado Huibin Shen', Oleksandr Shchur', Syama Sundar Rangapuram' Sebastian Pineda Arango Shubham Kapoor', , Jasper Zschiegner, Danielle C. Maddix', Hao Wang Michael W. Mahoney26, Kari Torkkola', Andrew Gordon Wilson Michael Bohlke-Schneider' Yuyang Wang
(ansarnd, stellalo) Bamazon. com
AWS Labs, AAmazon Supply Chain Optimization Technologies, UC San Diego, 4 University of Freibury. $ Rutgers University, UC Berkeley, "New York University
Reviewed on OpenReview: https://openrevie wttp://wwww./.oooooooo//////oo/o Code and Pretrained Models: https: thub. ctw/ wtt.///aazaaniiniaaaa wwt.a//mmsccceeeeeeeeeooooorrrre
Abstract
We introduce CURONOS, simple yet effective framework for pretrained probabilistic time series models. CHRONOS tokenizes time series values using scaling and quantization into a fixed vocabulary and trains existing transformer-based language model architectures on those tokenized time series via. the cross-entropy loss. We pretrained CHRONOS models based on the T5 family (ranging from 20M to 710M parameters) on a large collection of publicly available datasets, complemented by a synthetic dataset that we generated via Gaussian processes to improve generalization. In a comprehensive benchmark consisting of 42 datasets, and comprising both classical local models and deep learning methods, we show That CHRONOS models: (a) significantly oulperform other methods on datasets that were part of the training corpus; and (b) have comparable and occasionally superior zero-shot performance on new datasets, relative to methods that were trained specifically on them. Our results demonstrate that CHRONOS models can leverage time series data from diverse domains to improve zero-shot accuracy on unseen forecasting tasks, positioning pretrained models as a viable tool to greatly simplify forecasting pipelines.
1 Introduction
Time series forecasting an essential component of decision-making across various domains, including retail, energy, finance, healthcare, climate science, among others. Traditionally, forecasting has been dominated by statistical models such ARIMA and ETS. These have served as reliable tools, at least until the recent shift towards deep learning techniques (Hyndman & Athanasopoulos, 2018; Benidis et al., 2022). This shift can be attributed to the availability of large and diverse time series data sources, and the emergence of operational forecasting problems (Kolassa & Jamuschowski, 2019) that play to the strengths of deep forecasters, i.e., the ability to extract patterns out of large collection of time scries. Despite their impressive performance, deep forecasters still operate in the standard regime of training and prediction on the same dataset. While there have been works dedicated to transfer learning (Ye & Dai, 2018) and domain adaplation (Jin et al., 2022) for forecasting, the field has yet to converge unified, general-purposs forecasting model, goal that remains beacon for time scries researchers.
The emergence of large language models (LLMs) with zero-shot learning capabilities has ignited interest in developing "foundation models" for time series. In the context LLMs, this interest has been pursued through two main avenues: directly prompting pretrained LLMs in natural language (Gruver al., 2023;
"Equal contribution.
Xiyuan Zhang and Scbastian Pineda Arango contributed this work during their internships at AWS, Hao Wang, Michael Mahoney, and Andrew Gordon Wilson hold concurrent appointments Amazon and their corresponding universities, and this paper describes work performed Amazon.

---

Published in Transactions on Machine Learning Research (10/2024)
Xue & Salim, 2023) and fine-tuning LLMs for time series tasks (Zhou et al., 2023a; Jin et al., 2024). However, these methods face significant limitations, notably the need for prompt engineering or fine-tuning for each new task, or reliance on large-scale models (GPT-3 (Brown et al., 2020), Llama 2 (Touvron et al., 2023), etc.) that demand substantial computational resources and time for inference. Recent concurrent work (Dooley et al., 2023; Das et al., 2023; Rasul et al., 2023; Woo al., 2024) also explores pretraining transformer-Dase models with sophisticated time-scries-specifi designs on a large corpus of real and (or) synthetic time scries data.
In this work, we take step back and ask: what are the fundamental differences between language model that predicts the next token, and time scries forccasting model that predicts the next values? Despite the apparent distinction tokens from finite dictionary versus values from an unbounded, usually continuous domain - both endeavors fundamentally aim to model the sequential structure of the data to predict future patterns. Shouldn't 't good language models "just work" on time series? This naive question prompts us to challenge the necessity of time-series-speeific modifications, and answering it led us to develop CHRONOS, language modeling framework minimally adapted for time series forecasting. CHRONOS tokenizes time series into discrete bins through simple scaling and quantization of real values. In this way, we train off off-the-shelf language models on this "language of time scries," with no changes to the model architecture (see Figure/ 1 for a high-level depiction of CHRONOS). Remarkably, this straightforward approach proves to be effective and efficient, underscoring the potential for language model architectures to address a broad range of time series problems with minimal modifications.
Time Series Tokenization
Training
Inference
Historical Time Series
Context Tokens
Context Tokens
2400
21:2
2282
2246
23110
2400
21:42
/282
2246
2510
Moee Sooking
Time Series
Time Series Language Model
Language Model
Ouerttation
Predicted
trobabiitiia
Dequantztton Sampled Uniciing
2360
2400
2142
2282
2245
2310
2360
Context Tokens
Probubiistic Forecast
Figure 1: High-level depiction of CHRONOS. (Left) The input time series is scaled and quantized to obtain a sequence of tokens. (Center) The tokens are fed into language model which may either be an encoder-decoder or a decoder- only model. The model is trained using the eross-entropy loss. (Right) During inference, we autoregressively sample tokens from the model and map them back to numerical values. Multiple trajectories are sampled to obtain a predictive distribution.
For the development of a useful general-purpes time series forecasting model, the scarcity of publicly available time series datasets, both in quantity and quality, is arguably more critical than the modeling framework. In addition the comprchensive collection of public datasets we used to train CHRONOS, a central aspect of our approach is the integration of data augmentation strategies, including TSMixup and KernelSynth. TSMixup randomly samples a set of base time series from different training datasets, and generates new time series based on a convex combination of them; KernelSynth uses Gaussian processes to generate synthetic time series by randomly composing kernel functions. These techniques address the inherent. limitations of small training datasets in time series forecasting, enhancing model robustness and generalization.
Our comprehensive evaluation across 42 datasets establishes CHRONOS as a benchmark for both in-domain and zero-shot. forecasting, surpassing both traditional models and task-specific deep learning approaches.
2

---

Published in Transactions on Machine Learning Research (10/2024)
Notably, CHRONOS achieves impressive zero-shot forecasting performance out of the box, without necessi- tating task-specific adjustments. Its accuracy, coupled with its relatively modest model size, positions it as a preferable alternative to larger, more computationally demanding models for zero-shot forecasting appli- cations. By its very nature as a language model operating over a fixed vocabulary, CHRONOS can scamlessly integrate with future advancements LLMs, making an ideal candidate for further development as a generalist time series model.
The rest of the paper is organized follows. Section 2 introduces the background on time scries forecasting and language models, and discusses related work. In Section 3, we describe CHRONOS, our proposed language modeling framework for time series. Section 4 discusses our data augmentation technique and synthetic time series generation process. In Section 5, we present our main results and a rigorous analysis of different design choices. We discuss future directions in Section 6, and conclude the paper in Section 7. Additional material is presented in the appendices.
2 Background and Related Work
Time series forecasting concerns using historical data from a quantity of interest (typically real-valued) to predict their future values. Formally, given a uniformly-spaced time series æ1.0 -t,,...;.c),, we are interested in predicting the joint distribution of the next H steps, p( x1.C). In this work, we focus on univariate forecasting, where the observations are scalars, i.c., I2 E R for all i.
Time series lorecasting can be addressed with a variety of different mots which can be broadly categorized into classical forecasting methods and deep learning methods. Classical forecasting methods such as ETS, ARIMA (Hyndman et al., 2008), Theta (Assimakopoulos & Nikolopoulos, 2000) fit a separate model to each time series independently (hence referred to as local models). In contrast, deep learning forecasting models learn across time series in a given dataset (and are called global models). These methods leverage advances in deep learning, such as RNNs which are used by DeepState (Rangapuram et al., 2018), DeepFactor (Wang et al., 2019), DeepAR (Salinas et al., 2020), TimeGrad (Rasul et al., 2021), and transformers which are used by TFT (Lim et al., 2021) and TSL (Nie et al., 2023). Apart from the choice of architecture, these approaches differ in the way they model the target, with some modeling the density function while others directly predicting a set of quantiles (Wen et al., 2017; Gasthaus et al., 2019; Park et al., 2022). Nevertheless, not all models produce probabilistic forecasts: notably, models such as Informer (Zhou et al., 2021) and DLinear (Zeng et al., 2023) only produce point forecasts.
Large language models (LLMs) have demonstrated impressive performance on various natural language processing tasks (Brown et al., 2020; Chung et al., 2022; Touvron et al., 2023). Given a sequence of input to- kens, wisk = [wi...., we). language models aim to predict the next token, Wk+1, by modeling the conditional distribution, p(wp+1/////)/// w1+)). The tokens belong to a vocabulary, V, and may be characters, subwords (Sennrich et al., 2015), or words, depending on the lokenization scheme used.
Most modern LLMs (Brown al., 2020; Chung et al., 2022; Touvron et al., 2023) are based on the transformer architecture (Vaswani et al., 2017). The original transformer architecture is an encoder-decdeer model designed for machine translation. The encoder maps an input sentence of some language to a continuous representation, and the decoder generates the translation token-by-token using the input representation and previously decoded tokens. Many popular language models, such as BART (Lewis et al., 2019) and T5 (Raffel et al., 2020; Chung et al., 2022), belong to this family. Another popular architecture for LLMs is decoder-only, used in GPT-3 (Brown al., 2020) and Llama (Touvron et al., 2023), where the model only attends to tokens up to the current token. LLMs are typically trained on a very large corpus of text with their number of parameters ranging from millions (Raffel et al,, 2020) to hundreds of billions (Chowdhery et al., 2023). We refer the reader to Zhao et al. (2023) for recent survey on this area of research.
LLM-based forecasters. Inspired by the success of pretrained LLMs, recent work has shown that LLMs are general pattern recognizers (Mirchandani et al., 2023) and several methods adapting LLMs to the time scries domain have been developed. One line of work treats numerical time series data as raw text and directly uses the pretrained LLMs with minimal or no fine tuning to forecast unseen time series. PromptCast (Xue & Salim, 2023) leverages pretrained L.MSs for forecasting by transforming the time series data into text-based
3

---

Published in Transactions on Machine Learning Research (10/2024)
imput and output pairs and reformulating the forecasting problem question answering task. However, PromptCast requires datasel-specific templates for converting numerical data to text prompts. Perhaps the most straightforward LLM-based forecasting model is LLMTTime (Gruver et al., 2023), which shows clear evidence for zero-shot forccasting ability of pretrained LLMs on a variety of benchmark time series datasets. LLMTime proposes a new lokenization scheme that encodes real-valued data as a string of digits after fixing the numerical precision and scaling the data appropriately. Once encoded as strings, forecasts are obtained in a zero-shot setting from pretrained LLMs such as GPT-3 (Brown et al., 2020) and Llama 2 (Touvron et al., 2023). Nevertheless, the use of such compute-hungry models hampers the scalability and practical utility of LLMTime.
Zhou et al. (2023a) propose a unified one-fits-all model (GPT4TS) for different time series analysis tasks by using a pretrained GPT-2 model (Radford et al., 2019) as a backbone and only fine-tune the positional embeddings and the parameters of the layer normalization for each individual task. Instead of using tokenized input, they directly foed the model with patch emboddings, similar to PatchTST (Nic al., 2023). Recent concurrent work, Time-LLM (Jin et al., 2024), repurposes LLMs for time series Torecasting by aligning embeddings of time series patches with text prototypes, and prompting the (frozen) LLM with these aligned embeddings and natural language prefix describing the task. Unlike CHRONOS, both GPT4TS and Time- LLM require in-domain training or fine-tuning, i.e., they are fine-tuned and tested on each dataset separately. Furthermore, the aforementioned methods are based on prompting or fine-tuning pretrained LLMss Tn contrast, CURONOS trains language models from scratch on a large collection of time series, tokenized via scaling and quantization.
Zero-shot forecasting. Zero-shot forecasting is the ability of models to generate forecasts for time scries from unseen datasets. Some early work (Orozco & Roberts, 2020; Oreshkin et al., 2021; Jin et al., 2022) in zero-shot forecasting considers training on a single time series dataset and testing on a different dataset. ForecastPFN (Dooley et al., 2023) tackles the problem of zero-shot forecasting by training transformer- based model purely synthetic data generated according to predefined trend, seasonalities (daily, monthly, yearly). The trained transformer model is then used to forecast real-world time series in a zero-shot setting, In this work, we also propose method to generate synthetic time series data from Gaussian processes (Section 4.2); however, we use the synthetic data in combination with real data to train CHRONOS models, which improves the overall zero-shot performance. Furthermore, CHRONOS models are probabilistic, whereas Forecast can only generate point forecasts.
Recent concurrent works (Rasul et al., 2023; Goswami et al., 2024; Das et al., 2023; Woo al., 2024) also develop zero-shol forecasting models by pretraining transformer-basee architectures on a large corpus of time series data. These works operate on the real values of the time series and include time-series- specific designs such as time features, lags, patching, and real-valued distribution heads, among others. In contrast, CHRONOS follows a minimalist approach by tokenizing time series values into a fixed vocabulary and training existing language model architectures on these tokens without any time-series-ppecific design or features. That is, CURONOS uses a categorical distribution to model the observations, performing regression via classification.
Other time series tasks. Similar to Zhou et al. (2023a), recent works have studied general purpose models applicable across time series tasks including imputation, forecasting, classification and anomaly detection. Wu et al. (2023) develop task-generic backbone based on the Inception model (Szegedy et al., 2015). In order to use the CNN-based Inception model, one dimensional time series is transformed into a two dimensional image-like representation by essentially sogmenting the time scries based on the periodicity and stacking the segments. SimMTM (Dong et al., 2023) is a masked pretraining framework for time series which learns general time series representations that are then used for forecasting and classification via fino-tuning. Although we focus on univariate time series forecasting in this work, based on its excellent performance on unseen time series datasets, we hypothesize that CHrONOS learns general representations that can potentially be deployed for tasks beyond forecasting.
4

---

Published in Transactions on Machine Learning Research (10/2024)
3 Chronos: A Language Modeling Framework for Time Series
In this section we introduce CHRONOS, a framework adapting existing language model architectures and training procedures to probabilistic time series forecasting. While both language and time series are sequen- tial in nature, they differ in terms of their representation - natural language consists of words from a finite vocabulary, while time series are real-valued. This distinction necessitates specific modifications to existing language modcling frameworks, espocially concerning tokenization, to make them applicable to time series data. Nevertheless, since existing transformer models have excelled on language tasks, our design philosophy involves making minimal changes to the model architectures and training procedure.
3.1 Time Series Tokenization
where the first C time steps constitute the historical context, and the remaining H represent the forecast horizon. Language models operate on tokens from a finite vocabulary, so using them for time series data requires mapping the observations x; E R to a finite set of tokens. To this end, we first scale and then quantize observations into a fixed number of bins.
Consider time series D1.C+ =
Scaling. The scale of time series can differ significantly even within a single dataset. This poses optimiza- tion challenges for deep learning models. Therefore, individual time series are normalized to facilitate better optimization. In the case of CHRONOS, the goal of normalization is to map the time series values into a common normalization technique involves applying an affine transforma- tion to the time series, i.c., & = (z. ms/s. Several popular normalization schemes, such as mean scaling, standard scaling and min-max scaling, can be obtained by appropriately choosing m and s. We opt for mean scaling, a method that has proven effective in deep lcarning models commonly used for practical time series applications (Salinas et al., 2020; Rabanser et al., 2020), but other approaches are viable and only require minimal changes. An attractive feature of mean scaling is that it preserves zero values in the time series, which are often semantically meaningful, such as zero sales for a product or zero solar energy generation at night. Mean scaling normalizes individual entries of the time series by the mean of the absolute values in the historical context. Specifically, this involves setting m and & = C Ei= [ry]-
suitable range for quantization.
The scaled time series 01.+H "f............. #c+H #c+#.. is still real-valued and cannot be processed directly by language models. To convert these real values into discrete tokens, we employ quantization. Formally, we select B bin centers C1 <... < cв on the real line, and B - 1 edges b, separating them, G < b; < G+1, for i E (1...., B - 1). The quantization function R - (1, .2...,,], B), and dequantization d: 21. B} + R, are then defined as
Quantization.
- x < x <1. 2 if 5 b2,
(xx)=
and d(j) Cj.
(1)
B if bb- S
respectively. The positioning of bin centers and edges can either be data-dependent or uniform (Rabanser et al., 2020). Quantile binning, a type of data-dependent binning, exploits the cumulative distribution function (CDF) of the training datapoints to construct bins such that approximately equal number of dat- apoints are assigned to each bin. In contrast, uniform binning selects uniformly-spaced bin centers within the interval [c, CB] and the bin odges fall mid-way between the successive bin centers, i.c., b, = 2 E (1...., B 1}. Since the distribution of values for unseen downstream datasets can differ significantly from the training distribution, niques can be used. We refer the reader to Rabanser et al. (2020) for detailed discussion on quantization schemes for time series. A potential limitation of this approach is that the prediction range is restricted between [ci, en), making it theorelically infeasible to model time series with a strong trend. We explore this further in a practical setting in Section 5.7.
4 for
opt for uniform binning in our experiments, but other quantization tech-
Apart from the time series tokens {1, .......,. we include two special tokens, commonly used in language models, into the time series vocabulary, Ve.: PAD and EOS. The PAD token is used to pad time series of different
5

---

Published in Transactions on Machine Learning Research (10/2024)
lengths to a fixed length for batch construction and to replace missing values. The EOS token is appended to the quantized and padded time series to denote the end of the sequence. While the use of an EOS token is not strictly necessary in the case of time series, it makes training and inference using popular language modeling librarics convenient. The sequences of tokens from Vis can readily be processed by language models (both encoder-decoder and decoder only models), to train them as usual. A common approach in time series modeling is to incorporate time and frequency information, through features such as day-of-week, week- of-year, and so on. Perhaps counter-intuitively, in CIRONOS, we ignore time and frequency information, treating the "time series" simply sequence.
We primarily focus on the variants of the encodrrddcoddr T5 model (Raffel et al., 2020). Additionally, we conduct an experiment with the GPT-2 (Radford et al., 2019) model to demonstrate that our approach can be straightforwardly extended to decoder-only models. No modifications are required to the language model architecture, except adjusting the vocabulary size to [Vill which depends on the number of bins used for quantization and may be different from the vocabulary size of the original language model. Coneretely, adjusting the vocabulary size entails truncating (or extending) the input and output embedding layers of the language model.
3.2 Objective Function
As typical in language models, we use the categorical distribution over the elements of Ves as the output distribution, p(zc+++1 (z).(++))) where 21.4+ is the tokenized time series. CURONNO is trained to minimize the cross entropy between the distribution of the quantized ground truth label and the predicted distribution. Formally, the loss function for a single tokenized time scries (also accounting for EOS tokens) is given by,
#+1 |Pul
((0) = - --0=-EE 1 (0044-1=1) logp0 = i121044),
(2)
where po(=0+*+1 = i(21.0++) denotes the categorical distribution predicted by the model parameterized by 0. In practice, the loss is averaged over a batch of time series during training.
Note that the categorical cross entropy loss (Eq. 2) is not a distance-aware objective function, i.c., it does not explicitly recognize that bin is closer to bin i 1 than to i Instead, the model is expected to associate nearby bins together, based on the distribution of bin indices in the training dataset. In other words, CURONOS performs regression via classification (Torgo & Gama, 1997; Stewart et al., 2023). This is unlike typical probabilistic time series forecasting models, which either parametric continuous distributions such as Gaussian and Student's: s-t (Salinas et al., 2020) or perform quantile regression (Wen et al., 2017; Lim et al., 2021).
Opting for a categorical output distribution offers two key advantages. Firstly, it requires no modification to the language model architccture or training objective, enabling the use of popular language modeling libraries and the utilities they provide out of the box (Wolf et al., 2020). Secondly, it imposes no restrictions the structure of the output distribution, allowing the model to learn arbitrary distributions, including multimodal ones. This flexibility proves especially valuable for a pretrained model, as time series datasets from diverse domains may follow distinct output distribution patterns.
Arguably, modeling the output as an ordinal variable would be more appropriate, since the output domain is obtained by discretizing the real line. In fact, regression models for ordinal variables have been extensively studiod in the literature (McCullagh, 1980; Winship & Marc, 1984), including for neural networks and transformer models (Cheng et al., 2008; Hu et al., 2021). Imposing the ordinal nature of the classes on top of the models, in similar ways to the mentioned literature, could be an interesting extension of this work.
3.3 Forecasting
CHRONOS models are probabilistic by design and multiple realizations of the future can be obtained by autoregressively sampling from the predicted distribution, (itcc--+l221c4 for h E f1,2...., H). These sample paths come in the form of token IDs that need to be mapped back to real values and then unscaled
6

---

Published in Transactions on Machine Learning Research (10/2024)
to obtain the actual forecast. The dequantization function d from Eq. (1) maps the predicted tokens to real values: these are then unscaled by applying the inverse scaling transformation, which in the case of mean scaling involves multiplying the values by the scale s.
4 Data Augmentation
The quality and quantity of public time series data pales in comparison to the natural language processing (NLP) domain, which benefits from ample high-quality text datasets such as WikiTeext-103 (Merity et al., 2016), C4 (Raffel et al., 2020), and The Pile (Gao et al., 2020). This poses challenges for training models intended for zero-shot forecasting, which rely on large-scale time series data with diverse patterns. To address this issue, we propose enhancing the diversity of training data by generating mixup augmentations from real datasets and supplementing training with synthetic data.
4.1 TSMixup: Time Series Mixup
Mixup (Zhang et al., 2017) is a data augmentation scheme proposed in the context of image classi- fication. It generates convex combinations of random image pairs and their labels from the training dataset, which alleviates issues such as memorization and overfitting in deep learning models. Exist- ing works (Carmona et al., 2021; Zhou et al., 2023b) have extended Mixup to the time series domain. Building upon these works, propose TSMixup, which gen- eralizes the idea of Mixup to more than two datapoints. Con- cretely, TSMixup randomly samples k ~ [11, K} time series of a specific length, I ~ (flmin /max), , from the training datasets, scales them, and lakes their convex combination.
Original Time Series
TSMixup Augmentations
==1 V MMMMMM
DG
2
TSMixup T 1:1
A5 1:11
(3)
D.A
MWM MWMWMMM HM
Aa 0.4
where (() denotes the i-th scaled time series. The time series are scaled before mixing to ensure that time series with small and large values are given equal importance in the mixing pro- cess. The combination weights, i Au], are sampled from symmetric Dirichlet distribution, Dir(a), parameterized by the scalar concentration parameter The complete pseudocode of TSMixup can be found in Algorithm 1 in Appendix A. Intu- itively, TSMixup enhances the diversity of data by combining patterns from different time series. Figure/ 2 shows example augmentations generated by TSMixup and illustrates how different patterns are mixed.
3
Ag
Mwwmmmmw
Figure 2: An illustration of TSMixup angmen- tation for & = {1, 2,3. TSMixup improves pattern diversity by taking weighted combina- tions of randomly-sampled time series from dif- ferent datasets.
4.2 KernelSynth: Synthetic Data Generation using Gaussian Processes
While TSMixup improves pattern diversity, it may still prove insufficient for training a generalist time series model, especially when real data is limited. To further supplement the training dataset, we propose KernelSynth, a method to generate synthetic time series using Gaussian processes (GPs). KernelSynth is inspired by the Automatic Statistician (Duvenaud et al., 2013), where a compositional scarch over space of GP kernels is performed to explain the structure of a time series. We use the inverse of this process randomly compose GP kernels to generate new time series.
GPs are distributions over functions defined by the mean function, m(t), and the positive definite kernel, <(1, (1), where 1 E R is the domain. The kernel specifies a covariance function which defines the joint variability of the function values at an arbitrary pair of points, (t, (t,)'), in the input domain. Diverse patterns can be generated by appropriatcly selecting the kernel. We constructod a kernel bank, K, of basis kernels defining fundamental time series patterns. These include linear kernels for trend, RBF kernels for smooth local variation, and periodic kernels for seasonalities found in typical time series frequencies. The final kernel,
7

---

Published in Transactions on Machine Learning Research (10/2024)
Kernel Bank
Linear
Linear
sample
Linear
Linsar
[Linear Linear)
Poriodic
Periodic
Porocic
(a) KernelSynth
(b) Synthetic samples from KernelSynth
(a) An illustration of Kernelyynth, a Gaussian process (GP)-based synthetic time series generation method. Kernels are sampled from a kernel bank and then randomly combined using binary operator (x or +). The resultant kernel is used in GP prior to generate synthetic time series. Random samples from kernels at each step are shown in red and blue colors. (b) Example synthetic time series generated by KernelSyuth.
Figure 3:
R(t,t') is constructed by sampling j ~ J] kernels from K. with replacement and combining these kernels via random binary operations, + or x. A synthetic time series is generated by drawing a sample of length lyn from the GP prior, (P(m(t) 0, t(t, see Algorithm in Appendix A for details. Figure 3 depicts this generative process used in KernelSynth, illustrating how time series with intricate patterns arise from the composition of simple basis kernels.
5 Experiments
In this section, we present empirical results commonly used benchmark datasets. First, we give overview of the datasets, training strategy, baselines, and evaluation metrics (Section 5.1-5.4). Table 1 provides a high-level summary of the datasets and baselines used in our experiments. We then (a) evaluate the performance of CHRONOS models in the in-domain and zero-shot settings against local models and task-specific deep learning models (Section 5.5); (b) analyze the effect of various design choices such as model size, initialization, synthotic data proportion, context length, and vocabulary size on the performance of CHRONOS models (Section 5.6); and (c) analyze the qualitative performance of CHRONNS models and highlight their limitations (Section 5.7). We discuss our key findings in this section and relegate specific experiment details to the appendices.
Table 1: A high-level summary of the datasets and baselines used in our experiments.
Data Subset
# Datasets
# Series
Usage
Baselines
Pretraining-only
13 795,936
pretraining
Benclmark
15
97,272
pretraining and in- domain evaluation
Naive, SeasonalNaive, AutoETS, Auto- Theta, SCUM, AutoARIMA, DeepAR, TFT, PStchTST, DLincar, WaveNet, N-BEATS, N-HITS, GPT4TS, Lag-Llama, Moirai-1.0-R All the above, LLMTime and ForecastPFN
Benchmark II
27
190,674
zero-shot evalustion
5.1 Datasets
To train and evaluate CHRONOS models, we collected wide varioty of publicly available datasots spanning various application domains including energy, transport, healtheare, retail, web, weather, finance, and with sampling frequencies ranging from minutes up to yearly. The complete list of datasets, together with their
8

---

Published in Transactions on Machine Learning Research (10/2024)
respective sources and additional details, is given in Appendix B. In total, our dataset collection comprises 55 datasets from multiple sources, including the Monash Time Series Forecasting Repository (Godahewa et al., 2021), the M-competitions (Makridakis et al., 1979; Makridakis & Hibon, 2000; Makridakis et al., 2020; 2022), and public domain datasets from Kaggle.'
We categorize this collection into three subsets, based on how we use them for training and evaluating CIRONOS models: (a) datasets exclusively used for training (13 datasets); (b) Benchmark datasets, em- ployed for both training and evaluation, representing in-domaín evaluation (15 datasets); and (c) Bench- mark II datasets, used solely for evaluation, constituting zero-shot evaluation (27 datasets). In categorizing the datascts in this way, we tried to find good balance between keeping as many datasets as possible for the zero-shot evaluation of CIROOOS models, among the ones most commonly used in the literature, while still having enough variety of domains and sampling frequencies in the training data. Overall, we used 28 datasets for training CHRONOS models, consisting of about 890K univariate time series with approximately 84B observations (tokens) in total. For both in-domain (I) and zero-shot (II) benchmark datasets, we used the last H E N+ observations of each time series as held-out lest set: all models are judged by the accuracy of their forecast on such held-out set, which model had access to for training purposes. The prediction length H is task-specific (see Table 3 in Appendix B), where we define a task as a datasct and prodiction length pair. Tasks in both benchmarks exhibit diverse properties, in terms of the dataset size, frequency, history length, and prediction length, making them rich benchmarks reflective of real world scenarios.
5.2 Training Corpus and Protocols
We selected T5 (Raffel et al., 2020) as the main architecture for CHRONOS in our experiments, since it is available in a variety of sizes, ranging from 16M (Tiny) to 11B (XXL) parameters (Tay et al., 2021). We also conducted experiments with the decoder-only GPT-2 model to demonstrate the applicability of the CHRONOS framework to decoder-only models. In the following, we discuss the training configurations used for our main results (Section 5.5) and explore alternatives for some of the hyperparameters in Section 5.6.
We trained T5 models of 4 sizes, 2 namely, Mini (20M), Small (46M), Base (200M) and Large (710M), and the GPT-2 base model (90M), on 10M TSMixup augmentations (see Section 4.1) generated from the 28 training datasets, with K = 3 in Algorithm 1, and 1M synthetic time series generated using Gaussian processes (see Section 4.2). Note that with this setup, original time series are adequately represented since they are included in the TSMixup augmentations with probability 1/3. We sampled time series from the augmentations and synthetic data in the ratio 9:1 during training. Each model is trained with an effective batch size of 256 sequences, using distributed data parallelism and gradient accumulation, whenever necessary. These sequences were constructed by slicing random windows from the time series, and then scaling and quantizing them into equal-sized bins within the interval [c|=- - 15, cø= + 15], as described in Section 3.1. We set the vocabulary size, Vis, to 4096, including the special tokens (PAD and EOS). The context length of the sequences was set to 512, the default for T5 models, and the prediction length was set to 64, a value greater than the prediction lengths of all tasks we consider in our evaluation.
The models were optimized for 200K steps using the AdamW optimizer with a weight decay of 0.01. The learning rate was annealed linearly from its initial value of 0.001 to O over the training steps. The other model and training hyperparameters were set to their defaults used in the transformers library (Wolf et al., 2020). We used an AWS EC2 instance with 8 A100 (40GB) GPUs to train all CHRONOS models, and we employed faster floating point formats (TF32) and model compilation to speed up training. Table 6 in Appendix E reports the training time and the approximate cost. of training CHRONOS models of different sizcs.
5.3 Baselines
We assessed the performance of CHRONOS models against a variety of time scries forecasting basclines. From statistical forecasting literature (Hyndman & Athanasopoulos, 2018), we included Naive, Seasonal Naive, AutoETS, AutoARIMA (Hyndman et al., 2008), AutoTheta (Assimakopoulos & Nikolopoulos, 2000) and a
The datasets used in our experiments are available httpss///gggggggeeecooo/ttst///s htt.///tttet///tttglla////nooooa
2 'Our code and model checkpaints are available https:/ httess///tttttt com/ amazon whtpee//cccooooooooeeceeeee
9

---

Published in Transactions on Machine Learning Research (10/2024)
strong ensemble (SCUM) of statistical models (Petropoulos & Svetunkov, 2020). Additionally, we compared against several neural forecasting baselines, including WaveNet (Oord et al., 2016), DeepAR (Salinas et al., 2020), N-BEATS (Oreshkin et al., 2020), TFT (Lim et al., 2021), DLinear (Zeng et al., 2023), PatchTST (Nie et al., 2023), N-HiTS (Challu et al., 2023), and GPT4TS (Zhou et al., 2023a). Furthermore, from the recently proposed pretrained time series models, we included the ones with publicly available weights: Lag- Llama (Rasul et al., 2023) and Moirai-1.0-R (Woo et al., 2024). On Benchmark II (i..., zero-shot datasets for CIIRONOS models), we also evaluated against two zero-shot methods: ForecastPFN (Dooley et al., 2023) which is transformer model pretrained only on synthetic time series data and LLMTime (Gruver et al., 2023) which uses LLMs for zero-shot forecasting.
We categorize CURONOS models and the baselines into three groups: local models that estimate parameters for each time series individually; task-specific models trained or fine-tuned for each task separately; and pretrained models which do not perform task-specific training, instead using a single model across all tasks. Further details on the implementation and training of these basclines can be found in Appendix C.
5.4 Evaluation Metrics
Whenever possible, 3 we evaluated models both in terms of their probabilistic and point forecast performance. We used the weighted quantile loss (WQL) to assess the quality of the probabilistic forecasts: the WQL is related to the continuous ranked probability score (CRPS, Gneiting & Raftery (2007))* and is commonly used to evaluate probabilistic forecasts (Gasthaus et al., 2019; Shchur et al., 2023). The WQL measures the compatibility between the predictive distribution and the ground-truth observation at a uniformly- spaced grid of quantile levels; we compute the WQL on 9 uniformly-spaced quantile levels {0.1, 0.,. 0.9}. Quantile forecasters such as TFT were directly trained on these quantile levels. For methods requiring sampling, we estimated the quantiles using 20 sample forecast paths. We used the mean absolute scaled error (MASE, Hyndman & Koehler (2006)) to evaluate the point forecast performance. The MASE is defined as the absolute error of the forecast scaled by the historical seasonal error of the time series, and was selected due to its favorable properties over other point forecasting metries (Hyndman & Kochler, 2006). We used the median forecast (0.5-quantile) for computing the MASE for the probabilistic forecasters. See Appendix D for detailed discussion on the evaluation metrics.
Since the magnitude of the evaluation metrics can vary across datasets, we adopt a different approach to aggregate scores than naive averaging. For each dataset, we compute the relative score of each model as the model's score divided by the score of a bascline model (here, Scasonal Naive). The relative scores are aggregated across all datasets using the geometrie mean. The choice of the geometric mean is deliberate Fleming& & Wallace (1986) show that the arithmetic mean can yield misleading conclusions in this context,
and the geometric mean is provably the only meaningful way to aggregate such relative scores. Furthermore, the geometric mean is also not sensitive to the choice of the baseline, and the model ordering stays intact if another baseline is selected instead. We used Seasonal Naive due to its simplicity and popularity as a forecasting baseline. For models that failed or could not finish evaluation within the allotted time on certain datasots, we used a relative score of 1, i.c., the baseline relative score, when aggregating the results. We assign equal weights to all tasks during aggregation, reflecting real-world scenarios where datasets may have different numbers of time series, frequencies, history and prediction lengths.
5.5 Main Results
In this section, we present our main results on 42 datasots, which comprisc Benchmark I (15 datasots) and Benchmark II (27 datasets). CHRONOS models surpass classical statistical baselines, task-specific deep learning models, and other pretrained models on the in-domain datasets (Benchmark I; see Section 5.5.1). On the zero-shot datasets (Benchmark II; Section 5.5.2), CHRONOS models comfortably outperform statistical baselines and other pretrained models, while performing on par with the best deep learning models trained on these tasks. With an inexpensive fine-tuning regimen, our CHRONOS-T5 (Small) model achieves the top spot on Benchmark 11, significantly outperforming all baselines.
Some models (GPT4TS and ForecastPFN) only generate point forecasts and we only evaluate those.
4 Many existing works (Ansari et. al., 2021; Rasul et, al., 2023; Kollovieh al., 2023) use CRPS and WQL synonymously.
10

---

Published in Transactions on Machine Learning Research (10/2024)
5.5.1
Benchmark I: In-domain Results
Local
Spectfic Models
Protr Domain)
(Other)
Chronos-T5 (Large)
Chronos- (Large)
Chronos-TS (Dase)
0,580
Chrones-T5 TS (Base)
0.706
Chronos-T8 T5 (Min)
(Small)
PatchTST
0.601
Chronos-T3 (Min)
0.732
Chronas-T5 T5 (Small)
0.603
PatohTST
0.740
Chronos-GPT2
0.623
Chronos-GPT
0.741
N-HITS
0.656
Moiral-1.0 (Large)
0.806
0.664
DeepAR
Moiral-1.0 1.0 (Large)
0.670
W WeveNet
0.842
Model
DeepAR
Model
WaveNet
0.689
Molral- 1.0 (Base)
BEATS
0.857
Motral-1.0 (Base)
DLinear
0.864
DLinear
0.697
GPT4TS
0.471
TFT
0.734
0.939
0,876
SCUM
0.940
Lag-Uama
0.937
AutoARIMA
Seasonal Natvo
AutoETS
0,983
SCUM
1.060
1.000
AutoETS
AutoTheta
1.129
AutoTheta
1.083
Lag
Natvo
1.433
Naive
1.404
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1.6
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1.6
Agg. Relative WQL
Agg. Relative MASE
Figure 4: Performance of different models on Benchmark I, comprising 15 datasets also included in the training data of CHRONOS models. This benchmark showcases the in-domain performance of CHRONOS models against local statistical models, which fit parameters individually for each time series, task-specific models that train separate model for each task, and pretrained models trained on a large corpus time series data. Pretrained Models (Other) indicates that the in-domain setting does not apply to these models they CHRONOS. Specifically, this means that some datasets in Benchmark 1 were not part of their training corpus and (or) they were trained on the test sets of some datasets in Benchmark The probabilistic (WQL) and point (MASE) forecasting metrics (lower is better) are normalized using the scores of the Seasonal Naive baseline and aggregated through geometric mean obtain the aggregated relative WQL and MASE, respectively. Results for CHRONOS and task-specific models (except GPT4TS) have been averaged over random seeds. Models producing point-forecasts (GPT4TS) are only compared based on MASE.
trained different corpora than
Benchmark I comprises 15 datasets that also part of the training data of CHRONOS models, i.e., this benchmark evaluates the in-domain performance of CHRONOS models (sce Table 3). Figure 4 summarizes the probabilistic and point forecasting performance for all models on the held-out test windows, in terms of their aggregated relative scores, computed as described in Section 5.4. The bigger CHRONOS-TV models (Base and Large) significantly outperform bascline models, obtaining the best aggregated relative scores and average ranks (Figure 18 in Appendix E). These models not only perform better than local models (e.g., AutoETS and ARMOARIMA), but they also perform better than task-specific deep learning models trained or fine-tuned for each dataset (e.g., PSt and DeepAR) and other pretrained models (e.g., Lag- Llama and Moirai-L.0-R).
The smaller CHRONOS-T5 models (Mini and Small) and CHRONOS- GPT2 also perform better than the ma- jority of baselines. Between the two baseline pretrained models studied in this experiment, Moirai-1.0-P clearly outperforms Lag- Llama. Notably, the best Moirai-1.0-R model (Large, 311M) is still outperformed by the smallest Chronos-T5, model (Mini, 20M) even though Moirai-1.0-R models were trained on signif- icantly larger corpus of time series data. Task-specific deep learning models, trained across multiple time series for a specific task, perform better than local statistical models that fit parameters for each time series. Interestingly, the Seasonal Naive baseline performs competitively against other local models this bench- mark, suggesting that the datasets in this benchmark exhibit strong scasonal patterns. This is unsurprising since a majority of these datasets belong to domains such as energy and transport that tend to be highly seasonal in nature. The raw WQL and MASE values for individual datasets summarized in Figure 4 can be found in Tables and in Appendix E.
These results demonstrate the benefit of using models that are trained only once across multiple datasets, over fask-specific models trained individually for each task. Such models could streamline production forecasting
11

---

Published in Transactions on Machine Learning Research (10/2024)
systems, where forecasts from different time scries tasks required, by obviating the need for training separate models for each task.
5.5.2
Benchmark II: Zero-shot Results
Local Models
Task Specifii Models
Pretrained Models (Zero Shot)
Pretrained Models (Other)
TFT
0.659
PatchTST
0.810
Chronos- (Large)
(Large)
0.823
Thronos-TS (Base)
0.662
N-ITS
Chronos-T5
0.667
Chronos-TS (Bass)
0.832
N-iTS
0.672
N-BEATS
0.885
Chronos-TS (Min)
0.678
SCUM
BEATS
Chronas-TS (Small)
0.841
PatchTST
0.684
0.843
Chrones-GPT2
TFT
0.847
Model
Moirai-1.0. 1.0-R (Basa)
0.696
Model
Chrones-TS (Minl)
0.850
(Large)
AutoTheta
0.875
SCUM
0.728
Moiral-1.0 ,0 (Large)
0.876
0.733
DLinear
DLinear
0.757
GPT4TS
AstoARIMA
0.761
Moirai-1.0 (Base)
0.907
AutoTheta
AutoARIMA
0.908
LLMTime
0.804
WaveNet
AutoETS
AutoETS
0.953
WaveNet
0.842
0.962
Naive
Seasonal Nalve
Lag Liama
1.097
Naive
1.188
Naivo
1.152
Lag-UJama
1.291
Agg. Relative WQL
Agg. Relative MASE
Figure Performance of different models on Benchmark II, comprising 27 datasets not seen by CHRONOS models during training. This benchmark provides insights into the zero-shot performance of CHRONOS models against local statistical models, which fit parameters individually for each time series, task-specific models trained on each task, and pretrained models trained zero-shot setting does not apply to these models as they were pretrained on some datasets in Benchmark 11. The prohabilistic (WQL) and point (MASE) forecasting metrics (lower is better) the Seasonal Naive baseline and aggregated through a geometric mean to obtain the aggregated relative WQL and MASE, respectively. Results for CHRONOS and task-specific models (except GPT4TS) have been averaged over random seeds. Models producing foint-forecasts (GPT4TS and ForecastPPN) are only compared based on MASE.
large corpus of time series data. Pretrained Models (Other) indicates that the
normalized using the
of
Benchmark II consists of 27 datasets that were not used during CHRONOS models' training (see Table 3 in appendix B), ie., this benchmark evaluates the zero-shot performance of these models. These datasets belong to diverse domains and frequencies, some of which are not even part of the training data, making this challenging benchmark for CHRONOS. Figure 5 summarizes the results Benchmark II in terms of the aggregated relative scores. This benchmark is clearly more challenging than Benchmark (Figure 4), the best models tend to offer lower improvements relative to the bascline.
Nevertheless, despite never having seen these dalasets during training, CHRONOS models significantly outper- form standalone local statistical models. On probabilistic forecasting (aggregate relative WQL), CHRONOS models achieve the 2nd to 4th spots, performing better than most task-specific models that have been trained on these lasks. In terms of the point forecasting performance, CuRONOS- T5 (Large) places 2 surpass- ing most baselines, including the strong SCUM ensemble. CHRONOS models also significantly outperform other pretrained models such Moirai-1.0-R. Lag-Llama, LLMTime, and ForecastPFN, and even GPT4TS, which fine-tunes a pretrained GPT-2 model on each dataset. Moirai-1.0-F obtains the best performance after Chronos, although the evaluation setup may have been advantageous for Moirai-1.044 1.0-R as many datasets in Benchmark II were part of its pretraining corpus. The raw WQL and MASE values for individual datasets summarized in Figure can be found in Tables and 10 in Appendix E.
From rigorous standpoint, to prevent information leakage, the start time of any dataset within this category must be after the timestamp of the last. observation from the pretraining dataset and Benchmark Nevertheless, consider the risk minimal given that the datsets bear overlap beyond high-level conceptual categorization.
12

---

Published in Transactions on Machine Learning Research (10/2024)
The results this benchmark highlight the promise of CHRONOS generalist time series forecaster it performs significantly better than local models that are commonly used in a zero-shot setting, and it performs on par with the best task-specific deep learning models.
Fine tuning. Motivated by the remarkable zero-shot perfor- mance of CHRONOS models, we conducted a preliminary in- vestigation into fine-tuning CHRONOS models individually on datasets from Benchmark II.
Chronos-T5 (Small)
Zero Shot
Fine Tuned
Benchmark
0.667
WQL
0.597
We selected the CURONOS-T3 (Small) model for this experi- ment due to its good zero-shot performance with a relatively low training cost. We fine-tuned the model in a dataset- agnostic fashion with an initial learning rate of 0.001, annealed linearly to over 1000 steps. Figure shows that fine-tuning significantly improves the aggregate performance of the model Benchmark II. The fine-tuned CHRONOS-T5 (Small) model now takes the top spot Benchmark II overall, overtaking both larger (zero shot) CHRONOS models and the best lask- specific models. Notably, CHRONOS-T5 (Small) is not even the most accurate variant of CHRONOS on Benchmark II in the zero shot setting, suggesting that further improvements may be obtained by fine-tuning larger CHRONOS-T5 variants.
0.841
MASE
0.760
0.0
0.2
0.4
0.6
0.8
Agg. Relative Score
Figure When fine-tuned on individual datasets from Benchmark II, CIRONOS-T5 (Small) significantly improves over the zero- shot performance and becomes the best per- forming model on average (see Figure 5).
5.6 Analysis of Hyperparameters
Here, we explore the effect of different design choices on the downstream model performance, beginning with a comparison of different model sizes and initializations. We then analyze the effect of training steps, synthetic data proportion, context length, and vocabulary size, on the performance of CHRONOS- T5 (Small). We only vary the parameter of interest, keeping everything else fixed to the value used in the main results.
Model size. We experimented with four model sizes ranging from 20M to 710M parameters. 6 Unsurpris- ingly, the training loss improves with the model capacity, shown in Figure 7a. We also observe this trend in the downstream model performance benchmarks, as shown in Figure 7b. These trends suggest that even larger models may improve performance further. However, we did not explore larger models due to slow inference times which would render them impractical for rcal-world applications.
it improves with the model size for both in-domain and zero-shot
Initialization. We investigated whether initializing CHRONOS models to the corresponding T5 language models pretrained by Tay et al. (2021) on the C4 dataset (Raffel et al., 2020) has any impact on the training dynamics or the downstream performance. Figure 8 shows the training loss curve for models initialized randomly and those initialized with language model weights. Notably, models initialized randomly tend to converge to a lower training loss compared to their counterparts initialized with language model weights. For the larger models (Base and Large), models initialized with language model weights initially exhibit a faster decrease in training loss, but they ultimately converge a higher final loss.
Overall, these observations suggest that language model weights not particularly remarkable in the context of time series forecasting and offer no improvement over random initialization. These conclusions are further reinforced by Figure 9 which shows the downstream performance of models initialized with language model weights against three randomly-initialized models of each size. Across all model sizes, the performance of models initialized with language model weights either overlaps with or slightly underperforms compared to randomly initialized models. These results suggest that LLM initialization offers relatively little advantage in the context of time series forecasting, and instcad random initialization may be the preferable choice.
6These numbers differ from the original sixes of the models in Tay al. (2021) due to the change the vocabulary size.
13

---

Published in Transactions on Machine Learning Research (10/2024)
Benchmark (Metric)
In Domain (WQL)
Domain (MASE)
Zero Shot (WQL)
Zero (MASE)
4.4
Chronos-T5 (Mini)
0.95
Chronos-T5 (Small)
Agg. Relative Score
0.90
Training Loss
Chronos-T5 (Base)
Chronoe-T5 (Large)
0.80
0.75
0.70
3.6
0.65
3.4
3.2
0.55
0
50K
100K
150K
200K
20M
200M
710M
Training Step
Model Size
(a)
(b)
Figure 7: Model size. (a) Training loss curves of CHRONOS models of different sizes. (b) In-domain and zero-shot performance of CHRONOS models varying over model size (lower better).
Chronos-Te T5 (Mini)
Chronos-T5 (Small)
Chronos- T5 (Base)
Chronos-T5 (Large)
Training Loas
Inibalization
Initalization
Initalization
Inkialization
4.0
Language Model
Language
Language Model
Langunge Model
50K
150K
50K Training Step
50K
150K
50K Training Step
Training Step
Training Step
Figure 8: Initialization. Comparison of training loss of randomly-initializzed CHRONOS models of different sizes against. those initialized with language model weights.
Benchmark (Metric)
Domain (WQL)
Domain (MASE)
TSMixup augmentations. As described in Section 5.2, we trained CHRONOS models on TSMixup augmentations rather than directly on the original time series. In this experiment,
Zero Shot (WQL)
Zero Shot (MASE)
0.95
@ Random Init.
* Language Model Init.
0.90
we investigate whether using TSMixup augmentations is ad- vantageous for downstream performance. Figure 10a compares the performance of CHRONOS-T3 (Small, 46M) models trained with and without TSMixup augmentations. The model trained on TSMixup augmentations obtains similar in-domain perfor- mance to the model trained without augmentations. However, the zero-shot performance improves when using TSMixup aug- mentations. This suggests that TSMixup enchances the di- versity of training data which leads to improved performance on unseen datasets. Figure 10a also shows that the zero-shot performance obtains an additional boost with the inclusion of synthetic data. We investigate this further in the next experi- ment.
Agg. Relative Score
0.85
0.80
0.70
0.65
0.60
0.55
Model Size
Figure Comparison of the in-domain and zero-shot performance (lower is better) of mod- els initialized with language model weights (marked as star) and three randomly initial- ized models (marked as circles) across different model sizes.
Synthetic data proportion. We systematically explored the impact of KernelSynth on downstream model performance. We trained CURONOS-TS (Small, 46M) models with time series sampled from TSMixup augmentations and KernelSynth data in different ratios, ranging from 0% (i.e., trained solely on TSMixup augmentations) to 100% synthetic data.
Figure 10b shows the performance of models trained with different proportions of synthetic data. Both in-domain and zero-shot metrics improve with the incorporation of synthetic data in training. The most
14

---

Published in Transactions on Machine Learning Research (10/2024)
Benchmark (Metic)
TBMixup Syrth
TeMixup Only
(MASE)
90% T5Moup e 18% Syrths
Zero Shot (VOl)
Zero Shot (MASE)
Domain
0.513
0.003
Relative Score
Benchmark
Zore Shot (WOL)
0.700
0.735
0.80
0.75
0:738
0.727
Zero Shot (MASE)
0.007
0.60
0.6
20
Agg. Relative Score
% of Synthetic Data
(a)
(b)
Figure 10: (a) Comparison in-domain and zero-shot performance of CHRONOS-T5 (Small) models trained with and without TSMixup augienttttions. (b) In-domain and zero-shot performance of CHRONOS-T5 (Small) models with varying proportion KernelSynth data in the training corpus.
consistent improvement is observed around the 10% synthetic data proportion. Further increasing the proportion of synthetic data lends to worsen performance. This is unsurprising since the synthetic data generated using Gaussian processes is not representative of all real-world time series.
While the model trained only on synthetic data performs worse relative to models with real data in their training corpus, it performs reasonably well in terms of its absolute performance. Figure 20 (Appendix E) shows that performs significantly better than Forecast PFN (Dooley et al., 2023), another model that is trained solely on synthetic data (generated differently from KernelSynth). Surprisingly, it also outperforms several other baselines in our benchmarks," despite never having seen real data during training. These results attest the quality of our synthetic data, and they open up directions for future work to close the performance gap further.
Benchmark (Motric)
Benchmark (Merri)
Benchmark (Metric)
(WQL)
Zero Shat (WO1)
Zero Shot (MASEE
Zero Shot (WGL)
Zero Shot (AASE)
Zero Shot (WQL)
Zero Shot (MASE)
0.90
0.90
0.90
Agg. Relative Score
0.85
Agg. Relative Score
0.85
Relative Score
0.85
8.80
0.60
0.80
0.75
0.75
0.75
0.70
0,79
0.70
9.65
0.65
0.66
200K
4D0K
600K
512
Training Step
Context Length
Vocabulary Size
(a)
(c)
Figure 11: In-domain and zero-shot performance of CHRONOS-T5 (Small) models varying over (a) the number of training steps, (b) the training context length, and (c) the vocabulary size.
Training steps. We trained CHRONOS-TT (Small, 46M) for 1M training steps to study the effect of longer training on model performance. Figure 1la shows that the downstream model performance improves over the course of training, both on in-domain and zero-shot benchmarks. This suggests that performance of the larger models (Base and Large) can potentially be improved by training them for longer.
Context length. We studied the effect of the context length on downstream performance by training CHRONOS-T5 (Small, 46M) models with four distinct context lengths. Figure 11b shows how the performance varies with increasing context length. We observe improvements on both in-domain and zero-shot metrics
All benchmarks are zero-shot for this model, since was only trained synthetic data.
15

---

Published in Transactions on Machine Learning Research (10/2024)
context length increases up to 1024, showing that longer context helps the models to forecast better to certain degree. However, increasing the context length further tends to saturate worsen the performance, which may partly be due to limitation of our evaluation setup: it does not include enough high-frequency datascts (>= 15 min). Hence, further evaluation is required to conchusively study the impact of longer context. lengths. We posit that high-frequency datasets may benefit from a longer context, which may be necessary correctly capture the long-term scasonal patterns.
Vocabulary size. The vocabulary size governs the precision with which the model can process the scaled time scries. To explore its impact on performance, we trained CHRONOS-T3 (Small, 46M) models with varying vocabulary sizes. Figure 11c shows modest improvements in the point forecasting metric (MASE) the vocabulary size increases. In contrast, the WQL initially improves but deteriorates for larger vocabulary sizes. We hypothesize that this behavior is artifact of the chosen metrics. The MASE, which is invariant to the scale of individual series, is closely aligned to our training loss, which is also invariant to scale. Hence, MASE exhibits an improvement with increased precision, just as one expects for the training loss. Conversoly, WQL, a scale-dependent metric, does not correlate closely with the training loss and behaves less predictably as precision increases. See Appendix D for a discussion on the properties of these metrics. Beyond this experiment, we posit that selecting the vocabulary size the context of a model like CHRONOS would pose trade-off. A vocabulary that is too small would lead to poor forecasting accuracy due to large discretization errors; however, large vocabulary would lead to the bins being too fine, potentially leading to generalization errors due to fewer datapoints falling into each bin.
5.7
Qualitative Analysis and Limitations
Ground Truth
Medion Forecast
80% Interval
Ground
Medaa Forecast
interal
2.5
0.0
500
-2.5
300
500
100
0
100
200
300
400
500
600
100
200
300
400
500
600
(8))
(b)
Ground Truth
Medion Forecast
80% Interval
Ground
Media Forecast
Interal
100
200
300
400
500
600
100
200
300
400
500
600
(c)
(d)
Figure 12: Forecasts generated by CHRONOS-T5 (Base) syuthetically generated patterns. (a) Noise: CHRONOS generates reasonable forecasts for Gaussian noise with the 80% prediction interval matching the interval the underlying distribution (shown by the horizontal dashed blue line). (b) Trend: ChRONOS forecasts linear trend
(top) correctly but struggles with an exponential trend (bottom). (c) Seasonality: CHRONOS accurately models seasonal patterns of varying degrees of complexity (single seasonality at the top and three seasonalities the bottom). (d) Combined Patterns: CHRONOS forecasts time series generated by the additive (top) or multiplicative (bottom) combination of trend and seasonal patterns accurately.
16

---

Published in Transactions on Machine Learning Research (10/2024)
In this section, we analyze forecasts generated by CHRONOS models qualitatively, and we also highlight some limitations of our tokenization technique. We primarily focus on synthetically generated time series for a controlled analysis of different types of time series patterns. For example forecasts from real datasets, see Figures 22 to 24 in Appendix E.
L.I.D Noise. We generated time series comprised purely of Gaussian observations, (00,1) and (1000, 10), and used CHRONOS-T5 (Base) to forecast these. Figure 12a shows that CHRONOS generates plausible forecasts for such time series and the predicted 80% interval coincides with the ground truth 80% interval shown by the dashed blue lines.
Ground Truth
Medan Forecatt
60% Interval
490
200
1958
1992
1954
1958
1960
1952
750
280
Trend and seasonality. We generated time series following linear and exponential trends: CHRONOS-T5 (Base) predicts the linear trend accurately but struggles with the exponential trend, as shown in Figure 12b. This may be due to limitod representation of exponential trends in the training data. A po- tential resolution for generating better forecasts for time series with exponential trends is to perform logarithmic scaling be- fore feeding the time series into CHRONOS models. We also ob- served that CHRONOS models tend to underestimate the trend when the context not sufficiently long. This phenomenon is depicted in Figure 13 where the model forecasts the pattern correctly but underpredicts the trend when a short context is provided. However, with longer context, the model picks up the correct pattern and trend, In our analysis, we observed that CHRONOS models recognize scasonal patterns in time scries particularly well. We generated purely seasonal time series using sinusoids with different frequencies. As shown in Fig- ure 12c, CHRONOS-TSs 15 (Base) precisely forecasts both time series. When fundamental patterns such as trend and seasonality are combined, either additively or multiplicatively. CHRONOS forecasts them accurately. This is demonstrated in Figure 12d on time scries generated via addition and multiplication of a linear function with a sinusoid.
1950
1952
1954
1938
1958
1960
1982
Figure 13: When the context is not sufficiently long, CHRONOS-T5 (Base) tends to underes- timate trend, as shown in this example with the classic Air Passengers data (monthly) and forecast horizon of 24. Top: with only 120 observations as context, the median prediction plateaus compared to the previous trend. Bot- tom: with the full context of 144 observations, the prediction picks up the trend more closely.
Autoregressive processes. An autoregressive (AR) process of order p defined as
X, Σ
PiXi-i Exi
where E, ~ (O, 1) and f1.....,4 Sp are the parameters of the model. We generated time series from stationary AR processes of different orders ranging from to 4, and we compared the forecasts generated by CHRONOS- T5 (Base) against those generated by three models: (a) the ground truth AR model that was used to generate the time series; (b) an AR model with the correct order (p) fitted to the time series; and (c) an AutoARIMA model fitted to the time series. Figure 14 shows the results for the AR(1) and AR(4) processes, and Figure 21 (Appendix E) shows the results for AR(2) and AR(3). We observe that CHRONOS-T5 (Base) generates plausible forecasts across all four AR processes. The simpler AR(1) and AR(2) processes are casier for the correctly-specffie AR model and AutoARIMA model to fit, resulting in a better MSE than THRONOS-T5 (Base). However, with increasing complexity in AR(3) and AR(4) processes, CHRONOS-Th (Basc) not only outperforms the AutoARIMA model (which belongs the same family as the ground truth model) but also performss on par with the fitled AR model with correct order. These results highlight that CHRONOS models can recognize fundamental patterns present in time series data.
Flexible predictive distributions. Using a categorical distribution to encode predictions gives CHRONOS flexibility in producing predictive distributions of different shapes. This is shown in Figure 15, illustrating kernel density estimate (KDE) plots of token IDs sampled from a CHRONOS model, for the first five time steps in the forecast horizon, across three datasets. Despite the fact that cross-entropy is not distance- aware, CHRONOS outputs predictive distributions over contiguous set of tokens, and with different shapes,
17

---

Published in Transactions on Machine Learning Research (10/2024)
Forecast
Interaal
Median Ground Truth
Ground Truth
0.0
-2.5
350
400
450
500
550
350
400
459
509
550
AR (with correct order)
AR (with correct order)
M16 2418
Www Why
w
0.0
-255
-10
350
400
450
500
550
350
400
450
509
550
AUtOARIMA
AutOARIMA
MBG 455
MSG 1101
|uy Ww
-2.5
-10
350
400
450
500
550
350
400
450
500
550
(Base)
Chrones- (Base)
MSR 1.10
1765
Wwy Why
0
-2.5
-10
350
400
450
500
550
350
400
450
500
550
(a) AR(11
(b) AR(4)
Figure 14: Forecasts generated by CHRONOS-T5 (Base) for time series generated from AR(1) and AR(4) processes compared against forecasts generated by the ground truth AR model, a fitted AR model of the correct order, and AutoARIMA model. CHRONOS-T5 (Base) generates plausible forecasts and prediction intervals in both cases. All AR models fit the simpler AR(1) process correctly and obtain better MSE than CHRONOS-T5 (Base); however, with the increased complexity in the AR(4) process, CHRONOS-T5 (Base) performs second best after the ground truth AR model.
1
he
h
Am12
3
he.3
5-4
h-44
6-
5-5
h-
A-
Trkkn
ID
(a) NN5
(b) Traffic
(c) Hospital
Figure 15: Forecast distributions from CHRONOS model series from the NN5 (Daily), Traffic, and Hospital datasets respectively. Each plot shows the predictive distribution for five prediction steps (h - 1,...,5):: 5):
the densities were
obtained via kernel density estimation from sample forecasts. Even though the entropy is not distance-aware, the model learns to estimate distributions over neighboring tokens, and of diverse shapes, including multimodal ones.
including multi-modal ones. Although CHRONOS learns the topology the space directly from the data, we hypothesize that providing explicit topological information to the model during training may expedite the process and make the model robust for tokens where fewer datapoints are available. A potential method to inject topological information into the cross-entropy loss is through type of label smoothing assigning non-zero probability mass to tokens (i.e., bins) in the neighborhood of the the correct token. Farebrother et al. (2024) have obtained promising results with such distance-aware regression-via-cuee via-classificattion objective in the context. of reinforcement learning. An in-depth h theoretical and empirical analysis of the regression-via- classification paradigm in the context of time series forecasting would constitute interesting future research.
Overflow and loss of precision. One limitation of CHRONOS comes from the proposed tokenization approach (sce Section 3.1). Specifically, the tokens we select represent bin centers the range [-15, + 15). which ultimately represent original time series values in the range [-15s, 15s), where is the scale of the time series (mean absolute value). If is very small compared to the range of values in the series, then
18

---

Published in Transactions on Machine Learning Research (10/2024)
1.0
0.0
200
400
500
100
200
300
400
500
200 300
400
500
0.5
100
200
200
400
500
100 200 300 400
500
(a)
Figure 16: Loss of precision due to scaling and quantization. In (a), data consists of unit spikes every 10, 20, 50 observations (top bottom): the scale here 1/n, hence the maximum representable value 15/n. When 1 15/n then the model camnot possibly capture the spikes appropriately (all but the lop case), since their value is not represented accurately by lokens. In (b), data a sine wave shifted up by μ 1, 10, the scale here is μ,, and as the variance of the signal becomes smaller and smaller relative to μs, the tokens precision decreases.
some obscrvations will fall out of the representable range. An example of this behaviour is with sparse series, and as shown in Figure 16a. On the other hand, very large values of compared to the variance result in loss of precision: is the number of bins (we used B 4094 in our experiments); values closer than that to each other may be mapped to the same token, with an apparent loss of precision. An example of this behaviour is given Figure 16b. An inference-time heuristic solution to this problem is to preprocess the time series using alternative normalization scheme, such as standardization, for time series with large scale and small variance. Improving the tokenization to overcome these edge cases without heuristics is subject for future work, but the results from Section 5.5 suggest that the CHRONOS models performs well on real-world data despite the limitations.
the original space, tokens are spaced 30s/(B - 1) from each other, where B
6 Discussion
CHRONOS represents one of the first endeavours in practical pretrained time series forecasting models, with remarkable zero-shot performance a comprehensive collection of test datasets. This work opens up various research avenues, some of which we discuss below.
6.1 Beyond Zero-shot Univariate Forecasting
In our experiments, we evaluated CHRONOS in a zero-shot manner for most datasets. Such a setup highlights the competitiveness zero-shot. CHRONOS models against lask-specific baselines. We expect that both in-domain and zero-shot results could be enhanced further through fino-tuning, an avenue we briefly explored in Section 5.5.2. This can be done using any parameter-efficient fine-tuning methods such those based low-rank adapters (LoRA) (Hu et al., 2022; Zhang et al., 2023). Alternatively, CHRONOS can be calibrated for a specific task with conformal methods (Romano et al., 2019; Stankeviciute et al., 2021; Xu & Xie, 2021). CHRONOS is especially attractive in the context of conformal prediction since it requires no training set, so all available data can be used for calibration.
In this work, we have focused on univariate forecasting of uniformly-spaced time series since constitutes the most common of worl-world time series use-cases. Nevertheless, practical forecasting tasks often involve exogenous information that must be taken into account. may require modeling of irregularlyssample time
19

---

Published in Transactions on Machine Learning Research (10/2024)
series (Rubanova et al., 2019; Ansari et al., 2023). One example of exogenous information is covariates, that can be either time-independent (e-g., color of the product) or time-varying (e.g., on which days the product is on sale). Another closely related problem is multivariate forecasting, where historic values of one time series (o.g., interest rates) can influence the forecast for another time series (e.g., housing prices). The number of covariates or multivariate dimensions can vary greatly across lasks, which makes it challenging to train a single model that can handle all possible combinations. A possible solution may involve training task-specific adaptors that inject the covariates into the pretrained forecasting model (Rahman et al., 2020). As another option, can build stacking ensembles (Ting & Witten, 1997) of CHRONOS and other hight-weight models that excel at handling covariates such as LiM (Ke et al., 2017).
Thus far, our exploration has centered on the problem of time series forecasting. However, several other time series analysis tasks, such as classification, clustering, and anomaly detection (Dau et al., 2018; Wu & Keogh, 2021; Ismail Fawaz et al., 2019; Goswami et al., 2024), could potentially benefit from pretrained model like CHRONOS. We hypothesize that the representations Icarned by the encoders of CHRONOS-T3 models are universal and can be used for these tasks. An exploration of CHRONOS-TS representations for various downstream tasks would constitute interesting future work.
6.2 Inference
A potential limitation of the larger CHRONOS mod- els is their inference speed compared to tpek-specific deep learning models. Figure 17 illustrates the inference time of generating forecasts for a single time series, averaged across datasets. The inference speed of the larger CURONOS models is comparable to some statistical local models. Morcover, while CHRONOS models are slower than task-specific mod- cls, they are not too large to be prohibitively slow. Furthermore, lask-specific models need to be trained for each task individually, which requires additional time and compute. In contrast, CHRONOS models can be deployed for datasots with diverse history lengths, frequencies, prediction horizons, and con- text lengths. This makes model deployment signif- icantly casier and drastically simplifies forecasting pipelines, obviating the need for lask-specific training.
Nevve
Medel Type
Secoonal
TimesFM
Taok Specific Modals
PatchT'S
Pretrained Models
Worehen
DeepAR
CPU
Type
Model
Chronee- GP 14T4
N: N-BEATS
GPU (1 V
GPU (1 x xA100) (0)
Chrosas-Te TS (Smal)
Lag-
Moirai-1
ForecastFFN
AbbETS
Chrosex-15 (Larg)
AutrThets
SCUM
AutoAMMM
10P
10"
105
10'
Avg. Inference Time (ms)
Figure 17: Inference time of different models for forecasting a single time series, averaged across datasets. The compute requirements of individual models have been highlighted.
By leveraging a language modeling framework for time series, we make developments in the NLP commu- nity immediately transferable to CHRONOS models. For instance, inference speed can be improved by using CUDA kernels optimized for modern Ampere GPUs, quantization (Dettmers et al., 2022), and faster de- coding techniques, including speculative (Leviathan et al., 2023) and lookahead (Fu et al., 2023) decoding. Developments in long-context language models (Sun et al., 2022; Dao, 2023) may help improve CIRONNO models' applicability to high-frequency datasets that require longer contexts to capture seasonal patterns. Other techniques popularly used for text language models, such as temperature tuning, beam search (Freitag & Al-Onaizan, 2017), Top-K sampling (Fan et al., 2018), mucleus sampling (Holtzman et al., 2019), could enhance the quality of forecasts. These may particularly be helpful in improving the speed and quality of point forecasts, which currently require aggregation over multiple samples.
6.3 Data
Our findings underscore that training larger models on a large corpus of time series data yields excellent in-domain and zero-shot performance. Nevertheless, in contrast to NLP, high-quality public time series data remains limited. This poscs dilemma when training models on a large corpus of diverse datascts selecting more datasets for training leaves fewer for zero-shot evaluation. The time series community would benefit greatly from the availability of larger time series datasets that could be used to develop and improve
20

---

Published in Transactions on Machine Learning Research (10/2024)
pretrained model such CHRONOS. There have been some recent efforts building large-scale time series datasets for specific domains (Emami et al., 2023; Liu et al., 2023) and cross-domain (Borchert et al., 2022), albeit further investment is needed.
Another direction to address the problem of limited data involves developing better methods for generating synthetic time series. Our work has made significant strides this direction by clearly demonstrating the utility of synthetic data generated using Gaussian processes, improving model performance when incorpo- rated into the training data. Even models trained solely on synthetic data exhibit reasonable forecasting performance. Future research could delve into the failure modes of these models, proposing enhancements to bridge the gap between real and synthetic data.
7 Conclusion
In this work, we approach the problem of developing generalist pretrained forecasting models from the lens of a minimalist. We adapt existing language model architectures and training procedures for time series lorecasting, challenging the notion that time-series-speciiie features or architectures are necessary for forecasting. This results in CHRONOS, language modeling framework for time series that is, paradoxically, agnostic to time. The defining characteristic of CHRONOS is its compatibility with any language model architecture, only requiring minimal modifications tokenization though scaling and quantization. Our pretrained models significantly outperform existing local models and task-specific deep learning baselines in terms of their in-domain performance. More remarkably, CURONOS models obtain excellent results on unseen datasets (zero-shot performance), performing competitively with the best deep-learning baselines trained on these datasets, while showing promising evidence of further improvements through fine-tuning.
Our contributions are significant in two key aspects. First, we show that existing language model architec- tures are capable of performing forecasting without time-series-specifs customizations. This paves the way for accelerated progress by leveraging developments in the area of LLMs and through better data strategies. Second, on a practical level, the strong performance of CURONOS models suggests that large (by forecasting standards) pretrained language models can greatly simplify forecasting pipelines without sacrificing accuracy, offering an inference-only alternative the conventional approach involving training and tuning model on individual tasks.
Acknowledgements
We are indebted to Stefano Soatto for challenging us to think about the fundamental question regarding language models and time series modeling, ultimately leading to the creation of the present work. We are grateful lo our fellow researchers who have contributed to this work with insightful discussions and valuable feedback, including but not limited to George Karypis, Huzefa Rangwala, Devamanyu Hazarika, Imry Kissos, Laurent Callot, Baris Kurt, Valentin Flunkert, David Salinas, Boran Han, Xiaoyong Jin, Luke Huan, Youngsuk Park, Gaurav Gupta, Karthick Gopalswany, Tim Januschowski, Jan Gasthaus, Bing Xiang, Kashif Rasul, Juba Nait Saada, Matthias Karlbauer, Hugo Senetaire, Mononito Goswami and Gerald Woo.
References
Alexander Alexandrov, Konstantinos Benidis, Michael Bohike-Schnneider, Valentin Flunkert, Jan Gasthaus, Tim Januschowski, Danielle C Maddix, Syama Rangapuram, David Salinas, Jasper Schulz, et al. GluonTS: Probabilistic and Neural Time Series Modeling in Python. The Journal of Machine Learning Research, 21(1):4629-4635 2020. 33
Abdul Fatir Ansari, Konstantinos Benidis, Richard Kurle, Ali Caner Turkmen, Harold Soh, Alexander J Smola, Bernie Wang, and Tim Jamuschowski. Deep Explicit Duration Switching Models for Time Series. Advances in Newral Information Processing Systems, 34, 2021. 10
Abdul Fatir Ansari, Alvin Heng, Andre Lim, and Harold Soh. Neural contimnous-diserete state space models for prrgullaly-samplle time series. In International Conference on Machine Learning, pp. 926-951. PMLR, 2023. 20
21

---

Published in Transactions on Machine Learning Research (10/2024)
V. Assimakopoulos and K. Nikolopoulos. The theta model: a decomposition approach to forecasting. Inter- national Journal of Forecasting, (6(4):521-530 2000. 3, 9, 33
George Athanasopoulos, Rob J. Hyndman, Haiyan Song, and Doris C. Wu. The tourism forecasting compe- tition. International Journal of Forecasting, 27(3):822 844, 2011. 32
Konstantinos Benidis, Syana Sundar Rangapuram, Valentin Flunkert, Yuyang Wang, Danielle Maddix, Caner Turkmen, Jan Gasthaus, Michael Bohlke-Schneider, David Salinas, Lorenzo Stella, Pravier Aubet, Laurent Callot, and Tim Januschowski. Deep learning for time series forecasting: Tutorial and literature survey. ACM Compul. Surv., 55(6), 2022.
Oliver Borchert, David Salinas, Valentin Flunkert, Tim Jamuschowski, and Stephan Günnemann. Multi- objective model selection for time series forecasting. arXiv preprint arXiv:2202 00X.:22202.08485, 2022. 21
Tom B. Brown, Benjamin Mann, Nick Ryder, Mclanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Πla Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020. 2, 3, 4
Chris U Carmona, François-Xavii Aubet, Valentin Flunkert, and Jan Gasthaus. Neural Contextual Anomaly Detection for Time Series. arXic:2107.00700, 2021." 7
Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza Ramirez, Max Mergenthaler Canseco, and Artur Dubrawski. N-HiTS: Neural Hicrarchical Interpolation for Time Series Forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 2023. 10, 33
Jianlin Cheng, Zheng Wang, and Gianluca Pollastri. A neural network approach to ordinal regression. In 2008 IEEE international joint conference on neural networks (IEEE world congress on computational intelligence), pp.
IEFE, 2008. 6
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling Language Modeling with Pathways. Journal of Machine Learning Research, 24(240):1 --1,)))111111-11111 2023. 3
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling Instruction-Finttund Language Models. arXiv:2210.11420 2022. 3
Tri Dao. Attettntt: Faster Attention with Better Parallelism and Work Partitioning. arXiv:2307.08691 2023. 20
Abhimanyu Das, Weihao Kong, Rajat Scn, and Yichen Zhou. A decoder-only foundation model for time- series forecasting. arXiv:2310.1063 2023. 2, 4
Hoang Anh Dau, Eamonn Keogh, Kaveh Kamgar, Chin-Chia Michael Ych, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, Yanping, Bing Hu, Nurjahan Begum, Anthony Bagnall, Abdullah Mueen, Gustavo Batista, and HeXagon-ML. The UCR Time Series Classification Archive, October 2018. https: //www. cs. ucr. www..edu/eeammnnt wtwpieesseeeesdtaaaaeeeeeeeea
20
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. LLM.int(): 8-bit Matrix Multiplication
for Transformers at Scale. arXiv:2208.0 N0X0:22008.07339, 2022. 20
Jiaxiang Dong, Haixu Wu, Haoran Zhang, Li Zhang, Jianmin Wang, and Mingsheng Long. SimMTM: A
Simple Pre-Training Framework for Masked Time-Series Modeling, orXiv:2302.00861, 2023. 4
Samuel Dooley, Gurnoor Singh Khurana, Chirag Mohapatra, Siddartha Naidu, and Colin White. Forc- castPFN: Synthetically-Traine Zero-Shot Forecasting. In Advances in Neural Information Processing Systems, 2023. 2, 4, 10, 15, 33
22

---

Published in Transactions on Machine Learning Research (10/2024)
David Duvenaud, James Lloyd, Roger Grosse, Joshua Tenenbaum, and Ghahramani Zoubin. Structure Discovery in Nonparametric Regression through Compositional Kernel Search. In International Conference on Machine Learning, pp. 1166 1174. PMLR, 2013. 7
Patrick Emami, Abhiject Sahu, and Peter Graf. BuildingsBench: A Large-Scale Datasct of 900K Buildings and Benchmark for Short-Term Load Forecasting. arXiv:2307.00144, 2023. 21
Angela Fan, Mike Lewis, and Yann Dauphin. Hicrarchical Neural Story Generation. arXiv: 2018. 20
Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taiga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, et al. Stop regressing: Training value functions via classification for scalable deep rl. arXiv preprint arXi::2403.00 (03.00000 2024. 18
Philip J Fleming and John J Wallace. How not to lie with statistics: the correct way to summarize benchmark results. Communicatiins of the ACM, 29(3):218 221, 1986. 10
Markus Freitag and Yaser Al-Onaizan. Beam Search Strategies for Neural Machine Translation.
arXiv: 102::70020886 2017. 20
Yichao Fu, Peter Bailis; Ion Stoica, and Hao Zhang. Breaking the Sequential Dependency of LLM Inference Using Lookahead Decoding, November 2023. URL
20
Lco Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv:2101.0000 00027, 2020. 7
Federico Garza, Max Mergenthaler Canseco, Cristian Challi, and Kin G. Olivares. StatsForecast: Lightning fast forecasting with statistical and cconometric models. PyCon Salt Lake City, Utah, US 2022, 2022. 33
URL https: thub.
Jan Gasthaus, Konstantinos Benidis, Yuyang Wang, Syama Sundar Rangapuram, David Salinas, Valentin Flunkert, and Tim Januschowski. Probabilistic Forecasting with Spline Quantile Function RNNs. In Pro- ceedings of the Twenty-Seen Second International Conference on Artificial Intelligence and Statistics, volume 89
of Proccedings of Machine Learning Research, pp. 1901-1910. PMLR, 2019. 3, 10, 35
Tilmann Gneiting and Adrian E Raftery. Strietly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):359-31) 2007. 10, 35
Rakshitha Godahewa, Christoph Bergmeir, Geoffrey I. Webb, Rob J. Hyndman, and Pablo Montero-Manso. Monash Time Series Forecasting Archive. In Newral Information Processing Systems Track on Datasets and Benchmarks, 2021. 9, 30, 32, 33
Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. Moment: A family of open time-series foundation models. arXiv preprint arXiv:2402.0385 2024. 4, 20
Nate Gruver, Marc Finzi, Shikai Qiu, and Andrew Gordon Wilson. Large Language Models Are Zero-Shot Time Series Forecasters. In Advances in Newral Information Processing Systems, 2023. 1, 4, 10, 33, 34
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degener- 2019. 20
ation. arXiv:100,
Fdward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Rank Adaptation of Large Language Models. In International Conference on Learning Represenlations, 2022. 19
Shi Hu, Egill Fridgeirsson, Guido van Wingen, and Max Welling. Transformer-basee based deep survival analysis.
In Survival Coadiction-Algorilllee Challenges and Applications, pp. 132-148. PMLR, 2021. 6
23

---

Published in Transactions on Machine Learning Research (10/2024)
Rob Hyndman, Anne B Kochler, J Keith Ord, and Ralph D Snyder. Forecasting with exponential smoothing: the state space approach. Springer Science & Business Media, 2008. 3, 9
Rob Hyndman and George Athanasopoulos. Forecasting: principles and practice. OTexts, 2018. 1,
Rob J Hyndman and Anne B Koehler. Another look at measures of forecast accuracy. International journal of forecasting, 22 22(4))679 688, 2006. 10, 34
Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre-Alas Muller. Deep learning for time series classification: a review. Data mining and knowledge discovery, 33(4)9917 963, 2019. 20
Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y. Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Fuan-Fang Li, Shirui Pan, and Qingsong Wen. Time-LLM, LLM: Time series forecasting by reprogram- ming large language models. In The Twelfth International Conference on Learning Representations, 2024. 2,
Xiaoyong Jin, Youngsuk Park, Danielle Maddix, Hao Wang, and Yuyang Wang. Domain adaptation for time series forecasting via atlention sharing. In International Conference Machine Learning, pp. 10280- 10297. PMLR, 2022. 1, 4
Guolin Ke; Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. LBgh A Highly Efficient Gradient Boosting Decision Tree, Advances in neural information processing systems, 30, 2017. 20
Roger Koenker and Kevin F Hallock. Quantile regression. Journal of economic perspectives, 15(4):143-156, 2001. 35
Stephan Kolassa and Tim Januschowski. A classification of business forecasting problems. Foresight, 52, 2019.
Marcel Kollovieh, Abdul Fatir Ansari, Michael Bohlke-Schneider Jasper Zschiegner, Hao Wang, and Yuyang Wang. Predict, Refine, Synthesize: Self-Guiding Diffusion Models for Probabilistic Time Scries Fore- casting. In Advances Newral Information Processing Systems, volume 36, pp. 28341 18364.4 Curran Associates, Inc., 2023. 10
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decod- ing. In International Conference on Machine Learning, pp. 19274 19286. PMLR, 2023. 20
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zet.lemoyer. BART: Denoising Sequence-to-SSeqeence Pre-training for Natural Lan- guage Generation, Translation, and Comprehension. (rXtt:1910.13461 2019. 3
Bryan Lim, Sercan Ö Arik, Nicolas Loeff, and Tomas Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting, 37(4):1748 1764, 2021. 3, 6, 10, 33
Xu Liu, Yutong Xia, Yuxuan Liang, Junfeng Hu, Yiwei Wang, Lei Bai, Chao Huang, Zhenguang Liu, Bryan Hooi, and Roger Zimmermann. Largest: A benchmark dataset for large-scale traffic forecasting. arXiv:2306. 08259, 2023. 21
Spyros Makridakis and Michele Hibon. The M3-Competition: results, conclusions and implications. Inter- national journal of forecusting, 16 16(4):451 476, 2000. 33
Spyros Makridakis, Michele Hibon, and Claus Moser. Accuracy of forccasting: An empirical investigation. Journal of the Royal Statistical Sociely. Series A (General), 142 145, 1979. 9, 33
Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The M4 Competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting, 36(1):54 74, 2020. 9, 33
24

---

Published in Transactions on Machine Learning Research (10/2024)
Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. M5 accuracy competition: Results, findings, and conclusions. International Journal of Forecasting, 38(44)1346-136: 2022. 9, 33
Peter McCullagh. Regression models for ordinal data. Journal of the Royal Statistionl Sociely: Series B (Methodological), 42(2):109-12) 127, 1980. 6
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv:1609.0755 077444 2016. 7
Suvir Mirchandani, Fei Xia, Pete Florence, Brian Ichter, Danny Driess, Montserrat Gonzalez Arenas, Kan- ishka Rao, Dorsa Sadigh, and Andy Zeng. Large language models as general pattern machines. In Proceedings of The 7th Conference on Robot Learning, volume 229 of Proceedings of Machine Learning Research, pp.
PMLR, 2023. 3
Yuqi Nic, Nam H. Nguyen, Phanwadce Sinthong, and Jayant Kalagnanam. A time scries worth 64 words: Long-term forecasting with transformers. In International Conference on Learning Regresenlations, 2023.
3, 4, 10, 33
Kin G. Olivares, Cristian Challú, Federico Garza, Max Mergenthaler Canseco, and Artur Dubrawski. Neu- ralForecast: User friendly statc-of-the-art neural forecasting models. PyCon Salt Lake City, Utah, US 2022, 2022. URL https://githhbe http://./mmiiiiib/e/eaaareraaaae 33
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv:1609.08// 03499, 2016. 10, 33
Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-BEATS: Neural basis expansion analysis for interpretable time series forecasting. In International Conference Learning Representations, 2020. 10, 33
Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. Meta-learning framework with applications to zcro-shot time-scries forocasting. In Proceedings of the AAAI Conference Artificial Intelligence (AAAI), 2021. 4
Bernardo Pérez Orozco and Stephen J. Roberts. Zero-shot and few-shot time series forecasting with or- dinal regression recurrent neural networks. In 28th European Symposium on Arlificial Neural Nelworks, Computational Intelligence and Machine Learning, pp. 503-508, 2020. 4
Youngsuk Park, Danielle Maddix, François-Xavier Aubet, Kelvin Kan, Jan Gasthaus, and Yuyang Wang. Learning quantile functions without quantile crossing for distribution-free time series forecasting. Tn International Conference on Artificial Intelligence and Statistics, pp. 8127 8150. PMLR, 2022. 3
Fotios Petropoulos and Ivan Svetunkov. A simple combination of univariate models. International journal of forecasting, 36(1):100-115, 2020. 10
Stephan Rabanser, Tim Januschowski, Valentin Flunkert, David Salinas, and Jan Gasthaus. The effectiveness of discretization in forecasting: An empirical study on neural time series models. arXiv:2005.101 10111, 2020. 5
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019. 4, 6
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Tiu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485 5551, 2020. 3, 6, 7, 9, 13
Wasifur Rahman, Md Kamrul Hasan, Sangwu Lee, Amir Zadeh, Chengfeng Mao, Phils-Philippe Morency, and Ehsan Hoque. Integrating multimodal information in large pretrained transformers. In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 2020, pp. 2359. NIH Public Access, 2020. 20
25

---

Published in Transactions on Machine Learning Research (10/2024)
Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. Advances in neural informalion pro- cessing systems, 31, 2018. 3
Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising diffusion models for multivariate probabilistic time scries forecasting. In International Conference on Machine Learning, pp. 8857-8868. PMLR, 2021. 3
Kashif Rasul, Arjun Ashok, Andrew Robert Williams, Arian Khorasani, George Adamopoulos, Rishíka Bhag- watkar, Marin Biloš, Hena Ghonia, Nadhir Vincent Hassen, Anderson Schneider, Sahil Garg, Alexandre Drouin, Nicolas Chapados, Yuriy Nevmyvaka, and Irina Rish. Lag-Ilama: Towards foundation models for time series forecasting, 2023. 2, 4, 10, 33
Yaniv Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile regression. Adoances in neural information processing systems, 32, 2019. 19
Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for irregularly-sampled time series. Advances in neural information processing systems, 32, 2019. 20
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3):1181 1191, 2020. 3, 5, 6, 10, 33
Rico Semnrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. 1rXi::1508.077 07909, 2015.
Oleksandr Shchur, Ali Caner Turkmen, Nick Erickson, Huibin Shen, Alexander Shirkov, Tony Hu, and Bernie Wang. Autogluon-timest timeseries: Automl for probabilistic time series forecasting. In International Conference on Automated Machine Learning, pp. 9-1. PMLR, 2023. 10
Kamile Stankeviciute, Ahmed M Alaa, and Mihaela van der Schaar. Conformal timo-series forocasting. Advances in neural information processing systems, 34:6216 6228, 2021. 19
Lawrence Stewart, Francis Bach, Quentin Berthet, and Jean-Philippe Vert. Regression as classification: Influence of task formulation on neural network features. In International Conference on Artificial Intel- ligence and Statistics, pp. 11563 11582. PMLR, 2023. 6
Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. A length-extrapolatable transformer. 2rX1022212.10554, 2022. 20
Christian Szegedy, Vincent Vanhoucke, Sergey Toffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision, 2015.
Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fodus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. Scale elficiently: Insights from pre-training and fine-tuning transformers. arXiv:2109.10636, 2021. 9, 13
Kai Ming Ting and lan H Witten. Stacking bagged and dagged models. In Proceedings of the Fourteenth International Conference on Machine Learning, 1997. 20
Luis Torgo and Joao Gama. Regression using Classification Algorithms. Intelligent Data Analysis, 1(4): 275 275-292, 1997. 6
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bash- lykov, Soumya Baira, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Fer- rer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosscini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao,
26

---

Published in Transactions on Machine Learning Research (10/2024)
Xavier Martinct, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subra- manian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Niyan Zarov, Yuchen Zhang, Angela Fan, Melanic Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open Foundation and Tine-Tuned Chat Models, 2023. 2, 3,
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In Advances in Neural Information Processing Systems, 2017. 3
Yuyang Wang, Alex Smola, Danielle Maddix, Jan Gasthaus, Dean Foster, and Tim Jamuschowski. Deep factors for forecasting. In International conference on machine learning, pp. 6607-6611 PMLR, 2019.
Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamy, and Dhruv Madeka. A Multi-Horizen Quantile 2017. 3,
Recurrent Forecaster.
Christopher Winship and Robert D Mare. Regression models with ordinal variables. American sociological review, pp. 512-525, 1984. 6
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45. Association for Computational Linguistics, 2020. 6,
Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training universal time series forecasting transformers. arXie:2402. 02592, 2024. 2, 4, 10, 33
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. TimesNet: Tempo- ral 2D-Variation Modeling for General Time Series Analysis. In International Conference on Learning Representations, 2023. 4
Renjie Wu and Eamomn Keogh. Current Time Series Anomaly Detection Benchmarks are Flawed and are
Creating the Illusion of Progress. IEEE Transactions on Knowledge and Data Engineering, 2021. 20
Chen Xu and Yao Xie. Conformal Prediction Interval for Dynamic on Machine Learning, pp. 11559-11569, PMLR, 2021. 19
In International Conference
Hao Xue and Flora D. Salim. PromptCast: A New Prompt-based Learning Paradigm for Time Series Forecasting. arXi::220008800 2023. 2,
Rui Ye and Qun Dai. A novel transfer learning framework for time series forecasting, Snowledge-Basse Systems, 156:74-99) 2018.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are Transformers Effective for Time Series Forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, 2023. 3, 10, 33
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond Empirical Risk Minimization, arXiv:1710.09.0 122,, 2017. 7
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao Adaptive Budget Allocation for Farficten-fffiiinnt Finc- Eine-Tnnigg In International Conference on Learning Representations, 2023. 19
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang. Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. ar0iv:2303. 18223, 2023. 3
27

---

Published in Transactions on Machine Learning Research (10/2024)
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond Efficient. Transformer for Long Sequence Time-Series Forecasting. In The Thirly-Fiihh AAAT Conference on Artificial Intelligence, AAAI 2021, Virtual Conference, volume 35, pp. 11106 11115. AAAI Press, 2021. 3, 30
Tian Zhou, Peisong NIu, Xue Wang, Liang Sun, and Rong Jin. One Fits All: Power general time series analysis by pretrained LM. In Advances in Newral Information Processing Systems, 2023a. 2, 4, 10, 33
Yun Zhou, Liwen You, Wenzhen Zhu, and Panpan Xu. Improving time series forecasting with mixup data augmentation. In ECML PKDD 2023 International Workshop Machine Learning for Irregular Time Series, 2023b. 7
28

---

Published in Transactions on Machine Learning Research (10/2024)
A
Algorithms
Algorithm 1 and algorithm present the pseudocode for TSMixup and KernelSynth, respectively,
Algorithm TSMixup: Time Series Mixup
Input: Time series datasets (.,,..., NNaJ, maximum time series be mixed 3, symmetric Dirichlet concentration parameter 1.5, and (minimum, maximum) length of the augmented time scries (Imin 128, mmx 2048).
Output: An augmented time series.
1: k: ~ K}
number time series to mix b length of the augmented time series
2: I mllmin }
for 1, do
n [11
sample dataset index > sample a time series of length from dataset n apply mean scaling to the time series
5:
Xn
(f)
T1d
1=1 1=
end for
(A1,..., Akk
Dir((a1 a....., 0k = a))
sample mixing weights D take weighted combination of time series
9: return Σ A
Algorithm KernelSynth: Synthetic Data Generation using Gaussian Processes
Input: Kernel bank IC (see table 2), maximum kernels per time series 5, and length of the time scries Lann 1024.
Output: A synthetic time series Wi:ley"
j ]]
sample the number of kernels A sample j kernels from K.
2: (k.t,,")
ltd x
3: t') (t,
for i 2,j do
*~ {+,x]
sample a random binary operator compose kernels
(t,t") + n² xt(tt))(((())))
end for
8: 1llyyn GP(O, x (1)
sample from the GP prior
9: return F1:laa
Kernel
Formula
Hyperparameters
Constant
&Coust (z, x²) = C
C=1
White Noise
KWhiit #Whitn(II x') = on (z=r')
on E (0.1,1)
Linear
KLin (z, x') = 0² + x x!
a 10)
RBF
KRBF (x,,a') x²) = exp
|>--''1
1 10.1,1,10)
21
Rational Quadratic
KRQ (z, x²)
||2
a E 1.0
2a
p E {24, 48, 96,996,7 336, 7, 14, 30, 60, 365, 730, 4, 26, 52, 6, 12, 40, 10)
Periodic
KPer (z, x') = exp
(-2sin
π
Table 2: The kernel bank, X., used in KernelSynth (algorithm 2).
29

---

Published in Transactions on Machine Learning Research (10/2024)
B
Datasets
The complete list of datasets used for our empirical evaluation is provided in Table 3. The table is divided into three sections, representing how the datasets were used for CHRONOS models: in total, 55 datasets where used for experiments, 13 of which for pretraining only, 15 for in-domain ovaluation, and 27 for zero- shot evaluation (see also Section 5). In the following, we provide a brief description of each dataset, organized by its domain.
B.1 Energy
Australian Electricity (Godahewa et al., 2021) contains electricity demand data from 5 states in Australia.
Electricity (15 Min., Hourly, Weekly) contains electricity consumption (in kW) for 370 households. Original data has 15 minutes frequency and was obtained from /tttps//archiie... ics uci www.uuii.uuuu. aeii.eddddaase s1//elletriddttyyodddaaaammmm 20100001 hourly and weekly aggreations are from Godahewa et al. (2021).
ERCOT Load contains hourly energy load in 8 US regions between 2004 and 2021.
ETT (15 Min., Hourly) (Zhou et al., 2021) contains oil lemperatures and other covariates of electrical transformers from two stations in China, measured at 15 minutes granularity.
London Smart Meters contains half-hourly energy consumption 5561 houscholds in the UK between 2011 and 2014. Data was obtained from https://data, London.gov.uee gov. wkd.ua//atsse///////// smerrtmeter--eerr----e---d in- londonn-houseoolds
Solar (5 Min., Hourly) contains data about solar power generation in the US in 2006. The original data has 5 minule frequency and was obtained from http://www..el...gogg/////////// httpgrrrrr/ooaapoeerreeoee html; the hourly version was obtained via mean aggregation.
nrel.
Spanish Energy and Weather contains 4 years of electricity consumption, generation, pricing, and weather data for Spain. Electricity data is for all of Spain, weather data provided for each of 5 major Spanish citics. The data was obtained from https://ww..kag/t http://wwwwwkgggll..cc///at///// cot////ttttett///iccoossaaaaaaa energy-consumptioon oewprn/neeeeeeeetettoooeeeeeeeee wed-w.e
Wind Farms (Hourly, Daily) (Godahewa et al., 2021) contains energy production data from wind farms in Australia. Original data was collected at minute frequenccy, which we aggregated to hourly and daily using the mean.
B.2 Finance and economics
CIF 2016 (Godahewa et al., 2021) contains banking data, that was used in the CTF 2016 forecasting com- petition. Of all time series included, 24 are real data while the other 48 are artificially generated.
Exchange Rate contains daily exchange rates for currencies of eight countries (Australia, British, Canada, Switzerland, China, Japan, New Zealand and Singapore) between 1990 and 2016.
FRED-MD (Godahewa et al., 2021) contains mouthly macro-economic indicators from the Federal Reserve Bank. Data was extracted from the FRED-MD database, and the were differenced and log-transformed..
NN5 (Daily, Weckly) (Godahewa et al., 2021) contains cash withdrawal data from ATMs.
B.3 Healthcare
Covid Deaths (Godahewa et al., 2021) contains daily count data of COVID-19 deaths in a set of countries and states, between Jamuary and August, 2020.
Hospital (Godahewa al., 2021) contains monthly time series that represent the patient counts related medical products from January 2000 to December 2006.
30

---

Published in Transactions on Machine Learning Research (10/2024)
Table 3: All datasets that are used for experiments. 'hee datasets are partitioned according to how they are used for training and evaluation of CHRONOS models: pretraining only data is only used for CHRONOS training: in-domain evalution data is used for training CInONOS models and other task-specific baselines, except for the H observations that are held out for in-domain testing only; zero-shot evaluation data is not used training CuRONOS models, but only for evaluation (final H observations), well for training task-specific baselines (excluding the final H observations).
Series Length
Prediction
Dataset
Domain
Freq.
Num. Series
min
max
Length (H)
Pretraining-only
Brazilian Cities Temperature
nature
M
12
492
757
1320
Mexico City Hikes
transport
TH
494
780
78313
104449
Solar (5 Min.)
energy
5min
5166
105120
105120
105120
Solar (Hourly)
energy
5166
8760
8760
8760
Spenish Energy and Weather
energy
66
35064
35061
35061
Taxi (Hourly)
transport
2428
744
USIICN
nature
6090
5906
38653
59283
Weatherbench (Daily)
nature
ID
225280
14609
14609
14610
Weatherbench [Hourly]
nature
TH
2252800
350633
350639
350640
Weatherbench (Weekly)
natur
1W
225280
2087
2087
2087
Wild Daily (100k)
web
ID
100000
2741
2741
2741
Wind Farms (Daily)
energy
71
354
Wind (Hourly)
energy
1715
8514
In-domain evaluation
Electricity (15 Min.)
energy
15min
370
16032
113341
140256
Electricity (Hourly)
energy
321
26304
26304
26301
Electricity (Weekly)
energy
1W
136
156
156
KDD Cup 2018
nature
270
9504
10897
10920
48
Londou Smarr Meters
energy
30min
5560
288
29051
39648
48
M4 (Daily)
various
1D
4227
107
2371
9933
14
M4 (Hourly)
various
IH
414
748
901
1008
48
M4 (Monthly)
varions
IM
48000
60
234
2812
18
(Weekly)
various
93
1035
2610
Pedestrian Counts
transport
66
Rideshare
transport
23400
541
541
21
Taxi (30 Min.)
transport.
30min
2428
1469
1478
1488
48
Temperature-Rain
nature
ID
32072
725
725
725
30
Uber TLLC (Daily)
transport
1D
262
181
181
181
Uber TLC (Hourly)
transport
262
4344
4344
4344
24
Zero-shot evaluation
Australion Electricity
energy
30nin
5
230736
231052
232272
48
CIF 2016
banking
IM
72
28
98
120
12
Car Parts
retail
1M
2674
51
51
51
12
Covid Deaths
healthcare
Dominick
relail
100014
ERCOT Load
energy
151851
151834
15185-1
ETT (15 Min.)
15min
14
09680
69680
69680
24
ETT (Hourly)
energy
III
14
17420
17420
17420
24
Exchaange Rate
finance
1B
8
7588
7588
7588
30
FREDIMD
economics
IM
107
728
728
728
12
Hospital
healtheare
84
84
84
(Monthly)
various
90
(Quarterly]
3M
203
18
48
114
M1 [Yearly)
various
181
15
24
58
MB (Mouthll)
IM
1428
66
117
144
18
M3 (Quarterly
various
3M
756
24
48
72
M3 (Yearly)
various
IY
645
20
28
47
M4 (Quarterl]
varions
3M
24000
24
100
874
Yearlly)
various
19
37
retail
1969
NNS (Dady)
liaance
56
NN5 (Weekly)
luance
IW
111
113
113
113
Touriem (Monthly)
various
IM
366
91
298
333
21
Tourism (Qarterly)
varions
1Q
427
30
90
150
Tourism (Yearly)
varions
518
11
24
47
Trallie
transport
17544
17544
17544
24
Weather
nature
3010
1332
14296
65081
30
31

---

Published in Transactions on Machine Learning Research (10/2024)
B.4 Nature
Brazilian Cities Temperature contains monthly time series representing the weather at 12 different cities in Brazil. Data is originally from NOAA, and we used the post-processed version from https://ww..aggle..
KDD Cup 2018 (Godahewa al., 2021) contains various air quality indicators (including PM2.5, PMIO, NO2, CO, O3 and SO2); measured in 59 stations Beijing and London, wetn January 1, 2017 and March 31, 2018.
Temperature-Rain (Godahewa al., 2021) contains daily temperature observations and rain forecasts from 422 stations in Australia, between 2015 and 2017.
USHCN contains daily measurements of five climate indicators (precipitation, snow, snow depth, minimum temperature, maximum temperature) from climate stations located in 48 states in the USA. Data was obtained from https://cdiac. eesiiacceess-aee ess-dive, 1bl. tbbl.goo/ftp/uusccccaall/
Weather (Godahewa et al., 2021) contains daily time series of four weather variables (rain, mintemp, maxtemp and solar radiation) measured at weather stations in Australia.
Weatherbench (Hourly, Daily, Weekly) contains WeatherBench data at the spatial resolution of 5.625 (32x64 grid points). WeatherBench is a comprehensive benchmark dataset for weather prediction research and contains hourly values of the many weather-related variables over 40 years from 1979 to 2018 (including temperature, humidity, wind, precipitations). The original data has hourly frequency and was obtained from https://github. commppageo-dateee wareeeddddttWeeaeerBeeeeee we aggregatod it to daily and woekly using mcan, except for "total precipitation" which was aggregated by sum.
B.5 Retail
Car Parts (Godahewa et al., 2021) contains monthly sales data for various car parts, measured between January 1998 and March 2002.
Dominick (Godahewa et al., 2021) contains weekly time series representing the profit of individual stock keeping units from a retailer. Original data is from https://ww..chicagbbon htt..ottt..eee/rreeerrhh/kkl//// datasets/dominicks.
B.6 Mobility and transport
Mexico City Bikes contains hourly usage statistics for 494 bike stations in Mexico City from 2010 2022. Each value in the time series corresponds the number of bikes returned at the given station the given hour of the day. Data obtained from https://ecobici... www.biiii.ccc....mmmmmom http////ppeedaattt...mooomooo/oe Time scries that contain less than 50 non-zero observations were removed.
Pedestrian Counts (Godahewa et al., 2021) contains data from 66 sensors in Melbourne, counting pedes- trians between 2009 and 2020.
Rideshare contains various hourly statistics of Uber and Lyft services New York, between November 26, 2018 and Docember 18, 2018.
Taxi (30 Min., Hourly) contains spatio-temporal traffic time series of New York taxi rides taken at 1214 locations every 30 minutes in the months of January 2015 and January 2016. Original data has 30 minutes frequency, the hourly version was obtain by aggregation with sum.
Tourism (Monthly to Yearly) (Athanasopoulos et al., 2011; Godahewa et al., 2021) Tourism dataset from, used for the Kaggle Tourism Forecasting competition.
Traffic (Godahewa et al., 2021) contains hourly road occupancy readings from sensors in the San Francisco Bay area.
32

---

Published in Transactions on Machine Learning Research (10/2024)
Uber TLC (Hourly, Daily) contains the number of Uber pick-ups from various locations in New York, between January and June 2015. Data was obtained from https://github: http://hhhhbbcmm///eettttttttghh uber-tle-fiil-respons and aggregated hourly and daily.
B.7 Various
M1 (Monthly to Yearly) (Makridakis et al., 1979; Godahewa et al., 2021) contains the time time series used in the M1 forecasting competition. Data spans micro- pocro//macrocccoomiis, industry, and demographics.
M3 (Monthly to Yearly) (Makridakis & Hibon, 2000; Godahewa et al., 2021) contains the time time series used the M1 forecasting competition. Data spans microo//macroeccooomicc, industry, finance and demographics.
M4 (Hourly to Yearly) (Makridakis et al., 2020; Godahewa et al., 2021) contains data from various domains, at different sampling periods, used for the M4 forecasting competition. Domains include micro- /macroeconomics, demographic, industry, and finance.
M5 (Makridakis et al., 2022) contains products sales data, used for the M5 forecasting competition. The data includes sales up to the end of the validation set (end of public leaderboard), but not values for the test set (private leaderboard).
B.8 Web
Wiki Daily (100k) contains daily page views on the top-100k English Wikipedia articles between 2007 and 2022, ranked by number of obscrvations (non-missing). Data was obtained from httpss//dumps. wikimedia. Trgg/thher/pageieews.
C Baselines
We considered a total of 17 baseline methods for benchmarking CHRONOS. Local statistical baselines were AutoETS, AutoARIMA, Naive, Seasonal Naive, and AutoTheea (Assimakopoulos & Nikolopoulos, 2000); for these, relied on implementations in the StatsForocast library (Garza et al., 2022). For task-specific deep learning architectures, DeepAR (Salinas et al., 2020), PStchTST (Nie et al., 2023), TFT (Lim et al., 2021), DLinear (Zeng et al., 2023), and WaveNet (Oord et al., 2016), we based evaluations on the implementations in GluonTS (Alexandrov et al., 2020). However, N-BEATS (Oreshkin et al., 2020) and N-HiTS (Challu et al., 2023), experiments were based on implementations the NeuralForecast (Olivares et al., 2022) library. Finally, we used reference implementations of ForecastPPNN (Dooley al., 2023), GPT4TS (One-Fits-All) (Zhou et al., 2023a), LLM'ime (Gruver al., 2023), Lag-Llama!! (Rasul et al., 2023), and Moirai-1.0- !2 (Woo et al., 2024).
WaveNet and GPT4TS models were trained on AWS EC2 p3. 2xlarge instances which have 1 NVIDIA V100 GPUs with 16GB VRAM. All other baselines were trained on the CPU on Intel-based EC2 instances. Task- specific deep learning baselines not based on large language models (DeepAR, PatchTST, TFT, DLincar, WaveNet, N-BEATS, and N-HiTS) were trained and evaluated three times and their performance averaged in order account for high variance inherent their optimization.
For inference, we used EC2 CPU instances for local models, N-HiTS, and N-BEATS. The p3. 2xlarge instance (I X V100 16GB) was used for inference for other task-specific deep learning models and pretrained models such as Lag-aaama, Moirai-1.0-R, and ForecastPFN. Since LLMTime uses Llama-2 70B model which has significantly larger compule requirements, LLMTime inference was performed on the p3dn. 24xlarge AWS EC2 instance with 8 NVIDIA V100 32GB GPUs.
t/tttp:://gthub... cowp/bbacusai/Fooeccsttte
9https://githul http:///ghhuubb.oom//Do-------- -L--N00-DD--//Meur1122020------
10nttps://github com//ggruver/llmiine
Unttps://github.. cot/bicommeeeeeeeeeeee series- foundation foundationnnodeel/lla. models/lag-1 1lama
12https:/githubb wtt.c//alleeoooo.eeeeeeeeeeeee
33

---

Published in Transactions on Machine Learning Research (10/2024)
Table 4: The multiplier used to set the context length GPT4TS for each frequency. The context length equal to the multiplier times the prediction length, rounded to the nearest whole number.
Frequency
Multiplier
15min
20
30min
10
1H
10
1D or 1B
10
10
1.5
3M or 1Q
1.5
1Y
1.5
Statistical baselines (AutoFTS, AutoARIMA, Thtoa and SeasonalNaive) were used with their default hyperparameters in StatsForecast, but with season lengths implied by their frequencies. For example, daily froquency data had season length set to 7, hourly data 24, and on. For this hcuristic, used the helper function get_seasonality from GluonTS.
Unless otherwise specified, the default hyperparameter configurations provided in baseline implementations were kept as is, and no dataset specific or global hyperparameter tuning was performed. GluoTTS-base implementations were optimized with a batch size of 128, for a time limit of 4 hours and early stopping patience of 200 epochs. In PStchTST and DLincar, we experimented with two loss functions: original losses aimed at point forecasting (L1 L2 loss) well as default probabilistic forecasting heads used in their GhuonTS implementations, where the loss is set to the negative Student's-t log likelihood of the forecast horizon. Due to the consistently superior performance, our final results include the probabilistic versions of PStchTST and DLinear only. For GPT4TS, we set the context length equal to a multiple of the prediction length, with the multiplier depending on the frequency of the dataset (Table 4). We used the MASE loss function for ane-tuning in GPT4TS due to its superior performance.
For LLMTime, we experimented only with the Llama-2 70B due to the prohibitively high costs of running the benchmark through OpenAT APIs. We used the same hyperparameters used in the Monash experiment in the original paper (Gruver et al., 2023) with a few notable differences. We set the context length to 512, same for CHRONOS models, instead of 500. During hyperparameters may lead to a significant drop in the scale of the last prediction on some datasets. To alleviate this issue, we set the by (this extra prediction is removed before computing the metrics). The inference time for LLM'Time (Llama-2 70B) is =0.8 seconds per observation on p3dn. 24xlarge. As an example, this will take hours generate all the predictions on the Traffic dataset (862 time series, 24 as prediction length, 20 samples). Due to the very high compute cost, we skip the evaluation of LLMTime on some large datasets.
experiments, we observed that the default
to 1.4 (instead of 1.2) and increased the prediction length
A summary of the baseline models used along with details of hyperparameter values is provided in Table 5.
D Evaluation Metrics
In what follows, we consider a dataset of N time series {ι; (r,,,..,,,,.. c.C++) each spanning both the context length C and prodiction horizon H. We are interested in evaluating the accuracy of predictions for Bi,C41C+H, for all i NN, which can be either point forecasts or probabilistic ones.
A point forecast for æ; denoted as 2, [.,,+41.., mean absolute scaled error (MASE, Hyndman & Kochler (2006)). For each series, this is simply the mean absolute error (MAE) divided by the empirical error of seasonal naïve model:
To evaluate point forecasts, we use the
MASE(2,, 2x) C - S 14041 [a,
C+H
C-S
where S a seasonality parameter. Since the denominator scales proportionally æ,, this error metric is independent of the scale of the data. To aggregate MASE over the entire dataset, we average over all i.
34

---

Published in Transactions on Machine Learning Research (10/2024)
Table Baseline models and hyperparameter choices. Hyperparameters not specified are set to defaults their respective implementations. C' stands for context length, da for hidden layer dimension, for number of layers, wa for number heads, and vy for learning rate.
Model
Model Type Implementation
Probabilistic
Hyperparameters
Naive
Local
StatsForccust
Yes
N/A
NeivonalNaive
Local
StataForecast
Yes
N/A
AutoETS
Local
StatsForecust
Yes
C = 2500
AutoARIMA
Local
StataForecast
Yes
C= 1000
AutoThela
Local
StatsForecast
Yes
C 2500
DeepAR
Task-specifie
GSuonTSS
Yes
du 10, =2
TFT
Task-specifie
GluonTS
Yes
d, = 32, aH ==:==:0
PStchTST
Task-specifte
GluonTS
Yes
Patch length: 16, Stride: 8, da 32, nz
2,ng
DLinear
Task specific
GluonTS
Yes
Kernel size: 25, dg
WaveNet
Task specifie
Gh
Yes
Residual channels: 24, Skip channels:
N-BEATS
Taec-specifie
NeuralForecust
Iuput size multiplier: 5
N-HITS
Tpec-specific
NeuralForecast
Iuput size multiplier: 5
GPT4TS
Task-specifte
Reference
Fine-tuning epochs: 100, cos: 1, tmaa:: 10, viz 6,
with pretrained GPT-2 weights
PFN
Pretrained
Reference
No
(as the released pretrained model)
LLMTime
Pretzained
Reference
Yes
512, STEP_MULTIPLIER 1.4 (refer the text for details)
Lag-Llama
Pretrained
Reference
Yes
32
Moirai-1.0-R
Pretrained
Reference
Yes
1024, Patch length: selected by datasct-spocific validation
Probabilistic forecasts are given in terms of predicted quantiles (a) 11 c+ C+ levels a E (0, 1).
(a)
(a)
To evaluate the quality of such predicted quantiles, the weighted quantile loss (WQL): this is aggregation of the quantile loss (Koenker & Hallock, 2001), which is defined for the predicted a-quantile q of real observation z, as
{1 (1 (1-a)(q x), otherwise.
Sa(s - q), i 4
(4)
To aggregate Eq. (4) over multiple series and prediction instants, consider the weighted average
WOL 2E,r Quaaaai x4,e)
Ee, [aa
We average the above over finite set of levels .kk) to obtain
1 WQL K
WQT
In all experiments, we use quantiles at level a E o0.,,......,OA.. to compute WQL, so that K = 9. Note that, being a weighted average the quantile loss at different levels, WQL approximates (a weighted average of) the continuous ranked probability score (CRPS), a commonly used metric for evaluating probabilistic predictions (Gneiting & Raftery, 2007; Gasthaus et al., 2019). Unlike for MASE, where errors scaled by term proportional to the scale of each series, WQL aggregates absolute errors: as such, its value is affected by the relative scale of all series in the dataset.
E
Additional Results
This section complements Section 5.5 by providing additional details to the experimental results. Table 6 reports the training time and cost of CHRONOS-T3, models on p4d. 24xlarge EC2 instance. Tables and 8 report the raw WQL and MASE scores logether with the aggregale relative score and average rank obtained by all models on the datasets in Benchmark I. Similarly, Tables and 10 report these scores on Benchmark II. Figures 18 and 19 show the average ranks obtained by different models on Benchmark I and II, respectively. Figure 20 illustrates the zero-shot performance of CHRONOs-T3-S3 T5-Synth (Small), model trained solely on synthetic data generated using KernelSynth, against various baselines.
35

---

Published in Transactions on Machine Learning Research (10/2024)
Table 6: Training time and the cost of training CHRONOS models on a single p4d. 24x1arge instance. On-demand EC2 pricing of $32.773/hr was used to compute the cost (rounded to the nearest dollar).
Model
Training Time (hrs)
Cost (USD)
Chronos- T5
(Mini)
7.68
252
Chronos- T5 (Small)
7.73
253
Chronos- T5 (Base)
17.96
588
Chronos- T5
(Large)
63.05
2066
Table 7: WQL scores of different models for datasets in Benchmark 1, comprising 15 datasets also included in the training data of CHRONOS models. Models achieving the first, second, and third best scores have been highlighted. Scores for CURONOS and task-specific models have been averaged over 3 random seeds. The aggregated relative score was computed as described in Section 5.4.
Pestodani Mhaach Danatss
Pretaleed (0ktao)
Takk Sendi Madaa
1.80
1.40
0.33
1.10
1.66
111
1.85
1.90
1475
3.10
0.31
0-80
8.220
0.33
6:38
1:38
1:30
3232
6:380
0091
6.00
1.30
15.05
84.35
IM
0.10
6:18
01:01
1.10
1.29
8.04
1:14
5:10
6:34
6:86
8455
6.10
1.16
30:0
0450
01333
6130
8100
6100
0118
8138
6106
6150
0.73
00100
6:180
8000
0241
an the
011
1.11
0112
070
0.80
1109
100
1.00
Table 8: MASE
of different models for datasets in Benchmark I, comprising 15 datasets also included in the training data of CHRONOS models. Models achieving the first, second, and third best scores have been highlighted. Scores for CHRONOS and lask-specific models have been averaged over 3 random seeds. The aggregated relative score was computed as described in Section 5.4.
Prdemadd Nabha Du Denan!
Potisked Hodio S0lee
Dak Sywtk Markh
Locd Mades
1756
1148
1.50
Lab
198
1.80
177
1421
3,00
1.95
1123
3727
2000
11:4
3.79
3.30
202
11.2
0/23
1738
1984
1:000
1.21
2413
18:0
1.40
1311
1331
1.00
1.00
2.66
1190
11080
180
1154
1.14
1.11
1.20
0.34
0.80
1.00
0.40
1.20
1.94
1.49
136
0.83
186
0110
0.00
1871
130
1.00
1330
130
1114
1.59
179
(981
0154
Table 9: WQL scores of different models for datasets in Benchmark I1, comprising 27 datasets not by CHRONOS models during training. Models achieving the first, second, and third best scores have been highlighted. Scores for CuRONOS and lask-specific models have been averaged over random seeds. The aggregated relative score computed as described in Section 5.4.
Teareed Maril Blom Badl
Potreed Nadio W0
Teo Skrris
Lord Madio
6-10
1.3.
1.80
1881
6:29
640
0.25
0.33
1.00
1.8
1.50
2114
1.20
1.50
111
1180
4811
1.30
1113
1115
1720
11:53
198
1.80
190
6115
1.83
11:9
117
0.50
1110
2:38
1.67
1811
8100
140
1.33
138
130
19M
1180
140
01/3
6.25
9.10
0.30
0.34
1.00
1.50
4.10
1.20
6.10
0.10
10.0
1.00
1.100
1.300
1.20
01100
111
011
11:47
1756
$1:00
11100
1110
11100
1110
1330
120
1340
0138
1.84
1.:0
4475
6.150
1.93
6.200
1.90
1120
1331
0.50
1121
146
1831
6108
6411
0.50
18:
0.11
1.50
1.55
5050
135
5811
1.50
111
115
110
153
1101
1131
110
111:
1311
0110
133
0.30
1.0
1136
1/16
1585
1/08
11:15
1130
180
5.3
780
1Y8
1.09
1180
07/6
4.13
36

---

Published in Transactions on Machine Learning Research (10/2024)
Table 10: MASE scores of different models for datasets in Benchmark II, comprising 27 datasets not seen by CHRONOS models during training. Models achieving the first. second, and third best scores have been highlighted. Scores for CHRONOS and lask-specific models have been averaged over 3 random seeds. The aggregated relative score was computed as described Section
1.01
1.31
120
111
1.30
8.13
1.54
1.75
0/15
0.33
6.43
180
170
6.03
110
1.30
1110
1.30
1112
1.03
1100
1710
644
110
1100
1320
1.03
190
1.99
176
155
1100
1300
1110
1100
1.00
11100
1210
1320
1331
1.30
1110
142
1.51
1000
11.0
1100
112
1380
1100
1110
1320
1300
1.80
11901
1001
0436
1.00
1110
1000
1681
3.80
182
0.20
130
1000
2001
11:00
13.00
2339
133
13.00
0.1
610
1084
530
2791
280
1018
12101
100
0118
180
Local Models
Task Specific Medéls
Protrained Models (In Domain)
Pretrained Models (Other)
(Larga)
Chrones-TS (Base)
Chronas-T5 (Basa)
Chrenoe-T5 T5 (Mal]
6.07
PatchTST
Chrenos-T6 (Smal0)
6.20
Chronos-TS (Small)
6.07
PatchTST
Chronoa- T5 (Min))
6.47
Chronos-GP12
7.53
Chrence:Oee
6.00
N-HITS
8.20
8.50
Moirak-1.9- (Large)
Mobrai 1.0-R (.arge)
N-HITS
9.67
Model
DespAR
9.53
Model
Meirai-1.0-
DeepAR
10.87
11.58
11.80
WaveNet
10.73
WaveNet
12:13
Moirai-1.0-P
11.13
TFT
13:93
Lag
Lag-Llama
14.20
AutoTheta
15:39
16.50
AutoARIMA
16.57
Seasonal
AutoETS
16.67
SCUM
17.57
AutoTheta
17.39
Seasonal Naive
18.00
AutoARIM
17.57
Natre
19.67
19.87
0.9
2.
7.5
10.0
12.5
15.0
17.9
20.0
20
Avg. Rank (WQL)
Avg. Rank (MASE)
Figure 18: Average rank of different models on Benchmark I, comprising 15 datasets also included in the training data of CHRONOS models.
Models
Specttic Models
Pretrained Models (Zero
Pretrained Medels (Other)
PatcTTST
7.04
SCUM
TFT
7.11
PatchTST
8.22
Chrenos-T5
8.23
Chronos-T5 (.arge)
8.48
DeepAR
DeepAl
9.11
Chronas-TE (Bass)
p.30
N-HIS
9.04
9.46
Chronos-TS TS (Base)
9.41
AutoThata
9.59
Chronos-TS (Small)
9.80
10.00
Chronos-Ts (2mall)
Model
Moinsi 1.0 (rage)
Chronoe-GP12
AutoETS
11.19
11.26
11.30
Model
Moirai-1 1,0 (Large)
Chrones- T5 [Min)]
Chronoe GPT2
AutoETS
GPT4TS
16.70
11.63
12.04
12,11
12.44
DUineer
12.70
Meiral-1.c0 (Race)
Mairai-1.5
13.04
DLinear
12.33
13.44
AutoARMM
12:78
15.35
LLMTime
Lag
Seasonal
Sesoonal Neiva
Naive
Nalve
Lag sig--lama
19.67
12.5
Avg. Rank (WQL)
Rank (MASE)
Figure 19: Average rank of different models Benchmark II, comprising 27 datasets by CHRONOS models during training.
37

---

Published in Transactions on Machine Learning Research (10/2024)
Specitic Models
Pretrained Models
Pretrained Medels (Othed)
PatchTST
0.601
PatchTST
0.740
N-HITS
0.656
M
0.806
N-REATS
0.664
DespAR
0.821
M.0-a--1.0- (.arge)
0.670
Wavehet
DeepAR
0.676
WoveNet
0.689
Moinai-1.0-4
0.857
Moirai-1.08
0.091
N-BEATS
0.861
Model
DLinear
Model
DLinear
0.864
GPT4T8
0.871
0.724
Chronos-TS--ymi
Clrenos-TE Syath
(Smail)
0.739
(Small)
0.876
0.940
Lag
AutsARIMA
0.541
AutoETS
0.983
Sacoonal
1.000
Havve
1.076
AutoTheta
Leg-Llama
0.0
Agg. Relative WQL
Agg. Relative MASE
Benchmark I
Local Models
Tank Specific Models
Pretrained Models (Zaro Shot)
Pretrained Models (Other)
PatchTST
N-NITS
0.500
N-BEATS
PatcTTST
0.684
0.898
DeepAR
Moirai-1.0-R (Bese)
0.696
Mozrai-100 (Large)
0.720
AutoThata
0.875
SCUM
0.728
0.876
Model
DespAR
Model
DUneer
0.894
AutoARIMA
Mairai-1.0 (Bass)
0.007
0.772
AutoTheeta
(Small)
0.703
(Small)
0.917
LLMTTime
0.804
Warehlal
AutoETS
0.959
AutoETS
LLMTime
1.000
Sessonal
1.000
Lag-Ilama
1.291
Lamaa
1.097
2.450
1.2
1.0
Agg. Relative WQL
Agg. Relative MASE
(b) Benchmark 11
Figure 20: Performance of ChRovos-TTa-yyin (Small), CHRONOS model that was only trained synthetic data, on Benchmark I and II, against local and task-specific models. Note that unlike other CHRONOS models also trained on real data, both these benchmarks zero-shot for CHRONOS-T3, TT-Syttt (Small).
38

---

Published in Transactions on Machine Learning Research (10/2024)
Ground Truth
Median Forecast
80% Interva
Ground Truth
Median Forecast
80% Interval
Ground Truth
Ground Truth AR
2.5
1 MM
mas man
M52: 7.435
M02 1.741
AR (with correct order)
AR (with correct order)
MoD 1.440
W 1MMaoooow
Moc 1.790
M/ 1pw0 MAn
0.0
Auto ARIMA
ARItAARIMA
2.5
M 1/ M/ MM0 mm
MAD 144
W
1.900
0.0
550
450
550
Chronos- T5 (Base)
Chronos-TE T5 (Base)
MSE
2.5
MW
550
(a) AR((2
(b) AR(3)
Figure 21: Forecasts generated by CHRONOS-T5 (Base) time series generated from AR(2) and AR(3) processes compared against forecasts generated by the ground truth AR model, fitted AR model of the correct order, and an AutoARIMA model. CHRONOS-T5 (Base) generates plausible forecasts and prediction intervals in both cases. All AR models fit the simpler AR(2) process well and oblain better MSE than CuRONOS-T5 (Base); however, with the increased complexity in the AR(3) process, CHRONOS-T5 T5 (Base) performus belter than other models.
39

---

Published in Transactions on Machine Learning Research (10/2024)
3083
768
2583
00:28:08
02-28:1
034000 02-0112
00-0106
09-11
64-11
04-11
04-02
197601
147-37
1977-41
1977447
197301
197847
1976-01
1979-47
190101
Car Parts
Covid Durthe
1100
0.8
0.2
1996-07
106941
194647
2035-1
2808-07
2001-41
2001-47
2600-31
320-6-491--E-1 3655-6-)5500--6-15 318-77-5518-011 2036660013661
Dominick
BNCOT Load
53
19860
16800
14200
06-05.06
16-05
06:05
06-25:21
04:23
06-26.00
06-6612
16-25
Electricity (15 Min.
Electricity (Hourly)
Amm
15-35
11-35:00
11-05 52
11-2515 15
11-0511 13
154521
11-66 DO
11-06 30
11-06 06
12:20 03
12:0812
13:20
13-29
12-30 30
12-3012
12-31 00
12-21 12
01-01 de
Blectriiity (Weokt))
Biohange Rate
1.04
2009
23
14000
12500
13000
2015-27
301641
2856-37
2017-45
2017-47
33150n
2856-37
3016-45
2014-47
3304-01
2004-07
passos
2335477
2006
2006-67
2332-01
Figure 22: Example of forecasts from CHRONOS-TT5 (Base) on the test datasets used in experiments.
40

---

Published in Transactions on Machine Learning Research (10/2024)
cup
1.25
0.75
0:28
06-2212
(Quartert)
1089
20600
489
20000
1902
1983
1964
1982
1966
1907
1997
1948
1998
2300
2001
2002
2003
2031
(Yeart))
MB (Mondday)
1000
0.33
1992
1992
(Quarterly)
(Yearty)
1091
1992
1993
1994
1965
1976
1988
1904
1986
1998
1992
M4 (Dally)
M4 (4(oo)))))
2072
500
700
600
1453
400
1996-33.22 22
1996-11 017359 CET--CE5990 111999-1122
1996 311999 081399 12:15
271 01 312019 92 012619 32 922115 32 432815 oz 042315 092E19 02 092319 0122
(Manthy))
(4(Qarterry)
10000
6089
800
600
(Vearly)
34000
4000
20003
04:00
7800
6000
34000
2816-99
2816-11
231
2017-4a
2613-45
2812-37
1990
1964
2006
2001
2008
2012
2316
Figure Example forecasts from CHRONOS-T55 (Base) on test datasets used in experiments.
41

---

Published in Transactions on Machine Learning Research (10/2024)
NNS (Paly)
NNS
Pedestrian
180
143
1997-10
1997-11
1997-12
1998-01
1995
91209111 12 072819 BE20OT 092099 12 12219291 12. 112018 122211 8-12: 102219 12-16
Rideshere
Tacc (30 Min)
13.0
12.
12.
12:1000
12/8122
31:20:00
Temperabre-Rah
(Monthly)
M ww M
10033
28
2817-00 91174---0077-1-11
Touriam (Quarterly)
Tourium (Yeurly)
15000
12503
0060
25000
1907
1988
1999
1943
1991
1992
1993
1994
1983
1982
1904
1906
1181
1060
1992
Traffic
TLC (Deily)
2.0
3.54
13-28.00
13-1112
5015-34355 2015-0888--57-01
0.8
3.4
03
06.27.0
06.2712
05.20.03
06-20
06.20
06:20
06:31
12
07.01
1981-35-31991-80 35
1901 001.0038091.56-133 35 1981 07--0151--80 37-15
1991 201-96-31101110
Figure Example forecasts from CHRONOS-T: (Base) on test datasets used in experiments.
42


Reviews:
[{"review_id": "LQBxNbF5HG", "reviewer": "Reviewer_B6a4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n1. Given the impressive recent success of LLMs and foundations, which primarily operate on discrete text tokens / words, how we can extend this success to the real-valued time series forecasting task remains an important research question, with a high potential for real-world impact. This paper proposes a simple way to do so, by simply converting the time series forecasting problem into discrete tokens. This means that any future state-of-the-art language modelling improvements can also transfer easily to the time series forecasting problem.\n\n2. The paper demonstrates strong empirical results on various time series datasets, compared to both classical and deep learning methods. The zero-shot generalisation property is particularly appealing, demonstrating the pre-trained model's ability to generalise and transfer to new tasks.\n\n3. The experiments are rigorous and thorough. I especially appreciate the extensive and thorough ablation studies, which disentangle the impact of different factors such as model size, context length, etc.\n\n4. The paper is generally very clear and well-written.\n\n# Weaknesses\n\n1. I find some aspects of the proposed approach to be rather counter-intuitive. For example:\n    - My primary concern is regarding the tokenisation approach. In my understanding, the approach works by discretising the real values into fixed buckets, after doing mean scaling. However, some time series might have a much higher standard deviation than others. This means that time series with low standard deviations would have their range of values \"squeezed\" into a small number of bins, which makes predicting accurate values much more tricky (standard scaling, which takes into account both the mean and the standard deviation, should address this concern to some extent). Another issue, which is already pointed out in the paper, is that the model cannot predict values that are outside of the range of the bins, which may happen with a new, unseen time series. The root cause of both of these issues is that the model is using the same tokenisation scheme for all time series, which encourages generality, although it may come at the expense of the ability to adapt to unseen times series with very different scale & standard deviation properties than those seen at training time.\n    - Based on Eq. (2), it seems that the model is not trained on the context tokens (i.e. given C + H tokens, where C is the number of context tokens and H is the number of tokens to predict, the model is only trained on tokens 1 to H). Training the model to also predict the context tokens, as we do in language modelling (i.e. all tokens are used for training in language modelling, from the first to the last token), may alleviate the data scarcity issue further.\n    - It seems like there is a very clear ordering of values here, so not all mistakes are created equal (e.g. predicting a slightly wrong value should have a much less severe cost than predicting a value that is way off). The paper shows that the model can already learn this ordering to some extent, but intuitively, taking into account this \"cost weighting\" should improve the performance further. Even simple approaches that penalise the different mistakes differently should be able to take this into account.\n\n2. It seems that the paper does not compare against recent state-space models, which obtained strong results on time series modelling, such as the S4 model (Efficiently Modeling Long Sequences with Structured State Spaces, Albert Gu et al., 2021).\n\n3. There are some suggestions (particularly relating to the presentation, etc.) and questions that can be clarified in order to make the paper stronger. See the \"requested changes\" section below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Chronos, a simple yet effective way of modelling time series by casting it as a language modelling problem. To that end, the paper proposes a way of discretely tokenising real-valued time-series data through scaling and quantisation. Through this step, one can then simply apply any state-of-the-art language modelling architectures to predict the next discrete tokens in the sequence, in the exact same fashion as to how one traines a standard text-based language model that predicts the next word, conditional on the previous ones. The forecasts of the time series predictor can then be converted back into real-valued outputs by simply reversing the tokenisation process (i.e. de-scaling & de-quantisation) in a deterministic fashion.\n\nNevertheless, one remaining issue with time series forecasting is the relative lack of pre-training data, especially when compared to LLM training where very large amounts of unlabelled natural language text can be easily obtained by crawling the web. To that end, this paper uses two data augmentation techniques: (i) TSMixUp (combining different, randomly-selected time series data through convex combinations) and (ii) KernelSynth (generating synthetic time series data by composing different Kernel functions).\n\nExperiments on a wide range of time series forecasting problems demonstrate that the proposed T5-based Chronos model outperforms various classical and deep learning time series forecasting approaches. Furthermore, the approach yields a decent performance in a **zero-shot** setup, where the pre-trained model is applied to an unseen time series task out of the box without any fine-tuning. This demonstrates the plausibility of the pre-trained model as a **generalist** foundation model, although fine-tuning yields even better performance (as expected). The paper then conducted extensive analyses and ablation studies to disentangle the impact of various hyper-parameters, such as the model size and context length.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. **Critical**: More discussion and clarification around the tokenisation strategy, and how it handles time series with very different standard deviations (otherwise the time series with low standard deviations would have their values squeezed into a small number of buckets).\n\n2. **Recommended**: Addressing the questions below, which would further improve the clarity of the paper and resolve any doubt.\n    - For Eq. (2), would it help to train the model on all tokens, including the context tokens (in addition to the H tokens that need to be predicted)? This can at least be used for the autoregressive models, where standard LMs also use every token for training. For the T5-style encoder-decoder models, it makes sense to only train on the H tokens, and use the C context tokens with the bidirectional attention / encoder. (as is done currently)\n    - How easy would it be to incorporate a simple cost function that weighs different kinds of mistakes? This would penalise mistakes that are \"further off\", e.g. predicting the value \"1\" when the correct answer is \"1,000\".\n    - Regarding the KernelSynth (Section 4.2), how big is the Kernel bank?\n    - To confirm, are all deep learning approaches listed on the paper trained on the exact same pre-training dataset?\n    - Currently, the proposed time series foundation model only operates on time series data. Would adding natural language information help? E.g. one could imagine giving information in natural language about what kind of time series it is, whether there is seasonality aspect, where the data came from, what time period it encompasses, etc. Naturally this would require the model to represent and model both time series data and natural language text, but given the success of LLMs, this could probably work better.\n    - How does the method compare with state space models like S4 (Albert Gu et al., 2021)? These kinds of models have shown great promise for time series modelling and modelling long-range dependencies in linear time.\n   \n3. **Recommended**: Incorporate the suggestions below.\n    - For the results (e.g. Figures 4 - 6), it would be nice to include information on the caption that **lower is better**. This would help readers who are not familiar with the evaluation metric to better interpret the results. This holds for Figures 10 - 11 as well.\n    - It seems like not all the models in Figure 4, etc. are trained on the exact same dataset (due to these models being pre-trained off-the-shelf models that are trained on a different pre-training dataset). It would be nice to mark these models as such, for example by putting an asterisk next to the model name when listed in Figure 4, etc. This would let the readers know that these models are not directly comparable.\n    - On page 12, \"Fine tuning\" section, it seems like the fine-tuning is done with an initial learning rate and a linear decay. In my experience, using a \"triangular\" learning rate, where the learning rate is warmed up from 0 to the peak value (here 0.001), and then decayed linearly after that, would work better.", "broader_impact_concerns": "No broader impact concerns from my side.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BLuifvO8Lq", "reviewer": "Reviewer_WdST", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n(i) The adaptation of language models for time series modeling via scaling and quantization and using off-the-shelf language models with minimal modifications provides a practical solution to integrate with future advancements in LLMs\n\n(ii) The proposed data augmentation techniques and synthetic data generation are simple yet effective, providing solutions for addressing the limited availability of time series data\n\n(iii) The authors provide comprehensive experimental results across 42 datasets and analysis to support the design choices of Chronos\n\nWeaknesses: \n\nSee more details in comments below\n\n(i) More comprehensive comparison with baselines\n\n(ii) More details on finetuning results\n\n(iii) Model scalability\n\n(iv) Data contamination problem\n\nComment:\n1. Although the authors compared Chronos with more than ten baseline methods, they did not include comparisons with strong baselines such as ensembles. These models often require much less inference time, and it remains unclear how their accuracy compares with that of Chronos. A more thorough comparison with ensembling baselines would provide a clearer understanding of Chronos' relative performance.\n\n2. The paper lacks detailed information on fine-tuning techniques and results. The comparison is only limited to Chronos (small), and it would be beneficial to see the performance limits of the model under fine-tuning scenarios.\n\n3. While the authors conducted experiments to assess the impact of the portion of synthetic data, the paper does not address how the model’s accuracy changes with respect to the total amount of training data. It would be valuable to include an analysis of model scalability, illustrating how Chronos performs as the training dataset size varies.\n\n4. The authors used 55 public datasets, splitting them into different components for pretraining and evaluation. However, there is a concern regarding potential data contamination. It is crucial to highlight whether data from the same domain (similar time series) is used in both pretraining and zero-shot scenarios, as this could affect the validity of the zero-shot performance claims.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a framework for pretrained probabilistic time series models by directly adapting language model architectures for time series forecasting. The proposed Chronos is pre-trained on top of the T5 family with public and synthetic datasets with quantization techniques to get tokens. Chronos models achieve better performance under in-domain scenarios and on-par zero-shot performance compared with baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Address comments 1-4 above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xUGHOtRGbA", "reviewer": "Reviewer_Tq2a", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. The paper presents a strong foundation model, and is one of the first papers to concurrently show the possibility and benefits of training of a large corpus of time series data.\n2. The paper shows that discretization is an effective approach for modeling time series data. The benefits of such an approach is that it removes the complexities of dealing with real values, and probabilistic forecasting is made simper as the categorical distribution can easily approximate any complex distribution.\n3. The paper performs presents extensive details of model training and datasets. It also contains extensive empirical analysis to understand the strengths and weaknesses of the proposed method.\n\n### Weaknesses\n1. The method can only handle univariate time series.\n2. Prediction length is limited to a maximum of 64. Autoregressive models suffer from error accumulation - it is unclear whether such a method extend to longer horizons.\n3. The paper writing leans heavily into the idea of being motivated by LLMs. It ignores extensive related work of ordinal regression and regression via classification (RvC). Per my understanding, the main difference between this work and the aforementioned is that in this paper, inputs are also discretized, whereas in ordinal regression and RvC, it is mainly the outputs that are discretized.\n4. The writing related to quantization can be significantly improved. Currently it is quite confusing to understand the exact approach. [c_1,c_B], [l=-15, r=15] and [-15s, 15s] have been used to describe the prediction range. This is quite confusing. All the details of the exact approach used in training should be described in one place, and use a unified notation. The vocabulary size of the pre-trained models should also appear in Section 5.2.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a foundation model for time series forecasting. They introduce Chronos, an architecture which leverages the existing T5 architecture from the NLP literature. To bridge the gap between time series data and language, they discretize time series data into discrete tokens. They train the model on a collection of diverse time series data, and show that it has strong capabilities for both in-domain and zero-shot forecasting. They also introduce KernelSynth, a synthetic data generation process for time series data, and show that incorporating synthetic data improves overall performance of the model. Finally they perform extensive empirical analysis and ablations to provide greater understanding of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### Changes critical to securing recommendation for acceptance\n1. TSMixup has some issues that should be clarified:\n    a. Why is mean scaling performed before mixing? Is there any data leakage given that mean scaling is performed on the whole series?\n    b. What does it mean for $\\alpha=1.5$? Is this a vector of length $k$ with all elements as 1.5?\n2. More analysis or at least elaboration about the different scaling methods should be given. Why was mean scaling chosen over min-max, standardization, etc...? It seems that standardization would solve many of the problems related to loss of precision presented in the analysis section, why was mean scaling chosen over standardization?\n\n### Changes that would strengthen the work\n1. Please add in related work section to the extensive literature of ordinal regression and regression via classification. \n2. Please improve the writing regarding quantization.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Really interesting work. This paper is almost ready to be accepted. I have a few minor issues to address.

You say results are averaged over 3 runs. I am trying to understand what that means for Figure 4 and 5. 
1. What is the source of stochasticity here? 
2. What are the 3 runs averaging over, and how variable are outcomes?
3. Are the differences between algorithms significant?

Minor comment: another recent work found shifting from a regression loss to classification when using Transformers in RL allowed for significant improvements (https://arxiv.org/abs/2403.03950). This could be another useful citation, given the comment from a reviewer about regression as classification.
Decision: Accept (poster)