SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Nate Gruver* NYU
Marc Finzi* CMU
Shikai Qiu* NYU
Andrew Gordon Wilson NYU
By encoding time series as a string of numerical digits, we can frame time series forecasting as next-token prediction in text. Developing this approach, we find that large language models (LLMs) such as GPT-3 and LLaMA-2 can surprisingly zero-shot extrapolate time series at a level comparable to or exceeding the performance of purpose-built time series models trained on the downstream tasks. To facilitate this performance, we propose procedures for effectively tokenizing time series data and converting discrete distributions over tokens into highly flexible densities over continuous values. We argue the success of LLMs for time series stems from their ability to naturally represent multimodal distributions, in conjunction with biases for simplicity, and repetition, which align with the salient features in many time series, such as repeated seasonal trends. We also show how LLMs can naturally handle missing data without imputation through non-numerical text, accommodate textual side information, and answer questions to help explain predictions. While we find that increasing model size generally improves performance on time series, we show GPT-4 can perform worse than GPT-3 because of how it tokenizes numbers, and poor uncertainty calibration, which is likely the result of alignment interventions such as RLHF.
Despite similarities with other sequence modeling problems, such as text, audio, or video, time series has two particularly challenging properties. Unlike video or audio, which typically have consistent input scales and sampling rates, aggregated time series datasets often comprise sequences from radically different sources, sometimes with missing values. Moreover, common applications of time series forecasting, such as weather or financial data, require extrapolating from observations that contain a tiny fraction of the possible information, making accurate point predictions nearly impossible and uncertainty estimation especially important. While large-scale pretraining has become a key element of training large neural networks in vision and text, enabling performance to scale directly with data availability, pretraining is not typically used for time series modeling, where there is no consensus unsupervised objective and large, cohesive pretraining datasets are not readily available. Consequently, simple time series methods (e.g. ARIMA [8], and linear models [52]) often outperform deep learning methods on popular benchmarks [24].
In this paper, we demonstrate how large language models (LLM) can naturally bridge the gap between the simple biases of traditional methods and the complex representational learning and generative abilities of modern deep learning. In particular, we introduce an exceedingly simple method,  $\text{LLMTime}^2$ , to apply pretrained LLMs for continuous time series prediction problems, illustrated at a high level in Figure 1. At its core, this method represents the time series as a string of numerical digits, and views time series forecasting as next-token prediction in text, unlocking
2https://github.com/ngruver/llmtime
Figure 1: We propose LLMTIME, a method for time series forecasting with large language models (LLMs) by encoding numbers as text and sampling possible extrapolations as text completions. LLMTIME can outperform many popular time series methods without any training on the target dataset (i.e. zero-shot). The performance of LLMTIME also scales with the power of the underlying base model. Notably, models that undergo alignment (e.g. RLHF) do not follow the scaling trend. For example, GPT-4 demonstrates inferior performance to GPT-3 (Section 6).
the use of powerful pretrained models and probabilistic capacities, such as likelihood evaluation and sampling. To enable strong performance, we propose techniques to (1) effectively encode time series as a string of numerical digits and (2) adapt the discrete distributions of LLMs to continuous densities capable of modeling sophisticated multimodal distributions. Using these techniques, we find LLMTIME can exceed or match purpose-built time series methods over a range of different problems in a zero-shot fashion, meaning that LLMTIME can be used without any fine-tuning on the downstream data used by other models.
The zero-shot nature of LLMTime carries several natural advantages: (1) it facilitates the straightforward application of LLMs, eliminating the necessity for specialized knowledge of fine-tuning procedures and the substantial computational resources required for these procedures, as well as side-stepping access issues surrounding proprietary source code or APIs for LLM training or fine-tuning; (2) it is naturally suited to scenarios with limited data availability, where there is little information for training or fine-tuning; (3) by leveraging the broad pattern extrapolation capabilities of extensively pre-trained LLMs, it circumvents the extensive time, effort, and domain-specific expertise typically required for crafting dedicated time series models.
To understand the origins of LLMTime's impressive performance, we investigate how LLMs express preferences for simple or repetitive sequences [19] and show that these biases are in fact compatible with the salient structure of time series, such as seasonality. Aside from these biases, LLMs also can naturally accommodate missing data, and express multimodal distributions, which is particularly useful for time series. We also show how LLMs enable appealing functionality, such as the ability to provide additional side information through prompting, and query the LLM to explain its predictions.
Finally, in addition to broadly compelling forecasting performance, we find performance tends to improve with scale, and the quality of point predictions also improves with the quality of the uncertainty representation. However, we also find GPT-4 has worse uncertainty calibration than GPT-3, likely due to interventions such as reinforcement learning by human feedback (RLHF).
Language modeling Language models are trained on a collection of sequences,  $\mathcal{U} = \{U_1,U_2,\ldots U_i,\ldots ,U_N\}$ , where  $U_{i} = (u_{1},u_{2},\dots,u_{j},\dots,u_{n_{i}})$  and each token,  $u_{i}$ , belongs to a vocabulary  $\nu$ . Large language models typically encode an autoregressive distribution, in which the probability of each token is only dependent on the previous tokens in the sequence,  $p_{\theta}(U_i) = \prod_{j = 1}^{n_i}p_{\theta}(u_j|u_{0:j - 1})$ . The parameters,  $\theta$ , are learned by maximizing the probability of the entire dataset  $p_{\theta}(\mathcal{U}) = \prod_{i = 1}^{N}p_{\theta}(U_i)$ . Every language model has an associated tokenizer, which breaks an input string into a sequence of tokens, each belonging to  $\nu$ . Proper tokenization is extremely important, and small details can have surprisingly significant effects. The most common tokenization method for autoregressive language models is byte-pair encoding (BPE), which treats inputs like
bit strings and assigns tokens based on the rate of occurrence in the training corpus, optimizing for shorter sequences of tokens on average. Sampling from a language model typically starts with a prompt,  $u_{0:k}$ , and proceeds sequentially using  $p_{\theta}(u_j \mid u_{0:j-1})$ , which is often preprocessed, for example through temperature scaling or nucleus sampling [25].
Large language models Brown et al. [9] showed that increasing a language model's parameter count and training data size leads to new capabilities such as zero-shot generalization, in which a model can perform a text-formatted task without training the model parameters on any task-specific data. Large language models, for example GPT-3 [9] or LLaMA-2 [44], accomplish this form of generalization through in-context learning, which identifies patterns in the language model's prompt and extrapolates them through next-token prediction. Many authors have speculated that in-context learning emerges from a language model's extensive compression of the input data [19, 42, 15]. Compression favors learning algorithms that operate over the input data with programmatic abstractions, for example, context-free grammars [2] or induction heads [34], which can implement copy-and-paste type operations for generating samples with highly structured syntax. In this work, we show that the zero-shot generalization abilities of LLMs and their preference for compressible patterns extend well beyond language understanding and can be used for time series forecasting.
Zero-shot generalization has made LLMs significantly more useful as assistants, leading to the creation of methods to align LLMs with human preferences and instructions, for example reinforcement learning from human feedback (RLHF) [38] and instruction tuning [46]. While key to modern LLMs products, alignment methods can also significantly affect the abilities and calibration of the underlying model [36, 10]. Here we show these methods can also affect forecasting ability.
Time series data Time series data typically takes the exact same form as language modeling data, as a collection of sequences  $\{U_i = (u_1,\dots ,u_j,\dots ,u_{n_i})\}$ , but in time series  $u_{j}$  is numerical. Because language models are built to represent complex probability distributions over sequences, they are theoretically well-suited for time series modeling. In practice, however, language models are held back by the details of tokenizing numbers. BPE compresses numbers based on frequency of occurrence in the training data, so numbers can be broken down into awkward chunks that make learning basic numerical operations challenging. Touvron et al. [43] therefore designed the LLaMA tokenizer to map numbers to individual digits, which can lead to significant improvements in mathematical abilities, with small LLaMA models outperforming GPT-4 [30].
The other challenge of applying language models to time series data is proper evaluation. Mean absolute error (MAE) is commonly used but ignores uncertainty in the forecast which is highly limiting for stochastic data [24, 6]. Continuous ranked probability score (CRPS) captures distributional qualities and can compare models that generate samples without likelihoods. For a single prediction, the CRPS score is defined against the estimated cumulative distribution function (CDF),  $\hat{F}$  as  $\mathrm{CRPS}(\hat{F},y) = \int_{\mathbb{R}}\left(\hat{F} (z) - \mathbb{I}_{(z - y) > 0}\right)^{2}dz$ , where  $\hat{F} (z)$  is the empirical CDF produced by sampling forecasts and  $\mathbb{I}$  is the indicator function. While CRPS is an improvement over MAE, it also ignores key structures in the data, such as correlations between time steps. Fortunately, language models can assign likelihoods to full sequences of time series data, and we show how a small modification to an LLM's discrete likelihood can yield a continuous density that is useful for model comparison.
Language models for time series Several authors have explored using pretrained language model encoders as initializations for time series models. For example, Zhou et al. [56] propose FPT, which finetunes a BERT encoder to perform time series forecasting. Similarly, Zhang et al. [53] introduce Meta-Transformer, a framework for finetuning a language model for non-text modalities, including time series. Fewer papers explore using LLMs as forecasters without finetuning. The only method we are aware of is PromptCast [50], which poses forecasting as question answering with prompting.
Our work Unlike methods that leverage LLM backbones, our method is entirely zero-shot and does not require finetuning. Unlike PromptCast, we show that LLMs can be used directly as forecasters without any added text or prompt engineering, if we carefully preprocess the numerical values themselves. Our method solely relies on LLM's abilities to extrapolate patterns in general sequences and nothing particular to English or any other language. Going beyond prior work, we also cultivate the probabilistic nature of large language models and their ability to capture uncertainty over highly stochastic time series.
"151,167,...,267"
GPT-3 spaces
GPT-3 no spaces
Figure 2: Careful tokenization is important for good forecasting with LLMs. Using the Australian Wine dataset from Darts [23], with values [151, 167, ..., 267], we show the tokenization used by GPT-3 [9] and LLaMA-2 [44] and the corresponding effect on forecasting performance. Added spaces allow GPT-3 to create one token per digit, leading to good performance. LLaMA-2, on the other hand, tokenizes digits individually, and adding spaces hurts performance.
LLaMA spaces
LLaMA no spaces
3 LLMTIME: Forecasting with Language Models
Forecasting with LLMTIME has relatively few steps. Once the numerical values are processed into strings, making predictions with the language model follows standard sampling procedures. As we show next, however, correct pre-processing is not always intuitive but is extremely important, and incorrect handling can lead to unusable predictions.
Tokenization Tokenization is particularly important because it directly influences how patterns form within tokenized sequences and the types of operations that language models can learn. Unfortunately, common tokenization methods like BPE tend to break a single number into tokens that don't align with the digits, which can make arithmetic considerably more difficult [30]. For example, the number 42235630 gets tokenized as [422, 35, 630] by the GPT-3 tokenizer, and changes by even a single digit can result in an entirely different tokenization. By contrast, in many new open-source LLMs (e.g. LLaMA [43]), numbers are tokenized into individual digits by default. To remedy the tokenization of GPT models, we separate the digits with spaces to force a separate tokenization of each digit and use a comma (" ",") to separate each time step in a time series. Because decimal points are redundant given a fixed precision, we drop them in the encoding to save on context length. Thus, with e.g. 2 digits of precision, we pre-process a time series as follows before feeding into the tokenizer:
$$
0. 1 2 3, 1. 2 3, 1 2. 3, 1 2 3. 0 \rightarrow " 1 2 , 1 2 3 , 1 2 3 0 , 1 2 3 0 0".
$$
In Figure 2, we show that the added spaces of this encoding are helpful for GPT models, preventing the model from getting derailed by outputting an unusual token during sampling. For LLaMA models, with their unique tokenization of numbers, added spaces have the opposite effect. Each digit and space is already assigned its own token, and space tokens become nuisance inputs, adding to the sequence length without simplifying the sequence's structure and potentially making the sequence out-of-distribution to the model.
Rescaling To avoid wasting tokens when the inputs are very large, we scale values down so that the  $\alpha$ -percentile of rescaled time series values is 1. We avoid scaling by the maximum value so that the LLM can see some fraction of examples  $(1 - \alpha)$  where the number of digits changes and reproduce this behavior in its outputs to produce larger values than it has seen. We also experiment with an offset  $\beta$  based calculate as a percentile of the input data, and we tune these two parameters on validation log likelihoods (details in Appendix A).
Sampling / Forecasting To forecast, draw many samples (e.g. 20) from the LLM and use the statistics of the samples at each time step to construct a point estimate (e.g. as the median) or probabilistic forecast (e.g. as quantiles). To control sampling, we use temperature scaling, logit bias, and nucleus sampling (Appendix C).
Continuous likelihoods Modeling sequences of individual digits has additional benefits beyond good samples. With  $n$  digits of precision in base  $B$ , each sequence of  $n$  digits after the decimal place corresponds to one of  $B^n$  possible bins (Figure 3), each with width  $B^{-n}$ . As each distribution  $p(u_j \mid u_{0:j-1}; \theta)$  is a softmax over possible digits, we can view the distribution over each individual number as a hierarchical softmax [31], with  $p(u_1, \dots, u_n) = p(u_n | u_{n-1}, \dots, u_0) p(u_1 | u_0) p(u_0)$ . Though a language model's probability distribution is discrete, we can easily adapt it to provide a
Figure 3: Left: Autoregressive models over sequences of digits act like hierarchical softmax distributions over the corresponding numbers. When combined with uniform distributions in each discrete bucket, distributions over strings can become expressive distributions over continuous domains. Right: Using simple autoregressive models (e.g. RNNs) trained on a string representation of numbers, we can fit complex distributions that can be challenging for other methods, such as heavy-tailed or multimodal distributions. A simple autoregressive model can match or outperform well-known methods for density estimation, such as Gaussian mixture models (GMMs) or binning with a fixed resolution, as measured by Wasserstein distance between samples.
continuous density by placing a uniform distribution in each bin. Enumerating each of the countably infinite numbers that the model can produce (because the model can output an arbitrary number of digits before the decimal point) with an index  $k \in \mathbb{N}$  each with probability  $p_k$ , we can write out the distribution as a mixture of disjoint uniform distributions over the bins  $p(x) = \sum_{k \in \mathbb{N}} p_k U_k(x)$  where  $U_k(x) = B^n \mathbb{I}_{x \in [B^{-n}k, B^{-n}(k + 1))}$ . Therefore if a given data point lies in bin  $k$ , its continuous log likelihood is  $\log p(x) = \log p_k + n \log B$ . Finally, to obtain the likelihood  $\log p(z)$  in the original input space, we add a change of variables factor  $\log |\frac{dx}{dz}|$ , where  $z \mapsto x = s(z)$  is the rescaling operation in the pre-processing. As a result, the exponentially large number of bins and exponentially small bin widths enabled by our construction make it surprisingly efficient to represent flexible and high-resolution continuous distributions with LLMs, despite using a discrete tokenization of numbers.
Language models as flexible distributions The fact that LLMs can express flexible distributions over numbers is key for time series data. Uncertainty quantification is essential to forecasting, and typical approaches to representing uncertainty in time series can be limited by misspecification. For example, one common method for creating a probabilistic forecast is to fit a Gaussian or Laplace observation model. When the underlying data distribution is multimodal, both of these models will perform poorly. Methods like Gaussian mixture models (GMMs) solve the issue of multimodality but introduce additional challenges to optimization and model selection. We show that a language model is an underrated solution by training a small autoregressive model on a variety of one-dimensional distributions shown in Figure 3 (right). These distributions come from an exponential random variable, a mixture of a uniform and a student-t distribution, and the heavy-tailed distribution of time series prediction residuals from an ARIMA model on the MonthlyMilk dataset [23]. We evaluate these fits quantitatively by computing Wasserstein distances, and compare to a Laplace observation model, a GMM trained with expectation-maximization, and logistic regression over a flat binning of the data (with a tuned bin size). Each model is trained with only 200 samples from the distribution. The results show that the decimal autoregressive language model ("Decimal AR") performs extremely well, handling asymmetric, multimodal, and heavy-tailed distributions, which are among the diverse types characteristic of time series data.
We evaluate the zero-shot forecasting ability of LLMs by comparing LLMTime with GPT-3 and LLaMA-2 70B to many popular time series baselines on a variety of benchmark time series datasets. Not only is LLMTime able to generate plausible completions of the real and synthetic time series, it achieves higher likelihoods and CRPS values in zero-shot evaluation than the dedicated time series models like ARIMA, TCNs, and N-HiTS. When evaluated on deterministic metrics like MAE, LLMs also perform well, obtain the best or second best MAE values on each benchmark. As we are using
Figure 4: LLMTIME with base model GPT-3 or LLaMA-2 70B has the best or second best aggregated performance on several deterministic time series benchmarks [23, 18, 54] while being entirely zero-shot. Collectively, these benchmarks comprise 29 individual datasets with diverse sources, lengths, and noise levels. For Monash MAE numbers, established results are reported on unnormalized data, so we normalize values before aggregating (Appendix C.2). The informer datasets are multivariate, and we predict each covariate independently with LLMTIME (Appendix C.3). GPT-3 evaluation on the Informer datasets was skipped because of the cost of API queries. Error bars show standard errors over the individual datasets in each benchmark.
LLMs with undisclosed datasets, data leakage is an important concern that we address directly in Appendix B. Beyond strong performance on standard benchmarks, which are the most useful for comparison, we find that LLMTime also performs well on datasets that could not have been present in the base model's training data. The full set of hyperparameters used for LLMTime and the baseline methods are detailed in Appendix C.1. For some of the longer time series, not all of the history can be fit into the context window, and hence hyperparameters implicitly capture the trade-off between higher precision and capturing a larger temporal history.
Datasets We use three benchmark datasets that are common within deep learning research and many baseline methods that accompany the benchmark datasets.
- Darts [23]: A collection of 8 real univariate time series datasets. For Darts, we use several methods that are implemented directly in the package, including neural network models (TCN [28], N-BEATS [37], N-HiTS [11]) and simple moving average models (ARIMA [8]). Darts enables learning observation models with tractable likelihoods and is therefore especially useful for benchmarking the probabilistic predictions of LLMTime. We also include Spectral Mixture Gaussian Process (SM-GP) [48], a Bayesian nonparametric method (details in Appendix C.1).
- Monash [18]: The Monash forecasting archive contains 30 publicly available datasets along with baseline numbers for 12 forecasting models, including simple exponential smooth (e.g. ETS [26]), gradient boosting (e.g. CatBoost [39]) and deep learning models (e.g. DeepAR [40], WaveNet [35]). The Monash archive comprises over 400,000 individual time series, making it infeasible to use in its entirety with the largest available LLMs. To reduce the computational burden, we evaluate GPT-3's zero-shot performance on 19 datasets described in Appendix C.2.
- Informer [54]: We evaluated on multivariate datasets widely used for benchmarking efficient transformer models [16, 54]. In order to predict multivariate data with LLMTime, we forecast each covariate independently. We baseline against numbers obtained by running public implementations from the Autoformer [49] and FEDFormer [55] codebases (Appendix C.3).
Deterministic results To compute MAE values for LLMTime we use the pointwise median of 20 samples from the base model (GPT-2 or LLaMA-2 70B). Figure 4 shows that deterministic predictions from LLMTime are ranked best or second best on all the considered benchmarks while having no trainable parameters. We provide visualizations of the forecasts in Appendix C.5/C.7/C.8.
Probabilistic results In Figure 5, we show several probabilistic evaluations on the Darts datasets, including aggregated NLL and CRPS numbers, as well as analysis of how each model reacts to decreasing the input data size. Evaluated on log likelihood and CRPS, LLMTime considerably outperforms the baselines in aggregate and on almost every individual dataset (results per dataset included in Appendix C.5). Given the analysis of language model-derived densities in Section 3, it is unsurprising that language models excel in probabilistic evaluations, outperforming the baselines even