Large Language Models Are Zero-Shot Time Series Forecasters
Authors: Nate Gruver
Marc Anton Finzi
Shikai Qiu
Andrew Gordon Wilson
Venue: NeurIPS 2023 poster (2023)
Keywords: large language models
time series
probabilistic forecasting

Abstract: By encoding time series as a string of numerical digits, we can frame time series forecasting as next-token prediction in text. Developing this approach, we find that large language models (LLMs) such as GPT-3 and LLaMA-2 can surprisingly zero-shot extrapolate time series at a level comparable to or exceeding the performance of purpose-built time series models trained on the downstream tasks. To facilitate this performance, we propose procedures for effectively tokenizing time series data and converting discrete distributions over tokens into highly flexible densities over continuous values. We argue the success of LLMs for time series stems from their ability to naturally represent multimodal distributions, in conjunction with biases for simplicity, and repetition, which align with the salient features in many time series, such as repeated seasonal trends. We also show how LLMs can naturally handle missing data without imputation through non-numerical text, accommodate textual side information, and answer questions to help explain predictions.  While we find that increasing model size generally improves performance on time series, we show GPT-4 can perform worse than GPT-3 because of how it tokenizes numbers, and poor uncertainty calibration, which is likely the result of alignment interventions such as RLHF.

Large Language Models Are Zero-Shot Time Series Forecasters
Nate Gruver" NYU
Mare Finzi* CMU
Shikai Qiu NYU
Andrew Gordon Wilson NYU
Abstract
By encoding time series as string of numerical digits, we can frame time series forecasting as next-token prediction in text. Developing this approach, we find that large language models (LLMs) such as GPT-3 and LLaMA-2 can surprisingly zero- shot extrapolate time series level comparable of purpose-buil time series models trained on the downstream tasks. To facilitate this performance, we propose procedures for effectively tokenizing time series data and converting discrete distributions over tokens into highly flexible densities over continuous values. We argue the success of L.LMs for time series stems from their ability to naturally represent multimodal distributions, in conjunction with biases for simplicity, and repetition, which align with the salient features in many time series, such as repeated seasonal trends. We also show how LLMs can naturally handle missing data without imputation through non-numm numerical text, accommodate textual side information, and answer questions to help explain predictions. While we find that increasing model size generally improves performance on time series, show GPT-4 perform worse than GPT-3 because of how it tokenizes numbers, and poor uncertainty calibration, which is likely the result of alignment interventions such RLHF.
exceeding the performance
Introduction
Despite similarities with other sequence modeling problems, such text, audio, video, time series has two particularly challenging properties, Unlike video or audio, which typically have consistent input scales and sampling rates, aggregated time series datasets often comprise sequences from radically different sources, sometimes with missing values. Moreover, common applications of time series forecasting, such as weather or financial data, require extrapolating from observations that contain tiny fraction of the possible information, making accurate point predictions nearly impossible and uncertainty estimation especially important. While large-scale pretraining has become key element of training large neural networks in vision and text, enabling performance to scale directly with data availability, pretraining is not typically used for time series modeling, where there no consensus unsupervised objective and large, cohesive pretraining datasets are not readily available. Consequently, simple time series methods (e.g. ARIMA [8], and linear models [52]) often outperform deep learning methods on popular benchmarks [24].
In this paper, we demonstrate how large language models (LLM) can naturally bridge the gap between the simple biases of traditional methods and the complex representational learning and generative abilities of modern deep learning. In particular, we introduce an exceedingly simple method, LLMTIME", to apply pretrained LLMs for continuous time series prediction problems, illustrated high level in Figure of numerical digits, and views time series forecasting next-token prediction in text, unlocking
core, this method represents the time series as string
"Equal contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Samples
Prompt
N-BEATS
"and fell asleep" "and bit my leg"
N-H15
"The dog jumped up the bed"
ARUMA
LLM
0.00
0.05
D.10
0.15
0.20
"631, 656, 650, 4877.485.495
"479, 371, 364"
CRPS
"492. ... 499, 501"
0375
ho Myn
0.325
MMLUU Accuracy
We propose LLMTIME, a method for time series forecasting with large language models (LLMs)) by encoding numbers as text and sampling possible extrapolations as text completions. LLMTIME can outperform many popular time series methods without any training on the target dataset (i.e. zero-shot). The performance of L.M'TIME also scales with the power of the underlying base model. Notably, models that undergo alignment (e.g. RLHF) do not follow the scaling trend. For example, GPT-4 demonstrates inferior performance to GPT-3 (Section 6).
Figure 1:
the use of powerful pretrained models and probabilistic capacities, such as likelihood evaluation and sampling. To enable strong performance, we propose techniques to (1) effectively encode time series as a string of numerical digits and (2) adapt the discrete distributions of LLMs to continuous densities capable of modeling sophisticated multimodal distributions. Using these techniques, we find LLMTIME problems in a zero-shot fashion, meaning that LLMTIME can be used without any fine-tuning on the downstream data used by other models.
exceed or match purpose- crrpos--biilt time series methods over range of different
The zero-shot nature LLMTIME carries several natural advantages: (1) it facilitates the straight- forward application of LLMs, eliminating the necessity for specialized knowledge of fine-tuning procedures and the substantial computational resources required for these procedures, as well as side- stepping access issues surrounding proprietary source code or APIs for LLM training or fine-tuning; (2) it is naturally suited to scenarios with limited data availability, where there is little information for training fine-tuning; (3) by leveraging the broad pattern extrapolation capabilities of extensively pre-trained LLMs, it circumvents the extensive time, effort, and -ppeiii expertise typically required for erafting dedicated time series models.
To understand the origins of LLMTIME's impressive performance, we investigate how LLMs express preferences for simple or repetitive sequences [19] and show that these biases are in fact compatible with the salient structure of time series, such as seasonality. Aside from these biases, LLMs also can naturally acccommodate missing data, and express multimodal distributions, which is particularly useful for time series. We also show how LLMs enable appealing functionality, such as the ability to provide additional side information through prompting, and query the LLM to explain its predictions.
Finally, in addition broadly compelling forecasting performance, to improve with scale, and the quality of point predictions also improves with the quality of the uncertainty representation. However, we also find GPT-4 has worse uncertainty calibration than GPT-3, likely due to interventions such as reinforcement learning by human feedback (RLHF).
find performance tends
2 Background
Language modeling Language models are trained on a collection of sequences, 4 U2,UU,... UG, U,,..., U. where U; = (Wi, u2,.... Mj.. Mj...., Un,) and each token, u,, belongs to a vocabulary V. Large language models typically encode an autoregressive distribution, in which the probability of each token is only dependent on the previous tokens in the sequence, Po (U,) = I[ Pe (u, u0: The parameters, 6, are learned by maximizing the probability of the entire dataset (24) = I po (U,). Every language model has an associated tokenizer. which breaks an input string into sequence of tokens, each belonging to V. Proper tokenization is extremely important, and small details can have surprisingly significant effects. The most common tokenization method for autoregressive language models is byte-pair encoding (BPE), which treats inputs like

---

bit strings and assigns tokens based on the rate of occurrence in the training corpus. optimizing for shorter sequences of tokens on average. Sampling from a language model typically starts with prompt, uo:k, and proceeds sequentially using po (#z Mooj 1), which is often preprocessed, for example through temperature scaling or nucleus sampling [25].
Large language models Brown et al. [9] showed that increasing a language model's parameter count and training data size leads new capabilities such as zero-shot generalization, in which a model can perform a text-formatted task without training the model parameters on any task-specific data. Large language models, for example GPT-3 [9] or LLaMA-2 [44], accomplish this form of generalization through in-context learning, which identifies patterns in the language model's prompt and extrapolates them through next-token prediction. Many authors have speculated that in-context learning emerges from a language model's 's extensive compression of the input data [19, 42, 15]. Compression favors learning algorithms that operate over the input data with programmatic abstractions, for example, context-free grammars [2] or induction heads [34], which can implement copy-and-pasee type operations for generating samples with highly structured syntax. In this work, we show that the zero-shot generalization abilities of LLMs and their preference for compressible patterns extend well beyond language understanding and can be used for time series forecasting.
Zero-shot generalization has made LLMs significantly more useful as assistants, leading to the create of methods to align LLMs with human preferences and instructions, for example reinforcement learning from human feedback (RLHF) [38] and instruction tuning [46]. While key to modern LLMs products, alignment methods can also significantly affect the abilities and calibration of the underlying model [36, 10]. Here we show these methods can also affect forecasting ability.
Time series data Time series data typically takes the exact same form language modeling data, as a collection of sequences {U; = (u,,..., uj:.... (n but in time series uj is numerical. Because language models are built represent complex probability distributions over sequences, they are theoretically well-suited for time series modeling. In practice, however, language models are held back by the details of tokenizing numbers. BPE compresses numbers based on frequency of occurrence in the training data, so numbers can be broken down into awkward chunks that make learning basic numerical operations challenging. Touvron et al. [43] therefore designed the LLaMA tokenizer to map numbers to individual digits, which can lead to significant improvements in mathematical abilities, with small LLaMA models outperforming GPT-4 [30].
The other challenge of applying language models to time series data is proper evaluation. Mean absolute error (MAE) is commonly used but ignores uncertainty in the forecast which is highly limiting for stochastic data [24, 6]. Continuous ranked probability score (CRPS) captures distributional qualities and can compare models that generate samples without likelihoods. For a single prediction, the CRPS score is defined against the estimated cumulative distribution function (CDF). F as
CRPS(F, v) = Jx. (f(z) - (o--)>0) 2 dz, where F(z) is the empirical CDF produced by sampling forecasts and I is the indicator function. While CRPS is an improvement over MAE, it also ignores key structures in the data, such as correlations between time steps. Fortunately, language models can assign likelihoods to full sequences of time series data, and we show how a small modification to an LLM's discrete likelihood can yield a continuous density that is useful for model comparison.
Language models for time series Several authors have explored using pretrained language model encoders as initializations for time series models. For example, Zhou et al. [56] propose FPT, which finetunes BERT encoder perform time series forecasting. Similarly, Zhang et al. [53] introduce Meta- Transfrranfo,,,,,, a framework for finetuning a language model for non-text modalities, including time series. Fewer papers explore using LLMs as forecasters without finetuning. The only method we are aware of is PromptCast [50]. which poses forecasting as question answering with prompting.
Our work Unlike methods that leverage LLM backbones, our method is entirely zero-shot and does not require finetuning. Unlike PromptCast, we show that LLMs can be used directly as forecasters without any added text or prompt engineering, if we carefully preprocess the numerical values themselves. Our method solely relies on LLM's abilities to extrapolate patterns in general sequences and nothing particular to English or any other language. Going beyond prior work, we also cultivate the probabilistic nature of large language models and their ability to capture uncertainty over highly stochastic time series.
3

---

"151. 167 .67"
"151,167....255
115.167, 267
"151.16.....277
1151. 11. 167. ...., 266
1151,161 166,.....22... .6..267"
11 167 ...267
111,167....22 16.7....667"
GPT-3 spaces
GPT-3 spaces
LLaMA spaces
LLaMA no spaces
Figure 2: Careful tokenization is important for good forecasting with LLMs. Using the Australian Wine dataset from Darts [23], with values [151, 167, 267]. we show the tokenization used by
GPT-3 [9] and LLaMA-2 [44] and the corresponding effect forecasting performance. Added spaces allow GPT-3 to create one token per digit, leading to good performance. LLaMA-2, on the other hand, tokenizes digits individually, and adding spaces hurts performance.
3 LLMTIME: Forecasting with Language Models
Forecasting with LLMTTIME has relatively few steps. Once the numerical values are processed into strings, making predictions with the language model follows standard sampling procedures. As show next, however, correct pre-processing is not always intuitive but extremely important, and incorrect handling can lead unusable predictions.
Tokenization Tokenization is particularly important because it directly influences how patterns form within tokenized sequences and the types of operations that language models can learn. Unfortunately, common tokenization methods like BPE tend to break single number into tokens that don't 't align with the digits, which make arithmetic considerably more difficult [30]. For example, the number 42235630 gets tokenized as [422, 35, 630) by the GPT-3 tokenizer, and changes by even a single digit result in an entirely different tokenization. By contrast, in many new open-source L.MSs (e.g. LLaMA [43]), numbers are tokenized into individual digits by default. To remedy the tokenization of separate the digits with spaces force a separate tokenization of each digit and
GPT models, use a comma .""") to separate each time step in a time series. Because decimal points are redundant given a fixed precision, we drop them in the encoding to save on context length. Thus, with e.g. digits of precision, we pre-process a time series as follows before feeding into the tokenizer:
0.123, 1.23, 12.3, 123.0 - 1 2,, 23, 2 3 0, 2 23000.
In Figure 2, we show that the added spaces of this encoding are helpful for GPT models, preventing the model from getting derailed by outputting an unusual token during sampling. For LLaMA models, with their unique tokenization of numbers, added spaces have the opposite effect. Each digit and space is already assigned its own token, and space tokens become nuisance inputs, adding to the sequence length without simplifying the sequence's 's structure and potentially making the sequence out-of-distribution the model.
Rescaling To avoid wasting tokens when the inputs are very large, we scale values down so that the a-percentile of rescaled time series values is 1. We avoid scaling by the maximum value so that the LLM can see some fraction of examples (1 - a) where the number of digits changes and reproduce this behavior in its outputs produce larger values than it has seen. We also experiment with an offset 3 based calculate as a percentile of the input data, and we tune these two parameters on validation log likelihoods (details in Appendix A).
Sampling Forecasting To forecast, draw many samples (c.g. 20) from the LLM and use the statistics of the samples at each time step to construct point estimate (c.g. the median) or probabilistic forecast (c.g. as quantiles). To control sampling, we use temperature scaling, logit bias, and nucleus sampling (Appendix C).
Modeling sequences of individual digits has additional benefits beyond good samples. With digits of precision in base B, each sequence of n digits after the decimal place corresponds to one of Td possible bins (Figure 3), each with width B-" As each distribution p(uz | Mo:j-1; :):j--:0) () is softmax over possible digits, we can view the distribution over each individual number as a hierarchical softmax [31], with p(u1,..., un) = p(un (p ..., 24) p(u) I (uo) p( (uo)). Though a language model's probability distribution is discrete,
Continuous likelihoods
easily adapt it to provide

---

Exponentia
0.537
P("5")
-0.25
Wassersteln Distance
Encoder
P(0.537) =
Square Student
0.00
"537"
P('5") -
P("3"''5")
Donsity
0.2
Tokenizer
P("""""")
5(7''53")
0.0
["5","3";7"
P(7"''53")
ARIMA Residusis
P(0.537)
0.25
Language Model
0.00
P(uj (u0j-1)
Laplace
Fixed Bins
0.2
0.8
GMM
Decina AR
Figure 3: Left: Autoregressive models over sequences of digits act like hierarchical softmax dis- tributions over the corresponding numbers. When combined with uniform distributions in each discrete bucket, distributions over strings can become expressive distributions over continuous do- mains. Right: Using simple autoregressive models (e.g. RNNs) trained on a string representation of numbers. we can fit complex distributions that can be challenging for other methods, such as heavy-tailed or multimodal distributions. A simple autoregressive model can match or outperform well-known methods for density estimation, such as Gaussian mixture models (GMMs) or binning with fixed resolution, as measured by Wasserstein distance between samples.
continuous density by placing a uniform distribution in each bin. Enumerating each of the countably infinite numbers that the model can produce (because the model can output an arbitrary number of digits before the decimal point) with an index & E N each with probability Pk, we can write out the distribution as a mixture of disjoint uniform distributions over the bins p(x) Exen PsUx (z) where
(=(=)= = L " B-*K, 1))- Therefore if given data point lies in bin its continuous log
likelihood is log p(x) log Pk log B. Finally, obtain the likelihood log p(z)
z) in the original
s(2) is the rescaling operation in the pre-processing. As result, the exponentially large number of bins and exponentially small bin widths enabled by our construction make it surprisingly efficient represent flexible and high-resolution continuous distributions with LLMs, despite using a discrete tokenization of numbers.
input space,
add change of variables factor log de where 2
Language models flexible distributions The fact that LLMs express flexible distributions over numbers is key for time series data. Uncertainty quantification is essential to forecasting, and typical approaches representing uncertainty in time series can be limited by misspecification. For example, one common method for creating a probabilistic forecast is to fit a Gaussian or Laplace observation model. When the underlying data distribution is multimodal, both of these models will perform poorly. Methods like Gaussian mixture models (GMMs) solve the issue of multimodality but introduce additional challenges to optimization and model selection. We show that language model is an underrated solution by training a small autoregressive model on a variety of one-dimensional distributions shown in Figure (right). These distributions come from an exponential random variable, a mixture of a uniform and a student-t distribution, and the heavy-tailed distribution of time series prediction residuals from an ARIMA model on the MonthlyMil& dataset [23]. We evaluate these fits quantitatively by computing Wasserstein distances, and compare to a Laplace observation model, and logistic regression over a flat binning of the
GMM trained with iopectation-mxximiaae data (with a tuned bin size). Each model is trained with only 200 samples from the distribution. The results show that the decimal autoregressive language model ("Decimal AR") performs extremely well, handling asymmetric, multimodal, and heavy-tailed distributions, which are among the diverse types characteristic of time series data.
4 Experiments
We evaluate the zero-shot forecasting ability of LLMs by comparing LLMTIME with GPT-3 and LLaMA-2 70B to many popular time series baselines on a variety of benchmark time series datasets. Not only L.MTTME able to generate plausible completions of the real and synthetic time series, achieves higher likelihoods and CRPS values in zero-shot evaluation than the dedicated time series models like ARIMA, TCNs, and N-HSTS. When evaluated on deterministic metries like MAE, L.MSs also perform well, obtain the best or second best MAE values each benchmark. As we are using
5

---

Darts
Monash
Informer
Scaled MAE
2
0.4
MAE
0.2
MAE
1
0.2
0.0
0.0
N N-BEATS
TCN SM-
N-HITS LLaMA-2
GPT-3
ARIMA
Transformes
(DHR- IDHR-JARIMA
SES Theta
JAIMMA
DeepAR
PR GPT-3 FFNN N- N-BEATS
CatBoost
ETS TBATS
LLaMA-2
Tennsformer
ALaMA-2 FEDformer
Figure 4: LLMTIME with base model GPT-3 or LLaMA-2 70B has the best or second best aggregated performance on several deterministic time series benchmarks [23, 18, 54] while being entirely zero- shot. Collectively, these benchmarks comprise 29 individual datasets with diverse sources, lengths, and noise levels. For Monash MAE numbers, established results are reported on unnormalized data, so we normalize values before aggregating (Appendix C.2). The informer datasets are multivariate, and we predict each covariate independently with LLMTIME (Appendix C.3). GPT-3 evaluation on the Informer datasets was skipped because of the cost of API queries. Error bars show standard errors over the individual datasets in each benchmark.
LLMs with undisclosed datasets, data leakage is an important concern that we address directly in Appendix B. Beyond strong performance on standard benchmarks, which are the most useful for comparison, we find that LLMTIME also performs well on datasets that could not have been present in the base model's training data. The full set of hyperparameters used for LLMTIME and the baseline methods are detailed in Appendix C.I. For some of the longer time series, not all of the history can be fit into the context window, and hence hyperparameters implicitly capture the trade-off between higher precision and capturing a larger temporal history.
Datasets We three benchmark datasets that common within deep learning research and many baseline methods that accompany the benchmark datasets.
Darts [23]: A collection of real univariate time series datasets. For Darts, we use several meth- ods that are implemented directly in the package, including neural network models (TCN [28], N-BEATS [37], N-HiTS [II]) and simple moving average models (ARIMA [8]). Darts enables learning observation models with tractable likelihoods and is therefore especially useful for benchmarking the probabilistic predictions of LLMTIME. We also include Spectral Mixture Gaussian Process (SM-GP) [48]. Bayesian nonparametric method (details in Appendix C.I).
Monash [18]: The Monash forecasting archive contains 30 publicly available datasets along with baselinc numbers for 12 forecasting models, including simple exponential smooth (c.g. ETS [26]). gradient boosting (c.g. CatBoost [39]) and deep learning models (e.g. DeepAR [40], WaveNet [35]). The Monash archive comprises over 400,000 individual time series, making it infeasible use in its entirety with the largest available LLMs. To reduce the computational burden, we evaluate GPT-3's shot performance on 19 datasets described in Appendix C.2.
Informer [54]:
We evaluated on multivariate datasets widely used for benchmarking efficient transformer models [16, 54]. In order to predict multivariate data with LLMTIME, we fore- cast each covariate independently. We baseline against numbers obtained by running public implementations from the Autoformer [49] and FEDFormer [55] codebases (Appendix C.3).
Deterministic results To compute MAE values for LLMTIME we use the pointwise median of 20 samples from the base model (GPT-2 LLaMA-2 70B). Figure 4 shows that deterministic predictions from LLMTIME are ranked best or second best on all the considered benchmarks while having trainable parameters. We provide visualizations of the forecasts in Appendix C.5/C.7/C.8.
Probabilistic results In Figure 5, we show several probabilistic evaluations on the Darts datasets, including aggregated NLL and CRPS numbers, as well as analysis of how each model reacts to decreasing the input data size. Evaluated on log likelihood and CRPS, L.MMTIME considerably outperforms the baselines in aggregate and on almost every individual dataset (results per dataset included Appendix C.5). Given the analysis of language model-derived densities in Section 3, is unsurprising that language models excel in probabilistic evaluations, outperforming the baselines even

---

AirPassengers
NLL/D
w
10
NLL/D
CRPS
NLLID
GasRateCO2
NLLID
0.0
0.5 1.0
PrompiCast (LLaMA)
PromptCast (GPT-3)
LLMTime (LLaMA)
LLMTime (GPT-3)
Train Fraction
SM-GP
N-BEATS
TCN
N-HITS
ARIMA
Figure 5: Extended experiments on the Darts datasets. Left: Example probabilistic forecasts with baseline negative log likelihood per dimension (NLL/D). LLMs easily extrapolate trends (e.g. AirPassengers) and reproduce local patterns when data is noisy (e.g. GasRateCO2). Center: When using probabilistic metries like NLL and CRPS, LLMTIME outperforms all baselines, including PromptCast [50], a competing L.M method. Error bars show standard errors over datasets with Darts. Right: LLMTIME much more sample efficient than competing methods. While the performance of other methods degrades rapidly when we restrict them to a fraction of the original training set, LLMTIME can assign high likelihood with only a few examples.
more dramatically. In Figure 5 (left) we show two informative examples that capture the performance of LLMTIME. When extrapolating the AirPassengers dataset, LLMTIME successfully identifies and continues trend and period components, with uncertainty that grows predictions get further from the input data. On GasRateCO2, LLMTIME replicates local structure when there relatively little global structure. In Figure (right) we show that LLMTIME not only performs better than baselines with access to the full training data but also when restricted to small fractions the training data. As time series is frequently characterized by relative data scarcity and challenges in transfer learning, the data efficiency of LLMs is especially attractive.
Comparison with PromptCast Though included in the results described above, we want to explicitly highlight that LLMTIME significantly outperforms PrompiCast [50] when applied to both GPT-3 and LLaMA-2 70B, according to CRPS and MAE aggregated over the Darts datasets. This performance gap highlights important differences between the two approaches. Unlike our method, PromptCast formulates forecasting pre-trained language models with an explicit question about future values in a time series. For example, PromptCast feeds in the prompt "The values in the WoolyDataset for the past 95 time steps are 6172, 6709, 6633, 6077. What will the values for the next time steps be? The values for the next time steps will be". to extract predictions from LLM. PromptCast also does not apply our tokenization and data rescaling strategy (Section 3), which we show is crucial for good performance.
conventional question-answernee
task in NLP by prompting
5 Origins of Zero-Shot Performance
To understand why LLMs can extrapolate time series in a zero-shot manner, let's take step back and consider simple numerical sequences, for example [1, 4,9, 16.... or [0, 0, 0, 1, 0. 3. For any input sequence, there are arbitrarily many generation rules that are consistent with the input (e.g. f(x) = x² for E [1, 2, ...)), but some generation rules are overly complex and will generalize poorly. LLMs can forecast effectively because they prefer completions derived from simple rules, adopting form of Occam's 's razor prior [19, 42, 15]. To explicitly demonstrate this phenomenon, we create a synthetic example using the function f(x) = x + cos(x) with additive Gaussian noise. We fit symbolic expressions to the first 70% of timesteps using PySR [14] with symbols ["+", "." "_", "/", "sin". "cos", "cxp". ""suurrr"""" to identify generating rules with known complexity, quantified by the number of symbols in the regressed expression (Appendix D). Figure 6 (left) shows the likelihood that GPT-3 assigns the highest likelihood to symbolic regression generating rules that balance consistency with complexity.
In Figure (right) we show how program induction L.LMs leads to good zero-shot prediction for many deterministic patterns common in time series data. Along with samples, also show
7

---

Exponontial
Complesity: 3
AAMA Complesity: 9
-1.0
Normalized Value
10
10
NAAAA Complexity: 13
-0.5
Log Linehooo
Sigmoid
Sine
10
10
m0m Complexity 23
Beat Interference
Linear Sine
2 3
10
10
mmm Complexity: 60
Train Loss Complexity
Teat Loss
LLM NIL
TCN
ARIMA
GPT-3
Ground Truth
CPT-3 Median
Figure 6: LLMs can find low complexity explanations of the data, enabling them to zero-shot extrapolate numerical sequences. Left: GPT-3 likelihoods favor solutions from symbolic regression (PySR [14)) that balance training loss and complexity, leading to good generalization. Right: GPT-3 predicted median and 10-90th percentile prediction interval are shown given 140 timesteps of context. On the right of each time series, we show the log likelihoods compared to the ARIMA and TCN time series models. Overall, GPT-3 performs considerably better than the baselines, though composition and exponential growth are more challenging for the models (Appendix D.1).
likelihoods, comparing against standard time series models, which often struggle to extrapolate these simple patterns because they cannot identify programmatic generation rule to make predictions unlike those seen the observed history. While the generic simplicity bias is helpful for identifying and extrapolating patterns in the input, number of patterns common in time series models also translate directly to known capabilities of language models, for example
Repetition bias and periodicity: LLMs' bias towards repetitive sequences [25] (often unwanted in NLP) corresponds precisely the ability to identify and extrapolate periodic structure in the input. 4.2,8.6, 1.0, 4.2, 8.6 will lead to a 1.0 as a likely next output without any time series or arithmetic knowledge (I,- Tz-T).
Arithmetic and trend components: LLMs' ability to perform addition and multiplication [51, 30] maps on to extrapolating linear and exponential trends. For example, predicting the next element of 0.2, 1.6, 3., 4.4 the LLM needs only to add 1.4 to the last element (I'+11 = Iz + c). Similarly, exponential trends have the generating rule I++1 = c Ia and sigmoid trends have the generating rule r++1 = + cr, x.1).
Combining multiple patterns together presents a more difficult challenge, as it requires both iden- tifying the composite pattern and being able to perform the multiple operations within the same token budget. Supposing that model can perform copying in a single forward pass and addition in a single forward pass, that does not necessarily mean that it can do both simultaneously. We find that GPT-3 is only sometimes able to perform these compositions, though GPT-4 does more consistently as shown in Appendix E. It is likely that the limitations on compute and tokens spent may make this composition unnecessarily hard, and that additional recursive structure, for example from a scratchpad [33], Chain of Thought (CoT) prompting [47], or adaptive computation [41, 3], would make this task easier.
6 Special Properties of LLMs
So far we've ve shown that LLMs are effective forecasters across variety of datasets and that their fore- casting ability arises from biases created by generative pretraining. LLMTIME offers a mechanism for large-scale pre-training that uncommon in machine learning for time series. LLMs lessen the amount of time series data that must be aggregated for pretraining, substituting text pretraining in its place, and enable more powerful scaling results. Beyond escaping the limits of task-specific data, text pretraining also has many test-time benefits that stem from the base model's ability to process and generate natural language. As we show the following section, L.MSs can leverage their abilities in order to seamlessly incorporate missing data or answer questions about time series.
8

---

OpenAAl
Base
Chat
4.8
0.16
Li LttA
CRPS
CRPS
CRPS
0.14
0.1
4.4
0.12
0.10
0.
0.0
0.4 0.6
0.4
0.8
GPT-3 OPT-
7B
13B
70B
7B
13B
706
MMLU Accuracy
MMLU Accuracy
LLaMA Size
LLaMA Size
Figure 7: Left: Time series forecasting performance (NLL/D and CRPS on Darts [23]) improves with reasoning performance of the underlying model LLM, as judged by accuracy on the Massive Multitask Language Understanding (MMLU) benchmark [22]. Displayed results are for all GPT-3, LLaMA [43], and LLaMA-2 [44] base models. Center: GPT-4 performs worse than GPT-3. Right: Forecasting performance (NLL/D and CRPS on Darts) appears be negatively affected by alignment procedures (e.g. instruction tuning and RLHF) in general. LLaMA-2 chat models typically perform worse than the corresponding base model. Error bars show standard errors over individual datasets.
Base models and forecasting performance Given the rapid growth and improvement in open- source L.LMs [43, 44), the relationship between LLMTIME forecasting performance and the per- formance of the underlying base model is particular important and has broad implications. Steady increases in LLM benchmark performance can directly translate to steady improvements in forecast- ing ability. In Figure 7 (right), we show a study with OpenAl models (davinci, babbage, curie, ada), variants of LLaMA [43] (7B, 13B, 33B, 65B) and LLaMA-2 [44] models (7B, 13B, 70B) measuring accuracy on the Massive Multitask Language Understanding (MMLU) benchmark and probabilistic forecasting error. As we might hope, when reasoning (MMLU) performance increases forecasts also improve.
Chat models Though convenient scaling relationships appear to hold for base models, the corre- lation begins break down when we consider models that have been post-processed for chatbot applications. GPT-4 [36]. for example, demonstrates considerably more intelligence than GPT-3 and LLaMA models in natural language tasks. but effectively applying to time series is challenging. In Figure 7 (center), we show that GPT-4 has a forecasting error (CRPS) significantly larger than GPT-3's on the Darts datasets. The performance drop is the result of several small details in GPT-4's method. Due to the altered tokenization, GPT-4 cannot be easily forced to tokenize individual digits into an unbroken stream of numbers. Due the restrictions on the API, likelihood evaluations are also not permitted, which is why we present results for only CRPS. While GPT-4 can perform well on the synthetic examples discussed Section (shown Appendix E), we find that the GPT-4 calibration is much worse than GPT-3's on stochastic data, likely as a result of the preprocessing details above and the fact that the model has been treated with RLHF [13] which is known degrade calibration on anestion-answering tasks [36]. GPT-4 is not the only example of degraded performance in models designed for chat functionality. We observed the same phenomenon in LLaMA-2 models, which have corresponding chat versions for every model size. Figure 7 (right) shows that chat versions tend to have markedly worse forecasting error than their non-chat counterparts, though still maintain trends in size and reasoning ability.
Missing data major advantage of treating forecasting as a text completion task and using LLMs is that we can easily feed in any input that can be encoded as text. Often in time series, the time series will be incomplete and certain observations are missing. Simple imputation methods, such as nearest neighbor, are still core pre-processing steps in common data science workflows [32]. and the choice of imputation method is especially relevant to clinical data, which often contains irregularly sampled measurements and where missingness can be meaningful signal in itself [27]. Much like humans reading partial reports, LLMS can handle missing values without imputation by adopting special symbols, for instance,
[64. ., 49. 16, - "64, NaN, NaN, 49, NaN, 16, NaN"
In Figure 8 we compare likelihoods and CRPS value for forecasts from traditional time series methods and LLaMA-2 70B on data that has been corrupted with missing values and then processed with linear interpolation and the above string formatting. While the likelihoods of traditional methods rapidly deteriorate with corruptions, we find that LL-- 70B is more resilient, and when comparing CRPS values, LLaMA-2 70B is competitive with methods that use interpolation.

---

TCN N-HITS LLaMA (NaNs)
Lineer
0.20
Square
NUL/D
CRPS
Gauss Wave
0.15
Linear Cos
0.10
Sin
Sigmoid
Random
0%
50%
50%
0%
20%
40%
50%
% Missing
% Missing
Accuracy
Figure 8: Left: LLMTIME can handle missing values without interpolation by denoting missingness with text (e.g. 'NaN'). For baseline methods we perform linear interpolation and then the model usual. LLMTIME assigns higher log likelihood to datasets preprocessed with added 'NaN's ' than baseline methods assign to interpolated datasets. Forecasting performance, as judged by CRPS, is competitive between LLMTIME and alternative methods that use explicit interpolation. Filled area shows standard error over individual datasets and 3 random seeds. Right: LLMs can be used to answer questions about time series data posed as text. We show GPT-4's accuracy predicting the function that generated the time series, obtained using chain-of-thought prompting.
Connecting time series and textual understanding Because LLMs are designed for natural language and code, we can augment the numerical time series with useful text. We can do either by providing textual side information as inputs, or by producing textual outputs from a given time series. An interesting question is whether GPT-4 can explain in text its understanding of a given time series. We probe this quality by providing GPT-4 the code to generate our synthetic time series, provide the values of one these time series, and then ask it infer which of the functions produced the data in zero-shot manner. The prediction accuracies are shown in Figure 8, with the three remaining rows all being O. With CoT [47] prompting the model performs much better than random chance; however, its ability to identify patterns better when directly extrapolating the numerical data, suggesting that its numerical understanding the model often explains properties of the time series in order to select the right candidate from the list, and we show several of these sample explanations Appendix F. We also show how this task is encapsulated in simple (unprompted) next token prediction problem on cells of Jupyter notebook, illustrating why we expect such capabilities to emerge with a sufficiently powerful language model.
fully connected its textual understanding. In making predictions,
Discussion
We have demonstrated that large language models be used as pretrained time series forecasters by encoding numerical values as text. As with other "foundation" models, pretraining confers useful biases toward generalizable patterns that would otherwise be engineered into the model through architecture design [21], and enables natural scaling of performance with improvements in the base pretrained model. Because LLM forecasters are trained on language, they also confer unconventional capabilities, such as question answering. More broadly, framing time series forecasting as natural language generation can be seen as another step towards unifying more capabilities within a single large and powerful model, in which understanding can be shared between many tasks and modalities. Moreover, zero-shot forecasting can enable broadly compelling performance without requiring significant computational resources, domain expertise, or many downstream training data points.
While LLM forecasters benefit from the strengths of pretrained transformers, they also inherit their weaknesses, which can include a limited context window. While many univariate time series problems can fit comfortably within increasingly large context windows, multivariate problems pose a more significant challenge. There have been several recent advances extending LLM context windows to 10- (0-100K tokens [36, 4, 5, 1]. Combining these advances with time series forecasting is particularly exciting direction for future research. Another potential challenge of using current LLMs architectures could be their weakness in arithmetic and performing recursive and compositional operations, which could be a limitation on particularly challenging time series. On the other hand, many time series do not require precise arithmetic. Understanding the extent to which this is the case, and relaxing this limitation, is also promising avenue for future research. Separately from any limitation, it would also be promising to investigate effective procedures for fine-tuning L.LMs on time series. We hope that bridging LLM research with time series forecasting brings benefits to both communities.
10

---

Acknowledgements. We thank Micah Goldblum, Greg Benton, and Wesley Maddox for helpful discussions. This work is supported by NSF CAREER IIS-2145492, NSF I-DISRE 193471, NSF I1S-1910266, BigHat Biosciences, Capital One, and Amazon Research Award.
References
[1] Joshua Ainslie, Tao Lei, Michiel de Jong, Santiago Ontañón, Siddhartha Brahma, Yury Zemlyan- skiy, David Uthus, Mandy Guo, James Lee-Thorp, Yi Tay, et al. Colt5: Faster long-range transformers with conditional computation. arXiv preprint arXiv:2303.00755 2023.
[2] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 1, context- feteext-free grammar. arXiv preprint 2ttt::2335.13667.. 2023.
[3] Cem Anil, Ashwini Pokle, Kaiqu Liang, Johannes Treutlein, Yuhuai Wu, Shaojie Bai, J Zico Kolter, and Roger B Grosse. Path independent equilibrium models can better exploit test-time computation. Advances in Neural Information Processing Systems, 35:7796-780% 2022.
[4] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXív:2305. 10403, 2023.
[5] Anthropic. Introducing 100k context windows. Anthropic blog, 2023. URL https://ww. context- windows.
anthropic cot.pppicccm//ideee..e..
[6] Gregory Benton, Nate Gruver, Wesley Maddox, and Andrew Gordon Wilson. Deep prob- abilistic time series forecasting over long horizons. openreview preprint, 2022. URL https://openrevall nwt/ffeumfidddeeeeemoeoo
[7] Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin An- thony, Shivanshu Purohit, and Edward Raf. Emergent and predictable memorization in large language models. arXiv preprint arXiv:2304.11158 2023.
[8] George EP Box and Gwilym M Jenkins. Some recent advances in forecasting and control. Journal of the Royal Statistical Society. Series C (Applied Statistics), 17(2)91-109, 1968.
[9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877719011 2020.
[10] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint ar3iv: 12712, 2023.
[11] Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza, Max Mergenthaler, and Artur Dubrawski. N-hits: Neural hierarchical interpolation for time series forecasting. arXiv preprint 1201:22201118880 2022.
[12] Kent K Chang, Mackenzie Cramer, Sandeep Soni. and David Bamman. Speak, memory: An archacology of books known to chatgpt/gpt-4. arXiv preprint arXiv:2305.0011 2023.
[13] Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[14] Miles Cranmer. Interpretable machine learning for science with pysr and symbolicregression, jl. arXiv preprint arXin:2205.00558 2023.
[15) Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christo- pher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, et al. Language modeling is compression. arXiv preprint arXiv:2309. 10668, 2023.
[16] Dazhao Du, Bing Su, and Zhewei Wei. Preformer: Predictive transformer with multi- scale segment-wise correlations for long-term time series forecasting. arXiv preprint arXiv:220..11356 2022.

---

[17] Jacob R Gardner, Geoff Pleiss, David Bindel, Kilian Q Weinberger, and Andrew Gordon Wilson. Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu acceleration. In Advances in Neural Information Processing Systems, 2018.
[18] Rakshitha Godahewa, Christoph Bergmeir, Geoffrey Webb, Rob Hyndman, and Pablo Montero-Manso. Monash time series forecasting archive. arXiv preprint arXiv:2 2021.
[19] Micah Goldblum, Marc Finzi, Keefer Rowan, and Andrew Gordon Wilson. The no free lunch theorem, kolmogorov complexity, and the role of inductive biases in machine learning. arXiv preprint arXiv:2304.05566 2023.
[20] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
[21] Nate Gruver, Marc Finzi, Micah Goldblum, and Andrew Gordon Wilson. The lie derivative for measuring learned equivariance. arXiv preprint arXiv:2210.02984, 2022.
[22] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.0338 2020.
[23] Julien Herzen, Francesco Lässig. Samuele Giuliano Piazzetta, Thomas Neuer, Léo Tafti, Guil- laume Raille, Tomas Van Pottelbergh, Marek Pasicka, Andrzej Skrodzki. Nicolas Huguenin, et al. Darts: User-friendly modern machine learning for time series. The Journal of Machine Learning Research, 23(1)55442-544) 2022.
[24] Hansika Hewamalage, Klaus Ackermann, and Christoph Bergmeir. Forecast evaluation for data scientists: common pitfalls and best practices. Data Mining and Knowledge Discovery, 37(2): 788- 832, 2023.
[25] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint 1904.0971, 2019.
[26] Rob J Hyndman, Anne B Koehler, J Keith Ord, and Ralph D Snyder. Forecasting with exponential smoothing: the state space approach. Springer Science & Business Media, 2008.
[27] Shruti Kaushik, Abhinav Choudhury, Pankaj Kumar Sheron, Nataraj Dasgupta, Sayee Natarajan, Larry A Pickett, and Varun Dutt. Ai in healthcare: time-series forecasting using statistical, neural, and ensemble architectures. Frontiers in big data, 3:4, 2020.
[28] Colin Lea, Rene Vidal, Austin Reiter, and Gregory D Hager. Temporal convolutional networks: unified approach action segmentation. In Computer Vision-ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14, pages 47-54. Springer, 2016.
[29] Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison Bullison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021.
[30] Tiedong Liu and Bryan Kian Hsiang Low. Goat: Fine-tuned llama outperforms gpt-4 on arithmetic tasks. arXiv preprint 20Xiv:2305.14201, 2023.
[31] Andriy Mnih and Geoffrey E Hinton. A scalable hierarchical distributed language model. Advances in neural information processing systems, 21, 2008
[32] Steffen Moritz and Thomas Bartz-Beielstein. imputets: time series missing value imputation in
r. R.R. J., 9()):007, 2017.
[33] Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowyez, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114. 2021.
12

---

[34] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac HarfieldDDodde Danny Hernande7, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread, 2022. heads/index.html
[35] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. In 9th ISCA Speech Synthesis Workshop, pages 125-125. ISCA, 2016.
[36]
OpenAl. Gpt-4 technical report. arXiv, 2023.
[37] Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. Journal of Machine Learning Research, 2 ((111))1-63 2020.
[38] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744: 2022.
[39] Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. In Advances in Neural Information Processing Systems, volume 31, pages 6638 6648. NeurIPS, 2018.
[40] David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3):
1 1181-1191, 2020.
[41] Avi Schwarzschild, Eitan Borgnia, Arjun Gupta, Furong Huang, Uzi Vishkin, Micah Goldblum, and Tom Goldstein. Can you learn an algorithm? generalizing from easy hard problems with recurrent networks. Advances in Neural Information Processing Systems, 34:6695 016669966666,,6-0 2021.
[42] Ilya Sutskever. An observation on generalization. Workshop on Large Language Models and Transformers, 2023. URL https://wwii woutwwe....bb.......... cww/ http///tcchyv=AAKMt TV23Aabb. channel=SinmosInntiiii
[43] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Ani Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2300. 13971, 2023.
[44] Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Maric-Anne Lachaux. Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nic, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. ArXiv, abs/2307. 00988899 2023.
[45] Juan R Trapero, Nikolaos Kourentzes, and Robert Fildes. On the identification of sales forecast- ing models in the presence of promotions. Journal of the operational Research Society, 66(2): 299-307, 2015.
[46] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew Dai, and Quoe Le. Finetuned language models are zero-shot learners. arXiv preprint 21X::2109900652 2021.
13

---

[47] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le. and Denny Zhou. Chain of thought prompting elicits reasoning large language models. arXiv preprint UrXiv:2201.1190. 2022.
[48] Andrew Wilson and Ryan Adams. Gaussian process kemelss for pattern discovery and extrapo- lation. In International conference on machine learning, pages 1067-1075. PMLR, 2013.
[49] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition trans- formers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34, 2021.
[50] Hao Xue and Flora D. Salim. Promptcast: A new prompt-based learning paradigm for time series forecasting, 2023.
[51] Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, and Songfang Huang. How well large language models perform in arithmetic tasks? arXiv preprint arXiv:2304.02015, 2023.
[52] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? arXiv preprint arXiv:2205.13500 2022.
[53] Yiyuan Zhang, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue. Meta-transformer: unified framework for multimodal learning, arXiv preprint arXiv:2307. 10802, 2023.
[54] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of AAAI, 2021.
[55] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting, Proc. 39th International Conference Machine Learning (ICML 2022), 2022.
[56] Tian Zhou, Peisong Niu, Xue Wang, Liang Sun, and Rong Jin. One fits all: Power general time series analysis by pretrained Im. arXiv preprint arXiv:2302.1 //939, 2023.
14


Reviews:
[{"review_id": "CEQhay8wvt", "reviewer": "Reviewer_duBe", "summary": "This paper discusses a novel approach to time series forecasting using large language models (LLMs) like GPT-3. The authors encode time series as a string of numerical digits and view time series forecasting as next-token prediction in text. They find that LLMs can extrapolate time series at a level comparable or exceeding the performance of several classic time series models. The authors argue that LLMs excel at extrapolating deterministic numerical sequences because of biases for simple explanations introduced by their text pretraining.  The authors suggest that this approach has potential and enables new capabilities, such as integrating non-numerical text into the input or output. \n\nOverall, the paper studies an interesting problem setting that directly utilizes the in-context learning ability of Transformers to conduct zero-shot forecasting problems. In particular, the pre-trained transformer model (e.g., LLMs) can give promising performance. However, the analysis in the paper is not comprehensive enough. First of all, the paper fails to answer why pre-trained LLM can yield better performance. The arguments in the paper either focus on RNN (e.g., L121) or directly assume LLMs can perform extrapolation on time series sequences (e.g., L144 and L154). Based on the current presentation, it looks like the authors design a specific feature engineering method (digits representation) that can properly utilize the LLMs' extrapolation or in-context learning ability. If it is the case, the most important research question, why pretrained LLMs can conduct arbitrary time series' extrapolation better than other models is not answered or well discussed. From my perspective, if the major innovation of the paper is a feature engineering method, the current numerical performance would not be strong enough to lead to a publication in the top machine learning conferences like Neurips.\n\n\n\nMoreover, the experimental settings in the paper contain learnable parameters that are tuned based on the forecasting error. From my perspective, I would not treat it as zero-shot learning setting, since parameters can be viewed as trained. The authors also don't provide codes to help reviewers verify the results of experiments.\n\nIn summary, this paper studies an interesting problem that utilizing pretrained LLMs to help time series forecasting. However, based on the current presentation, the analysis and the numerical performance are not strong enough. ", "questions": "Please see my comments in Summary section.", "limitations": "The authors don't directly discuss the limitations of the proposed methods in the main paper.", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "This paper studies an interesting problem that utilizing pretrained LLMs to help time series forecasting", "weaknesses": "Please see my comments in Summary section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your review. \n\n**Understanding why LLMs can perform extrapolation on time series:**\nWe have demonstrated empirically that LLMs indeed can perform extrapolation on both deterministic and stochastic time series in Section 3.1 and 3.2.  We explicitly pose the ability to extrapolate simple deterministic trends as the ability to identify and implement simple arithmetic operations on the numbers in the sequence, for example addition by a constant for linear growth, multiplication for exponential growth, or copying for periodicity. Prior works [4,5] have shown that larger LLMs are increasingly capable at performing addition and multiplication at the token level, and the repetition bias is also a well-studied phenomenon [3]. We also relate these well-known phenomena with recent findings that LLMs have a preference for numerical sequences generated by a small number of arithmetic operations [2]. While we devoted a significant portion of the paper to investigating why LLMs are well suited to time series prediction, ultimately a good theory will hinge on a deeper understanding of the inductive biases of the transformer architecture and the effects of extensive generative pretraining. These questions are among the most important open problems in the field of deep learning at large and extend beyond the scope of our paper.\n\n**Comprehensiveness of the analysis**\nRNN vs LLM for Figure 2 experiments (fitting continuous distributions): For figure 2 we chose to use an RNN as the autoregressive model over the digits in order to disentangle uncertainty representation from a model’s ability to perform in-context learning. While we can train a small RNN or transformer directly to fit the distribution via gradient descent, the practical setup required in time series requires adapting to the distribution purely using in context learning, a considerably more challenging task. In section 3.1 we wanted to evaluate this parametrization of continuous densities in isolation. We have previously run GPT-3 on this identical problem with in context learning and the results are similar (though slightly worse due to the burden of only using in context learning).\n\n**Zero shot setting:**\nIn general we find that there is little value in tuning the hyperparameters of our LLM time series prediction method on the data at hand. Using a fixed set of hyperparameters reproduces very nearly the same performance, and in the additional LLaMA-2 experiments we always use the fixed sampling parameters temperature (T) = 1.0, nucleus size (top_p) = 0.9, and 3 decimals of precision. Regardless, by zero shot we simply mean the setting without additional data besides the given series, for example no other instances of completed or uncompleted time series, and we feel that this usage is consistent with how zero shot is used in computer vision and NLP ( e.g. a zero shot image classification method that ingests a test image and performs test-time fine-tuning [1]).\n\n**Code:**\nWe fully intend to release the code for our method on the acceptance of the paper.\n\n**Limitations:**\nWhile we do discuss several limitations within the paper (such as the limited context window, and the challenge of multivariate series), we recognize that it would be better to have these limitations aggregated into a dedicated limitations section. We have listed this new limitations section in the general response, and we will add it into the revised version of the paper.\n\n[1] Shu, Manli, et al. \"Test-time prompt tuning for zero-shot generalization in vision-language models.\" Advances in Neural Information Processing Systems 35 (2022): 14274-14289.\n[2] Goldblum, Micah, Marc Finzi, Keefer Rowan, and Andrew Gordon Wilson. \"The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learning.\" arXiv preprint arXiv:2304.05366 (2023).\n[3] Holtzman, Ari, et al. \"The curious case of neural text degeneration.\" arXiv preprint arXiv:1904.09751 (2019).\n[4] Yuan, Z., Yuan, H., Tan, C., Wang, W., & Huang, S. (2023). How well do Large Language Models perform in Arithmetic tasks?. arXiv preprint arXiv:2304.02015.\n[5] Liu, T., & Low, B. K. H. (2023). Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks. arXiv preprint arXiv:2305.14201."}, {"review_id": "K68z9xazVQ", "reviewer": "Reviewer_eErN", "summary": "Briefly summarize the paper and its contributions. This is not the place to critique the paper; the authors should generally agree with a well-written summary.\nThis paper proposes that LLMs like GPT-3 can be employed as time series forecasters due to the similarities between text and time series such as periodicity. The authors show how LLM can be used to model multimodal distributions through a hierarchical discretization of the output space and uniform distribution in each discrete bucket, enabling training of LLM with negative log-likelihood. The hierarchical discretization also allows the LLMs to circumvent the requirement of a fixed output range with binning values and a large number of bins for an expressive output space.\nExperiments show that GPT-3 fitted with the proposed modifications can forecast time series (e.g. linear, periodic, exponential) better than baselines such as TCN and ARIMA. The authors also explore optimal ways to encode time series data in LLMs and found that using a base of 10 and use of space to individually tokenize digits in numbers result in better performance. When benchmarked against existing models, the proposed approach perform competitively with top baselines in the DARTS and Monash datasets. Further experiments were conducted to show that LLMs as time series forecasters are sample efficient, have performances scaled with model size and handles missing data well.\n", "questions": "a) The experiments seem to be mostly in a zero-shot setting, have the authors studied the few-shot setting where few-shot examples are in the prompt and how would that affect forecasting performance?\n\nb) What prompts, if any, are given to the LLM, other than the initial time series data and what effects would the choice of prompts have on its performance?\n", "limitations": "No, a separate section addressing the approach's limitations and potential societal impact would be recommended.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The approach to using LLM as a time series forecaster is innovation and the authors found ways to elegantly unlock this capability without the need for extensive effort (e.g. retraining or finetuning).\nThis finding will be interesting and relevant to a wide audience in this community.\nExperiments were extensively conducted to show optimal approaches for LLMs’ time series forecasting ability and other aspects such as sample efficiency etc.\n", "weaknesses": "Limitations of the proposed approach were not discussed in the paper", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your supportive review!\n\nTo answer your questions,\nWe did not study the few-shot setting, but it would be a great direction for future work. We suspect that both few-shot prompting and fine tuning could substantially improve forecasting performance. One challenge of few-shot prompting, however, is the limited context window of current LLMs. In the zero-shot setting, we can use the full context window to extend the number of past observations used by the model. In the few-shot setting, this same context window might get consumed by the few-shot examples, limiting the number of past observations.\nAll the results in paper that use GPT-3 or LLaMA models are obtained without any natural language prompting. We experimented with prompting with a description of the dataset, but the initial results showed no improvement in performance. For ChatGPT and GPT-4, we used the following prompt (which was necessary for good forecasts): \"You are a helpful assistant that performs time series predictions. The user will provide a sequence and you will predict the remaining sequence.\"\n\nWe appreciate the question about limitations. There are several key limitations that are worth noting and we have added them into a unified section shown in the general response, which will be integrated into the paper. The limitations of our method are largely the limitations of LLMs in general, specifically limited input sequence length (small context window), subsequent challenges in handling multivariate data, and computational cost. Additionally, black box APIs have limited control over system prompts, RLHF, tokenization and likelihoods. As each of these limitations is an active area of research in itself, we expect that all progress should also improve the practicality of our method. By connecting LLMs with time series forecasting, we have also opened new avenues in evaluating progress in LLM performance on a task that requires understanding complex, quantitative patterns.\n"}, {"review_id": "Mib0donxR2", "reviewer": "Reviewer_URMj", "summary": "The papers presents an analysis of larger language models for the task of time series prediction. The paper claims that these large language models being very generalizable are able to work very efficiently with numbers in these tasks since each digit prediction is a conditional over the previously predicted digits and hence acts in a way similar to hierarchical softmax to predict the final output.\n\n", "questions": "Some typos\n- line 59: \"from from\" -> \"from\"\n- line 63-64: \"without any major changes by directly numbers as strings of digits.\" --> needs to be reworded", "limitations": "- Since we depend on the LM model to perform the forecasting, it is much harder to understand why a particular forecasting was chosen. There are always multiple ways to interpret a series for completion and relying that the LM with follow Occum razor is dangerous. Only the other hand, traditional time series forecasting methods are much more interpretable.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Even though a lot of the work presented here is known to be useful, less is known about how these models work and performing a closer analysis with simplified datasets and well defined examples gives a much better understanding about what works and what does not and helps figure out areas of improvement.\n\nFor example the approach to split numbers by digits, previous works have achieved similar results by removing multi-digit entries from the vocabulary of the model which eventually has the same impact as this paper. But this paper presents intuition around how this might help the model in predicting one digit at a time and be more performant using the analogy to heirarchical softmax.\n\nThe well defined set of experiments show that this approach works very well to improve the ability of LLMs to work with numbers and the analysis on time series datasets confirms the same.", "weaknesses": "There are already papers showing that:\n1. Mathematical ability of LLMs improves significantly if we process digit by digit (approached generally by removing multi-digit entries from the vocab).\n2. Application of LLMs on time series datasets which generalize better then regular approaches has also been shown.\n\nSee \"PROMPTCAST: A NEW PROMPT-BASED LEARNING PARADIGM FOR TIME SERIES FORECASTING\"\n\nSo there isn't much novelty provided by this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the supportive review and your comments on related work. \n\n**Comparison to related work**\n\nThough other papers have investigated the use of LLMs in basic arithmetic (e.g Goat [1]) and forecasting (e.g. PromptCast), we believe the findings of our paper are unique in their scope and focus. We delve deep into understanding why LLMs can extrapolate on time series, emphasizing their bias for low-complexity completions, ability to identify and execute arithmetic operations on numerical sequences, and superior representation of uncertainties. We also uncover scaling results and the effect of alignment methods which are not present in related work (e.g. PromptCast).\n\n**Following Occam’s Prior** \n\nRegarding your comment on Occam’s razor, of course there are many consistent completions of a time series (just as there can be many consistent extrapolations of a numerical pattern). Following the Occam’s razor prior explored in [17] does not mean that we should _select_ the simplest trajectory consistent with the observed history, but rather that we should form a predictive distribution and _sample_ according to their posterior likelihoods using this prior conditioned on the observations, resulting in simpler completions being sampled more often. This process is formalized through Solomonoff induction, whereby the Occam’s razor prior (also known as Solomonoff prior) $p(x) = 2^{-K(x)}/Z$ (where $x$ is the complete time series and $K$ is the prefix Kolmogorov complexity) is used as a prior to form a posterior $p(x|x_{[:t]}) = p(x_{[:t]}|x)p(x)/p(x_{[:t]}) \\propto 2^{-K(x)}$ for all series $x$ that match the history. We believe that LLMs capture a very rough and approximate version of this Solomonoff prior. With the deterministic time series in Figure 3, these signals can be explained by a generating program of short length, and other programs which correctly fit the previous history but make substantially different predictions on the future require many more bits to express. These solutions are still represented in $p(x|x_{[:t]})$ but are just exponentially less likely to get sampled (if we sample enough we should recover them). On the other hand, for real time series that have noise, the shortest generating program for the data is still very large since it must encode the noise. Therefore, other explanations of the data are close in size and therefore sampled frequently in the posterior distribution of the Occam’s prior, and that reflects what we see in the LLM predictions in e.g. Figure 6 that have considerably more variation.\n\n[1] Liu, T., & Low, B. K. H. (2023). Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks. arXiv preprint arXiv:2305.14201."}, {"review_id": "Ay17FxWo9i", "reviewer": "Reviewer_s41P", "summary": "The authors propose a way to use autoregressive large language models to solve time series problems. Several tricks make the time series inputs amenable to LLMs (e.g., separating digits by a space to circumvent tokenization problems). GPT-3 outperforms existing methods on several time series benchmarks. LLMs can also perform imputation on missing data in the time series.", "questions": "**I don't understand the argument in Section 3.2: LLMs extrapolate patterns in numerical sequences.**\n\nThe discussion in Section 3.2 is confusing to me. The authors suggest that LLMs can handle sequences with linear growth, exponential growth, etc. What's confusing is that these are all fixed functions that have no dependence on the previously occurring values. Something that may be more interesting is e.g. the Fibonacci sequence, where the next value is a function of past values. I don't really get the point of Figure 3 + Table 1 because these are just simple functions, and if I wanted to fit them, I would directly fit them using the appropriate function class (i.e., not huge, overparametrized neural networks).\n\nI also don't really get why it is a given that LLMs can do \"simple arithmetic operations\". Although these are simple to us, they may be complicated functions to learn in the embedding space, and they depend on the tokenization used. Note that the cited paper [17] is  focused on low-complexity sequences in the sense of Kolmogorov complexity, which doesn't necessarily correspond to simple arithmetic operations.\n\n**Can the authors motivate their dataset choices?**\n\nThe transformer-related time series models don't use the DARTS dataset. ", "limitations": "**Not enough LLMs**\n\nThe writing indicates that the authors are trying to make the broader claim that all LLMs (that can perform in-context learning) can serve as time series models. To actually substantiate this claim, they need to test other LLMs besides GPT-3 and GPT-4. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper considers the interesting setting of applying pre-trained LLMs to \"classical\" (i.e., 1-D) time series problems. The tricks introduced to format the time series as input to an LLM are interesting in their own right, though I am not fully sure if they are novel or not. The authors also conduct thorough ablations on these tricks to see how they might affect the performance of GPT-3 on these types of problems. ", "weaknesses": "**Costs of using LLMs**\n\nOne crucial discussion missing from this paper is the inference cost. If using GPT-3 or GPT-4, then this is a literal payment. Even for open-sourced LLMs, I would imagine that the inference cost is much higher than that of other time series models. Similarly, using LLMs for imputation is much more expensive than standard methods (e.g., interpolation). The authors should acknowledge this weakness clearly.\n\n**No comparison to transformer time series models**\n\nThe authors ought to include comparisons to other transformer time series models, such as the Informer and Autoformer, which were clear leaps forward in the field. It's hard to tell if the gains observed in this paper are from the massive amounts of pre-training data (as suggested by Section 3.2?) or from the self-attention mechanism, which does seem well-adapted to time series. Please also compare the size of GPT-3 to those models. This is my main reason to not give it an accept rating.\n\n**Writing**: Please define MAE, CRPS, etc before using them in the text. The colors used for \"Ground Truth\" and \"GPT-3\" are too close to each other in Figure 3, so it is hard to read. The authors should include a discussion of how positional embeddings make it easier for an LLM to interpret the input data and operate across the time domain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your thoughtful and detailed review. In our understanding, your main concerns about the paper are the need for testing additional LLMs, the lack of comparison against Informer and Autoformer, and the purpose of the argument that we present in section 3.2. \n\n**Evaluating other LLMs** In fact we did evaluate other available LLMs shown in Figure 7 (right) and in Appendix B.3, though to further address your question, we have added the recent LLaMA-2 70B model to our investigations and find that in many cases it outperforms GPT-3 Davinci. These comparisons are shown in Figure 1, 2, and 3 of the attached PDF. \n\nRegarding your point about transformer methods like Informer and Autoformer, we have added comparisons against these methods shown in Figure 1 in the attached PDF. We compare LLaMA-2 70B with Autoformer, Informer, Reformer, and a transformer model on 5 of the autoformer benchmark datasets with two different prediction lengths. We find that LLaMA-2 70B performs on-par with Autoformer and better than Informer in aggregate. We were surprised to find that the model performed this well out of the box without any hyperparameter tuning or modification of the sampling temperature. While we think this performance is quite impressive, it’s worth noting that the LLaMA model is about 10,000 times larger than the autoformer models it is compared against, and the LLaMA model appears to experience degrading performance with longer sequence lengths, but the LLaMA model is predicting completely zero-shot and is capable of performing all kinds of text-generation tasks in addition to forecasting.\n\n**Inference costs** Regarding inference costs, because we do not perform any training of the base LLM, these are actually quite manageable. A single forecast with maximum history length costs only 4 cents with the largest GPT-3 Davinci model. In general these costs are very low, except if one requires to draw many many samples from the distribution over trajectories. In total we have spent less than 1000 dollars on GPT-3 queries for prototyping our method and evaluating on the many datasets.  With the ability to run models locally like LLaMA-2, this cost decreases even further.\n\n**Section 3.2** Regarding Section 3.2 and the deterministic synthetic time series, these experiments provide a controlled environment for uncovering which patterns are easier for the LLMs to identify and extrapolate, and provide an understanding of why this is the case. Ultimately we do not care about the model’s performance on these deterministic signals for their own sake, and of course one could fit these functions very well by directly parameterizing a function in the right family. Although these functions are easy to express as a mathematical formula, as you allude to, that does not mean that the functions are easy to express at a token level inside an LLM and even less so when the task is combined with identifying the recurring pattern from the history. Hence, the ability to identify and extrapolate these signals is a nontrivial capability of the language models and we try to make sense of how this is possible in simple cases through the arithmetic operations that are entailed. While the discussion is focused on Kolmogorov complexity in [17], the experiments on the low complexity bias of LLMs actually measure complexity with respect to a reduced language of expressions that can be formed from an expression tree with constant, addition, multiplication, and integer division primitives. These primitives coincide with those we discuss in Section 3.2 for simple pattern extrapolation.\n\nGiven that we have performed the necessary experiments to specifically address your concerns, we hope that you will consider raising your score.\n"}, {"review_id": "ojTTDRm2gt", "reviewer": "Reviewer_Zuxw", "summary": "This paper studies the usage of large language models such as GPT-3 for zero-shot time series forecasting task. To construct the LLMs input sequence in line with how they are pre-trained, it proposes to encode the continuous time series as a string of numerical digits with spaces added between single digits. As a result, time-series forecasting can be transformed into a next-token prediction task suitable for LLMs. Results on standard univariate time-series datasets show that LLMs can perform comparably to time-series models such as ARIMA, etc. in a zero-shot fashion i.e., without any finetuning.", "questions": "Minor: \n\n1. On page 4, I believe the figure references should be attributed to Figure 2 instead of Figure 3.\n", "limitations": "I could not find a section addressing the limitations and negative societal impact of this work.\n", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. The paper is well-written and most of the sections are easy to comprehend.\n\n2. The studied problem is straightforward and interesting considering the capabilities exhibited by large language models.\n", "weaknesses": "1. Comparison to other recent transformer-based time-series forecasting methods (listed in the references below, but not limited to) is missing.\n\n2. The paper considers simple univariate time-series forecasting datasets, it would be helpful to consider more complex multivariate time-series datasets and how they should be encoded.\n\n3. In Appendix C, details about how other existing time-series forecasting models/methods perform compared to GPT-3 are missing on the time-series data that was recorded after GPT-3's training data cutoff date.\n\nReferences:\n\n[1] Informer: Beyond efficient transformer for long sequence time-series forecasting, Zhou et al., 2021\n\n[2] Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting, Wu et al., 2021\n\n[3] FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting, Zhou et al., 2022\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your review and comments on the paper.\n\n**Transformer based time series methods** As we mentioned in the general response, we have added an additional evaluation of our model using the LLAMA-2 LLM which includes comparisons against Informer, Autoformer, and FEDformer as you suggested. We find that our LLM based time series prediction method compares favorably, with aggregate MAE values on par with or better than these transformer methods, but with completely zero-shot predictions. \n\n**Complex multivariate time-series** We have added a limitations section to the paper and we list this section in the general response. Our paper is focused on univariate time series, and as we discuss in the limitations section, multivariate time series are challenging to accommodate given the short context windows afforded by the LLMs. In the additional experiments on Autoformer benchmark datasets, we demonstrate that the LLaMA-2 LLM can perform well on multivariate data by considering it as a collection of univariate series. Developing methodology to accommodate all non-temporal correlations from the multivariate time series with LLMs is also a worthwhile endeavor, but we leave it to future work.\n\n**GPT-3 training cutoff date evaluation** We would like to direct you to figure 9 in appendix C where we evaluate the LLM predictions on three time series datasets recorded after the GPT-3 training data cutoff date. We evaluate the predictions both qualitatively (showing forecasts and prediction intervals) and quantitatively (using the negative log likelihood). Measured by NLL, GPT-3 performs strongly on the three datasets, and the forecasts are consistent with performance on the other time series we evaluate in the paper.\n\n**Limitations** We have added a limitations section in the general response that we will be adding to the main text.\n\nAs you mention, we believe the question of performing time series predictions with LLMs is of significant interest. We hope that given the new experiments performed and clarification of your concerns you would consider raising your score. \n"}, {"review_id": "Gl09f1lMXx", "reviewer": "Reviewer_z6DB", "summary": "This paper proposes a method to use LLMs for time series prediction. \nPrecisely, they develop a way to tokenize and encode numerical digits and prompt LLMs to generate future numbers given the past. \nTheir method is comparable to or exceeds the state-of-the-art time series models.", "questions": "Please see my questions and suggestions in the Weakness.", "limitations": "Please see limitations in the Weakness.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper proposes a very novel method to use LLMs for time series prediction. The problem is a high-impact problem, and their way of using LLMs is creative. I was excited when reading this paper. \n\nThe method is very carefully designed and is very convincing. Each component of the method (e.g., digit representation, tokenization) has been carefully thought through and demonstrated with insightful experiments and analysis. \n\nThe results are positive.", "weaknesses": "I would like to recommend this paper for acceptance, and I gave a positive score. \n\nThe main reason that I didn't give a higher score is its presentation and the limitation of the proposed method. \n\nFor presentation, I think the current structure needs improvement and some terminologies need clarification. \n\nThe current presentation is a little bottom-up: 3.1 explains how to represent numbers and 3.2 discusses how LLMs process and generate sequences of numbers; then 4 explains how these things are put together to form the method. \nHonestly, I was lost until I read the end of page-5 (of a 9-page paper!) since I did not know why I had to read section 3 and I didn't know how 3.1 and 3.2 are connected. \nMaybe a better way is top-down: give a high-level overview of the method first and then dive into each phase/component for their details. If the authors do not want to change the structure dramatically, maybe at least add a paragraph explaining the overall picture of the method at the beginning. \n\nI would like the authors to give a precise clarification whenever they mention a term that has multiple meanings.\nMaybe I am wrong but \"multimodality\" can mean \"the distribution has multiple modes\" or \"the data has multiple forms\". In this paper, it seems that the authors have used it for both cases (3.1 and 5.5) but that made me confused. \n\nFor limitation of the method, it seems that the proposed method requires something that modern blackbox LLMs do not like to offer: i.e., tokenization and log-likelihood computation. This reliance will restrict the impact of this work: this is fine, since I think the method is already very good and it is a reasonable limitation. But I would appreciate it if the paper has an aggregated section/paragraph discussing this limitation explicitly, instead of mentioning it here and there scattered across the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and supportive review. Your comments about improving the readability of the paper through section 3 are well received. We have added a paragraph at the start of section 3 giving a high-level overview of the method, and we have revised the experiments in Figure 2 to use the decimal encoding. Hopefully these changes will make it clearer how the findings in 3.1 and 3.2 connect to the method we propose.\n\nOn multimodality, your interpretation is correct and we have somewhat overloaded the usage of multimodal. We will retitle section 5.5 “Connecting Time Series and Language Understanding” to avoid confusion. Regarding the limitations of using black-box LLMs, we have added a complete limitations section in the general response discussing this and other issues that we will add to the main text of the paper. As you point out, tokenization and log-likelihood are often not accessible for the models exposed only through APIs (for example GPT-3 provides log-likelihoods but GPT-4 does not), and this can be a hindrance to the method and make comparison more difficult. However, the proliferation of highly performant open source models like LLaMA-2 provide reason for optimism on this front.\n\nGiven that your main concern, and the reason for your weak accept rating, was the limitations of black-box LLM APIs, we hope you will considering raising your score in light of our new LLaMA results, which do not suffer from the same limitations. "}, {"review_id": "cYpefaW3VV", "reviewer": "Reviewer_AFnc", "summary": "This paper investigates to which extent an off-the-shelf large language model like GPT-3 can be used for time series forecasting. Its main claim is that forecasting performance can be very good, provided some care is taken in correctly encoding the input. Some limited experiments are proposed to support the claim.", "questions": "- L19 time series has\n- L19 \"time series are relatively unstructured\": this is arguable. There is host of research precisely leveraging structure in time-series. For a very clear example, just consider speech, audio or music processing, where structure is everywhere.\n- L30 \"pretraining has not impacted time series forecasting to the same degree\": pretty arguable and gratuitous statement\n- L35: in a natural way\n- L52 \"instead of building towards a foundation model paradigm\". I would say that this statement is not only arguable, it is very wrong and can even be felt as pretty pejorative for the host of researchers working in the field. The authors could for instance check out the many foundational books or papers that were written on modelling stochastic processes or in developing information theory and that all emerged in a time-series setup. Regarding recent foundational approaches, one could for instance mention neural processes or related fundamental research.\n- L54-64: \"uncertainty quantification for time series also tends to be domain-specific and ad hoc\": I wonder what makes the authors think that NLP is free of any such bias. The whole paragraph has a strange pejorative feel to it that I cannot understand and that I think is inappropriate.\n- L64: limited expressive\n-  L65 \"by directly numbers as strings of digits\" ?\n- Section 3.1: there is no reference whatsoever to source coding or arithmetic coding and their host of deep-learning based recent variants, which appear to me as a clear _foundational_ setup for the _adhoc_ method proposed here.\n- L130-L139: Evaluation. It doesn't really make sense to me to compare the proposed RNN-method for quantization with parametric distribution models like GMM or Laplace, etc. The authors should have compared instead with a quantile-based model where the number of quantiles corresponds to the number of parameters of the RNN they fitted. \n- The \"occam razor\" discussion from section 3.2 is interesting, where the authors show that in some toyish cases, a LLM is able to extrapolate a numerical sequence. Still, they then generalize this interesting feature they observed on toyish examples as a general feature of LLM, which is not supported in any way but is only a \"working assumption\" and should be mentioned as such.\n- L149: \"time series generated according to a deterministic function can be considered as only a small generalization of these numerical sequences\": This is of course not true and the authors should check the host of different ways that were proposed in the past to model deterministic signals and their incredible complexity. \n- L157: \"maps on precisely to extrapolating linear trends\" ?\n- L168: it is important that the authors detail how the method is extended to autoregressively predict digit sequences to multiple numbers, since this is a very important ingredient of the approach.\n- Figure 7: I do not understand where history length appears here\n- Figure 8: As a transformer, I would expect TCN to actually easily handle missing values.\n\n- In the conclusion: I would not say that the toy experiments proposed are enough to write that \"LLM can act as extremely capable time series forecasters\" in a _general_ manner. Once again, the paper makes a very nice case for very small sequences and asks a relevant question, but this remains only a funny curiosity for practitioners at this stage, considering how hard scaling to realistic time series appears. For instance, one second of audio sampled at 44.1kHz means 44100 samples. If each one ends up as a word, this would mean something like 50 pages of text, and this is only for mono. People routinely process hours of multichannel audio. Of course, other time-series practitioners have similar or even much more challenging scales to handle. \n- The authors are apparently unaware of the fact that a very fruitful research has long been done on multimodal signal processing in time-series contexts.", "limitations": "What I wonder after reading this paper is whether time-series models could be useful for text processing.\n\n-- \nAfter reading the rebuttal, I don’t think the authors can be trusted to make the huge amount of changes it would take to address my concerns.", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is definitely timely in the sense that  large language models (LLM) are very trendy and that many other works already showed that they are good at transfer learning. In this respect, attempting to use them for forecasting makes perfect sense and this paper could possibly have a good impact.\n- It cannot be denied that the reported performance is good, which supports the claim in a fascinating way.\n\nThere are some other limited contributions, notably regarding how the input should be encoded before using the LLM for forecasting, but I see them as very secondary when compared to the two mentioned above. Although I am recommending rejection for the reasons that I will describe below, I still must acknowledge that this paper asks a relevant and interesting question and is bound to meet some good impact, even if it is not presented at Neurips.", "weaknesses": "- If I had to summarize what I feel is my biggest concern with this paper, I would say that I feel uncomfortable with its _attitude_. As I mentioned above, it asks a question that I definitely think is interesting, but as it currently reads, it basically claims forecasting is solved by the method proposed, that outperforms state of the art in time-series forecasting.\nIn my view, the authors should have rather mentioned that using a LLM for forecasting interestingly seems to bring some very good performance in the experiments they did, but should have insisted much more on how much this should be understood today as some kind of funny curiosity at this stage. Indeed, there is something that is completely forgotten here: those LLM just cannot reasonably be used for forecasting in any realistic setup, that requires handling hundred thousands or million samples. What the paper convincingly shows is only that LLM can yield very fascinating performance for forecasting in the toy setups that were considered, and this is already a big deal.\n\n- the paper contains many arguable statements that I will detail below. As a researcher in time-series, I must say that many statements there read pretty pejorative towards research in time-series and sound like \"look how NLP will solve this problem that you have been struggling with for years\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and for your detailed review. We appreciate your recognition of the paper's relevance, fascinating results, and potential for impact. We apologize if the tone of our paper comes off as dismissive of prior work. We never intended this to be the case. In several places we believe this perception may come about from a difference of terminology and perspective employed in the two research communities, for example the term “foundation models” which is commonly used to refer to large pretrained models [1], not to suggest that they are more foundational in nature than other research. We assure you that we respect the contributions and nuances of the time-series community and any perceived negative tone was unintended. While we acknowledge some of your concerns and will adjust the manuscript accordingly, including certain uses of language, there are other points we feel require further clarification. \n\n**Handling missing values with TCN:** We note that TCN stands for Temporal Convolutional Networks and is not a transformer. On the other hand, actual transformer based models may be able to more easily accommodate missingness, by using a designated token or omitting the missing entries. However, for trained transformer time series models, if missing entries are not encountered at training time, then with missing values the statistics inside the model will be different and the model will not be able to handle the missing entries gracefully. However, as our LLM-based method has not adapted to the statistics of fully observed time series (because it is only using in-context learning) it has a significant advantage in this scenario.\n\n**Uncertainty Quantification:** In our description of prior work in time-series uncertainty we agree that \"ad hoc\" was a poor choice of words (and commonly overused in machine learning papers). What we meant was that there is no standard approach that resolves the problem of parametrizing aleatoric uncertainty in time series models. Instead there are many different approaches, each with its own set of shortcomings, and no satisfactory singular resolution of this problem. We will revise this sentence on line 54 to express this viewpoint more clearly.\n\n**Time Series Structure:** Our remarks on time series being relatively unstructured primarily pertain to popular domains of applications like finance and climate. While these series have inherent structure, they are typically more stochastic in nature than data types like images, where deep learning has seen much more substantial success. Our aim wasn't to downplay the complexity of time series modeling. On the contrary, the lack of easily identifiable structures beyond basic ones like periodicity can itself be a challenge when developing forecasting models. We will ensure that our revised manuscript provides a clearer context to avoid any misinterpretations.\n\n**Pre-training in Time Series:** While pre-training has its applications in the time-series domain, its prevalence in NLP or vision is evidently much more pronounced. We don’t think it is fair to call our statement “arguable and gratuitous” without any specific evidence or references. \n\n**LLM’s Low-complexity Bias:** Our demonstration of LLM’s low-complexity bias in simple deterministic settings was to provide insight into why LLMs can perform well on real world timeseries which also contain low-complexity deterministic patterns, rather than to establish it as a universal phenomenon. Nevertheless, recent work such as [17] also supports our finding that LLMs exhibit a preference for simple completions with low Kolmogorov complexity. \n\n**Significance of Our findings:** We strongly disagree that our LLM-based time series prediction method is “only a funny curiosity”. While it’s true that it is not suitable for time series with a large number of samples, there are plenty of real prediction problems with short lengths as demonstrated by the DARTS and Monash datasets, on which LLMs perform among the best models.\n\n**Figure 7 History Length:** In the univariate setting, N-HITS and N-BEATS can learn a model by chunking up the time series into segments and training on the batched segments. When the history is short, not many chunks are available and it is insufficient for effectively training a neural network. Smaller models with less learnable parameters (e.g. ARIMA) tend to do better in this scenario, and LLMs also do very well because no parameters are being trained on the data (the inference is zero shot).\n\n**L168:** For the autoregressive prediction on the sequence, the tokenized numbers are just concatenated with comma separators as presented in section 4.  To compute the continuous NLL of the time series the total NLL is just the sum of the NLLs from each conditional distribution over a given number. Mathematically, $\\mathrm{ContinuousLogLikelihood} = \\log p(x_{1:T}) = \\sum_{i=1}^T \\big(\\log p(x_i\\in U_k|x_{1:i-1}) + nlog B\\big) = \\mathrm{DiscreteLogLikelihood}+ Tn\\log B$ where $U_k$ is bin $k$.\n\n\n[1] Bommasani, Rishi, et al. \"On the opportunities and risks of foundation models.\" arXiv preprint arXiv:2108.07258 (2021)."}, {"review_id": "2Scb5D0KAv", "reviewer": "Reviewer_kUEg", "summary": "The paper discusses a novel approach for time series forecasting by encoding time series as numerical digits and treating it as next-token prediction in text. Surprisingly, it is found that large language models (LLMs) like GPT-3 can effectively extrapolate time series even without being specifically trained for this task. The success of LLMs in time series forecasting is attributed to their ability to extrapolate deterministic patterns, such as seasonality, and model uncertainty by using expressive distributions on continuous variables. It is argued that LLMs excel in predicting numerical sequences due to biases introduced by their text pre-training, which favors simple explanations. Additionally, LLMs demonstrate excellent representation of multimodal distributions by decomposing the distribution over digits. This combination of qualities allows LLMs to achieve strong performance on time series data and enable new capabilities such as integrating non-numerical text and handling missing data. Furthermore, the study shows that the performance of LLMs in time series prediction improves with increased scale, suggesting significant potential for this unconventional approach.", "questions": "1. The contributions are not very clear, and it seems that the authors are only utilized the GPT models.\n\n2. The authors claim they explain why LLMs can be used for timing prediction, but there is not enough analysis in the paper.\n\n3. Why are most experiments conducted on virtual datasets and not on more real-world datasets?", "limitations": "See weakness and questions.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper proposed a method that utilize the pretrained language model to solve time series forecasting problems.\n2. This paper proposes a novel timing coding approach that can help LLMS understand timing data.\n3. Also, the authors identify why language models are able to perform well time-series prediction task.", "weaknesses": "1. The experimental results are not enough to support the viewpoint put forward in the paper. For example, most experiments are conducted on virtual data sets, but how are the effects on real data sets is not enough.\n\n2. For some of the methods proposed in this paper, their effectiveness has not been fully analyzed, and there is a lack of analysis of Tokenization, scale, etc.\n\n3.As can be seen from the results in Figure 4, the proposed method is effective only for some simple time series data (such as obvious periodicity or trend), but not for some more complex time series data (such as wooly or HeartRate).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. While we appreciate your feedback, we believe that based on the concerns you raise, some of the experiments we performed may have been overlooked.\n\n**\"most experiments are conducted on virtual datasets\"**\nWe do not believe this is an accurate assessment. For time series extrapolation, we have performed comparisons on $8$ real datasets from the Darts project (shown in figure 6, and figure 4), the $10$ most impactful real datasets from the Monash archive (figure 5), and $3$ additional real world datasets collected after the GPT-3 cutoff date (figure 9 and appendix C). With the figure 1 in the attached PDF of the general comment, we have extended these further with an additional $5$ autoformer datasets. This is in comparison to the $8$ synthetic datasets that we evaluate in figure 3. \n\n**\"For some of the methods proposed, there effectiveness has not been fully analyzed, and there is a lack of analysis of Tokenization, scale, etc\"**\nIn section 4 and figure 4 we analyze precisely these design details, considering tokenization with and without spaces and varying the base of the encoding of the numbers. We evaluate these choices both qualitatively and quantitatively.\n\n**\"... proposed method is effective only for some simple time series data\"**\nWe have evaluated on a large variety of real time series datasets and on these datasets our method consistently ranks in the top performing methods. Keep in mind that we are showing the median and 10-90th percentiles, which for time series with a high variance will look quite different from realizations of the time series or samples from the model. On Wooly ours is indeed not the best performing method, but on HeartRate due to the considerable variance in the underlying process our method achieves the lowest NLLs among the competing methods.\n\n**\"authors are only using GPT-3 models\"**\nWe would like to point to figure 7 (right) in the main text where we compare against publicly accessible Cohere, Forefront, and Aleph-Alpha models as the base LLM used in our method (with additional details in appendix B.3). Since the reviews, we have made an effort to extend this even further, with the powerful LLaMA-2 models. We have added additional comparisons using LLaMA-2 70B as the base LLM in the response PDF figures (1,2) and the other sized LLaMA-2 models in figure (3).\n\nAltogether we hope that in light of these clarifications about experiments in the text and additional ones that we have run, that you may consider reevaluating the paper.\n"}]

Author rebuttal:
General Comment to All Reviewers

We are thankful for the thoughtful and supportive feedback. We were happy to see many of the reviewers think that the goals and results of the paper are interesting, relevant, and useful. Few ML researchers would confidently assert that language models are zero-shot time series forecasters. This finding is simultaneously surprising and supported by diverse experiments on real-world datasets. To unpack this result, we perform a comprehensive set of experiments to understand why language models are well-suited to forecasting and what factors impact the performance. Because there were a few recurring themes in reviewer questions, we would like to make a few general points before responding individually.

**Evaluation of Other Large Language Models** Several reviewers mentioned the need for evaluating other LLMs to better demonstrate the generality of the method. We would like to highlight Figure 7 (right) in the original submission’s main text, where we compare GPT-3 against many other LLMs that are accessible via open APIs, evaluated on the Darts datasets. Additionally, we are happy to share new results with the recent open-source LLaMA-2 models. These results, shown in Figure 1,2,3 of the attached PDF, demonstrate very strong performance, with metrics comparable to GPT-3 on Darts and significantly exceeding GPT-3 on Monash, making it the best performing model overall on those datasets. These results help demonstrate that our method is not limited to GPT-3, that more powerful LLMs will likely yield continued improvements on time series prediction, and that the limitations of working with LLMs locked behind APIs can be alleviated with open source models.

**Additional Comparisons Against Transformer Time Series Models** Following suggestions from the reviewers, we have added comparisons against the recent Autoformer, Informer, Reformer, and FEDFormer methods. LLaMA-2 demonstrates strong performance against these models across the board.

**Additional Limitations Section** Several reviewers suggested that it would be helpful to have a unified limitations section. We present the following limitations section below that we will add into the revised version of the paper:

“While we have demonstrated that text-pretrained LLMs are surprisingly capable at time series extrapolation, our approach has a few notable drawbacks, resulting from limitations of the underlying LLMs. Most notably the relatively short context window of LLMs limits the amount of information that can be processed. Our approach is therefore best suited for time series that are short in length, univariate, and consist of a single instance. Longer time series are currently truncated in order to fit into the context window (though subsampling could also be used), and multivariate time series pose an additional challenge, as the length is multiplied by the number of covariates. One possible solution for multivariate time series is to make predictions on the distinct dimensions separately, ignoring the correlations between the channels, but this independence assumption is less than ideal. 

Computational cost can also be a limiting factor. While most time series models are lightweight and fast to run over a large number of series, LLMs are significantly more expensive, even when used solely for inference. Beyond the properties of LLMs themselves, relying on API access to black-box LLMs can lead to its own set of drawbacks, as we lack full control over tokenization and model alignment methods. As we show in the paper, correct tokenization is essential to good forecasting performance and safety-driven prompts and RLHF may directly interfere with forecasting ability. Black box models also increasingly do not provide likelihood values (GPT-3 does, GPT-4 does not) that are necessary for computing the continuous probability density functions for time series. While these are significant obstacles when using LLM APIs, we hope that open source LLMs such as LLaMA can alleviate many of these issues, enabling direct control over prompting, tokenization, and likelihood calculations.”


Meta-review: This paper was a borderline paper and judging based on scores alone this paper should be rejected. However, I took the chance to read some of the review comments and also the paper itself. Skimming the paper, I find the paper to be of potential value to the community and I think it's interesting enough to be accepted especially with the impact multiplier of LLMs. I personally would be interested to read this paper so this is worth acceptance. 

I also noticed the reviewer comments about not liking the attitude of the paper, which initially I thought could be a case of the authors "overclaiming". However, taking a look at the paper itself I find that that is not the case. In fact, I thought it was quite a tasteful presentation of the research questions.

Therefore, I recommend acceptance.
Decision: Accept (poster)