Title: Large Language Models Are Zero-Shot Time Series Forecasters
Venue: NeurIPS 2023
Accepted: yes
Keywords: large language models

time series

probabilistic forecasting
Abstract: By encoding time series as a string of numerical digits, we can frame time series forecasting as next-token prediction in text. Developing this approach, we find that large language models (LLMs) such as GPT-3 and LLaMA-2 can surprisingly zero-shot extrapolate time series at a level comparable to or exceeding the performance of purpose-built time series models trained on the downstream tasks. To facilitate this performance, we propose procedures for effectively tokenizing time series data and converting discrete distributions over tokens into highly flexible densities over continuous values. We argue the success of LLMs for time series stems from their ability to naturally represent multimodal distributions, in conjunction with biases for simplicity, and repetition, which align with the salient features in many time series, such as repeated seasonal trends. We also show how LLMs can naturally handle missing data without imputation through non-numerical text, accommodate textual side information, and answer questions to help explain predictions.  While we find that increasing model size generally improves performance on time series, we show GPT-4 can perform worse than GPT-3 because of how it tokenizes numbers, and poor uncertainty calibration, which is likely the result of alignment interventions such as RLHF.

Large Language Models Are Zero-Shot Time Series Forecasters

Abstract

By encoding time series as a string of numerical digits, we can frame time series forecasting as next-token prediction in text. Developing this approach, we find that large language models (LLMs) such as GPT-3 and LLaMA-2 can surprisingly zeroshot extrapolate time series at a level comparable to or exceeding the performance of purpose-built time series models trained on the downstream tasks. To facilitate this performance, we propose procedures for effectively tokenizing time series data and converting discrete distributions over tokens into highly flexible densities over continuous values. We argue the success of LLMs for time series stems from their ability to naturally represent multimodal distributions, in conjunction with biases for simplicity, and repetition, which align with the salient features in many time series, such as repeated seasonal trends. We also show how LLMs can naturally handle missing data without imputation through non-numerical text, accommodate textual side information, and answer questions to help explain predictions. While we find that increasing model size generally improves performance on time series, we show GPT-4 can perform worse than GPT-3 because of how it tokenizes numbers, and poor uncertainty calibration, which is likely the result of alignment interventions such as RLHF.

1 Introduction

Despite similarities with other sequence modeling problems, such as text, audio, or video, time series has two particularly challenging properties. Unlike video or audio, which typically have consistent input scales and sampling rates, aggregated time series datasets often comprise sequences from radically different sources, sometimes with missing values. Moreover, common applications of time series forecasting, such as weather or financial data, require extrapolating from observations that contain a tiny fraction of the possible information, making accurate point predictions nearly impossible and uncertainty estimation especially important. While large-scale pretraining has become a key element of training large neural networks in vision and text, enabling performance to scale directly with data availability, pretraining is not typically used for time series modeling, where there is no consensus unsupervised objective and large, cohesive pretraining datasets are not readily available. Consequently, simple time series methods (e.g. ARIMA [8], and linear models [52]) often outperform deep learning methods on popular benchmarks [24].

In this paper, we demonstrate how large language models (LLM) can naturally bridge the gap between the simple biases of traditional methods and the complex representational learning and generative abilities of modern deep learning. In particular, we introduce an exceedingly simple method, LLMTIME 2 , to apply pretrained LLMs for continuous time series prediction problems, illustrated at a high level in Figure 1. At its core, this method represents the time series as a string of numerical digits, and views time series forecasting as next-token prediction in text, unlocking

LLM

Prompt

Samples

"and fell asleep"

"The dog jumped up on the bed"

"631, 656, 650, ..., 487, 485, 487"

"479, ..., 371, 364"

"492, ..., 499, 501"

"and bit my leg"

the use of powerful pretrained models and probabilistic capacities, such as likelihood evaluation and sampling. To enable strong performance, we propose techniques to (1) effectively encode time series as a string of numerical digits and (2) adapt the discrete distributions of LLMs to continuous densities capable of modeling sophisticated multimodal distributions. Using these techniques, we find LLMTIME can exceed or match purpose-built time series methods over a range of different problems in a zero-shot fashion, meaning that LLMTIME can be used without any fine-tuning on the downstream data used by other models.

The zero-shot nature of LLMTIME carries several natural advantages: (1) it facilitates the straightforward application of LLMs, eliminating the necessity for specialized knowledge of fine-tuning procedures and the substantial computational resources required for these procedures, as well as sidestepping access issues surrounding proprietary source code or APIs for LLM training or fine-tuning; (2) it is naturally suited to scenarios with limited data availability, where there is little information for training or fine-tuning; (3) by leveraging the broad pattern extrapolation capabilities of extensively pre-trained LLMs, it circumvents the extensive time, effort, and domain-specific expertise typically required for crafting dedicated time series models.

To understand the origins of LLMTIME's impressive performance, we investigate how LLMs express preferences for simple or repetitive sequences [19] and show that these biases are in fact compatible with the salient structure of time series, such as seasonality. Aside from these biases, LLMs also can naturally acccommodate missing data, and express multimodal distributions, which is particularly useful for time series. We also show how LLMs enable appealing functionality, such as the ability to provide additional side information through prompting, and query the LLM to explain its predictions.

Finally, in addition to broadly compelling forecasting performance, we find performance tends to improve with scale, and the quality of point predictions also improves with the quality of the uncertainty representation. However, we also find GPT-4 has worse uncertainty calibration than GPT-3, likely due to interventions such as reinforcement learning by human feedback (RLHF).

2 Background

Language modeling Language models are trained on a collection of sequences, U = { U 1 , U 2 , . . . U i , . . . , U N } , where U i = ( u 1 , u 2 , . . . , u j , . . . , u n i ) and each token, u i , belongs to a vocabulary V . Large language models typically encode an autoregressive distribution, in p θ ( U i ) = n i p θ ( u j u 0: j 1 ) . The parameters, θ , are learned by maximizing the probability of the entire dataset p θ ( U ) = i =1 p θ ( U i ) . Every language model has an associated tokenizer , which breaks an input string into a sequence of tokens, each belonging to V . Proper tokenization is extremely important, and small details can have surprisingly significant effects. The most common tokenization method for autoregressive language models is byte-pair encoding (BPE), which treats inputs like

which the probability of each token is only dependent on the previous tokens in the sequence, ∏ j =1 | -∏ N

bit strings and assigns tokens based on the rate of occurrence in the training corpus, optimizing for shorter sequences of tokens on average. Sampling from a language model typically starts with a prompt , u 0: k , and proceeds sequentially using p θ ( u j | u 0: j -1 ) , which is often preprocessed, for example through temperature scaling or nucleus sampling [25].

Large language models Brown et al. [9] showed that increasing a language model's parameter count and training data size leads to new capabilities such as zero-shot generalization , in which a model can perform a text-formatted task without training the model parameters on any task-specific data. Large language models, for example GPT-3 [9] or LLaMA-2 [44], accomplish this form of generalization through in-context learning , which identifies patterns in the language model's prompt and extrapolates them through next-token prediction. Many authors have speculated that in-context learning emerges from a language model's extensive compression of the input data [19, 42, 15]. Compression favors learning algorithms that operate over the input data with programmatic abstractions, for example, context-free grammars [2] or induction heads [34], which can implement copy-and-paste type operations for generating samples with highly structured syntax. In this work, we show that the zero-shot generalization abilities of LLMs and their preference for compressible patterns extend well beyond language understanding and can be used for time series forecasting.

Zero-shot generalization has made LLMs significantly more useful as assistants, leading to the create of methods to align LLMs with human preferences and instructions, for example reinforcement learning from human feedback (RLHF) [38] and instruction tuning [46]. While key to modern LLMs products, alignment methods can also significantly affect the abilities and calibration of the underlying model [36, 10]. Here we show these methods can also affect forecasting ability.

Time series data Time series data typically takes the exact same form as language modeling data, as a collection of sequences { U i = ( u 1 , . . . , u j , . . . , u n i ) } , but in time series u j is numerical. Because language models are built to represent complex probability distributions over sequences, they are theoretically well-suited for time series modeling. In practice, however, language models are held back by the details of tokenizing numbers. BPE compresses numbers based on frequency of occurrence in the training data, so numbers can be broken down into awkward chunks that make learning basic numerical operations challenging. Touvron et al. [43] therefore designed the LLaMA tokenizer to map numbers to individual digits, which can lead to significant improvements in mathematical abilities, with small LLaMA models outperforming GPT-4 [30].

The other challenge of applying language models to time series data is proper evaluation. Mean absolute error (MAE) is commonly used but ignores uncertainty in the forecast which is highly limiting for stochastic data [24, 6]. Continuous ranked probability score (CRPS) captures distributional qualities and can compare models that generate samples without likelihoods. For a single prediction, the CRPS score is defined against the estimated cumulative distribution function (CDF), ˆ F as CRPS ( ˆ F,y ) = ∫ R ( ˆ F ( z ) -I ( z -y ) > 0 ) 2 dz, where ˆ F ( z ) is the empirical CDF produced by sampling forecasts and I is the indicator function. While CRPS is an improvement over MAE, it also ignores key structures in the data, such as correlations between time steps. Fortunately, language models can assign likelihoods to full sequences of time series data, and we show how a small modification to an LLM's discrete likelihood can yield a continuous density that is useful for model comparison.

Language models for time series Several authors have explored using pretrained language model encoders as initializations for time series models. For example, Zhou et al. [56] propose FPT, which finetunes a BERT encoder to perform time series forecasting. Similarly, Zhang et al. [53] introduce Meta-Transformer, a framework for finetuning a language model for non-text modalities, including time series. Fewer papers explore using LLMs as forecasters without finetuning. The only method we are aware of is PromptCast [50], which poses forecasting as question answering with prompting.

Our work Unlike methods that leverage LLM backbones, our method is entirely zero-shot and does not require finetuning. Unlike PromptCast, we show that LLMs can be used directly as forecasters without any added text or prompt engineering, if we carefully preprocess the numerical values themselves. Our method solely relies on LLM's abilities to extrapolate patterns in general sequences and nothing particular to English or any other language. Going beyond prior work, we also cultivate the probabilistic nature of large language models and their ability to capture uncertainty over highly stochastic time series.

"151，167，..,267"

"151,167.,...,267""

"151,167,..,267"

"151,167.,.,267"

7

5

6

7

267"

"151,167,

267"

151,167,

,267

151,167,

267

MM

MMM

GPT-3 spaces

GPT-3nospaces

LLaMAspaces

LLaMAnospaces

3 LLMTIME: Forecasting with Language Models

Forecasting with LLMTIME has relatively few steps. Once the numerical values are processed into strings, making predictions with the language model follows standard sampling procedures. As we show next, however, correct pre-processing is not always intuitive but is extremely important, and incorrect handling can lead to unusable predictions.

Tokenization Tokenization is particularly important because it directly influences how patterns form within tokenized sequences and the types of operations that language models can learn. Unfortunately, common tokenization methods like BPE tend to break a single number into tokens that don't align with the digits, which can make arithmetic considerably more difficult [30]. For example, the number 42235630 gets tokenized as [422 , 35 , 630] by the GPT-3 tokenizer, and changes by even a single digit can result in an entirely different tokenization. By contrast, in many new open-source LLMs (e.g. LLaMA [43]), numbers are tokenized into individual digits by default. To remedy the tokenization of GPT models, we separate the digits with spaces to force a separate tokenization of each digit and use a comma (" ,") to separate each time step in a time series. Because decimal points are redundant given a fixed precision, we drop them in the encoding to save on context length. Thus, with e.g. 2 digits of precision, we pre-process a time series as follows before feeding into the tokenizer:

In Figure 2, we show that the added spaces of this encoding are helpful for GPT models, preventing the model from getting derailed by outputting an unusual token during sampling. For LLaMA models, with their unique tokenization of numbers, added spaces have the opposite effect. Each digit and space is already assigned its own token, and space tokens become nuisance inputs, adding to the sequence length without simplifying the sequence's structure and potentially making the sequence out-of-distribution to the model.

Rescaling To avoid wasting tokens when the inputs are very large, we scale values down so that the α -percentile of rescaled time series values is 1 . We avoid scaling by the maximum value so that the LLM can see some fraction of examples ( 1 -α ) where the number of digits changes and reproduce this behavior in its outputs to produce larger values than it has seen. We also experiment with an offset β based calculate as a percentile of the input data, and we tune these two parameters on validation log likelihoods (details in Appendix A).

Sampling / Forecasting To forecast, draw many samples (e.g. 20) from the LLM and use the statistics of the samples at each time step to construct a point estimate (e.g. as the median) or probabilistic forecast (e.g. as quantiles). To control sampling, we use temperature scaling, logit bias, and nucleus sampling (Appendix C).

Continuous likelihoods Modeling sequences of individual digits has additional benefits beyond good samples. With n digits of precision in base B , each sequence of n digits after the decimal place corresponds to one of B n possible bins (Figure 3), each with width B -n . As each distribution p ( u j | u 0: j -1 ; θ ) is a softmax over possible digits, we can view the distribution over each individual number as a hierarchical softmax [31], with p ( u 1 , ..., u n ) = p ( u n | u n -1 , ..., u 0 ) p ( u 1 | u 0 ) p ( u 0 ) . Though a language model's probability distribution is discrete, we can easily adapt it to provide a

P(u j  | u 0:j-1 )

P("3"|"5")

P("3"|"5")

P("7"|"53")

P("7"|"53")

P("5")

P("5")

["5","3","7"]

P(0.537)

P(0.537) =

Tokenizer

0.537

"537"

Encoder

.

.

continuous density by placing a uniform distribution in each bin. Enumerating each of the countably infinite numbers that the model can produce (because the model can output an arbitrary number of digits before the decimal point) with an index k ∈ N each with probability p k , we can write out the distribution as a mixture of disjoint uniform distributions over the bins p ( x ) = ∑ k ∈ N p k U k ( x ) where U k ( x ) = B n I x ∈ [ B -n k,B -n ( k +1)) . Therefore if a given data point lies in bin k , its continuous log likelihood is log p ( x ) = log p k + n log B . Finally, to obtain the likelihood log p ( z ) in the original input space, we add a change of variables factor log | dx dz | , where z ↦→ x = s ( z ) is the rescaling operation in the pre-processing. As a result, the exponentially large number of bins and exponentially small bin widths enabled by our construction make it surprisingly efficient to represent flexible and high-resolution continuous distributions with LLMs, despite using a discrete tokenization of numbers.

Language models as flexible distributions The fact that LLMs can express flexible distributions over numbers is key for time series data. Uncertainty quantification is essential to forecasting, and typical approaches to representing uncertainty in time series can be limited by misspecification. For example, one common method for creating a probabilistic forecast is to fit a Gaussian or Laplace observation model. When the underlying data distribution is multimodal, both of these models will perform poorly. Methods like Gaussian mixture models (GMMs) solve the issue of multimodality but introduce additional challenges to optimization and model selection. We show that a language model is an underrated solution by training a small autoregressive model on a variety of one-dimensional distributions shown in Figure 3 (right). These distributions come from an exponential random variable, a mixture of a uniform and a student-t distribution, and the heavy-tailed distribution of time series prediction residuals from an ARIMA model on the MonthlyMilk dataset [23]. We evaluate these fits quantitatively by computing Wasserstein distances, and compare to a Laplace observation model, a GMM trained with expectation-maximization, and logistic regression over a flat binning of the data (with a tuned bin size). Each model is trained with only 200 samples from the distribution. The results show that the decimal autoregressive language model ('Decimal AR') performs extremely well, handling asymmetric, multimodal, and heavy-tailed distributions, which are among the diverse types characteristic of time series data.

4 Experiments

We evaluate the zero-shot forecasting ability of LLMs by comparing LLMTIME with GPT-3 and LLaMA-2 70B to many popular time series baselines on a variety of benchmark time series datasets. Not only is LLMTIME able to generate plausible completions of the real and synthetic time series, it achieves higher likelihoods and CRPS values in zero-shot evaluation than the dedicated time series models like ARIMA, TCNs, and N-HiTS. When evaluated on deterministic metrics like MAE, LLMs also perform well, obtain the best or second best MAE values on each benchmark. As we are using

LLMs with undisclosed datasets, data leakage is an important concern that we address directly in Appendix B. Beyond strong performance on standard benchmarks, which are the most useful for comparison, we find that LLMTIME also performs well on datasets that could not have been present in the base model's training data. The full set of hyperparameters used for LLMTIME and the baseline methods are detailed in Appendix C.1. For some of the longer time series, not all of the history can be fit into the context window, and hence hyperparameters implicitly capture the trade-off between higher precision and capturing a larger temporal history.

Datasets We use three benchmark datasets that are common within deep learning research and many baseline methods that accompany the benchmark datasets.

Darts [23]: A collection of 8 real univariate time series datasets. For Darts, we use several methods that are implemented directly in the package, including neural network models (TCN [28], N-BEATS [37], N-HiTS [11]) and simple moving average models (ARIMA [8]). Darts enables learning observation models with tractable likelihoods and is therefore especially useful for benchmarking the probabilistic predictions of LLMTIME. We also include Spectral Mixture Gaussian Process (SM-GP) [48], a Bayesian nonparametric method (details in Appendix C.1).

Monash [18]: The Monash forecasting archive contains 30 publicly available datasets along with baseline numbers for 12 forecasting models, including simple exponential smooth (e.g. ETS [26]), gradient boosting (e.g. CatBoost [39]) and deep learning models (e.g. DeepAR [40], WaveNet [35]). The Monash archive comprises over 400,000 individual time series, making it infeasible to use in its entirety with the largest available LLMs. To reduce the computational burden, we evaluate GPT-3's zero-shot performance on 19 datasets described in Appendix C.2.

Informer [54]: We evaluated on multivariate datasets widely used for benchmarking efficient transformer models [16, 54]. In order to predict multivariate data with LLMTIME, we forecast each covariate independently. We baseline against numbers obtained by running public implementations from the Autoformer [49] and FEDFormer [55] codebases (Appendix C.3).

Deterministic results To compute MAE values for LLMTIME we use the pointwise median of 20 samples from the base model (GPT-2 or LLaMA-2 70B). Figure 4 shows that deterministic predictions from LLMTIME are ranked best or second best on all the considered benchmarks while having no trainable parameters. We provide visualizations of the forecasts in Appendix C.5/C.7/C.8.

Probabilistic results In Figure 5, we show several probabilistic evaluations on the Darts datasets, including aggregated NLL and CRPS numbers, as well as analysis of how each model reacts to decreasing the input data size. Evaluated on log likelihood and CRPS, LLMTIME considerably outperforms the baselines in aggregate and on almost every individual dataset (results per dataset included in Appendix C.5). Given the analysis of language model-derived densities in Section 3, it is unsurprising that language models excel in probabilistic evaluations, outperforming the baselines even

more dramatically. In Figure 5 (left) we show two informative examples that capture the performance of LLMTIME. When extrapolating the AirPassengers dataset, LLMTIME successfully identifies and continues trend and period components, with uncertainty that grows as predictions get further from the input data. On GasRateCO2, LLMTIME replicates local structure when there is relatively little global structure. In Figure 5 (right) we show that LLMTIME not only performs better than baselines with access to the full training data but also when restricted to small fractions of the training data. As time series is frequently characterized by relative data scarcity and challenges in transfer learning, the data efficiency of LLMs is especially attractive.

Comparison with PromptCast Though included in the results described above, we want to explicitly highlight that LLMTIME significantly outperforms PromptCast [50] when applied to both GPT-3 and LLaMA-2 70B, according to CRPS and MAE aggregated over the Darts datasets. This performance gap highlights important differences between the two approaches. Unlike our method, PromptCast formulates forecasting as a conventional question-answering task in NLP by prompting pre-trained language models with an explicit question about future values in a time series. For example, PromptCast feeds in the prompt 'The values in the WoolyDataset for the past 95 time steps are 6172, 6709, 6633, . . . , 6077. What will the values for the next 5 time steps be? The values for the next 5 time steps will be", to extract predictions from an LLM. PromptCast also does not apply our tokenization and data rescaling strategy (Section 3), which we show is crucial for good performance.

5 Origins of Zero-Shot Performance

To understand why LLMs can extrapolate time series in a zero-shot manner, let's take a step back and consider simple numerical sequences, for example [ 1 , 4 , 9 , 16 , . . . ] or [ 0 , 0 , 1 , 0 , 1 , 2 , 0 , 1 , 2 , 3 , . . . ]. For any input sequence, there are arbitrarily many generation rules that are consistent with the input (e.g. f ( x ) = x 2 for x ∈ [1 , 2 , 3 , 4 , ... ] ), but some generation rules are overly complex and will generalize poorly. LLMs can forecast effectively because they prefer completions derived from simple rules, adopting a form of Occam's razor prior [19, 42, 15]. To explicitly demonstrate this phenomenon, we create a synthetic example using the function f ( x ) = x +cos( x ) with additive Gaussian noise. We fit symbolic expressions to the first 70% of timesteps using PySR [14] with symbols [" + ", " · ", "-", " / ", " sin ", " cos ", " exp ","square"] to identify generating rules with known complexity, quantified by the number of symbols in the regressed expression (Appendix D). Figure 6 (left) shows the likelihood that GPT-3 assigns the highest likelihood to symbolic regression generating rules that balance consistency with complexity.

In Figure 6 (right) we show how program induction in LLMs leads to good zero-shot prediction for many deterministic patterns common in time series data. Along with samples, we also show

.

likelihoods, comparing against standard time series models, which often struggle to extrapolate these simple patterns because they cannot identify a programmatic generation rule to make predictions unlike those seen in the observed history. While the generic simplicity bias is helpful for identifying and extrapolating patterns in the input, a number of patterns common in time series models also translate directly to known capabilities of language models, for example

Repetition bias and periodicity : LLMs' bias towards repetitive sequences [25] (often unwanted in NLP) corresponds precisely to the ability to identify and extrapolate periodic structure in the input. 4 . 2 , 8 . 6 , 1 . 0 , 4 . 2 , 8 . 6 will lead to a 1 . 0 as a likely next output without any time series or arithmetic knowledge ( x t = x t -T ) .

Arithmetic and trend components : LLMs' ability to perform addition and multiplication [51, 30] maps on to extrapolating linear and exponential trends. For example, predicting the next element of 0 . 2 , 1 . 6 , 3 ., 4 . 4 the LLM needs only to add 1 . 4 to the last element ( x t +1 = x t + c ). Similarly, exponential trends have the generating rule x t +1 = c · x t and sigmoid trends have the generating rule x t +1 = x t + cx t (1 -x t ) .

Combining multiple patterns together presents a more difficult challenge, as it requires both identifying the composite pattern and being able to perform the multiple operations within the same token budget. Supposing that a model can perform copying in a single forward pass and addition in a single forward pass, that does not necessarily mean that it can do both simultaneously. We find that GPT-3 is only sometimes able to perform these compositions, though GPT-4 does so more consistently as shown in Appendix E. It is likely that the limitations on compute and tokens spent may make this composition unnecessarily hard, and that additional recursive structure, for example from a scratchpad [33], Chain of Thought (CoT) prompting [47], or adaptive computation [41, 3], would make this task easier.

6 Special Properties of LLMs

So far we've shown that LLMs are effective forecasters across a variety of datasets and that their forecasting ability arises from biases created by generative pretraining. LLMTIME offers a mechanism for large-scale pre-training that is uncommon in machine learning for time series. LLMs lessen the amount of time series data that must be aggregated for pretraining, substituting text pretraining in its place, and enable more powerful scaling results. Beyond escaping the limits of task-specific data, text pretraining also has many test-time benefits that stem from the base model's ability to process and generate natural language. As we show in the following section, LLMs can leverage their abilities in order to seamlessly incorporate missing data or answer questions about time series.

Base models and forecasting performance Given the rapid growth and improvement in opensource LLMs [43, 44], the relationship between LLMTIME forecasting performance and the performance of the underlying base model is particular important and has broad implications. Steady increases in LLM benchmark performance can directly translate to steady improvements in forecasting ability. In Figure 7 (right), we show a study with OpenAI models (davinci, babbage, curie, ada), variants of LLaMA [43] (7B, 13B, 33B, 65B) and LLaMA-2 [44] models (7B, 13B, 70B) measuring accuracy on the Massive Multitask Language Understanding (MMLU) benchmark and probabilistic forecasting error. As we might hope, when reasoning (MMLU) performance increases forecasts also improve.

Chat models Though convenient scaling relationships appear to hold for base models, the correlation begins to break down when we consider models that have been post-processed for chatbot applications. GPT-4 [36], for example, demonstrates considerably more intelligence than GPT-3 and LLaMA models in natural language tasks, but effectively applying it to time series is challenging. In Figure 7 (center), we show that GPT-4 has a forecasting error (CRPS) significantly larger than GPT-3's on the Darts datasets. The performance drop is the result of several small details in GPT-4's method. Due to the altered tokenization, GPT-4 cannot be easily forced to tokenize individual digits into an unbroken stream of numbers. Due to the restrictions on the API, likelihood evaluations are also not permitted, which is why we present results for only CRPS. While GPT-4 can perform well on the synthetic examples discussed in Section 5 (shown in Appendix E), we find that the GPT-4 calibration is much worse than GPT-3's on stochastic data, likely as a result of the preprocessing details above and the fact that the model has been treated with RLHF [13] which is known to degrade calibration on question-answering tasks [36]. GPT-4 is not the only example of degraded performance in models designed for chat functionality. We observed the same phenomenon in LLaMA-2 models, which have corresponding chat versions for every model size. Figure 7 (right) shows that chat versions tend to have markedly worse forecasting error than their non-chat counterparts, though still maintain trends in size and reasoning ability.

Missing data A major advantage of treating forecasting as a text completion task and using LLMs is that we can easily feed in any input that can be encoded as text. Often in time series, the time series will be incomplete and certain observations are missing. Simple imputation methods, such as nearest neighbor, are still core pre-processing steps in common data science workflows [32], and the choice of imputation method is especially relevant to clinical data, which often contains irregularly sampled measurements and where missingness can be meaningful signal in itself [27]. Much like humans reading partial reports, LLMs can handle missing values without imputation by adopting special symbols, for instance,

. In Figure 8 we compare likelihoods and CRPS value for forecasts from traditional time series methods and LLaMA-2 70B on data that has been corrupted with missing values and then processed with linear interpolation and the above string formatting. While the likelihoods of traditional methods rapidly deteriorate with corruptions, we find that LLaMA-2 70B is more resilient, and when comparing CRPS values, LLaMA-2 70B is competitive with methods that use interpolation.

Connecting time series and textual understanding Because LLMs are designed for natural language and code, we can augment the numerical time series with useful text. We can do so either by providing textual side information as inputs, or by producing textual outputs from a given time series. An interesting question is whether GPT-4 can explain in text its understanding of a given time series. We probe this quality by providing GPT-4 the code to generate our synthetic time series, provide the values of one these time series, and then ask it to infer which of the functions produced the data in a zero-shot manner. The prediction accuracies are shown in Figure 8, with the three remaining rows all being 0 . With CoT [47] prompting the model performs much better than random chance; however, its ability to identify patterns better when directly extrapolating the numerical data, suggesting that its numerical understanding is not fully connected to its textual understanding. In making predictions, the model often explains properties of the time series in order to select the right candidate from the list, and we show several of these sample explanations in Appendix F. We also show how this task is encapsulated in a simple (unprompted) next token prediction problem on cells of a Jupyter notebook, illustrating why we expect such capabilities to emerge with a sufficiently powerful language model.

7 Discussion

We have demonstrated that large language models can be used as pretrained time series forecasters by encoding numerical values as text. As with other 'foundation' models, pretraining confers useful biases toward generalizable patterns that would otherwise be engineered into the model through architecture design [21], and enables natural scaling of performance with improvements in the base pretrained model. Because LLM forecasters are trained on language, they also confer unconventional capabilities, such as question answering. More broadly, framing time series forecasting as natural language generation can be seen as another step towards unifying more capabilities within a single large and powerful model, in which understanding can be shared between many tasks and modalities. Moreover, zero-shot forecasting can enable broadly compelling performance without requiring significant computational resources, domain expertise, or many downstream training data points.

While LLM forecasters benefit from the strengths of pretrained transformers, they also inherit their weaknesses, which can include a limited context window. While many univariate time series problems can fit comfortably within increasingly large context windows, multivariate problems pose a more significant challenge. There have been several recent advances extending LLM context windows to 10-100K tokens [36, 4, 5, 1]. Combining these advances with time series forecasting is a particularly exciting direction for future research. Another potential challenge of using current LLMs architectures could be their weakness in arithmetic and performing recursive and compositional operations, which could be a limitation on particularly challenging time series. On the other hand, many time series do not require precise arithmetic. Understanding the extent to which this is the case, and relaxing this limitation, is also a promising avenue for future research. Separately from any limitation, it would also be promising to investigate effective procedures for fine-tuning LLMs on time series. We hope that bridging LLM research with time series forecasting brings benefits to both communities.

Acknowledgements. We thank Micah Goldblum, Greg Benton, and Wesley Maddox for helpful discussions. This work is supported by NSF CAREER IIS-2145492, NSF I-DISRE 193471, NSF IIS-1910266, BigHat Biosciences, Capital One, and an Amazon Research Award.

References

Joshua Ainslie, Tao Lei, Michiel de Jong, Santiago Ontañón, Siddhartha Brahma, Yury Zemlyanskiy, David Uthus, Mandy Guo, James Lee-Thorp, Yi Tay, et al. Colt5: Faster long-range transformers with conditional computation. arXiv preprint arXiv:2303.09752 , 2023.

Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 1, context-free grammar. arXiv preprint arXiv:2305.13673 , 2023.

Cem Anil, Ashwini Pokle, Kaiqu Liang, Johannes Treutlein, Yuhuai Wu, Shaojie Bai, J Zico Kolter, and Roger B Grosse. Path independent equilibrium models can better exploit test-time computation. Advances in Neural Information Processing Systems , 35:7796-7809, 2022.

Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403 , 2023.

Anthropic. Introducing 100k context windows. Anthropic blog, 2023. URL https://www. anthropic.com/index/100k-context-windows .

Gregory Benton, Nate Gruver, Wesley Maddox, and Andrew Gordon Wilson. Deep probabilistic time series forecasting over long horizons. openreview preprint , 2022. URL https://openreview.net/forum?id=22h1XSEiN0 .

Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raf. Emergent and predictable memorization in large language models. arXiv preprint arXiv:2304.11158 , 2023.

George EP Box and Gwilym M Jenkins. Some recent advances in forecasting and control. Journal of the Royal Statistical Society. Series C (Applied Statistics) , 17(2):91-109, 1968.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 , 2023.

Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza, Max Mergenthaler, and Artur Dubrawski. N-hits: Neural hierarchical interpolation for time series forecasting. arXiv preprint arXiv:2201.12886 , 2022.

Kent K Chang, Mackenzie Cramer, Sandeep Soni, and David Bamman. Speak, memory: An archaeology of books known to chatgpt/gpt-4. arXiv preprint arXiv:2305.00118 , 2023.

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 30, 2017.

Miles Cranmer. Interpretable machine learning for science with pysr and symbolicregression. jl. arXiv preprint arXiv:2305.01582 , 2023.

Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, et al. Language modeling is compression. arXiv preprint arXiv:2309.10668 , 2023.

Dazhao Du, Bing Su, and Zhewei Wei. Preformer: Predictive transformer with multiscale segment-wise correlations for long-term time series forecasting. arXiv preprint arXiv:2202.11356 , 2022.

Jacob R Gardner, Geoff Pleiss, David Bindel, Kilian Q Weinberger, and Andrew Gordon Wilson. Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu acceleration. In Advances in Neural Information Processing Systems , 2018.

Rakshitha Godahewa, Christoph Bergmeir, Geoffrey I Webb, Rob J Hyndman, and Pablo Montero-Manso. Monash time series forecasting archive. arXiv preprint arXiv:2105.06643 , 2021.

Micah Goldblum, Marc Finzi, Keefer Rowan, and Andrew Gordon Wilson. The no free lunch theorem, kolmogorov complexity, and the role of inductive biases in machine learning. arXiv preprint arXiv:2304.05366 , 2023.

Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning . MIT press, 2016.

Nate Gruver, Marc Finzi, Micah Goldblum, and Andrew Gordon Wilson. The lie derivative for measuring learned equivariance. arXiv preprint arXiv:2210.02984 , 2022.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 , 2020.

Julien Herzen, Francesco Lässig, Samuele Giuliano Piazzetta, Thomas Neuer, Léo Tafti, Guillaume Raille, Tomas Van Pottelbergh, Marek Pasieka, Andrzej Skrodzki, Nicolas Huguenin, et al. Darts: User-friendly modern machine learning for time series. The Journal of Machine Learning Research , 23(1):5442-5447, 2022.

Hansika Hewamalage, Klaus Ackermann, and Christoph Bergmeir. Forecast evaluation for data scientists: common pitfalls and best practices. Data Mining and Knowledge Discovery , 37(2): 788-832, 2023.

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751 , 2019.

Rob J Hyndman, Anne B Koehler, J Keith Ord, and Ralph D Snyder. Forecasting with exponential smoothing: the state space approach . Springer Science & Business Media, 2008.

Shruti Kaushik, Abhinav Choudhury, Pankaj Kumar Sheron, Nataraj Dasgupta, Sayee Natarajan, Larry A Pickett, and Varun Dutt. Ai in healthcare: time-series forecasting using statistical, neural, and ensemble architectures. Frontiers in big data , 3:4, 2020.

Colin Lea, Rene Vidal, Austin Reiter, and Gregory D Hager. Temporal convolutional networks: A unified approach to action segmentation. In Computer Vision-ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14 , pages 47-54. Springer, 2016.

Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499 , 2021.

Tiedong Liu and Bryan Kian Hsiang Low. Goat: Fine-tuned llama outperforms gpt-4 on arithmetic tasks. arXiv preprint arXiv:2305.14201 , 2023.

Andriy Mnih and Geoffrey E Hinton. A scalable hierarchical distributed language model. Advances in neural information processing systems , 21, 2008.

Steffen Moritz and Thomas Bartz-Beielstein. imputets: time series missing value imputation in r. R J. , 9(1):207, 2017.

Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114 , 2021.

Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread , 2022. https://transformer-circuits.pub/2022/in-context-learning-and-inductionheads/index.html.

Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. In 9th ISCA Speech Synthesis Workshop , pages 125-125. ISCA, 2016.

OpenAI. Gpt-4 technical report. arXiv , 2023.

Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. Journal of Machine Learning Research , 21(111):1-63, 2020.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022.

Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. In Advances in Neural Information Processing Systems , volume 31, pages 6638-6648. NeurIPS, 2018.

David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting , 36(3): 1181-1191, 2020.

Avi Schwarzschild, Eitan Borgnia, Arjun Gupta, Furong Huang, Uzi Vishkin, Micah Goldblum, and Tom Goldstein. Can you learn an algorithm? generalizing from easy to hard problems with recurrent networks. Advances in Neural Information Processing Systems , 34:6695-6706, 2021.

Ilya Sutskever. An observation on generalization. Workshop on Large Language Models and Transformers, 2023. URL https://www.youtube.com/watch?v=AKMuA_TVz3A&ab_ channel=SimonsInstitute .

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. ArXiv , abs/2307.09288, 2023.

Juan R Trapero, Nikolaos Kourentzes, and Robert Fildes. On the identification of sales forecasting models in the presence of promotions. Journal of the operational Research Society , 66(2): 299-307, 2015.

Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 , 2021.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 , 2022.

Andrew Wilson and Ryan Adams. Gaussian process kernels for pattern discovery and extrapolation. In International conference on machine learning , pages 1067-1075. PMLR, 2013.

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems , 34, 2021.

Hao Xue and Flora D. Salim. Promptcast: A new prompt-based learning paradigm for time series forecasting, 2023.

Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, and Songfang Huang. How well do large language models perform in arithmetic tasks? arXiv preprint arXiv:2304.02015 , 2023.

Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? arXiv preprint arXiv:2205.13504 , 2022.

Yiyuan Zhang, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue. Meta-transformer: A unified framework for multimodal learning. arXiv preprint arXiv:2307.10802 , 2023.

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of AAAI , 2021.

Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Proc. 39th International Conference on Machine Learning (ICML 2022) , 2022.

Tian Zhou, Peisong Niu, Xue Wang, Liang Sun, and Rong Jin. One fits all: Power general time series analysis by pretrained lm. arXiv preprint arXiv:2302.11939 , 2023.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper discusses a novel approach for time series forecasting by encoding time series as numerical digits and treating it as next-token prediction in text. Surprisingly, it is found that large language models (LLMs) like GPT-3 can effectively extrapolate time series even without being specifically trained for this task. The success of LLMs in time series forecasting is attributed to their ability to extrapolate deterministic patterns, such as seasonality, and model uncertainty by using expressive distributions on continuous variables. It is argued that LLMs excel in predicting numerical sequences due to biases introduced by their text pre-training, which favors simple explanations. Additionally, LLMs demonstrate excellent representation of multimodal distributions by decomposing the distribution over digits. This combination of qualities allows LLMs to achieve strong performance on time series data and enable new capabilities such as integrating non-numerical text and handling missing data. Furthermore, the study shows that the performance of LLMs in time series prediction improves with increased scale, suggesting significant potential for this unconventional approach.\n\nSTRENGTHS:\n1. The paper proposed a method that utilize the pretrained language model to solve time series forecasting problems.\n2. This paper proposes a novel timing coding approach that can help LLMS understand timing data.\n3. Also, the authors identify why language models are able to perform well time-series prediction task.\n\nWEAKNESSES:\n1. The experimental results are not enough to support the viewpoint put forward in the paper. For example, most experiments are conducted on virtual data sets, but how are the effects on real data sets is not enough.\n\n2. For some of the methods proposed in this paper, their effectiveness has not been fully analyzed, and there is a lack of analysis of Tokenization, scale, etc.\n\n3.As can be seen from the results in Figure 4, the proposed method is effective only for some simple time series data (such as obvious periodicity or trend), but not for some more complex time series data (such as wooly or HeartRate).\n\nQUESTIONS:\n1. The contributions are not very clear, and it seems that the authors are only utilized the GPT models.\n\n2. The authors claim they explain why LLMs can be used for timing prediction, but there is not enough analysis in the paper.\n\n3. Why are most experiments conducted on virtual datasets and not on more real-world datasets?\n\nLIMITATIONS:\nSee weakness and questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates to which extent an off-the-shelf large language model like GPT-3 can be used for time series forecasting. Its main claim is that forecasting performance can be very good, provided some care is taken in correctly encoding the input. Some limited experiments are proposed to support the claim.\n\nSTRENGTHS:\n- The paper is definitely timely in the sense that  large language models (LLM) are very trendy and that many other works already showed that they are good at transfer learning. In this respect, attempting to use them for forecasting makes perfect sense and this paper could possibly have a good impact.\n- It cannot be denied that the reported performance is good, which supports the claim in a fascinating way.\n\nThere are some other limited contributions, notably regarding how the input should be encoded before using the LLM for forecasting, but I see them as very secondary when compared to the two mentioned above. Although I am recommending rejection for the reasons that I will describe below, I still must acknowledge that this paper asks a relevant and interesting question and is bound to meet some good impact, even if it is not presented at Neurips.\n\nWEAKNESSES:\n- If I had to summarize what I feel is my biggest concern with this paper, I would say that I feel uncomfortable with its _attitude_. As I mentioned above, it asks a question that I definitely think is interesting, but as it currently reads, it basically claims forecasting is solved by the method proposed, that outperforms state of the art in time-series forecasting.\nIn my view, the authors should have rather mentioned that using a LLM for forecasting interestingly seems to bring some very good performance in the experiments they did, but should have insisted much more on how much this should be understood today as some kind of funny curiosity at this stage. Indeed, there is something that is completely forgotten here: those LLM just cannot reasonably be used for forecasting in any realistic setup, that requires handling hundred thousands or million samples. What the paper convincingly shows is only that LLM can yield very fascinating performance for forecasting in the toy setups that were considered, and this is already a big deal.\n\n- the paper contains many arguable statements that I will detail below. As a researcher in time-series, I must say that many statements there read pretty pejorative towards research in time-series and sound like \"look how NLP will solve this problem that you have been struggling with for years\".\n\nQUESTIONS:\n- L19 time series has\n- L19 \"time series are relatively unstructured\": this is arguable. There is host of research precisely leveraging structure in time-series. For a very clear example, just consider speech, audio or music processing, where structure is everywhere.\n- L30 \"pretraining has not impacted time series forecasting to the same degree\": pretty arguable and gratuitous statement\n- L35: in a natural way\n- L52 \"instead of building towards a foundation model paradigm\". I would say that this statement is not only arguable, it is very wrong and can even be felt as pretty pejorative for the host of researchers working in the field. The authors could for instance check out the many foundational books or papers that were written on modelling stochastic processes or in developing information theory and that all emerged in a time-series setup. Regarding recent foundational approaches, one could for instance mention neural processes or related fundamental research.\n- L54-64: \"uncertainty quantification for time series also tends to be domain-specific and ad hoc\": I wonder what makes the authors think that NLP is free of any such bias. The whole paragraph has a strange pejorative feel to it that I cannot understand and that I think is inappropriate.\n- L64: limited expressive\n-  L65 \"by directly numbers as strings of digits\" ?\n- Section 3.1: there is no reference whatsoever to source coding or arithmetic coding and their host of deep-learning based recent variants, which appear to me as a clear _foundational_ setup for the _adhoc_ method proposed here.\n- L130-L139: Evaluation. It doesn't really make sense to me to compare the proposed RNN-method for quantization with parametric distribution models like GMM or Laplace, etc. The authors should have compared instead with a quantile-based model where the number of quantiles corresponds to the number of parameters of the RNN they fitted. \n- The \"occam razor\" discussion from section 3.2 is interesting, where the authors show that in some toyish cases, a LLM is able to extrapolate a numerical sequence. Still, they then generalize this interesting feature they observed on toyish examples as a general feature of LLM, which is not supported in any way but is only a \"working assumption\" and should be mentioned as such.\n- L149: \"time series generated according to a deterministic function can be considered as only a small generalization of these numerical sequences\": This is of course not true and the authors should check the host of different ways that were proposed in the past to model deterministic signals and their incredible complexity. \n- L157: \"maps on precisely to extrapolating linear trends\" ?\n- L168: it is important that the authors detail how the method is extended to autoregressively predict digit sequences to multiple numbers, since this is a very important ingredient of the approach.\n- Figure 7: I do not understand where history length appears here\n- Figure 8: As a transformer, I would expect TCN to actually easily handle missing values.\n\n- In the conclusion: I would not say that the toy experiments proposed are enough to write that \"LLM can act as extremely capable time series forecasters\" in a _general_ manner. Once again, the paper makes a very nice case for very small sequences and asks a relevant question, but this remains only a funny curiosity for practitioners at this stage, considering how hard scaling to realistic time series appears. For instance, one second of audio sampled at 44.1kHz means 44100 samples. If each one ends up as a word, this would mean something like 50 pages of text, and this is only for mono. People routinely process hours of multichannel audio. Of course, other time-series practitioners have similar or even much more challenging scales to handle. \n- The authors are apparently unaware of the fact that a very fruitful research has long been done on multimodal signal processing in time-series contexts.\n\nLIMITATIONS:\nWhat I wonder after reading this paper is whether time-series models could be useful for text processing.\n\n-- \nAfter reading the rebuttal, I don’t think the authors can be trusted to make the huge amount of changes it would take to address my concerns.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to use LLMs for time series prediction. \nPrecisely, they develop a way to tokenize and encode numerical digits and prompt LLMs to generate future numbers given the past. \nTheir method is comparable to or exceeds the state-of-the-art time series models.\n\nSTRENGTHS:\nThe paper proposes a very novel method to use LLMs for time series prediction. The problem is a high-impact problem, and their way of using LLMs is creative. I was excited when reading this paper. \n\nThe method is very carefully designed and is very convincing. Each component of the method (e.g., digit representation, tokenization) has been carefully thought through and demonstrated with insightful experiments and analysis. \n\nThe results are positive.\n\nWEAKNESSES:\nI would like to recommend this paper for acceptance, and I gave a positive score. \n\nThe main reason that I didn't give a higher score is its presentation and the limitation of the proposed method. \n\nFor presentation, I think the current structure needs improvement and some terminologies need clarification. \n\nThe current presentation is a little bottom-up: 3.1 explains how to represent numbers and 3.2 discusses how LLMs process and generate sequences of numbers; then 4 explains how these things are put together to form the method. \nHonestly, I was lost until I read the end of page-5 (of a 9-page paper!) since I did not know why I had to read section 3 and I didn't know how 3.1 and 3.2 are connected. \nMaybe a better way is top-down: give a high-level overview of the method first and then dive into each phase/component for their details. If the authors do not want to change the structure dramatically, maybe at least add a paragraph explaining the overall picture of the method at the beginning. \n\nI would like the authors to give a precise clarification whenever they mention a term that has multiple meanings.\nMaybe I am wrong but \"multimodality\" can mean \"the distribution has multiple modes\" or \"the data has multiple forms\". In this paper, it seems that the authors have used it for both cases (3.1 and 5.5) but that made me confused. \n\nFor limitation of the method, it seems that the proposed method requires something that modern blackbox LLMs do not like to offer: i.e., tokenization and log-likelihood computation. This reliance will restrict the impact of this work: this is fine, since I think the method is already very good and it is a reasonable limitation. But I would appreciate it if the paper has an aggregated section/paragraph discussing this limitation explicitly, instead of mentioning it here and there scattered across the paper.\n\nQUESTIONS:\nPlease see my questions and suggestions in the Weakness.\n\nLIMITATIONS:\nPlease see limitations in the Weakness.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the usage of large language models such as GPT-3 for zero-shot time series forecasting task. To construct the LLMs input sequence in line with how they are pre-trained, it proposes to encode the continuous time series as a string of numerical digits with spaces added between single digits. As a result, time-series forecasting can be transformed into a next-token prediction task suitable for LLMs. Results on standard univariate time-series datasets show that LLMs can perform comparably to time-series models such as ARIMA, etc. in a zero-shot fashion i.e., without any finetuning.\n\nSTRENGTHS:\n1. The paper is well-written and most of the sections are easy to comprehend.\n\n2. The studied problem is straightforward and interesting considering the capabilities exhibited by large language models.\n\nWEAKNESSES:\n1. Comparison to other recent transformer-based time-series forecasting methods (listed in the references below, but not limited to) is missing.\n\n2. The paper considers simple univariate time-series forecasting datasets, it would be helpful to consider more complex multivariate time-series datasets and how they should be encoded.\n\n3. In Appendix C, details about how other existing time-series forecasting models/methods perform compared to GPT-3 are missing on the time-series data that was recorded after GPT-3's training data cutoff date.\n\nReferences:\n\n[1] Informer: Beyond efficient transformer for long sequence time-series forecasting, Zhou et al., 2021\n\n[2] Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting, Wu et al., 2021\n\n[3] FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting, Zhou et al., 2022\n\nQUESTIONS:\nMinor: \n\n1. On page 4, I believe the figure references should be attributed to Figure 2 instead of Figure 3.\n\nLIMITATIONS:\nI could not find a section addressing the limitations and negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a way to use autoregressive large language models to solve time series problems. Several tricks make the time series inputs amenable to LLMs (e.g., separating digits by a space to circumvent tokenization problems). GPT-3 outperforms existing methods on several time series benchmarks. LLMs can also perform imputation on missing data in the time series.\n\nSTRENGTHS:\nThe paper considers the interesting setting of applying pre-trained LLMs to \"classical\" (i.e., 1-D) time series problems. The tricks introduced to format the time series as input to an LLM are interesting in their own right, though I am not fully sure if they are novel or not. The authors also conduct thorough ablations on these tricks to see how they might affect the performance of GPT-3 on these types of problems.\n\nWEAKNESSES:\n**Costs of using LLMs**\n\nOne crucial discussion missing from this paper is the inference cost. If using GPT-3 or GPT-4, then this is a literal payment. Even for open-sourced LLMs, I would imagine that the inference cost is much higher than that of other time series models. Similarly, using LLMs for imputation is much more expensive than standard methods (e.g., interpolation). The authors should acknowledge this weakness clearly.\n\n**No comparison to transformer time series models**\n\nThe authors ought to include comparisons to other transformer time series models, such as the Informer and Autoformer, which were clear leaps forward in the field. It's hard to tell if the gains observed in this paper are from the massive amounts of pre-training data (as suggested by Section 3.2?) or from the self-attention mechanism, which does seem well-adapted to time series. Please also compare the size of GPT-3 to those models. This is my main reason to not give it an accept rating.\n\n**Writing**: Please define MAE, CRPS, etc before using them in the text. The colors used for \"Ground Truth\" and \"GPT-3\" are too close to each other in Figure 3, so it is hard to read. The authors should include a discussion of how positional embeddings make it easier for an LLM to interpret the input data and operate across the time domain.\n\nQUESTIONS:\n**I don't understand the argument in Section 3.2: LLMs extrapolate patterns in numerical sequences.**\n\nThe discussion in Section 3.2 is confusing to me. The authors suggest that LLMs can handle sequences with linear growth, exponential growth, etc. What's confusing is that these are all fixed functions that have no dependence on the previously occurring values. Something that may be more interesting is e.g. the Fibonacci sequence, where the next value is a function of past values. I don't really get the point of Figure 3 + Table 1 because these are just simple functions, and if I wanted to fit them, I would directly fit them using the appropriate function class (i.e., not huge, overparametrized neural networks).\n\nI also don't really get why it is a given that LLMs can do \"simple arithmetic operations\". Although these are simple to us, they may be complicated functions to learn in the embedding space, and they depend on the tokenization used. Note that the cited paper [17] is  focused on low-complexity sequences in the sense of Kolmogorov complexity, which doesn't necessarily correspond to simple arithmetic operations.\n\n**Can the authors motivate their dataset choices?**\n\nThe transformer-related time series models don't use the DARTS dataset.\n\nLIMITATIONS:\n**Not enough LLMs**\n\nThe writing indicates that the authors are trying to make the broader claim that all LLMs (that can perform in-context learning) can serve as time series models. To actually substantiate this claim, they need to test other LLMs besides GPT-3 and GPT-4.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe papers presents an analysis of larger language models for the task of time series prediction. The paper claims that these large language models being very generalizable are able to work very efficiently with numbers in these tasks since each digit prediction is a conditional over the previously predicted digits and hence acts in a way similar to hierarchical softmax to predict the final output.\n\nSTRENGTHS:\nEven though a lot of the work presented here is known to be useful, less is known about how these models work and performing a closer analysis with simplified datasets and well defined examples gives a much better understanding about what works and what does not and helps figure out areas of improvement.\n\nFor example the approach to split numbers by digits, previous works have achieved similar results by removing multi-digit entries from the vocabulary of the model which eventually has the same impact as this paper. But this paper presents intuition around how this might help the model in predicting one digit at a time and be more performant using the analogy to heirarchical softmax.\n\nThe well defined set of experiments show that this approach works very well to improve the ability of LLMs to work with numbers and the analysis on time series datasets confirms the same.\n\nWEAKNESSES:\nThere are already papers showing that:\n1. Mathematical ability of LLMs improves significantly if we process digit by digit (approached generally by removing multi-digit entries from the vocab).\n2. Application of LLMs on time series datasets which generalize better then regular approaches has also been shown.\n\nSee \"PROMPTCAST: A NEW PROMPT-BASED LEARNING PARADIGM FOR TIME SERIES FORECASTING\"\n\nSo there isn't much novelty provided by this paper.\n\nQUESTIONS:\nSome typos\n- line 59: \"from from\" -> \"from\"\n- line 63-64: \"without any major changes by directly numbers as strings of digits.\" --> needs to be reworded\n\nLIMITATIONS:\n- Since we depend on the LM model to perform the forecasting, it is much harder to understand why a particular forecasting was chosen. There are always multiple ways to interpret a series for completion and relying that the LM with follow Occum razor is dangerous. Only the other hand, traditional time series forecasting methods are much more interpretable.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nBriefly summarize the paper and its contributions. This is not the place to critique the paper; the authors should generally agree with a well-written summary.\nThis paper proposes that LLMs like GPT-3 can be employed as time series forecasters due to the similarities between text and time series such as periodicity. The authors show how LLM can be used to model multimodal distributions through a hierarchical discretization of the output space and uniform distribution in each discrete bucket, enabling training of LLM with negative log-likelihood. The hierarchical discretization also allows the LLMs to circumvent the requirement of a fixed output range with binning values and a large number of bins for an expressive output space.\nExperiments show that GPT-3 fitted with the proposed modifications can forecast time series (e.g. linear, periodic, exponential) better than baselines such as TCN and ARIMA. The authors also explore optimal ways to encode time series data in LLMs and found that using a base of 10 and use of space to individually tokenize digits in numbers result in better performance. When benchmarked against existing models, the proposed approach perform competitively with top baselines in the DARTS and Monash datasets. Further experiments were conducted to show that LLMs as time series forecasters are sample efficient, have performances scaled with model size and handles missing data well.\n\nSTRENGTHS:\nThe approach to using LLM as a time series forecaster is innovation and the authors found ways to elegantly unlock this capability without the need for extensive effort (e.g. retraining or finetuning).\nThis finding will be interesting and relevant to a wide audience in this community.\nExperiments were extensively conducted to show optimal approaches for LLMs’ time series forecasting ability and other aspects such as sample efficiency etc.\n\nWEAKNESSES:\nLimitations of the proposed approach were not discussed in the paper\n\nQUESTIONS:\na) The experiments seem to be mostly in a zero-shot setting, have the authors studied the few-shot setting where few-shot examples are in the prompt and how would that affect forecasting performance?\n\nb) What prompts, if any, are given to the LLM, other than the initial time series data and what effects would the choice of prompts have on its performance?\n\nLIMITATIONS:\nNo, a separate section addressing the approach's limitations and potential societal impact would be recommended.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper discusses a novel approach to time series forecasting using large language models (LLMs) like GPT-3. The authors encode time series as a string of numerical digits and view time series forecasting as next-token prediction in text. They find that LLMs can extrapolate time series at a level comparable or exceeding the performance of several classic time series models. The authors argue that LLMs excel at extrapolating deterministic numerical sequences because of biases for simple explanations introduced by their text pretraining.  The authors suggest that this approach has potential and enables new capabilities, such as integrating non-numerical text into the input or output. \n\nOverall, the paper studies an interesting problem setting that directly utilizes the in-context learning ability of Transformers to conduct zero-shot forecasting problems. In particular, the pre-trained transformer model (e.g., LLMs) can give promising performance. However, the analysis in the paper is not comprehensive enough. First of all, the paper fails to answer why pre-trained LLM can yield better performance. The arguments in the paper either focus on RNN (e.g., L121) or directly assume LLMs can perform extrapolation on time series sequences (e.g., L144 and L154). Based on the current presentation, it looks like the authors design a specific feature engineering method (digits representation) that can properly utilize the LLMs' extrapolation or in-context learning ability. If it is the case, the most important research question, why pretrained LLMs can conduct arbitrary time series' extrapolation better than other models is not answered or well discussed. From my perspective, if the major innovation of the paper is a feature engineering method, the current numerical performance would not be strong enough to lead to a publication in the top machine learning conferences like Neurips.\n\n\n\nMoreover, the experimental settings in the paper contain learnable parameters that are tuned based on the forecasting error. From my perspective, I would not treat it as zero-shot learning setting, since parameters can be viewed as trained. The authors also don't provide codes to help reviewers verify the results of experiments.\n\nIn summary, this paper studies an interesting problem that utilizing pretrained LLMs to help time series forecasting. However, based on the current presentation, the analysis and the numerical performance are not strong enough.\n\nSTRENGTHS:\nThis paper studies an interesting problem that utilizing pretrained LLMs to help time series forecasting\n\nWEAKNESSES:\nPlease see my comments in Summary section.\n\nQUESTIONS:\nPlease see my comments in Summary section.\n\nLIMITATIONS:\nThe authors don't directly discuss the limitations of the proposed methods in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-28"}