Title: Are Language Models Actually Useful for Time Series Forecasting?
Venue: NeurIPS 2024
Accepted: yes
Keywords: Time Series

Language Models

Time Series Forecasting
Abstract: Large language models (LLMs) are being applied to time series forecasting. But are language models actually useful for time series? In a series of ablation studies on three recent and popular LLM-based time series forecasting methods, we find that removing the LLM component or replacing it with a basic attention layer does not degrade forecasting performance---in most cases, the results even improve! We also find that despite their significant computational cost, pretrained LLMs do no better than models trained from scratch, do not represent the sequential dependencies in time series, and do not assist in few-shot settings. Additionally, we explore time series encoders and find that patching and attention structures perform similarly to LLM-based forecasters. All resources needed to reproduce our work are available: https://github.com/BennyTMT/LLMsForTimeSeries.

Are Language Models Actually Useful for Time Series Forecasting?

Abstract

Large language models (LLMs) are being applied to time series forecasting. But are language models actually useful for time series? In a series of ablation studies on three recent and popular LLM-based time series forecasting methods, we find that removing the LLM component or replacing it with a basic attention layer does not degrade forecasting performance-in most cases, the results even improve! We also find that despite their significant computational cost, pretrained LLMs do no better than models trained from scratch, do not represent the sequential dependencies in time series, and do not assist in few-shot settings. Additionally, we explore time series encoders and find that patching and attention structures perform similarly to LLM-based forecasters. 1

1 Introduction

Time series analysis is a critical problem across many domains, including disease propagation forecasting [8], retail sales analysis [3], healthcare [26, 17] and finance [31]. A great deal of recent work in time series analysis (constituting repositories with more than 1200 total stars on GitHub) has focused on adapting pretrained large language models (LLMs) to classify, forecast, and detect anomalies in time series [15, 50, 22, 4, 5, 32, 14, 44, 16]. These papers posit that language models, being advanced models for sequential dependencies in text, may generalize to the sequential dependencies in time series data. This hypothesis is unsurprising given language models are now pervasive in machine learning research. However, direct connections between language modeling and time series forecasting remain largely undefined. So to what extent is language modeling really beneficial for traditional time series tasks?

Our claim is simple but profound: popular LLM-based time series forecasters perform the same or worse than basic LLM-free ablations, yet require orders of magnitude more compute . Derived from extensive ablations, this reveals a worrying trend in contemporary time series forecasting literature. Our goal is not to imply that language models will never be useful for time series. In fact, recent works point to many exciting and promising ways that language and time series interact, like time series reasoning [25, 7, 45, 42, 37], social understanding [6] and financial reasoning [36, 20]. Rather, we aim to highlight surprising findings that existing methods do very little to use the innate reasoning power of pretrained language models on established time series tasks.

We substantiate our claim by performing three ablations of three popular and recent LLM-based forecasting methods [50, 15, 22] using eight standard benchmark datasets from reference methods

and another five datasets from MONASH [13]. First, we successfully reproduce results from the original publications. Then, we show that replacing language models with simple attention layers, basic transformer blocks, randomly-initialized language models, and even removing the language model entirely , yields comparable or better performance. The same performance was observed on another five datasets that were not studied by the reference methods.

Next, we compare the training and inference speed of these methods against their ablations, showing that these simpler methods reduce training and inference time by up to three orders of magnitude while maintaining comparable performance. Then, to investigate the source of LLM forecaster's performance, we further explore time series encoders. We find that a simple linear model with an encoder composed of patching and attention can achieve forecasting performance similar to that of LLMs. Next, we test whether the sequence modeling capabilities of LLMs transfer to time series by shuffling input time series and find no appreciable change in performance. Finally, we show that LLMs do not even help forecasting in few-shot settings with 10% of the training data. We discuss the implications of our findings and suggest that time series methods that use large language models are better left to multimodal applications [4, 12, 38] that require textual reasoning.

The key contributions we make in this paper are as follows:

We propose three straightforward ablation methods for methods that pass time series into LLMs for forecasting. We then ablate three top-tier methods on thirteen standard datasets and find that LLMs fail to convincingly improve time series forecasting. However, they significantly increase computational costs in both training and inference.

We study the impact of an LLM's pretraining by re-initializing their weights prior to forecasting. We find that this has no impact on forecasting performance. Additionally, in shuffling input time series, we find no evidence the LLMs successfully transfer sequence modeling abilities from text to time series and no indication they help in few-shot settings.

We find a very simple model, with patching and attention as encoder, can achieve performance similar to LLMs. This suggests a massive gap between the benefits LLMs pose and the time series forecasting problem, despite a rapid rush to adopt LLMs.

2 Related Work

Here, we summarize the key works relevant to LLM-based time series models. They can be broadly classified into three sections: (i) time series forecasting using LLMs; (ii) encoders in LLM time series models; and (iii) smaller and efficient neural models for time-series.

Time Series Forecasting Using LLMs. Recently, with the development of Large Language Models (LLMs) [10, 29, 34] and their demonstrated multi-modal capabilities, more researchers have successfully applied LLMs to time series forecasting tasks [14, 16, 5, 4]. Chang et al., [5] used finetuning the transformer module and positional encoding in GPT-2 to align pre-trained LLMs with time series data for forecasting tasks. Zhou et al. [50] proposed a similar finetuning method, named 'OneFitsAll', for time series forecasting with GPT-2. Additionally, Jin et al. [15] introduced a reprogramming method to align LLM's Word Embedding with time series embeddings, showing good representation of time series data on LLaMA [34]. Similarly, CALF [22] and TEST [32] adapted word embeddings to enable LLMs to forecast time series data effectively. In addition to time-series forecasting models, Liu et al. [23] show that these models can be extended to classifying health-time series, such as heart-rate and daily-footsteps. These models have also been shown to outperform supervised neural models in few-shot settings.

Encoders in LLM Time Series Models. In order for an LLM to learn from text it must first be discretized and encoded as word tokens which are 1 × d vectors [10, 29, 34]. Similarly, LLM-based methods for time series learn discrete time series tokens. One method is to segment the time series into overlapping patches, which effectively shortens the time series while retaining its features [15, 50, 5, 4, 28, 27, 11]. Other methods decompose time series into trend, seasonal components, and residual components [4, 28]. Lastly, Liu et al. [22] feed the multivariate time series using a Transformer to enable different channels to learn the dynamics of other channels. These embedding procedures are followed by a linear neural network layer that projects the time series encoding to the same dimensions used by the pre-trained LLM.

Small and Efficient Neural Forecasters. In addition to LLMs, there has been a large body of research on smaller yet efficient frameworks that outperform their bulky counterparts in time series forecasting [19, 47, 33, 24, 2]. For example, Zeng et al. [46] present DLinear, an incredibly simple model that combines decomposition techniques and achieves better forecasting performance than state-of-the-art transformer-based time series architectures at the time, such as Informer [48], FEDformer [49], and Autoformer [40]. Furthermore, Xu et al. [43] introduces a lightweight model with only 10k parameters, which captures both amplitude and phase information in the time-series to outperform transformer-based models.

3 Experimental Setup

We use three state-of-the-art methods for time series forecasting and propose three ablation methods for LLMs: (i) ' w/o LLM '; (ii) ' LLM2Attn '; (iii) and ' LLM2Trsf '. To evaluate the effectiveness of LLMs in time series forecasting, we test these methods on eight standard datasets.

3.1 Reference Methods for Language Models and Time Series

We experiment with three recent methods for time series forecasting using LLMs. All models were published between December 2023 and May 2024 and are popular, with their GitHub repositories collectively amassing 1,245 stars. These methods are summarized in Table 2, and use either GPT2 [29] or LLaMA [34] as base models, with different alignment and fine-tuning strategies.

OneFitsAll [50] : OneFitsAll, sometimes called GPT4TS, applies instance norm and patching to the input time series and then feeds it into a linear layer to obtain a input representation for the language model. The multi-head attention and feed forward layers of the language model are frozen while the positional embeddings and layer norm are optimized during training. A final linear layer is used to transform the language model's final hidden states into a prediction.

Time-LLM [15] : In Time-LLM the input time series is tokenized via patching and aligned with a low-dimensional representation of word embeddings using multi-head attention. The outputs of this alignment, combined with the embeddings of descriptive statistical features, are passed to a frozen pre-trained language model. The output representations of the language model are then flattened and passed through a linear layer to obtain a forecast.

CALF [22] : CALF embeds the input time series by treating each channel as a token. One half of the architecture is a 'textual branch' which uses cross attention to align the time series representation with a low dimensional representation of the language model's word embeddings. This representation is then passed through a pretrained, frozen language model to obtain a 'textual prediction'. Simultaneously, a 'temporal' branch learns a low-rank adapter for a pretrained language model based on the input time series to produce a 'temporal prediction' which is used for inference. The model includes additional loss terms that enforce similarity between these representations.

Reproducibility Note. While experimenting with each model, we tried to replicate the conditions of their original papers. We used the original hyper-parameters, runtime environments, and code, including model architectures, training loops, and data-loaders. To ensure a fair comparison, we have included error metrics from the original papers alongside our results wherever possible.

Instance Norm

Pre-trained LLM

Input Time Series

Encoding

Projection

🔥

Alignment

🔥

/

❄

🔥

🔥

🔥

Trainable

Aligning

🔥

Attention

Instance Norm

Input Time Series

Encoding

Projection

🔥

🔥

Instance Norm

Input Time Series

Encoding

Projection

🔥

🔥

🔥

Transformer

Instance Norm

Input Time Series

Encoding

Projection

🔥

🔥

(a) w/ LLM

(b) w/o LLM

(c) LLM2Attn

(d) LLM2Trsf

❄

Freezing

3.2 Proposed Ablations

To isolate the influence of the LLM in an LLM-based forecaster, we propose three ablations: removing the LLM component or replacing it with a simple block. Specifically, for each of the three methods we make the following three modifications:

w/o LLM (Figure 1 (b)). We remove the language model entirely, instead passing the input tokens directly to the reference method's final layer.

LLM2Attn (Figure 1 (c)). We replace the language model with a single randomly-initialized multi-head attention layer.

LLM2Trsf (Figure 1 (d)). We replace the language model with a single randomly-initialized transformer block.

In the above ablations, we keep left parts of the forecasters unchanged (trainable). For example, as shown in Figure 1 (a), after removing the LLM, the input encodings are passed directly to the output projection. Alternatively, as shown in Figure 1 (b) or (c), after replacing the LLM with attention or a transformer, they are trained along with the remaining structure of the original method.

3.3 Datasets and Evaluation Metrics

Benchmark Datasets. We evaluate on the following real-world datasets: (1) ETT [21]: encompasses seven factors related to electricity transformers across four subsets: ETTh1 and ETTh2, which have hourly recordings, and ETTm1 and ETTm2, which have recordings every 15 minutes; (2) Illness [40]: includes the weekly recorded influenza illness among patients from the Centers for Disease Control, which describes the ratio of patients seen with influenza-like illness to the total number of patients; (3) Weather [40]: local climate data from 1,600 U.S. locations, between 2010 and 2013, and each data point consists of 11 climate features; (4) Traffic [40]: is an hourly dataset from California transportation department, and consists of road occupancy rates measured on San Francisco Bay area freeways; (5) Electricity [35]: contains the hourly electricity consumption of 321 customers from 2012 to 2014. The train-val-test split for ETT datasets is 60%-20%-20%, and for Illness, Weather, and Electricity datasets is 70%-10%-20% respectively. The statistics for all datasets is given in Table 1. We highlight that these datasets, with the same splits and size, have been extensively used to evaluate time-series forecasting ability of LLM-based and other neural models for time-series data [48, 50, 4, 15, 5, 46, 40, 49]. (6) Exchange Rate [18]: collected between 1990 and 2016, it contains daily exchange rates for the currencies of eight countries (Australia, British, Canada, Switzerland, China, Japan, New Zealand and Singapore). (7) Covid Deaths [13]: contains daily statistics of COVID-19 deaths in 266 countries and states between January and August 2020. (8) Taxi (30 min) [1]: contains taxi rides from 1,214 locations in New York City between January 2015 and January 2016. The data is collected every 30 minutes, with an average of 1,478 samples. (9) NN5 (Daily) [13]: contains daily cash withdrawal data from 111 ATMs in the UK, with each ATM having

0.420

0.450

MAE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

ETTh1-OneFitsAll

0.310

0.322

MAE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

ETTm2-OneFitsAll

0.247

0.260

MAE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

ECL-OneFitsAll

0.426

0.429

MAE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

ETTh1-CALF

0.319

0.322

MAE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

ETTm2-CALF

0.261

0.264

MAE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

ECL-CALF

0.420

0.435

MAE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

ETTh1-TimeLLM

0.310

0.325

MAE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

ETTm2-TimeLLM

0.256

0.268

MAE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

ECL-TimeLLM

791 data points. (10) FRED-MD [13]: contains 107 monthly macroeconomic indices released by the Federal Reserve Bank since 01/01/1959. It was extracted from the FRED-MD database.

Evaluation Metrics and Setup. We report the results in terms of mean absolute error (MAE) and mean squared error (MSE) between predicted and true values of the time-series. Mathematically, given a test-set with D elements, MAE = 1 |D| ∑ t i ∈D [ | c i -̂ c i | ] and MSE = 1 |D| ∑ t i ∈D ( c i -̂ c i ) 2 , where c i and ̂ c i denote the true value and predicted value at the i -th index of the time-series respectively.

4 Results

In this section, we provide the details of our comprehensive evaluation of all baseline LLM models for time-series forecasting. Specifically, we ask the following research questions. (RQ1) Do pretrained language models contribute to forecasting performance? (RQ2) Are LLM-based methods worth the computational cost? (RQ3) Does language model pretraining help performance on forecasting tasks? (RQ4) Do LLMs represent sequential dependencies in time series? (RQ5) Do LLMs help with few-shot learning? (RQ6) Where does the performance come from?

4.1 Do pretrained language models contribute to forecasting performance? (RQ1)

Our results show that pretrained LLMs are not useful for time series forecasting tasks yet . Overall, as shown in Table 3, across 13 datasets and two metrics, ablations out perform Time-LLM methods in 26 / 26 cases, CALF in 22 / 26 cases, and OneFitsAll in 19 / 26 cases. We averaged results over different predicting lengths, as in [50, 15, 22]. Across all prediction lengths (thirteen datasets and four prediction lengths) ablations outperformed Time-LLM, CALF, and OneFitsAll in 35 / 40 , 31 / 40 , and 29 / 40 cases as measured by MAE, respectively. To ensure a fair comparison, we also report results from each method's original paper alongside our replication. For specific results refer to Appendix E.1. To better evaluate the effectiveness of LLMs and ablation methods, we include 95% bootstrapped confidence intervals for each task. In tasks where LLMs performed better, such as OneFitsAll with ETTh1, shown in Figure 2, there is still substantial overlap in the confidence intervals with the ablation method ' w/o LLM ' in MAE. Other datasets results and MSE metrics are shown in Figure 5 and Figure 6 in the Appendix. To summarize, our results on the evaluation above, it is hard to conclude that LLMs are effective in time series forecasting.

4.2 Are LLM-based methods worth the computational cost? (RQ2)

In the previous section, we showed that LLMs do not meaningfully improve performance on time series forecasting tasks. Here, we evaluate the computational intensity of these methods with their

Time-LLM [15]

Model

→

Time-LLM

w/o LLM

LLM2Attn

LLM2Trsf

From Original Paper

Dataset

↓

MAE

MSE

MAE

MSE

MAE

MSE

MAE

MSE

MAE

MSE

ETTh1

0.432

0.417

0.419

0.405

0.437

0.422

0.439

0.429

0.423

0.408

ETTh2

0.396

0.360

0.383

0.345

0.389

0.353

0.394

0.359

0.383

0.334

ETTm1

0.377

0.356

0.371

0.350

0.376

0.356

0.377

0.359

0.371

0.329

ETTm2

0.315

0.260

0.307

0.252

0.314

0.259

0.310

0.253

0.329

0.250

Illness

0.894

2.017

0.924

1.956

0.849

1.789

0.837

1.795

0.801

1.435

Weather

0.270

0.243

0.272

0.243

0.254

0.224

0.254

0.226

0.257

0.225

Traffic

0.281

0.421

0.295

0.428

0.276

0.416

0.275

0.416

0.263

0.387

Electricity

0.259

0.164

0.269

0.171

0.260

0.167

0.254

0.161

0.252

0.158

Exchange Rate

0.448

0.422

0.413

0.384

0.432

0.403

0.442

0.422

-

-

0.089

0.189

0.080

0.198

0.058

0.086

0.054

0.079

-

-

Taxi (30 Min)

0.277

0.163

0.286

0.176

0.269

0.157

0.255

0.141

-

-

NN5 (Daily)

0.432

0.402

0.425

0.379

0.411

0.364

0.401

0.347

-

-

FRED-MD

0.0004

5e-7

0.0002

3e-7

0.0046

2.53e-5

0.0008

2.6e-6

-

-

# Wins

0

12

2

12

-

#Parameters

6651.82M

0.55M

0.55M

0.66M

-

CALF [22]

Model

→

CALF

w/o LLM

LLM2Attn

LLM2Trsf

From Original Paper

Dataset

↓

MAE

MSE

MAE

MSE

MAE

MSE

MAE

MSE

MAE

MSE

ETTh1

0.431

0.431

0.428

0.436

0.430

0.428

0.430

0.430

0.428

0.432

ETTh2

0.383

0.351

0.383

0.352

0.382

0.349

0.383

0.350

0.382

0.349

ETTm1

0.391

0.396

0.390

0.397

0.390

0.396

0.390

0.394

0.390

0.395

ETTm2

0.323

0.283

0.322

0.282

0.321

0.281

0.320

0.281

0.321

0.281

Illness

0.869

1.699

0.861

1.639

0.892

1.748

0.860

1.630

-

-

Weather

0.273

0.251

0.277

0.257

0.279

0.258

0.277

0.255

0.274

0.250

Traffic

0.284

0.443

0.278

0.439

0.275

0.430

0.271

0.426

0.281

0.439

Electricity

0.266

0.175

0.262

0.174

0.264

0.175

0.261

0.172

0.265

0.175

Exchange Rate

0.417

0.388

0.409

0.367

0.417

0.389

0.409

0.367

-

-

0.084

0.163

0.066

0.115

0.131

0.431

0.066

0.106

-

-

Taxi (30 Min)

0.258

0.142

0.264

0.147

0.267

0.150

0.267

0.150

-

-

NN5 (Daily)

0.403

0.362

0.386

0.336

0.463

0.445

0.415

0.381

-

-

FRED-MD

0.0012

2.9e-6

0.0011

2.7e-6

0.0015

4.9e-6

0.0017

4.5e-6

-

-

# Wins

4

7

4

11

-

#Parameters

180.25M

8.17M

10.5M

13.68M

-

OneFitsAll [50]

Model

→

OneFitsAll

w/o LLM

LLM2Attn

LLM2Trsf

From Original Paper

Dataset

↓

MAE

MSE

MAE

MSE

MAE

MSE

MAE

MSE

MAE

MSE

ETTh1

0.420

0.417

0.422

0.417

0.452

0.465

0.474

0.525

0.426

0.427

ETTh2

0.388

0.353

0.389

0.356

0.402

0.375

0.397

0.367

0.394

0.354

ETTm1

0.377

0.362

0.374

0.358

0.379

0.369

0.379

0.369

0.383

0.351

ETTm2

0.310

0.253

0.309

0.255

0.312

0.256

0.310

0.254

0.326

0.266

Illness

0.852

1.871

0.924

1.960

0.829

1.763

0.850

1.830

0.903

1.925

Weather

0.254

0.226

0.272

0.245

0.256

0.226

0.256

0.228

0.270

0.236

Traffic

0.273

0.420

0.273

0.439

0.266

0.415

0.256

0.409

0.294

0.414

Electricity

0.262

0.169

0.254

0.165

0.250

0.162

0.245

0.157

0.263

0.166

Exchange Rate

0.378

0.357

0.361

0.323

0.376

0.350

0.393

0.387

-

-

0.057

0.075

0.050

0.073

0.058

0.103

0.078

0.162

-

-

Taxi (30 Min)

0.252

0.138

0.259

0.143

0.259

0.145

0.257

0.140

-

-

NN5 (Daily)

0.438

0.438

0.422

0.385

0.423

0.390

0.420

0.386

-

-

FRED-MD

0.0006

1.2e-6

0.0002

4e-7

0.0006

1.5e-6

0.0012

2.4e-6

-

-

# Wins

7

11

3

5

-

#Parameters

91.36M

9.38M

10.71M

13.54M

-

nominal performance in mind. The language models in our reference methods use hundreds of millions and sometimes billions of parameters to perform time series forecasting. Even when the parameters of the language models are frozen they still contribute to substantial overhead during training and inference. For instance, Time-LLM has 6642 M parameters and takes 3003 minutes to train on the Weather dataset whereas ablation methods have only 0.245 M parameters and take 2.17 minutes on average. Information about training other methods on ETTh1 and Weather datasets are shown in Table 4. In the case of inference time, we divide by the maximum batch size to give an estimate of inference time per example. Time-LLM, OneFitsAll, and CALF take, on average, 28.2 ,

0

100

200

0.306

0.312

0.318

MAE

Time-LLM-ETTm2

1.26

1.32

1.38

0.305

0.310

0.315

OneFitsAll-ETTm2

1.95

2.10

0.320

0.324

CALF-ETTm2

1

2

0.280

0.290

0.300

MAE

Time-LLM-Traffic

0.30

0.45

0.60

0.260

0.270

OneFitsAll-Traffic

30.00

60.00

90.00

0.272

0.280

0.288

CALF-Traffic

4

8

Inference Time (ms)

0.256

0.264

0.272

MAE

Time-LLM-ECL

0.30

0.45

0.60

Inference Time (ms)

0.250

0.260

OneFitsAll-ECL

12.00

18.00

Inference Time (ms)

0.260

0.265

0.270

CALF-ECL

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

2.3 , and 1.2 times longer than the modified models. Examples can be seen in Figure 3, where the green marks (ablation methods) are typically below the red one (LLM) and are positioned towards the left of the axis, indicating a lower computational costs and better forecasting performance. Other datasets and MSE metric refer to Figure 7 and Figure 8 in Appendix. In conclusion, the computational intensity of LLMs in time series forecasting tasks does not result in a corresponding performance improvement.

4.3 Does language model pretraining help performance on forecasting tasks? (RQ3)

Our evaluation in this section indicates that pretraining with language datasets is unnecessary for time series forecasting. To test whether the knowledge learned during pretraining meaningfully improves forecasting performance we experimented with different combinations of pretraining and finetuning CALF's [22] language model on time series.

Pretrain + Finetune (Pre+FT) . This is the original method, wherein a pretrained language model is finetuned on time series data. In the case of CALF, the base language model is frozen and low rank adapters (LoRA) are learned.

Random Initialization + Finetune (woPre+FT) . Does the textual knowledge from pretraining aid time series forecasting? In this method we randomly initialize the weights of the language model (thereby erasing any effect of pretraining) and train the LLM from scratch.

Pretrain + No Finetuning (Pre+woFT) . How much does finetuning on time series improve prediction performance? For this baseline we again leave the language model frozen and forgo

learning LoRAs. Results from this model are therefore indicative of the base language model's performance without additional guidance on processing time series.

Random Initialization + No Finetuning (woPre+woFT) . This baseline is effectively a random projection from the input time series to a forecasting prediction and serves as a baseline comparison with the other methods.

Overall, as shown in Table 5, across 8 datasets using MAE and MSE metrics, the "Pretraining + Finetune" method performed the best 3 times, while "Random Initialization + Finetune" achieved this 8 times. This indicates that language knowledge offers very limited help for forecasting. However, "Pretrain + No Finetuning" and the baseline "Random Initialization + No Finetuning" performed the best 5 times and 0 times, respectively, suggesting that Language knowledge does not contribute meaningfully during the finetuning process. Detailed results refer to Table 20 in Appendix.

In summary, textual knowledge from pretraining provides very limited aids for time series forecasting.

4.4 Do LLMs represent sequential dependencies in time series? (RQ4)

Most time series forecasting methods that use LLMs finetune the positional encoding to help understand the position of time steps in the sequence [4, 50, 22, 5, 32]. We would expect a time series model with good positional representations to show a significant drop in predictive performance when the input is shuffled [46]. We applied three types of shuffling to the time series: shuffling the entire sequence randomly ("sf-all"), shuffling only the first half of the sequence ("sf-half"), and swapping the first and second halves of the sequence ("ex-half"). As shown in Table 6, LLM-based methods were no more vulnerable to input shuffling than their ablations. This implies that LLMs do not have unique capabilities for representing sequential dependencies in time series.

4.5 Do LLMs help with few-shot learning in forecasting? (RQ5)

In this section, our evaluation demonstrates that LLMs are still not meaningfully useful in few-shot learning scenarios.

While our results indicate that LLMs are not useful for time series forecasting, it is nonetheless possible that knowledge encoded in pretrained weights could help performance in few-shot settings where data are scarce. To evaluate whether this is the case we trained models and their ablations on 10% of each dataset. Specifically, we evaluated LLaMA in Time-LLM methods. The results for LLaMA, shown in Table 7, compared LLaMA with completely removing the LLM ( w/o LLM ). There was no difference, with each performing better in 8 cases. We conducted similar experiments with CALF, a GPT-2-based method. Our results in Table 8 indicate that our ablations can perform better than LLMs in few-shot scenarios.

4.6 Where does the performance come from? (RQ6)

In this section, we evaluate common encoding techniques used in LLM time series models. We find that combining patching with one-layer attention is a simple and effective choice.

In subsection 3.2, we found that simple ablations of LLM-based methods did not decrease performance. To understand why such simple methods work so well we selected some popular techniques used for encoding in LLM time series tasks, such as patching [50, 15, 5, 32, 22, 11], decomposition [4, 28]. A basic transformer block also can be used to aid in encoding [22].

The specific results, shown in Table 18 in the Appendix, indicate that a structure combining patching and attention, named 'PAttn', performs better than most other encoding methods on small datasets (with time stamps less than 1 million) and is even comparable to LLM methods. Its detailed structure, as shown in Figure 4, involves applying "instance norm" to the time series, followed by patching and projection. Then, one-layer

Patching

(Channel-Independence)

🔥

Projection

🔥

Projection

🔥

Attention

Inp

attention enables feature learning between patches. For larger datasets, such as Traffic (~15 million) and Electricity (~8 million), a model named "LTrsf," using the encoder from CALF [22], performs better. In those methods, finally, time series embedding will be projected with a single linear layer to forecast. Details of other encoders is in Appendix subsection D.3.

Overall, patching plays a crucial role in encoding. Additionally, basic Attention and Transformer blocks also effectively aid in encoding.

5 Conclusion

In this paper we showed that despite the recent popularity of LLMs in time series forecasting they do not appear to meaningfully improve performance. We experimented with simple ablations, showing that they maintain or improve the performance of the LLM-based counterparts while requiring considerably less compute. Once more, our goal is not to suggest that LLMs have no place in time series analysis. To do so would likely prove to be a shortsighted claim. Rather, we suggest that the community should dedicate more focus to the exciting tasks could be unlocked by LLMs at the interface of time series and language such as time series reasoning [25, 7, 45, 37], or social understanding [6].

References

A. F. Ansari, L. Stella, C. Turkmen, X. Zhang, P. Mercado, H. Shen, O. Shchur, S. S. Rangapuram, S. P. Arango, S. Kapoor, J. Zschiegner, D. C. Maddix, H. Wang, M. W. Mahoney, K. Torkkola, A. G. Wilson, M. Bohlke-Schneider, and Y. Wang. Chronos: Learning the language of time series. arXiv preprint arXiv:2403.07815 , 2024.

Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi, et al. Audiolm: a language modeling approach to audio generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 2023.

J.-H. Böse, V. Flunkert, J. Gasthaus, T. Januschowski, D. Lange, D. Salinas, S. Schelter, M. Seeger, and Y. Wang. Probabilistic demand forecasting at scale. VLDB , 2017.

D. Cao, F. Jia, S. O. Arik, T. Pfister, Y. Zheng, W. Ye, and Y. Liu. Tempo: Prompt-based generative pre-trained transformer for time series forecasting. In ICLR , 2024.

C. Chang, W.-C. Peng, and T.-F. Chen. Llm4ts: Two-stage fine-tuning for time-series forecasting with pre-trained llms. arXiv preprint arXiv:2308.08469 , 2023.

J. Cheng and P. Chin. Sociodojo: Building lifelong analytical agents with real-world text and time series. In ICLR , 2024.

W. Chow, L. Gardiner, H. T. Hallgrímsson, M. A. Xu, and S. Y. Ren. Towards time series reasoning with llms. arXiv preprint arXiv:2409.11376 , 2024.

J. Chung, C. Gulcehre, K. Cho, and Y. Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 , 2014.

A. Das, W. Kong, R. Sen, and Y. Zhou. A decoder-only foundation model for time-series forecasting. 2024.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

S. Gao, T. Koker, O. Queen, T. Hartvigsen, T. Tsiligkaridis, and M. Zitnik. UniTS: Building a unified time series model. In NeurIPS , 2024.

R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, and I. Misra. Imagebind: One embedding space to bind them all. In CVPR , 2023.

R. Godahewa, C. Bergmeir, G. I. Webb, R. J. Hyndman, and P. Montero-Manso. Monash time series forecasting archive. In NeurIPS , 2021.

N. Gruver, M. Finzi, S. Qiu, and A. G. Wilson. Large language models are zero-shot time series forecasters. In NeurIPS , 2023.

M. Jin, S. Wang, L. Ma, Z. Chu, J. Y. Zhang, X. Shi, P.-Y. Chen, Y. Liang, Y.-F. Li, S. Pan, et al. Time-llm: Time series forecasting by reprogramming large language models. In ICLR , 2024.

M. Jin, Y. Zhang, W. Chen, K. Zhang, Y. Liang, B. Yang, J. Wang, S. Pan, and Q. Wen. Position paper: What can large language models tell us about time series analysis. In ICML , 2024.

M. Jörke, S. Sapkota, L. Warkenthien, N. Vainio, P. Schmiedmayer, E. Brunskill, and J. Landay. Supporting physical activity behavior change with llm-based conversational agents. arXiv preprint arXiv:2405.06061 , 2024.

G. Lai, W.-C. Chang, Y. Yang, and H. Liu. Modeling long-and short-term temporal patterns with deep neural networks. In SIGIR , 2018.

S. Lee, T. Park, and K. Lee. Learning to embed time series patches independently. In ICLR , 2024.

Y. Li, B. Luo, Q. Wang, N. Chen, X. Liu, and B. He. A reflective llm-based agent to guide zero-shot cryptocurrency trading. In EMNLP , 2024.

Z. Li, S. Qi, Y . Li, and Z. Xu. Revisiting long-term time series forecasting: An investigation on linear mapping. arXiv preprint arXiv:2305.10721 , 2023.

P. Liu, H. Guo, T. Dai, N. Li, J. Bao, X. Ren, Y. Jiang, and S.-T. Xia. Calf: Aligning llms for time series forecasting via cross-modal fine-tuning. arXiv preprint arXiv:2403.07300 , 2024.

X. Liu, D. McDuff, G. Kovacs, I. Galatzer-Levy, J. Sunshine, J. Zhan, M.-Z. Poh, S. Liao, P. Di Achille, and S. Patel. Large language models are few-shot health learners. arXiv preprint arXiv:2305.15525 , 2023.

Y. Liu, T. Hu, H. Zhang, H. Wu, S. Wang, L. Ma, and M. Long. iTransformer: Inverted transformers are effective for time series forecasting. In ICLR , 2024.

M. A. Merrill, M. Tan, V. Gupta, T. Hartvigsen, and T. Althoff. Language models still struggle to zero-shot reason about time series. In EMNLP , 2024.

M. A. Morid, O. R. L. Sheng, and J. Dunbar. Time series prediction using deep learning methods in healthcare. ACM Trans. Manage. Inf. Syst. , 14(1), 2023.

Y. Nie, N. H. Nguyen, P. Sinthong, and J. Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In ICLR , 2023.

Z. Pan, Y. Jiang, S. Garg, A. Schneider, Y. Nevmyvaka, and D. Song. s 2 ip-llm: Semantic space informed prompt learning with llm for time series forecasting. In Forty-first International Conference on Machine Learning , 2024.

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

K. Rasul, A. Ashok, A. R. Williams, A. Khorasani, G. Adamopoulos, R. Bhagwatkar, M. Biloš, H. Ghonia, N. Hassen, A. Schneider, et al. Lag-llama: Towards foundation models for time series forecasting. In R0-FoMo: Robustness of Few-shot and Zero-shot Learning in Large Foundation Models , 2023.

O. B. Sezer, M. U. Gudelek, and A. M. Ozbayoglu. Financial time series forecasting with deep learning: A systematic literature review: 2005-2019. Applied soft computing , 90, 2020.

C. Sun, Y. Li, H. Li, and S. Hong. Test: Text prototype aligned embedding to activate llm's ability for time series. In ICLR , 2024.

S. J. Talukder and G. Gkioxari. Time series modeling at scale: A universal representation across tasks and domains. 2023.

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models (2023). arXiv preprint arXiv:2302.13971 , 2023.

A. Trindade. ElectricityLoadDiagrams20112014. UCI Machine Learning Repository, 2015. DOI: https://doi.org/10.24432/C58C86.

S. Wang, T. Ji, L. Wang, Y. Sun, S.-C. Liu, A. Kumar, and C.-T. Lu. Stocktime: A time series specialized large language model architecture for stock price prediction. arXiv preprint arXiv:2409.08281 , 2024.

X. Wang, M. Feng, J. Qiu, J. Gu, and J. Zhao. From news to forecast: Iterative event reasoning in llm-based time series forecasting. In Neural Information Processing Systems , 2024.

C. Wimmer and N. Rekabsaz. Leveraging vision-language models for granular market change prediction. arXiv preprint arXiv:2301.10166 , 2023.

G. Woo, C. Liu, A. Kumar, C. Xiong, S. Savarese, and D. Sahoo. Unified training of universal time series forecasting transformers. 2024.

H. Wu, J. Xu, J. Wang, and M. Long. Autoformer: Decomposition transformers with autocorrelation for long-term series forecasting. NeurIPS , 2021.

H. Wu, T. Hu, Y. Liu, H. Zhou, J. Wang, and M. Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In International Conference on Learning Representations , 2023.

Z. Xu, Y. Bian, J. Zhong, X. Wen, and Q. Xu. Beyond trend and periodicity: Guiding time series forecasting with textual cues. arXiv preprint arXiv:2405.13522 , 2024.

Z. Xu, A. Zeng, and Q. Xu. Fits: Modeling time series with 10 k parameters. In ICLR , 2024.

H. Xue and F. D. Salim. Promptcast: A new prompt-based learning paradigm for time series forecasting. IEEE Transactions on Knowledge and Data Engineering , 2023.

W. Ye, Y. Zhang, W. Yang, L. Tang, D. Cao, J. Cai, and Y. Liu. Beyond forecasting: Compositional time series reasoning for end-to-end task execution, 2024. URL https: //arxiv.org/abs/2410.04047 .

A. Zeng, M. Chen, L. Zhang, and Q. Xu. Are transformers effective for time series forecasting? In AAAI , 2023.

L. Zhao and Y. Shen. Rethinking channel dependence for multivariate time series forecasting: Learning from leading indicators. In ICLR , 2024.

H. Zhou, S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong, and W. Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI , 2021.

T. Zhou, Z. Ma, Q. Wen, X. Wang, L. Sun, and R. Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In ICML , 2022.

T. Zhou, P. Niu, L. Sun, R. Jin, et al. One fits all: Power general time series analysis by pretrained lm. In NeurIPS , 2023.

Are Language Models Actually Useful for Time Series Forecasting? (Appendix)

A Limitations

Here, we discuss the limitations of our paper.

We evaluate the ability of LLMs using time-series forecasting. However, to get a better picture of how LLMs can work with time-series, this ability should be evaluated across other downstream tasks as well, such as time-series classification and question-answering.

Our evaluation is limited to only time-series datasets, i . e ., sequences with even time-intervals. However, there also exists a large fraction of data in the form of non-uniform series, such as payment records, online purchases, etc. Understanding the ability of LLMs in forecasting non-uniform sequences is also necessary to verify our claim on the usefulness of LLMs for time-series data.

B Broader Societal Impact

One of the major impacts our study will have is on the influx of models that use LLMs for modeling time-series. Our results will help researchers to not simply follow the trend of using LLMs in all applications, but to check their usability in detail. Specifically, these findings will help them determine if the LLM component is necessary and if the computational costs are reasonable for the specific setting. In addition to the research community, our findings on the better performance of smaller and simpler models will help develop scalable models that are easy to understand, interpret, and can be deployed cheaply in real-world applications. While we agree that a majority of our results are experimental and limited to selected datasets, we feel that these results will also help researchers narrow down their search space for better models in time-series forecasting, and not simply neglect the simpler models.

C License

All our contributions will be released under the MIT License.

D Additional Experimental Details

D.1 System Configuration

We train and evaluate each reference method and each architecture modification using the same device. For Time-LLM [15], applying LlaMA-7B [34], we use NVIDIA A100 GPU with 80GB memory. For other methods [50, 22], applying GPT-2 [29], we use NVIDIA RTX A6000 GPU with 48GB memory. Though an analysis of memory footprint is beyond scope of this research, we note that training the baselines in the absence of LLM can be done within smaller GPUs as well.

D.2 Baseline Hyper-Parameter Details

When reproducing the reference methods, we used the original repository's hyper-parameters and model structures. In the ablation study, due to the smaller model parameters, we adjusted the learning rate or increased the batch size in some cases. All other training details remained identical with the reference methods. The specific training details and hyper-parameters are provided in the code and run scripts of the repository 2 . Note that the training process and hyper-parameters for the simple methods can also be accessed via this link.

D.3 Details of Encoder Exploration and Simple Methods

To investigate the source of LLM method performance, we conducted further research on encoders. We used various encoders to encode time series data, followed by a linear layer to project the time

series embeddings to the forecast. The encoder structure combining patching and attention is shown in Figure 4. The key difference between PAttn and PatchTST [27] is the absence of position embedding and the Feed Forward in PatchTST [27], or, more specifically, replacing the Transformer Encoder with a simple single-layer Attention structure. It could be a simple yet effective method to help evaluate the trade-off between cost and performance for newly proposed methods.

In addition, we propose three different neural models (i) 'LTrsf': which performs better on larger datasets, uses CALF 's [22] encoder without cross-modal attention. This could be a potential source of CALF 's performance; (ii) 'D-LTrsf'; and (iii) 'D-PAttn': both of them decompose the time series into three sub-sequences and forecast each using the above two methods respectively, and then linearly combine the results for the final forecast. Across our results in Table 18, we note that even simpler models significantly outperform LLM-based time-series models. In detail, the LLM-based models, all combined we able to appear 33 times as the best and the second-best performer. However, 'PAttn' was outperforming them by appearing 34 times as the best and the second-best performer. In addition, the comparison between 'PAttn', 'LTrsf' and other state-of-the-art methods is shown in Table 19.

E Additional Experiments

Here we present the results of additional experiments that also highlight the ability of LLMs in modeling time-series data.

E.1 Confidence Intervals for Forecasting

Since LLMs and deep learning models in general are probabilistic in nature, their predictions can vary across different runs and different random initializations. Thus, we report the confidence intervals (CIs) for all MAE and MSE predictions made by our baseline models. We report the CIs for MAE prediction by Time-LLM, CALF, OneFitsAll in Tables 9, 11, and 13, for MSE predictions in Tables 10, 12, and 14, respectively. Across all results, we note that the range of variation is quite small, and these intervals do not affect our observations. To illustrate the subtle differences more clearly, we present the visualized results in Figure 5 and Figure 6.

Confidence Intervals for other Datasets. To evaluate the generality of the ablations in the paper, we introduce five additional datasets that have not been studied by the reference methods [50, 15, 22]. The above datasets are used in many time series forecasting studies [30, 9, 39, 1, 46]. The prediction lengths for the 'Exchange Rate' are "96, 192, 336, 720", as in [46, 18]. The prediction lengths for the other four datasets are 30, 48, 56, and 12, respectively, following the settings in Chronos [1]. As shown in Tables 15, 16, and 17, the forecasting performance on the five new datasets, using the three methods [50, 15, 22] we referenced, still demonstrates that language models are unnecessary for forecasting tasks.

E.2 Complete Results

Here we provide the results for all methods and datasets that we were unable to add to the main paper.

Inference Times. All results regarding 'inference time' and forecast performance are shown in Figure 7 and Figure 8 respectively.

Randomized Parameters and Random-Shuffling of Inputs. The results of the randomized parameters are shown in Table 20. The remaining results for shuffled input are shown in Table 21,22,23 and 24.

0.390

0.400

MAE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

ETTh2-OneFitsAll

0.380

0.383

MAE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

ETTh2-CALF

0.380

0.390

MAE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

ETTh2-TimeLLM

0.375

0.380

MAE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

ETTm1-OneFitsAll

0.392

0.400

MAE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

ETTm1-CALF

0.372

0.376

MAE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

ETTm1-TimeLLM

0.840

0.900

MAE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

Illness-OneFitsAll

0.850

0.900

MAE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

Illness-CALF

0.800

0.880

MAE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

Illness-TimeLLM

0.255

0.270

MAE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

Weather-OneFitsAll

0.272

0.276

MAE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

Weather-CALF

0.255

0.270

MAE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

Weather-TimeLLM

0.260

0.280

MAE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

Traffic-OneFitsAll

0.272

0.280

MAE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

Traffic-CALF

0.270

0.285

MAE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

Traffic-TimeLLM

0.420

0.480

MSE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

ETTh1-OneFitsAll

0.424

0.432

MSE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

ETTh1-CALF

0.400

0.420

MSE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

ETTh1-TimeLLM

0.350

0.360

0.370

MSE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

ETTh2-OneFitsAll

0.344

0.348

MSE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

ETTh2-CALF

0.345

0.360

MSE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

ETTh2-TimeLLM

0.360

0.370

MSE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

ETTm1-OneFitsAll

0.400

0.410

MSE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

ETTm1-CALF

0.340

0.360

MSE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

ETTm1-TimeLLM

0.256

0.264

MSE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

ETTm2-OneFitsAll

0.279

0.282

MSE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

ETTm2-CALF

0.252

0.258

MSE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

ETTm2-TimeLLM

0.160

0.168

MSE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

ECL-OneFitsAll

0.171

0.174

MSE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

ECL-CALF

0.160

0.168

MSE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

ECL-TimeLLM

1.800

2.000

MSE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

Illness-OneFitsAll

1.600

1.800

MSE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

Illness-CALF

1.600

2.000

MSE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

Illness-TimeLLM

0.225

0.240

MSE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

Weather-OneFitsAll

0.252

0.258

MSE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

Weather-CALF

0.225

0.240

MSE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

Weather-TimeLLM

0.420

0.440

MSE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

Traffic-OneFitsAll

0.430

0.440

MSE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

Traffic-CALF

0.400

0.425

MSE

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

Traffic-TimeLLM

0

80

160

0.420

0.430

0.440

MAE

Time-LLM-ETTh1

2.60

2.80

3.00

0.425

0.450

0.475

OneFitsAll-ETTh1

2.40

2.60

2.80

0.424

0.428

0.432

CALF-ETTh1

0

80

160

0.384

0.392

0.400

MAE

Time-LLM-ETTh2

2.60

2.80

0.392

0.400

OneFitsAll-ETTh2

2.25

2.40

2.55

0.380

0.384

0.388

CALF-ETTh2

0

100

200

0.370

0.375

0.380

MAE

Time-LLM-ETTm1

1.20

1.30

0.375

0.380

OneFitsAll-ETTm1

2.20

2.40

2.60

0.388

0.392

CALF-ETTm1

40

60

0.840

0.880

0.920

MAE

Time-LLM-Illness

8.00

8.50

9.00

0.840

0.880

0.920

OneFitsAll-Illness

34.40

35.20

0.870

0.885

CALF-Illness

0

80

160

Inference Time (ms)

0.260

0.270

MAE

Time-LLM-Weather

0.30

0.45

0.60

Inference Time (ms)

0.260

0.270

4.00

8.00

Inference Time (ms)

0.270

0.275

0.280

CALF-Weather

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

0

80

160

0.420

0.430

0.440

MSE

Time-LLM-ETTh1

2.60

2.80

3.00

0.425

0.450

0.475

OneFitsAll-ETTh1

2.40

2.60

2.80

0.424

0.428

0.432

CALF-ETTh1

0

80

160

0.384

0.392

0.400

MSE

Time-LLM-ETTh2

2.60

2.80

0.392

0.400

OneFitsAll-ETTh2

2.25

2.40

2.55

0.380

0.384

0.388

CALF-ETTh2

0

100

200

0.370

0.375

0.380

MSE

Time-LLM-ETTm1

1.20

1.30

0.375

0.380

OneFitsAll-ETTm1

2.20

2.40

2.60

0.388

0.392

CALF-ETTm1

0

100

200

Inference Time (ms)

0.306

0.312

0.318

MSE

Time-LLM-ETTm2

1.26

1.32

1.38

Inference Time (ms)

0.305

0.310

0.315

OneFitsAll-ETTm2

1.95

2.10

Inference Time (ms)

0.320

0.324

CALF-ETTm2

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

(a) Inference Time and Performance for ETTh1, ETTh2, ETTm1, and ETTm2.

40

60

0.840

0.880

0.920

MSE

Time-LLM-Illness

8.00

8.50

9.00

0.840

0.880

0.920

OneFitsAll-Illness

34.40

35.20

0.870

0.885

CALF-Illness

0

80

160

0.260

0.270

MSE

Time-LLM-Weather

0.30

0.45

0.60

0.260

0.270

4.00

8.00

0.270

0.275

0.280

CALF-Weather

1

2

0.280

0.290

0.300

MSE

Time-LLM-Traffic

0.30

0.45

0.60

0.260

0.270

OneFitsAll-Traffic

30.00

60.00

90.00

0.272

0.280

0.288

CALF-Traffic

4

8

Inference Time (ms)

0.256

0.264

0.272

MSE

Time-LLM-ECL

0.30

0.45

0.60

Inference Time (ms)

0.250

0.260

OneFitsAll-ECL

12.00

18.00

Inference Time (ms)

0.260

0.265

0.270

CALF-ECL

w/ LLM

w/o LLM

LLM2Attn

LLM2Trsf

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Abstract and the Introduction sections include the the paper's contributions and scope.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Appendix A discusses the limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not make any theoretical contributions.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Section 4 and Appendix D shows the details of the baselines used and their hyper-parameter details respectively.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have shared the code as a .zip file in the supplementary material.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Experiments 3.3 and Appendix D describe them.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

We have provided the details of confidence intervals for all our results in

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of computing workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We mention the hardware details in Appendix D

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes, we do follow the Code Of Ethics.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Appendix B describes broader impact.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA] .

Justification: The paper poses no such risks.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper properly credited, and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We provide the license details in Appendix C.

13. New Assets

Question: Are new assets introduced in the paper well documented, and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We attached the code in supplementary material. All implementation and hyper-parameter details are also reported in the paper.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing or research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing or research with human subjects.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper questions the effectiveness of LLMs in time series forecasting. Through a series of ablation studies on 3 LLM-based forecasting methods, the authors find that removing or replacing the LLM component often improves forecasting results. They conclude that LLMs do not provide significant benefits for time series forecasting tasks, despite their significant computational cost. The paper suggests that simpler models or encoders with basic attention mechanisms perform equally well, challenging the current trend of employing LLMs for time series forecasting.\n\nSTRENGTHS:\n- The paper offers an interesting perspective on the use of LLMs in time series forecasting, challenging prevalent assumptions and trends in the field.\n- The empirical analysis seems to be valid, with well-executed ablation studies and comparisons across multiple methods and datasets.\n- The paper is easy to read, with well-organized sections and effective use of figures and tables to present relevant results.\n- The findings may have implications for the research community, potentially redirecting efforts towards more efficient and effective methods for time series forecasting.\n\nWEAKNESSES:\n- The scope is limited to time series forecasting and does not explore other potential applications of LLMs in time series analysis, such as imputation, classification or anomaly detection.\n- The datasets used are all evenly spaced time series, which may not fully represent the variety of real-world time series data, such as those with irregular intervals.\n- The paper could benefit from a more detailed discussion on the potential reasons why LLMs fail to outperform simpler models in this context, providing deeper theoretical insights.\n\nQUESTIONS:\nRelated to my last point in \"weakness\", can the authors provide more theoretical explanations or hypotheses on why LLMs do not perform well in time series forecasting compared to simpler models?\n\nLIMITATIONS:\nThe authors acknowledge that the study focuses solely on time series forecasting and uses only evenly spaced datasets. Future research should explore LLMs’ effectiveness in other time series tasks and with non-uniform datasets. Additionally, potential negative societal impacts, such as the environmental cost of training large models, should be considered.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-29"}

comments: SUMMARY:
A recent surge of papers have popularised the usage of pre-trained LLMs for time series forecasting. The paper analyses the claim that LLMs are useful for time series forecasting, by performing a series of ablation studies. Their conclusion is that LLMs bring little to no benefit for the task, and are significantly more costly.

STRENGTHS:
1. The paper is very well written, the hypothesis is stated clearly and the experiments and results are also clearly stated.
2. Experimental approach is very sound, experiments are well thought out and very comprehensive.
3. The paper reveals very significant findings regarding a recent trend in time series forecasting, in which many papers at top conferences have been publishing about. It turns out that the performance of many of these methods is not due to a pre-trained LLM as touted by these papers, but due to other factors such as patching and channel independence is the most important.

WEAKNESSES:
1. Experiments in section 4.3 are only presented for LLaTa, it should be presented for all the models. Importantly, the OneFitsAll paper showed that pre-trained weights performed much better than random initialized weights. Can the authors comment on this?

2. The experiments in 4.6 lead to a model that is very similar to PatchTST and some other similar patch-based Transformers, which the LLM-based models claim to be better than. Can the authors comment on this?

QUESTIONS:
See weaknesses

LIMITATIONS:
Authors have adequately addressed limitations in appendix.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-05

comments: SUMMARY:
This paper explores the effectiveness of language models in time series forecasting. The authors substantiate their claim by performing three straightforward ablations of three popular and recent LLM-based forecasting methods. After extensive experiments, the authors find that patching and attention structures perform similarly to state-of-the-art LLM-based forecasters.

STRENGTHS:
1. The paper is well-structured and easy to follow.
2. The author question existing LLM-based forecasting models by conducting comprehensive ablation studies and provide very interesting and insightful observations.

WEAKNESSES:
1. Although the paper focuses on the effectiveness of LLM in TSF, existing state-of-the-art forecasting models have been omitted. 
2. The authors conduct ablation studies on the patching and decomposition of LLM-based models to find out where the performance comes from. However, there is a lack of discussion on LLaTA, which embeds the input time series by treating each channel as a token instead of using patching.

QUESTIONS:
1. Why the author choose ETTh1 and ILI datasets in RQ4? Results on other benchmark datasets are expected.
2. How does the author implement the "LTrsf" model, and what is the difference between "LTrsf" and the existing iTransformer model? 
3. Can you compare the results between "PAttn" and PatchTST?

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-06

comments: SUMMARY:
This paper presents an extensive empirical study on the effect of pre-trained LLMs in time series forecasting tasks. By ablating popular LLM adaptations on widely adopted time series benchmarks, experiments in the paper show LLMs do not benefit from pre-training on text data to gain improvement in forecasting performance. It suggests that the patching and first layer of attention makes most of contribution in forecasting setups.

STRENGTHS:
This paper is very well written and populated with extensive empirical evidences to support its main arguments. 
1. The experiment setting covers various types of LLM adaptation to forecasting, including simple fine-tuning, PEFT and modality alignment, as well as popular time series forecasting benchmarks, which makes the conclusions fairly solid. 
2. The reasoning behind the 6 research questions are convincing with rigorous ablation studies, and it covers multiple aspects of applicability of LLM in forecasting, from efficacy to efficiency. 
3.  Great reproducibility evidences are provided for an empirical study.

WEAKNESSES:
A questionable piece of results is the random baseline in Table 5. It is counter-intuitive that, while the `woPre+woFT` variant is completely unfitted according to line 208-210, but its metrics are quite close to other variants in all experiments. 

Minor typos/errors:
1. line 76: One method ~this~ is

QUESTIONS:
See weakness

LIMITATIONS:
Limitations are pointed out in the paper by the authors.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-10