Are Language Models Actually Useful for Time Series Forecasting?

Authors: mingtian  tan
mike a merrill
vinayak  gupta
tim  althoff
thomas  hartvigsen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 04:06:03

Abstract: Large language models (LLMs) are being applied to time series forecasting. But are language models actually useful for time series? In a series of ablation studies on three recent and popular LLM-based time series forecasting methods, we find that removing the LLM component or replacing it with a basic attention layer does not degrade forecasting performance---in most cases, the results even improve! We also find that despite their significant computational cost, pretrained LLMs do no better than models trained from scratch, do not represent the sequential dependencies in time series, and do not assist in few-shot settings. Additionally, we explore time series encoders and find that patching and attention structures perform similarly to LLM-based forecasters. All resources needed to reproduce our work are available: https://github.com/BennyTMT/LLMsForTimeSeries.

Main paper: 
 introduction: 
 Time series analysis is a critical problem across many domains, including disease propagation forecasting [8] , retail sales analysis [3] , healthcare [26, 17] and finance [31] . A great deal of recent work in time series analysis (constituting repositories with more than 1200 total stars on GitHub) has focused on adapting pretrained large language models (LLMs) to classify, forecast, and detect anomalies in time series [15, 50, 22, 4, 5, 32, 14, 44, 16] . These papers posit that language models, being advanced models for sequential dependencies in text, may generalize to the sequential dependencies in time series data. This hypothesis is unsurprising given language models are now pervasive in machine learning research. However, direct connections between language modeling and time series forecasting remain largely undefined. So to what extent is language modeling really beneficial for traditional time series tasks?Our claim is simple but profound: popular LLM-based time series forecasters perform the same or worse than basic LLM-free ablations, yet require orders of magnitude more compute. Derived from extensive ablations, this reveals a worrying trend in contemporary time series forecasting literature. Our goal is not to imply that language models will never be useful for time series. In fact, recent works point to many exciting and promising ways that language and time series interact, like time series reasoning [25, 7, 45, 42, 37] , social understanding [6] and financial reasoning [36, 20] . Rather, we aim to highlight surprising findings that existing methods do very little to use the innate reasoning power of pretrained language models on established time series tasks.We substantiate our claim by performing three ablations of three popular and recent LLM-based forecasting methods [50, 15, 22] using eight standard benchmark datasets from reference methods and another five datasets from MONASH [13] . First, we successfully reproduce results from the original publications. Then, we show that replacing language models with simple attention layers, basic transformer blocks, randomly-initialized language models, and even removing the language model entirely, yields comparable or better performance. The same performance was observed on another five datasets that were not studied by the reference methods.Next, we compare the training and inference speed of these methods against their ablations, showing that these simpler methods reduce training and inference time by up to three orders of magnitude while maintaining comparable performance. Then, to investigate the source of LLM forecaster's performance, we further explore time series encoders. We find that a simple linear model with an encoder composed of patching and attention can achieve forecasting performance similar to that of LLMs. Next, we test whether the sequence modeling capabilities of LLMs transfer to time series by shuffling input time series and find no appreciable change in performance. Finally, we show that LLMs do not even help forecasting in few-shot settings with 10% of the training data. We discuss the implications of our findings and suggest that time series methods that use large language models are better left to multimodal applications [4, 12, 38] that require textual reasoning.The key contributions we make in this paper are as follows:• We propose three straightforward ablation methods for methods that pass time series into LLMs for forecasting. We then ablate three top-tier methods on thirteen standard datasets and find that LLMs fail to convincingly improve time series forecasting. However, they significantly increase computational costs in both training and inference.• We study the impact of an LLM's pretraining by re-initializing their weights prior to forecasting. We find that this has no impact on forecasting performance. Additionally, in shuffling input time series, we find no evidence the LLMs successfully transfer sequence modeling abilities from text to time series and no indication they help in few-shot settings.• We find a very simple model, with patching and attention as encoder, can achieve performance similar to LLMs. This suggests a massive gap between the benefits LLMs pose and the time series forecasting problem, despite a rapid rush to adopt LLMs. 
related work: 
 Here, we summarize the key works relevant to LLM-based time series models. They can be broadly classified into three sections: (i) time series forecasting using LLMs; (ii) encoders in LLM time series models; and (iii) smaller and efficient neural models for time-series.Time Series Forecasting Using LLMs. Recently, with the development of Large Language Models (LLMs) [10, 29, 34] and their demonstrated multi-modal capabilities, more researchers have successfully applied LLMs to time series forecasting tasks [14, 16, 5, 4] . Chang et al., [5] used finetuning the transformer module and positional encoding in GPT-2 to align pre-trained LLMs with time series data for forecasting tasks. Zhou et al. [50] proposed a similar finetuning method, named "OneFitsAll", for time series forecasting with GPT-2. Additionally, Jin et al. [15] introduced a reprogramming method to align LLM's Word Embedding with time series embeddings, showing good representation of time series data on LLaMA [34] . Similarly, CALF [22] and TEST [32] adapted word embeddings to enable LLMs to forecast time series data effectively. In addition to time-series forecasting models, Liu et al. [23] show that these models can be extended to classifying health-time series, such as heart-rate and daily-footsteps. These models have also been shown to outperform supervised neural models in few-shot settings.Encoders in LLM Time Series Models. In order for an LLM to learn from text it must first be discretized and encoded as word tokens which are 1 × d vectors [10, 29, 34] . Similarly, LLM-based methods for time series learn discrete time series tokens. One method is to segment the time series into overlapping patches, which effectively shortens the time series while retaining its features [15, 50, 5, 4, 28, 27, 11] . Other methods decompose time series into trend, seasonal components, and residual components [4, 28] . Lastly, Liu et al. [22] feed the multivariate time series using a Transformer to enable different channels to learn the dynamics of other channels. These embedding procedures are followed by a linear neural network layer that projects the time series encoding to the same dimensions used by the pre-trained LLM. OneFitsAll [50] GPT-2 Add&Norm Fine-Tune Time-LLM [15] LLaMA None Freeze CALF [22] GPT-2 LoRA Fine-Tune Table 2 : Three popular methods for time series forecasting with Large Language Models. Small and Efficient Neural Forecasters. In addition to LLMs, there has been a large body of research on smaller yet efficient frameworks that outperform their bulky counterparts in time series forecasting [19, 47, 33, 24, 2] . For example, Zeng et al. [46] present DLinear, an incredibly simple model that combines decomposition techniques and achieves better forecasting performance than state-of-the-art transformer-based time series architectures at the time, such as Informer [48] , FEDformer [49] , and Autoformer [40] . Furthermore, Xu et al. [43] introduces a lightweight model with only 10k parameters, which captures both amplitude and phase information in the time-series to outperform transformer-based models. 
experimental setup: 
 We use three state-of-the-art methods for time series forecasting and propose three ablation methods for LLMs: (i) "w/o LLM"; (ii) "LLM2Attn"; (iii) and "LLM2Trsf". To evaluate the effectiveness of LLMs in time series forecasting, we test these methods on eight standard datasets.We experiment with three recent methods for time series forecasting using LLMs. All models were published between December 2023 and May 2024 and are popular, with their GitHub repositories collectively amassing 1,245 stars. These methods are summarized in Table 2 , and use either GPT-2 [29] or LLaMA [34] as base models, with different alignment and fine-tuning strategies.• OneFitsAll [50] : OneFitsAll, sometimes called GPT4TS, applies instance norm and patching to the input time series and then feeds it into a linear layer to obtain a input representation for the language model. The multi-head attention and feed forward layers of the language model are frozen while the positional embeddings and layer norm are optimized during training. A final linear layer is used to transform the language model's final hidden states into a prediction. • Time-LLM [15] : In Time-LLM the input time series is tokenized via patching and aligned with a low-dimensional representation of word embeddings using multi-head attention. The outputs of this alignment, combined with the embeddings of descriptive statistical features, are passed to a frozen pre-trained language model. The output representations of the language model are then flattened and passed through a linear layer to obtain a forecast. • CALF [22] : CALF embeds the input time series by treating each channel as a token. One half of the architecture is a "textual branch" which uses cross attention to align the time series representation with a low dimensional representation of the language model's word embeddings. This representation is then passed through a pretrained, frozen language model to obtain a "textual prediction". Simultaneously, a "temporal" branch learns a low-rank adapter for a pretrained language model based on the input time series to produce a "temporal prediction" which is used for inference. The model includes additional loss terms that enforce similarity between these representations.Reproducibility Note. While experimenting with each model, we tried to replicate the conditions of their original papers. We used the original hyper-parameters, runtime environments, and code, including model architectures, training loops, and data-loaders. To ensure a fair comparison, we have included error metrics from the original papers alongside our results wherever possible. To isolate the influence of the LLM in an LLM-based forecaster, we propose three ablations: removing the LLM component or replacing it with a simple block. Specifically, for each of the three methods we make the following three modifications:• w/o LLM (Figure 1 (b)). We remove the language model entirely, instead passing the input tokens directly to the reference method's final layer. • LLM2Attn (Figure 1 (c)). We replace the language model with a single randomly-initialized multi-head attention layer. • LLM2Trsf (Figure 1 (d) ). We replace the language model with a single randomly-initialized transformer block.In the above ablations, we keep left parts of the forecasters unchanged (trainable). For example, as shown in Figure 1 (a), after removing the LLM, the input encodings are passed directly to the output projection. Alternatively, as shown in Figure 1 (b) or (c), after replacing the LLM with attention or a transformer, they are trained along with the remaining structure of the original method.Benchmark Datasets. We evaluate on the following real-world datasets: ( 1 . We highlight that these datasets, with the same splits and size, have been extensively used to evaluate time-series forecasting ability of LLM-based and other neural models for time-series data [48, 50, 4, 15, 5, 46, 40, 49] . (6) Exchange Rate [18] : collected between 1990 and 2016, it contains daily exchange rates for the currencies of eight countries (Australia, British, Canada, Switzerland, China, Japan, New Zealand and Singapore). ( 7 [|c i -c i |] and MSE = 1 |D| ti∈D (c i -c i ) 2, where c i and c i denote the true value and predicted value at the i-th index of the time-series respectively. 
results: 
 In this section, we provide the details of our comprehensive evaluation of all baseline LLM models for time-series forecasting. Specifically, we ask the following research questions. (RQ1) Do pretrained language models contribute to forecasting performance? (RQ2) Are LLM-based methods worth the computational cost? (RQ3) Does language model pretraining help performance on forecasting tasks? (RQ4) Do LLMs represent sequential dependencies in time series? (RQ5) Do LLMs help with few-shot learning? (RQ6) Where does the performance come from? 4.1 Do pretrained language models contribute to forecasting performance? (RQ1)Our results show that pretrained LLMs are not useful for time series forecasting tasks yet. Overall, as shown in Table 3 , across 13 datasets and two metrics, ablations out perform Time-LLM methods in 26/26 cases, CALF in 22/26 cases, and OneFitsAll in 19/26 cases. We averaged results over different predicting lengths, as in [50, 15, 22] . Across all prediction lengths (thirteen datasets and four prediction lengths) ablations outperformed Time-LLM, CALF, and OneFitsAll in 35/40, 31/40, and 29/40 cases as measured by MAE, respectively. To ensure a fair comparison, we also report results from each method's original paper alongside our replication. For specific results refer to Appendix E.1. To better evaluate the effectiveness of LLMs and ablation methods, we include 95% bootstrapped confidence intervals for each task. In tasks where LLMs performed better, such as OneFitsAll with ETTh1, shown in Figure 2 , there is still substantial overlap in the confidence intervals with the ablation method "w/o LLM" in MAE. Other datasets results and MSE metrics are shown in Figure 5 and Figure 6 in the Appendix. To summarize, our results on the evaluation above, it is hard to conclude that LLMs are effective in time series forecasting.In the previous section, we showed that LLMs do not meaningfully improve performance on time series forecasting tasks. Here, we evaluate the computational intensity of these methods with their 2.3, and 1.2 times longer than the modified models. Examples can be seen in Figure 3 , where the green marks (ablation methods) are typically below the red one (LLM) and are positioned towards the left of the axis, indicating a lower computational costs and better forecasting performance. Other datasets and MSE metric refer to Figure 7 and Figure 8 in Appendix. In conclusion, the computational intensity of LLMs in time series forecasting tasks does not result in a corresponding performance improvement.Our evaluation in this section indicates that pretraining with language datasets is unnecessary for time series forecasting. To test whether the knowledge learned during pretraining meaningfully improves forecasting performance we experimented with different combinations of pretraining and finetuning CALF's [22] language model on time series.• Pretrain + Finetune (Pre+FT). This is the original method, wherein a pretrained language model is finetuned on time series data. In the case of CALF, the base language model is frozen and low rank adapters (LoRA) are learned. • Random Initialization + Finetune (woPre+FT). Does the textual knowledge from pretraining aid time series forecasting? In this method we randomly initialize the weights of the language model (thereby erasing any effect of pretraining) and train the LLM from scratch. • Pretrain + No Finetuning (Pre+woFT). How much does finetuning on time series improve prediction performance? For this baseline we again leave the language model frozen and forgo Overall, as shown in Table 5 , across 8 datasets using MAE and MSE metrics, the "Pretraining + Finetune" method performed the best 3 times, while "Random Initialization + Finetune" achieved this 8 times. This indicates that language knowledge offers very limited help for forecasting. However, "Pretrain + No Finetuning" and the baseline "Random Initialization + No Finetuning" performed the best 5 times and 0 times, respectively, suggesting that Language knowledge does not contribute meaningfully during the finetuning process. Detailed results refer to Table 20 in Appendix.In summary, textual knowledge from pretraining provides very limited aids for time series forecasting.Most time series forecasting methods that use LLMs finetune the positional encoding to help understand the position of time steps in the sequence [4, 50, 22, 5, 32] . We would expect a time series model with good positional representations to show a significant drop in predictive performance when the input is shuffled [46] . We applied three types of shuffling to the time series: shuffling the entire sequence randomly ("sf-all"), shuffling only the first half of the sequence ("sf-half"), and swapping the first and second halves of the sequence ("ex-half"). As shown in Table 6 , LLM-based methods were no more vulnerable to input shuffling than their ablations. This implies that LLMs do not have unique capabilities for representing sequential dependencies in time series. In this section, our evaluation demonstrates that LLMs are still not meaningfully useful in few-shot learning scenarios.While our results indicate that LLMs are not useful for time series forecasting, it is nonetheless possible that knowledge encoded in pretrained weights could help performance in few-shot settings where data are scarce. To evaluate whether this is the case we trained models and their ablations on 10% of each dataset. Specifically, we evaluated LLaMA in Time-LLM methods. The results for LLaMA, shown in Table 7 , compared LLaMA with completely removing the LLM (w/o LLM).There was no difference, with each performing better in 8 cases. We conducted similar experiments with CALF, a GPT-2-based method. Our results in Table 8 indicate that our ablations can perform better than LLMs in few-shot scenarios.Where does the performance come from? (RQ6)In this section, we evaluate common encoding techniques used in LLM time series models. We find that combining patching with one-layer attention is a simple and effective choice. In subsection 3.2, we found that simple ablations of LLM-based methods did not decrease performance. To understand why such simple methods work so well we selected some popular techniques used for encoding in LLM time series tasks, such as patching [50, 15, 5, 32, 22, 11] , decomposition [4, 28] . A basic transformer block also can be used to aid in encoding [22] .The specific results, shown in Table 18 in the Appendix, indicate that a structure combining patching and attention, named "PAttn", performs better than most other encoding methods on small datasets (with time stamps less than 1 million) and is even comparable to LLM methods. Its detailed structure, as shown in Figure 4 , involves applying "instance norm" to the time series, followed by patching and projection. Then, one-layer attention enables feature learning between patches. For larger datasets, such as Traffic (~15 million) and Electricity (~8 million), a model named "LTrsf," using the encoder from CALF [22] , performs better. In those methods, finally, time series embedding will be projected with a single linear layer to forecast. Details of other encoders is in Appendix subsection D.3.Overall, patching plays a crucial role in encoding. Additionally, basic Attention and Transformer blocks also effectively aid in encoding. 
conclusion: 
 In this paper we showed that despite the recent popularity of LLMs in time series forecasting they do not appear to meaningfully improve performance. We experimented with simple ablations, showing that they maintain or improve the performance of the LLM-based counterparts while requiring considerably less compute. Once more, our goal is not to suggest that LLMs have no place in time series analysis. To do so would likely prove to be a shortsighted claim. Rather, we suggest that the community should dedicate more focus to the exciting tasks could be unlocked by LLMs at the interface of time series and language such as time series reasoning [25, 7, 45, 37] , or social understanding [6] . 
 
 Appendix: Time Series Forecasting? (Appendix)A LimitationsHere, we discuss the limitations of our paper.1. We evaluate the ability of LLMs using time-series forecasting. However, to get a better picture of how LLMs can work with time-series, this ability should be evaluated across other downstream tasks as well, such as time-series classification and question-answering.2. Our evaluation is limited to only time-series datasets, i.e., sequences with even time-intervals. However, there also exists a large fraction of data in the form of non-uniform series, such as payment records, online purchases, etc. Understanding the ability of LLMs in forecasting non-uniform sequences is also necessary to verify our claim on the usefulness of LLMs for time-series data.One of the major impacts our study will have is on the influx of models that use LLMs for modeling time-series. Our results will help researchers to not simply follow the trend of using LLMs in all applications, but to check their usability in detail. Specifically, these findings will help them determine if the LLM component is necessary and if the computational costs are reasonable for the specific setting. In addition to the research community, our findings on the better performance of smaller and simpler models will help develop scalable models that are easy to understand, interpret, and can be deployed cheaply in real-world applications. While we agree that a majority of our results are experimental and limited to selected datasets, we feel that these results will also help researchers narrow down their search space for better models in time-series forecasting, and not simply neglect the simpler models.All our contributions will be released under the MIT License.We train and evaluate each reference method and each architecture modification using the same device. For Time-LLM [15] , applying LlaMA-7B [34] , we use NVIDIA A100 GPU with 80GB memory. For other methods [50, 22] , applying GPT-2 [29] , we use NVIDIA RTX A6000 GPU with 48GB memory. Though an analysis of memory footprint is beyond scope of this research, we note that training the baselines in the absence of LLM can be done within smaller GPUs as well.When reproducing the reference methods, we used the original repository's hyper-parameters and model structures. In the ablation study, due to the smaller model parameters, we adjusted the learning rate or increased the batch size in some cases. All other training details remained identical with the reference methods. The specific training details and hyper-parameters are provided in the code and run scripts of the repository2 . Note that the training process and hyper-parameters for the simple methods can also be accessed via this link.To investigate the source of LLM method performance, we conducted further research on encoders. We used various encoders to encode time series data, followed by a linear layer to project the time series embeddings to the forecast. The encoder structure combining patching and attention is shown in Figure 4 . The key difference between PAttn and PatchTST [27] is the absence of position embedding and the Feed Forward in PatchTST [27] , or, more specifically, replacing the Transformer Encoder with a simple single-layer Attention structure. It could be a simple yet effective method to help evaluate the trade-off between cost and performance for newly proposed methods.In addition, we propose three different neural models (i) "LTrsf": which performs better on larger datasets, uses CALF 's [22] encoder without cross-modal attention. This could be a potential source of CALF 's performance; (ii) "D-LTrsf"; and (iii) "D-PAttn": both of them decompose the time series into three sub-sequences and forecast each using the above two methods respectively, and then linearly combine the results for the final forecast. Across our results in Table 18 , we note that even simpler models significantly outperform LLM-based time-series models. In detail, the LLM-based models, all combined we able to appear 33 times as the best and the second-best performer. However, "PAttn" was outperforming them by appearing 34 times as the best and the second-best performer. In addition, the comparison between "PAttn", "LTrsf" and other state-of-the-art methods is shown in Table 19 .Here we present the results of additional experiments that also highlight the ability of LLMs in modeling time-series data.Since LLMs and deep learning models in general are probabilistic in nature, their predictions can vary across different runs and different random initializations. Thus, we report the confidence intervals (CIs) for all MAE and MSE predictions made by our baseline models. We report the CIs for MAE prediction by Time-LLM, CALF, OneFitsAll in Tables 9, 11, and 13, for MSE predictions in Tables 10, 12 , and 14, respectively. Across all results, we note that the range of variation is quite small, and these intervals do not affect our observations. To illustrate the subtle differences more clearly, we present the visualized results in Figure 5 and Figure 6 .Confidence Intervals for other Datasets. To evaluate the generality of the ablations in the paper, we introduce five additional datasets that have not been studied by the reference methods [50, 15, 22] .The above datasets are used in many time series forecasting studies [30, 9, 39, 1, 46] . The prediction lengths for the "Exchange Rate" are "96, 192, 336, 720", as in [46, 18] . The prediction lengths for the other four datasets are 30, 48, 56, and 12, respectively, following the settings in Chronos [1] . As shown in Tables 15, 16 , and 17, the forecasting performance on the five new datasets, using the three methods [50, 15, 22] we referenced, still demonstrates that language models are unnecessary for forecasting tasks.Here we provide the results for all methods and datasets that we were unable to add to the main paper.Inference Times. All results regarding "inference time" and forecast performance are shown in Figure 7 and Figure 8 or replace it with a single-layer attention ("LLM2Attn") or Transformer ("LLM2Trsf"), the performance on time series forecasting tasks with MSE metric does not decline and even improves, compared with original methods, such as "GPT-2" or "LLaMA". The vertical dashed line in the figures represents the results from the original paper.Methods CALF [22] OneFitAll [50] Time-LLM [ PAttn CALF [22] OneFitsAll [50] Time-LLM [15] DLinear [46] RLinear [21] PatchTST [27] iTransformer [24] TimesNet [41] FEDformer [49] Metric Table 20 : Pretraining on language datasets is not necessary for time series forecasting tasks. The table shows the performance of using pretraining models versus not using pretraining, as well as the combination of fine-tuning and not fine-tuning LLMs in time series forecasting. All resources needed to reproduce our work are available: https://github.com/BennyTMT/ LLMsForTimeSerieshttps://github.com/BennyTMT/LLMsForTimeSeries 


Decision: The paper presents an interesting empirical study by questioning the effectiveness of LLMs in time series forecasting.  The study considers ablations of LLM adaptation to forecasting and popular time series forecasting benchmarks. The finding that pretrained LLMs are not better than models trained from scratch is worth publishing.

Reviews: review: paper_summary: paper_summary: This paper presents an extensive empirical study on the effect of pre-trained LLMs in time series forecasting tasks. By ablating popular LLM adaptations on widely adopted time series benchmarks, experiments in the paper show LLMs do not benefit from pre-training on text data to gain improvement in forecasting performance. It suggests that the patching and first layer of attention makes most of contribution in forecasting setups.
strength_weakness: strength_weakness: A questionable piece of results is the random baseline in Table 5. It is counter-intuitive that, while the `woPre+woFT` variant is completely unfitted according to line 208-210, but its metrics are quite close to other variants in all experiments. 

Minor typos/errors:
1. line 76: One method ~this~ is
questions: questions: See weakness
limitations: limitations: Limitations are pointed out in the paper by the authors.
paper_summary: paper_summary: This paper explores the effectiveness of language models in time series forecasting. The authors substantiate their claim by performing three straightforward ablations of three popular and recent LLM-based forecasting methods. After extensive experiments, the authors find that patching and attention structures perform similarly to state-of-the-art LLM-based forecasters.
strength_weakness: strength_weakness: 1. Although the paper focuses on the effectiveness of LLM in TSF, existing state-of-the-art forecasting models have been omitted. 
2. The authors conduct ablation studies on the patching and decomposition of LLM-based models to find out where the performance comes from. However, there is a lack of discussion on LLaTA, which embeds the input time series by treating each channel as a token instead of using patching.
questions: questions: 1. Why the author choose ETTh1 and ILI datasets in RQ4? Results on other benchmark datasets are expected.
2. How does the author implement the "LTrsf" model, and what is the difference between "LTrsf" and the existing iTransformer model? 
3. Can you compare the results between "PAttn" and PatchTST?
limitations: limitations: yes
paper_summary: paper_summary: A recent surge of papers have popularised the usage of pre-trained LLMs for time series forecasting. The paper analyses the claim that LLMs are useful for time series forecasting, by performing a series of ablation studies. Their conclusion is that LLMs bring little to no benefit for the task, and are significantly more costly.
strength_weakness: strength_weakness: 1. Experiments in section 4.3 are only presented for LLaTa, it should be presented for all the models. Importantly, the OneFitsAll paper showed that pre-trained weights performed much better than random initialized weights. Can the authors comment on this?

2. The experiments in 4.6 lead to a model that is very similar to PatchTST and some other similar patch-based Transformers, which the LLM-based models claim to be better than. Can the authors comment on this?
questions: questions: See weaknesses
limitations: limitations: Authors have adequately addressed limitations in appendix.
paper_summary: paper_summary: This paper questions the effectiveness of LLMs in time series forecasting. Through a series of ablation studies on 3 LLM-based forecasting methods, the authors find that removing or replacing the LLM component often improves forecasting results. They conclude that LLMs do not provide significant benefits for time series forecasting tasks, despite their significant computational cost. The paper suggests that simpler models or encoders with basic attention mechanisms perform equally well, challenging the current trend of employing LLMs for time series forecasting.
strength_weakness: strength_weakness: - The scope is limited to time series forecasting and does not explore other potential applications of LLMs in time series analysis, such as imputation, classification or anomaly detection.
- The datasets used are all evenly spaced time series, which may not fully represent the variety of real-world time series data, such as those with irregular intervals.
- The paper could benefit from a more detailed discussion on the potential reasons why LLMs fail to outperform simpler models in this context, providing deeper theoretical insights.
questions: questions: Related to my last point in "weakness", can the authors provide more theoretical explanations or hypotheses on why LLMs do not perform well in time series forecasting compared to simpler models?
limitations: limitations: The authors acknowledge that the study focuses solely on time series forecasting and uses only evenly spaced datasets. Future research should explore LLMs’ effectiveness in other time series tasks and with non-uniform datasets. Additionally, potential negative societal impacts, such as the environmental cost of training large models, should be considered.

Comments: comment: Thanks for your response. The rebuttal addressed my concerns. I have raised my score to 6.