Context is Key: A Benchmark for Forecasting with Essential Textual Information
Authors: Andrew Robert Williams
Arjun Ashok
Étienne Marcotte
Valentina Zantedeschi
Jithendaraa Subramanian
Roland Riachi
James Requeima
Alexandre Lacoste
Irina Rish
Nicolas Chapados
Alexandre Drouin
Venue: ICML 2025 poster (2025)
Keywords: time series
forecasting
multimodality
foundation models
contextual forecasting
deep learning
machine learning
natural language processing
Primary area: applications->time_series
TL;DR: A forecasting benchmark with problems that require the combined use of numerical historical data and textual context.

Abstract: Forecasting is a critical task in decision-making across numerous domains. While historical numerical data provide a start, they fail to convey the complete context for reliable and accurate predictions. Human forecasters frequently rely on additional information, such as background knowledge and constraints, which can efficiently be communicated through natural language. However, in spite of recent progress with LLM-based forecasters, their ability to effectively integrate this textual information remains an open question. To address this, we introduce "Context is Key" (CiK), a time-series forecasting benchmark that pairs numerical data with diverse types of carefully crafted textual context, requiring models to integrate both modalities; crucially, every task in CiK requires understanding textual context to be solved successfully. We evaluate a range of approaches, including statistical models, time series foundation models, and LLM-based forecasters, and propose a simple yet effective LLM prompting method that outperforms all other tested methods on our benchmark. Our experiments highlight the importance of incorporating contextual information, demonstrate surprising performance when using LLM-based forecasting models, and also reveal some of their critical shortcomings. This benchmark aims to advance multimodal forecasting by promoting models that are both accurate and accessible to decision-makers with varied technical expertise.
The benchmark can be visualized at https://servicenow.github.io/context-is-key-forecasting/v0.

Context is Key: A Benchmark for Forecasting with Essential Textual Information
Andrew R. Williams!2 123 * *Arjun Ashok 123
Étienne Marcotte'
Valentina Zantedeschi 14 Jithendaraa Subramanian 125 Roland Riachi
James Requeima Alexandre Lacoste'
Irina Rish (Nicolas Chapados 127 Alleandre Drouin 124
Abstract
1. Introduction
Forecasting is a critical task in decision-making across numerous domains. While historical nu- merical data provide a start, they fail to convey the complete context for reliable and accurate predictions. Human forecasters frequently rely additional information, such background knowledge and constraints, which can efficiently be communicated through natural language. How- ever, in spite of recent progress with LLM-based forecasters, their ability to effectively integrate this textual information remains an open question. To address this, we introduce "Context is Key" (CiK), a time series forecasting benchmark that pairs numerical data with diverse types of care- fully crafted textual context, requiring models to integrate both modalities; crucially, every task in CiK requires understanding textual context to be solved successfully. We evaluate a range of ap- proaches, including statistical models, time series foundation models, and LLM based forecasters, and propose simple yet effective L.M prompt- ing method that outperforms all other tested meth- ods on our benchmark. Our experiments high- light the importance of incorporating contextual information, demonstrate surprising performance when using LLM-based forecasting models, and also reveal some of their critical shortcomings. This benchmark aims to advance multimodal fore- casting by promoting models that are both accu- rate and accessible to decision-makers with varied technical expertise. The benchmark can be visual- ized https: /ttpricewww..wweeeeeece.e/ss.wwc github.co 1o/conte
The prediction of future states of the world is cornerstone of decision making (Hyndman & Athanasopoulos, 2018) and intelligence (Wang, 2019). Articulated as time series forecasting, this problem pervades much of science and commerce.
Accurate forecasting relies on several decisions up to the practitioner (Hyndman & Athanasopoulos, 2018): Model selection: choosing appropriate forecasting model for given problem, and 2. Incorporating prior information: determining what relevant information to integrate into the model and how to do so effectively. This involves deci- sions about statistical priors, inductive biases in the model architecture, and other forms of domain knowledge integra- tion, all of which traditionally rely on expert knowledge and manual intervention. However, recent advancements in machine learning have shown promise in automating both model selection and the incorporation of prior information, accelerating the democratization of time series forecasting.
In the wake of the foundation model paradigm shift (Bom- masani et al., 2021), several works such as Liang et al. (2024); Chen et al. (2023): Lim & Zohren (2021), have addressed automatic model selection by learning flexible, adaptable models applicable across various problem sce- narios. However, these approaches are much more costly than traditional statistical methods, and provide debat- able improvements in performance (Garza & Mergenthaler- Canseco, 2024). Typically, these models process purely numerical time series, excluding the context that human forecasters rely on to incorporate prior information.
An alternative class of recent approaches (Jin et al., 2024; Liu et al., 2024c: Requeima et al., 2024) adapt large lan- guage models (LLMs) for forecasting and leverage natural language (NL) as an intuitive interface to integrate side in- formation. These methods overcome a significant limitation of traditional forecasting techniques by eliminating the need to manually encode priors or design specialized models. They further hold the promise of capturing broader range of prior knowledge and context, potentially leading to more comprehensive and accurate forecasts.
Equul contribution; order decided by coin toss. 'Core con- tributors. 'ServiceNow Research 'Mila- Québec AI Institute "nniversiéé de Montréal "Université Laval 'MeGill University University of Toronto 'Polytechnique Montréal. Correspondence to: Andrew R. Williams wandrew.williams williams @esrvicceow.com>> Arjun Ashok <arjun.ashot ashok servicenow.com....
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
Forecast
Additional contextual information: "Thi seres contains the power production photovatiic piant the state Alsbema, Over the 12:22:2:
power production happanad average
1500
58--5%%
10%--90%
1250
25%L-75%
50%
1000
access
sccess fistory the relevant context. forecast
the numerical history, insutf cient forecast accurateyy
much more accurate.
750
250
12:23
12:33 06
12-22 09
12-23
15
12-23 15
32223 23
12:3000
122303
12:350
12-23
1233
1222
12.33
Figure 1. An example task from the proposed Context is Key (CiK) benchmark with GPT-4o forecasts in blue and the ground truth in yellow. Left: Forecasts based on the numerical history alone are inaccurate, as nothing indicates reversion zero. Right: The context enables better forecasts because it reveals that the series represents photovoltaic power production. Hence. the model can deduce that no power produced at night. context also enables better estimation of peak hour of production by providing statistics the history.
Unfortunately, there are as yet no systematic evalua- tions of these models' abilities to jointly leverage historical observations and natural language for forecasting. While several benchmarks for context-aided forecasting have been recently released (Zhang al., 2023; Liu et al., 2024a; Xu et al., 2024; Emami et al., 2024; Merrill et al., 2024), their contexts are not guaranteed to be useful for improving per- formance. As such, it remains unknown whether existing models can enhance their forecasts by utilizing crucially relevant textual context.
only numerical data, and LLM-based forecasters capable of incorporating natural language context. Our analysis explores key factors such the impact of context con- ditioning, tradeoff's in model size and performance, and discusses failure modes of models (Sec. 5).
2. Problem Setting
Context-Aided Forecasting This work addresses the problem of context-aided forecasting, where the goal is to produce statistical forecasts by incorporating relevant side information (i.e., context). Let X = (Xi,.... X.] repre- sent series of random variables corresponding to historical observations in discrete time, where each X, E X C RR and let X [X.+1...., Xy] represent future observations. In classical statistical forecasting, the goal is to estimate the joint distribution of future observations given the historical ones, P(X, F XH). We further assume access to context, denoted C, which consists of additional data of arbitrary nature containing information relevant for predicting X and complementary the history XH. The task then be- comes estimating the distribution P(Xp P XH, C). Cru- cially, we restrict our focus to relevant context, which we define as context that does not degrade forecasts. Formally, for x Xx X : C, given some loss function L assess- ing a predictive distribution over Xxp against a realization Xr. L: P(Xp) x Xx - IR, we are interested models where, in expectation, forecasts that leverage context per- form better:
To this end, propose the Context is Key (CiK, pro- nounced kick) benchmark of forecasting tasks. CiK consists of tasks designed to assess forecasting model's s ability to use both 1. numerical input-output pairs and 2. essential textual context. As shown in Figure I, accurate forecasts in CiK are made possible only by effectively leveraging both numerical data and key information contained within the accompanying text.
Our contributions are:
CiK Benchmark: A collection of manually designed forecasting tasks spanning seven real-world domains, each requiring the integration of diverse contextual information that has non-trivial impact on forecasts (Sec. 3).
Region of Interest CRPS (RCRPS): A scoring rule to eval- uate context-aided forecasting performance, which pri- oritizes context-sensitiv windows and accounts for con- straint satisfaction (Sec. 4).
Direct Prompt Forecasters: A simple yet effective prompt- based approach to using LLMs as context-aided forecast- ers, which serves as a surprisingly strong baseline on CiK (Sec. 5.2).
E C(P(Xp XH. C),K) S E C(P(XF XH).xF). XF
Furthermore, although the nature of the context can vary widely, we specifically concentrate context communi- cated through natural language, which we refer to as "con- text" or "contextual information".
Extensive evaluation of diverse models CiK, including statistical models, time series foundation models using
2

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
Retail
Energy
Mechanics
Synthetic
4.2%
8.5%
9.9%
Economics
2%
Traffic
15.5%
20
Future
Causal
History
Intemporal
Covariate
16. 9%
36.6%
Climatology
Figure 3. Number of tasks per context type in CiK.
Public Safety
cally for the tasks. Overall, the time series in CiK exhibit diverse sampling frequencies. with observations ranging from every 10 minutes monthly intervals; additional de- tails data sourees can be found in Appendix A.I.
Figure 2. The tasks in the CiK benchmark rely on real-world nu- merical data from domains.
3. Context is Key: a Natural Language Context-Aided Forecasting Benchmark
Memorization mitigation: Building tasks with publicly available data introduces contamination risk: pretrained L.MSs and time series foundation models may have mem- orized portions of the data, potentially inflating evaluation performance. We employ several mitigation strategies. First, we prioritize live data sources that are continuously updated, such as Chen et al. (2001) and Ville de Montréal (2020), ensuring the data is collected after the training cut-off dates of the models that we evaluate. Second, where applica- ble, we use derived series that are not directly available in the raw data, such as incident logs converted into time series (Ville de Montréal, 2020). Finally, as a last resort, we apply minor transformations, such adding noise or shifting timestamps, but use these sparingly to limit their potential impact on the tasks, and to avoid misalignment between common-sense knowledge (e.g.. holiday dates) and the numerical data. The exact mitigation methods used, per data source, together with the number of tasks on which they are applied are given in Appendix A.
We introduce the Context Key (CiK) benchmark, collec- tion of probabilistic forecasting tasks where accurate pre- dictions require integrating both numerical data and natural language contextual information. CiK comprises 71 distinct tasks spanning seven real-world application domains (see Sec. 3.1), each featuring various stochastic components that can be instantiated into thousands of task instances (e.g.. time series, time windows, natural language formulation). These tasks encompass diverse types of contextual informa- tion that reveal various aspects of dynamical processes (see Sec. 3.2). Moreover, they are designed such that context is key in that it non-trivially unlocks accurate forecasts, e.g., by conveying causal relationships that reveal the effect covariate on the time series of interest. example task is shown in Figure I and others can be found in Appendix B.
Availability: CiK is open source. The complete set of tasks can be explored at text-is-key- forec/sttnrccaan//oooooa/ee
gf thub... ia/con and the source code, at https::/ggttie com/ coom//erviceoow//eoee http://owwcootettttoeeooeoo -eeeeeettti-------eeeeee for ecasting. All data sources used for CiK's tasks are openly available (see Appendix A.
3.2. Natural Language Context
For each task in the benchmark, we jointly sample numer- ical data from one of the series described in Sec. 3.1 and then manually craft the natural language context necessary to unlock accurate forecasts. In some cases, this context purely descriptive, providing information about the general nature of the target variable and its historical behavior, as seen in the task illustrated in Figure 1. In other cases, the raw numerical data is adjusted to reflect the influence of the context. For example, in one task based on data from Goda- hewa et al. (2021), ), an ATM is expected to be inaccessible during a specific time period in the future, leading to zero withdrawals (see Appendix B.3). In another task, electricity demand is projected surge due an incoming weather event (see Appendix B.2). For such cases, modify the series to incorporate patterns described by the context.
3.1. Domains and Numerical Data Sources
As illustrated in Figure 2, the majority (95%) of tasks in CiK based on real-world application domains. We leverage 2,644 time series sourced from publicly available datasets across seven domains: Climatology (solar irradiance and cloud coverage (Sengupta et al., 2018)); Economics (un- employment rates across states and counties (U.S. Bureau of Labor Statistics, 2024)); Energy (electricity consump- tion and production (Godahewa et al., 2021)); Mechan- ies (experimental properties of physical systems (Gamella et al., 2024)); Public Safety (fire department intervention counts (Ville de Montréal, 2020)); Transportation (high- way segment occupancy rates and average speeds (Chen et al., 2001)); and Retail (cash withdrawals from various ATMs (Godahewa et al., 2021)). The remaining 5% of tasks use simulated data from dynamical systems crafted specifi-
Overall, we include diverse forms of natural language con- text, each capturing a different aspect of the process under- lying the time series and providing complementary knowl- edge that a human expert could leverage for more accurate
3

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
forecasting. The types of context are described below and exemplified in the task illustrated in Figure 4. For additional clarity, further examples are provided in Appendix B and the distribution of tasks per context type is shown in Figure 3.
the number of trash C
3
CFF
tiee incidents responded to by Montreal firefighters the Marcier-
Assume that the number of trash fire incidants never be more than 20.
Hochelaga-Maiise Meisoneeue borough.
20.0
Intemporal information (cr) Information about the pro-
17.5
4
CH
cess that remains invariant in time. For example, a descrip-
12.5
In other years, yearly average number of incidents was 99 and the month with the most incidents was August.
10.0
tion of the process and the nature of the target variable,
7.5
Liebserved histrry observed history
as in Figure 4 (point 2). This includes patterns that can-
5.0
Ground Witt
2.5
Forecast Regler
not be inferred from the available numerical data, such
long-period seasonalities, or constraints values, such as positivity.
30
20
ppllicy aiming to erradicate field fires will implemented from July to August this year.
Conv
10
Future information (CF) Information about the future behavior of the time series. For example, a scenario to be simulated as in Figure 4 point 3, or expected events along with any entailed constraints, such as an inventory shortage restricting future sales amounts.
3019-11
2019-44
2019-07
2019-10 to
2920-01
2020-04
2028-07
2020-10
Ccausal
This series associated with the number field fires. However, field fires do not cause
Fiels
Trassh lires
fres
Historical information (cи) Information about the past behavior of the series that the available numerical history does not reveal. For example, statistics on past values of the series, as in Figure 4 (point 4), or an explanation for spurious patterns that should be be disregarded at inference, such as periodic anomalies caused by sensor maintenance.
Figure 4. Illustration of a CiK task annotated with types of natural language context: ① The short numerical history is misleading, suggesting increasing trend. However, contextual information compensates and enables accurate forecasts: 2 The intemporal information (ey) reveals the nature of the series, implying a sea- sonal pattern with greater prevalence in the summer months due to weather. ? The future information (CF) reveals that the series cannot continue its increasing trend. ④ The historical information (Czz) ) complements the short history by providing high-level statis- ties on past values. 5 The covariate information (Cow) reveals an association with another quantity: field fires. Could its behavior impacts future values of the target series? mation (Cemest) provides the answer.
Covariate information (Ccor) Information about additional variables that are statistically associated with the variable of interest. For example, a series correlated with the target values (as in Figure 4 point 5.)
Causal information (Ceausal) Information about causal re- lationships between covariates and the target variable. For example, the covariates are known to cause or are con- founded with the target variable, as in Figure 4 point O.
④ No, the causal infor-
Finally, for completeness, Appendix A.7 provides the distri- butions of lengths of the numerical historical data, prediction horizons and natural language context.
proper scoring rule designed specifically for context-aided probabilistic forecasting. This new scoring rule is an exten- sion of the Continuous Ranked Probability Score (CRPS; Gneiting & Raftery (2007)), a proper scoring rule that pro- vides comprehensive assessment of forecast quality by evaluating the entire predictive distribution rather than point forecasts. Since is based on the CRPS, the RCRPS can be calculated using only samples from the predictive distribu- tion, and so can be used even in cases where closed- form distributions are unavailable. The RCRPS extends the CRPS via two key components: region of interest and measure of constraint satisfaction.
3.3. Validating the Relevance of The Context
Related efforts on context-aided forecasting, outlined in Sec. 6, rely on scraping and/or LLMs to obtain natural lan- guage context (Zhang et al., 2023; Merrill et al., 2024; Liu et al., 2024a; Emami et al., 2024). In contrast, to ensure both the quality and relevance of tasks in CiK, we manually craft all contextual information and associated data transforma- tions according to the procedure described in Appendix A.2. To validate the importance of the context, we subject each task to review by panel of human and LLM evaluators tasked with assessing if the context enables better forecasts. The results are overwhelmingly positive, with humans find- ing the context to relevant for 95% of evaluated instances (details in Appendix A.4).
Region of interest (RoI): The RCRPS assigns more weight to errors a task's Rol, which is a subset of time steps I S [++1,.... T] for which the context is particularly relevant. For example, in the ATM task from Sec. 3.2 (vi- sualized in Appendix B.3), the Rol denotes the time steps during which the ATM is expected to be unavailable. In other tasks, such as those in Figures I and 4, where the con- text informs the value of all future time points, the RCRPS assigns equal weights to all time steps (for readability, we report the definition of RCRPS for this special case in Ap-
4. Region of Interest CRPS
Alongside the tasks, we introduce the Region of Interest Continuous Ranked Probability Score (RCRPS), a novel
4

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
Table 1. Results of selected models the CiK benchmark. Starting from the left, the first column shows the RCRPS averaged over all
tasks. The second column shows the rank of each method w.r.t other models, averaged over all tasks. The remaining columns show the average RCRPS stratified by types of context (Sec. 3.2). All averages are weighted according to the scheme described in Sec. 5.1 and accompanied by standard errors. Lower is better and the best averages are in bold. An asterisk (*) denotes models that do not use natural language context. For results on all models and with alternative aggregation strategies, see Appendix C.
MODEL
AVERAGE RCRPS
AVERAGE RANK
INTEMPORAL INFORMATION
HISTORICAL INFORMATION
FUTURE INFORMATION
COVARIATE INFORMATION
CAUSAL INPORMATION
DIRECT PROMPT (ours)
1 11ama-3. 1 1-495B-Tnss
0.159 + 0.008
4,516 + 0.233
0.174 + 0.010
0,146 + 0.001
0.075 + 0.005
0.164 + 0.010
0.398 + 0.045
Llama-3 3------7------ Inst
0,286 + 0.004
7.803 + 0.106
0.336 +: 0.006
0.180 + 0.003
0.194 + 0.006
0.228 + 0.004
0.629 + 0.019
Mixtral- 8x78- Inst
0,523 +: 0.023
14.473 +: 0.147
0.723 +: 0.037
0.236 + 0.002
0.241 1 0.001
0.359 +: 0.028
0.875 1 0.128
Owen 2.5 78 Inst
0,290 +: 0.003
11.330 1: 0.253
0.290 1: 0.004
0,176 + 0.003
0.287 1 0.007
0.240 1 0.002
0.525 1: 0.003
Owen 2.5 0.58 Inst
0,463 +:2 0.012
12.694 +: 0.173
0.609 + 0.019
0.165 + 0.004
0.218 = 0.012
0,476 1 0.015
0.429 + 0.006
GPT-40
0.274 + 0.010
4,381 + 0.159
0.218 + 0.007
0,118 0,001
0.121 = 0.001
0,250 - 0.011
0.858 +
SPT-40-mini
0,354 + 0.022
9.056 + 0.194
0.475 + 0.035
0.139 + 0.002
0.143 = 0.002
0,341 = 0.028
0.1228
LLMP
1amm---700--nst
0.539 + 0.013
8,243 0.231
0.438 + 0.017
0.516 0.028
0.847 + 0.024
0.546 + 0.016
0.392 0.028
1ama-3-708
0.236 + 0.006
6.522 + 0.244
0.212 + 0.005
0.121 0.008
0.299 0.017
0.193 0.004
0.360 0.011
Mtxtral-6x78-Ine
0.264 + 0.004
8.519 0.264
0.242 + 0.007
0.173 0.004
0.324 + 0.005
0.219 0.008
0.437 0.007
Mixtral-Bxe 6x78
0,262 + 0.008
8,540 4: 0.198
0.250 1: 0.008
0.119 0.003
0,310 1 0.019
0.229 1 0.006
0.457 0.011
Owen 2.5 78 Inst
1.974 1: 0.027
18,443 1: 0.276
2,509 1: 0.044
2,857 1 0.056
1.653 1 0.008
1,702 + 0.035
1.333 1 0.144
Owen 2.5 78
0.910 + 0.037
16.051 +: 0.341
1.149 1 0.047
1,002 0.053
0,601 1 0.071
0.639 0.047
0,928 1 0.129
Qwen-2. 2........8... Inst
1.937 + 0.024
20.136 + 0.191
2.444 + 0.038
1,960 0.063
1,443 0.010
1.805 = 0.030
1.199 1 0.129
Qwen-2.5-0.00
1,995 + 0.024
19.686 + 0.275
2.546 + 0.039
2.083 0.052
1.579 = 0.015
1.821 = 0.030
1.225 0.128
MULTIMODAL MODELS
UniTime
0,370 0.001
14.675 + 0.091
0.457 + 0.002
0.155 + 0.000
0.194 + 0.003
0.395 + 0.001
0.423 1 0.001
Time -LM (ETThT)
0,476 + 0.001
17.932 + 0.075
0.518 + 0.002
0.183 + 0.000
0.403 : 0.002
0.441 + 0.001
0,482 + 0.001
TS FOUNDATION MODELS*
Lag-Llame
0.327 + 0.004
13.370 + 0.233
0.330 + 0.005
0.167 + 0.005
0.292 = 0.009
0.294 + 0.004
0.495 + 0.014
Chronos-Large
0.326 + 0.002
12.298 + 0.148
0.314 + 0.002
0.179 + 0.003
0.379 + 0.003
0.255 + 0.002
0.460 + 0.004
TimeGFN
0.353 + 0.000
15.047 + 0.098
0.332 + 0.000
0.177 + 0.000
0.405 + 0.000
0.292 + 0.000
0.474 + 0.000
Moirai-Large
0.520 + 0.006
12.873 + 0.263
0.596 +: 0.009
0.140 + 0.001
0.431 + 0.002
0.499 + 0.007
0.438 0.011
STATISTICAL MODELS*
ARIMA
0.475 + 0.006
12.721 1 0.167
0.557 + 0.009
0,200 0.007
0,350 : 0.003
0.375 + 0.006
0,440 ::000000 0.011
ETS
0.530 : 0.009
15.001
+ 0.198
0.639 + 0.014
0,362 0.014
0.315 = 0.006
0.402 = 0.010
0,508 1 0.017
Exp Exp-Smoothing
0.605 : 0.013
15.689 + 0.152
0.702 + 0.020
0.493 + 0.016
0.397 = 0.006
0.480 = 0.015
0.827 + 0.060
pendix E).
across tasks; its calculation is described in Appendix E. Finally. / is scaling factor that controls the impact of constraint violation on the score; we use / = 10 in our ex- periments. We refer the reader to Appendix E for additional details.
Constraint satisfaction: The RCRPS penalizes constraint violations according to a task-specific function Vi: whose value is positive for any trajectory that violates the con- straints. Concrete examples are given in Appendix E.4.
Given an inferred forecast distribution X and a ground truth Xy, the scoring rule is defined as:
5. Experiments and Results
In this section, we define our evaluation protocol (Sec. 5.1) and outline the models that we evaluate on CiK (Sec. 5.2). We then present results on the benchmark (Sec. 5.3), along with an analysis of factors affecting model performance. Finally, we look at areas for improvement by analyzing forecasting errors (Sec. 5.4) and inference cost (Sec. 5.5).
RCRPS(X xp)xp)
Σ x..)+
2
iEI
Σ
CRPS(X..2) + B CRPS(
2|-2
where the terms respectively account for the CRPS inside the Rol, the CRPS outside of the Rol, and the constraint violation penalty. The last term, which is inspired by the threshold-weighheed CRPS of Gneiting& & Ranjan (2011), 1), vanishes when all constraints are satisfied. The a term is a task-dependent normalization factor to make the RCRPS ccale-independent which enables fair RCRPS aggregation
5.1. Evaluation Protocol
Each task in CiK has many unique instances arising from the selection of time series and windows in the associated numerical data, as well as minor variations in natural lan- guage context. We deterministically sample five instances of each task in order to make the evaluation reproducible and affordable. For every instance, we generate 25 independent
5

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
forecasts per model for evaluation. Many of the tasks in the benchmark share similarities due to sharing the same data sources or using variants of the same context. Therefore, we identify clusters of similar tasks and design a weighting scheme such that each cluster has equal total weight in our aggregate score (see Appendix A.5 for more details).
Beats all
Beats or none
Beats 3 or
Beats or
With context
Without context
DP Laaa--311-440 3.44--.1-408B-nnt
Lamma-3770--nnt
LLMP Llama-a-7B-innt
LLMP Liama 3-708
LLMP Miktral-8k7B-nnt
DP GPT-4o-mini
5.2. Methods
DP Qwwen-2.-78-innt
LLMP Mixtral-kx7B
DP Mixtral-8x78-ine
We evaluate a wide variety of models including methods based on LLMss, state-of-the-art numerical time series foun- dation models (TSFMs) and classical statistical forecasting methods. Since CiK is meant to be evaluation bench- mark, it does not have corresponding training set. We therefore only evaluate models that can produce forecasts directly based on the history of series. This includes LLMs and TSFMs that support zero-shot inference, and traditional statistical models that can be fit directly to the history of given instance. We outline these methods below and refer the reader Appendix D for additional details.
DP Qwen- 0.5--8---ne
LLMP Oween-25-78-Ins
LLMP 0wen-2.5-76
LLMP Qwen-2.5-0.58
LLMP Swwen-25-0058--1 (ETTh1)
0% 25%
50% 75%
100%00%
25% 50% 75% 100%
Figure 5. Proportion of tasks for which LLM-based methods out- perform the 7 quantitative forecasting methods (see Sec. 5.2). A method is considered to outperform another task if its average RCPRS is lower on said task. Results are shown for variants that use (left) and do not use (right) the natural language context. full green bar would indicate that the method is better on all tasks, whereas a full red bar would indicate that it is worse everywhere Tasks are weighted according to Sec. 5.1.
LLM-based Forecasters: We consider two prompt- based approaches. We propose "DIRECT PROMPT", a simple ap- preach where we instruct the model to directly output a forecast as a structured output for all of the required timestamps (see Appendix D.1 for more details). We also evaluate LLM Processes (LLMP: Requeima et al. (2024)), method which autoregressively prompts the LLM multiple times to output a forecast. For each of these. we evaluate a variety of LLMs with diverse archi- tectures and sizes, such as GPT-4o, GPT-40-mini (Achiam et al., 2023), Qwen-2. 6-.0.58, 1.5B, 7B} (Yang et al., 2024), Mixtral-8x77 (Jiang et al., 2024), 3 (8R, 708} (Dubey et al., 2024), Llama-3. 1--4-3 1-4058 (Dubey et al., 2024).² We also evaluate multimodal forecasting models, ChatTime (Wang et al., 2025) (zero-shot), UniTime (Liu et al., 2024c) and Time-LLM (ETTh1) (Jin et al., 2024) (trained accord- ing to their respective authors' guidelines) (details in Ap- pendix D.4). For all of these approaches, inference is per- formed zero-shot on the benchmark and we compare their performance with and without the natural language context.
task instance, which is the same input that the TSFMs pro- cess to produce forecasts zero-shot.
5.3. Results on CiK
Tab. I shows our main results. At a high level, we observe that the best-performing methods combine pretrained LLMs with prompting strategies like DIRECT PROMPT and I.I.P,, with a bias towards the largest models. In terms of RCRPS, Llama-3. 1-.11440IIss (DIRECT PROMPT) significantly outper- forms all of its counterparts. GPT-40 (DIRECT PROMPT) per- forms worse with respect to RCRPS, but compares favorably in terms of average rank. This discrepancy is due strong failures on some of the tasks, which we discuss in Sec. 5.4. Other models like Llama- 3-766 (LLMP), Mixtral-8x78- Inst (LLMP) and Mixtral-8x7B (LLMP) are on par with GPT-4o (DIRECT PROMPT), Llama-3 3- 3-788- Inst (DIRECT PROMPT) and 2. 5-sstt (DIRECT PROMPT) in terms of RCRPS. In- terestingly, all of these methods outperform UniTine and Time-LLM, which also rely on LLMs (GPT-2 & LLaMA-7B). We discuss this gap in Appendix D.4. Finally, as empha- sized in Figure 5, we observe that the best-performing LLM methods significantly outperform purely quantitative mod- els. In what follows, we examine various aspects of these results (and refer to Appendix C for additional results).
Quantitative Forecasting Models: We also evaluate number of models that process only numerical data, but cannot integrate natural language. We evaluate Exponential Smoothing (Gardner Jr. (1985)), ETS (Hyndman et al., 2008), and ARIMA (Box et al., 2015), three simple, but time-tested statistical approaches. We also evaluate four state-of-the-as TSFMs: Lag-Llama (Rasul al., 2023), Chronos (Ansari al., 2024), Koirai (Woo al., 2024) and TimeGEN (Garza et al., 2023). We fit Exponential Smoothing, ETS and ARIMA to the individual numerical history of each
Explaining the performance of LLM--bsed ap- proaches: The strong performance of LLM-based methods could be due to two factors: (i) properly leveraging the natural language context and (ii) being more proficient at numerical forecasting. We thus aim to disentangle their contributions.
For LLMP, we do not consider Llamo-3. 1--450 and GPT models as LLMP requires loading model weights into memory, which is infeasible due to resource limitations and confidentiality of the respective models (see Appendix D.2.2 for more details).
On the one hand, Figure 6 shows clear evidence that meth- ods with access to the context improve their forecasts. For
6

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
example, Llama-3. 3.ma-3.1-4058 1--011------- Inst (DIRECT ProMPT) improves by 67.1% with context. We find these differences sta- tistically significant across many of the models (see Ap- pendix C.6). This is reflected in the quality of the example forecasts in Appendix C.I1, where we observe clear im- provements in regions of interest, as well as improved con- straint satisfaction. Other models show lesser improvements and, in some cases, even a degradation in performance. Our analysis in Sec. 5.4 shows that this can be explained either by the context being ignored, or by significant failures in using context, worsening overall performance.
2,00
With Context
Without Context
Aggregate RCRPS
(Lower is Better)
1.00
LLM Process
Direct Prompt
Multimodal
0.50
0.25
Uiama 1.4058-inst
0.10
Liama 3-70
Mixtral-Ba a8278
Hixtral Bx B--B-tt
Liama -nn-
Owen-2 2. 5-78-inof
GFT-
Owen S. SB-inot
Unitime
Mixtral-,, 8x77----r-
Time-Li
Liama 7.708 inet
Owen--2.5 5.0. SB-inst
Qwen-2.5 Qwen-2.5577 5-78
Owen-2. 5-78-inst
Owen-2 2-0.225-.... 58
On the other hand, Figure (right) shows that LLM-based forecasters, when evaluated without context, no longer dom- inate the quantitative forecasting models. However, some LLM-based forecasters remain surprisingly competitive. For instance, multiple L1ama-3 (LLMP) models outperform at least 5 of the quantitative models on the majority of tasks. The extended results in Appendix C.1 further substanti- ate this. In contrast, models such as L1ana-3. (DIRECT PROMPT) and GPT-4o (DIRECT PrOMPT) show signif- icantly weaker forecasting performance without context. This suggests that such models are especially preferable in cases where context is available. This is also reflected in their aggregate scores without context (in Tab. 3).
Figure 6. RCRPS with and without context (log scale, lower is better). Full hars show performance with context; striped hars show performance without context. In general. larger models outperform smaller models and benefit much more from con- text. DIRECT PROMPT models all improve with context, other than 2.5 0.58 Instruct. For LLMP, larger models benefit from context, but smaller models fail to do so and perform worse in general.
Inst
benchmark remains unsolved, leaving significant room for advancements from the research community.
5.4. Error Analysis
Foundation models are known to make mistakes return factually inaccurate information (Bommasani et al., 2021). We find that models occasionally return forecasts that miss the ground truth by large margin, We use the term signif- icant failure to denote a forecast that over or undershoots by at least 500% of the range of the ground truth; we clip the RCRPS of such instances tionately skewing the aggregate score. Despite this, such significant failures impact the results in Tab. 1: GPT-4o with DIrEcT PROMPT, while emerging as a top-performer in most tasks (as reflected in its average rank), has a significantly higher aggregate RCRPS than models ranked worse, such as Mixtral-8x78 with LLMP. As an example, GPT-4o with DIREcT PROMPT fails significantly in a task with a context in- volving scientific notation (see Figure 25 and more examples in Appendix C.12). Notably, while model may generally achieve a high win rate, a few significant failures can domi- nate its aggregate performance, as observed for GPT-4o with DIREcT PROMPT. In the case of Qwen-2. 5.0. -.-...III Inst with DIRECT PROMPT, this leads to an aggregate RCRPS that is worse with context than without. We analyze this in detail in Appendix 1. These findings underscore the need for future work to develop more robust models that can handle context effectively while avoiding significant failures.
Comparing LLMP and DIRECT PROMPT: Figure 5 (right) shows that without context, LLMP models exhibit stronger numerical forecasting performance than DIRECT PROMPT models. This advantage likely stems from LLMP closer alignment with the forecasting task: LLMP simply prompts the LLM autoregressively predict the next value in the time series, task well suited to base models with no in- struction tuning. In contrast, DIRECT PROMPT requires output forecasts to be structured, which relatively complicates the task.
avoid them dispropor-
This line of reasoning leads us to another observation on the impact of instruction tuning; as reflected in Tab. 1 and Fig- ure 5, instruction tuning appears to generally degrade LLMP performance. Llama-3 models show twofold decrease in performance after instruction tuning, a behavior previously observed by Gruver et al. (2024). Interestingly, instruction tuning does not degrade the performance of Mixtral-8x78. Finally, whereas L.MP mostly suffers from instruction tun- ing, DIRECT PROMPT requires forecasts be produced in specific structure, skill that instruction-tumed models might be better at (see Appendix 1. for details).
No Method Excels Across All Context Types: Tab. shows that some methods can effectively produce better forecasts with the provided contextual information. How- ever, no single method is the best across context types. Llama 1.1.......0 1--888B-Ins Inst (DIRECT prompT), the top-performing method on average. outperforms its counterparts only 3 out of 5 context types. This finding indicates that the
5.5. Inference Cost
A key practical aspect for forecasting applications is the inference time of models and their associated cost. Fig- ure 7 shows that, while Llame-3. 3.mme-3..-405.. 1-4058-Instruct performs the best on average, it comes at the cost of a significantly higher parameter count than the quantitative forecasters.
7

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
abilistic forecasting metrics, e-g.. the continuous ranked probability score (CRPS; c.f. Gneiting & Raftery, 2007), to assess the quality of entire forecast distributions. Other re- lated datasets include Time-MMD (Liu et al., 2024a), which integrates text extracted from reports and web searches, TGTSF (Xu et al., 2024), which incorporates information such as weather reports and news articles, SysCaps (Emami et al.. 2024), which includes LLM -ggeeratdd descriptions of building energy consumption systems, TS-Insights (Zhang et al., 2023), which includes LLM-generated descriptions of trends and seasonalities, and Dual-Forecaster (Wu et al., 2025) where time series datasets are captioned with trend and seasonality information. Several works (Sawhney et al., 2021; Liu et al., 2024b; Wang et al., 2024b;a; Kim et al., 2024) automatically construct datasets of paired textual and numerical information. The key distinction between these works and ours lies in the importance of the textual infor- mation: while in the above works, the text is not guaranteed to be essential for high-quality forecasts, in CiK, all tasks are handcrafted to ensure that accurate forecasts cannot be achieved without relying on the provided natural language context, thereby making it a high-quality evaluation bench- mark for context-aided forecasting.
Direct Prompt LLM Process
A
Multimodal
Quantitative ve With Context Without Context
Aggregate RCRPS
Exp Smoothing ng
0.6
ETS
0.4
Arima
0.3
Lag-Uama
Owen-2.5 www.-mo 0.-5-2.--..88-imn 0.58-inst Ino context))
0.2
Qwen-2.5-73 2-3-7tt
Mixtral-8x78
0.15
Uama-3-770
708
Llama-3.1 1-18-55---stt
0
10
100
10
10a
10
10
10a
Parameter Count
Figure 7. Comparison of average RCRPS (per Tab. 1), vs. the parameter count of each method(lower is better for both). The GPT family. as well as TimeGEN, are left out as there is no information on them about parameter count. The dashed line illustrates the Pareto front: models above and to the right of this line are dominated. Quantitative forecasters dominate the low-parameter regime, while LLM-based methods such as Qwen-2. 5--8---7-Is Inst (DIRECT PROMPT), Llame-3-788 (L.MP) and Llama-3. 1-4958--Ims Inst (DIRECT prompT) offer superior performance for a higher parameter count.
This emphasizes that while LLMs can be powerful context- aware forecasters, they come with steep computational costs, which highlights the need for efficient models that balance both accuracy and resource demands. Of note is also that many LLM-based forecasters are pareto-dominated by quan- titative forecasters such Lag-Llama and Chronos. This sug- gests that, beyond the ability to process text, a careful choice of prompting strategy and LLM is crucial for Pareto effi- ciency. Further, due to their high parameter count, the L.MSs have inference times that are orders of magnitude longer than quantitative forecasters, which are far more efficient for sustained usage (see Figure 13 for a comparison). LLMs require significant computational power, making them un- suitable for real-world practical forecasting at scale where speed and cost matter. However, additional research could improve the cost-efiectiveness of context-aided forecasting models to match that of traditional models.
Repurposing LLMs for Forecasting A natural approach to context-aided forecasting is to build methods based on LLMs. Xue & Salim (2023) showed that forecasting could be framed as a question-answerna problem. Subsequently, Gruver et al. (2024) and Requeima et al. (2024) showed that LLMs could generate accurate forecasts with sequence completion, and that textual side-information could be used to influence forecasts. However, their analysis is limited to illustrative examples rather than comprehensive evaluation. Some works have explored the ability of LLMs to reason about time series (Chow et al., 2024; Kong et al., 2025; Aksu et al., 2024; Potosnak et al., 2024; Ye et al., 2024). Other approaches have incorporated time series into pretrained LLMs (Jin et al., 2024; Liu et al., 2024c: Zhang et al., 2024) by introducing special tokens used to represent patched time series patterns, or modifying their encoders to account for time series data (Jia et al., 2024). While these methods show promising results, their evaluations primarily rely on datasets where the contextual information is not guaranteed to improve forecasts over numerical data alone, As result, it remains unclear whether their performance is due by ac- curate numerical forecasting or by effectively incorporating context; this shortcoming motivates our investigation into this question.
6. Related Work
We review two streams of related work: (i) works that intro- duce related benchmarks and datasets, and (ii) works that repurpose L.LMs for forecasting.
Benchmarks and Datasets Merrill et al. (2024) present a benchmark designed to evaluate LLMs' ability to rea- son about time series, with context-aided forecasting as one assessed capability. They focus on purely synthetic time series, which may not accurately reflect real-world dynamics, whereas our benchmark is based primarily on real-world data. Further, their evaluation is limited to point forecasting metrics, which do not measure the quality of the full forecast distribution, In contrast, we adopt prob-
7. Discussion
In this work, propose the Context is Key (CiK) bench- mark: collection of forecasting tasks that require pro- cessing historical data with essential natural language con- text. We evaluate a range of models on CiK, including our
8

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
proposed LLM prompting method, DIRECT PROMPT, which achieves the best aggregate performance. We analyze and discuss the failure modes of these models, and our find- ings underscore the critical role of contextual information in improving forecasts, while also revealing both the unex- pected strengths and notable limitations of the investigated LLM-based forecasters.
Impact Statement
This paper's goal is to advance the development of fore- casting methods which can leverage contextual information, which can increase the likelihood that such methods will be adopted by various organizations. Therefore, this paper shares the positive and negative impacts that such meth- ods will have. Such methods could increase forecasting accuracy and efficiency, increasing the ability to foresee future courses and plan in consequence. A secondary effect is an increased democratization of access to high-quality forecasts. On the flip side, this democratization could in- crease reliance on automated methods for decision-makee -making purposes.
Limitations: While our benchmark provides valuable in- sights into the integration of contextual information in time series forecasting, it is important to acknowledge its limita- tions. Our study limits itself to the natural language modality for context, and excludes multivariate time series scenarios. Although we deliberately designed the tasks to assess how well forecasting models can integrate contextual informa- tion, our benchmark does not evaluate whether models can leverage latent relationships that might elude human obser- vation. While we have taken steps to mitigate memorization concerns, as discussed in Sec. 3.1, achieving absolute cer- tainty in this regard is challenging without strictly held-out data. Finally, the performance of dataset-specific methods such as ChatTime (Wang et al., 2024a), UniTime (Liu et al., 2024c) and Time-LLM (Jin et al., 2024) may improve in the presence of dataset-specific training set.
Acknowledgements
The authors are grateful to Andrei Dinin, Christian Hudon, Ethan Honey, Gabrielle Gauthier Melançon, Ghazwa Dar- wiche, Kiarash Mohammadi, Léo Boisvert, Loic Man- dine, Megh Vipul Thakkar, Oluwanifemi Isaac Bamgbose, Orlando Marquez, Raymond Li, Thibault Le Sellier de Chezelles and Thomas Lai for participating in the human study on the relevance of context. The authors are grateful to Midan Kim, Torsten Scholak, Mohammad Reza Sam- sami, Oussama Boussif and Can Chen for their valuable feedback and suggestions. This research was supported by Mitacs Accelerate Grants and enabled by compute resources provided by ServiceNow Research and the Frontier super- computer. The latter resources were awarded through the Frontier DD allocation and INCITE 2023 program for the project "Scalable Foundation Models for Transferable Gen- eralist Al" and were supplied by the Oak Ridge Leaersship Computing Facility at the Oak Ridge National Laboratory, with support from the Office of Science of the U.S. Depart- ment of Energy.
Future work: There are several promising avenues for future work. Extensions to CiK could include multivariate forecasting tasks, or tasks that incorporate other modalities such as images, databases or spatiotemporal data. Tasks that deliberately challenge context length limitations, probe specific weaknesses of language models or include domain- specific expert knowledge would also be valuable additions. More research is also needed to better understand what drives the catastrophic failures that LLMs exhibit. The anal- ysis of catastrophic failures could benefit from searching for systematic failure patterns, or failures associated with specific linguistic patterns. Training datasets for context- aided forecasting would enable better evaluation of dataset- specific methods. In fact, methods to improve the automatic generation of large, high-quality datasets for context-aided forecasting could complement CiK. Furthermore, this moti- vates research into developing more accurate and efficient multimodal forecasting models, which our benchmark is rell-positioned to support. Other avenues of research in- clude exploring different input/output structures for forecast- ing with LLMs, finetuning specialized LLMs for context- aided forecasting, allowing models to scale test-time compu- tation, and compressing the context to reduce the required amount of computation. Lastly, as models become more ro- bust, they could be integrated into agentic systems with con- versational interfaces, allowing forecasts to be augmented with human expertise and automatically retrieved informa- tion. Such advancements would represent significant step toward automating and democratizing access to powerful forecasting tools.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L.,, Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Aksu, T., Liu, C., Saha, A., Tan, S., Xiong. C., and Sahoo, D. Xforecast: Evaluating natural language explanations for time series forecasting. arXiv preprint arXiv:2410. 14180, 2024.
Alexandrov, A., Benidis, K., Bohlke-Schneider, M.,
Flunkert, V., Gasthaus, J., Januschowski, T., Maddix, D.C. C., Rangapuram, S., Salinas, D., J., Stella, L., Türkmen, A. C., and Wang. Y. GluonTS: Probabilistic and Neural Time Series Modeling in Python. Journal of

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
Machine Learning Research, 21(116)1166, 2020. URL http:///rrrggpppp//////... 820 h0mmm.1m1.5 55
Garza, A. and Mergenthaler-Canseco,,, M. Nixtla foundation- time-series-arena. https://github. cow/Nixtla/niii 1a/ la/tree/main/ wwta////e/eeetooo/ddddioiooooooo serie s- s-arena, 2024. 1
Allen, S., Ginsbourger, D., and Ziegel, J. Evaluating fore- casts for high-impact events using transformed kernel scores. SIAM/ASA Journal Uncertainty Quantifica- tion, 11(3):906-940) 2023. doi: 10.1137/22M1532114. 55
Garza, A., Challu, C., and Mergenthaler-Canseoo,, M. TimeGPT-1. arXiv preprint arXiv:2310.03589, 2023. 6, 54
Gneiting, T. and Raftery, A. E. Strictly proper scoring rules, predictoe,,,on, and estimation. Journal the the American statistical Association, 102(477):359-377 2007. 4, 8, 55
Ansari, A. F., Stella, L.. Turkmen, C., Zhang, X., Mercado, P., Shen, H., Shchur, O., Rangapuram, S. S., Arango, S. P., Kapoor, S., et al. Chronos: Learning the language of time series. arXiv preprint arXiv:2403.0785 05 2024. 6,
Gneiting, T. and Ranjan, R. Comparing density forecasts us- ing threshold- and quantile-weighted scoring rules. Jour- nal of Business & Economic Statistics, 29(3):411-42 2011. doi: 10. 908/..es.2210.0055 5, 55
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg. J., Bosse- lut, A., Brunskill, E., et al. On the opportumities and risks of foundation models. arXiv preprint arXiv:2 108.0588, 2021. 7
Godahewa, R., Bergmeir, C., Webb, G. I., Hyndman, R. J., and Montero-Manso, Monash time series forecasting archive. arXiv preprint arXhv:2105.06643 2021. 3, 14, 15
Box, G. E. P., Jenkins, G. M., Reinsel, G. C., and Ljung, G. M. Time series analysis: forecasting and control. John Wiley & Sons, fifth edition, 2015. 6
Gruver, N., Finzi, M., Qiu, S., and Wilson, A. G. Large language models are zero-shot time series forecasters. Advances in Neural Information Processing Systems, 36, 2024. 7,8
Chen, C., Petty, K., Skabardonis, A., Varaiya, P., and Jia, Z. Freeway performance measurement system: mining loop detector data. Transportation research record, 1748(1): 96-102, 2001. 3, 14,
Hyndman, R., Koehler, A. B., Ord, J. K., and Snyder, R. D. Forecasting with exponential smoothing: the state space approach. Springer Science & Business Media, 2008.
Chen, Z., Ma, M., Li, T., Wang, H., and Li, C. Long sequence time-series forecasting with deep learning: A survey, Information Fusion, 97:101819, 2023.
Hyndman, and Athanasopoulos, G. Forecasting; prin- ciples and practice. OTexts, 2018. 1, 17
Jia, F., Wang, K., Zheng, Y., Cao, D., and Liu, Y. Gpt4mts: Prompt-based large language model for multimodal time- series forecasting. Proceedings of the AAAI Confer- ence on Artificial Intelligence, 38(21):23443-2335 Mar. 2024. doi: 10.1609/aaai.v3388 v38i21............... 333221..33.11. 30383. URL /ttps:// jjs ww..aaii.org.. wwwpg/////e...///eeeeeee pttp://AAA//rriicl//eeeeee 8
Chow, W., Gardiner, L., Hallgrímsson, H. T., Xu, M. A., and Ren, S. Y. Towards time series reasoning with 1lms. arXiv preprint arXiv:2409. 1376, 2024. 8
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A. et al. The Llama 3 herd of models. arXiv preprint MoXiv:2407.21788 2024. 6, 48
Jiang, A. Q. Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., de las Casas. D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L. R., Saulnier, L.., Lachaux, M.-A., Stock, P., Subramanian, S., Yang. S., Antoniak, S., Scao, T. L., Gervet, T., Lavril, T., Wang. T., Lacroix, T., and Sayed, W. Mixtral of experts, 2024. URL https: /aaxiv..or g/ g/abs/248 04088.
Emami, P., Li, Z., Sinha, S., and Nguyen, T. Syscaps: Language interfaces for simulation surrogates of complex systems. arXiv preprint vrXiv:2405.19653 2024. 2, 4,
Gamella, L., Bühlmann, P., and Peters, The causal chambers: Real physical systems a testbed for AI methodology. arXiv preprint arXiv: 1aee:2444.144444 11311, 2024. 3, 14, 26
Jin, M., Wang, S., Ma, L.. Chu, Z., Zhang, J. Y., Shi, X.. Chen, P.-Y,, Liang, Y., Li, Y.-F., Pan, S., and Wen, Q. Time-LLM: Time series forecasting by reprogramming large language models. In The Twelfth International Conference on Learning Representations, 2024. URL hetps://openret wtt.:t//oorummuooonssorreea 1, 6,
Gardner Jr., F. S. Exponential smoothing: The state of the art. Journal of Forecasting, 4(1):1-28, 1985. doi: 100200000 URL https: / //onlinelibrary. www.yy.cmm//o//aa///r..e//////// 3980048188 6
8, 52
10

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
Kim, K., Tsaí, H., Sen, R., Das, A., Zhou, Z., Tanpure, A., Luo, M., and Yu, R. Multi-modal forecaster: Jointly predicting time series and textual data. arXiv preprint arXiv:2411.067 2024. 8
Sawhney, R., Wadhwa, A., Agarwal, S., and Shah, R. Fast: Financial news and tweet based time aware network for stock trading. In Proceedings of the 16th conference of the european chapter of the association for computational linguistics: main volume, pp. 2164-2175, 2021. 8
Kong, Y. Yang, Y. Wang, S., Liu, C., Liang, Y., Jin, M., Zohren, S., Pei, D., Liu, Y., and Wen, Q. Position: Em- powering time series reasoning with multimodal Ilms. arXiv preprint arXiv:2502.014777 2025. 8
Sengupta, M., Xie, Y., Lopez, A., Habte, A., Maclaurin, G., and Shelby, J. The national solar radiation data base (NSRDB). Renewable and sustainable energy reviews, 89:51-60, 2018. 3, 14
Liang, Y., Wen, H., Nie, Y., Jiang, Y., Jin, M., Song, D., Pan, S., and Wen, Q- Foundation models for time se- ries analysis: A tutorial and survey. arXiv preprint arXiv:2403. 14735, 2024. 1
Taillardat, M., Mestre, O., Zamo, M., and Naveau, P. Calibrated ensemble forecasts using quantile regression forests and ensemble model output statistics. Monthly Weather Review, 144(6):2375 - 2393, 2016. doi: 10.117 5/WWR-D-15-0260) 55, 56
Lim, B. and Zohren, S. Time-series forecasting with deep learning: survey. Philosophical Transactions of the Royal Society A, 379(2194::202002099 2021.
U.S. Bureau of Labor Statistics. Unemployment rate [vari- ous locations], 2024. URL https: //te// stlouisfed org/. Accessed on 2024-08-30, retrieved from FRED. 3, 15
Liu, H., Xu, S., Zhao, Z., Kong, L., Kamarthi, H., Sasanur, A. B., Sharma, M., Cui, J., Wen, Q.. Zhang, C., et al. Time-MMD: A new multi-domain multimodal dataset for time series analysis. arXiv preprint arXiv:2406.0882 086277 2024a. 2, 4,
Ville de Montréal. Interventions des pompiers de montréal, 2020. URL https:://wiit http://www.doneeeueeeeeeeee rche rche/dateae rchee/taraaeet//nt2--neeeeevvee http:/eessuueeeeeeeeeeeeeeee
Liu, M., Zhu, M., Wang, X., Ma. G., Yin, J., and Zheng, X. Echo-gl: Earnings calls- driven heterogeneous graph learning for stock movement prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- 38, pp. 13972-13980, 2024b. 8
service- ur ii-ite- incendie- Updated on 2024-09-12, accessed on 2024-09-13 3, 14
Wang. C., Qi, Q., Wang, J., Sun, H., Zhuang, Z., Wu, J., Zhang. I.., and Liao, J. Chattime: A unified multimodal time series foundation model bridging numerical and textual data. arXiv preprint arXiv:2412.11377 2024a. 8,
Liu, X., Hu, J., Li, Y., Diao, S., Liang, Y., Hooi, B., and
Zimmermann, R. Unitime: A language-empowered uni- fied model for cross-domain time series forecasting. In Proceedings of the ACM on Web Conference 2024, pp.
4095-4106, 2024c. 1, 6, 8, 9. 52
Wang, C., Qi. Q. Wang. J., Sun, H., Zhuang, Z., Wu, J., Zhang, L., and Liao, J. Chattime: A unified multimodal time series foundation model bridging numerical and textual data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 12694 12702, 2025.
Merrill, M. A., Tan, M., Gupta, V., Hartvigsen, T., and Al- thoff, T. Language models still struggle to zero-shot rea- son about time series. arXiv preprini arXiv:2404.1173
2024. 2, 4, 8
Potosnak, W., Challu, C., Goswami, M.. Wiliński, M., Žukowska, N., and Dubrawski, A. Implicit reason- ing in deep time series forecasting. arXiv preprint arXiv:2409. 10840, 2024. 8
Wang, P. On defining artificial intelligence. Journal of Artificial General Intelligence, 10(2):1-37, 2019.
Wang, X., Feng. M., Qiu, J., Gu, J., and Zhao, J. From news to forecast: Integrating event analysis in LLM- time series forecasting with reflection. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b. URL 2id=tj8nsfxiirr.
Rasul, K., Ashok, A., Williams, A. R., Khorasani, A., Adamopoulos, G., Bhagwatkar, R., Biloš, M., Ghonia, H., Hassen, N. V., Schneider, A., et al. Lag-Llamar Towards foundation models for time series forecasting. arXiv preprint arXiv:2310.0823
2023. 53
Requeima, J., Bronskill, J., Choi, D., Turner, R. E., and Duvenaud, D. LLM processes: Numerical predictive dis- tributions conditioned on natural language. arXiv preprint arXiv:2405. 2256, 2024. 6, 8, 50
Wei, J.. Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned lan- guage models are zero-shot learners. arXiv preprint 10X.:2209.0//522 2021. 48

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
Woo, G., Liu, C., Kumar, A., Xiong, C., Savarese, S., and Sahoo, D. Unified training of universal time series fore- casting transformers. arXiv preprint arXiv:2402.025592,
2024. 6, 53
Wu, W., Zhang, G., zheng, Wang, Y., and Qi, H. Dual- forecaster: A multimodal time series model integrating descriptive and predictive texts, 2025, URL https://op 8
enreview. net enreview.net/forumm// het/forumm1d::@F101s21100
Xu, Z., Bian, Y., Zhong, J., Wen, X., and Xu, Q. Beyond trend and periodicity: Guiding time series forecasting with textual cues. arXiv preprint arXi::2405.13552, 2024.
2, 8
Xue. H. and Salim, D. Promptcast: A new prompt- based learning paradigm for time series forecasting. IEEE Transactions Knowledge and Data Engineering, 2023.
Yang. A.. Yang, B., Hui, B., Zheng, B.. Yu, B., Zhou, C., Li C., Li, C., Liu, D., Huang, F,, et Qwen2 technical report. CoRR, 2024. 6
Ye, W., Zhang, Y., Yang. W., Tang, L., Cao, D., Cai, J., and Liu, Beyond forecasting: Compositional time series reasoning for end-to-end task execution. arXiv preprint arXiv:2410.040477 2024. 8
Zamo, M. and Naveau, P. Estimation of the continuous ranked probability score with limited information and applications to ensemble weather forecasts. Mathematical Geosciences, 50(2):209 234, 2018. doi: 10.1007/s110
04-017-9709-7. 55, 56
Zhang, W. Ye, J., Z., Li, J., and Tsung, Dualtime: A dual-adapter multimodal language model for time series representation. arXiv preprint wxXiv2446.00662) 2024.
Zhang, Y., Zhang, Y., Zheng, M., Chen, K., Gao, C., Ge, R., Teng, S., Jelloul, A., Rao, J., Guo, X., Fang, C.-W,, Zheng, Z. and Yang, Insight miner: A large-scale multimodal model for insight mining from time series. In NeurlPS 2023 Al for Science Workshop, 2023. URL https://opennreview.nn wtwp:/ee/oouum/id-eeeeeseduuud 2, 4,
Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., and Zhang, W. Informer: Beyond efficient transformer for long sequence time-series forecasting. In The Thirty- Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Virtual Conference, volume 35, pp. 11106-11115.
AAAI Press, 2021. 52
12

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
Appendix
Table of Contents
Additional Details the Benchmark
14
A.1 Data Sources
14
A.2 Task Creation Process
A.3 Model Capabilities...
16
A.4 Validating the Relevance of the Context
17
A.5 Weighting scheme for tasks
A.6 Standard enrors and average ranks
19
A.7 Task lengths
20
B Examples of tasks from the benchmark
21
B.I Task: Constrained Predictions
21
Task: Electrical Consumption Increase
B.3 Task: ATM Maintenance...
23
B.4 Task: Montreal Fire High Scason
24
B.5 Task: Solar Prediction
25
B.6 Task: Speed From Load
26
C Additional Results
26
C.I Extended results on all models
27
results partitioned by types of context
................................
28
C,3 Full results partitioned by model capabilities
29
C,4 Inference Time
30
C,5 Significant failures per model
................................
30
C.6 Testing the Statistical Significance of the Relevance of Context
................................
32
Cost API-hased models
................................
32
C88 Impact of Relevant and Irrelevant Information in Context
................................
33
C.9 Impact of Solely Irrelevant Information in Context
33
C.10 The effect of significant failures on aggregate performance models ................................
C.11 Visualizations of successful context-aware forecasts
36
C. Visualizations significant failures
D Implementation Details of Models
48
D.1 DIRECT PROMPT
48
D.2 L.MP
50
D.3 ChatTime
52
D.4 UniTime Time LLM
D.5 alag-l 1ama
53
D.6 Chronos
53
D.7 Moirai
53
D.8 TimeGEN
54
Exponential Smoothing
13

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
D.10 ETS and ARIMA
54
E Details of the proposed metric
54
E.1 Scaling for cross-task aggregation
55
E.2 CRPS and twCRPS
56
Estimating the CRPS using samples
56
E,4
Constraint-violatio. functions
57
Covariance two CRPS estimators
E.6 Comparison of Statistical Properties of Various Scoring Rules
58
A. Additional Details on the Benchmark
A.1. Data Sources
We list here the domains and the respective sources of time series data in the various tasks in the CiK benchmark. We also show the number of tasks that use each source's data and list any memorization mitigation strategies used for each dataset.
Traffic (11 tasks):
Traffic occupancy rate: We use traffic occupancy rate (%) data from the California Performance Measurement System (PeMS) (Chen al., 2001), with frequency hourly. This dataset contains total of 446 series. As this is live dataset (updated frequently), we use data from 2024 (i.e. data after the cutoff dates of LLMs used) and do not apply any memorization mitigation strategy.
Climatology (12 tasks):
Solar irradiance and cloud cover data (9 tasks): We use solar irradiance and cloud cover data for the Americas in 2022 (Sengupta et al., 2018), with frequency either 10 minutes or hourly. We extract subset of 45 series from this dataset for the benchmark.
To mitigate memorization, we shift the dates by day ahead.
Solar photovoltaic power production (3 tasks): Time series reflecting solar power production in Alabama during 2006 (Godahewa et al., 2021), with a frequency 10 minutes. This dataset contains a total of 137 series, but our tasks only use a single aggregated series generated from them.
To mitigate memorization, we add gaussian noise to the data with a standard deviation of 3% of the standard deviation of the data in each respective sampled window.
Public Safety (26 tasks):
Logs of number of interventions carried out by the Montreal Fire Depart- ment due to the occurence of various kinds of incidents (such as trash fires, field fires, nautical accidents, bike accidents) (Ville de Montréal, 2020). The data was processed from a raw log and aggregated to monthly frequency. This dataset contains a total of 48 series.
Fire Department Intervention Logs:
Due being processed, do not apply any special memorization mitigation strategy on top.
Mechanics (3 tasks):
Causal Chambers: Experimental data collected from the wind tunnel physical system from Gamella et al. (2024), released in April 2024. We make use of the load_in, pressure_des wdd.wwwidd, ressure_amb and speed_in series (downsampling them to frequency) to build out-of-distribution forecasting tasks where the target values can be inferred from the driver variate provided as covariate and the description of the physical system given in the context. We select subset of 17 series from this dataset for the benchmark.
* Since the data is released in 2024 and after the cutoff dates the LLMs used, we do not apply any memorization mitigation technique to transform the data.
14

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
Table Summary of Transformations Applied Tasks Per Domain
Domain
Number of tasks
Transformations
Date shift
Gaussian noise
Public Safety
26
None
None
Traffic
11
None
None
Mechanics
3
None
None
Economics
3
None
None
Synthetic
3
None
None
Total
46
None
None
Climatology
12
9
Energy
7
None
Retail
6
None
Total
25
9
16
Economics (3 tasks):
American unemployment data at the state and county levels, from the Federal Reserve Bank of St. Louis (U.S. Bureau of Labor Statistics, 2024), with frequency monthly. We extract subset of 1769 series from this dataset for the benchmark.
FRED:
As this is live dataset (updated frequently), we use data from 2024 (i.e. data after the cutoff dates of LLMS used) and do not apply any memorization mitigation strategy.
Retail (6 tasks):
NNS ATM cash withdrawals: The NN5 dataset of ATM cash withdrawals in the UK from the Monash Time Series Forecasting Repository (Godahewa et al., 2021), with frequency daily. This dataset contains a total of 111 series.
To mitigate memorization, we add gaussian noise to the data with a standard deviation of 3% of the standard deviation of the data in each respective sampled window.
Energy (7 tasks):
Electricity consumption: Electricity usage from 2012 to 2014 from the Monash Time Series Forecasting Repository (Godahewa et al., 2021), with frequency daily. This dataset contains total of 321 series.
To mitigate memorization, we add gaussian noise to the data with a standard deviation of 3% of the standard deviation of the data in each respective sampled window.
Synthetic Data (3 tasks): We employ bivariate setup where the parent variable is drawn from a categorical distribution, and the child variable generated using a continuous linear Structural Vector Autoregressive (SVAR) model with Gaussian noise, with a lag of and noise scale of 0.1.
Since this data synthetic, we do not apply any mitigation technique on top of data to mitigate memorization. Since models assume timestamp, use dates from 2025, and frequency of daily when input this data our models.
Depending on the task and the context used in the task, appropriate history and prediction lengths are used in the task.
summary of the number of tasks with either types of memorization strategy (shifting the dates by one day, or adding gaussian noise) is presented in Tab. 2.
A.2. Task Creation Process
All tasks were manually designed, from scratch, by the authors of this work without resorting to external annotators, crowdsourcing, or LLMss. We used the following procedure to create the tasks in the benchmark.
15

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
Task Diversity: First, we identified high-quality sources of public time series data from various application domains (Sec. 3.1). Special care taken find data sources that continuously updated to facilitate future benchmark updates. Second, we established the categorization for types of context (Sec. 3.2). Third, we posited various reasoning capabilities that one could potentially use to infer non-trivial information about numerical forecasts from contextual information (e.g., using common sense, making analogies to covariates, etc.; Appendix A.3). The task ideation process that followed aimed to ensure sufficient coverage of these three aspects.
Task Ideation: With this framework established, all authors contributed to the ideation of new tasks. summary, the process consisted of:
1. Selecting data source
2. Implementing time series window selection strategy (e.g., short or long history)
3. Brainstorming about the nature of contextual information that could help achieve better forecasts (c.g., information about the past) and the capabilities that might potentially serve to apply it to the forecasting problem.
4. Writing code to verbalize the context (e.g., calculating statistics of the series beyond the observed numerical history, creating template render such statistics text, etc.), and
5. Finally, required, writing code to modify the time series data to reflect the context (e.g., introducing some spikes in future values).
Then, the tasks were peer-reviewed by committee composed of all other authors (each with time series research experience). The creator of each task was not allowed to participate in the review. The review ensured that the contextual information was of high quality, that undoubtedly enabled a better forecast, and that the context types used in the task were tagged correctly. If task was deemed of not high enough quality, it was either returned for revisions or excluded from the benchmark.
Peer Review:
Code availability: The code for all tasks is available here: http::/gh.......oooo//////////// ting. An example task can be found here: https: //g/: thub. com/Servicens wttrriiiieo/////oeeeeeeeeee -oraaa--oorrattinn/bbb.o.o /main / /cik k benchmark tasks /oontt real. fire/short history-poe -y, where the time series window selection occurs from 1.94-112 and the context generation occurs from L1-4-5
A.3. Model Capabilities
As mentioned in Appendix A.2, we designed the tasks with consideration of the capabilities model might potentially use to incorporate contextual information into its forecasts. All tasks in CiK are tagged with such capabilities. However, these tags are inherently subjective and not intended as formal attributions. Rather, they serve as broad categories to help readers identify examples of interest within the benchmark. These are as follows:
Common-Sense (24 Tasks): Using direct instructions available in the context, Instructions could express constraints be satisfied, or the exact expected effect of an event, for example.
Retrieval: Retrieving facts from memory or context.
Retrieval from memory (35 Tasks): Retrieving from memory facts that enable interpretation of the context, such as relevant physical constants or quantitative laws.
Retrieval from context (25 Tasks): Retrieving relevant information from context and distinguishing it from irrelevant information.
Reasoning: Reasoning about information in context or memory.
Analogical Reasoning (6 tasks): Making analogies between entities events, for instance, applying knowledge from past event that is similar to an upcoming one.
Mathematical Reasoning (32 tasks): Performing calculations over the context, e.g. solving an equation.
Deductive Reasoning (39 tasks): Inferring new facts not explicitly mentioned in the context, e.g. inferring from the context that certain values are logically impossible to occur.
Causal Reasoning (22 tasks): Deriving or using causal information from the context to reason about actions (such as interventions).
16

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
Example: To illustrate the rationale, provide the following example. To solve the task in Figure 4, one could retrieve from memory that Montreal experiences snowfall and cold weather during the winter months. It could then infer that trash fires are less likely to occur during this period through deductive reasoning. This chain of thought reveals seasonal pattern that is not apparent in the short numerical history. Additionally. through causal reasoning, it is apparent that, despite strong association between field fires and trash fires, the intervention described in 5 is unlikely to reduce the frequency of the latter. Failure to recognize this distinction would lead to inaccurate forecasts.
A.4. Validating the Relevance of the Context
To evaluate the relevance of the contextual information for tasks in CiK, we query both human and LLM annotators on the relevance of the context. Our findings confirm that the context is relevant for all tasks (see Figure 8).
100%
Proportion of LLM Responses that Label the Context as Useful
95%
75%
0%
5
50%
95% 1005s
Proportion of Annotators that Label the Context Useful
Figure 8. Ratings of the relevance of the context for both human annotations (x-axis) and L.I.M annotations (y-axis). There are 5 ratings per task for the LLM, and between and 10 ratings per task for the human annotators. Each cell represents the number of tasks that correspond to given pair of (human, LLM) ratings for the relevance of the context. For example, 53 (top right) of the 71 tasks have over 95% of annotators tagging the context as useful, as well as more than 95% of the LLM annotations tagging the context as useful. All tasks considered relevant by either the LLM the human annotators. Furthermore, the majority of tasks are considered relevant across more than 95% of ratings.
Human Evaluation of the Relevance of Context To ensure that the context used in the tasks is relevant to the tasks, we ask 11 human annotators to evaluate the relevance of the context across 5 seeds. After brief presentation of two example tasks based on examples from (Hyndman & Athanasopoulos, 2018), we ask the annotators whether the context enables better forecasts. Annotators are instructed that tasks are designed for the purpose of context-aided forecasting, and that we are seeking to identify tasks for which the context not useful.
The results of this study can be found in Figure 9: the vast majority of tasks are always annotated useful, while annotators disagree small minority of tasks, such as Tetaaussallatttttttellllltttttto at prttps://sett http::////evviieeer/r/eo thub. io/coootetttiiii key- forecasting wttp/uullaallllootttttttmmm inSVAR (visualized nSVAR. html) likely due to the highly statistical nature of such task(s). Overall, 94.7% of annotations report that the context is useful.
An LLM-- Critique of the Relevance of Context To further validate the quality of the tasks, we build an LLM-based critique by prompting GPT-4c with the historical and future numerical data, as well as the context, and asking to assess
17

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
Task-wise Proportion of Annotators Who Find the Context Useful
ATMBuliddngC esekkakk
ATMUnderPeried Wantieddaainnnnnnaaaaaaaa waskwe/eettttttthcaaoo ChCocaai
ATMUndeeeerrosctteeee ntens nee bescWWinConncussoleee splict
ATMUnderreriodd Kaintenance IeseWittruut
Task
CashDapietedinnATiM MoMoSinnn Task
DecreaseinT Pafiiaarddiiiiiiiiioo Task
Difhusetorizenttlll licee Conorooausnnnn
Nomm Frod Clenniyy
Dreettermalimad aancafoom tuuus
Electricityl incresseeln
Blectrkiitynncceaae wwwpel/neeeiiiiiiiiiioo...
pectricitylncreeaee setrresseeiitee
Flectricltylncceessee
sogliattotvaeiiongeeaaaaimm ancerromiooutttaee ExplictD
Sygliothresurrrroe
Smailat ocation Day.eaaaaaaaaaaaaa
Explic FTafficaaaeaaaTakk hnfolldda pilnPredit WinnWinday
hwwphhoneeeennnyyyyytahhheeeeeee sthaskow ysiii pinneeditttnniindd
ExplcitWhhDays Traific ofeecc hiiloldds pinnredittonninnoo
CullCca usai EaxeeElettt Biva nSVVR
Enttassiiinneeetinglll aiciiiiauutttiiiiiiii nSVAl
ClobalHorzorta Iradianc ofromirnao
wtw.cttrrseeeernnsee.. portaas edTask
t trafiicorreastttsss hitolidd pinneeddtttniiind
increaseoiitee drawall
Local SaaiennttaarDo war.Spaafoocccsstta
LorgNowsElectiinn www.eeeeeecccciineeeee wwt.iiittlnnnneeeeee. Podictneiddettna Task
Med Mesteeeeeestiieeee httpe//wiiettcttyceeeeeeeeee
MirimaiC Cu Cortinat
nSVAR
MinimalinfoHa
Montrealf Firefiee attdeaas Exp
onfound
FruF FruFailAnddSar www.linnnaaaaaiinnnnnnooooooo
Tast
ng
MontraaiFieeFeisde Conv gnwoeimm
Exo otCausal
Taskkkk
MontrealFirefie
Contoune
Montreal FFreeFe ashiieed
Task
Montrea FireFieldnnd Neutral Tone
onfound nnghsk
Mantre Firefield splinil
Mo ntreal Fiel of impl olt ortt sstk
Montre ailtreieeeessuu uet oplic hrrt estoyy lask
Montreal ceses uolm pilii Sh ou Task
MontredFineouricalleeecc aleee lngyf
wint in syy Vate Tosk
ContteaaFieNautiiill www.caaaccccacaaaaaaaoaa. igell
Wate Taok
Montreal Il rekautic ca apiic hhott istory Tack
Montrea lirefaattic Hese
lack
Montrea wwweerrshwndBiyycceeeeeeeeoo Exo cit
Tack
Montreal railRrethsstnnn wawaaaanniiiddeeoeeoooeoo weleyyeeooninnnnnnnnnnn www.emmliiiiiiiiiiccccmmcm of
in Tack
MandovalFiiiTa Trask And wic.cillleuurrrallacocooo And Too
on Tusk
Montrealline fire Trast An--i-ddllleeee cycle Neutral
Montreal fire Trashhnnddannnnn
Convincing toretxp
Montreal www.rrasrrrraaaaaaaa www.//nnnnoonnniiiiioooo www.nnnoonoiiiiiiooomooo
Montrealf reirashhnndaaate Iraleeerral Exp
Fiu TwdTrachndNNaatiaaa /Nuutaa wmpppitaaaaaaaaa
Montrealfiveh selicittthortttii
Sensortia atcnanceinPeed cassskk
SensorPerodiick aintenance
Semort
Sensorfi umula
ShortWeeat www.re/eesctttyyyeeeseeeeeo caskk
Simllar, ocation Day. warforoooo batk
SimilartccatinWii
Purtirteeetttiiiiieeeeeeeeee RaferanceeayySS aeeeeeeaaaaaaeeeaaatttaa castaskk
Speedfromt acTiss
Unemp boutety Coutyiinnn ittaaaatttaeaaaa
Un unamploymett ne ConontyUssnggiiii Mws.ggutttttoo Ststtetaa
Unemp CommpplmmmnttCuaae wyy.sggiinggeeee. www.uggngggeetteeeeeee. tetatetta
Zerithinfeta toySouuffeeecatttttt
0.6
Preportion
Contest Useld
Figure 9. The task-wise proportion of annotators (n=11) who tag the context as useful. The overall rate of tasks tagged as useful across all annotators 94.7%.
18

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
Given the context, how would your prediction of the future values change? (5 seeds task)
30
20
10
Worse
Unchanged
Slightly
Significantly
Better
Better
Score
Figure 10. A histogram of results from the LLM based critique of the relevance of context. Given the historical data, the future data and the associated context of tasks, GPT-4o is asked assess whether its predictions would be "significantly better", "slightly better", "unchanged", or "worse" (see Appendix A.4 for the details). The context in all tasks is considered as enabling better forecasts, with the majority tasks having context that enable "significantly better" forecasts.
whether its estimation of future values would be "significantly better", "slightly better", "unchanged", or "worse" when the context provided compared to when is not provided. Note that this experiment was ran after the benchmark was created, as an analysis tool to further validate the quality of the tasks.
We this critique instances of each of the 71 tasks and report results in Figure 10. All tasks are assessed enabling better forecasts when given context, with the majority of tasks assessed having contexts that enable "significantly better" forecasts. The prompt used in the critique is in Figure 11.
A.5. Weighting scheme for tasks
To take full advantage of the available data, we create multiple tasks using each data source, by varying the specific contextual information we provide to the models. Since we do not want our aggregate results to be dominated by the few datasets for which there are larger number of tasks, we weight the contribution of each task to the various aggregated results.
To define the weight of each task, we first group the tasks clusters. These clusters are primarily defined based on the original data source used create the tasks. However, when tasks are fundamentally different, due not testing the same capabilities, put them in different clusters despite them using the same data source. For example, for tasks created using the Solar irradiance and cloud cover data, all of which ask models to forecast the irradiance, the tasks form three distinct clusters: one for tasks asking models to do forecast with very short history (less than day), one for tasks giving the cloud cover as covariate, and the final one for tasks where the models are given a tight upper bound on the possible irradiance. Once we define these clusters, we simply give equal weight to each cluster, and equal weight to each task inside each cluster.
A.6. Standard errors and average ranks
To get the standard errors shown in Tab. 1, we first compute the standard error for tasks using the method described in Appendix E.5. We then aggregate them according to each task weight, by assuming that errors for each are independent and thus using the formula for the variance weighted sum of independent variables.
To take into consideration the uncertainty have for the scores, we compute average ranks through a simple simulation. In
FFor Figure 8, we consider both "significantly better" and "slightly better" as meaning the context useful.
19

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
You are a critic whose role is to evaluate the quality of tasks in the "context is key" time series forecasting benchmark.
"Context Key" (C1K) time series forecasting benchmark pairs numerical with diverse types carefully crafted textual context, requiring models to integrate both modalities to arrive at accurate predictions.
Here task evaluate.
<hisstory>
(thistory))
</history>
<pontext>>
<background>
((background))
</backgrourd>
<scenarid>
((scenario))
</scenarid>
<constraints>
((constraints))
</oonstraints>
</context>
<future>
((future))
</future>
Assume the following two scenarios: 1) You are given only the numerical data in <history> and have additional information about the nature of the time series. You must ignore the <context> section completely.
2) You are given the <context> section in addition to the nunerical data in <history>.
Now, assume you had estimate the probability distribution of the <future> volues given the information available in each scenario. would the quality your estimation change scenario compared scenario 1?
First show reasoning in Creasont<<reasonn tags, then answer in <answer></answer, tags with either "significantly better". "slightly better", "unchanged", "worse" (no other ressonses allowed).
Figure 11. The prompt used query the L.M critique.
this simulation, we replace the RCRPS for each task and model pair by an independent Gaussian variable of mean equals the one we measured, and of standard deviation equals to the standard error. We then draw from this distribution and compute the weighted average ranks for each model. The results shown in Tab. the mean and standard deviation measured from 10,000 repetitions of this simulation.
A.7. Task lengths
Figure 12 provides an overview of the distribution of the lengths of the natural language context, numerical history and target (prediction horizon) for set of five instances for each task in the CiK benchmark.
20

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
140
120
Instance Court.
60
750 length (characters)
Hilstory length (fleats)
Target length (fleats)
Figure 12. Histograms depicting the distribution of lengths for the context, numerical history and target length set of five instances for each task in CiK. We measure the length of the natural language context in characters, and the numerical sequences in floats.
B. Examples of tasks from the benchmark
In this section, we feature multiple examples from the benchmark to exemplify exactly what a task is, what context types represent (Sec. 3.2), and how we tag these tasks with descriptive capabilities (Appendix A.3). To visualize tasks in the benchmark, refer the reader https: / sew.icenov.......e gi github tot. context-ettt-oooooooooeootoooeee is- fs-kky-ffee forcassttretttin//ve..
B.1. Task:
Constrained Predictions
Domain: Traffic
Context types: Future information
Context: "Suppose that in the forecast, the values are bounded above by 11.88, the values are bounded below by 7.06."
History Ground Truth Forecast
14
10
6-
2024-04-18
2024-04-19
2024-04-20
2024-04-21
2024-04-22
2024-04-23
2024-04-24
2024-04-25
This task, which we refer to as "Bounded Prediction Constraint Based On Prediction Quantiles", is forecasting task where we modify the forecast horizon (in green in the plot) by bounding one or both of its extremes according to its unmodified ground truth's quantile values. We verbalize these bounds in the context, and the model is expected to interpret and respect them.
Since we draw this series from the PeMS dataset (Chen al., 2001), we tag domain as "Traffic". The context directly
21

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
refers the future, hence the context source tagged as "Future information".
Since the context contains constraints, the Region of Interest CRPS metric that introduce (Sec. 4) heavily penalizes forecasts that exceed these constraints: models that do not incorporate the information about bounds in the context, such as quantitative forecasting models, would not be able to predict the ground truth (orange line) because its lower bound is much higher than that of the history. In this case, the region of interest for the metric is the entire forecast horizon because the context applies everywhere. Although statistical forecasters may pick up on the seasonality present the history (black line), they would obtain worse scores than models capable of processing the context and adjusting the lower bound of their predictions.
B.2. Task: Electrical Consumption Increase
Domain: Energy
Context types: Future information, Covariate information
Context: "This is the electricity consumption recorded in Kilowatt (kW) in city A. A heatwave struck the city, which began on 2012-10-09 18:00:00 and lasted for approximately hours, saw temperatures soar to unprecedented levels. According to the city's electricity provider, power consumption during the peak of the heatwave reached approximately 5 times the typical usage for this time of year."
History
16000
Ground Truth Forecast
14000
Region Interest
12000
10000
8000
6000
4000
2000
2012-10-04
2012-10-05
2012-10-06
2012-10-07-
2012-10-08-
2012-10-09-
2012-10-10-
The "Short News Electricity Increase" task introduces large shock in the forecast horizon that is only referred to in the context. Hence, the model must interpret the context appropriately to forecast the spike.
Since this series represents electricity consumption (Sec. 3.1), we tag a coming from the "Energy" domain. The context types for this task are twofold: the first context source is "Future information", which represents knowledge of the five-fold increase in typical usage during the shock. The second source context, "Covariate information".. represents the occurrence of a heatwave, which coincides with the timing and duration of the shock. The model must therefore interpret both the information the magnitude of the shock from the future information, well the timing and duration of the sock from the covariate information. Together, these pieces of information enable accurate forecast despite the lack of information about the shock in the task's numerical history.
In this task, we also see a "Region of Interest" (Rol), characterized by a darker region of the forecast horizon. This Rol represents the region of the forecast horizon for which the context is relevant, i.e. the period during which the increased power consumption occurred. As detailed in Sec. 4, this region of interest is taking into account in the RCRPS metric.
22

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
B.3. Task: ATM Maintenance
Domain:
Retail
Context types: Intemporal information, Covariate information
Context: "This is the number of cash withdrawals from automated teller machine (ATM) in an arbitrary location in England. The ATM was under maintenance for 7 days, periodically every 14 days, starting from 1996-1 100-11-300 1-30 00:00:00. Assume that the ATM will not be in maintenance in the future."
History
Ground Truth
40
Forecast
Region Interest
10
1996-12 12
1997-01-
1997-02
1997-03
1997-04
1997-05
The "Automated Teller Machine (ATM) Under Period Maintenance" task represents the history of withdrawals from ATM that undergoes regular maintenance. This maintenance introduces periodic, easily forecastable signal into the history. However, the context explicitly states that the forecast should assume the ATM will not be in maintenance during the forecast. Therefore, forecasting models are expected to ignore this signal.
Since this series represents ATM withdrawals, we tag as "Retail". The context includes information such as the location of the ATM, and therefore provides "Intemporal information". As the maintenance frequency and duration is also described, the context types include "Covariate information".
The Rol represents when the maintenance periods would have occurred in the forecast horizon, which is likely where forecasting models that do not leverage the context will forecast O. While quantitative forecasting model would find such signal irresistible, context-aware models should avoid repeating the pattern in the forecast.
We also note that the series is not quite 0 during the maintenance periods. This is a consequence of using one of our memorization mitigation schemes (Appendix A.I, paragraph "Memorization mitigation").
23

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
B.4. Task: Montreal Fire High Season
Domain: Public Safety
Context types: Intemporal information, Historical information
Context: "The Montreal Fire Department is in charge of responding to various kind of public safety incidents.
This is the number of field fire incidents responded to by Montreal firefighters in the borough of Rivière-des- Peatiis-PPonteeaxxe Tremmlles In other years, the yearly average number of incidents was 106 with the busiest month being June."
History
Ground Truth
17.5
Forecast
15.0
12.5
10.0
7.5
5.0
2.5
0.0
2022-01
2022-03-
2022-05
2022-07
2022-09
2022-11
The "Montreal Field Fire With Explicit Short History" task requires predicting the number of field fire incidents during the summer, so we tag it as being part of the "Public Safety" domain.
The context contains information from two different sources: it contains "Intemporal information". such as the location and nature of the incidents. However, it also contains "Historical information", which verbalizes statistics about past values of the series, beyond the numerical data. That is, the yearly average number of incidents, along with the knowledge that June the month with the most incidents.
24

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
B.5. Task: Solar Prediction
Domain:
Climatology
Context types: Intemporal information
"This series estimates the power production for given day of a new solar power plant located in the state of Georgia, which has climate similar Alabama's."
Context:
25
History
Ground Truth
Forecast
20
5
08-07
08-07
08-07 06
08-07 09
08-07
05-07
08-07
21
The "Explicit Similar Location and Day Solar Forecast" task requires forecasting the power production of solar power plant based on very short history and information about the similarity between its climate and that of an adjacent location. We therefore tag the domain of this series as "Climatology".
Without the "Intemporal information" that the context provides, accurate forecasts of the parabola-like shape of the ground truth are unlikely: the history contains very few defining characteristics, which makes it interchangeable with that many potential processes and therefore many possible forecasts.
25

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
B.6. Task: Speed From Load
Domain: Mechanics
Context types: Causal information, Intemporal information, Covariate information
Context: "The wind tunnel is chamber with one controllable fan that pushes air through it. We can control the load of the fan (corresponding to the duty cycle of the seree--iith-mmodlla
signal) and measure its speed (in revolutions per minute). The fan designed so its steady-state speed scales broadly linearly with the load. Unless completely powered off, the fan never operates below certain speed, corresponding effective load between 0.1 and 0.2. The task is to forecast the speed of the fan. The load is between 0 and 1. At full load (=1), the fan turns at a maximum speed of 3000 rpm. The load is set to: 0.0 until 05:47:09, 0.1 from 05:47:09 until 05:47:29, 0.0 from 05:47:29 until 05:48:01, 0.2 from 05:48:01 until 05:48:27, 0.1 from 05:48:27 until 05:48:49, 0.0 from 05:48:49 until 05:49:00."
minimum
History
650
Ground Truth Forecast
600
550
450
400
350
300
05:46:30
05:47:00
05:47:30
05:48:00
05:48:30
The "Speed From Load" task combines many different context types and capabilities produce forecast of the revolutions minute (RPM) of fan in wind tunnel based on its load. This task, based on the Causal Chambers dataset (Gamella et al., 2024), is therefore tagged as part of the "Mechanics" domain.
As the plot shows, producing an accurate forecast of the ground truth (orange line) from the numerical history alone (black line) is essentially impossible. However, the context the task quite rich: provides "Intemporal information" on the nature of the task, such the limits of the load and the fan, "Covariate information" that describes the load during the history and future, well as "Causal information" the control that the load exerts the fan, well as the proportionality of their relationship. With this information, it is possible to forecast the series nearly perfectly, excepting some noise.
C. A Adiiiional Results
The full unaggregated results from our benchmark can be found at the following location: https: :ttp::///gttee http//gtttttttteeeeeeeeteee com/ com/Service Now/ / cont e is key- forecast innccastinnnn htlgg//lb///ateee/ /main/minn////i / resul result is camplete, csv..
26

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
C.1. Extended results all models
Table Results all models in CiK benchmark aggregated over all tasks. The first column shows RCRPS averaged over tasks. The second column shows the rank of each method w.r.t. other baselines, averaged over all tasks. All averages are weighted according to the scheme described in Sec. 5.1 and accompanied by standard errors. Lower hetter and the hest means are hold.
Model
Avenge RCRPS
Average Rank
With Context
DIRECT FROMPS
Liang-3. 1 -4888-Inst
0.159 0.008
7,962 0.591
L2ans 3-00 Inst
0,256 0.004
14.906 + 0.200
L-ana-3-98
0,461
28.020 00,579
Moxtrol-B<78-In6:
Overr 2-5- 3-78tIn
0.250 #: 0.003
22 540 +: 0.751
0ven-2.5 200.-2.-11.8-2200 2nst
0,575 d: 0.014
32.643 1: 0.850
0.550
GPT-4o
0.274
GPT-tp-miist
0.354 + 0.022
17.574 + 0.498
LLMP
0.539
18:039 0.569
L2ama-3-740
0,727
11 11 -o--------mr Trat
0,483 + 0.000
18.597- + 0.471
Llbem-3-03 2-0
0,311 + 0.023
18.641) + 1.042
Maxcral Bx78-
0.264
Moceral-0278
0.262
Oven 5-20- 502-
1,974 + 0.027
45.227 + 0.747
Oven-2 3-78
0,910 :: 0.037
36 133 1.049
Qven-2
0ven-2.3-1 5-1.
1,731 0.036
0.525
Oven-2 3-9. 58-2not
1,937 + 0.024
40. + 0.611
0ven-2.3-0.56
1,995 0.024
48.511 0.823
Un1Time
0,370 0.001
35.459 0,155
T:me-LiM (ETTh1)
0,476 :: 0.001
44 086
: 149
ChacTinc-Sasc
0.735 0.002
39.033 0.312
Cha:Tine-Chat
0.005
0.369
Without Contest
DIRECT PROMPT (surs)
Lana-3 1-485B- Tost
0,473 :: 0.005
34,312 :: 0.299
Llaex-3--0 ---0--00---00 23--.
0.602
41.872 0.406
Moctral-Brr BBx778---
0,712 :: 0.021
46 505 :: 0.384
Swen2 4-78 Tixt
0.431 + 0.022
37 162 0.599
Oven-2. Genn.t
0.621
Qven-2 3-9. S0l-nnst
0,305 0.007
817 0.351
8PT-40
0,641 1: 0.008
31.511 1 0.380
GPT-40-midd
0,423 0.006
35.716
0.278
L1MP
1.ans-3-700-Imss
0.378 0.004
26 036 0.477
Liana-3-798
0,311 1: 0.006
21.319 1 0.472
Liana-3-B3-Inst
0.503 0.009
31.156 0.435
Liana-8-82
Mextrol-B<78-I105
0,353 0.015
21.55 595
0.471
Maxtral-B478
0,306 1: 0.007
22 565 1 0,514
0ven-2.8-78
1,020
0ven-2.5-78
Oven 52- Snstt
1,515 =: 0.033
47.008 :: 1,000
0ven-2.5-1.85
1,700 1: 0.028
41.194 1: 1.057
Qven
1,821 0.027
47.759 0.706
Multarudall Modela
UriTine (ETTh1)
0,405 :: 0.002
37.247 :: 0.178
0.488
43.019
ChacTinee
ChacTine-Cher Chat-
0,781 A: 0.015
35.260 J: 0,446
TB Foundation Modola
Lag
0.004
30.441 0.810
Chronns- Tiny
27.407
Chronoo- Mini
0.341 2: 0.001
28.855 J: 0.421
Cheonuxx Sewll
0.328 + 0.0022
26.533 + 0.368
0.672
30.601 0.379
Dheonox-1 arge
0.326
TLewGEN
0.353 + 0.000
35.92% + 0.167
Notrat-Gasti
0.565 :: 0.031
36 037 : 0, 426
0.406
Motrst-Large
0,520 L. 0.006
28.641 0.864
Linti Models
ARTMA
0.475 :: 0.006
041 :: 0
Exp Exp-Snowiiing
0,605
Tab. provides the extended results with all models evaluated on the CiK benchmark, aggregated over tasks.
27

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
C.2. Full results partitioned by types of context
Table 4. Results of all models on the CiK benchmark. Starting from the left, the first column shows the RCRPS averaged over all tasks. The second column shows the rank of each method w.r.t. other baselines, averaged over all tasks. The remaining columns show the average RCRPS stratified by types of context. All averages weighted according the scheme described in Sec. 5.1 and accompanied by standard errors. Lower is better and the best averages are in bold.
Modd
Average
Average
Interpoccal Infoemation
Hisooiical Information
Futrre Infocmaatio
Conarizze Informaticn
Cuusal Information
RCRPS
Rank
With Contest
01159 0.008
0.587
I.174
0.001
0,775 0
0,164 0,010
0,308
TTnna
0.004
0.000
0:228 0.004
-Lano 88 Inac
0.461
0.000
28.013
0.588
0,572 0.011
0.253
0.017
0.262 0,002
0,531
Mixt o1: 3x78- Inst
0.523
0.023
33.058
2: 0.415
0.223 = 0.037
0.256
0.002
0.241 2: 0.001
0.359 2: 0,02
0,875
0.125
owen-22 -5--ttt-- Lins
0.200 = 0.003
22:842 :: 0.735
02200 0.004
0.176
0.003
0.287 = 0.007
0.240 :: 0.002
0.525 0.003
Owen-2 5-1 TR-Tast
0.575 = 0.014
32.642
: 0.864
0.64 0.025
0.264
0100
0.370 - 0.010
0.450 + 0.004
1,270 + 0:009
Oven 2.5 0. Inat
0.463
0.012
27.218
:: 0.530
06699 = 0.019
0.165
0.004
0,218 :: 0.012
0.476 2: 0.015
0.429 0:005
GPI-do
0.274 0.010
8.651
0.443
02218 = 0.007
0
0.001
0.121
= 0.001
0:250
6 0.011
0.858 0.053
GPT-do-wint
0.354 0.022
0.506
0.475
0.00
0,143
0.002
0,541 1 0,02
0,646 0.128
0.539 0.013
18.034 0.571
0.438
0.017
0.516
0.028
0.847
0.024
0.546 0,016
0,392
0.006
0.716
02212
0.461 0.000
18.997 + 0.452
0.476 0.013
0.161
0.006
0.326
0.000
0.304 : 0.008
0.878
_Jano- 3- 56
0.311 =: 0.023
18.677 :: 1,000
01352
0.035
0.123
0.0M
0.271 :: 0.010
0.288 1: 0,029
0,739
0.134
Maxt-all (octt-a--xx7---- Sx-88-1tt
0.264 = 0.004
16005 :: 0.667
0.242 = 0.007
0.173
0.004
0.324 = 0.005
0.219 2: 0.005
0.437
0.007
0.763 + 0.008
16262- + 0.833
[1.200 - 0.038
0.119
0.110 + 0.019
0.229 + 0.006
0.457
0.011
0ven-2.5-78 2.5-78 Inst
1,974 =: 0.027
45,235
:: 0,742
2,509 = 0.044
2,457
0.056
LA51 :: 0.008
1.702 à: 0.035
1,333 :: 0.344
own--2 1--1
0,910 :: 0.057
38,144 :: 1.039
1,10 = 0.047
1,00 0.53
0.601 : 0.071
0.639 : 0.047
0.928
0.129
Gwen-2.5-1 2051
0.866
2614
1,433 0.029
2.057 0.033
2.448
Owen-7
0.519
0.943
0eon-2.5:8 Inss
1.37 0.24
50.93 à 0.002
1960 0,03
1 0.010
1.805 0,030
Owen-2 5-3 30
1.995 0.024
48.499
:: 0.834
2.546 = 0.039
2,083
0.052
1.579 0.015
1.821 4 0,030
1,225 0.128
Miltsmoda: Podele
UntTiiee
0.330
0.001
35-456 a: 0.152
0.45
0.002
0.155
0.000
0.194 0.003
0.99 a: 0.001
0,425 0:001
Tame-LUM (ETTh1)
0.476 = 0.001
44.057
:: 0.148
0.518
= 0.002
0.183
0.000
0
= 0.002
0.44]
#: 0.001
0,482 a: 0.001
Char Baree
0.785
0.002
99
+ 0.811
0663
0.002
0.181
0.001
0.374
+ 0.000
0.794 + 0.000
2.727 00003
ChesTine thit
0,247 0.005
34.150 à: 0.391
0.693
0.007
0,405
0.033
0,347
: 0.007
0.532 à: 0,005
2,971 0.015
Without Context
DOnECT rRoser (ouss)
-.am--1 1-4850-1100
0.473
0.005
34:334 + 0.256
0.522
0.007
0.713 +113+:10 0.014
0.393 = 0.000
0,320 + 0.002
0.587
0.005
_Jano-3
0,470
0.008
35.150 0.356
0.532
0.389
0.00
0.317 0.002
0,615
3-58-2nat
0.602
0.006
0.404
0.748
0.008
0.335
0.000
Maxt soctrol:-ee 8x78- Inat
0.712 0.021
46,907
0.389
11906= 0025
0.55
0.15
0,400 0.001
0.485 1 0.022
0,893
0.125
Owen-2 m-1---
0.421 0.022
37.155
0.547
0.479 0035
0.115
0017
0:322
0.008
0:357 + 0.028
0,200
+ 0.128
Over-2. 5-1. re-Inst
0.450 0.007
362
0.615
0.994 = 0011
0.324
0.00
0.360
0.008
0.315 1: 0.000
0999 : 0000>
owen-22 5-5.: 38 28-2
0,305
0.007
26.81 :: 0.353
0.241
= 0.07
0.185
0.004
0.236
= 0.016
0.255 1: 0.005
0396
:: 0.002
0.441
0.008
31. 10.505-11
+ 0.887
0.492
0013
0.200
0007
0.376
- 0.002
0.276
+ 0.001
0.504
+ 0.002
OPT 40--0---0 wind
0.006 0,423
35.711 à: 0.275
0.490 0.009
0.391
0.007
0,338 = 0.004
0.280
à: 0,001
0,531 a: 0.003
LLMP
0.375 =: 0.004
26,030 + 0.469
0.405
0.006
0.186
0.004
0.353 =: 0.004
0:253 + 0.002
0.481 0.004
-1ano-3-78
0,311 0.006
21:812
0.410
0.311
0.003
0.0M
0,321 0.018
0,245 0.002
0,419 0:005
00:000
0.500
0:266
_Jano 58
0.000
25.053
à: 0,375
0,357
0.004
0.007
0.250 0.001
0,491
Maxt -akt-aal////Iiet
0.383 0.015
24.557
= 0.471
= 0.024
0.08
0.340 0.004
0.349 :: 0.019
0.470
Mixt -11-s77
0.306 0.007
22,567 a: 0,516
(129%
8006
0.100
0.3%
0.021
0,242 à: 0.001
0,489
0:003
0ven-2.5-70 Irst
1,020 = 0.026
39:951 :: 1,232
1.435
= 0041
0,389
0.032
0 =: 0.018
0,512 1: 0,032
0,810
:: 0.125
Owen-2 5-m
0.782 = 0.050
38,091
+ 1,150
0.923
0045
0.00
0.04
0.441 = 0.020
0.545 + 0.034
0.792
0.128
own--2.5- 3-1 20-2nnt
1.515 = 0.033
47.055 + 1.063
21088
0.047
0.607
0.038
0.971
= 0.050
1.300 :: 0.041
0.926
0.125
Ownn-2 5-1 10
1,000 0.028
41
1,055
1.296
0.046
0.19
0.650 0.011
0,855 0.0%
0,785
0.128
2.5--2.5...12 Ins1
1.515 0.033
1.063
2.10
0.050
1,300
à 0.041
0,926
Owen-2
0.027
0.704
2253
1,644 0.024
1.642 0.034
Podele
UniTlre
0.002
37.250 à: 0.178
0.330
0.003
0,384 à 0.002
0,443
iime-LiM (t: (111)
0.45
0.000
43.016
0.164
0.467
0.000
0.406
0.004
0.499 :: 0.001
0.465
CharTi we-Bane
0.725
0.00
36
0.35%
0.688
0003
0.171
0.00
0.367
0.00
0.083
e 0.000
2.719
ChatTave Shick
0,781
0.015
35:267
:: 0,436
0.241 = 0.024
0.150
0.001
0,425
:: 0.035
0,791 1: 0.001
2,592 1: 0,000
T5 Fourdation modola
-ag- Ties
0.323 = 0.004
30:446 + 0.818
0.330 - (.005
0.167
0.293 = 0.000
0.294 + 0.004
0.405
0.014
Chronos- Tany
0.328 :: 0.001
21.895
:: 0.440
1,300
00002
0.163
0,393 :: 0.002
0.264 :: 0,002
0,486
0.003
Chronos-mtnn
0.341 0.001
268897
0.415
0:318
0.171 0.00
0.40)
0.002
0:272 0.002
0.481
00004
Swall
0.328 0.002
0.371
0.308
0.393 0.000
0.257
0.002
0.453
01673 0.003
30.992
0,510
0.392 0.00
0.697
0.000
2.481
Chronos-..rgg
0.326 0.002
250001
0.401
01314:
0.379 0.00
0.255
0,000
0,460
0.353 0.000
35.923
:: 0.168
0.332
0.000
0,405 0.000
0.292 à 0.000
0,474
M 1
0.565 0.01
36
14048
0.418
0.60
0.000
0010
0.4%
0.00
0.558 + 0.040
0.464
0.013
Mosrat-base
0.624 = 0.013
35.361
+ 0.410
0.629 = 0.021
0.172 0.00
0.390
= 0.004
0.630
# 0.017
0.486
0.015
Motrot-Tang -Tanee
0,520 : 0.006
28.666
3: 0,600
11596
0.009
0.140
0.001
0.431
:: 0.002
0.499 8: 0,000
0,438 a 0.011
Statiotio:1 Models
ARLIA
0.475 =: 0.006
27,039 :: 0.483
n.557
0.009
0.200
0.607
0.350 :: 0.000
0,375 4: 0.006
0,440
0.011
FTS
0.590-000 0.000
33/766
+ 0.615
0.60
0014
0.362 0014
0.315 0.006
0:402 + 0.110
0.508 0.17
Sroothing
0,605 0.013
36.426
0.549
0,702
0,397 0.006
0,480 0,015
0,827 0,000
Tab. 4 provides the results of all tested models, partitioned by the types of context.
28

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
C.3. Full results partitioned by model capabilities
We provide an additional view of the results of all models in Tab. 5, partitioned by model capabilities.
Table 5. Results of all models on the CiK benchmark. Starting from the left, the first column shows the RCRPS averaged over all tasks. The second column shows the rank of each method w.r.t. other baselines, averaged over all tasks. The remaining columns show the average RCRPS stratified by model capabilities. All averages are weighted according the scheme described in Sec. 5.1 and accompanied by standard errors. Lower is better and the best averages are in hold.
Model
Averge
RCRPS
Average
Rank
Sosman-Sesse
From Contest
Retieval
From Mensry
Deluctive
Analogical
Reasoning
Mathenatical
Causal
Context
DIRECT PaoMrT (ours)
Llann-3: 1-4808-00 Test
0.889:00000 0.008
7,971
0.585
0.140
0.013
0.109:1:0000
0.191 0.006
0.132 :: 0.001
0.167 : 0.008
0,316 1: 0.028
0.376 1: 0.009
1 11ww-22200--1 tritt
0:286++00 0.014
14,802
0.203
0.323 0.005
0.122 0.001
0.408 +0.02
0.168 + 0.000
0.492 - -0.019
0.473+ +0012
0.577 + +-0.077
Liono- 3-88 38-I
0.461 1: 0.0000
28,016
1: 0.581
0,325 a 0.010
0,374 1: 10000
0249 1 1.0221
0.407 1: 0,0014
1.245 1 0.009
0.437 1: 0.001
0.494 1 1,000
Mixtrt1-8e78--1 Inst
0.525 0.023
33,069
0.413
0.433 0.043
0.204 0.000
0.64 0.029
0.426 0.004
1,245 0.006
0.644 0.080
0,789 0:1112
Owen-2
0:200-00000
22.852
0,343
0.127
0.324
0.205 0.006
0.400 0.002
0.575
32.631
0.361
0.60=
0.314-00000
0.88 0.04
0.809
1.077 0.122
0.896
1.151 + 0.008
Oven-2. 3--.58-Innst
0.463 0.012
27,231 1: 0.333
0.267 0.008
1,29 1 0.055
0.764 0.009
0.244 1: 0.0007
2,043 1 0.104
0.330 1 0.001
0.392 1 1:000
OPT-4e
0:224::00000 0.010
8.640
= 0.436
0.179 :: 0.044
0 : 0.003
0.519 0:029
0 :: 0.006
0.447
= 0.009
0.590 +: 0.03
0,709
+ 0.046
GPT--o-estes
0.354 0.022
17,573 + 0.505
0.296 + 0.043
0.419 0.014
0.471 + 0.012
0.219 + 0.005
1.024 0.083
0.475 + 0.090
0.578 + +.111
L1MP
0.530=00.013
18.042
0.641 0.01
0:284
0.015
0.392 + 0.014
0.495 0.025
0,312
0.019
0.453
0.020
0.495 0.008
0.236 0.006
12.377
0,309 0.011
0.126
0.009
0217 0.007
0.152 0.003
0.241 0.019
0.2944
0.008
0.329 à 0,010
0483-=0000 0.010
18.585
0.477
0,345 0.00
0:138 0.004
0810 0.030
0.242 :: 0.008
1,278 0.069
0.617
0.023
0.777 1 0.000
I. 11mm-22-m
0.311 + 0.023
18.634
1.090
0.403 0.043
0.124 0.001
0.280 + 0.026
0.177 + 0.014
0.267 - 0.015
0.530
0.084
0.661 + 0.11
Moctral-Ueett Ln
0.264 = 0.004
16.07%
0.666
0,344 = 0.004
0.127 0.003
11224 0.025
0.179 +: 0.010
0.173 = 0,009
0,348
0.005
0.405 + 0.007
Maxtral-Berk
0.006
16,302
0,348
0,112
1.230
0.153 0.002
0,230 0.041
0.414 0,009
(wwn-7
1.924
45,211
0.048
1/02
2.215
2.758
1,733 0.002
1,677
(bwn-2. 5-20
0:010=0.01 0.037
38.157
1.041
0,691 0.063
0.7944 0.083
1.558 0.042
0393 0.028
2,538
0.155
0.878 0.084
0881 + 0.113
2.158 0.127
50,654
0.863
2,056 =: 0.846
1.566 :: 0.113
2.671 0.038
2.165 :: 0.135
3,635 = 11.055
2,480 :: 0.085
2.323 : 0.113
Owen-7 1o1 1n
1.731 + 0.0%
45.118
0 528
1.343 0.61
1.737 + 0.074
2.594 0.042
2:256 + 0.042
3,275 0. 0.182
2.086
0.033
1.536 + 0. 114
Down-7 3-40 30-Inst
1.37
0.024
50,482 0.62
1,740 + 0.043
1 1.500 0.021
2,193 + 0.025
2:305 + 0.078
3,439
= 0.004
1.485 + 0.084
1.398 + 0.111
Uwen-2.2.0.00
0.124
1,529
0,125
1.967
2,009 0.023
1,804 0.086
1.443 0,113
Multieedee
Oni
0
35.453
3,002
0.379 0.001
11321 0.001
0.511 à 0.003
0.337
0.001
0.333
0.001
0.385 à 0:001
Time-LLM (ETTh13
0 0.001
44,084
0.150
0,448 a: 0.002
0.192 1: 0.000
0L373 0.000
0.338 1: 0.003
0.397
= 0.000
0 4: 0.001
0.440 1: 0,001
ChalT:nee-aa
0.735 0.002
50,033
0.312
0.543 0.802
0.215 0.002
1,990 + 0.002
0.263
+ 0.004
0.374 = 0.004
1.788 0.002
2.407
+ 0.002
Chatl 1ame chat
0.747 0.005
34,182 0.389
0.525 e: 0.004
0.299
0.020
1.198 à: 0.015
0.315 :: 0.012
0.277 0.003
2,015 à 0.016
2.691 +: 0,022
Without Context
L 1-4808 Inst
0.473
0,393
0:325:1:
0.752 0.015
0.494
0.720 0.007
0.594
0.67
0.420
35.143
0.357
0.404 0.00
0.304 0.000
0.117 +0015
0488
0.012
0.694
0.004
0
0.000
0.61 + 00007
L-ana----e 3-88--88888 Ii-t
0,002 1: 0.006
41.875
0.395
0,390 1: 0.005
0.322 1. 0.000
1,123 1 0.018
0.643
1: 0.002
1,446 = -00005
0,581 1 0.008
0.617 1 1,000
M5xtrcl-Bw78 B--0----
0,712 0.021
46,999
0.375
0.624 1 0.043
0.324 1 0.006
1.553 1 0.007
0.763 1: 0.005
1,237 1 0.007
03858 1. 0.000
0,872 1 1.1112
Oven-2. 1-28-Irot
0421= 0.022
57.154
0.546
0.447 0.043
0:259 0.006
0.505 0.00
0:375 0.007
0.41
0.006
0662
0.000
0,792 0.111
Owen-7
0450-=000 0.007
16.308
0.377
0:232
0:377 0.013
0.000
0.476 0.008
Oven-2, B-C. IB-nst
0.007
26,819
0.002110000
0
0.300
0,440 1 0.015
0,315
6002
0.367
1.000
GPT-40
0.411 0.000
31,507
0.385
0.381 0.002
0.179:::000 0.000
0 0.028
0:357 : 0.007
0,953
0.067
0.422
0.00
0.471
1 0.002
6PP-4o-estet
0.233-2 + 0.006
55.715 + 0.275
0.359 + 0.003
0.214 + 0.002
0.640 + 0.019
0.391 + 0.008
0,771 + 0.041
0.461 + 0.00
0.511 + 0.003
LLMP
0.378 + 0.004
26,031
0.474
0.368 0.833
0.150 + 0.001
0.513 + 0.012
(1292 + 0.006
0.668 0.025
0.384 0.601
0.440 + 0.033
Llare
0.006
21,810
0.002
11351
0.215 0.004
0.017
0.434 0.005
Llana-3-88- Te-T
0.503
31,156
0.41 0.010
0.000
0.467 0:001
0.345-0000 0.00
25.067
0.326 0.044
0.150
0.497 0.009
0.266 0.005
0.640
0.017
0 0.00
0.44 + 0.002
0.383 = 0.1115
24.512 0.471
= 0.003
0.550 = 0.112
11450 +: 0.947
02111 + 0.008
1.027 = 0.135
(137] + 0.004
0.42 + 0.004
Mixtral-Be70
0,305 0.007
22,560
:: 0.512
0.360 2 0.013
0.345 0
IL327 2: 0.038
0.212 :: 0.005
0,340 = 0.016
IL382 2 0.002
0.445 4 0,003
Owen-7 1-28-Trnt
1,200 0.036
50 50.922
1.333
0.521 + 0.045
1.157 0.022
163441 0.017
1.06 + 0.014
3,300 - 0.087
03853 + 0.01
0,790 + 0. 0.112
0.732= 0.030
0.045
0.751
1.333 0.09
0,730 0.112
1.515 0.033
47,071 1.066
1,316 0.555
1.159 0.157
0.031
16652 0.042
3,583 0.300
1,108 0.082
0.448 4 0.112
Oven-2 18
1:020 0.028
41.187
1.53
1,005 0.43
0:287: 0.036
1.350 0.023
1:264 : 0.040
1,28 0.647
0,771 0.066
0.730 + 0. 112
(wwn-) 2-0. 10-Inst
1.318 0.057
44.303
0.750
1644
0.064
0:239 0.0331
1.192 0.019
1453 + 0.047
1.675
= 0.072
0.0900 + 0.082
0.743
+ 0.112
Uben-2. 2-40 20
0.027 1.21
47,763
0.715
1,705 :: 0.645
0.572
0.040
1.722 à: 0.033
2.498 :: 0.136
1.671 0.064
1.492 2: 0.083
1.113 + 0,114
Multinedel Modele
Uni
0,455
0.002
37.248
0.177
0 0.092
0.366 0.001
1.319 0.001
0.496 à 0.005
0.314
0.002
0.352
0.002
0.409 à 0.003
TLM (ETTM1)
0.002
43,014
0,440
0.003
0.000
0,371
0.499
0.002
0.001
0,368
0.001
0.424
1 0:001
ChacTaac-Bose
0.725
0.002
38.771
0.351
0,537
0.002
0.205
0.000
1,000 0.001
0.250
1: 0.004
0,365
0.005
1,779 0.000
2.396
1 0.005
Chatl Lome che-
0
= 0.015
35.260
= 0.446
0.365 = 0.022
0.165
= 0.002
1 4 0.036
0.282 # 0.020
0,445
= 0.064
1.5%6
0.000
2.536 # 0.000
T5 Foundation Podelo
Lag Llana i
0.327 :: 0.004
30,451 :: 0.319
0.353 d: 0.807
0.181 :: 0.003
IL324 1: 0.003
0.269 d: 0.006
0.342 =: 0.006
1.386 2: 0.009)
0.449 : 0.012
Chronco
0,328 0.001
27.487
0,400 0.002
0.344 0.000
0.297
0.229
0.002
0.290
0.005
0,382
0.440 0.005
Ghroncs-WW Monz
0.000
28,893
0,412 0.002
0.147
0.000
0,302
0.000
0.255
0.002
0.508
0.004
0,378 0.000
0.436 1. 0.000
Chronce- Sno.1
0.328 0.000
26,504
0.372
0.288 : 0.00
0,14 1: 1000
0287
0.000
0.248 :: 0.003
0.250 : -0003
0.358
0.005
0.412 1 0.006
Chronca Busa
0.672 :: 0.003
30,001
:: 0.375
00712 2: 0.002
0.145 -: 01002
1.223 1 0.00
0.261 1: 0.003
0.259 = 00000
1.643 1: 0.009
2.187 1 0.012
Longge
0.326 2: 0.002
23, 25,602
0.399
0385 1 0.002
0.158
1 0.002
11288 1 0.00
0.259 1 0.002
0.296 - 0,003
0.362 1. 0.000
0.417 1 0.004
TsmedaN
0.353 0.000
35, $5.92411
0. 0.167
0.61 0.000
0.178 0.000
0.508 0.000
0.278
0.000
0.324 0.000
0.377
0.000
0.431 1 0.000
Moirai-awal:
0.565 0.131
36,03%
0.429 0.005
0.146
11468 0.776
0.566
0.017
1,204
0.271
0.426 1 0.012
Modras Coze
0.624 0.013
35,263
0,410 0.005
0,600
0.053
0.680
0.029
0.690
1 0.019
1,147
0.099
0,375
0.010
0.441
1 0:013
Motrot-large
0.520 L 0.006
0.362 28.635
0,414 1L 0.004
0.155
1. 0.004
0.260 1 0.000
0.751
1. 0.015
0.008 0,276
0.337
1 0.000
0.397 1 0,0000
Stotioticol Podelo
ARIMA
0.475 L: 0.006
27,047 a 0-485
0,395 a: 0.005
0.160 L. 0.000
0,512 1. 0.012
0.513 1 0.012
0,206 : 0.008
0.354 1. 0.000
0 1 0,010
E78
0,530 0.009
33,786
0.635
0.48 + 0 0009
0:226 0.010
0.682: + 0.18
0.577 + 0.019
0.855 0.085
0.453 + 0012
0.479 +.015
Eap-Sreething
0.013
36.435
0.569
0.013
0.743
0.563 0.016
0.005
0.72 0.053
29

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
C.4. Inference Time
10a
Direct Prompt
LLM Process
102
Runtime (seconds)
Quantitative
Multimodal
10a
No Context
100
10-1
10-2
uniting Dartiin Exp smaothin Mariel Chroges Khr Chigges Chrgtaaa
Chrones El
Timee ad
Chalfinn Chnte
Liama 3-708 iast
Chise
Chat
Lame Latt
Figure 13, Inference time in seconds, for all baselines, averaged over all tasks. Several quantitative methods are much faster average than LLs-based methods. However, there are significant differences in inference time between the LLM-based forecasters: for the Lama models, LLMP takes about an order of magnitude more time to run average than DIRECT PROMPT.
Figure 13 provides the inference time of all tested models on the benchmark. Note that these values have not been normalized based on the computing resources made available each model during inference; please refer to Appendix D for information about how much compute resources were allocated to each of them.
C.5. Significant failures per model
We observe that in few instances in the benchmark, some models tend obtain significantly worse performance when evaluated with context. In our evaluation, we term all instances where the RCRPS value of model is greater than 5, as significant failures of the model on those instances. We found 5 as a suitable value for analyzing such failures, as it intuitively represents the value forecast would get if the distance between the forecast and the ground-truth was 5 times bigger than the range of the ground-truth for the task. When we aggregate the RCRPS of instances in the benchmark (such as in Tab. 1), we cap the RCRPS of such significant failures to 5, to avoid outliers with a much higher RCRPS affecting the aggregate score. In Tab. 6, we show the number of such instances in our evaluation of the benchmark where we found models to have significant failures (out of total of 355 evaluated instances). Interestingly, some models such as DIRECT PROMPT with Llama-3. 1-485B-Instruc and LLMP with Llama-3-700 and Llama-3-ge 8B are more robust such significant failures, and do not incur such failures. On the other hand, models such as Qwen family of models (that are notably significantly smaller than the rest) with LLMP achieve the most significant failures, followed by Llama-3- 3------------t--- with LLMP. We postulate that this is because of models misinterpreting context. It is still an open question as to how to increase the robustness of models to prevent or reduce such significant failures. We visualize such significant failures in Appendix 12
and LLama-3-an 3-88-Innstruct
30

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
Table 6. Number of instances with significant failures in models that support context
Model
Number of instances with significant failures
With Context
PROMPP (ours)
Liana 3.1-486B- Inst
0
1Laro-3-788-Ind
Liana 88 Inat
Mixtral-8xCB-Ing
Over-2: Over-2.5-7e 5--8--Inst
2. 9B--B--n Instt
10
Over-2.
(P1-4o
5
LLMP
Llama-3-/AM-Inge
18
Llara-2-89-Ins:
12
Mixtral 8x78 Inst
Mixtral-8x28
0
Ower 2.5
107
2vor-2.5-75
27
Over 2..--2.5 2.5-1158
99
100
0ver-2.5-0.30
109
Multimodal Models
Tine-LLy (ETTh1)
ShatTinn-Base ShatTone-Chat
Without Context
PROMPT (ours)
3.1-0-- Instt
9
708
Llana-3-88 Inat
8
Mixtral-BxCB-Ing
14
Oven-2 b-/8-Inst
2ver-2.5-1, 5B.-IIIn
Ower-2,
GPT-40
GPT-40-minn
LLMP
11ara-3-790-111
Liara-3-78B
Llara-3-88--1mg
Nixtral-8x78 Inst
Nixtrwl-RR28
0
Swen 5:78
Qver-2.5-78
15
1.58
52
Qver-2.5-1.5B
36
Over-2.5-0. 5.-.2----.
42
Over-2.b- 2.9.0.00
Nultivodal Modols
Time LLV (ETTa1)
ChatTAnn--Bsee Chatltme--t Chme-Chat
23
TB Foundation Mockls
Log-Llana
Chronos-Tary tary
Shrenos-Srell
Chronos-BBane
18
Chronos Large
FameGEM
Small
rodraal Base
Noirai-Large
Statiatica" Modela
ARIMA
Exp Sxoothiing
31

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
C.6. Testing the Statistical Significance of the Relevance of Context
Table 7. p-value of the one-sided paired (-test between the RCPRS values with and without context for models who can use it. Since this test is done on unweighted RCRPS values, the average RCPRS presented table are also unweighted.
Average RCRPS
Average RCRPS
Model
With context
Without context
p-value
DIRECT PROMPT Llama-3.1 -485B- Inst
0.165 = 0.005
0.544 + 0.007
6.92 x 10-13
LLMP Llama-3-70a
0.191 + 0.004
0.249 = 0.004
1.85 10-9
LLMP Mixtral-8x77
0.202 = 0.005
0.245 = 0.004
9.17 10
LLMP 1.1ama33-20
0.214 + 0.007
0.283 + 0.003
4.66 x 10-
LLMP dixtral-8x76B Inst
0.223 = 0.002
0.290 # 0.009
0.002
DIRECT PROMPT 2- Inst Inst
0.244 = 0.003
0.403 = 0.009
7.99 x 1
DIRECT PROMPT L1ama-3-70B-Ing
0.246 = 0.003
=
1.07 x 10-10
DIRECT PROMPT GPT-4o-mini
0,250 + 0.003
0.403 + 0.005
2.85 x 10
DIRECT PROMPT GPT
0,252 + 0.010
0.387 = 0.007
6.21 1044
UniTime
0,290 = 0.001
0.321 0.001
0.016
DIRECT PROMPT 2.50n-2.5-0... 1B.B-Inntt
0.343 = 0.011
0.258 + 0.004
0.987
Time-LLM (ETTh1)
0.378 = 0.001
0.364 + 0.001
1 10
DIRECT PROMPT Mixxtral-xx7B-Ins
0.413 = 0.007
0.699 # 0.006
1.88 x 10-6
LLMP Lapm-Laam--3 3-8B-Inst
0.413 # 0.009
0.432 = 0.010
0.287
DIRECT PROMPT Llama-3-80-Ins.
0.416 + 0.007
3.31 x 10-10
DIRECT PROMPT Qwen-2.5- 5-1.5-156 5B- Inst
0.481 + 0.016
0.406
0.975
ChatTime-Chat
0.557 + 0.001
0.004
0.086
ChatTime-Base
0.568 # 0.001
0.556
1 1.35 10
LLMP LLama-3-70B-Ing
0.57
0.313
1.93 10
LLMP Qwen-2.5-78
0.909 0.025
0.618 + 0.018
- 8.48 10-6
LLMP Qwen-2. 501.5--1-----1 Inst
2.038 # 0.025
1.181 = 0.022
LLMP Qwen-2. 5-0.5-0.555 5B Inst
2.067 # 0.025
1.047 = 0.017
LLMP Qwen-2. 5-0.5B
2.144 = 0.021
1.766 = 0.025
1 4.83 X 10
LLMP Qwen-2.551
2.162 + 0.028
0.861 = 0.016
LLMP Qwen-2 2...-2..11711
2.275 # 0.025
0.895 + 0.014
To whether the lower average RCPRS using context than without using context we observe for the best performing model in benchmark is statistically significant, ran analysis using the paired t-test. We used the paired t-test implemented in the scipy Python package as the scipy stats. ttest_rel method, with al alternatives "less" to make the test one-sided. As can be seen in Tab. 7, for many models the improved RCPRS when using the context is statistically significant, with p-values lower than x 10 for 7 out of the 23 models under consideration. Furthermore, the best performing models are those for which the improvement is statistically significant, with the 9 best models all having p-values below 0.01.
C.7. Cost of APl-based models
Tab. provides the cost incurred in evaluating GPT-4o (version spt-4o-2024-05-13 and GPT-4o-mini (version gpt-4o mini- 2024-07-18) with the Direct Prompt method on CiK (as per the evaluation protocol used, described in Sec. 5.1).
32

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
Table Costs (SCAD) evaluating 6PT-4o family models on CiK. "Total" represents the total enst evaluating each model on the CiK benchmark. The "Per-instance average" and the "Per-instance median" are the average and median cost of running single instance for given task, other words the average median cost of generating sample trajectories given example a task, reminder, cach task in CiK is evaluated over 5 instances in evaluation protocol.
Model
Total
Per-insfance average
Periustance median
GPT-4c
$143.83
S0.288
$0.170
contexi)
$139.50
$0.160
GPT-4c-mini
$13.79
50,040
$0.040
6PT-4o-mini (no context)
$13.32
$0.038
$0.040
C.8. Impact of Relevant and Irrelevant Information in Context
RCPS
bR 40
DR Llama-d 3 4000 Erer
LLAP Moera dld-e Irattrecc
Model
Figure 14. A comparison of RCRPS (lower better) for tasks predicting the Unemployment Rate of county. Both contain the context needed to solve the task. However, the Unempl UnemploymentCoum symmentoouttllsinggnnnn tagtttelatt task (dark green) is filtered to only contain the relevant context. Other the other hand, the linemployment ooyeentountyyy Fxp 1ic1 teMu1tipl wellipppettteeeeeeeee helppstteaaat task (light green) also contains other unrelated context. We visualize three models here, all which perform hetter when the context only includes the most relevant information.
We study here if models perform better on context that has already been filtered to only contain relevant information. To assess this, we compare two tasks on predicting the Unemployment Rate of a county.
1. For the Unemploymen which the county belongs to, tagged with the name of the state. See ey- www.eorttigg//aaaaaaaaaooo www.iiggg////// /ttp:////////mlnnnn/e twg.ueeennnooeoootttuttuuuuiiinn
task, the context contains the unemployment rate of the state
github. co/contexttaa is-k html for visualization.
2. In the rate, the context includes unemployment rates 2 other randomly selected states, also tagged with state names. See nttps://servicee github. http:/onnextii-ieee -orrree--i---k----oeeettttttttee eStateData. html for a visualization.
task, in addition to the county's state unemployment
forecas /0 ott.re/eeeottiieeeeeeeeeeooooooo http////wwwooeoooooo////o//aaaaa
Results of three randomly picked models from the benchmark is visualized in Figure 14. We find that models perform much better when only the relevant state's 's data is provided, as opposed to the context also containing data from other states.
C.9. Impact of Solely Irrelevant Information in Context
Many of our tasks include covariates in its context which are highly useful for the models to accurately predict the target time series. One question is: Do the LLM- -ased models perform well for such tasks due to correctly understanding that said covariates are helpful or because they blindly use the provided data without asking themselves the data is actually relevant?
33

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
As way get some insight on this question, we took a task where the models have to forecast the unemployment data of an American county, given the unemployment data of the state the county is in (Task Unemployment byment Sungnnnngnnnn teData). We then modify this task by first trying to mislead the model by wrongly saying that the state-level data was from another state (without changing the data itself), then by giving the data from the other state (while explicitly telling the model that data is from said other state), before finally removing the state-level data altogether. The result for this experiment with instances per task for DIRECT PROMPT - GPT-4 is shown in Tab. 9, while the forecasts for a single instance are shown in Figure 15. From these, we see that the model aggressively used data which is marked as being from an other state, even though if the data was actually from said other state, the performance would be closer to not having any state-level data. This shows that the model is liable to take any information provided being useful, even though its usefulness is marginal.
Table Ability of the DIRECT PROMPT GPT-4o model to accurately predict unemployment level of an American county, given various covariates. These results are averaged over instances.
Available data
RCPRS
Data from the correct state, accurately tagged
0.0583
Data from the correct state, inaccurately tagged
0.0557
Data from incorrect state, accurately tagged
0.1966
No state-level data
0.2630
C.10. The effect of significant failures on the aggregate performance of models
35
30
20
10
"lok PCRPS
Figure 16. Histogram of the RCPRS (lower is better) of the DIRECT PROMPT Owen-2 --en- 580.58-1rr AB-Instruct model each task, with and without context. With context, the RCRPS is close to zero for a large number of tasks, but there is also a long tail of tasks with high RCRPS values, dominating and worsening the model's aggregate RCRPS.
As discussed in Sec. 5.4, in a few instances from the benchmark, some models return forecasts that miss the ground truth by large margin, which we term significant failures (detailed in Appendix C.5). We analyse the effect of such significant failures on the results here. We use the DIRECT PROMPT Mixtral 8x7B model as an example here, while the same phenomenon may apply to other models. Figure 6, we can find that the aggregate RCRPS of DIRECT PROMPT Mixtral 8x7B worsens when uses context. However, in Figure 5 (left), the win rate of the model vs quantitative baselines improves when it uses context. These figures show results that seem contradictory, but are in fact compatible: adding context improves the model's RCRPS for most tasks, but greatly worsens it for minority of tasks where the model achieves significant failures.
To further illustrate this effect, we visualize the task-wise RCRPS of the DIRECT PROMPT Mixtral-8x7B Insttuutt model, both with and without context, in Figure 16. With context, the model gets an RCRPS close to zero in a large number of tasks. However, there is also a long tail of tasks with high RCRPS values with context, dominating and worsening the model's aggregate RCRPS.
34

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
3.50
Hatery
History
3.25
Ground Trulh
Ground Youh
93%93%
3.00
10N-901%
25%-75
25%-75%%
50%
2.6
50%
2.0
1.50
2023
2023-09
2023 30
2023 0223-22
2024 01
202465
2024 04
2024 45
2024 06
2023
2023
09
2023-10
2023-11
2023-22
2024-1
2024
2024
2024 04
2024 05
2024-00
202400
(a) The task in our benchmark: the context contains memploommnn nate of
(b) The contest only mentions that this time series is on snempleyment rote, and
the the county correctly togged with
which county state-level unemployement data is provided.
3.25
Histery
History
Ground Trush
3.00
Ground Truth
596-955
5%--55%
10%-90%%
2.75
50%
2.75
50%
75%
2.50
2.25
2.25
2.00
3.75
1.75
1.50
2023-88
2021-99
2023 50
2023-22
2028 01
2026 45
2024 06
2024-77
2023
2023-09
2023-1
127-2
2044-1
2024-22
63
85
202--0
2024
2223
200
(e) The state-level mnemployment rate incorrectly lagged as being from an-
(d) The context contains the unemployment rate of another state than the one the county which correctly togged.
other
Figure Forecasts done by Direct Prompt GPT-4o, with varying information in the context. The task forecast the unemployment
rate of an American county.
35

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
C.1I. Visualizations of successful context-aware forecasts
Context: " 'This series represents the occupancy rate (%) captured by a highway sensor.
Consider that the meter will be offline for maintenance between 2024-04-11 11 13:00:00 and 2024-04-11 15:00:00. which results in zero readings."
25
Forecast
History
Ground Truth
20
Region of Interest
5%-95%
10%-90%
15-
25%-775%
10
2024-04-05
2024-04-06
2024-04-07
2024-04-08
2024-04-09
2024-04-10
2024-04-11
2024-04-12
(a) Without Context
45
Forecast
History
Ground Truth
20
Region of Interest
5%-95%%
10%-90%
15
25%6-75%
2024-04-05
2024-04-06 00-006
2024-04-07-
2024-04-08
2024-04-09
2024-04-10-
2024-04-11-
2024-04-12-
(b) Widh Context
Figure 17, Example of successful context-aware forecasting by DIRECT PROMPT with Llama-3, 1lama-3.1-48566
36

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
This series contains Diftuse Horizontal Irradiance for location in Sinaloa, Mexico. The Diffuse Horizontal Irradiance is the total amount of energy (in Watts per squared meter) arriving indirectly horizontal surface, ignoring the direct sunlight. Even when there still be some Diffuse Horizontal Irradiance, since clouds are not the only cause of light scattering. When there are no clouds, the Diffuse Horizontal Irradiance is mostly a function of the position of the sun in the sky, with only small variations from factors such as water vapour and dust particles levels. If the cloud cover is light, the Diffuse Horizontal Irradiance will increase due to the increase scattering of sun light. but heavy cloud cover will decrease it due to some sun light no longer being able to reach the ground. At the beginning of the series, the weather was cloudy. At 2022-07-11 12 11:00:00, the weather became clear. At 2022-07-12 19:00:00, the weather became cloudy.
Context:
clouds scatter the sun light, there will
At 2022-07-13 12:00:00, the weather became clear.
At 2022-07-13 13:00:00, the weather became cloudy.
At 2022-07-14 06:00:00, we expect that the weather will become clear.
At 2022-07-14 07:00:00, we expect that the weather will become cloudy.
At 2022-07-14 10:00:00, we expect that the weather will become clear.
At 2022-07-14 18:00:00, we expect that the weather will become cloudy.
400
Forecast
History
350
Ground Truth
5%-95%
300
25%-75 %
50%
250
200
150
100
07-13 00
07-1312
07-14
07-14
07-15 00
07-15
(a) Without Context
Forecast
History
350
Ground Truth
5%-95%
300
10%-90%
50%
200
150
100
0
07-13 00
07-13 12-
00
07-14
07-15 00.
07:14
(b) With Context
Figure 18. Example of successful context- aware forecasting by DIRECT PROMPT with Llama-3. 1.ttaa3..114180 1-4058- -eett-Inttteeeeeee Instruct
37

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
Context:
This the number of cash withdrawals from an automated teller machine (ATM) in an arbitrary location in England.
Consider that the building which contains the ATM is closed from 1997-09-05 00:00:00, for days.
60
40
30
20
10
Forecast
History
Ground Truth
Region of Interest
5%-95%
10%-990%
25%--75%
-2
19977 7o0
1997400
199740
1997-07
1997-09
1997-10
1
19977
(a) Without Context
60
Forecast
History
Ground Truth
50
Region of Interest
$%-95%
10%-90%
40
25%-775%
20
10
1997-04
1997-05
1997-06
1997-07
1997-08
1997-09
1997-10
(b) With Context
Figure 19, Example of successful context-aware forecasts by DIRECT PROMPT with GPT-40
38

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
"The Montreal Fire Department in charge of responding various kind of public safety incidents. This is the number of field fire incidents responded by Montreal firefighters in the -eviere-de--Prairiss-- Pointe- aux- Trembles borough. In other years, the yearly average number of incidents 106 with the busiest month being June.
Context:
The Mayor is determined to completely cradicate this kind of incident. Fortunately, the city's 's public safety research group identified that field fires and trash fires tend to co-occur. When the amount of field fires increases, the amount of trash fires also tends to increase. The same holds when they decrease.
The Mayor has a plan: they will implement daily spraying all piles of trash with water starting on 2022-06.
120
Forecast
History
Ground Truth
100
5%-95%
10%--90%
25%6-75%
80
50%
60
40
20
2022-01
2022-03
2022-05
2022-07-
2022-09.
2022-11
(a) Without Context
Forecast
17.5
History
Ground Truth
15.0
5%6955%
10%-90%%
25%-75%
12.5
50%
10.0
7.5
5.0
2.5
2022-01
2022-03
2022-05
2022-07
2022-09
2022-11
(b) With Context
Figure 20. Example of successful context-aware forecasts by DIRECT PROMPT with GPT-4o
39

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
Context: " This the Unemployment Okaloesa County, in Florida. For relerence, here the Unemployment Rate for lew American states during the same period: Penncylvania
(2023-08-01 00:00:00, 4.2)
(2023-09-01 00:0000, 3.0)
(2023-10-01 00:0:00, 3.1)
(2023-11-01 00:00:00, 2.9)
(2023 12 01 00:00:00, 2.9)
(2024-01-01 00:00:00, 3.5)
(2024-02-01 00:00:00, 3.7)
(2024-03-01 00:00:00, 3.4)
(2024 04-01 00:00:00, 2.9)
(2024-05-01 00:0:000 3.2)
(2024-06-01 00:00:00, 3.7)
(2024-07-01 00:00:00, 4.0)
Florida
(2023 08 01 00:00:00, 3.3)
(2023-09-01 00:00:00, 3.1)
00:00:00,33 3.1)
(2023-11-01 00:00:00, 3.0)
(2023-12-01 00:00:00, 2.9)
(2024-01-01 00:00:00, 3.1)
(2024-02-0) 00:00:00, 3.1)
(2024-03-01 00:00:00, 3.3)
22024-04-0 0I 00:00:00, 3.1)
(2024-05
2.9)
(2024-06-01 00:00:00, 3.5)
(2024-07-01 00:00:00, 3.8)
Wisconsin
(2023-0801 00:00:00, 3.4)
(2023-09-01 00:00:00, 2.9)
(2023-10.01 00:00:00, 2.8)
(2023-11-01 00:00:00, 2.7)
(2023-12-01 00:00:00, 2.9)
(2024-01-01 00:00:00, 2.8)
(2024-02-01 00:00:00, 3.3)
(2024-03-01 00:00:00, 3.5)
(2024-04-01 00:00:00,30 3.0)
(2024-05-01 00:00:00, 3.0)
00:00:00,333)
(2024-07.0) 00:00:00, 3.3)
Forecast
Forecast
3,6
5%-95% %
10%-90%%
10%-90%
3.2
25%-75%6
25%-75%
3.0
2.6
2021.00
2023 69
3023-30
2022
2023-12
2024-01
3024-02
2024-03
7024-04
2024-05
2024-88-
2024 07
3023
2023 09
0023.00
2023-3
2023-11
2023-12
7024-01
200403
2024
2024 06
0024-07
(a)
Contert
(b)
Figure Example successful context-aware forecasts by L.MPP with Mixtral 8x78 Instruct
40

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
Context: Suppose that in the forecast, the values are bounded below by 0.80.
Forecast
2.5
History
Ground Truth
5%.95%
2.0
10%-90%
25%-75%
50%
1.5
W
1.0
0.5
0.0
2024-01 30
2024-01-31
2024-02-01
2024-02-02
2024-02-03
2024-02-00 04
2024-02-05
2024-02-06
(a) Without Context
Forecast
2.5
History
Ground Truth
5%-95%
2.0
90%
75%
50%
1.5
1.0
0.5
0.0
2024-01-30
2024-01-31
2024-02-01
2024-02-02
2024-02-03
2024-02-04
2024-02-05
2024-02-06
(b) With Context
Figure 22, Example of successful context-aware forecasts by LLMP with Nixtrall-x78-Instrree

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
Context:
This series contains the amount of sunlight (in Watts per squared meter) arriving on horizontal surface, for location in Alaska, United States."
1750
Forecast
History
Ground Truth
1500
5%-95%
10%-90%%
1250
25%-75%
50%
1000
750
500
250
12-23 00
12-23 03
12-23 06
12-23 09
12-23 12
12-23
12-23 18
12-23 21
Without Context
800
Forecast
History
700
Ground Truth
5%-95%
10%-90%
600
25%-75%
50%
500
400
300
200
100
12-23 00
12-23 03
12.23 06
12-23 09
12-23 12
12-23 15
12-23 18
12-23
(b) Widh Context
Figure 23, Example of successful context-aware forecasts by LLMP with Llama-3-700
42

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
The Montreal Fire Department is in charge of responding to various kind of public safety incidents. This series contains the number of field fire incidents responded to by the Montreal Fire Department in the Rosemont-Laa Petite-Patrie borough. On average, they respond to 58 incidents per year and the month with the most incidents was June,
Context:
The Mayor is determined to completely eradicate this kind of incident. Fortunately, the city's 's public safety research group, team of highly qualified experts, identified that field fires and gas leaks tend co-occur. When the amount of field fires increases, the amount of gas leaks also tends to increase. The same holds when they decrease.
The Mayor has a plan: they will implement a strict prohibition of using any form of combustible gas in the city starting on 2023-06. recent interview, they claimed, "This a bulletproof plan, and am certain it will immediately put an end to field fires." "
Forecast
175
History
Ground Truth
5%695%%
150
10%-90%
25%-75%
50%
100
75
50
25
2023-01
2023-03-
2023-05
2023-09
2023-11-
(8) Without Context
0.0
Forecast
History
7.5
Ground Truth
5%-95%
5.0
10%-
25%-75%%
50%
2.5
0.0
2023-01
2023-03-
2023-05
2023-07
2023-09 -9
2023-11
(b) With Contest
Figure 24. Example of successful context-aware forecasts by L.IPP with L.lmma- 3-788
43

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
C.12. Visualizations of significant failures
Context: " Given are variables X and X.I, where X_0 is covariate and X_1 is the variable forecast. Variables are generated from linear Structural Vector Autoregressive (SVAR) model with additive gauss noise and noise scale of 1.487c-03, with lag 3.
The task is to forecast the value of the variable X 1 at time I, given the values of the covariate X O and the variable X_1 itself at times t-1, ... 1-3, For the first 128 days, the covariate X_0 takes value of 8 from 2024-02-21 to 2024-03-11, 12 from 2024-03-12 to 2024-05-06, 12 from 2024-05-07 to 2024-06-27. For the next 32 days, the covariate X_0 takes a value of 30 from 2024-06-28 to 2024-07-13, 60 from 2024-07-14 to 2024-07-14, 60 from 2024-07-15 to 2024-07-29. Each day can be treated as timestep for the forecasting task. The causal parents affect the child variables different lags.
The causal parents for each variable is given below:
No parents for X_0 at any lag.
Parents for X 1 at lag 1: ['X O', 'X I'] affect the forecast variable as 0.527 X 0++ -0.895 X I.
Parents for X_1 at lag 2: ['X_0", 'X_I"] affect the forecast variable as 1.380 X_00+ +0.758 X.I.
Parents for X_1 at lag 3: ['X_0', 'X_1'] affect the forecast variable as -0.661 * X_00 + -0.793 X.1.
Forecast
0.8
Ground Truth
10%-900%
0.6
25%6-75%
www.mm
2024-04
2024 05
2024 06
2024-07-
2024-0
Without Context
0.0
1615
Forecast
History
Ground Truth
5%-95%
1.0% 90%
-0.4
5.0%
-0.6
-1.0
-1.2
2024 03
2024 04
2024-03 05
2024-06
2024-07
(b) Widh Context
Figure 25. Example to show significant failure case of DIRECT PROMPT with GPT-4o where its performance worsens with context
44

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
Context: This series contains the road occupancy rates on a freeway the San Francisco Bay area. The days for which the forecast is required Thursday 2024-07-04, Friday 2024-07-05, Saturday 2024-07-06. Note that 2024-07-00 is holiday due to Independence Day. Note that traffic on this freeway typically reduces holidays."
40
35
30
Forecast
History
25
Ground Truth
Region of Interest
20
95%
10%-90%
15
25 %5%%
10
2024-06-28
2024-06-29
2024-06-30
2024-07-01
2024-07-02
2024-07-03
2024-07-04
2024-07-05
2024-07-06
2024-06-
2000
Forecast
History
1750
Ground Truth
Region of Interest
1500
5%-95%
10%-90%
1250
25 25%
1000
750
250
2024-06-27
2024-06-28
2024-06-29
2024-06-30
2024-07-01
2024-07-02
2024-07-03
2024-07-04
2024-07-05
2024-07-06
(b) With Context
Figure 26. Example to show significant failure case of L.IMP with
1ama--3798 788 where its performance worsens with context
45

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
This series represents the occupancy rate (%) captured by highway sensor. The sensor had a calibration problem starting from 2024-04-20 13:00:00 which resulted in additive trend in the series that increases by 0.0072 every hour. At timestep 2024-04-2 13:00:00, the sensor repaired and this additive trend will disappear."
Context:
Forecast
2.5
History Ground Truth
5%-95%
10%%90%
2.0
25%-75%
50%
1.5
1.0
0.5
0.0
2024-04-18
2024-04-19
2024-04-20
2024-04-21
2024-04-22
2024-04-23
2024-04-24
2024-04-25
2000
Forecast
History
1750
Ground Truth
5%-955 9%%955%
1500
100%900%
25%-75%
1250
50%
1000
750
500
250
0
2024-04-18
2024-04-19
2024-04-20
24-4-04-21-
2024-04-22
2024-04-23
2024-04-24-
2024-04-25
(b) With Context
Figure 27. Example to show significant failure case of L.IMP with
1ama-3-22 768 where its performance worsens with context
46

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
The Montreal Fire Department is in charge of responding to various kind of public safety incidents. This series contains the number field fire incidents responded to by the Montreal Fire Department the borough. On average, they respond to 19 incidents per year with the busiest
Context:
L'fle-Bizarde Seeedd-Sittt--eeeeee month being June.
The Mayor is determined to completely eradicate this kind of incident. Fortunately, the city's public safety research group, team of highly qualified experts, identified that field fires and trash fires tend to co-occur. When the amount of field fires increases, the amount of trash fires also tends to increase. The same holds when they decrease.
The Mayor has a plan: they will implement daily spraying of all piles of trash with fire retardant foam starting on 2023-06. In a recent interview, they claimed, "This a bullctproof plan, and am certain will immediately put an end field fires."
2.00
Forecast
History
1.75
Ground Truth
5%-95%
1.50
10%-90%%
25%-75%
1.25
50%
1.00
0.75
0.25
0.00
2023-01
2023-03
2023-05
2023-07
2023-09
2023-11
(a) Without Context
Forecast
History
Ground Truth
5%-95%
10%-900%
25%-755%
50%
4-
2
2023-01 -0-
2023-00 03
2023-05
2023-07
2023-09
2023-11
With Context
Figure Example to show significant failure case DIRECT PROMPT with Llama 3-88 Instruct where misinterprets context
47

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
D. Implementation Details of Models
D.I. DIRECT PROMPT
D.1.1. METHOD
For DIRECT PROMPT, we propose to use a simple prompt template that we describe below, where ((context)) is replaced with the context of the respective task, ((history) replaced with the historical values the given format, and ((pred.timg time)) replaced with the prediction timesteps. The prompted model expected output predictions in the given template style (i.e. within the given forecast tags, in the given format) for all prediction timesteps in the prompt. Notably, unlike LLMP which consists of predicting the single next digit in loop. Direct Prompt expects models to forecast in single pass in highly structured format, which requires models to understand and adhere to the template.
I have tine series forecasting task for you.
Here is some context sbout the task. Make sure factor any background knowledge, satisfy any constraints, and respect any scenarios.
<context>
((context))
</context>
Here is a historical time series in (timestamp, value) format:
<history>>
((histery))
</history>
Now please predict the value at the following timestamps: ((pred.time) .time))
Return the forecast in (timestamp, value) fornat between <forecast> and </forecast> tags. not include any other information (e.g., comments) forecast.
Example:
<history>
(t1, v1)
(t2, v2)
(t3, v3)
</history>
<farecast>
(t4, v4)
(t5, v5)
</forecast>>
To constrain the output of the model to follow the specific structure, constrained decoding using the ln-format-enfoncer tool (https:/ / g/tbbbbbb//b.b.b/////gg//bbtb. thub. ccm/ hoom/noaaatt// 11m format-enforcen and regular expression that only allows models output the values corresponding to the prediction timestamps. Without constrained decoding, we observe that models often produce samples that fail to adhere to the structure and are therefore rejected. Specifically, larger models (Llama-3. GPT-4o and GPT-40-min1) can produce 25 valid forecasts with to 3 trials. However with the smaller models (such as Llama-3-700- Instructt Llama-3-88 1tama-3-88-11 88-Innttrus and Mixtral-8x77 -erelllxxx---nnsee Innttrccc,, up to 10 trials can required to obtain 25 valid forecasts.
Further, found that without explicit "Do not include any other information (e.g., comments) in the forecast.", models often included unwanted information along with the forecasts.
Instruction-tuned models are more amenable for DIRECT PROMPT DIRECT PROMPT requires forecasts be produced in specific structure. To generate structured outputs, models need be steerable (Dubey et al., 2024), capability that typically elicited from base models with post-training methods such instruction tuning (Wei et al., 2021). We observe this in our evaluations we find that several base models, including Llama-3-B8 3-85, Llama-3-788 Mixtral-8xe 8877, and even the biggest base model we tried, Llama-3. 1-4058, are incapable of generating outputs adhering to the structure required for direcT PROMPT, despite increasing the number retries to as high as 50 retries. With DIRECT PROMPT, these models often output irrelevant information, sometimes completing solely the context as a text completion task, and in other cases regurgitating forecasting datasets that they have memorized.
48

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
While very simple, such prompt templates be powerful tools to understand how LLMs perform context-aided forecasting: as the prompt gives control over the structure and content of the output (particularly for instruction-tuned models), one may construct other, more involved template structures in the prompt. For instance, prompt template could ask LLMs to explain the reasoning behind their (context-aidess forecasts, and more. We leave it to future work to understand how such prompt-based techniques can lead to more detailed evaluations and give us better insights into what the models are capable of.
Extensions of DIRECT PROMPT
D. 1.2. IMPLEMENTATION DETAILS
We used single H100 GPU run the DIRECT PROMPT approach for 1emma-3-8-Inssteie and HI00 GPUs for Qwen-2. (-..555000000nn.... 5B-Instruct, 1.S-Instruct, 7B-Instruct), -Lama-3-70B-nnstraee and Hixtral-9x7B-nnsrrut. We queried Llama-3. 1. 1-485b-Insttucc from externally-hotted server running 8 H100s. We use the OpenAl API to perform inference on the proprietary 4o and GPT-4o-mini models. We provide the cost incurred in the inference of these models with the DIRECT PROMPT method in Appendix C.7.
D.1.3. EXAMPLE PROMPT
prompt used an example task from the benchmark is given below.
49

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
I have a time series forecasting task for you.
Here is some context about the task. Make sure to factor in any background knowledge, any constraints, and respect any scenarios. <context>
Background; : This is hourly traffic data.
Scenario: Suppose that there is an accident on the road and there is 40. .8.0% of the usual traffic from 2024 84 24 17:00:00 for hours.
</context>
Here is historical time series in in (timestamp, value) format: <history>
(2024-84-23 :0,, 8. (2824 1004--44-77
:0, (28244844-2) 02 :0:000 8) (2624-84-22 00: 0)2224--84
84: 04:00:00, 8.1, 8.1) (2824-84422
85 00,
-8-22----23 06 06:00:00, (.3))2224-04-2) 87:00:000
8984--44-8
08:00:00, 0.5) (2824-04-
00,
(28)444--8--22 10:00:30, (2824-284--4-2 11:00:00. (2824-24-44
12:00:00, (2924-2-42 13:00:00,
(2924--8884-22 14:00:00,
0.8)(2224-04-2) 15:00:00, (222
16:00:00.
(2024---42 17:00:00,
(2824224--4- 18:00:00,
0.84-2284-8--22 19:00:00. (.82
00,00
(2824--4-22 21:00:00
(2824288488 22:00:00 00. 8.1)(2624-04-23 23:00 :0:0000000 (284(2224-84-24
00:00:00.
(2824---244 01 10:00000 80, 0)22444-84 82:80:00, 8) (2824-64-24 03 08: (2824-84-2
04:00:08
(2824284--4-22
0s 80,
(2824484-4 00:00:00, 8. 8.3)(2624-04-24 07:00
0 (2824-84-4
08:00:00,
(2824-22 04-24 09:00:00,
0. (2824 84-24
18:00:30, 8.4)(2824-06) 04 11:00:00,
0 0.5) (2824 84-24
12:00:00, 0.6)
</history>
Now please predict the value at the following timestamos: [ 2924-84-24
13:00:00'
'2824-84-24
14:00:00'
"2824-04-24 15:00:00'
24-4-44-24
15:00:00 '2024-84-24
17:00:00'
'2824-04-24
18:00:09
'2824-04-24
19:00:00 '2824-04-2
20 20:00:00 22024-04-24
21 20:00:00
'2024-84-24 22:00:00:
22024-04-24
23:00:00'
'2824-04-20 80:00 :0: eea
"2024-84-25
01 :0"
"2894-84-25
62 :0:000"
'2024-04-25
83 00:00:0001
'2024-84-25
84:00:00" 0982484422
65 06:00:00 8e 22624-04-25
06 06:00:88 80 '2824-84-25
07:00:08
2824-84-25
08. 98:00:00"
2222-04-22 89 :9:00:00 80' 22024-84-25
18: :00 22824-84-25 11 10:00:00
"2624-04-25 12:00:001
Return the forecast i (timestamp, value) format in between <forecast> and </forecast> tags.Do other information (e.g.. commntts) in the forecast.
Do not include any
Example:
<history>
(t1, v1)
(t2, v2)
(t3, v3)
</history>
<torecast>
(t4, v4)
(ts, v5)
forecaat
D.2. LLMP
D.2.1. METHOD
In this section we outline proceseesssss (LLMP; Requeima et (2024)), one of the prompt- based baselines evaluated in Sec. 5.3. Prompts are constructed by first providing textual information followed by the numerical history. The context may include background knowledge, scenario description and task constraints, replaced by (bbakkgound)). ((scenario) and (( (ccosttraints), respectively, in the prompt template below. The numerical history ((history))) is provided by converting the numerical data to text where values are separated by commas () and tuples by newline characters ('n)). The LLM then outputs the continuation of the string prompt, forecasing the the value for the next time index (((next index))). This forecast and the next time index is appended to the prompt allowing the LLM to autoregressively complete the entire forecast. Numerical samples are rejected if they do not adhere to decimal representation format. See Requeima et al. (2024)) for full
50

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
details.
The following is the prompt template used to construct prompts for the LLMP baseline:
Forecast the future values of this time series, while considering the following background knowledge, scenario, and constraints.
Background knowledge: ((background))
Scenario:
((scenario))
Constraints:
((constraints))
((history))
((next index))
A prompt used in an example task from the benchmark is given below:
Forecast the future values of this time series, while considering the following background knowledge, scenario, and constraints.
Background knowledge:
This is hourly traffic data.
Spenario:
Suppose that there is accident on the road and there is 40. of the usual traffic from 2024-64-24 17:00:00 for 6 hours.
Constraints:
2824-84-23 00:000:000000 ,0000 1122244-44223 01: 01:00:200 81 n2824-8--22 02 00 n2824-88--2222 03:00:00 02224-84-23
04:00:00, 1222224
05:00:00,
06:00:00
0.3.0...3333222222 04 07: 07:00:00. 5222::2222....222444 -44
08:00:00, 8.5)n2222 84-23 09:00:00, 422224 94 10:20:00,0 6| 23 11 11:00:00,0. 80 84-23
12:00:00, 8. 0.4%)2202--04
13:00:00, 622222 84-23 14:20:00, 000:000000.88)2222 82)22222)2222282 84 23 15:00:00, 16:00:00,1..2)222200 1.2.2.24-22222
16:00:00 :00000001 2.2)22224-88423 17:00:00,1 17:00:00,1.3))22200. 2.3.42222--8-22 18:00:00,
0.2222244222222333 18:00:.00.0.6)222244-28
19:00:00, 6 0.3
20:00:00 :00, 8.3442224- 21:09:00. 22.2222222--4-2 22:00:00, 0.11..2-1.111222222--2220 23:00:00. 022244-4--2
00:00:00 :00, 8. 111/22244-04-22 01:00:00, 0122224-84-224 02:00:00,
83 00:0000000000 9422244--444 04:00:00,0 0.1
\n2824-
444444 84-24 05 05:00:00, 0.24022944-8--44 84-24 06 08:00,
07 00:000 52
(0::80,00 00, 62,222
-04-24 89: :0 :08, 8. 51.22224444422222
10:30:00 00, 8. 4.24))2224-442222
00:00:00000
12 10:00:08 0..26.622222--2222
13:00:00,
D.2.2. IMPLEMENTATION DETAILS
We used single H100 GPU to run the LLMP approach for following models: Llama-3--8 3-88, and Llama-3-8B -eeee----B-Inttte Instruct. used 2 H100 GPUs for the Qwen-2.5 family of models, Mixtral-8x7B, and Mextrrl--x7B--- -t and used used H100 GPUs for the following models: L1ama-3-700, and Llama 1lama-3-788---1 ---77--Insttcct.
Since the code of L.MP (ht //g thub. com/ reque iaeeemmamll11 /m processes/) only supports using open-source models (such as those available in HuggingFace) and requires loading the weights into memory, it does not support experimenting with the GPT-4o and GPT-40-mini models. Further, due to the memory requirements LLMP, we were unable to experiment with the L1ama-3.1--a 1-405B and Llana-3. 1-405B-Insttruc Instruct models that required more than 24 H100 GPUs in parallel to process a single instance from the benchmark, which exceeded our available resources.
51

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
D.3. ChatTime
We evaluate the released ChatTime-Base (nttps://huugiingaaee www./hhhenneeeeaaagnhaaaaaiiiiee -aseeae) and ChatTime- Chat (https: (nttps://hugging /ttgghggggggcceeeec/////// co co//hhhengsammeengg/che11m----1- 7hathat) models zero-shot, per the instructions in the authors' GitHub repository
D.4. UniTime and Time-LLM
For multimodal models, jointly train UniTime (Liu et al., 2024c) its ensemble of datasets: ETTm1, ETTm2, ETTh2, Electricity, Weather, Exchange and Illness.
We also evaluate Time-LLM (Jin et al., 2024), another multimodal model built on top of the Llama architecture. We train Time-LLM on ETThl according to the authors' suggested specifications, and we compare the performance of both models with and without context.
UniTime: We train UniTime (Liu al., 2024c) with their codebase (https://githil com/1iuxu77/ /nni/ using single seed one AMD Instinct MI200 GPU for approximately 14 hours. It features lightweight transformer with maximum context length of 200 and a pre-trained GPT2 language model as backbone, of which only the first half of the transformer layers are used. The time series baseline employs non-overlapping patch embeddings generated with a kernel size and stride of 16, and a maximum input sequence length of 96. When the total tokenized length exceeds the architecture's 's capacity, we truncate the context.
Unlike TLMe-LLM, UniTime is jointly trained on all datasets simultaneously. Batches were generated by first choosing dataset uniformly at random then returning a batch from the associated data loader. To account for domain convergence speed imbalance, mask rate of 0.5 is used and the training batch size is varied according to the dataset (details in the data config directory of the UniTime GitHub repository). Training was conducted for 10 epochs of the mixed dataset, with cosine decay from an initial learning rate of le-4 to minimum of le-6 over a maximum period of 20 epochs. The results of our training on the original datasets are given in Tab. 10.
Finally, in order to accelerate training, we added BF16 automatic mixed precision training and gradient accumulation to the original training procedure.
Time-LLN: We train Time-LLM (Jin al., 2024) with their codebase (ht //gi thub. com/ KimMeen Twte/mimeeii/ L.M) the ETTh1 dataset (Zhou al., 2021) with prediction length of 96. We train using single seed on four AMD Instinct MI200 GPUs, with average training time per run of approximately 13 hours. Training was conducted using batch size of 8 per device and 4 gradient accumulation steps, along with a ICycle learning rate schedule with a maximum learning rate of le-3, In addition, runs were accelerated using DeepSpeed Stage 2 and BF16 automatic mixed precision.
Training was conducted over a maximum of 50 epochs with early stopping, and a time-based split of 70% for training, 10% for validation, and 20% for testing, where the most recent windows were reserved for the test set. All runs were trained with
Table 10. Evaluation results for Un on their test splits. Results are comparable to the original paper, although MSE Illness is approximately 20% higher for prediction lengths 36,48,60.
Dataset
Mean Squared Error (MSE)
Prediction Length
96
192
336
720
ETTh1
0.395
0.435
0.469
0.468
ETTh2
0.291
0.368
0.413
0.422
ETTml
0.336
0.377
0.409
0.465
ETTm2
0.181
0.248
0.315
0.417
Exchange
0.090
0.180
0.322
0.862
Weather
0.179
0.224
0.278
0.354
Electricity
0.198
0.202
0.217
0.257
24
36
48
60
Illness
2.284
2.515
2.572
2.455
52

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
an input sequence length of 512, with overlapping patch embeddings generated with kernel size of 16 and stride of 8. The results on the ETTh1 test set given in Tab. 11.
When evaluating on CiK tasks which do not conform to Time-LLM's requirements, we make the following modifications to the method:
For short history tasks where the history length X is less than 5, we change the topk operator's & value from to |Hl in the calculate.lagst function.
For tasks where the length of the prediction window |X jexceeds the trained projection head's output dimension (in our case, 96), we repeat the last predicted value |Xpl 96 times. This occurs for very few tasks (3 tasks) with prediction windows of 97 or 98 steps depending the sampled instance, which we assume leads negligible impact evaluated results.
Table 11. ETThl test set results for Time-LJ. trained on ETThI.
Time-LLM
MSE
MAE
-T96 0.3846123
0.4149854
Why Do Time-LLM and UniTime Not Benefit (More) From Context? Looking table Appendix C.I, we that context actually harms the performance of Time-L forecasts. Two possible reasons for this are: 1) Time-LLM's adaptation procedure is unlikely to retain the backbone LLM's language-processngg capabilities, and 2) Time-LLM's single-dataset training procedure is unlikely to generalize to unseen time series patterns. Part of Time-LLM's model adaptation involves training linear layers at the input and output of the language model. Although the backbone LLM remains frozen, these linear layers must be trained, and T'me-LLM opts for highly structured prompting format which involves domain knowledge, task instructions and input statistics. Since the training data for the linear layers consists of output representations based on these highly structured prompts, it is not evident that the resulting architecture will generalize more diverse contextual descriptions such as those found in CiK. Furthermore, although we have not conducted a formal analysis of the diversity of the ETTh1 dataset, it is not a priori obvious that such dataset would have sufficient diversity of patterns to train a time series foundation model.
Interestingly, UniTime's performance does benefit from context for some tasks (see Figure 29). However, the aggregate RCRPS and rank of UniTime with respect to other models indicate that it still struggles to produce forecasts competitive with even quantitative forecasting methods.
D.5. Lag-Llama
We use the publicly available implementation of Lag-Llama (Rasul al., 2023) located at https:///gthhii com/ time- ser inn-foouuttii--modelll.. and its associated pre-trained weights. The model inference was done on single H100 GPU.
D.6. Chrones
We use the publicly available implementation of Chronos (Ansari et al., 2024) located n- socience/chronoos- http://hoooooocoeeaaeeeeeeeeeeee forecasting. We evaluated (see Appendix C. our tasks on all 5 available models: chronos-tiny, chronos-mini, chronos-smale chronos-base and chronos-large and reported the results of the best performing model, chronos-large in Tab. 1. The model inference was done on a single H100 GPU.
wwm.thth.cooooooooccooooooooo com/ amazo
D.7. Moirai
We use the publicly available implementation of Moirai (Woo et al., 2024) located at https://githubb wtt.hhu...oo//aaaaffa/aao
CeeaTReeserrch/mmi22t We evaluated (see Appendix C.1) our tasks on the 3 following models: moirai-1 moirai-1..-RR (0-Rssmal) (located at https: /htggggggaaaaceeece//////gg/gace co co/Salesforce /wwo/mmoiiiiiieeie 1..0-..0000 R- small), moirai-1.0-R-bon / //huggingface wwtp:eeewcac.c.eeee/e/a/aaaaaaae /mocc//mmrrraaa rai-1. 1.0----- R-Rebas)) and moirai- 1.0-111--RR- R-O-e- (located at https: http::///hggggg/g///e/e co/Sal esforce porcee/mooraa /mmirai-1111./ -----large and reported the results of the best performing model, moirai-1.00 l-O-R-large in Tab. 1. The model inference was done on a single H100 GPU.
(located at https:
53

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
Context: "Suppose that in the forecast, the values are bounded above by 6.29."
Forecast History
Forecast
History
Ground Truth
Ground Truth
95%
95%
10% 90%
10% 90%
25%6-75%
25%6775%
50%
50%
2024-01-12
2024-01-13
7024-01-14
2024-01-55
2024-01-16
2024-0117
2024-01-11
70240112
2024-01-14
7024-11-55
2024-01-16
202401-17
2024-01-18
Figure 29. A comparison of forecasts from UniTime without context (left) and with context (right). On average across instances, UniTime's
RCRPS is 64% better with context than without on the "Bounded Prediction Constraint Based On Prediction Quantiles" task.
D.8. TimeGEN
access TimeGEN-1, an optimization of the TimeGPT model (Garza et al., 2023), using the API made available through the nixtla Python package. Unlike other baselines, we only generate point forecasts with TimeGEN due to its probabilistic mode requiring much longer historical data than is available in instances evaluated in the benchmark. This is the reason the RCPRS values for TimeGEN have zero standard error.
D.9. Exponential Smoothing
We used the Exponential Smoothing implementation from the statemodels Python package, namely the statsmodels tsa. www.wwitewss.....mmmmoo.ooo Exponential Smoo class. Both trend and seasonal components of the models are set to be additive. The seasonal period length is either set manually for tasks where the simple guess using the time series frequency is incorrect. If there is not at least two full seasonal periods in the history window of the time series, we disable the seasonal component of the model. Since some of the benchmark tasks can have as few as time steps in the history window, we also disable the trend component if we have less than time steps in said window.
D.10. ETS and ARIMA
We used the implementations ETS and ARIMA from the forecast package, using rpy? for compatibility with Python. For ETS, we use the ets method, which we call with automatic error, trend, and seasonality components. In the rare cases where the ETS forecast contains NaN values, we manually switch off the trend component and rerun the forecast. TheARIMA results are computed using the auto. arima method. If the ARIMA fits fail, we rerun it with restricted parameter and disabled seasonality.
E. Details of the proposed metric
The CiK benchmark is designed to determine whether models can improve their probabilistic forecasts by leveraging associated textual information (see Sec. 2). To support this goal, the evaluation metric:
1. should be a proper scoring rule, such that a model who perfectly knows what the correct forecast is should have no reason to favor another prediction;
2. must be easy to compute using a finite sample from the forecast distribution, since many models do not provide a functional form for their forecasts.
54

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
To account for the importance of leveraging relevant context, the metric should also:
1. penalize obviously impossible forecasts, i.e. that can be inferred as implausible from the contextual information;
2. take a similar range of values across different tasks, to prevent some tasks to dominate the score as we average the results across tasks;
3. prioritize forecast quality for timesteps with relevant context, even if these timesteps are a small portion of the forecast horizon.
To satisfy the first two properties, we start with the Continuous Ranked Probability Score (CRPS) (Gneiting & Raftery, 2007), reliable strictly proper scoring nule for univariate probability distribution, and take its mean over all time steps. To compute the CRPS from finite number of samples, we use the estimator based on its probability weighted moment form (Taillardat et al., 2016), since it is unbiased (Zamo & Naveau, 2018). See Appendix E.3 for more details about this estimator.
Many of our tasks are built to include information about a hard constraint on Xxr in their C, which can be written as vc(xF) = 0. If we were only interested to measure by how much a forecast breaks the constraint, we could take inspiration from the whristold-eeighted CRPS (Gneiting& & Ranjan, 201 by using ve as its chaining function (Allen et al., 2023):
twCRPS oc (Xp.x)) xF) = CRPS (vo(XE), (c(x)))
where Xp is the forecast of X be evaluated. Since, by construction, the ground-truth x always satisfy the constraints, we have (x(xx) 0. But since we do not care only about whether forecasts break constraints, we sum both the original CRPS and this twCRPS, but we weight the later by factor of 3 = 10, to denote the additional interest we show to these errors. See Appendix E.4 for the various ve used in the benchmark.
One common approach to normalize the CRPS to get similar ranges for multiple problems is to divide it by the mean absolute value of the target ground-truth of the forecasted series (Alexandrov et al., 2020). This has two issues: the metric no longer proper, and leads to much larger values for series close to zero than those far from it. To solve the first issue, we take advantage that we can generate many more instances from each of our tasks, by computing normalization factor from 25 instances not included in the benchmark. The details of this calculations are in Appendix E.
Many tasks in our benchmark contains contextual information which is highly relevant for small fraction of the time steps in the forecasting window, while being only marginally relevant for the majority of the time steps. If we were to weight these two categories equally, then the score for a model which ignores the context would be hard to distinguish from the score of one who does not. We correct this issue by identifying the subset of time steps with relevant information, which we call the Region of Interest (Rol). We then weight the CRPS to give half weight to the Rol time steps and half weight to the non-Rol time steps. Therefore, we obtain our metric, which we call the ofgiionoo-IIneeeee CRPS (RCRPS):
ax
Σ CRPS ai) + प
Σ CRPS X x.) + 3- (RPS(vc(XE).0) IE-I
if [ >
0
(CRPS(XF,XF)+
EE
Σ
CRPS X,, 2:) + 3. CRPS( (((((
[ =
where I is the of time steps in the Rol,- -I is the set of time steps in the forecast but not in the Rol, is the aforementioned scaling factor, and we drop the factor of two and the first sum for tasks where there is meaningful Rol.
E. Scaling for cross-task aggregation
The rationale behind scaling the RCPRS is to allow us to average its value from diverse tasks without the average being dominated by the forecast quality for tasks with time series with large values. An alternative argument is: all other conditions being equal, a forecaster that is wrong by 10 in its forecast for time series which goes from 25 to 30 is worse than one that is wrong by 100 in its forecast for a time series which goes from 2500 to 3000. Furthermore, we have multiple tasks for which some instances have constant Xp or nearly so, often with values close to zero Due to these tasks, we cannot simply use a scaling which only depends on said instances Xp. Instead, we take advantage of our benchmark ability to generate a very large number of instances for each tasks by using M = 25 instances not included in our benchmark. Given
55

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
the ground-truth future values x for these instance, the scaling factor 3 for an individual task is follow:
(max; 2."" min, aT")
(2)
M
Properness In an ideal scenario, all instances of tasks would be fully independent. In that case then Equation (2) would not contain any information about the target time series in the benchmark instances, making the RCPRS a proper scoring rule. However, due to possible overlaps in the time windows used when creating the instances and to auto-correlan correlaorla,,,ns, we cannot guarantee independence between instances, and thus we cannot guarantee that the RCPRS is actually proper scoring rule. Note that this deviation from a proper scoring rule minor, and has much smaller effect than the one due the common approach of normalizing the CRPS using the Mean Absolute Value of the ground-truth.
E.2. CRPS and twCRPS
Given a univariate forecast X and a ground-trute truth realization z, the Continuous Ranked Probability Score (CRPS) can be defined in its integral as follow:
CRPS X x) / dy x (v) 1v 2 x))²
where 4 x (1) is the Cumulative Distribution Function of X, and 1 is the indicator function.
There are multiple ways to compute the CRPS, but particularly interesting one which showcases its link to the Mean Absolute Error the energy form of the CRPS:
CRPS(X, Ex XX - 2| ,
x |X XX-X
(4)
We get the threshold-weighte CRPS (tWCRPS) from Equation (4) by adding weighting function w(x) to it:
twCRPS(X, | dyw(y)
(v) ( 2 x)]²
(5)
To get the energy form of the twCRPS, we must compute the chaining function v(x) from w((x):
v(x)- u(x') f (2,,))) dyw(y).
(6)
Using v(x), we can write the twCRPS as:
twCRPS(X, z) = E Ex |u(X) v(x)] -m x |o(X)- -(X')-
Equation (7) can readily be generalized to a multivariate forecast, by using any Rd - IR chaining function.
E.3. Estimating the CRPS using samples
Computing the CRPS using Equation (3) or Equation (4) directly would be extremely hard for most of the baselines included in our experiments. Instead, it is more computationally convenient to use estimator of the CRPS which uses finite number of samples Ii, ... TM from the forecasting distribution. An unbiased estimator of the CRPS created from Equation (4) is:
M
M
M
CRPS(X,
Σ
FA - x
Σ
Fnn
(8)
M
2M - 1)
1=
n=ln'=l
However, this estimator is relatively costly, having O(M²) time complexity.
A faster estimator which gives the same result as Equation (8) (up to numerical accuracy) is the one based on the probability weighted moment form of the CRPS (Taillardat et al., 2016; Zamo & Naveau, 2018):
M
M
M
CRPS(X,,) R
1
In
+
Σ
In
2
(n 1)
(9)
M
M
M(M - 1) nm
12
56

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
where the Tn have been sorted in ascending order. We used Equation (9) in our metric, since it is accurate as Equation (8), while only having a O(M log M) time complexity.
E.4. Constraint-violation functions
In selecting constraint-violation functions ve for our various tasks, have the following requirements: it should be invariant to the number of timesteps in the forecasting window and it should be multiplied by if all numerical data in task is transformed using + + B. Here are the te use in some of our benchmark tasks:
Constant upper-bound constraint 24 r+:
vc(xr) T
E it+1
max(0, 22 - -r+),
Constant lower-hound constraint T; 2 T
vc(xF)
max - xz),
T
Constant lower-bound and upper- upper-bound constraints T S xj S +t:
vc(xF) T i=
max(0, - xz) + max(0, +*),
and variable upper-bound constraints, on a subset of time steps V; S Vi E C:
eo (x()))
max(0, 24 - 7.
|C'
IEC
E.5. Covariance of two CRPS estimators
One approach compute standard error on the RCRPS is to compute the empirical standard deviation based the instances for each task. However, such method would overestimate the standard error, since it would consider both the variance coming from the selection of instances of given task, and the variance coming from the models sampling processes. Since all models are tested using the exact same instances, the variance coming from their selection is not relevant, and thus we need a way to ignore
To do so, we take advantage that the RCRPS is weighted sum of multiple CRPS estimates. Since those estimates are not independent from one another, we can compute an estimate of the variance of the RCPRS under the sampling process by computing estimate of the covariance matrix between the various CRPS estimates, followed by the appropriate weighted
Let says we want to compute the covariance between the CRPS for variable and the CRPS for variable j. using M independent and identically distributed samples from the joint distribution of X; and X3.
Cov (CRPS/ Nr, x.) CRPS( X {),0;))
Cov
Σ
Xin xill
Σ ngttn
X - Xznncb
M
2M 1)
n
1
Σ
jn - Iz5
1
E1
M
2M
non'
where the sums are over the various samples and x; and xy are the ground-truth values.
57

---

Context is Key: A Benchmark for Forecasting with Essential Textual Information
After some tedious algebraic manipulations, we obtain the final formula for the covariance of two CRPS estimates:
Cov (CRPS( Xx, ,CRPS( X {;)
1
E
X
E
x
M
Xx
E
-
E X3 X"
E
X
X
M X,
1
E
E
XX
X
XX
M X; X
X5
2M 3
EE
[X
X
E E 1X"
XX
2M(M - 1)
X; X;
x2 X5"
1
E
1x:-4l -1,1 - zzl
M
(X..X))
1
E
E
- 1X, zzg
M
(X.,X,)
X
1
E
E
X
23
1 X11
M
(X,,X,)
X3
E
E
X X 18,
2M - 1) (X..X,)
(X,,XX
M
E
E
X
-1
X1I
M(M - XX
where variables with the same number of apostrophes (') are drawn together and those with different number of apostrophes are independent variables.
To get estimate of the covariance using our M samples, we can estimate each of these terms using their respective unbiased estimators. Once we have computed an estimate of the variance for a single task instance, the overall variance for full task is computed using the formula for the variance of the average of multiple independent variables. One slight disadvantage of using this method is that it offers no guarantee that the RCPRS variance estimate will be non-negative, so in the rare cases where the estimate for the variance of a full task is negative, we clip it to 0.
E.6. Comparison of Statistical Properties of Various Scoring Rules
Table 12, Comparison of Statistical Propertics of Various Scoring Rules. The * indicates that, be proper, it would need that different seeds are independent, which cannot be guaranteed by CiK, but could happen in other applications.
Invariance
Metric
Proper Scoring Rule
Domain
Additive
Multiplicative
Brier Score
Yes
Discrete
Yes
Yes
CRPS
Yes
Continuous
Yes
No
TwCRPS
Yes
Continuous
Yes
No
CRPS skill score
No
Continuous
Yes
Yes
MAV-Scaled CRPS
No
Continuous
No
Yes
RCRPS
N
Continuous
Yes
Yes
Tab. 12 describes a few statistical properties for both commonly used scoring rules and our RCPRS. For the invariance (additive and multiplicative) properties, we indicate whether the scoring rule remains unchanged if all relevant quantities (forecast, ground truth, threshold, and constraint parameters) are modified by adding a constant to them or by multiplying them by constant, By Scaeec CRPS, we denote the common approach in the forecasting literature to normalize the CRPS by dividing by the Mean Absolute Values of the ground-truth, instead of reporting the original CRPS values.
58


Reviews:
[{"review_id": "mg4riBy0cS", "reviewer": "Reviewer_m7ph", "summary": "This paper introduces \"Context is Key\" (CiK), a benchmark for time-series forecasting models that incorporate textual context alongside numerical data. The authors design 71 tasks across seven domains where textual information is essential for accurate forecasting, propose a Region of Interest CRPS (RCRPS) evaluation metric, and evaluate statistical models, time series foundation models, and LLM-based forecasters. Their \"Direct Prompt\" method performs better than other tested approaches, showing the value of textual context while highlighting current limitations.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough feedback. We appreciate the recognition of our contributions, including the comprehensive empirical evaluations, task diversity and quality, the soundness of the proposed metric, and the thoroughness in handling data contaminations. Below, we clarify several points raised in the review.\n\n---\n\n## Transformations to prevent data contamination\n\nWe acknowledge your concerns. We present the time series domains and the exact transformations used in App A.1. We include a summary table [here](https://raw.githubusercontent.com/anon-forecast/benchmark_report_dev/refs/heads/main/rebuttal_images/data_transformation_table.png).\n\nWe’d like to note that a significant portion of our tasks did not require any transformation. And wherever required, mitigation strategies were chosen extremely carefully and applied intelligently (the exact details of which are in App A.1). The tasks where we shifted the date by 24 hours are tasks with solar irradiance data, where the date shift would have minimal impact as the seasonal effects change smoothly over the year. For the tasks to which we added a very small amount of gaussian noise, we visually inspected that the noise barely changed the data.\n\nWe’d also like to emphasize that the code uses transformations as an option, and the tasks can be obtained without the transformation too. Nevertheless, we will explicitly point to these transformations and their potential impact in the main text, as suggested. For the camera-ready version, we will also analyze the performance of all models to the sensitivity to the transformations.\n\n---\n\n## Failure pattern analysis\n\n- [**Specific linguistic patterns and systematic failures**] We thank the reviewer for this point. The analysis of specific linguistic patterns could yield interesting further insights. More analyses are also required to understand systematic failure specific models (such as for DP - GPT-4o with scientific notation). We propose to add a note on these points in the future work.  \n- [**Additional analyses**] We would like to point out that we provide the number of catastrophic failures per model (in App. C.5.). In the camera-ready version, we will add a view of this table broken down by context type and domain.\n\n---\n\n## Other comments\n\nThank you for bringing these aspects to our attention. \n\n- [**Computational costs**] Apart from the pareto analysis which was aimed at this direction (Fig 7), we currently also provide the average inference time of all models per task (App C.4). We propose to further complement this analysis that provides the per-timestep average inference time, enabling estimation of real-world performance. We are open to any other suggestions.  \n- [**Bidirectional interactions**] We focus on the forecasting task in CiK, but other tasks that use numerical data to predict text are very relevant as well. We will add this point to future work.  \n- [**How models can better use contextual information**] We believe encouraging models to “think” or reason about the context explicitly can be useful here. We also believe fine tuning LLMs on datasets with text and time series would allow them to generalize better to tasks in CiK. We add these directions in our future work section.\n- [**Domain-specific contexts**] Extending CiK to build domain-specific tasks and benchmarks is definitely a valuable direction of future work that will present additional changes in context integration. We will add this point to future work.\n---\n\n## Clarifications to other questions\n\n- [**Impact of instruction tuning**] We agree that the differences in the performance of LLMP and DP may point to fundamental differences in how the methods use contextual information. We hypothesize that, while the instruction-tuned models respond better to DP’s direct instructions, the iterative prediction strategy used by LLMP may rely on a base (non instruction-tuned) model’s proper calibration for forecasting, which might be more calibrated (as instruction-tuning might affect calibration as per [the GPT-4 technical report](https://arxiv.org/abs/2303.08774) (Fig 8)). We agree that this requires further investigation and leave it to future work.\n- [**Compression**] We have not tested this but agree that it is a very interesting idea, especially given recent works on LLM-based lossless compression such as [FineZip](https://arxiv.org/abs/2409.17141). We will add this point to future work.\n- [**Structured context**] While we restrain this work to study textual context, the codebase is extensible and natively supports adding more modalities. We agree this is a very interesting direction of future work, that we have already noted in the future work section.  \n- [**Context-source specific models**] We believe both approaches, i.e. unified models for context-aided forecasting, and ensemble approaches such as expert-based methods and [learning to defer](https://arxiv.org/abs/1711.06664) merit investigation. We will add this point to future work.", "claims_and_evidence": "The paper's main claims are backed by empirical results:\n- The authors demonstrate context necessity through their task design, with 95% of instances confirmed by evaluators as benefiting from contextual information.\n- Figure 6 shows quantified performance gains from textual context, with larger models seeing significant improvements (67.1% for Llama-3.1-405B-Inst), with statistical significance confirmed in Appendix C.6.\n- The Direct Prompt method outperforms other approaches in comprehensive evaluations (Table 1), though no single method excels across all context types.\n\nHowever, I have concerns about real-world applicability due to the data transformations used to mitigate memorization. The authors apply several techniques described in Appendix A.1, including prioritizing live data sources, using derived time series, and adding noise or shifting timestamps. While these modifications help prevent LLMs from leveraging memorized data during evaluation, they create an artificial evaluation setting that may not reflect performance on unmodified real-world data. For instance, shifting timestamps could disrupt natural seasonal patterns that models might otherwise leverage. The paper claims these transformations are used \"sparingly,\" but provides no quantitative analysis of how extensively they were applied or how they might affect model behavior. This creates a disconnect between benchmark performance and what practitioners might experience in deployment scenarios. The main text should explicitly acknowledge this limitation rather than burying it in an appendix, ideally with ablation studies showing performance differences between original and transformed data where possible.", "methods_and_evaluation_criteria": "The benchmark draws from diverse data sources and includes various context types (intemporal, future, historical, covariate, causal). The authors address memorization issues by applying transformations to the data, though this potentially affects realism.\n\nThe RCRPS metric improves on standard metrics by accounting for context-sensitive regions and constraint satisfaction, making it more suitable for evaluating contextual forecasting.\n\nThe experiment design uses multiple instances per task and multiple forecasts per instance, with a weighting scheme to handle task similarity clusters. This helps provide statistical reliability.\n\nThe model comparison covers different architectural approaches and scales, allowing for meaningful performance comparisons across the forecasting methodology spectrum.", "theoretical_claims": "The paper is mostly empirical. The RCRPS formulation extending CRPS is mathematically sound. The context-aided forecasting problem formalized in probabilistic terms in Section 2 seems correct.", "experimental_designs_or_analyses": "The task creation process addresses data contamination through several methods. The sample size (5 instances × 25 forecasts per task) seems adequate for statistical analysis.\n\nTable 1 shows comprehensive evaluation results with significance testing in the appendix. The failure analysis in §5.4 reveals important model limitations, like GPT-4o struggling with scientific notation.\n\nThe decision to clip RCRPS scores for major failures at 5 prevents outliers from dominating results but also hides how badly models can fail. The authors acknowledge this trade-off and include win-rate analyses that aren't affected by clipping.\n\nFigure 7's parameter-efficiency analysis provides practical deployment insights beyond raw performance numbers.", "relation_to_broader_scientific_literature": "The paper positions itself between numerical approaches (time series foundation models) and language-integrated methods. It acknowledges prior benchmarks (Merrill et al., 2024; Zhang et al., 2023; Liu et al., 2024a) while explaining CiK's focus on essential contextual information.\n\nThe Direct Prompt approach builds on prior prompt-based forecasting methods (Requeima et al., 2024; Gruver et al., 2024), though the paper could better discuss architectural innovations in multimodal forecasting.", "essential_references_not_discussed": "None that I'm aware of.", "other_strengths_and_weaknesses": "Strengths:\n\n- Manual creation and validation of tasks ensures quality\n- Context type taxonomy provides structure for analyzing text-forecast relationships\n- Model failure and inference cost analysis offers practical insights\n\nWeaknesses:\n\n- Missing analysis of which linguistic patterns most impact forecast quality\n- Limited exploration of computational costs for real-time applications\n- One-way focus (text informing forecasts) misses potential bidirectional interactions\n- Limited insight into how models could better use contextual information", "other_comments_or_suggestions": "The paper could benefit from a more systematic analysis of the failture patterns across context types and model architectures. Developing clearer guidelines for matching specific context types to the most appropriate models based on your findings could also be useful. Have you considered extensions to domain-specific ontexts like healthcare or finance? The specialized terminology may present additional challenges for contextual integration.", "questions_for_authors": "1. What specific aspects of instruction tuning hurt LLMP performance while not affecting Direct Prompt? Does this suggest fundamental differences in how these methods use model capabilities?\n2. Besides scientific notation issues, have you found other systematic failure patterns that could guide model selection?\n3. Have you explored compression techniques that might preserve contextual integration while reducing computational costs?\n4. How might the benchmark extend to structured data formats (databases, tables) common in operational forecasting?\n5. Since no method excels across all context types, would you recommend ensemble approaches based on context detection, or do you think unified models could overcome these limitations?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r3AEl7Fuwh", "reviewer": "Reviewer_F7ED", "summary": "The paper introduces Context is Key (CiK), which is a benchmark aiming to evaluate forecasting models’ ability to integrate both numerical time-series data and essential textual context. Unlike traditional forecasting benchmarks that rely solely on numerical data, CiK explicitly requires models to process and leverage natural language context to improve prediction accuracy. CiK consists of 71 manually designed forecasting tasks spanning seven real-world domains (climatology, economics, energy, mechanics, public safety, transportation, and retail). The authors evaluate statistical models, time-series foundation models, and large language model-based forecasters. Notably, the authors introduced a new forecasting metric, RCRPS, to specifically evaluate forecasts when context is relevant.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful response. We are grateful that you highlighted the thoroughness of our experiments, the value of the RCRPS metric, and the quality of the writing. We are pleased that you see the CiK benchmark as filling a critical gap in the literature, with real-world implications across multiple industries that rely on additional non-numerical information for forecasting.\n\nBelow, we present point by point clarifications to your concerns. We are happy to clarify the points further.\n\n---\n\n## ***Formal theoretical proofs to support claims about context-aided forecasting***\n\n### *RCRPS*\n\nWe thank the reviewer for highlighting the added value of a formal justification of the RCRPS. We emphasize that we give further details on the RCRPS metric and why it is a good choice of metric for context-aided forecasting in Appendix E. We propose the following:\n\n\\[**Addition to Main Text from Appendix E**\\] We will include key elements of Appendix E in the main text, including the desiderata for the RCRPS and statistical properties such as properness. \n\n\\[**Additional Appendix Section: Statistical Properties**\\] We will include in the Appendix a detailed comparison of the statistical properties of different metrics, including the CRPS, CRPSS, twCRPS, Brier Score, and the proposed RCRPS. \n\n### *Formal characterization of the context-aided forecasting setting*\n\nThank you for your suggestion regarding a formal characterization of the problem setting of context-aided forecasting. We emphasize that the problem setting is defined formally in Section 2, according to which, we are interested in models where, in expectation, forecasts that leverage context perform better. Critically, all tasks in the benchmark are designed accordingly.\n\n**We seek clarification on this.** We are happy to include aspects of formalization beyond what is discussed in the problem setting. Please let us know what you had in mind.\n\n---\n\n## ***Related Work***\n\n\\[**Addition to Related Work**\\] Thank you for providing these valuable references; we will make sure to reference them all in the revised paper.\n\n---\n\n## ***Clarifications to Questions***\n\n\\[**Time series alignment strategies for LLMs**\\] Thank you for your question. We hope that the analyses we presented are of use:  \n\n- \\[**Sensitivity to input format and post-training**\\] We find that the forecasting performance of the LLMs is sensitive to the input-output format, as shown by differences between the LLMP and DP methods which each prompt the model differently. We also find that the post-training strategy plays a role, as can be evidenced by the gap in performance of models with the DP and LLMP models. Sec 5.3 (“Comparing LLMP and DIRECT PROMPT”) contains more details on both these results. Both these factors could play important roles in the ability of LLMs to process and forecast time series.   \n- \\[**Addition to Future Work**\\] Future work that explores better input/output formatting strategies would benefit the community. As also suggested by reviewer M7ph, the gap between DP and LLMP suggests that prompting strategies might elicit different capabilities too. Further, fine-tuning LLMs on time series data may further adapt them to process time series better. We leave all these directions to future work. \n\n\\[**Impact of the ability of LLMs to natively process time series**\\] Thank you for this insightful question. \n\n- **\\[Comparison of performance of LLMs to other models in the no-context setup\\]**  We compare the performance of LLMs to that of quantitative models that can natively process time series (such as foundation models and statistical models) in a no-context setup where all models consume the same input (only the numerical historical data). These results are presented in App C.1. We find that many of the LLMs (with both DP and LLMP) are competitive and sometimes better than the quantitative models. With that said, as all LLMs consider time series as text, we agree that the lower performance of some LLMs could be due to their inability to “natively” process time series, among other reasons (such as poor forecasting ability). Improving the ability of LLMs to process time series is an active area of research, and we believe CiK can be of high value in this front.  \n- \\[**Extended discussion of catastrophic failures**\\] While successful examples of LLMs generating good forecasts hints at the capability to process time series data as text, the catastrophic failures (Sec 5.4)  may suggest that LLMs are not extremely reliable when processing time series, and more research is needed in understanding when they can fail catastrophically.", "claims_and_evidence": "Firstly, the authors explicitly designed various tasks where textual context is necessary to make accurate forecasts. And with human and LLM-based evaluations, it further confirmed that with contextual guidance, the forecast quality is improved. The experiments are comprehensive and detailed.", "methods_and_evaluation_criteria": "The experiments in this paper are exceptionally well-designed and detailed, effectively filling a critical gap in the literature by providing time series data paired with contextual information—a resource that has been largely missing until now. Additionally, the authors introduce a new evaluation metric, RCRPS, which prioritizes context-sensitive forecast accuracy. This ensures that models are assessed based on their ability to effectively integrate and utilize textual context, making the evaluation more meaningful and relevant to real-world forecasting scenarios.", "theoretical_claims": "The paper does not include formal theoretical proofs to support its claims about context-aided forecasting. Instead, it relies primarily on empirical evidence to demonstrate that textual context improves forecasting accuracy. While RCRPS is intuitively appealing, the paper does not formally prove that it is optimal metric for context-aware forecasting.", "experimental_designs_or_analyses": "The paper’s experimental designs are well-structured and comprehensive, which are one of the key strengths. The paper evaluates a wide range of models, including Statistical Models (ARIMA, ETS, Exponential Smoothing), Time-Series Foundation Models (Lag-Llama, Chronos, Moirai, TimeGEN), LLMs (GPT-4o, Llama-3, Mixtral-8x7B, Qwen-2.5), Multimodal Models (UniTime, Time-LLM). This allows for a fair and meaningful comparison between traditional, modern, and context-aware forecasting methods. Additionally, the authors conducted detailed ablation studies. For example, in figure 5, it shows that removing textual context significantly reduces forecasting accuracy. And the authors also designed a new evaluation metrics specifically for context-aware forecasting. They also included various examples tasks, which would be really good for further implementation.", "relation_to_broader_scientific_literature": "This paper is one of the early contributions in the emerging field of integrating contextual information with time-series forecasting, primarily due to the lack of paired datasets in this domain. It effectively fills a gap by introducing a benchmark specifically designed to test models’ ability to incorporate textual context into forecasting. Additionally, the lack of appropriate evaluation metrics has been a significant challenge in this area. The authors address this by proposing Region of Interest CRPS, which has the potential to enhance the evaluation of context-aware forecasting models.", "essential_references_not_discussed": "Essential references are discussed but several suggested references are listed in Suggestion section for time series reasoning.", "other_strengths_and_weaknesses": "The paper is among the first to systematically evaluate how textual context improves time-series forecasting. The CiK benchmark fills a major gap in the literature by providing a dataset where context is essential. The proposed evaluation metric (RCRPS) is specifically designed for context aware. The findings have practical implications for industries like finance, energy, and public safety, where human forecasters already use external knowledge to refine numerical predictions. The paper is well-structured and clearly written, with detailed explanations of the benchmark design, evaluation methods, and experimental setup. The supplementary material provides extensive details on dataset construction, evaluation metrics, and failure cases, which ensures reproducibility.\n\nThe weakness lies on the lack of theoretical justification. If a theoretical framework could be designed (for example using Bayesian reasoning or causal inference), this would strengthen the argument. RCRPS is well-motivated but not formally compared against other scores (e.g., CRPSS, Brier Score) in terms of statistical properties.", "other_comments_or_suggestions": "In addition to citing Merrill et al.’s work on the challenges large language models (LLMs) face in zero-shot time-series reasoning, several other studies have explored this area:\n\n1. “Towards Time-Series Reasoning with LLMs”: they propose a novel multi-modal approach that integrates a lightweight time-series encoder with an LLM. \n\n2. “Beyond Forecasting: Compositional Time Series Reasoning for End-to-End Task Execution”: they introduce a program-aided inference agent that leverages LLMs’ reasoning capabilities to decompose complex time-series tasks into structured execution pipelines. \n\n3. “Implicit Reasoning in Deep Time Series Forecasting”: This study delves into how deep learning models implicitly perform reasoning during time-series forecasting, shedding light on the internal mechanisms that contribute to their predictive capabilities.\n\t\n4. “XForecast: Evaluating Natural Language Explanations for Time Series Forecasting”: The authors of this paper focus on the generation and evaluation of natural language explanations accompanying time-series forecasts, aiming to enhance the interpretability and transparency of predictive models.\n\n5. “Position: Empowering Time Series Reasoning with Multimodal LLMs”: This work advocates for the integration of multimodal data—such as textual descriptions, visual data, and audio signals—with LLMs to bolster time-series reasoning.", "questions_for_authors": "1. Did you experiment with any alignment strategies to help LLMs better process time-series data?\n2. Could the lower performance of certain LLM-based forecasting approaches be attributed to the fact that LLMs struggle to natively process time-series sequences?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fmaQYkPphk", "reviewer": "Reviewer_hoT6", "summary": "This paper introduces the \"Context is Key\" benchmark to evaluate the capability of models in leveraging textual information for time series forecasting. By designing 71 tasks and evaluating them with both human and large language model (LLM) annotators, the study confirms the significant role of contextual information in enhancing prediction accuracy, especially during relevant time periods. The authors propose a novel evaluation metric combining CRPS and twCRPS to more accurately assess model performance. Experimental results show that providing contextual information significantly improves forecasting quality, offering valuable insights for future developments in multimodal forecasting and automated predictive systems.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful response. We appreciate your recognition of our work as addressing a gap in the field, offering valuable insights for future developments, and providing clear and convincing findings. We address your concerns below and are happy to clarify any further points.\n\n---\n\n\n## ***Training Approach of Time-LLM and UniTime***\n\nThank you for your questions regarding the training approaches for Time-LLM and UniTime. We included Time-LLM and UniTime in our evaluation due to their respective contributions to multimodal forecasting. These models require training to adapt them for context-aided forecasting, however CiK is an evaluation benchmark, and has no training set.  Therefore, we rely on the original papers’ training recipes to train their models, which adapt them for specific time series datasets with paired templates of context text. We discuss these limitations of the models in Appendix D.3 (“Why Do Time-LLM and UniTime Not Benefit (More) From Context?”). To make this more clear, we propose to:\n\n- \\[**Expanded Results Discussion**\\] Add to the main text an abridged version of the discussion in Appendix D.3, which describes the limitations we faced when evaluating Time-LLM and UniTime. \n\nFinally, if you can suggest a more appropriate training recipe, we are happy to rerun the models and report the results in the camera-ready version.\n\n---\n\n## ***Stringent comparison setups for Time-LLM and UniTime, and potential overestimation of the importance of event text***\n\nThank you for pointing this out. We acknowledge the reviewer’s claim that the comparison setups for Time-LLM and UniTime are stringent, and may fail to fully leverage the strength of the baseline models. We attribute this to the training approaches of the respective models (described above), which adapt the baseline models’ capabilities to process contextual text in highly specialized templates, and for specific time series domains. CiK is however intended to evaluate context-aided forecasting across several time series domains and types of textual context, which is a different, more broader setup compared to those used in the papers of Time-LLM and UniTime. The training dataset of the models would have to be much more diverse to enable these models to generalize well enough to perform well on CiK, which contains linguistic variations and requires many types of reasoning. To make this more clear, we propose to \n\n- \\[**Addition to Limitations**\\] Add to limitations the impact of the lack of training datasets when evaluating dataset-specific approaches, such as UniTime and Time-LLM.  \n- \\[**Addition to Future Work**\\] Add to future work the need for training datasets for context-aided forecasting.\n\n---\n\n## ***Additional baselines***\n\n\\[**TimeMMD**\\] We contacted the authors of [Time-MMD](https://arxiv.org/abs/2406.08627) for access to checkpoints of models from their paper, but have yet to hear back. Unfortunately, we found that their paper does not contain the hyperparameters used to train these models and thus we could not recover their checkpoints ourselves. We will however add the results of these models to the paper if we receive the checkpoints or manage to reproduce their original results.\n\n\\[**ChatTime**\\] Thank you for these suggestions. We evaluate the publicly available checkpoints ([Base](https://huggingface.co/ChengsenWang/ChatTime-1-7B-Base), [Chat](https://huggingface.co/ChengsenWang/ChatTime-1-7B-Chat)) of [ChatTime](https://arxiv.org/abs/2412.11376) on CiK. Here are the aggregate results:\n\n| Model | Average RCRPS (± std) |\n|---|--|\n|**With Context**||\n| ChatTime-Base | 0.735 ± 0.002 |\n| ChatTime-Chat | 0.747 ± 0.005 |\n|**Without Context**||\n| ChatTime-Base | 0.725 ± 0.002 |\n| ChatTime-Chat | 0.781 ± 0.015 |\n\nCompared to other models evaluated on CiK (shown in Table 1), these results (with and without context) are not competitive and are, in fact, much worse than those of statistical models that cannot process context. ChatTime-Base shows a very small improvement with context, while ChatTime-Chat degrades with context. This is likely because ChatTime’s training dataset is limited to 3 time series datasets with highly specific contextual text templates. Similar to Time-LLM and UniTime, the training approach likely fails to generalize outside its training domains, and fails to leverage the strength of the baseline models. We will add these results to the paper.", "claims_and_evidence": "This paper provides evidence supporting the significant role of contextual information in time series forecasting by designing 71 tasks and evaluating them with both human and large language model (LLM) annotators. It introduces a novel evaluation metric (CRPS and twCRPS) to more accurately measure prediction performance. Experimental results show that providing contextual information improves forecasting quality, demonstrating the effectiveness of this approach. Overall, the main findings of the paper are clear and convincing, highlighting the critical importance of context in forecasting tasks.", "methods_and_evaluation_criteria": "This paper collects a large number of datasets and designs reasonable evaluation metrics, filling a gap in the field.", "theoretical_claims": "I have reviewed the correctness of the theoretical proofs in this paper.", "experimental_designs_or_analyses": "The experiments in this paper are relatively comprehensive. However, I have reservations about the training approach for UniTime and Time-LLM. Such a stringent comparison setup fails to fully leverage the strength of the baseline models and may lead to an overestimation of the role of event text. Furthermore, if the authors could include some multimodal time series forecasting baselines, such as dataset-specific models (e.g. TimeMMD) and foundational models (e.g. ChatTime), it would make the work even more complete.", "relation_to_broader_scientific_literature": "This paper further advances the development of multimodal time series forecasting.", "essential_references_not_discussed": "I believe the related work section of this paper is sufficient.", "other_strengths_and_weaknesses": "Refer to the above comments.", "other_comments_or_suggestions": "Refer to the above comments.", "questions_for_authors": "Refer to the above comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper received three reviews, all of which were positive (two weak accepts and one accept). Reviewers commended the paper’s comprehensive experimental design, which spans 71 tasks across seven domains, where textual context is essential for accurate time series forecasting. The paper proposes a region-of-interest variant of the CRPS evaluation metric (RCRPS), and provides a structured evaluation of classical statistical models, time series foundation models, and  LLM-based predictors. Overall, the reviewers recognized this work as one of the early contributions to the emerging area of integrating contextual information into time series forecasting.

The authors provided a rebuttal addressing the reviewers' comments, which were mostly requests for clarification and additional analysis. No reviewer raised further objections or concerns during the discussion phase. The consensus remained positive.

The AC concurs with the reviewers' assessments. This work offers a timely and well-structured benchmark in a rapidly growing area, and makes a valuable contribution by proposing a new metric and systematically evaluating a broad range of models. AC recommends acceptance. The AC would also like to note that a shorter version of this work was presented at NeurIPS 2024 Workshop. After confirming that the workshop paper was under 4 pages and does not violate the dual submission policy outlined at https://icml.cc/Conferences/2025/CallForPapers, AC discussed the matter with the SAC and both agreed that no further action is necessary.
Decision: Accept (poster)