Context is Key: A Benchmark for Forecasting with Essential Textual Information

## Abstract

Abstract Forecasting is a critical task in decision making across various domains. While numerical data provides a foundation, it often lacks crucial context necessary for accurate predictions. Human forecasters frequently rely on additional information, such as background knowledge or constraints, which can be efficiently communicated through natural language. However, the ability of existing forecasting models to effectively integrate this textual information remains an open question.
To address this, we introduce “Context is Key” (CiK), a time series forecasting benchmark that pairs numerical data with diverse types of carefully crafted textual context, requiring models to integrate both modalities.
We evaluate a range of approaches, including statistical models, time series foundation models, and LLM-based forecasters, and propose a simple yet effective LLM prompting method that outperforms all other tested methods on our benchmark.
Our experiments highlight the importance of incorporating contextual information, demonstrate surprising performance when using LLM-based forecasting models, and also reveal some of their critical shortcomings.
By presenting this benchmark, we aim to advance multimodal forecasting, promoting models that are both accurate and accessible to decision-makers with varied technical expertise.
The benchmark can be visualized at https://servicenow.github.io/context-is-key-forecasting/v0/ .

### 1 Introduction

The estimation of future conditions is the foundation of decision making (Hyndman & Athanasopoulos, ) and intelligence (Wang, ).
Articulated as time-series forecasting, this problem pervades many domains of science and commerce.
Accurate forecasting relies on several crucial decisions up to the practitioner (Hyndman & Athanasopoulos, ), in particular on:
1. *Model selection*: Choosing the appropriate forecasting model for a given problem, and 2. *Incorporating prior information*: Determining what relevant information should be included in the model and how to effectively integrate it.
This involves decisions about statistical priors, inductive biases in the model architecture, and other forms of domain knowledge integration. Traditionally, these decisions have heavily relied on expert knowledge and manual intervention.
However, recent advancements in machine learning have shown particular promise in democratizing time-series forecasting by automating both model selection and the integration of prior information.

In the wake of the foundation model paradigm shift (Bommasani et al., ), several works (e.g.,  Liang et al. (); Chen et al. (); Lim & Zohren ()) have addressed automatic model selection by learning flexible, adaptable models that can be applied across various problem scenarios.
Unfortunately, when compared to traditional statistical methods, current approaches provide debatable performance improvements while requiring significantly more resources (Garza & Mergenthaler-Canseco, ).
Moreover, these models typically cast inputs and outputs as purely numerical time series, which leaves no room for the context that human experts typically rely on to focus their modelling efforts.

Figure: Figure 1: An example task from the proposed Context is Key (CiK) benchmark with forecasts produced by a context-aware model. Left: Using the numerical history alone leads to poor forecasts, as nothing indicates a reversion to zero. Right: Awareness of the context significantly improves the forecasts as it reveals that no power will be produced during the night (via deductive reasoning) and enables estimating the peak hour of production.
Refer to caption: x1.png

An alternative class of recent approaches (Jin et al., ; Liu et al., ; Requeima et al., ) adapt large language models (LLMs) for forecasting and leverage natural language as an intuitive interface to integrate side information.
These methods overcome a significant limitation of traditional forecasting techniques by
eliminating the need to manually encode priors or design specialized models.
They further hold the promise to capture a broader range of prior knowledge and context, potentially leading to more comprehensive and accurate forecasts.
Unfortunately, there are as of yet no systematic evaluations of these models’ abilities to jointly leverage historical observations and natural language for forecasting.
While several benchmarks for context-aided forecasting have been recently released (Zhang et al., ; Liu et al., ; Xu et al., ; Emami et al., ; Merrill et al., ), their contexts are not guaranteed to be useful for improving performance.
Hence, it is still an open question as to what extent existing methods can improve their predictions by leveraging crucially-relevant information provided in textual form.

To this end, we propose the Context Is Key (CiK, pronounced *kick*) benchmark of forecasting tasks with numerical input-output pairs and essential textual context.
The benchmark is designed to assess a forecaster’s ability to utilize both the numerical data and key information contained within the accompanying text, as the accuracy of the forecasts relies heavily on effectively leveraging both; see Fig. 1 for an example where context is imperative to forecast accuracy.

Our contributions are:

- •
We manually curate and release 71 forecasting tasks (Sec. 3) spanning 7 domains,
which cover various kinds of contextual information (Sec. 3.2), and in addition to basic natural language-processing and time-series analysis, require various capabilities (Sec. 3.3).
- •
We introduce the Region of Interest CRPS metric (RCRPS) to evaluate context-aided forecasting performance (Sec. 4), which prioritizes context-sensitive windows in the prediction and accounts for constraint satisfaction.
- •
We evaluate various methods on CiK (Sec. 5), including statistical models, time series foundation models using only numerical data, and LLM-based forecasters capable of incorporating context. We introduce Direct Prompt, a simple prompting method that achieves the best results on CiK. Our analysis explores key factors such as the impact of context conditioning, prompting techniques, model capabilities, and discusses failure modes of models.

### 2 Problem Setting

###### Context-Aided Forecasting

This work addresses the problem of *context-aided forecasting*, where the goal is to produce statistical forecasts by incorporating relevant side information (i.e., context). Let $\mathbf{X}_{H}=[X_{1},\ldots,X_{t}]$ represent a series of random variables corresponding to historical observations, where each $X_{\tau}\in\mathcal{X}\subseteq\mathbb{R}$, and let $\mathbf{X}_{F}=[X_{t+1},\ldots,X_{T}]$ represent future observations. In the classical statistical forecasting problem, the objective is to estimate the joint distribution of the future observations given the historical data:

$$ $P(\mathbf{X}_{F}\mid\mathbf{X}_{H}).$ $$

We further assume access to *context*, denoted $\mathbf{C}\in\mathcal{C}$, which is additional data of arbitrary nature ($\mathcal{C}$) that contains information relevant for predicting $\mathbf{X}_{F}$ and complementary to the history $\mathbf{X}_{H}$.
The task then becomes estimating the distribution:

$$ $P(\mathbf{X}_{F}\mid\mathbf{X}_{H},\mathbf{C}).$ $$

Crucially, we restrict our focus to *relevant context*, which we define as context that does not degrade the prediction of future time steps.
Formally, for
$\mathbf{x}_{F}\sim\mathbf{X}_{F}\mid\mathbf{X}_{H},\mathbf{C},$
given some loss function $\mathcal{L}$ assessing a predictive distribution over $\mathbf{X}_{F}$ against a realization $\mathbf{x}_{F}$, $\mathcal{L}:P(\mathbf{X}_{F})\times\mathbf{x}_{F}\rightarrow\mathbb{R}$, we are interested in systems where, in expectation, forecasts that leverage context perform better:(^1^11Using the negative log-probability as the loss function would make this statement equivalent to: the entropy of $P(\mathbf{X}_{F}\mid\mathbf{X}_{H},\mathbf{C})$ must be lower than the cross entropy of $P(\mathbf{X}_{F}\mid\mathbf{X}_{H},\mathbf{C})$ and $P(\mathbf{X}_{F}\mid\mathbf{X}_{H})$.)

$$ $\displaystyle\mathop{\mathbb{E}}_{\mathbf{x}_{F}}\mathcal{L}(P(\mathbf{X}_{F} \mid\mathbf{X}_{H},\mathbf{C}),\mathbf{x}_{F})\leq\mathop{\mathbb{E}}_{\mathbf {x}_{F}}\mathcal{L}(P(\mathbf{X}_{F}\mid\mathbf{X}_{H}),\mathbf{x}_{F}).$ $$

Furthermore, although the nature of the context $\mathcal{C}$ can vary widely, we specifically concentrate on *context communicated through natural language*.

### 3 Context is Key: a Natural Language Context-Aided Forecasting Benchmark

Figure: Figure 2: Overview: The tasks in the CiK benchmark rely on real-world numerical data, from 7 domains, as well as synthetic data (left), coupled with natural language context capturing up to 5 different aspects of the dynamical process (center), and require up to 7 non-trivial capabilities to unlock accurate forecasts (right).
Refer to caption: x2.png

We present the *Context is Key* (CiK) benchmark, a collection of probabilistic forecasting tasks that cannot be solved without integrating natural language contextual information with numerical data.
CiK consists of 71 distinct tasks spanning seven application domains (Sec. 3.1) and that can be instantiated in different ways, e.g., by changing target time series or by selecting different time windows.
These tasks encompass diverse types of contextual information (e.g., past events and known causal relationships; Sec. 3.2), and are designed such that various capabilities (e.g., causal reasoning; Sec. 3.3) are required to fully leverage the context and *unlock* accurate forecasts (see Fig. 2 for an overview). An example task is illustrated in Fig. 1 and others are given in Appendix B.
The complete set of tasks can be explored at [https://servicenow.github.io/context-is-key-forecasting/v0/](https://servicenow.github.io/context-is-key-forecasting/v0/) and the source code is available at [https://github.com/ServiceNow/context-is-key-forecasting](https://github.com/ServiceNow/context-is-key-forecasting).

#### 3.1 Domains and Numerical Data Sources

The vast majority (95%) of tasks in CiK draw numerical data from 2,644 real-world time series acquired from public sources.
These series cover a range of domains: Climatology (solar irradiance and cloud coverage (Sengupta et al., )); Economics (unemployment rates across states and counties (U.S. Bureau of Labor Statistics, )); Energy (electricity consumption and production (Godahewa et al., )); Mechanics (experimental properties of physical systems (Gamella et al., )); Public Safety (fire department intervention counts (Ville de Montréal, )); Transportation (highway segment occupancy rates and average speeds (Chen et al., )); and Retail (cash withdrawals from various ATMs (Godahewa et al., )). The remaining 5% of tasks use simulated data from dynamical systems crafted specifically for the tasks. Overall, the time series in CiK exhibit diverse sampling frequencies, with observations ranging from every 10 minutes to monthly intervals. Additional details on data sources can be found in Sec. A.1.

Memorization mitigation:
Using publicly available real-world data introduces the risk that pretrained LLMs and time-series foundation models may have memorized portions of the data, potentially inflating evaluation performance.
To mitigate this, we employ several strategies.
First, we prioritize live data sources that are continuously updated, such as Chen et al. () and Ville de Montréal (), ensuring the data is collected after the training cut-off dates of the models being evaluated.
Second, where applicable, we use derived series that are not directly available in the raw data, such as converting an incident log into time series (Ville de Montréal, ).
Finally, as a last resort, we apply minor transformations, such as adding noise or shifting timestamps, but use these sparingly to avoid misalignment between common-sense knowledge (e.g., holiday dates) and the numerical data. Details on the mitigation methods used for each data source are provided in Sec. A.1.

#### 3.2 Natural Language Context

Figure: Figure 3: Illustration of a CiK task annotated with types of natural language context: \raisebox{-.9pt}{1}⃝ The short numerical history is misleading, suggesting an increasing trend. However, contextual information compensates and enables accurate forecasts: \raisebox{-.9pt}{2}⃝ The intemporal information ($\mathbf{c}_{I}$) reveals the nature of the series, implying a seasonal pattern with greater prevalence in the summer months due to weather. \raisebox{-.9pt}{3}⃝ The historical information ($\mathbf{c}_{H}$) complements the short history by providing high-level statistics on past values. \raisebox{-.9pt}{4}⃝ The covariate information ($\mathbf{c}_{\text{\text{cov}}}$) reveals an association with another quantity: field fires, reinforcing potential seasonal behavior. \raisebox{-.9pt}{5}⃝ In addition, the future information ($\mathbf{c}_{F}$) reveals a future effort to reduce field fires. Could this impact future values of the target series? \raisebox{-.9pt}{6}⃝ No, the causal information ($\mathbf{c}_{\text{causal}}$) provides the answer.
Refer to caption: x3.png

For each task in the benchmark, we jointly sample numerical data from one of the series described in Sec. 3.1 and then carefully craft the natural language context necessary to unlock accurate forecasts.
In some cases, this context is purely descriptive, providing information about the general nature of the target variable and its historical behavior, as seen in the task illustrated in Fig. 1.
In other cases, the raw numerical data is adjusted to reflect the influence of the context.
For example, in one task based on data from Godahewa et al. (), an ATM is expected to be inaccessible during a specific period, leading to zero withdrawals (visualized in Sec. B.3).
In another task, electricity demand is projected to surge due to an incoming weather event (visualized in Sec. B.2).
In such cases, we modify the series to incorporate these patterns and include such information into the corresponding context.

Overall, we include diverse forms of natural language context, capturing various aspects of the process underlying the time series and revealing complementary knowledge that could be provided by a human expert or an external information source.
The types of context are described below and exemplified in the task illustrated in Fig. 3.
Several additional examples are given in Appendix B.

Intemporal information ($\mathbf{c}_{I}$)  Information about the process that remains invariant in time.
For example, a description of the process and the nature of the target variable, as in Fig. 3 (point \raisebox{-.9pt}{2}⃝), patterns that cannot be inferred from the available numerical data (e.g., long-period seasonalities), or constraints on values (e.g., positivity).

Historical information ($\mathbf{c}_{H}$) Information about the past behavior of the series that is not reflected in the available numerical history. For example, statistics on past values of the series, as in Fig. 3 (point \raisebox{-.9pt}{3}⃝), or an explanation for spurious patterns to be disregarded at inference (e.g., periodic anomalies due to sensor maintenance).

Covariate information ($\mathbf{c}_{\text{cov}}$)  Information about any additional variables that are statistically associated with the variable of interest and that may help prediction. For instance, a variable correlated with the target values (as in Fig. 3 point \raisebox{-.9pt}{4}⃝).

Future information ($\mathbf{c}_{F}$)  Information relevant to the future behavior of the time series. For example, a scenario to be simulated (as in Fig. 3 point \raisebox{-.9pt}{5}⃝) or expected events along with any entailed constraints (e.g., an inventory shortage restricting future sales amounts).

Causal information ($\mathbf{c}_{\text{causal}}$)  Information about causal relationships between covariates and the target variable. For example, if the covariates are known to cause or are confounded with the target variable (as in Fig. 3 point \raisebox{-.9pt}{6}⃝).

Finally, we note that, in contrast with the work of Zhang et al. (); Merrill et al. (); Liu et al. (); Emami et al. () which rely on LLMs or scraped news articles, all contextual information and data transformations in the CiK benchmark are manually crafted to ensure quality and relevance.

#### 3.3 Model Capabilities

In addition to forecasting and natural language understanding,
all tasks are designed such that fully utilizing the contextual information requires a range of capabilities,
including instruction following, various forms of reasoning, and retrieval.

For example, to solve the task in Fig. 3 , the model could
*retrieve from memory* that Montreal experiences snowfall and cold weather during the winter months.
It could then infer that trash fires are less likely to occur during this period through *deductive reasoning*.
This chain of thought reveals a seasonal pattern that is not apparent in the short numerical history. Additionally, through *causal reasoning*, it is apparent that, despite a strong association between field fires and trash fires, the intervention described in \raisebox{-.9pt}{5}⃝ is unlikely to reduce the frequency of the latter. Failure to recognize this distinction would lead to inaccurate forecasts.

A list of all capabilities with definitions is available in Sec. A.4 and the capabilities required to solve each task are documented at [https://servicenow.github.io/context-is-key-forecasting/v0/](https://servicenow.github.io/context-is-key-forecasting/v0/).
The distributions of tasks per capability and context type are shown in Fig. 2, while the distribution of lengths of the numerical historical data, prediction horizons and natural language context can be found in Sec. A.5.
Multiple example tasks from CiK are provided in Appendix B, along with an explanation of their sources of natural language context and the capabilities required to solve them.

### 4 Region Of Interest Continuous Ranked Probability Score

Alongside the tasks, we introduce the Region of Interest CRPS (RCRPS), a novel proper scoring rule designed specifically for context-aided probabilistic forecasting.
This new scoring rule is an extension of the Continuous Ranked Probability Score (CRPS; Gneiting & Raftery ()), a proper scoring rule that provides a comprehensive assessment of forecast quality by evaluating the entire predictive distribution rather than focusing solely on summary statistics.
Importantly, since it is based on the CRPS, the RCRPS can be calculated using only samples from the predictive distribution, and so can be used even in cases where closed-form distributions are unavailable. The RCRPS extends the CRPS via two key components: a *region of interest* and a measure of *constraint satisfaction*. This allows assessing both forecast accuracy and the integration of contextual information.

Region of interest (RoI):  The score reweighs a strict subset of time steps, denoted by $\mathcal{I}\subseteq[t\!+\!1,\dots,T]$, whose values are heavily informed by the context.
For example, in the ATM task described in Sec. 3.2 (visualized in Sec. B.3), this would correspond to the time intervals during which the ATM is expected to be unavailable.
In other tasks, such as those in Figs. 1 and 3, where the context informs the value of all future time points, we set the RoI to an empty set, essentially weighting all time steps equally (for readability, we report the definition of RCRPS for this special case in Appendix E).

Constraint satisfaction:  The score penalizes violations of constraints, whether explicitly or implicitly included in the context, by measuring a task-specific function, denoted by $v_{\mathbf{C}}$, whose value is zero for any trajectory that satisfies the constraints and $>0$ for any trajectory that violates them.
Concrete examples are given in Sec. E.4.
For tasks whose context does not imply constraints, we use $v_{\mathbf{C}}(\cdot)\equiv 0$.

Given an inferred forecast distribution $\widetilde{\mathbf{X}}_{F}$ and a ground truth $\mathbf{x}_{F}$, the scoring rule is defined as:
$\text{RCRPS}(\widetilde{\mathbf{X}}_{F},\mathbf{x}_{F})\mathrel{\mathop{:}}=
\mathcal{\alpha}\,\cdot\left[\frac{1}{2|\mathcal{I}|}\cdot\sum_{i\in\mathcal{I
}}\text{CRPS}\!\!\left(\widetilde{X}_{i},x_{i}\right)+\frac{1}{2|\neg\mathcal{
I}|}\cdot\sum_{i\in\neg\mathcal{I}}\text{CRPS}\!\!\left(\widetilde{X}_{i},x_{i
}\right)+\beta\cdot\,\text{CRPS}\!\!\left(v_{\mathbf{C}}(\widetilde{\mathbf{X}
}_{F}),0\right)\right],$

where the terms respectively account for CRPS inside the RoI, CRPS outside of the RoI, and constraint satisfaction.
We note that the last term is inspired by the threshold-weighted CRPS of Gneiting & Ranjan () and that it vanishes when
all constraints are satisfied.
The $\alpha$ term is a task-dependent scaling factor that is used to ensure that score values for tasks with numerical data of various scales can be aggregated; its calculation is described in Sec. E.1.
Finally, $\beta$ is a scaling factor that controls the impact of constraint violation on the score; we use $\beta=10$ in our experiments.
For additional details and discussion on the RCRPR properness, we refer the reader to Appendix E.

### 5 Experiments and Results

In this section, we define our evaluation protocol (Sec. 5.1) and outline the baseline models evaluated on CiK (Sec. 5.2). We then present results on the benchmark (Sec. 5.3), along with an analysis of factors affecting model performance.
Finally, we look at areas for improvement by analyzing forecasting errors (Sec. 5.4) and inference cost (Sec. 5.4).

#### 5.1 Evaluation Protocol

Each task in CiK has many unique specifications, i.e. instances arising from the various time series and windows in the associated numerical data, as well as minor variations in natural language context.
In order to make the evaluation reproducible and affordable, we deterministically select 5 instances of each task for evaluation. For each instance, we generate 25 independent samples from each model for evaluation.
Since many of the tasks in the benchmark share similarities due to having been created from the same data sources or using variants of the same context, we identify these clusters of similar tasks, and design a weighting scheme such that each cluster has equal total weight in our aggregate score (see Sec. A.2 for more details).
Finally, to prevent the aggregate scores from being dominated by rare instances where some models give forecasts which are orders of magnitudes away from the ground truth, we introduce an upper bound of 5 to the RCPRS value for each instance, which intuitively represents the value a forecast would get if the distance between the forecast and the ground-truth was 5 times bigger than the range of the ground-truth of the instance.

#### 5.2 Baselines

We evaluate a wide variety of models ranging from methods based on language models to state-of-the-art numerical time series foundation models and classical statistical forecasting methods.
Since CiK is meant to be an evaluation benchmark and hence does not have a corresponding training set, we only directly evaluate models that support zero-shot inference (such as LLMs and time series foundation models),
and those which can be fit directly to the few historical data points of each task instance evaluated, such as traditional statistical models.
We outline these methods below and refer the reader to Appendix D for additional details.

LLM-based Forecasters:  We consider two prompt-based approaches: LLM-processes (LLMP; Requeima et al. ()) and a simple approach which we propose, called “Direct Prompt”, where we instruct the model to directly output a forecast as a structured output, rather than prompting it multiple times as in (Requeima et al., ) (described in detail in Sec. D.1).
For each of these, we evaluate a variety of LLMs with diverse architectures and sizes, such as GPT-4o (Achiam et al., ), Mixtral-8x7B (Jiang et al., )), Llama-3-8b (Dubey et al., ), and Llama-3.1-405b (Dubey et al., ).
(^2^22For LLMP, we do not consider Llama-3.1-405b and GPT models as LLMP requires loading model weights into memory, which is infeasible due to resource limitations and confidentiality, respectively.)
Next, we evaluate multimodal forecasting models, UniTime (Liu et al., ) and Time-LLM (ETTh1) Jin et al. () each trained according to their respective authors’ guidelines (detailed in Sec. D.3).
For all of these approaches, inference is performed zero-shot on the benchmark and we compare their performance with and without the natural language context.

Quantitative Forecasting Models:  To contrast the performance of LLM-based forecasters, we also evaluate a number of models that are only capable of processing numerical data (no natural language). This includes exponential smoothing (Gardner Jr., ), ETS (Hyndman et al., ), and ARIMA (Box et al., ), three simple, but time-tested statistical approaches, as well as four state-of-the-art time series foundation models: Lag-Llama (Rasul et al., ), Chronos (Ansari et al., )
(^3^33Results reported here are on Chronos-Large and Moirai-Large. Results on all versions are in App. C.2.)
, Moirai (Woo et al., ), and TimeGEN (Garza et al., ).
We note that exponential smoothing, ETS, and ARIMA are fitted to each task instance’s numerical history,
while the foundation models are evaluated zero-shot.

**Table 1: Results on the CiK benchmark. Starting from the left, the first column shows the RCRPS averaged over all tasks. The second column shows the rank of each method w.r.t. other baselines, averaged over all tasks. The remaining columns show the average RCRPS stratified by model capabilities (Sec. 3.3). All averages are weighted according to the scheme described in Sec. 5.1 and accompanied by standard errors. Lower is better and the best averages are in bold. An asterisk (*) denotes models that do not use natural language context.**
|  | Average<br>RCRPS | Average<br>Rank | Instruction<br>Following | Retrieval | Reasoning |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | From Context | From Memory | Deductive | Analogical | Mathematical | Causal |  |  |  |
| Direct Prompt (ours) |  |  |  |  |  |  |  |  |  |
| Llama-3.1-405B-Inst | 0.159 $\pm$ 0.008 | 4.469 $\pm$ 0.192 | 0.140 $\pm$ 0.013 | 0.109 $\pm$ 0.002 | 0.191 $\pm$ 0.006 | 0.133 $\pm$ 0.001 | 0.167 $\pm$ 0.008 | 0.316 $\pm$ 0.028 | 0.376 $\pm$ 0.039 |
| Llama-3-70B-Inst | 0.518 $\pm$ 0.030 | 10.406 $\pm$ 0.190 | 0.504 $\pm$ 0.038 | 0.371 $\pm$ 0.071 | 0.523 $\pm$ 0.048 | 0.461 $\pm$ 0.048 | 0.694 $\pm$ 0.117 | 0.573 $\pm$ 0.044 | 0.643 $\pm$ 0.049 |
| Llama-3-8B-Inst | 1.647 $\pm$ 0.069 | 15.130 $\pm$ 0.171 | 1.604 $\pm$ 0.131 | 0.199 $\pm$ 0.010 | 1.568 $\pm$ 0.067 | 2.133 $\pm$ 0.082 | 1.555 $\pm$ 0.008 | 1.589 $\pm$ 0.177 | 1.840 $\pm$ 0.238 |
| Mixtral-8x7B-Inst | 1.061 $\pm$ 0.058 | 13.381 $\pm$ 0.230 | 0.857 $\pm$ 0.077 | 0.296 $\pm$ 0.049 | 1.077 $\pm$ 0.078 | 1.352 $\pm$ 0.117 | 1.145 $\pm$ 0.144 | 1.000 $\pm$ 0.086 | 1.096 $\pm$ 0.106 |
| GPT-4o | 0.276 $\pm$ 0.010 | 4.368 $\pm$ 0.149 | 0.180 $\pm$ 0.004 | 0.087 $\pm$ 0.003 | 0.519 $\pm$ 0.029 | 0.113 $\pm$ 0.006 | 0.447 $\pm$ 0.029 | 0.590 $\pm$ 0.033 | 0.769 $\pm$ 0.046 |
| GPT-4o-mini | 0.353 $\pm$ 0.022 | 8.930 $\pm$ 0.177 | 0.296 $\pm$ 0.043 | 0.419 $\pm$ 0.014 | 0.471 $\pm$ 0.012 | 0.218 $\pm$ 0.005 | 1.024 $\pm$ 0.033 | 0.475 $\pm$ 0.080 | 0.578 $\pm$ 0.112 |
| LLMP |  |  |  |  |  |  |  |  |  |
| Llama-3-70B-Inst | 0.550 $\pm$ 0.013 | 8.038 $\pm$ 0.205 | 0.645 $\pm$ 0.018 | 0.284 $\pm$ 0.015 | 0.392 $\pm$ 0.014 | 0.519 $\pm$ 0.026 | 0.312 $\pm$ 0.019 | 0.453 $\pm$ 0.020 | 0.495 $\pm$ 0.028 |
| Llama-3-70B | 0.237 $\pm$ 0.006 | 6.560 $\pm$ 0.254 | 0.310 $\pm$ 0.011 | 0.126 $\pm$ 0.009 | 0.217 $\pm$ 0.007 | 0.134 $\pm$ 0.003 | 0.241 $\pm$ 0.019 | 0.294 $\pm$ 0.008 | 0.329 $\pm$ 0.010 |
| Llama-3-8B-Inst | 0.484 $\pm$ 0.010 | 9.457 $\pm$ 0.166 | 0.345 $\pm$ 0.002 | 0.138 $\pm$ 0.004 | 0.910 $\pm$ 0.030 | 0.242 $\pm$ 0.008 | 1.278 $\pm$ 0.069 | 0.617 $\pm$ 0.022 | 0.787 $\pm$ 0.030 |
| Llama-3-8B | 0.313 $\pm$ 0.023 | 9.499 $\pm$ 0.323 | 0.404 $\pm$ 0.043 | 0.124 $\pm$ 0.003 | 0.280 $\pm$ 0.026 | 0.179 $\pm$ 0.014 | 0.267 $\pm$ 0.015 | 0.530 $\pm$ 0.084 | 0.661 $\pm$ 0.117 |
| Mixtral-8x7B-Inst | 0.264 $\pm$ 0.004 | 8.496 $\pm$ 0.256 | 0.344 $\pm$ 0.004 | 0.127 $\pm$ 0.003 | 0.224 $\pm$ 0.005 | 0.179 $\pm$ 0.010 | 0.173 $\pm$ 0.009 | 0.348 $\pm$ 0.005 | 0.405 $\pm$ 0.007 |
| Mixtral-8x7B | 0.262 $\pm$ 0.008 | 8.619 $\pm$ 0.208 | 0.348 $\pm$ 0.012 | 0.146 $\pm$ 0.022 | 0.230 $\pm$ 0.016 | 0.153 $\pm$ 0.002 | 0.230 $\pm$ 0.041 | 0.354 $\pm$ 0.007 | 0.414 $\pm$ 0.009 |
| Multimodal Models |  |  |  |  |  |  |  |  |  |
| UniTime | 0.371 $\pm$ 0.002 | 13.495 $\pm$ 0.091 | 0.271 $\pm$ 0.003 | 0.179 $\pm$ 0.001 | 0.318 $\pm$ 0.001 | 0.510 $\pm$ 0.003 | 0.333 $\pm$ 0.001 | 0.332 $\pm$ 0.001 | 0.384 $\pm$ 0.001 |
| Time-LLM (ETTh1) | 0.476 $\pm$ 0.001 | 16.662 $\pm$ 0.075 | 0.448 $\pm$ 0.002 | 0.192 $\pm$ 0.000 | 0.373 $\pm$ 0.000 | 0.538 $\pm$ 0.003 | 0.397 $\pm$ 0.001 | 0.382 $\pm$ 0.001 | 0.440 $\pm$ 0.001 |
| TS Foundation Models* |  |  |  |  |  |  |  |  |  |
| Lag-Llama | 0.329 $\pm$ 0.004 | 13.157 $\pm$ 0.235 | 0.355 $\pm$ 0.007 | 0.181 $\pm$ 0.003 | 0.324 $\pm$ 0.003 | 0.272 $\pm$ 0.006 | 0.342 $\pm$ 0.006 | 0.386 $\pm$ 0.009 | 0.449 $\pm$ 0.012 |
| Chronos | 0.326 $\pm$ 0.002 | 11.962 $\pm$ 0.139 | 0.385 $\pm$ 0.002 | 0.138 $\pm$ 0.002 | 0.288 $\pm$ 0.002 | 0.249 $\pm$ 0.002 | 0.295 $\pm$ 0.003 | 0.362 $\pm$ 0.003 | 0.417 $\pm$ 0.004 |
| TimeGEN | 0.354 $\pm$ 0.000 | 14.345 $\pm$ 0.090 | 0.402 $\pm$ 0.000 | 0.176 $\pm$ 0.000 | 0.308 $\pm$ 0.000 | 0.279 $\pm$ 0.000 | 0.324 $\pm$ 0.000 | 0.377 $\pm$ 0.000 | 0.431 $\pm$ 0.000 |
| Moirai | 0.520 $\pm$ 0.006 | 12.458 $\pm$ 0.266 | 0.414 $\pm$ 0.004 | 0.155 $\pm$ 0.004 | 0.260 $\pm$ 0.003 | 0.751 $\pm$ 0.015 | 0.276 $\pm$ 0.008 | 0.337 $\pm$ 0.007 | 0.397 $\pm$ 0.010 |
| Statistical Models* |  |  |  |  |  |  |  |  |  |
| ARIMA | 0.480 $\pm$ 0.006 | 12.320 $\pm$ 0.180 | 0.399 $\pm$ 0.006 | 0.160 $\pm$ 0.002 | 0.517 $\pm$ 0.012 | 0.522 $\pm$ 0.013 | 0.706 $\pm$ 0.026 | 0.354 $\pm$ 0.007 | 0.403 $\pm$ 0.010 |
| ETS | 0.522 $\pm$ 0.009 | 14.310 $\pm$ 0.203 | 0.407 $\pm$ 0.009 | 0.228 $\pm$ 0.010 | 0.682 $\pm$ 0.018 | 0.571 $\pm$ 0.019 | 0.855 $\pm$ 0.035 | 0.453 $\pm$ 0.012 | 0.479 $\pm$ 0.015 |
| Exp-Smoothing | 0.603 $\pm$ 0.013 | 14.936 $\pm$ 0.132 | 0.571 $\pm$ 0.021 | 0.334 $\pm$ 0.013 | 0.743 $\pm$ 0.018 | 0.557 $\pm$ 0.019 | 0.899 $\pm$ 0.035 | 0.673 $\pm$ 0.038 | 0.782 $\pm$ 0.053 |

#### 5.3 Results on the benchmark

Our main results are shown in Tab. 1.
At a high level, we observe that the best-performing baselines combine pretrained LLMs with prompting strategies like Direct Prompt and LLMP, with a bias toward the largest models.
In terms of RCRPS, Llama-3.1-405B-Inst (Direct Prompt) significantly outperforms all of its counterparts.
As can be seen in Fig. 4, it achieves this only with context.
GPT-4o (Direct Prompt) performs worse with respect to RCRPS, but compares favorably in terms of average rank, taking the best average rank by a small margin.
This discrepancy is due to strong failures on some of the tasks, which we discuss in Sec. 5.4.
Other models like Llama-3-70B (LLMP), Mixtral-8x7B-Inst (LLMP), Mixtral-8x7B (LLMP), and Llama-3-8B (LLMP) are on par with GPT-4o (Direct Prompt) in terms of RCRPS.
Interestingly, all of these baselines outperform UniTime and Time-LLM, which also rely on LLMs (GPT-2 & LLaMA-7B).
We discuss this gap in Tab. 8.
Finally, as emphasized in Fig. 5, we observe that the best-performing LLM baselines significantly outperform purely quantitative models.
In what follows, we examine various aspects of these results.

Explaining the performance of
LLM-based approaches

Figure: Figure 4: Performance with and without context (lower is better). Full bars show performance with context; striped bars show performance without. All models improve with context, except DP Mixtral-8x7B-Inst, LLMP Llama-3-70B-Inst and Time-LLM. Llama-3.1-405B-Inst improves significantly with context, exhibiting the best aggregate RCRPS.
Refer to caption: x4.png

The stronger performance of LLM baselines could be due to two factors: (i) properly leveraging the natural language context and (ii) being more proficient at numerical forecasting. We thus attempt to disentangle their contributions.
On the one hand, Fig. 4 shows clear evidence that most baselines make use of the context to improve their forecasts.
For example, Llama-3.1-405B-Inst (Direct Prompt) improves by $67.1\%$ with context.
This is reflected in the quality of the forecasts, where we observe clear improvements especially in regions of interest and improved satisfaction of constraints (see Sec. C.4 for examples).
Other models show much slighter improvements and, in three cases, even a degradation in performance.
These can be explained either by the context being ignored, or by significant failures in using context, impoverishing overall performance (see Sec. 5.4).

Figure: Figure 5: Proportion of tasks for which LLM-based baselines outperform the 7 quantitative forecasting baselines (see Sec. 5.2). A baseline is considered to outperform another on a task if its mean RCPRS is lower on said task. Results are shown for variants that use (left) and do not use (right) the natural language context. A full green bar would indicate that the baseline is better on all tasks, whereas a full red bar would indicate that it is worse everywhere. Averages are weighted according to Sec. 5.1.
Refer to caption: x5.png

On the other hand, Fig. 5 (right) shows that some LLM baselines are surprisingly good forecasters when compared to quantitative forecasting models in a no-context setting.
For instance, multiple Llama-3-based models used with the LLMP strategy outperform at least $5$
of the quantitative baselines on the majority of tasks.
This is further substantiated by results in Sec. C.2.
In contrast, other baselines, including the best models Llama-3.1-405B-Inst (Direct Prompt) and GPT-4o (Direct Prompt), show much weaker numerical forecasting abilities without context, suggesting that their performance is mostly due to leveraging the context.

Comparing the LLMP and Direct Prompting Strategies

Clear patterns emerge when comparing these strategies.
First, as shown in Fig. 5 (right), LLMP baselines exhibit stronger numerical forecasting performance without context than Direct Prompt baselines.
This advantage likely stems from LLMP’s closer alignment with the forecasting task: LLMP simply prompts the LLM to autoregressively predict the next value in the time series – a task well suited for non-instruction tuned LLMs.
This contrasts with Direct Prompting which requires output forecasts to be structured, complicating the overall task.

This line of reasoning leads us to our second observation; as reflected in Tab. 1 and Fig. 5, instruction tuning appears to degrade LLMP performance, with Llama-3 models showing a twofold decrease in performance after tuning—a behavior previously observed by Gruver et al. (). Interestingly, instruction tuning does not degrade Mixtral-8x7B performance. Finally, while instruction tuning generally harms LLMP, it is essential for models used with the Direct Prompt strategy. Again, Direct Prompt requires forecasts to be produced in a specific structure, a skill that base models typically hone during post-training (see Sec. D.1.1 for details).

No Baseline Excels Across All Capabilities

Based on the results in Tab. 1, it is evident that some models possess the necessary capabilities to effectively utilize the contextual information provided. However, we observe that no single model is the best across all capabilities. Llama-3.1-405B-Inst (Direct Prompt), our overall top-performing baseline, outperforms its counterparts in only 4 out of 7 capabilities. This finding indicates that the benchmark remains unsolved, leaving significant room for advancements from the research community.

#### 5.4 Error Analysis

We find that models occasionally return forecasts that miss the ground truth by a large margin.
A *significant failure* denotes a forecast that over or undershoots by at least five times the range of the ground truth; at that point, we clip the RCRPS to 5 as explained in Sec. 5.1.
Despite this cap, such unpredictable behaviour impacts the results of Tab. 1:
GPT-4o with Direct Prompt, while emerging as a top-performer in most tasks (as reflected in its average rank), provides significantly higher aggregate RCRPS than models ranked worse, such as Mixtral-8x7B with LLMP.
As an example, Direct Prompt with GPT-4o fails significantly in a task with a context involving scientific notation (see Fig. 16; more examples can be found in Sec. C.5).
These findings underscore the need for future work to develop more robust models that can handle context effectively while avoiding significant failures.

#### 5.5 Inference Cost

A key practical aspect for forecasting applications is the inference time of models and their associated cost.
Fig. 6 (left) shows that, while Llama-3.1-405B-Instruct has the best RCRPS, it comes at the cost of a significantly higher parameter count than the quantitative forecasters.
This emphasizes that, while LLMs can be powerful context-aware forecasters, they come with a steep computational cost, highlighting the need for efficient models that balance both accuracy and resource demands.
Of note is also that many LLM baselines are Pareto dominated by quantitative forecasters such as Lag-Llama and Chronos.
This suggests that the ability to ingest text is not enough and that a careful choice of LLM and prompting strategy is crucial for Pareto efficiency.

Fig. 6 (right) emphasizes the disparity in inference time between LLMs and quantitative models.
LLMs take significantly longer to make predictions, with the most accurate LLMs having inference times that are orders of magnitude higher than their quantitative counterparts. Quantitative models, in contrast, maintain much lower inference times, making them far more efficient for practical use.
The high computational demands of context-aware LLMs hinder their practical use in real-world forecasting, especially where speed and cost matter. The clear benefits of incorporating context warrants research into making them more efficient, aiming to match the cost-effectiveness of traditional models and enabling their deployment in large-scale forecasting.

Figure: Figure 6: Overview of inference costs. (Left) Comparison of average RCRPS (per Tab. 1), vs. the parameter count of each baseline model (lower is better for both). The GPT family, as well as TimeGEN, are left out as there is no information on them about parameter count. The dashed line illustrates the Pareto front: models above and to the right of this front are dominated. Quantitative forecasters dominate the low-parameter regime, while LLM-based methods such as LLMP Llama-3-8B or 3-70B and DP 3.1-405B-Inst offer superior performance for a higher parameter count. (Right) Inference time in seconds, for all baselines, averaged over all tasks. Several quantitative methods are much faster on average than LLM-based methods. However, there are significant differences in inference time between the LLM-based forecasters: for the Llama models, LLM Process takes about an order of magnitude more time to run on average than Direct Prompt.
Refer to caption: x6.png

### 6 Related Work

We review two streams of related work: (i) work that introduce related benchmarks and datasets, and (ii) work that repurpose LLMs to obtain foundation models for context-aided forecasting.

Benchmarks and Datasets
Merrill et al. () present a benchmark designed to evaluate LLMs’ ability to reason about time series, with context-aided forecasting as one assessed capability.
Their approach differs from ours in several important ways. First, they focus on purely synthetic time series, which may not accurately reflect real-world dynamics, whereas our benchmark is based primarily on real-world data.
Second, their evaluation is limited to point forecasting metrics, which do not measure the quality of the full forecast distribution.
In contrast, we adopt probabilistic forecasting metrics, e.g., the continuous ranked probability score (CRPS; *c.f.* , ), to assess the quality of entire forecast distributions.
Other related datasets include Time-MMD (Liu et al., ), which integrates text extracted from reports and web searches, TGTSF (Xu et al., ), which incorporates information such as weather reports and news articles, SysCaps (Emami et al., ), which includes LLM-generated descriptions of building energy consumption systems, and TS-Insights (Zhang et al., ), which includes LLM-generated descriptions of trends and seasonalities.
The key distinction between these works and ours lies in the role of textual information:
while in these works, the text is not essential to generating high-quality forecasts, in our benchmark, accurate forecasts *cannot be achieved*, by design, without effectively leveraging the provided textual information.

Repurposing LLMs for Forecasting
A natural approach to this problem is to build forecasting methods based on LLMs.
Xue & Salim () showed that forecasting could be framed as a question-answering problem.
Subsequently, Gruver et al. () and Requeima et al. () showed that zero-shot sequence completion could generate accurate forecasts and that textual side-information could be used to influence forecasts. However, their analysis is limited to illustrative examples rather than a comprehensive evaluation. Other approaches incorporate time series into pretrained LLMs (Jin et al., ; Liu et al., ; Zhang et al., )
by introducing special tokens used to represent patched time series patterns; or modifying their encoders to account for time series data (Jia et al., ).
While these methods show promising results, their evaluations primarily rely on datasets where the contextual information is not guaranteed to improve forecasts over numerical data alone.
As a result, it remains unclear whether their success is driven by accurate numerical forecasting or by effectively incorporating context;
this shortcoming motivates our investigation into this question.

### 7 Discussion

In this work, we propose the Context is Key (CiK) benchmark:
a collection of forecasting tasks that require combining historical data with critical natural language context. We evaluate a range of models on CiK, including our proposed LLM prompting method, Direct Prompt, which achieves the best performance. We analyse and discuss the failure modes of these models, and our findings underscore the critical role of contextual information in improving forecasts, while also revealing both the unexpected strengths and notable limitations of the investigated LLM-based forecasters.

Limitations:  While our benchmark provides valuable insights into the integration of contextual information in time series forecasting, it is important to acknowledge its limitations. Our study excludes modalities other than time series data and text, and excludes multivariate time series scenarios. Although we carefully and deliberately designed the tasks to assess how well time series forecasters can integrate contextual information, our focus was on relationships between context and forecasts that are discernible to humans. Hence, our benchmark does not explicitly evaluate a models’ capacity to leverage latent relationships that might elude human observation.
Moreover, while tasks are designed to require certain capabilities, we do not guarantee that alternative approaches to solving them do not exist.
Our collection of capabilities and context types was not intended to be exhaustive but rather to serve as tools for analyzing forecasters’ performance on the benchmark.
While we have taken steps to mitigate memorization concerns, as discussed in Sec. 3.1, achieving absolute certainty in this regard is challenging without strictly held-out data.

Future work:  There are several promising avenues for future work in the context-aided forecasting setting. Enhancements to the benchmark could include tasks that require multivariate forecasting or incorporate additional modalities, such as images, structured databases, or spatiotemporal data. Tasks that deliberately challenge context length limitations or probe specific weaknesses of language models would also be valuable additions. Furthermore, this benchmark strongly motivates research into developing more accurate and efficient multimodal forecasting models, which it is well-positioned to support. Lastly, as models become more robust, they could be integrated into agentic systems with conversational interfaces, allowing forecasts to be augmented with human expertise and automatically retrieved facts (e.g., via search engines). Such advancements would represent a significant step toward automating and democratizing access to powerful forecasting tools.

### Acknowledgements

The authors are grateful to Midan Kim, Torsten Scholak, Mohammad Reza Samsami, Oussama Boussif and Can Chen for their valuable feedback and suggestions. This research was supported by Mitacs Accelerate Grants and enabled by compute resources provided by ServiceNow Research and the Frontier supercomputer. The latter resources were awarded through the Frontier DD allocation and INCITE 2023 program for the project “Scalable Foundation Models for Transferable Generalist AI” and were supplied by the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, with support from the Office of Science of the U.S. Department of Energy.

## Appendix

### Appendix A Additional Details on the Benchmark

#### A.1 Data Sources

We list here the domains and the respective sources of time series data we use in the various tasks in the CiK benchmark.
We also show the number of tasks that use each source’s data and list any memorization mitigation strategies used for each dataset.

- •
Traffic (11 tasks):
–
Traffic occupancy rate: We use traffic occupancy rate (%) data from the California Performance Measurement System (PeMS) (Chen et al., ), with frequency hourly.
This dataset contains a total of 446 series.
*
As this is a live dataset (updated frequently), we use data from 2024 (i.e. data after the cutoff dates of LLMs used) and do not apply any memorization mitigation strategy.
- •
Climatology (12 tasks):
–
Solar irradiance and cloud cover data (9 tasks): We use solar irradiance and cloud cover data for the Americas in 2022 (Sengupta et al., ), with frequency either 10 minutes or hourly.
We extract a subset of 45 series from this dataset for the benchmark.
*
To mitigate memorization, we shift the dates by one day ahead.
–
Solar photovoltaic power production (3 tasks): Time series reflecting solar power production in Alabama during 2006 (Godahewa et al., ), with a frequency 10 minutes.
This dataset contains a total of 137 series, but our tasks only use a single aggregated series generated from them.
*
To mitigate memorization, we add gaussian noise to the data with a standard deviation of 3% of the standard deviation of the data in each respective sampled window.
- •
Public Safety (26 tasks):
–
Fire Department Intervention Logs: Logs of number of interventions carried out by the Montreal Fire Department due to the occurence of various kinds of incidents (such as trash fires, field fires, nautical accidents, bike accidents) (Ville de Montréal, ). The data was processed from a raw log and aggregated to monthly frequency.
This dataset contains a total of 48 series.
*
Due to it being processed, we do not apply any special memorization mitigation strategy on top.
- •
Mechanics (3 tasks):
–
Causal Chambers: Experimental data collected from the wind tunnel physical system from Gamella et al. (), released in April 2024. We make use of the load_in, pressure_downwind, pressure_ambient and speed_in series (downsampling them to 1s frequency) to build out-of-distribution forecasting tasks where the target values can be inferred from the driver variate provided as covariate and the description of the physical system given in the context.
We select a subset of 17 series from this dataset for the benchmark.
*
Since the data is released in 2024 and after the cutoff dates of the LLMs used, we do not apply any memorization mitigation technique to transform the data.
- •
Economics (3 tasks):
–
FRED: American unemployment data at the state and county levels, from the Federal Reserve Bank of St. Louis (U.S. Bureau of Labor Statistics, ), with frequency monthly.
We extract a subset of 1769 series from this dataset for the benchmark.
*
As this is a live dataset (updated frequently), we use data from 2024 (i.e. data after the cutoff dates of LLMs used) and do not apply any memorization mitigation strategy.
- •
Retail (6 tasks):
–
NN5 ATM cash withdrawals: The NN5 dataset of ATM cash withdrawals in the UK from the Monash Time Series Forecasting Repository (Godahewa et al., ), with frequency daily.
This dataset contains a total of 111 series.
*
To mitigate memorization, we add gaussian noise to the data with a standard deviation of 3% of the standard deviation of the data in each respective sampled window.
- •
Energy (7 tasks):
–
Electricity consumption: Electricity usage from 2012 to 2014 from the Monash Time Series Forecasting Repository (Godahewa et al., ), with frequency daily.
This dataset contains a total of 321 series.
*
To mitigate memorization, we add gaussian noise to the data with a standard deviation of 3% of the standard deviation of the data in each respective sampled window.
- •
Synthetic Data (3 tasks):
We employ a bivariate setup where the parent variable is drawn from a categorical distribution, and the child variable is generated using a continuous linear Structural Vector Autoregressive (SVAR) model with Gaussian noise, with a lag of $3$ and a noise scale of $0.1$.
–
Since this data is synthetic, we do not apply any mitigation technique on top of data to mitigate memorization. Since our models assume a timestamp, we use dates from $2025$, and a frequency of daily when we input this data to our models.

Depending on the task and the context used in the task, appropriate history and prediction lengths are used in the task.

#### A.2 Weighting scheme for tasks

To take full advantage of the available data, we create multiple tasks using each data source, by varying the specific contextual information we provide to the models.
Since we do not want our aggregate results to be dominated by the few datasets for which there are a larger number of tasks, we weight the contribution of each task to the various aggregated results.

To define the weight of each task, we first group the tasks in clusters.
These clusters are primarily defined based on the original data source used to create the tasks.
However, when tasks are fundamentally different, due to not testing the same capabilities, we put them in different clusters despite them using the same data source.
For example, for tasks created using the Solar irradiance and cloud cover data, all of which ask models to forecast the irradiance, the tasks form three distinct clusters: one for tasks asking models to do forecast with very short history (less than a day), one for tasks giving the cloud cover as covariate, and the final one for tasks where the models are given a tight upper bound on the possible irradiance.
Once we define these clusters, we simply equal weight to each cluster, and equal weight to each task inside each cluster.

#### A.3 Standard errors and average ranks

To get the standard errors shown in Tab. 1, we first compute the standard error for tasks using the method described in Sec. E.5.
We then aggregate them according to each task weight, by assuming that errors for each are independent and thus using the formula for the variance of a weighted sum of independent variables.

To take into consideration the uncertainty we have for the scores, we compute average ranks through a simple simulation.
In this simulation, we replace the RCRPS for each task and model pair by an independent Gaussian variable of mean equals to the one we measured, and of standard deviation equals to the standard error.
We then draw from this distribution and compute the weighted average ranks for each model.
The results shown in Tab. 1 are the mean and standard deviation measured from 10,000 repetitions of this simulation.

#### A.4 Model Capabilities

We provide a detailed explanation of each model capability here. Note that tasks in the CiK benchmark need not be mutually exclusive with the model capabilities they require; tasks are tagged with one or more model capabilities.

Instruction following (24 Tasks): Using direct instructions available in the context. Instructions could express constraints to be satisfied, or the expected effect of an event, for example.

Retrieval: Retrieving facts from memory or context.

- •
Retrieval from memory (35 Tasks): Retrieving from memory facts that enable interpretation of the context, such as relevant physical constants or quantitative laws.
- •
Retrieval from context (25 Tasks): Retrieving relevant information from context and distinguishing it from irrelevant information.

Reasoning: Reasoning about information in context or memory.

- •
Analogical Reasoning (6 tasks): Making analogies between entities or events, for instance, applying knowledge from a past event that is similar to an upcoming one.
- •
Mathematical Reasoning (32 tasks): Performing calculations over the context, e.g. solving an equation.
- •
Deductive Reasoning (39 tasks): Inferring new facts not explicitly mentioned in the context,
e.g. inferring from the context that certain values are logically impossible to occur.
- •
Causal Reasoning (22 tasks): Deriving or using causal information from the context to reason about actions (such as interventions).

#### A.5 Task lengths

Fig. 7 provides an overview of the distribution of the lengths of the natural language context, numerical history and target (prediction horizon) for a set of five instances for each task in the CiK benchmark.

Figure: Figure 7: Histograms depicting the distribution of lengths for the context, numerical history and target length of a set of five instances for each task in CiK. We measure the length of the natural language context in characters, and the numerical sequences in floats.
Refer to caption: extracted/5951995/figures/appendix/context_lengths/lengths.png

### Appendix B Examples of tasks from the benchmark

In this section, we feature multiple examples from the benchmark to exemplify exactly what a task is, what context sources represent (Sec. 3.2), and how these tasks encourage the use of capabilities (Sec. 3.3). To visualize all tasks in the benchmark, we refer the reader to [https://servicenow.github.io/context-is-key-forecasting/v0/](https://servicenow.github.io/context-is-key-forecasting/v0/).

#### B.1 Task: Constrained Predictions

[Uncaptioned image]: extracted/5951995/figures/example_tasks/bounded_pred.png

This task, which we refer to as “Bounded Prediction Constraint Based On Prediction Quantiles”, is a forecasting task where we modify the forecast horizon (in green in the plot) by bounding one or both of its extremes according to its unmodified ground truth’s quantile values.
We verbalize these bounds in the context, and the model is expected to interpret and respect them.

Since we draw this series from the PeMS dataset (Chen et al., ), we tag its domain as “Traffic”.
The context directly refers to the future, hence the context source is tagged as “Future information”.
Finally, since the model is expected to obey the constraints in the context, we tag the evaluated capability as “Instruction following”.

Since the context contains constraints, the Region of Interest CRPS metric that we introduce (Sec. 4) heavily penalizes forecasts that exceed these constraints: models that do not incorporate the information about bounds in the context, such as quantitative forecasting models, would not be able to predict the ground truth (orange line) because its lower bound is much higher than that of the history.
In this case, the region of interest for the metric is the entire forecast horizon because the context applies everywhere.
Although statistical forecasters may pick up on the seasonality present in the history (black line), they would obtain worse scores than models capable of processing the context and adjusting the lower bound of their predictions.

#### B.2 Task: Electrical Consumption Increase

[Uncaptioned image]: x8.png

The “Short News Electricity Increase” task introduces a large shock in the forecast horizon that is only referred to in the context.
Hence, the model must interpret the context appropriately to forecast the spike.

Since this series represents electricity consumption (Sec. 3.1), we tag it a coming from the “Energy” domain.
The context sources for this task are twofold: the first context source is “Future information”, which represents knowledge of the five-fold increase in typical usage during the shock.
The second source of context, “Covariate information”, represents the occurrence of a heatwave, which coincides with the timing and duration of the shock.
The model must therefore interpret both the information on the magnitude of the shock from the future information, as well as the timing and duration of the sock from the covariate information.
Together, these pieces of information enable an accurate forecast despite the lack of information about the shock in the task’s numerical history.

The skills for this task are tagged as “Instruction following” and “Retrieval from context”.
While instruction following involves interpreting the context to include the shock in the prediction, the model must also retrieve from the context the relevant information, as there is unneeded information in the context as well: an accurate forecast does not require knowing that the temperature has reached unprecedented levels.

In this task, we also see a “Region of Interest” (RoI), characterized by a darker region of the forecast horizon.
This RoI represents the region of the forecast horizon for which the context is relevant, i.e. the period during which the increased power consumption occurred.
As detailed in Sec. 4, this region of interest is taking into account in the RCRPS metric.

#### B.3 Task: ATM Maintenance

[Uncaptioned image]: x9.png

The “Automated Teller Machine (ATM) Under Period Maintenance“ task represents the history of withdrawals from an ATM that undergoes regular maintenance.
This maintenance introduces a periodic, easily forecastable signal into the history.
However, the context explicitly states that the forecast should assume the ATM will not be in maintenance during the forecast.
Therefore, forecasting models are expected to ignore this signal.

Since this series represents ATM withdrawals, we tag it as “Retail”.
The context includes information such as the location of the ATM, and therefore provides “Intemporal information”.
As the maintenance frequency and duration is also described, the context sources include “Covariate information”.

This task is tagged with two capabilities. “Instruction following” is necessary because the model must assume that the ATM will not be in maintenance in the future.
However, the model must use “Deductive reasoning” to determine what and when the impact of the maintenance was – reducing the number of withdrawals to 0 every 14 days –, and avoid including that pattern in the forecast.
The RoI represents when the maintenance periods would have occurred in the forecast horizon, which is likely where forecasting models that do not leverage the context will forecast 0.
While a quantitative forecasting model would find such a signal irresistible, context-aware models should avoid repeating the pattern in the forecast.

We also note that the series is not quite 0 during the maintenance periods.
This is a consequence of using one of our memorization mitigation schemes (Sec. A.1, paragraph “Memorization mitigation”).

#### B.4 Task: Montreal Fire High Season

[Uncaptioned image]: extracted/5951995/figures/example_tasks/montreal_fire.png

The “Montreal Field Fire With Explicit Short History” task requires predicting the number of field fire incidents during the summer, so we tag it as being part of the “Public Safety” domain.

The context contains information from two different sources: it contains “Intemporal information”, such as the location and nature of the incidents.
However, it also contains “Historical information”, which verbalizes statistics about past values of the series, beyond the numerical data.
That is, the yearly average number of incidents, along with the knowledge that June is the month with the most incidents.

This task is tagged with many skills and involves several steps of interpretation to arrive at a reasonable forecast.
We first note that the task requires “Retrieval from memory”: an important piece of information for this prediction is that winters in Montreal, a city in the northern hemisphere, are long and harsh, with temperatures reaching $-40^{\circ}$C.
Secondly, the task requires the model to use “Deductive reasoning” to deduce that, since temperatures are so cold during the winter months, fields are likely covered in snow and are rather unlikely to catch fire.
Finally, the model can employ “Mathematical reasoning” to determine how many field fires are likely to occur on average in the forecast horizon, given the total number of field fires that have already blazed during the history.

Note that “Retrieval from memory” tasks do not explicitly ask the model to return information retrieved from memory; rather, we tag tasks as such because they cannot be solved without key information that is not present in the history or the context.

#### B.5 Task: Solar Prediction

[Uncaptioned image]: x10.png

The “Explicit Similar Location and Day Solar Forecast” task requires forecasting the power production of a solar power plant based on a very short history and information about the similarity between its climate and that of an adjacent location.
We therefore tag the domain of this series as “Climatology”.

Without the “Intemporal information” that the context provides, it is quite possibly impossible to accurately forecast the parabola-like shape of the ground truth: the history contains very few defining characteristics, which makes it interchangeable with that of many potential processes and therefore many possible forecasts.
The model must use “Deductive reasoning” to foresee this reversion to zero based on the fact that solar panels do not produce electricity at night.

However, the information in the context alone is not sufficient to provide an accurate forecast: nothing indicates the time at which production should peak.
It must therefore rely on “Retrieval from memory” to retrieve information about Alabama’s climate and then “Analogical reasoning” to apply it to the present problem.

#### B.6 Task: Speed From Load

[Uncaptioned image]: x11.png

The “Speed From Load” task combines many different context sources and capabilities to produce a forecast of the revolutions per minute (RPM) of a fan in a wind tunnel based on its load.
This task, based on the Causal Chambers dataset (Gamella et al., ), is therefore tagged as part of the “Mechanics” domain.

As the plot shows, producing an accurate forecast of the ground truth (orange line) from the numerical history alone (black line) is essentially impossible.
However, the context of the task is quite rich: it provides “Intemporal information” on the nature of the task, such as the limits of the load and of the fan, “Covariate information” that describes the load during the history and future, as well as “Causal information” on the control that the load exerts on the fan, as well as the proportionality of their relationship.

To leverage the context requires multiple skills: firstly, “Instruction following” is necessary to understand that the task is to forecast the speed of the fan (as opposed to e.g. the load) and to apply the correct loads at the right moments. Secondly, the model must use “Causal reasoning” to understand that the changes in the load will directly impact the speed of the fan. Finally, the model must leverage “Mathematical reasoning” to calculate the speed of the fan as a function of the load.

### Appendix C Additional Results

#### C.1 Results partitioned by types of context

Table Tab. 2 provides a view of the results partitioned by the types of context. One can observe that Direct Prompt - Llama-3.1-405B-Instruct achieves the best performance at tasks where the context involves intemporal, future or covariate information, while GPT-4o has an upper hand at tasks involving historical context information. LLMP with Llama-3-70B-Instruct achieves the best performance in tasks that involve causal information in the context. This provides a view complementary to that of partitioning by model capabilities (as in Tab. 1), and emphasizes that no single model is the best at processing all types of context, leaving room for advancements in models in the future.

**Table 2: Results on the CiK benchmark aggregated over all tasks and kinds of context. The first column shows the RCRPS averaged over all tasks. The second column shows the rank of each method w.r.t. other baselines, averaged over all tasks. The remaining columns show the average RCRPS stratified by context source. All averages are weighted according to the scheme described in Sec. 5.1 and accompanied by standard errors. Lower is better and the best means are in bold. * denotes models that do not use natural language context.**
| Model | Average RCRPS | Average Rank | $\mathbf{c}_{I}$ | $\mathbf{c}_{H}$ | $\mathbf{c}_{F}$ | $\mathbf{c}_{cov}$ | $\mathbf{c}_{\text{causal}}$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Direct Prompt (ours) |  |  |  |  |  |  |  |
| Llama-3.1-405B-Inst | 0.159 $\pm$ 0.008 | 4.469 $\pm$ 0.192 | 0.174 $\pm$ 0.010 | 0.146 $\pm$ 0.001 | 0.085 $\pm$ 0.003 | 0.169 $\pm$ 0.010 | 0.398 $\pm$ 0.045 |
| Llama-3-70B-Inst | 0.518 $\pm$ 0.030 | 10.406 $\pm$ 0.190 | 0.621 $\pm$ 0.042 | 0.308 $\pm$ 0.064 | 0.301 $\pm$ 0.033 | 0.452 $\pm$ 0.032 | 0.704 $\pm$ 0.056 |
| Llama-3-8B-Inst | 1.647 $\pm$ 0.069 | 15.130 $\pm$ 0.171 | 2.355 $\pm$ 0.100 | 0.813 $\pm$ 0.115 | 1.332 $\pm$ 0.094 | 1.185 $\pm$ 0.087 | 2.041 $\pm$ 0.271 |
| Mixtral-8x7B-Inst | 1.061 $\pm$ 0.058 | 13.381 $\pm$ 0.230 | 1.263 $\pm$ 0.082 | 0.561 $\pm$ 0.111 | 0.691 $\pm$ 0.094 | 0.724 $\pm$ 0.053 | 1.232 $\pm$ 0.121 |
| GPT-4o | 0.276 $\pm$ 0.010 | 4.368 $\pm$ 0.149 | 0.220 $\pm$ 0.007 | 0.118 $\pm$ 0.001 | 0.108 $\pm$ 0.001 | 0.265 $\pm$ 0.012 | 0.858 $\pm$ 0.053 |
| GPT-4o-mini | 0.353 $\pm$ 0.022 | 8.930 $\pm$ 0.177 | 0.474 $\pm$ 0.035 | 0.139 $\pm$ 0.002 | 0.141 $\pm$ 0.001 | 0.345 $\pm$ 0.030 | 0.644 $\pm$ 0.128 |
| LLMP |  |  |  |  |  |  |  |
| Llama-3-70B-Inst | 0.550 $\pm$ 0.013 | 8.038 $\pm$ 0.205 | 0.455 $\pm$ 0.018 | 0.516 $\pm$ 0.028 | 0.690 $\pm$ 0.018 | 0.588 $\pm$ 0.018 | 0.392 $\pm$ 0.028 |
| Llama-3-70B | 0.237 $\pm$ 0.006 | 6.560 $\pm$ 0.254 | 0.213 $\pm$ 0.005 | 0.121 $\pm$ 0.008 | 0.233 $\pm$ 0.012 | 0.198 $\pm$ 0.004 | 0.360 $\pm$ 0.011 |
| Llama-3-8B-Inst | 0.484 $\pm$ 0.010 | 9.457 $\pm$ 0.166 | 0.477 $\pm$ 0.013 | 0.161 $\pm$ 0.006 | 0.264 $\pm$ 0.003 | 0.316 $\pm$ 0.008 | 0.878 $\pm$ 0.035 |
| Llama-3-8B | 0.313 $\pm$ 0.023 | 9.499 $\pm$ 0.323 | 0.334 $\pm$ 0.035 | 0.123 $\pm$ 0.004 | 0.232 $\pm$ 0.012 | 0.291 $\pm$ 0.031 | 0.739 $\pm$ 0.134 |
| Mixtral-8x7B-Inst | 0.264 $\pm$ 0.004 | 8.496 $\pm$ 0.256 | 0.242 $\pm$ 0.007 | 0.173 $\pm$ 0.004 | 0.268 $\pm$ 0.009 | 0.220 $\pm$ 0.002 | 0.437 $\pm$ 0.007 |
| Mixtral-8x7B | 0.262 $\pm$ 0.008 | 8.619 $\pm$ 0.208 | 0.250 $\pm$ 0.008 | 0.119 $\pm$ 0.003 | 0.254 $\pm$ 0.013 | 0.229 $\pm$ 0.007 | 0.457 $\pm$ 0.011 |
| Multimodal Models |  |  |  |  |  |  |  |
| UniTime | 0.371 $\pm$ 0.002 | 13.495 $\pm$ 0.091 | 0.455 $\pm$ 0.002 | 0.154 $\pm$ 0.000 | 0.226 $\pm$ 0.003 | 0.396 $\pm$ 0.001 | 0.422 $\pm$ 0.001 |
| TimeLLM | 0.476 $\pm$ 0.001 | 16.662 $\pm$ 0.075 | 0.517 $\pm$ 0.002 | 0.183 $\pm$ 0.000 | 0.376 $\pm$ 0.002 | 0.446 $\pm$ 0.001 | 0.482 $\pm$ 0.001 |
| TS Foundation Models* |  |  |  |  |  |  |  |
| Lag-Llama | 0.329 $\pm$ 0.004 | 13.157 $\pm$ 0.235 | 0.333 $\pm$ 0.005 | 0.167 $\pm$ 0.005 | 0.277 $\pm$ 0.006 | 0.301 $\pm$ 0.004 | 0.495 $\pm$ 0.014 |
| Chronos | 0.326 $\pm$ 0.002 | 11.962 $\pm$ 0.139 | 0.314 $\pm$ 0.002 | 0.179 $\pm$ 0.003 | 0.316 $\pm$ 0.002 | 0.252 $\pm$ 0.002 | 0.460 $\pm$ 0.004 |
| TimeGEN | 0.354 $\pm$ 0.000 | 14.345 $\pm$ 0.090 | 0.333 $\pm$ 0.000 | 0.177 $\pm$ 0.000 | 0.348 $\pm$ 0.000 | 0.291 $\pm$ 0.000 | 0.474 $\pm$ 0.000 |
| Moirai | 0.520 $\pm$ 0.006 | 12.458 $\pm$ 0.266 | 0.596 $\pm$ 0.009 | 0.140 $\pm$ 0.001 | 0.364 $\pm$ 0.002 | 0.510 $\pm$ 0.008 | 0.438 $\pm$ 0.011 |
| Statistical Models* |  |  |  |  |  |  |  |
| ARIMA | 0.480 $\pm$ 0.006 | 12.320 $\pm$ 0.180 | 0.565 $\pm$ 0.010 | 0.200 $\pm$ 0.007 | 0.307 $\pm$ 0.003 | 0.390 $\pm$ 0.006 | 0.440 $\pm$ 0.011 |
| ETS | 0.522 $\pm$ 0.009 | 14.310 $\pm$ 0.203 | 0.627 $\pm$ 0.014 | 0.362 $\pm$ 0.014 | 0.323 $\pm$ 0.008 | 0.401 $\pm$ 0.010 | 0.508 $\pm$ 0.017 |
| Exp-Smoothing | 0.603 $\pm$ 0.013 | 14.936 $\pm$ 0.132 | 0.700 $\pm$ 0.020 | 0.493 $\pm$ 0.016 | 0.438 $\pm$ 0.009 | 0.492 $\pm$ 0.017 | 0.827 $\pm$ 0.060 |

#### C.2 Extended results on all models

**Table 3: Extended results on the CiK benchmark aggregated over all tasks. The first column shows the RCRPS averaged over all tasks. The second column shows the rank of each method w.r.t. other baselines, averaged over all tasks. All averages are weighted according to the scheme described in Sec. 5.1 and accompanied by standard errors. Lower is better and the best means are in bold.**
| Model | Average RCRPS | Average Rank |
| --- | --- | --- |
| With Context |  |  |
| Direct Prompt (ours) |  |  |
| Llama-3.1-405B-Inst | 0.159 $\pm$ 0.008 | 6.369 $\pm$ 0.384 |
| Llama-3-70B-Inst | 0.518 $\pm$ 0.030 | 17.350 $\pm$ 0.399 |
| Llama-3-8B-Inst | 1.647 $\pm$ 0.069 | 27.425 $\pm$ 0.360 |
| Mixtral-8x7B-Inst | 1.061 $\pm$ 0.058 | 24.010 $\pm$ 0.499 |
| GPT-4o | 0.276 $\pm$ 0.010 | 6.973 $\pm$ 0.304 |
| GPT-4o-mini | 0.353 $\pm$ 0.022 | 13.704 $\pm$ 0.315 |
| LLMP |  |  |
| Llama-3-70B-Inst | 0.550 $\pm$ 0.013 | 13.469 $\pm$ 0.399 |
| Llama-3-70B | 0.237 $\pm$ 0.006 | 9.646 $\pm$ 0.542 |
| Llama-3-8B-Inst | 0.484 $\pm$ 0.010 | 14.465 $\pm$ 0.356 |
| Llama-3-8B | 0.313 $\pm$ 0.023 | 14.497 $\pm$ 0.713 |
| Mixtral-8x7B-Inst | 0.264 $\pm$ 0.004 | 12.505 $\pm$ 0.482 |
| Mixtral-8x7B | 0.262 $\pm$ 0.008 | 12.826 $\pm$ 0.381 |
| Multimodal Models |  |  |
| UniTime | 0.371 $\pm$ 0.002 | 24.928 $\pm$ 0.135 |
| Time-LLM | 0.476 $\pm$ 0.001 | 31.632 $\pm$ 0.117 |
| Without Context |  |  |
| Direct Prompt (ours) |  |  |
| Llama-3.1-405B-Inst | 0.473 $\pm$ 0.005 | 25.194 $\pm$ 0.219 |
| Llama-3-70B-Inst | 0.714 $\pm$ 0.035 | 28.141 $\pm$ 0.348 |
| Llama-3-8B-Inst | 1.900 $\pm$ 0.059 | 34.027 $\pm$ 0.265 |
| Mixtral-8x7B-Inst | 0.847 $\pm$ 0.045 | 26.253 $\pm$ 0.494 |
| GPT-4o | 0.441 $\pm$ 0.008 | 23.450 $\pm$ 0.289 |
| GPT-4o-mini | 0.423 $\pm$ 0.006 | 26.131 $\pm$ 0.199 |
| LLMP |  |  |
| Llama-3-70B-Inst | 0.378 $\pm$ 0.004 | 19.742 $\pm$ 0.349 |
| Llama-3-70B | 0.312 $\pm$ 0.006 | 16.413 $\pm$ 0.376 |
| Llama-3-8B-Inst | 0.503 $\pm$ 0.009 | 23.125 $\pm$ 0.337 |
| Llama-3-8B | 0.345 $\pm$ 0.003 | 18.819 $\pm$ 0.278 |
| Mixtral-8x7B-Inst | 0.383 $\pm$ 0.015 | 18.454 $\pm$ 0.361 |
| Mixtral-8x7B | 0.306 $\pm$ 0.007 | 17.097 $\pm$ 0.386 |
| Multimodal Models |  |  |
| UniTime | 0.405 $\pm$ 0.002 | 26.875 $\pm$ 0.128 |
| Time-LLM | 0.454 $\pm$ 0.002 | 30.067 $\pm$ 0.125 |
| TS Foundation Models |  |  |
| Lag-Llama | 0.329 $\pm$ 0.004 | 22.781 $\pm$ 0.626 |
| Chronos-Tiny | 0.328 $\pm$ 0.001 | 20.783 $\pm$ 0.335 |
| Chronos-Mini | 0.341 $\pm$ 0.001 | 21.926 $\pm$ 0.331 |
| Chronos-Small | 0.328 $\pm$ 0.002 | 20.126 $\pm$ 0.269 |
| Chronos-Base | 0.672 $\pm$ 0.003 | 22.519 $\pm$ 0.283 |
| Chronos-Large | 0.326 $\pm$ 0.002 | 19.255 $\pm$ 0.311 |
| TimeGEN | 0.354 $\pm$ 0.000 | 26.700 $\pm$ 0.121 |
| Moirai-Small | 0.565 $\pm$ 0.031 | 26.323 $\pm$ 0.353 |
| Moirai-Base | 0.624 $\pm$ 0.013 | 26.065 $\pm$ 0.275 |
| Moirai-Large | 0.520 $\pm$ 0.006 | 21.041 $\pm$ 0.718 |
| Statistical Models |  |  |
| ARIMA | 0.480 $\pm$ 0.006 | 19.780 $\pm$ 0.386 |
| ETS | 0.522 $\pm$ 0.009 | 24.259 $\pm$ 0.476 |
| Exp-Smoothing | 0.603 $\pm$ 0.013 | 25.858 $\pm$ 0.241 |

#### C.3 Significant failures per model

We observe that in a few instances in the benchmark, some models tend to obtain significantly worse performance when evaluated with context.
In our evaluation, we term all instances where the RCRPS value of a model is greater than $5$, as significant failures of the model on those instances. We found $5$ as a suitable value for analyzing such failures, as it intuitively represents the value a forecast would get if the distance between the forecast and the ground-truth was 5 times bigger than the range of the ground-truth for the task.
When we aggregate the RCRPS of instances in the benchmark (such as in Tab. 1), we cap the RCRPS of such significant failures to $5$, to avoid outliers with a much higher RCRPS affecting the aggregate score. In Tab. 4, we show the number of such instances in our evaluation of the benchmark where we found models to have significant failures (out of a total of $355$ evaluated instances). Interestingly, some models such as Direct Prompt with Llama-3.1-405B-Instruct and LLMP with Llama-3-70B and Llama-3-8B are more robust to such significant failures, and do not incur such failures. On the other hand, models such as LLMP with Llama-3-70B-Inst and Llama-3-8B-Inst achieve significant failures in a large number of instances from the benchmark. We postulate that this is because of models misinterpreting context. It is still an open question as to how to increase the robustness of models to prevent or reduce such significant failures. We visualize such significant failures in Sec. C.5.

**Table 4: Number of instances with significant failures in models that support context**
| Model | Number of instances with significant failures |
| --- | --- |
| Direct Prompt (ours) |  |
| Llama-3.1-405B-Inst | 0 |
| Llama-3-70B-Inst | 1 |
| Llama-3-8B-Inst | 3 |
| Mixtral-8x7B-Inst | 5 |
| GPT-4o | 5 |
| GPT-4o-mini | 2 |
| LLMP |  |
| Llama-3-70B-Inst | 18 |
| Llama-3-70B | 0 |
| Llama-3-8B-Inst | 12 |
| Llama-3-8B | 0 |
| Mixtral-8x7B-Inst | 1 |
| Mixtral-8x7B | 1 |
| Multimodal Models |  |
| UniTime | 0 |
| Time-LLM | 2 |

#### C.4 Visualizations of successful context-aware forecasts

Figure: (a) Without Context
Refer to caption: x12.png

Figure: (a) Without Context
Refer to caption: x14.png

Figure: (a) Without Context
Refer to caption: x16.png

Figure: (a) Without Context
Refer to caption: x18.png

Figure: (a) Without Context
Refer to caption: x20.png

Figure: (a) Without Context
Refer to caption: x22.png

Figure: (a) Without Context
Refer to caption: x24.png

Figure: (a) Without Context
Refer to caption: x26.png

#### C.5 Visualizations of significant failures

Figure: (a) Without Context
Refer to caption: x28.png

Figure: (a) Without Context
Refer to caption: x30.png

Figure: (a) Without Context
Refer to caption: x32.png

Figure: (a) Without Context
Refer to caption: x34.png

#### C.6 Cost of API-based models

Tab. 5 provides the cost incurred in evaluating GPT-4o (version gpt-4o-2024-05-13) and GPT-4o-mini (version gpt-4o-mini-2024-07-18) with the Direct Prompt method on CiK (as per the evaluation protocol used, described in Sec. 5.1).

**Table 5: Costs ($CAD) of evaluating the GPT-4o family of models on CiK. “Total” represents the total cost of evaluating each model on the CiK benchmark. The “Per-instance average” and the “Per-instance median” are the average and median cost of running a single instance for a given task, in other words the average and median cost of generating 25 sample trajectories for a given example of a task. As a reminder, each task in CiK is evaluated over 5 instances in our evaluation protocol.**
| Model | Total | Per-instance average | Per-instance median |
| --- | --- | --- | --- |
| GPT-4o | $143.83 | $0.288 | $0.170 |
| GPT-4o (no context) | $139.50 | $0.279 | $0.160 |
| GPT-4o-mini | $13.79 | $0.040 | $0.040 |
| GPT-4o-mini (no context) | $13.32 | $0.038 | $0.040 |

#### C.7 Impact of Irrelevant Information in Context

Many of our tasks include covariates in its context which are highly useful for the models to accurately predict the target time series.
One question is: Do the LLM-based models perform well for such tasks due to correctly understanding that said covariates are helpful or because they blindly use the provided data without asking themselves if the data is actually relevant?

As a way to get some insight on this question, we took a task where the models have to forecast the unemployment data of an American county, given the unemployment data of the state the county is in.
We then modify this task by first trying to mislead the model by wrongly saying that the state-level data was from another state (without changing the data itself), then by giving the data from the other state (while explicitly telling the model that data is from said other state), before finally removing the state-level data altogether.
The result for this experiment with 5 instances per task for Direct Prompt - GPT-4o is shown in Tab. 6, while the forecasts for a single instances are shown in Fig. 20.
From these, we see that the model aggressively used data which is marked as being from an other state, even though if the data was actually from said other state, the performance would be closer to not having any state-level data.
This shows that the model is liable to take any information provided as being useful, even though its usefulness is marginal.

**Table 6: Ability of the Direct Prompt - GPT-4o model to accurately predict the unemployement level of an American county, given various covariates. These results are averaged over 5 instances.**
| Available data | RCPRS |
| --- | --- |
| Data from the correct state, accurately tagged | 0.0583 |
| Data from the correct state, inaccurately tagged | 0.0557 |
| Data from an incorrect state, accurately tagged | 0.1966 |
| No state-level data | 0.2630 |

Figure: (a) The task in our benchmark: the context contains the unemployment rate of the state the county is in, correctly tagged with the state name.
Refer to caption: x36.png

### Appendix D Implementation Details of Models

#### D.1 Direct Prompt

##### D.1.1 Method

For Direct Prompt, we propose to use a simple prompt template that we describe below, where $(($context$))$ is replaced with the context of the respective task, $(($history$))$ is replaced with the historical values in the given format, and $(($pred_time$))$ is replaced with the prediction timesteps.
The prompted model is expected to output predictions in the given template style (i.e. within the given forecast tags, in the given format) for all prediction timesteps in the prompt.
Notably, unlike LLMP which consists of predicting the single next digit in a loop, Direct Prompt expects models to forecast in a single pass in a highly structured format, which requires models to understand and adhere to the template.

| [⬇](data:text/plain;base64,SSBoYXZlIGEgdGltZSBzZXJpZXMgZm9yZWNhc3RpbmcgdGFzayBmb3IgeW91LgoKSGVyZSBpcyBzb21lIGNvbnRleHQgYWJvdXQgdGhlIHRhc2suIE1ha2Ugc3VyZSB0byBmYWN0b3IgaW4gYW55IGJhY2tncm91bmQga25vd2xlZGdlLApzYXRpc2Z5IGFueSBjb25zdHJhaW50cywgYW5kIHJlc3BlY3QgYW55IHNjZW5hcmlvcy4KPGNvbnRleHQ+CigoY29udGV4dCkpCjwvY29udGV4dD4KCkhlcmUgaXMgYSBoaXN0b3JpY2FsIHRpbWUgc2VyaWVzIGluICh0aW1lc3RhbXAsIHZhbHVlKSBmb3JtYXQ6CjxoaXN0b3J5PgooKGhpc3RvcnkpKQo8L2hpc3Rvcnk+CgpOb3cgcGxlYXNlIHByZWRpY3QgdGhlIHZhbHVlIGF0IHRoZSBmb2xsb3dpbmcgdGltZXN0YW1wczogKChwcmVkX3RpbWUpKS4KClJldHVybiB0aGUgZm9yZWNhc3QgaW4gKHRpbWVzdGFtcCwgdmFsdWUpIGZvcm1hdCBpbiBiZXR3ZWVuIDxmb3JlY2FzdD4gYW5kIDwvZm9yZWNhc3Q+IHRhZ3MuCkRvIG5vdCBpbmNsdWRlIGFueSBvdGhlciBpbmZvcm1hdGlvbiAoZS5nLiwgY29tbWVudHMpIGluIHRoZSBmb3JlY2FzdC4KCkV4YW1wbGU6CjxoaXN0b3J5PgoodDEsIHYxKQoodDIsIHYyKQoodDMsIHYzKQo8L2hpc3Rvcnk+Cjxmb3JlY2FzdD4KKHQ0LCB2NCkKKHQ1LCB2NSkKPC9mb3JlY2FzdD4KIg==)<br>I have a time series forecasting task for you.<br>Here is some context about the task. Make sure to factor in any background knowledge,<br>satisfy any constraints, and respect any scenarios.<br><context><br>((context))<br></context><br>Here is a historical time series in (timestamp, value) format:<br><history><br>((history))<br></history><br>Now please predict the value at the following timestamps: ((pred_time)).<br>Return the forecast in (timestamp, value) format in between <forecast> and </forecast> tags.<br>Do not include any other information (e.g., comments) in the forecast.<br>Example:<br><history><br>(t1, v1)<br>(t2, v2)<br>(t3, v3)<br></history><br><forecast><br>(t4, v4)<br>(t5, v5)<br></forecast><br>" |
| --- |

We observe that models often produce samples that fail to adhere to the structure and are therefore rejected. When sampling $25$ samples from the model, with Direct Prompt, we allow retrying for a maximum of $K$ times until we obtain $25$ valid samples. If we do not have $25$ valid samples from the model at the end of $K$ retries, we record a failure of the model and attribute the model the RCRPS upper bound of 5 for that task. In practice, we find that larger models (Llama-3.1-405B-Instruct, GPT-4o and GPT-4o) can produce $25$ valid forecasts with $1$ to $3$ retries. However with the smaller models (such as Llama-3-70B-Instruct, Llama-3-8B-Instruct and Mixtral-8x7B-Instruct), up to $10$ retries can be required to obtain $25$ valid forecasts. Further, we found that without an explicit “Do not include any other information (e.g., comments) in the forecast.”, models often included unwanted information along with the forecasts.

###### Instruction-tuning is necessary for models to work with Direct Prompt

Direct Prompting requires forecasts to be produced in a specific structure.
To generate structured outputs, models need to be steerable (Dubey et al., ), a capability that is typically elicited from base models with post-training methods such as instruction tuning (Wei et al., ). We observe this in our evaluations as we find that several base models, including Llama-3-8B, Llama-3-70B, Mixtral-8x7B, and even the biggest base model we tried, Llama-3.1-405B, are incapable of generating outputs adhering to the structure required for Direct Prompt, despite increasing the number of retries to as high as $50$ retries. With Direct Prompt, these models often output irrelevant information, sometimes completing solely the context as a text completion task, and in other cases regurgitating forecasting datasets that they have memorized.

###### Extensions of Direct Prompt

While very simple, such prompt templates can be powerful tools to understand how LLMs perform context-aided forecasting: as the prompt gives control over the structure and content of the output (particularly for instruction-tuned models), one may construct other, more involved template structures in the prompt.
For instance, a prompt template could ask LLMs to explain the reasoning behind their (context-aided) forecasts, and more. We leave it to future work to understand how such prompt-based techniques can lead to more detailed evaluations and give us better insights into what the models are capable of.

##### D.1.2 Implementation Details

We used a single H100 GPU to run the Direct Prompt approach for Llama-3-8B-Instruct, and 2 H100 GPUs for Llama-3-70B-Instruct and Mixtral-8x7B-Instruct.
We queried Llama-3.1-405b-Instruct from an externally-hosted server running on 8 H100s.
We use the OpenAI API to perform inference on the proprietary GPT-4o and GPT-4o-mini models. We provide the cost incurred in the inference of these models with the Direct Prompt method in Sec. C.6.

##### D.1.3 Example Prompt

A prompt used in an example task from the benchmark is given below.

| [⬇](data:text/plain;base64,SSBoYXZlIGEgdGltZSBzZXJpZXMgZm9yZWNhc3RpbmcgdGFzayBmb3IgeW91LgoKSGVyZSBpcyBzb21lIGNvbnRleHQgYWJvdXQgdGhlIHRhc2suIE1ha2Ugc3VyZSB0byBmYWN0b3IgaW4gYW55IGJhY2tncm91bmQga25vd2xlZGdlLHNhdGlzZnkgYW55IGNvbnN0cmFpbnRzLCBhbmQgcmVzcGVjdCBhbnkgc2NlbmFyaW9zLgo8Y29udGV4dD4KQmFja2dyb3VuZDogVGhpcyBpcyBob3VybHkgdHJhZmZpYyBkYXRhLgpTY2VuYXJpbzogU3VwcG9zZSB0aGF0IHRoZXJlIGlzIGFuIGFjY2lkZW50IG9uIHRoZSByb2FkIGFuZCB0aGVyZSBpcyA0MC4wJSBvZiB0aGUgdXN1YWwgdHJhZmZpYyBmcm9tIDIwMjQtMDQtMjQgMTc6MDA6MDAgZm9yIDYgaG91cnMuCjwvY29udGV4dD4KCkhlcmUgaXMgYSBoaXN0b3JpY2FsIHRpbWUgc2VyaWVzIGluICh0aW1lc3RhbXAsIHZhbHVlKSBmb3JtYXQ6CjxoaXN0b3J5PgooMjAyNC0wNC0yMyAwMDowMDowMCwgMC4xKSgyMDI0LTA0LTIzIDAxOjAwOjAwLCAwKSgyMDI0LTA0LTIzIDAyOjAwOjAwLCAwKSgyMDI0LTA0LTIzIDAzOjAwOjAwLCAwKSgyMDI0LTA0LTIzIDA0OjAwOjAwLCAwLjEpKDIwMjQtMDQtMjMgMDU6MDA6MDAsIDAuMikoMjAyNC0wNC0yMyAwNjowMDowMCwgMC4zKSgyMDI0LTA0LTIzIDA3OjAwOjAwLCAwLjUpKDIwMjQtMDQtMjMgMDg6MDA6MDAsIDAuNSkoMjAyNC0wNC0yMyAwOTowMDowMCwgMC40KSgyMDI0LTA0LTIzIDEwOjAwOjAwLCAwLjUpKDIwMjQtMDQtMjMgMTE6MDA6MDAsIDAuNSkoMjAyNC0wNC0yMyAxMjowMDowMCwgMC40KSgyMDI0LTA0LTIzIDEzOjAwOjAwLCAwLjYpKDIwMjQtMDQtMjMgMTQ6MDA6MDAsIDAuOCkoMjAyNC0wNC0yMyAxNTowMDowMCwgMS4yKSgyMDI0LTA0LTIzIDE2OjAwOjAwLCAxLjIpKDIwMjQtMDQtMjMgMTc6MDA6MDAsIDEuMykoMjAyNC0wNC0yMyAxODowMDowMCwgMC42KSgyMDI0LTA0LTIzIDE5OjAwOjAwLCAwLjMpKDIwMjQtMDQtMjMgMjA6MDA6MDAsIDAuMykoMjAyNC0wNC0yMyAyMTowMDowMCwgMC4zKSgyMDI0LTA0LTIzIDIyOjAwOjAwLCAwLjEpKDIwMjQtMDQtMjMgMjM6MDA6MDAsIDAuMSkoMjAyNC0wNC0yNCAwMDowMDowMCwgMC4xKSgyMDI0LTA0LTI0IDAxOjAwOjAwLCAwKSgyMDI0LTA0LTI0IDAyOjAwOjAwLCAwKSgyMDI0LTA0LTI0IDAzOjAwOjAwLCAwLjEpKDIwMjQtMDQtMjQgMDQ6MDA6MDAsIDAuMSkoMjAyNC0wNC0yNCAwNTowMDowMCwgMC4yKSgyMDI0LTA0LTI0IDA2OjAwOjAwLCAwLjMpKDIwMjQtMDQtMjQgMDc6MDA6MDAsIDAuNSkoMjAyNC0wNC0yNCAwODowMDowMCwgMC42KSgyMDI0LTA0LTI0IDA5OjAwOjAwLCAwLjUpKDIwMjQtMDQtMjQgMTA6MDA6MDAsIDAuNCkoMjAyNC0wNC0yNCAxMTowMDowMCwgMC41KSgyMDI0LTA0LTI0IDEyOjAwOjAwLCAwLjYpCjwvaGlzdG9yeT4KCk5vdyBwbGVhc2UgcHJlZGljdCB0aGUgdmFsdWUgYXQgdGhlIGZvbGxvd2luZyB0aW1lc3RhbXBzOiBbJzIwMjQtMDQtMjQgMTM6MDA6MDAnICcyMDI0LTA0LTI0IDE0OjAwOjAwJyAnMjAyNC0wNC0yNCAxNTowMDowMCcgJzIwMjQtMDQtMjQgMTY6MDA6MDAnICcyMDI0LTA0LTI0IDE3OjAwOjAwJyAnMjAyNC0wNC0yNCAxODowMDowMCcgJzIwMjQtMDQtMjQgMTk6MDA6MDAnICcyMDI0LTA0LTI0IDIwOjAwOjAwJyAnMjAyNC0wNC0yNCAyMTowMDowMCcgJzIwMjQtMDQtMjQgMjI6MDA6MDAnICcyMDI0LTA0LTI0IDIzOjAwOjAwJyAnMjAyNC0wNC0yNSAwMDowMDowMCcgJzIwMjQtMDQtMjUgMDE6MDA6MDAnICcyMDI0LTA0LTI1IDAyOjAwOjAwJyAnMjAyNC0wNC0yNSAwMzowMDowMCcgJzIwMjQtMDQtMjUgMDQ6MDA6MDAnICcyMDI0LTA0LTI1IDA1OjAwOjAwJyAnMjAyNC0wNC0yNSAwNjowMDowMCcgJzIwMjQtMDQtMjUgMDc6MDA6MDAnICcyMDI0LTA0LTI1IDA4OjAwOjAwJyAnMjAyNC0wNC0yNSAwOTowMDowMCcgJzIwMjQtMDQtMjUgMTA6MDA6MDAnICcyMDI0LTA0LTI1IDExOjAwOjAwJyAnMjAyNC0wNC0yNSAxMjowMDowMCddLgoKUmV0dXJuIHRoZSBmb3JlY2FzdCBpbiAodGltZXN0YW1wLCB2YWx1ZSkgZm9ybWF0IGluIGJldHdlZW4gPGZvcmVjYXN0PiBhbmQgPC9mb3JlY2FzdD4gdGFncy5EbyBub3QgaW5jbHVkZSBhbnkgb3RoZXIgaW5mb3JtYXRpb24gKGUuZy4sIGNvbW1lbnRzKSBpbiB0aGUgZm9yZWNhc3QuCgpFeGFtcGxlOgo8aGlzdG9yeT4KKHQxLCB2MSkKKHQyLCB2MikKKHQzLCB2MykKPC9oaXN0b3J5Pgo8Zm9yZWNhc3Q+Cih0NCwgdjQpCih0NSwgdjUpCjwvZm9yZWNhc3Q+CiI=)<br>I have a time series forecasting task for you.<br>Here is some context about the task. Make sure to factor in any background knowledge,satisfy any constraints, and respect any scenarios.<br><context><br>Background: This is hourly traffic data.<br>Scenario: Suppose that there is an accident on the road and there is 40.0% of the usual traffic from 2024-04-24 17:00:00 for 6 hours.<br></context><br>Here is a historical time series in (timestamp, value) format:<br><history><br>(2024-04-23 00:00:00, 0.1)(2024-04-23 01:00:00, 0)(2024-04-23 02:00:00, 0)(2024-04-23 03:00:00, 0)(2024-04-23 04:00:00, 0.1)(2024-04-23 05:00:00, 0.2)(2024-04-23 06:00:00, 0.3)(2024-04-23 07:00:00, 0.5)(2024-04-23 08:00:00, 0.5)(2024-04-23 09:00:00, 0.4)(2024-04-23 10:00:00, 0.5)(2024-04-23 11:00:00, 0.5)(2024-04-23 12:00:00, 0.4)(2024-04-23 13:00:00, 0.6)(2024-04-23 14:00:00, 0.8)(2024-04-23 15:00:00, 1.2)(2024-04-23 16:00:00, 1.2)(2024-04-23 17:00:00, 1.3)(2024-04-23 18:00:00, 0.6)(2024-04-23 19:00:00, 0.3)(2024-04-23 20:00:00, 0.3)(2024-04-23 21:00:00, 0.3)(2024-04-23 22:00:00, 0.1)(2024-04-23 23:00:00, 0.1)(2024-04-24 00:00:00, 0.1)(2024-04-24 01:00:00, 0)(2024-04-24 02:00:00, 0)(2024-04-24 03:00:00, 0.1)(2024-04-24 04:00:00, 0.1)(2024-04-24 05:00:00, 0.2)(2024-04-24 06:00:00, 0.3)(2024-04-24 07:00:00, 0.5)(2024-04-24 08:00:00, 0.6)(2024-04-24 09:00:00, 0.5)(2024-04-24 10:00:00, 0.4)(2024-04-24 11:00:00, 0.5)(2024-04-24 12:00:00, 0.6)<br></history><br>Now please predict the value at the following timestamps: [’2024-04-24 13:00:00’ ’2024-04-24 14:00:00’ ’2024-04-24 15:00:00’ ’2024-04-24 16:00:00’ ’2024-04-24 17:00:00’ ’2024-04-24 18:00:00’ ’2024-04-24 19:00:00’ ’2024-04-24 20:00:00’ ’2024-04-24 21:00:00’ ’2024-04-24 22:00:00’ ’2024-04-24 23:00:00’ ’2024-04-25 00:00:00’ ’2024-04-25 01:00:00’ ’2024-04-25 02:00:00’ ’2024-04-25 03:00:00’ ’2024-04-25 04:00:00’ ’2024-04-25 05:00:00’ ’2024-04-25 06:00:00’ ’2024-04-25 07:00:00’ ’2024-04-25 08:00:00’ ’2024-04-25 09:00:00’ ’2024-04-25 10:00:00’ ’2024-04-25 11:00:00’ ’2024-04-25 12:00:00’].<br>Return the forecast in (timestamp, value) format in between <forecast> and </forecast> tags.Do not include any other information (e.g., comments) in the forecast.<br>Example:<br><history><br>(t1, v1)<br>(t2, v2)<br>(t3, v3)<br></history><br><forecast><br>(t4, v4)<br>(t5, v5)<br></forecast><br>" |
| --- |

#### D.2 LLMP

In this section we outline LLM-processes (LLMP; Requeima et al. ()), one of the prompt-based baselines evaluated in Sec. 5.3. Prompts are constructed by first providing textual information followed by the numerical history. The context may include background knowledge, a scenario description and task constraints, replaced by $(($background$))$, $(($scenario$))$ and $(($constraints$))$, respectively, in the prompt template below. The numerical history ($(($history$))$) is provided by converting the numerical data to text where values are separated by commas (,) and tuples by newline characters (\n). The LLM then outputs the continuation of the string prompt, forecasing the the value for the next time index ($(($next index$))$). This forecast and the next time index is appended to the prompt allowing the LLM to autoregressively complete the entire forecast. Numerical samples are rejected if they do not adhere to a decimal representation format. See Requeima et al. ()) for full details.

The following is the prompt template used to construct prompts for the LLMP baseline:

| [⬇](data:text/plain;base64,IgpGb3JlY2FzdCB0aGUgZnV0dXJlIHZhbHVlcyBvZiB0aGlzIHRpbWUgc2VyaWVzLCB3aGlsZSBjb25zaWRlcmluZyB0aGUgZm9sbG93aW5nIGJhY2tncm91bmQga25vd2xlZGdlLCBzY2VuYXJpbywgYW5kIGNvbnN0cmFpbnRzLgoKQmFja2dyb3VuZCBrbm93bGVkZ2U6CigoYmFja2dyb3VuZCkpCgpTY2VuYXJpbzoKKChzY2VuYXJpbykpCgpDb25zdHJhaW50czoKKChjb25zdHJhaW50cykpCgooKGhpc3RvcnkpKQooKG5leHQgaW5kZXgpKQoi)<br>"<br>Forecast the future values of this time series, while considering the following background knowledge, scenario, and constraints.<br>Background knowledge:<br>((background))<br>Scenario:<br>((scenario))<br>Constraints:<br>((constraints))<br>((history))<br>((next index))<br>" |
| --- |

A prompt used in an example task from the benchmark is given below:

| [⬇](data:text/plain;base64,IgpGb3JlY2FzdCB0aGUgZnV0dXJlIHZhbHVlcyBvZiB0aGlzIHRpbWUgc2VyaWVzLCB3aGlsZSBjb25zaWRlcmluZyB0aGUgZm9sbG93aW5nIGJhY2tncm91bmQga25vd2xlZGdlLCBzY2VuYXJpbywgYW5kIGNvbnN0cmFpbnRzLgoKQmFja2dyb3VuZCBrbm93bGVkZ2U6ClRoaXMgaXMgaG91cmx5IHRyYWZmaWMgZGF0YS4KClNjZW5hcmlvOgpTdXBwb3NlIHRoYXQgdGhlcmUgaXMgYW4gYWNjaWRlbnQgb24gdGhlIHJvYWQgYW5kIHRoZXJlIGlzIDQwLjAlIG9mIHRoZSB1c3VhbCB0cmFmZmljIGZyb20gMjAyNC0wNC0yNCAxNzowMDowMCBmb3IgNiBob3Vycy4KCkNvbnN0cmFpbnRzOgoKMjAyNC0wNC0yMyAwMDowMDowMCwwLjFcbjIwMjQtMDQtMjMgMDE6MDA6MDAsMFxuMjAyNC0wNC0yMyAwMjowMDowMCwwXG4yMDI0LTA0LTIzIDAzOjAwOjAwLDBcbjIwMjQtMDQtMjMgMDQ6MDA6MDAsMC4xXG4yMDI0LTA0LTIzIDA1OjAwOjAwLDAuMlxuMjAyNC0wNC0yMyAwNjowMDowMCwwLjNcbjIwMjQtMDQtMjMgMDc6MDA6MDAsMC41XG4yMDI0LTA0LTIzIDA4OjAwOjAwLDAuNVxuMjAyNC0wNC0yMyAwOTowMDowMCwwLjRcbjIwMjQtMDQtMjMgMTA6MDA6MDAsMC41XG4yMDI0LTA0LTIzIDExOjAwOjAwLDAuNVxuMjAyNC0wNC0yMyAxMjowMDowMCwwLjRcbjIwMjQtMDQtMjMgMTM6MDA6MDAsMC42XG4yMDI0LTA0LTIzIDE0OjAwOjAwLDAuOFxuMjAyNC0wNC0yMyAxNTowMDowMCwxLjJcbjIwMjQtMDQtMjMgMTY6MDA6MDAsMS4yXG4yMDI0LTA0LTIzIDE3OjAwOjAwLDEuM1xuMjAyNC0wNC0yMyAxODowMDowMCwwLjZcbjIwMjQtMDQtMjMgMTk6MDA6MDAsMC4zXG4yMDI0LTA0LTIzIDIwOjAwOjAwLDAuM1xuMjAyNC0wNC0yMyAyMTowMDowMCwwLjNcbjIwMjQtMDQtMjMgMjI6MDA6MDAsMC4xXG4yMDI0LTA0LTIzIDIzOjAwOjAwLDAuMVxuMjAyNC0wNC0yNCAwMDowMDowMCwwLjFcbjIwMjQtMDQtMjQgMDE6MDA6MDAsMFxuMjAyNC0wNC0yNCAwMjowMDowMCwwXG4yMDI0LTA0LTI0IDAzOjAwOjAwLDAuMVxuMjAyNC0wNC0yNCAwNDowMDowMCwwLjFcbjIwMjQtMDQtMjQgMDU6MDA6MDAsMC4yXG4yMDI0LTA0LTI0IDA2OjAwOjAwLDAuM1xuMjAyNC0wNC0yNCAwNzowMDowMCwwLjVcbjIwMjQtMDQtMjQgMDg6MDA6MDAsMC42XG4yMDI0LTA0LTI0IDA5OjAwOjAwLDAuNVxuMjAyNC0wNC0yNCAxMDowMDowMCwwLjRcbjIwMjQtMDQtMjQgMTE6MDA6MDAsMC41XG4yMDI0LTA0LTI0IDEyOjAwOjAwLDAuNlxuMjAyNC0wNC0yNCAxMzowMDowMCwKIg==)<br>"<br>Forecast the future values of this time series, while considering the following background knowledge, scenario, and constraints.<br>Background knowledge:<br>This is hourly traffic data.<br>Scenario:<br>Suppose that there is an accident on the road and there is 40.0% of the usual traffic from 2024-04-24 17:00:00 for 6 hours.<br>Constraints:<br>2024-04-23 00:00:00,0.1\n2024-04-23 01:00:00,0\n2024-04-23 02:00:00,0\n2024-04-23 03:00:00,0\n2024-04-23 04:00:00,0.1\n2024-04-23 05:00:00,0.2\n2024-04-23 06:00:00,0.3\n2024-04-23 07:00:00,0.5\n2024-04-23 08:00:00,0.5\n2024-04-23 09:00:00,0.4\n2024-04-23 10:00:00,0.5\n2024-04-23 11:00:00,0.5\n2024-04-23 12:00:00,0.4\n2024-04-23 13:00:00,0.6\n2024-04-23 14:00:00,0.8\n2024-04-23 15:00:00,1.2\n2024-04-23 16:00:00,1.2\n2024-04-23 17:00:00,1.3\n2024-04-23 18:00:00,0.6\n2024-04-23 19:00:00,0.3\n2024-04-23 20:00:00,0.3\n2024-04-23 21:00:00,0.3\n2024-04-23 22:00:00,0.1\n2024-04-23 23:00:00,0.1\n2024-04-24 00:00:00,0.1\n2024-04-24 01:00:00,0\n2024-04-24 02:00:00,0\n2024-04-24 03:00:00,0.1\n2024-04-24 04:00:00,0.1\n2024-04-24 05:00:00,0.2\n2024-04-24 06:00:00,0.3\n2024-04-24 07:00:00,0.5\n2024-04-24 08:00:00,0.6\n2024-04-24 09:00:00,0.5\n2024-04-24 10:00:00,0.4\n2024-04-24 11:00:00,0.5\n2024-04-24 12:00:00,0.6\n2024-04-24 13:00:00,<br>" |
| --- |

We used a single H100 GPU to run the LLMP approach for the following models: Llama-3-8B, and Llama-3-8B-Instruct.
We used 2 H100 GPUs for the following models: Mixtral-8x7B, and Mixtral-8x7B-Instruct, and used
used 8 H100 GPUs for the following models: Llama-3-70B, and Llama-3-70B-Instruct.

#### D.3 UniTime and Time-LLM

For multimodal models, we jointly train UniTime (Liu et al., ) on its ensemble of datasets: ETTm1, ETTm2, ETTh1, ETTh2, Electricity, Weather, Exchange and Illness.

We also evaluate Time-LLM (Jin et al., ), another multimodal model built on top of the Llama architecture. We train Time-LLM on ETTh1 according to the authors’ suggested specifications, and we compare the performance of both models with and without context.

UniTime:  We train UniTime (Liu et al., ) using a single seed on one AMD Instinct MI200 GPU for approximately 14 hours. It features a lightweight transformer with maximum context length of 210 and a pre-trained GPT2 language model as backbone, of which only the first half of the transformer layers are used. The time series baseline employs non-overlapping patch embeddings generated with a kernel size and stride of 16, and a maximum input sequence length of 96.
When the total tokenized length exceeds the architecture’s capacity, we truncate the context.

Unlike Time-LLM, UniTime is jointly trained on all datasets simultaneously. Batches were generated by first choosing a dataset uniformly at random then returning a batch from the associated data loader. To account for domain convergence speed imbalance, a mask rate of 0.5 is used and the training batch size is varied according to the dataset (details in the data config directory of the UniTime GitHub repository). Training was conducted for 10 epochs of the mixed dataset, with cosine decay from an initial learning rate of 1e-4 to a minimum of 1e-6 over a maximum period of 20 epochs. The results of our training on the original datasets are given in Tab. 7.

Finally, in order to accelerate training, we added BF16 automatic mixed precision training and gradient accumulation to the original training procedure.

**Table 7: Evaluation results for UniTime on their test splits. Results are comparable to the original paper, although MSE on Illness is approximately 20% higher for prediction lengths 36,48,60.**
| Dataset | Mean Squared Error (MSE) |  |  |  |
| --- | --- | --- | --- | --- |
| Prediction Length | 96 | 192 | 336 | 720 |
| ETTh1 | 0.395 | 0.435 | 0.469 | 0.468 |
| ETTh2 | 0.291 | 0.368 | 0.413 | 0.422 |
| ETTm1 | 0.336 | 0.377 | 0.409 | 0.465 |
| ETTm2 | 0.181 | 0.248 | 0.315 | 0.417 |
| Exchange | 0.090 | 0.180 | 0.322 | 0.862 |
| Weather | 0.179 | 0.224 | 0.278 | 0.354 |
| Electricity | 0.198 | 0.202 | 0.217 | 0.257 |
|  | 24 | 36 | 48 | 60 |
| Illness | 2.284 | 2.515 | 2.572 | 2.455 |

Time-LLM:  We train Time-LLM (Jin et al., ) on the ETTh1 dataset (Zhou et al., ) with a prediction length of 96.
We train using a single seed on four AMD Instinct MI200 GPUs, with an average training time per run of approximately 13 hours.
Training was conducted using a batch size of 8 per device and 4 gradient accumulation steps, along with a 1Cycle learning rate schedule with a maximum learning rate of 1e-3.
In addition, runs were accelerated using DeepSpeed Stage 2 and BF16 automatic mixed precision.

Training was conducted over a maximum of 50 epochs with early stopping, and a time-based split of 70% for training, 10% for validation, and 20% for testing, where the most recent windows were reserved for the test set. All runs were trained with an input sequence length of 512, with overlapping patch embeddings generated with a kernel size of 16 and a stride of 8. The results on the ETTh1 test set are given in Tab. 8.

When evaluating on CiK tasks which do not conform to Time-LLM’s requirements, we make the following modifications to the method:

- •
For short history tasks where the history length $|\mathbf{X_{H}}|$ is less than 5, we change the topk operator’s $k$ value from 5 to $|\mathbf{X_{H}}|$ in the calculate_lags() function.
- •
For tasks where the length of the prediction window $|\mathbf{X_{F}}|$exceeds the trained projection head’s output dimension (in our case, 96), we repeat the last predicted value $|\mathbf{X_{F}}|-96$ times. This occurs for very few tasks (3 tasks) with prediction windows of 97 or 98
steps depending on the sampled instance, which we assume leads to a negligible impact on evaluated results.

**Table 8: ETTh1 test set results for Time-LLM trained on ETTh1.**
| Time-LLM | MSE | MAE |
| --- | --- | --- |
| ETTh1-pl96 | 0.3846123 | 0.4149854 |

Why Do Time-LLM and UniTime Not Benefit (More) From Context?
Looking at table Sec. C.2, we see that context actually harms the performance of Time-LLM’s forecasts.
Two possible reasons for this are: 1) Time-LLM’s adaptation procedure is unlikely to retain the backbone LLM’s language-processing capabilities, and 2) Time-LLM’s single-dataset training procedure is unlikely to generalize to unseen time series patterns.
Part of Time-LLM’s model adaptation involves training linear layers at the input and output of the language model.
Although the backbone LLM remains frozen, these linear layers must be trained, and Time-LLM opts for a highly structured prompting format which involves domain knowledge, task instructions and input statistics.
Since the training data for the linear layers consists of output representations based on these highly structured prompts, it is not evident that the resulting architecture will generalize to more diverse contextual descriptions such as those found in CiK.
Furthermore, although we have not conducted a formal analysis of the diversity of the ETTh1 dataset, it is not a priori obvious that such a dataset would have a sufficient diversity of patterns to train a time series foundation model.

Interestingly, UniTime’s performance does benefit from context for some tasks
(see Fig. 21).
However, the aggregate RCRPS and rank of UniTime with respect to other models indicate that it still struggles to produce forecasts competitive with even quantitative forecasting methods.

Figure: Figure 21: A comparison of forecasts from UniTime without context (left) and with context (right). On average across 5 instances, UniTime’s RCRPS is 64% better with context than without on the “Bounded Prediction Constraint Based On Prediction Quantiles” task.
Refer to caption: x40.png

#### D.4 Lag-Llama

We use the publicly available implementation of Lag-Llama (Rasul et al., ) located at [https://github.com/time-series-foundation-models/](https://github.com/time-series-foundation-models/), and its associated pre-trained weights.
The model inference was done on a single H100 GPU.

#### D.5 Chronos

We use the publicly available implementation of Chronos (Ansari et al., ) located at [https://github.com/amazon-science/chronos-forecasting](https://github.com/amazon-science/chronos-forecasting). We evaluated (see Sec. C.2) our tasks on all $5$ available models: chronos-tiny, chronos-mini, chronos-small, chronos-base and chronos-large, and reported the results of the best performing model, chronos-large in Tab. 1.
The model inference was done on a single H100 GPU.

#### D.6 Moirai

We use the publicly available implementation of Moirai (Woo et al., ) located at [https://github.com/SalesforceAIResearch/uni2ts](https://github.com/SalesforceAIResearch/uni2ts). We evaluated (see Sec. C.2) our tasks on the $3$ following models: moirai-1.0-R-small (located at [https://huggingface.co/Salesforce/moirai-1.0-R-small](https://huggingface.co/Salesforce/moirai-1.0-R-small)), moirai-1.0-R-base (located at [https://huggingface.co/Salesforce/moirai-1.0-R-base](https://huggingface.co/Salesforce/moirai-1.0-R-base)) and moirai-1.0-R-large (located at [https://huggingface.co/Salesforce/moirai-1.0-R-large](https://huggingface.co/Salesforce/moirai-1.0-R-large)) and reported the results of the best performing model, moirai-1.0-R-large in Tab. 1.
The model inference was done on a single H100 GPU.

#### D.7 TimeGEN

We access TimeGEN-1, an optimization of the TimeGPT model (Garza et al., ), using the API made available through the nixtla Python package.
Unlike all other baselines, we only generate point forecasts with TimeGEN due to its probabilistic mode requiring much longer historical data than is available in instances evaluated in the benchmark.
This is the reason the RCPRS values for TimeGEN have zero standard error.

#### D.8 Exponential Smoothing

We used the Exponential Smoothing implementation from the statsmodels Python package, namely the statsmodels.tsa.holtwinters.ExponentialSmoothing class.
Both trend and seasonal components of the models are set to be additive.
The seasonal period length is either set manually for tasks where the simple guess using the time series frequency is incorrect.
If there is not at least two full seasonal periods in the history window of the time series, we disable the seasonal component of the model.
Since some of the benchmark tasks can have as few as 3 time steps in the history window, we also disable the trend component if we have less than 5 time steps in said window.

#### D.9 ETS and ARIMA

We used the implementations of ETS and ARIMA from the forecast R package, using rpy2 for compatibility with Python.
For ETS, we use the ets method, which we call with automatic error, trend, and seasonality components.
In the rare cases where the ETS forecast contains NaN values, we manually switch off the trend component and rerun the forecast.
The ARIMA results are computed using the auto.arima method.
If the ARIMA fits fail, we rerun it with restricted parameter and disabled seasonality.

### Appendix E Details of the proposed metric

The CiK benchmark is designed to determine whether models can improve their probabilistic forecasts by leveraging associated textual information (see Sec. 2).
To support this goal, the evaluation metric:

- 1.
should be a proper scoring rule, such that a model who perfectly knows what the correct forecast is should have no reason to favor another prediction;
- 2.
must be easy to compute using a finite sample from the forecast distribution, since many models do not provide a functional form for their forecasts.

To account for the importance of leveraging relevant context, the metric should also:

- 1.
penalize obviously impossible forecasts, i.e. that can be inferred as implausible from the contextual information;
- 2.
take a similar range of values across different tasks, to prevent some tasks to dominate the score as we average the results across tasks;
- 3.
prioritize forecast quality for timesteps with relevant context, even if these timesteps are a small portion of the forecast horizon.

To satisfy the first two properties, we start with the Continuous Ranked Probability Score (CRPS) (Gneiting & Raftery, ), a reliable strictly proper scoring rule for univariate probability distribution, and take its mean over all time steps.
To compute the CRPS from a finite number of samples, we use the estimator based on its probability weighted moment form (Taillardat et al., ), since it is unbiased (Zamo & Naveau, ).
See Sec. E.3 for more details about this estimator.

Many of our tasks are built to include information about a hard constraint on $\mathbf{X}_{F}$ in their $\mathcal{C}$, which can be written as $v_{\mathcal{C}}(\mathbf{x}_{F})=0$.
If we were only interested to measure by how much a forecast breaks the constraint, we could take inspiration from the threshold-weighted CRPS (Gneiting & Ranjan, ) by using $v_{\mathcal{C}}$ as its chaining function (Allen et al., ):

$$ $\text{twCRPS}_{v_{\mathcal{C}}}(\widetilde{\mathbf{X}}_{F},\mathbf{x}_{F}) \equiv\text{CRPS}\left(v_{\mathcal{C}}(\widetilde{\mathbf{X}}_{F}),v_{\mathcal {C}}(\mathbf{x}_{F})\right),$ (1) $$

where $\widetilde{\mathbf{X}}_{F}$ is the forecast of $\mathbf{X}_{F}$ to be evaluated.
Since, by construction, the ground-truth $\mathbf{x}_{F}$ always satisfy the constraints, we have $v_{\mathcal{C}}(\mathbf{x}_{F})=0$.
But since we do not care only about whether forecasts break constraints, we sum both the original CRPS and this twCRPS, but we weight the later by a factor of $\beta=10$, to denote the additional interest we show to these errors.
See Sec. E.4 for the various $v_{\mathcal{C}}$ used in the benchmark.

One common approach to normalize the CRPS to get similar ranges for multiple problems is to divide it by the mean absolute value of the target ground-truth of the forecasted series (Alexandrov et al., ).
This has two issues: the metric is no longer proper, and it leads to much larger values for series close to zero than those far from it.
To solve the first issue, we take advantage that we can generate many more instances from each of our tasks, by computing a normalization factor $\alpha$ from 25 instances not included in the benchmark.
The details of this calculations are in Sec. E.1.

Many tasks in our benchmark contains contextual information which is highly relevant for a small fraction of the time steps in the forecasting window, while being only marginally relevant for the majority of the time steps.
If we were to weight these two categories equally, then the score for a model which ignores the context would be hard to distinguish from the score of one who does not.
We correct this issue by identifying the subset of time steps with relevant information, which we call the Region of Interest (RoI).
We then weight the CRPS to give half weight to the RoI time steps and half weight to the non-RoI time steps.
Therefore, we obtain our metric, which we call the Region-of-Interest CRPS (RCRPS):

$$ $\text{RCRPS}(\widetilde{\mathbf{X}}_{F},\mathbf{x}_{F})\mathrel{\mathop{:}}= \begin{cases}\mathcal{\alpha}\,\cdot\left[\frac{1}{2|\mathcal{I}|}\cdot\sum \limits_{i\in\mathcal{I}}\text{CRPS}\!\!\left(\widetilde{X}_{i},x_{i}\right)+ \frac{1}{2|\neg\mathcal{I}|}\cdot\sum\limits_{i\in\neg\mathcal{I}}\text{CRPS} \!\!\left(\widetilde{X}_{i},x_{i}\right)+\beta\cdot\,\text{CRPS}\!\!\left(v_{ \mathbf{C}}(\widetilde{\mathbf{X}}_{F}),0\right)\right]&\text{if}\;|\mathcal{I }|>0\\ \mathcal{\alpha}\,\cdot\left[\frac{1}{|\neg\mathcal{I}|}\cdot\sum\limits_{i\in \neg\mathcal{I}}\text{CRPS}\!\!\left(\widetilde{X}_{i},x_{i}\right)+\beta\cdot \,\text{CRPS}\!\!\left(v_{\mathbf{C}}(\widetilde{\mathbf{X}}_{F}),0\right) \right],&\text{if}\;|\mathcal{I}|=0\end{cases}$ $$

where $\mathcal{I}$ is the set of time steps in the RoI, $\neg\mathcal{I}$ is the set of time steps in the forecast but not in the RoI, $\alpha$ is the aforementioned scaling factor,
and we drop the factor of two and the first sum for tasks where there is no meaningful RoI.

#### E.1 Scaling for cross-task aggregation

The rationale behind scaling the RCPRS is to allow us to average its value from diverse tasks without the average being dominated by the forecast quality for tasks with time series with large values.
An alternative argument is: all other conditions being equal, a forecaster that is wrong by 10 in its forecast for a time series which goes from 25 to 30 is worse than one that is wrong by 100 in its forecast for a time series which goes from 2500 to 3000.
Furthermore, we have multiple tasks for which some instances have constant $\mathbf{x}_{F}$ or nearly so, often with values close to zero.
Due to these tasks, we cannot simply use a scaling which only depends on said instances $\mathbf{x}_{F}$.
Instead, we take advantage of our benchmark ability to generate a very large number of instances for each tasks by using $M=25$ instances not included in our benchmark.
Given the ground-truth future values $\mathbf{x}_{F}^{m}$ for these instance, the scaling factor $\beta$ for an individual task is as follow:

$$ $\alpha=\left[\frac{\sum_{m}\left(\max_{i}x_{i}^{m}-\min_{i}x_{i}^{m}\right)}{M }\right]^{-1}.$ (2) $$

###### Properness

In an ideal scenario, all instances of a tasks would be fully independent.
In that case then Eq. 2 would not contain any information about the target time series in the benchmark instances, making the RCPRS a proper scoring rule.
However, due to possible overlaps in the time windows used when creating the instances and to auto-correlations, we cannot guarantee independence between instances, and thus we cannot guarantee that the RCPRS is actually a proper scoring rule.
Note that this deviation from a proper scoring rule is minor, and has a much smaller effect than the one due to the common approach of normalizing the CRPS using the Mean Absolute Value of the ground-truth.

#### E.2 CRPS and twCRPS

Given a univariate forecast $\widetilde{X}$ and a ground-truth realization $x$, the Continuous Ranked Probability Score (CRPS) can be defined in its integral as follow:

$$ $\text{CRPS}(\widetilde{X},x)=\int_{-\infty}^{\infty}dy\left[\Phi_{\widetilde{X }}(y)-\mathds{1}(y\geq x)\right]^{2},$ (3) $$

where $\Phi_{\widetilde{X}}(y)$ is the Cumulative Distribution Function of $\widetilde{X}$, and $\mathds{1}$ is the indicator function.

There are multiple ways to compute the CRPS, but a particularly interesting one which showcases its link to the Mean Absolute Error is the energy form of the CRPS:

$$ $\text{CRPS}(\widetilde{X},x)=\mathbb{E}_{X\sim\widetilde{X}}\left|X-x\right|- \frac{1}{2}\mathbb{E}_{X,X^{\prime}\sim\widetilde{X}}\left|X-X^{\prime}\right|.$ (4) $$

We get the threshold-weighted CRPS (twCRPS) from Eq. 4 by adding a weighting function $w(x)$ to it:

$$ $\text{twCRPS}(\widetilde{X},x)=\int_{-\infty}^{\infty}dyw(y)\left[\Phi_{ \widetilde{X}}(y)-\mathds{1}(y\geq x)\right]^{2}.$ (5) $$

To get the energy form of the twCRPS, we must compute the chaining function $v(x)$ from $w(x)$:

$$ $v(x)-v(x^{\prime})=\int_{[x,x^{\prime})}dyw(y).$ (6) $$

Using $v(x)$, we can write the twCRPS as:

$$ $\text{twCRPS}(\widetilde{X},x)=\mathbb{E}_{X\sim\widetilde{X}}\left|v(X)-v(x) \right|-\frac{1}{2}\mathbb{E}_{X,X^{\prime}\sim\widetilde{X}}\left|v(X)-v(X^{ \prime})\right|.$ (7) $$

Eq. 7 can readily be generalized to a multivariate forecast, by using any $\mathbb{R}^{d}\rightarrow\mathbb{R}$ chaining function.

#### E.3 Estimating the CRPS using samples

Computing the CRPS using Eq. 3 or Eq. 4 directly would be extremely hard for most of the baselines included in our experiments.
Instead, it is more computationally convenient to use an estimator of the CRPS which uses a finite number of samples $x_{1}$, …, $x_{M}$ from the forecasting distribution.
An unbiased estimator of the CRPS created from Eq. 4 is:

$$ $\text{CRPS}(\widetilde{X},x)\approx\frac{1}{M}\sum_{n=1}^{M}\left|x_{n}-x \right|-\frac{1}{2M(M-1)}\sum_{n=1}^{M}\sum_{n^{\prime}=1}^{M}\left|x_{n}-x_{n ^{\prime}}\right|.$ (8) $$

However, this estimator is relatively costly, having a $O(M^{2})$ time complexity.

A faster estimator which gives the same result as Eq. 8 (up to numerical accuracy) is the one based on the probability weighted moment form of the CRPS (Taillardat et al., ; Zamo & Naveau, ):

$$ $\text{CRPS}(\widetilde{X},x)\approx\frac{1}{M}\sum_{n=1}^{M}\left|x_{n}-x \right|+\frac{1}{M}\sum_{n=1}^{M}x_{n}-\frac{2}{M(M-1)}\sum_{n=1}^{M}(n-1)x_{n},$ (9) $$

where the $x_{n}$ have been sorted in ascending order.
We used Eq. 9 in our metric, since it is as accurate as Eq. 8, while only having a $O(M\log M)$ time complexity.

#### E.4 Constraint-violation functions

In selecting constraint-violation functions $v_{\mathcal{C}}$ for our various tasks, we have the following requirements: it should be invariant to the number of timesteps in the forecasting window and it should be multiplied by $\alpha$ if all numerical data in a task is transformed using $x\rightarrow\alpha x+\beta$.
Here are the $v_{\mathcal{C}}$ we use in some of our benchmark tasks:

- •
*Constant upper-bound constraint* $x_{i}\leq\tau^{+}$:
$v_{\mathcal{C}}(\mathbf{x}_{F})=\frac{1}{T-t}\sum_{i=t+1}^{T}\max(0,x_{i}-\tau
^{+}),$
- •
*Constant lower-bound constraint* $x_{i}\geq\tau^{-}$:
$v_{\mathcal{C}}(\mathbf{x}_{F})=\frac{1}{T-t}\sum_{i=t+1}^{T}\max(0,\tau^{-}-x
_{i}),$
- •
*Constant lower-bound and upper-bound constraints* $\tau^{-}\leq x_{i}\leq\tau^{+}$:
$v_{\mathcal{C}}(\mathbf{x}_{F})=\frac{1}{T-t}\sum_{i=t+1}^{T}\max(0,\tau^{-}-x
_{i})+\max(0,x_{i}-\tau^{+}),$
- •
and *Variable upper-bound constraints, on a subset of time steps* $x_{i}\leq\tau^{+}_{i}\ \forall i\in C$:
$v_{\mathcal{C}}(\mathbf{x}_{F})=\frac{1}{|C|}\sum_{i\in C}\max(0,x_{i}-\tau^{+
}_{i}).$

#### E.5 Covariance of two CRPS estimators

One approach to compute standard error on the RCRPS is to compute the empirical standard deviation based on the 5 instances we use for each task.
However, such a method would overestimate the standard error, since it would consider both the variance coming from the selection of instances of a given task, and the variance coming from the models sampling processes.
Since all models are tested using the exact same instances, the variance coming from their selection is not relevant, and thus we need a way to ignore it.

To do so, we take advantage that the RCRPS is a weighted sum of multiple CRPS estimates.
Since those estimates are not independent from one another, we can compute an estimate of the variance of the RCPRS under the sampling process by computing an estimate of the covariance matrix between the various CRPS estimates, followed by the appropriate weighted sum.

Let says we want to compute the covariance between the CRPS for variable $i$ and the CRPS for variable $j$, using $M$ independent and identically distributed samples from the joint distribution of $\widetilde{X}_{i}$ and $\widetilde{X}_{j}$.

$$ $\mathrm{Cov}\left(\text{CRPS}\!\!\left(\widetilde{X}_{i},x_{i}\right),\text{ CRPS}\!\!\left(\widetilde{X}_{j},x_{j}\right)\right)=\\ \mathrm{Cov}\Biggl{(}\frac{1}{M}\sum_{n}|\widetilde{X}_{i,n}-x_{i}|-\frac{1}{2 M(M-1)}\sum_{n\neq n^{\prime}}|\widetilde{X}_{i,n}-\widetilde{X}_{i,n^{\prime} }|,\\ \frac{1}{M}\sum_{n}|\widetilde{X}_{j,n}-x_{j}|-\frac{1}{2M(M-1)}\sum_{n\neq n^ {\prime}}|\widetilde{X}_{j,n}-\widetilde{X}_{j,n^{\prime}}|\Biggr{)},$ $$

where the sums are over the various samples $n$ and $x_{i}$ and $x_{j}$ and the ground-truth values.

After some tedious algebraic manipulations, we obtain the final formula for the covariance of two CRPS estimates:

$$ $\mathrm{Cov}\left(\text{CRPS}\!\!\left(\widetilde{X}_{i},x_{i}\right),\text{ CRPS}\!\!\left(\widetilde{X}_{j},x_{j}\right)\right)=\\ \begin{aligned} -\frac{1}{M}&\operatorname*{\mathbf{E}}_{\widetilde{X}_{i}}| \widetilde{X}_{i}-x_{i}|\operatorname*{\mathbf{E}}_{\widetilde{X}^{\prime}_{j} }|\widetilde{X}^{\prime}_{j}-x_{j}|\\ +\frac{1}{M}&\operatorname*{\mathbf{E}}_{\widetilde{X}_{i}}|\widetilde{X}_{i}- x_{i}|\operatorname*{\mathbf{E}}_{\widetilde{X}^{\prime}_{j}}\operatorname*{ \mathbf{E}}_{\widetilde{X}^{\prime\prime}_{j}}|\widetilde{X}^{\prime}_{j}- \widetilde{X}^{\prime\prime}_{j}|\\ +\frac{1}{M}&\operatorname*{\mathbf{E}}_{\widetilde{X}_{i}}\operatorname*{ \mathbf{E}}_{\widetilde{X}^{\prime}_{i}}|\widetilde{X}_{i}-\widetilde{X}^{ \prime}_{i}|\operatorname*{\mathbf{E}}_{\widetilde{X}^{\prime\prime}_{j}}| \widetilde{X}^{\prime\prime}_{j}-x_{j}|\\ -\frac{2M-3}{2M(M-1)}&\operatorname*{\mathbf{E}}_{\widetilde{X}_{i}} \operatorname*{\mathbf{E}}_{\widetilde{X}^{\prime}_{i}}|\widetilde{X}_{i}- \widetilde{X}^{\prime}_{i}|\operatorname*{\mathbf{E}}_{\widetilde{X}^{\prime \prime}_{j}}\operatorname*{\mathbf{E}}_{\widetilde{X}^{\prime\prime\prime}_{j} }|\widetilde{X}^{\prime\prime}_{j}-\widetilde{X}^{\prime\prime\prime}_{j}|\\ +\frac{1}{M}&\operatorname*{\mathbf{E}}_{(\widetilde{X}_{i},\widetilde{X}_{j}) }|\widetilde{X}_{i}-x_{i}|\cdot|\widetilde{X}_{j}-x_{j}|\\ -\frac{1}{M}&\operatorname*{\mathbf{E}}_{(\widetilde{X}_{i},\widetilde{X}_{j}) }\operatorname*{\mathbf{E}}_{\widetilde{X}^{\prime}_{i}}|\widetilde{X}_{i}- \widetilde{X}^{\prime}_{i}|\cdot|\widetilde{X}_{j}-x_{j}|\\ -\frac{1}{M}&\operatorname*{\mathbf{E}}_{(\widetilde{X}_{i},\widetilde{X}_{j}) }\operatorname*{\mathbf{E}}_{\widetilde{X}^{\prime}_{j}}|\widetilde{X}_{i}-x_{ i}|\cdot|\widetilde{X}_{j}-\widetilde{X}^{\prime}_{j}|\\ +\frac{1}{2M(M-1)}&\operatorname*{\mathbf{E}}_{(\widetilde{X}_{i},\widetilde{X }_{j})}\operatorname*{\mathbf{E}}_{(\widetilde{X}^{\prime}_{i},\widetilde{X}^{ \prime}_{j})}|\widetilde{X}_{i}-\widetilde{X}^{\prime}_{i}|\cdot|\widetilde{X} _{j}-\widetilde{X}^{\prime}_{j}|\\ +\frac{M-1}{M(M-1)}&\operatorname*{\mathbf{E}}_{(\widetilde{X}_{i},\widetilde{ X}_{j})}\operatorname*{\mathbf{E}}_{\widetilde{X}^{\prime}_{i}}\operatorname*{ \mathbf{E}}_{\widetilde{X}^{\prime\prime}_{j}}|\widetilde{X}_{i}-\widetilde{X} ^{\prime}_{i}|\cdot|\widetilde{X}_{j}-\widetilde{X}^{\prime\prime}_{j}|,\\ \end{aligned}$ $$

where variables with the same number of apostrophes (^′) are drawn together and those with different number of apostrophes are independent variables.

To get an estimate of covariance using our $M$ samples, we can estimate each of these terms using their respective unbiased estimators.
Once we have compute an estimate of the variance for a single task instance, the overall variance for a full task is computed using the formula for the variance of the average of multiple independent variables.
One slight disadvantage of using this method, is that it offers now guarantee that the RCPRS variance estimate will be non-negative, so in the rare cases where the estimate for the variance of a full task is negative, we clip it to 0.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. How the textual contexts were annotated, including any guidelines, annotator expertise, and inter-annotator agreement (IAA) metrics?

2. Do the authors envision methods to automate or partially automate this process, such as using existing NLP techniques to generate context?

3. Could the authors provide explicit definitions for each capability and clarify the criteria used to categorize tasks?

4. Could the authors clarify whether this discrepancy suggests limitations of the CRPS metric or other factors in the benchmark design?
soundness: 1
strengths: 1. Good writing and easy to follow
2. CiK uniquely requires essential textual context for forecasting, marking a new direction in multimodal prediction.
3. The benchmark is robust, with real-world tasks and a novel, context-focused RCRPS metric.
summary: This paper introduces a new benchmark, CiK, to evaluate how well forecasting models can use essential textual context alongside numerical data to improve prediction accuracy. The benchmark includes 71 tasks across various fields, where models need to integrate natural language information—like historical trends or future events—with time series data for accurate forecasts. To assess performance, the authors develop the Region of Interest CRPS (RCRPS) metric, which emphasizes context-sensitive parts of the forecast and accounts for constraints stated in the text. Through experiments, they show that a simple prompting method for large language models (LLMs) outperforms traditional forecasting methods, underscoring the importance of context for improved predictions.
weaknesses: 1. Missing Information on Context Annotations: 
The paper relies on carefully crafted textual contexts but omits crucial details about the annotation process, such as the guidelines provided to annotators, the number and qualifications of annotators, methods used to resolve disagreements, and quantitative measures of inter-annotator agreement (IAA). This lack of information raises questions about the consistency and reliability of the annotations. Including examples or a sample of the annotation guidelines, a description of annotator expertise, and the process for calculating IAA would strengthen the benchmark’s credibility and demonstrate rigorous annotation practices.

2. Limited Benchmark Novelty:  
While the CiK benchmark combines existing time-series datasets with manually created textual contexts, its contribution to multimodal benchmarks is limited in novelty. The approach resembles prior work that integrates time-series data with textual sources like news or social media.[1][2] To clarify its uniqueness, the authors could provide comparisons to specific existing work and clearly articulate the novel contributions or improvements over these prior works. Additionally, the manual creation of contexts raises concerns about scalability; introducing semi-automated methods or leveraging AI to generate contexts could make the benchmark more practical for real-world applications and future expansions.

3. Ambiguous Task Type Annotations: 
The paper lacks clarity in task categorization, with no explicit definitions provided for each model capability category. For instance, “instruction following” is inconsistently applied, leaving tasks like “Public Safety” uncategorized, despite requiring instruction interpretation. It would be helpful if the authors included definitions for each capability category, specified criteria for categorizing tasks, and offered examples illustrating why certain tasks fall into each category. This additional information would clarify the task taxonomy and improve the interpretability of the benchmark structure.

4. Unexplained Results Discrepancies:  
Certain performance discrepancies raise concerns about the validity of the benchmark’s metrics. For example, LLMP Mixtral-8x7B shows lower CRPS performance with context compared to without in Figures 4 and 5, yet it still appears to outperform traditional quantitative models when using context. This inconsistency suggests that CRPS may not fully capture the forecast quality in multimodal contexts. The authors could benefit from including a discussion on why CRPS was chosen, exploring alternative or complementary metrics, or providing a deeper analysis of the observed discrepancies to enhance the reliability of the reported results.

5. Limited Model Variety:  
The benchmark’s experimental setup primarily includes larger models like Llama-3 series, limiting the variety across model sizes and architectures, as well as smaller models like Mistral, Qwen, and Falcon. A more diverse set of models, including smaller or less resource-intensive models, could offer broader insights and improve the benchmark’s generalizability. Explaining any practical or strategic reasons for the current model selection would provide additional context. Exploration of smaller models or discussing plans for future testing would also enhance the paper’s impact.

[1] Sawhney, Ramit, Arnav Wadhwa and Shivam Agarwal. “FAST: Financial News and Tweet Based Time Aware Network for Stock Trading.” Conference of the European Chapter of the Association for Computational Linguistics (2021).

[2] Liu, Mengpu, Mengying Zhu, Xiuyuan Wang, Guofang Ma, Jianwei Yin and Xiaolin Zheng. “ECHO-GL: Earnings Calls-Driven Heterogeneous Graph Learning for Stock Movement Prediction.” AAAI Conference on Artificial Intelligence (2024).

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Are all the time series is univariate? 
2. In this benchmark construction, have you tried very refined text rather than information at a specific time step or overall? How did it perform?
3. Regarding retrieval, if I use the time series segment corresponding to a specific text as the retrieval "text", will the performance be better? Because this is more direct.
soundness: 3
strengths: The paper provides rigorous evaluation, testing various model types and prompting techniques. The introduction of the RCRPS metric enhances assessment accuracy by factoring in context relevance and constraint adherence. The combination of text and time series has always been something that researchers in the field want to try, and this benchmark provides a good research foundation. The writing structure of the paper is very clear
summary: This paper explores the integration of contextual textual data with numerical time series to improve time series forecasting. It introduces the CiK benchmark, consisting of 71 diverse forecasting tasks across multiple domains. Unlike existing benchmarks, CiK requires models to process both numerical data and associated textual context, reflecting real-world complexities such as seasonal trends or future constraints (e.g., maintenance periods). The authors also propose a novel metric, the Region of Interest Continuous Ranked Probability Score (RCRPS), which weights context-sensitive time windows and penalizes constraint violations.
weaknesses: From the experimental results, we can see that text provides a good auxiliary role, but this method should be limited to models with LLM as the backbone.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: Questions:
1) Does the covariate information always imply the availability of the future values of the covariates, or are there examples with covariate information provided only for the history time series?
2) While I believe it would be handled in a discussion of the manual curation process, I wanted to know if the entire manual curation for so many datasets was done by the authors who would be credited for the paper or if any form of crowdsourcing was utilized for the manual tasks. Were the annotators paid fairly for their efforts if any crowdsourcing was utilized? Was any LLM used during the manual curation process?

Suggestions for the authors:
1) Include a discussion on the manual curation process with information on the data sources and the selection of relevant context from them.
2) Include benchmark descriptions mentioning the sequence lengths, prediction horizon, and number of sequences present in each benchmark to build confidence in the robustness of the work.
3) Provide some ideas for separating different contextual information in the text.
4) Highlight the efforts taken to ensure that any contextual information paired with the time series is actually correct/relevant. Do a similar task to highlight the relevance of the said "region of interest."
soundness: 3
strengths: This work is original and highly relevant to the current trend of LLM-based forecasting. The benchmark is well-designed, offering broad coverage across various domains and tasks, with results for multiple forecasters included to showcase its capabilities. The analysis and results are clearly presented, with examples and figures that effectively illustrate key benchmark characteristics and greatly enhance comprehension. Additionally, the paper introduces an intriguing new metric for evaluating forecast quality within relevant regions, which adds depth to the evaluation framework.
summary: The paper addresses a pertinent issue: the scarcity of benchmarks for context-enhanced forecasting. Although many recent studies focus on predicting future values using textual cues, there is limited data available for training and evaluating such models. This paper introduces a manually curated benchmark specifically for text-aided time-series forecasting, featuring numerical time series data paired with contextual text. The benchmark is extensive in its selection of domains and tasks, providing a comprehensive resource. It is also thoroughly evaluated across a wide range of forecasting models. Furthermore, the paper introduces a novel metric for forecast evaluation.
weaknesses: While this work is both novel and relevant, it lacks analytical rigor in its benchmark evaluation. The reader is supposed to assume that all provided textual information contributes meaningfully to the forecasts, with minimal evidence beyond the overall performance improvements seen for the entire dataset. Evaluating the relevance of the textual context—perhaps through methods such as LLM-based or human assessments—would strengthen the claim that these textual data are correct and relevant descriptions for the time series. Additionally, some covariate information appears to include future events, which a causal forecaster would not typically access (e.g., “Speed From Load” in Appendix B.2). This raises concerns about causal consistency, as there is no mechanism for systematically separating different types of contextual data other than through manual or LLM editing. Such limitations could present challenges for users who want to avoid incorporating future or irrelevant covariate information in their experiments.

The paper also lacks clarity regarding the historical context length and forecasting horizon—key details that should be specified. Furthermore, the reliability of the benchmark results hinges on the sample size, yet no information about the number of samples for the datasets is provided.

Perhaps the most notable contribution of the paper is its manual curation of the dataset. However, this process remains underexplained. Details such as the curation methodology, sources of textual data, and the criteria used for selecting relevant data are absent, which limits the transparency of this work.  A more comprehensive discussion of these aspects would significantly enhance the credibility and utility of the dataset for future research.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. can you just open-source all the related-materials and I think it's better to let all people to judge if it has enough value and easy to use.
2. how do you choose the best-quality text and how would you access this quality?
3. Is there any better way to cooperate the numbers in prompt to evaluate the effect of the benchmark?
5. text brings more calculation, but the effect of the texts is not well discussed. How to balance and choose the most effective one?
soundness: 3
strengths: I would say this is a 'huge' work, congratulations!
The following are some points I agree:
1. the benchmark is relatively complete and has a potential to have impact, which include textual context (time-invariants, history, covariates, future and causals).This work may lead to complex or graph modelling with these information.
2. the proposed strategy to evaluate is simple but useful.
3. the paper contains some use case and discussion of existed models
summary: The paper introduces "Context is Key" (CiK), a benchmark designed to evaluate the ability of forecasting models to integrate numerical data with essential textual context (time-invariants, history, covariates, future and causals...). Recognizing that accurate forecasting often requires more than just numerical inputs, the authors aim to bridge this gap by creating a collection of tasks that necessitate the use of both modalities. Some key contributions:
1. a relatively complete benchmark named cik
2. analysis of different models
3. propose direct prompt, which is a simple strategy to prompt LLM to do time-serise prediction
weaknesses: 1. lack of discussion of noise in texts. The texts are complete but the quality filter is not well designed. In general, we need more well-designed texts which is really useful
2. what is the importance of texts? are they hidden in time-series itself? For me, time-series is the sampling result of a complex system, and even you have already done a lot and try to give more complete one, but the system is hard to define. As a result, the time-series itself may contain more information than the texts.
3. besides model evaluation, the benchmark is hard to use as in real world, people may prefer to use simple model and may lack of texts.
4. The cooperation between texts and numbers is not well-designed. Generally, LLMs including GPT are not good at dealing with numbers, and they are not sensitive to the symbols in numbers.

Meta-review: This work proposes Context is Key (CiK) benchmark to evaluate how well forecasting models can integrate textual context with numerical time series data, spanning 71 tasks across various domains. It is indeed a comprehensive coverage across multiple domains, and it further introduced a novel Region of Interest CRPS metric for context-sensitive evaluation. The benchmark provides a valuable research foundation for combining text and time series analysis, and the experimental evaluation includes testing various model types and prompting techniques.

However, several critical weaknesses lead to rejection. As pointed out by most reviewers, this workr lacks crucial information about the context annotation process, including guidelines, annotator expertise, and inter-annotator agreement metrics (Reviewer 7GBq, kJ9d). There are concerns about the benchmark's novelty, as it resembles prior work integrating time-series data with textual sources (Reviewer 7GBq). The analytical rigor is insufficient, with minimal evidence supporting the relevance of provided textual information to forecasts (Reviewer kJ9d). Important details about historical context length, forecasting horizon, and sample sizes are missing (Reviewer kJ9d). The work lacks proper discussion of text quality filtering and the relative importance of textual versus time-series information (Reviewer RToE). Additionally, there are unexplained performance discrepancies in the results (Reviewer 7GBq), and the benchmark's real-world applicability is limited due to its reliance on LLM-backbone models and the challenge of obtaining high-quality textual data (Reviewers 6GZP, RToE).

Decision: Reject