Context Matters: Leveraging Contextual Features for Time Series Forecasting

## Abstract

Abstract Time series forecasts are often influenced by exogenous contextual features in addition to their corresponding history. For example, in financial settings, it is hard to accurately predict a stock price without considering public sentiments and policy decisions in the form of news articles, tweets, etc. Though this is common knowledge, the current state-of-the-art (SOTA) forecasting models fail to incorporate such contextual information, owing to its heterogeneity and multimodal nature. To address this, we introduce ContextFormer, a novel plug-and-play method to surgically integrate multimodal contextual information into existing pre-trained forecasting models. ContextFormer effectively distills forecast-specific information from rich multimodal contexts, including categorical, continuous, time-varying, and even textual information, to significantly enhance the performance of existing base forecasters. ContextFormer outperforms SOTA forecasting models by up to 30% on a range of real-world datasets spanning energy, traffic, environmental, and financial domains.

### 1 Introduction

Figure: Figure 1: Forecasting with context. A context-aware forecaster like ContextFormer can incorporate multimodal contextual information, such as daily news articles, online search trends, and market data, to enhance the accuracy of time-series forecasts.
Refer to caption: x1.png

Numerous state-of-the-art (SOTA) solutions to time series forecasting (Lin et al., ) have predominantly depended only on the time series history. However, in many real-world forecasting applications, such as predicting stock prices, air quality, or household energy consumption, future values are frequently influenced by external contextual factors like geographical and economic indicators. Industrial solutions for forecasting, such as predicting the demand for online food delivery (Chad Akkoyun, ), have shown the potential to improve forecasting accuracy by incorporating macroeconomic factors like tax refunds.
However, the current SOTA forecasting models (Liu et al., ; Nie et al., ) still are unable to handle these contextual factors and solely rely on the historical time series to predict the future. We attribute this to the inherent diversity and multimodality of these contextual factors. For example, consider the task of predicting the price of a stock. The contextual factors can vary from categorical indicators like stock category (e.g., energy, technology, or healthcare), continuous and time-varying indicators like market cap and interest rates, or even textual information in the form of news articles. We refer to this multimodal contextual information as metadata and use both these terms interchangeably. Incorporating metadata into forecasting models is hard for the following reasons:

- 1.
Lack of multimodal metadata encoders. We note that the time series domain lacks the availability of foundation models trained on multimodal datasets to extract aligned representations (e.g., CLIP Radford et al. ()). These are key to mapping the time series history and multimodal metadata into the same representation space from which the forecast can be decoded.
- 2.
Non-uniformity in metadata across datasets. The metadata associated with stock price prediction, such as new articles, tweets and opinions, interest rates, etc., are completely different from the metadata associated with weather prediction, such as rainfall levels, pollution levels, wind direction, and speed, etc. This prevents us from pooling datasets together to train a *context-aware* foundation model for forecasting.
- 3.
Diversity of metadata within datasets. For a given dataset, the metadata could be categorical (e.g., national holidays), continuous (e.g., interest rates), or even time-varying (e.g., oil prices). Current approaches
often end up modeling such diverse metadata through simple linear regressors (Das et al., ), which may be insufficient to capture the complex correlations.

Figure: Figure 2: ContextFormer estimates the true effect of the explainable contextual factors on the forecast. Here, we show an example of a bitcoin price forecast using news articles and the historical price for the past four days. Note that the sharp decline in price is attributed to an ongoing market correction. Existing *context-agnostic* forecasting models treat this as a transient shock, leading to an overcorrection in price recovery. In contrast, our *ContextFormer* comprehends the underlying market dynamics, resulting in a more accurate and reliable forecast.
Refer to caption: x2.png

Consequently, for these exact reasons, we note that the recent wave of foundation models for forecasting relies only on history. To this end, we propose a plug-and-play approach to build *context-aware* forecasting models on top of *context-agnostic* SOTA forecasting models. Our approach includes novel architectural additions to handle categorical, continuous, time-varying, and even textual metadata. Additionally, we introduce novel training modifications to ensure that the *context-aware* forecast is at least as good as the *context-agnostic* forecast with respect to the traditional forecasting metrics. Our architectural and training modifications are inspired by theoretical insights on improving any regression model with new, correlated features. Our primary contributions are as follows:

- 1.
We propose ContextFormer, a novel framework for incorporating diverse multimodal metadata into any context-agnostic forecasting model. ContextFormer surgically inserts aligned representation of metadata using cross-attention blocks (Vaswani et al., ) into the existing forecasting model architectures.
- 2.
We introduce a plug-and-play fine-tuning approach to
effectively incorporate metadata and ensure that the resulting forecasts are at least as good as that of the context-agnostic base model.
- 3.
We show definitive improvements in the forecasting performance of state-of-the-art context-agnostic forecasting models, such as PatchTST (Nie et al., ) and iTransformer (Liu et al., ) across a wide range of real-world forecasting tasks spanning retail, finance, energy, and environmental domains.

### 2 Related Works

Classical methods. These methods predict future values for time series using statistical techniques. Established approaches include ARMA (AutoRegressive Moving Average), which captures temporal dependencies through autoregression and moving averages, and exponential smoothing methods like Holt-Winters and STL (Seasonal-trend decomposition using LOESS), which account for trends and seasonality. The Box-Jenkins methodology is also used for building models to handle non-stationary data (Shumway & Stoffer, ; Hyndman & Athanasopoulos, ). These techniques have long been the foundation of time series forecasting, providing reliable ways to analyze past trends and make accurate predictions. Prophet (Taylor & Letham, ), developed by Facebook, builds upon the traditional techniques by incorporating additional features such as holiday effects and non-linear trends, thus providing greater flexibility and accuracy. Prophet is capable of managing complex seasonal patterns and irregularities and is known for its robustness in handling missing data and outliers. Classical time series forecasters struggle against deep learning models because they lack the adaptability to fully utilize large datasets and learn intricate temporal dependencies.

Deep-learning methods. These methods have been the go-to method to learn the time series features and perform forecasting, utilizing the recent advancements in neural network architectures, such as RNNs (Sherstinsky, ) and transformers (Vaswani et al., ). Notable RNN-based approaches include DeepAR (Salinas et al., ) and LSTNet (Lai et al., ). SOTA transformer-based approaches include iTransformer (Liu et al., ), which applies the attention and feed-forward network on the inverted dimensions and PatchTST (Nie et al., ), a channel-independent transformer which takes time series segmented into subseries-level patches as input tokens. Other prominent approaches include Autoformer (Wu et al., ), Informer (Zhou et al., ), FEDformer (Zhou et al., ) and TimesNet (Wu et al., ). Recent works have focused on foundation models, which are deep models pre-trained on large amounts of data, enabling them to learn extensive information and a variety of patterns. They can be fine-tuned or adapted to specific tasks with relatively small amounts of task-specific data, showcasing remarkable flexibility and efficiency. Popular foundation models include Time-LLM (Jin et al., ), CHRONOS (Ansari et al., ), Lag-Llama (Rasul et al., ), and TimesFM (Das et al., ). Existing deep learning forecasters fail against context-aware models because they lack the ability to incorporate external factors and dynamic contextual information into predictions.

Forecasting with covariates. The current literature in this field predominantly focuses on domain-specific forecasting (Zhou et al., ; Wu et al., ; ) to enhance downstream task performance (Li et al., ), or on the development of foundation models for universal forecasting (Das et al., ; Ansari et al., ), often overlooking correlated time-varying conditions that may be available as categorical or continuous metadata. One of the earliest works in conditional forecasting, a CNN-based approach by Borovykh et al. (), utilizes stacks of dilated convolutions to capture a broad range of historical data for forecasting. More recent models, such as TiDE (Das et al., ) and TimesFM (Das et al., ), incorporate covariates into their forecasting processes. TiDE encodes the past of a time series along with covariates using dense MLPs and then decodes the series with future covariates, again using dense MLPs. TimesFM treats covariates as exogenous regressors and fits linear models outside its decoder at the time of inference (check Appendix B for further details). Another recent development is TimeWeaver (Narasimhan et al., ), a diffusion-based architecture designed for conditional synthesis, which integrates contextual metadata. Our approach builds on these advancements by integrating valuable contextual information from commonly used datasets into the training process of existing SOTA forecasters. We aim to improve forecasting accuracy by utilizing the rich contextual metadata, which is typically neglected in existing methods.

### 3 Problem Formulation

In this section, we formally define the context-aware time series forecasting problem.
We are given a multivariate time series $\bm{X}_{\mathrm{hist}}=\left({\bm{x}}^{1},{\bm{x}}^{2},\dots,{\bm{x}}^{L}\right)$, with ${\bm{x}}^{i}\in\mathbb{R}^{F}$. Here, $L$ denotes the history time series length, while $F$ represents the number of channels. Each sample ${\bm{x}}^{i}$ is associated with contextual metadata ${\bm{c}}^{i}$, comprising both categorical features ${\bm{c}}^{i}_{\mathrm{cat}}\in\mathbb{N}^{K_{\mathrm{cat}}}$ and continuous features ${{\bm{c}}^{i}_{\mathrm{cont}}}\in\mathbb{R}^{K_{\mathrm{cont}}}$. The symbols $K_{\mathrm{cat}}$ and $K_{\mathrm{cont}}$ represent the number of categorical and continuous metadata features, respectively. Together, these features form a vector ${\bm{c}}^{i}={\bm{c}}^{i}_{\mathrm{cat}}\oplus{{\bm{c}}^{i}_{\mathrm{cont}}}$, where $\oplus$ denotes vector concatenation. Note that ${\bm{c}}^{i}$ can include both time-varying and time-invariant metadata features. Now, we can define a metadata sequence as $\bm{C}_{\mathrm{hist}}=\left({\bm{c}}^{1},{\bm{c}}^{2},\dots,{\bm{c}}^{L}\right)$, having the same number of timesteps as $\bm{X}_{\mathrm{hist}}$.

To understand this better, let us take the example of the Beijing AQ dataset, which contains the time series data of six air pollutant concentrations: $CO$, $NO_{2}$, $SO_{2}$, $O_{3}$, $PM2.5$, and $PM10$ concentration $\left(F=6\right)$, sampled on an hourly basis for four days $\left(L=96\right)$. The metadata here includes information on the location, air pressure, amount of rainfall, temperature, dew point, wind speed, and wind direction for each time series sample. In this dataset, location (12 unique labels) and wind direction (17 unique labels) are categorical values $\left(K_{\mathrm{cat}}=2\right)$, while the other features are continuous $\left(K_{\mathrm{cont}}=5\right)$. All metadata except for location is time-varying. Therefore, given such historical time series data and its paired metadata, the task is to predict the future data samples $\bm{X}_{\mathrm{future}}=\left({\bm{x}}^{L+1},{\bm{x}}^{L+2},\dots,{\bm{x}}^{L
+T}\right)$ for a forecasting horizon $T$.

We now go on to define a dataset ${\mathcal{D}}=\left\{\left({\bm{X}}^{n}_{\mathrm{hist}},{\bm{C}}^{n}_{\mathrm{
hist}},{\bm{X}}^{n}_{\mathrm{future}}\right)\right\}_{n=1}^{N},$ which consists of
$N$ independent and identically distributed $\left({\bm{X}}^{n}_{\mathrm{hist}},{\bm{C}}^{n}_{\mathrm{hist}},{\bm{X}}^{n}_{
\mathrm{future}}\right)$ triplets sampled from a joint distribution $P_{\mathrm{data}}$. Formally, the context-aware time series forecasting problem is stated as follows.

Problem. For a dataset ${\mathcal{D}}$ with history length $L$ and forecasting horizon $T$, the goal is to learn the parameters of a model $f(\bm{X}_{\mathrm{hist}},\bm{C}_{\mathrm{hist}};\theta_{\mathrm{forecast}})$ that predicts the forecast $\hat{\bm{X}}_{\mathrm{future}}=\left(\hat{{\bm{x}}}^{L+1},\hat{{\bm{x}}}^{L+2}
,\dots,\hat{{\bm{x}}}^{L+T}\right)$ for the input $\left(\bm{X}_{\mathrm{hist}},\bm{C}_{\mathrm{hist}}\right)$, where $\left(\bm{X}_{\mathrm{hist}},\bm{C}_{\mathrm{hist}},\bm{X}_{\mathrm{future}}
\right)\in{\mathcal{D}}$, such that the following loss function is minimized.

$$ $\displaystyle\mathcal{L}\left(\theta_{\mathrm{forecast}}\right)=\mathbb{E}_{ \bm{X}_{\mathrm{hist}},\bm{C}_{\mathrm{hist}},\bm{X}_{\mathrm{future}}\sim P_{ \mathrm{data}}}{\|\bm{X}_{\mathrm{future}}-\hat{\bm{X}}_{\mathrm{future}}\|}_{ 2},$ (1) $\displaystyle\textrm{where}\ \hat{\bm{X}}_{\mathrm{future}}=f(\bm{X}_{\mathrm{ hist}},\bm{C}_{\mathrm{hist}};\theta_{\mathrm{forecast}}).$ (2) $$

Note that here $\|.\|_{2}$ represents the $\l_{2}$ norm, while $\mathbb{E}$ denotes the expectation over a distribution. Thus, the optimal learned parameters are given by

$$ $\theta_{\mathrm{forecast}}^{\ast}=\operatorname*{arg\,min}_{\theta_{\mathrm{ forecast}}}\mathcal{L}\left(\theta_{\mathrm{forecast}}\right).$ (3) $$

The above forecasting model is said to be context-aware as the forecast $\hat{\bm{X}}_{\mathrm{future}}$ is modeled on both $\bm{X}_{\mathrm{hist}}$ and $\bm{C}_{\mathrm{hist}}$. In contrast, for a context-agnostic model, the forecast will be based only on the history time series $\bm{X}_{\mathrm{hist}}$, with no contribution from the contextual metadata.

### 4 Theoretical Motivation

Figure: (a) Context-aware Forecaster
Refer to caption: x3.png

This section provides theoretical justifications for enhancing forecasting accuracy by incorporating context. From an information-theoretic perspective, we show that including context reduces forecasting uncertainty, thereby improving model accuracy. We then examine the integration of contextual information into a simple linear regression model, illustrating how this approach improves the performance of a simple autoregressive forecaster.

#### 4.1 A Perspective From Information Theory

Taking inspiration from a recent work on retrieval-based forecasting (Jing et al. ()), we illustrate the relationship between the variables $\bm{X}_{\mathrm{hist}}$, $\bm{C}_{\mathrm{hist}}$, $\hat{\bm{X}}_{\mathrm{future}}$, and $\bm{X}_{\mathrm{future}}$ for context-aware and context-agnostic forecasters in the form of the graphical models given in Fig. 3.

On analyzing these models from an information theoretic perspective, for the context-aware forecaster (Fig. 3(a)), we can show the following.

$$ $\mathcal{I}\left(\bm{X}_{\mathrm{future}};\hat{\bm{X}}_{\mathrm{future}}\right )\geq\mathcal{I}\left(\bm{X}_{\mathrm{future}};\bm{X}_{\mathrm{hist}},\bm{C}_{ \mathrm{hist}}\right)\geq\mathcal{I}\left(\bm{X}_{\mathrm{future}};\bm{X}_{ \mathrm{hist}}\right),$ (4) $$

where the quantity $\mathcal{I}\left(A;B\right)$ represents the mutual information between the variables $A$ and $B$. The first inequality in Eq. 4 is a direct result of the very well-known data processing inequality, while the second inequality comes from

$$ $\displaystyle\mathcal{I}\left(\bm{X}_{\mathrm{future}};\bm{X}_{\mathrm{hist}}, \bm{C}_{\mathrm{hist}}\right)=$ $\displaystyle\;\mathcal{I}\left(\bm{X}_{\mathrm{future}};\bm{X}_{\mathrm{hist} }\right)+\mathcal{I}\left(\bm{X}_{\mathrm{future}};\bm{C}_{\mathrm{hist}}|\bm{ X}_{\mathrm{hist}}\right),$ (5) $\displaystyle\mathcal{I}\left(\bm{X}_{\mathrm{future}};\bm{C}_{\mathrm{hist}}| \bm{X}_{\mathrm{hist}}\right)$ $\displaystyle\geq 0\;\forall\;\bm{X}_{\mathrm{future}},\bm{X}_{\mathrm{hist}}, \bm{C}_{\mathrm{hist}}.$ (6) $$

The increase in the mutual information between the input variables and the forecast on the inclusion of context to the input data would imply a lower uncertainty for the context-aware forecast over the context-agnostic one, given their respective inputs. Furthermore, with an additional assumption of Gaussian noise between $\bm{X}_{\mathrm{future}}$ and $\hat{\bm{X}}_{\mathrm{future}}$, Jing et al. () proves the equivalence between MSE and mutual information and states that minimizing the MSE loss is equivalent to maximizing the mutual information. This can be mathematically expressed as

$$ $\min\mathbb{E}_{P_{\mathrm{data}}}\;{\|\bm{X}_{\mathrm{future}}-\hat{\bm{X}}_{ \mathrm{future}}\|}_{2}\iff\text{max }\mathcal{I}\left(\bm{X}_{\mathrm{future} };\hat{\bm{X}}_{\mathrm{future}}\right),$ (7) $$

thus, context-aware models (Fig. 3(a)) are more suitable for forecasting under an MSE loss objective. Further discussion on Eq. 7 has been provided in Appendix A.
Having demonstrated how incorporating contextual information can enhance the forecasting accuracy of general forecasting models, we now turn our attention to integrating context into a simple autoregressive model. We will also explore the guarantees we can provide for this approach.

#### 4.2 Adding Context to an Autoregressive Forecaster

Let ${\bm{y}}^{t}$ be a time-varying quantity influenced by past values and additional contextual information. Here, the underlying assumption is that the true forecast ${\bm{y}}^{t}$ is a linear combination of $p$ lag terms and $q$ context terms. First, we model ${\bm{y}}^{t}$ only as a $p$-order autoregressive (AR) process ${\bm{y}}^{t}={\bm{x}}^{t}{\bm{\beta}}+{\bm{\epsilon}}$. Here, ${\bm{x}}^{t}=\begin{bmatrix}{\bm{y}}^{t-1}&{\bm{y}}^{t-2}&\cdots&{\bm{y}}^{t-p
}\end{bmatrix}$ is a vector of the previous $p$ lagged values, ${\bm{\beta}}$ is a vector of AR coefficients with ${\bm{\beta}}\in\mathbb{R}^{p}$, and ${\bm{\epsilon}}$ is the error term. Note that even though ${\bm{y}}_{t}$ depends on additional contextual information, the assumed linear model only depends on the lag parameters, reflecting the context-agnostic case.

Given $n+p$ observations, we construct an $n\times p$ matrix ${\bm{X}}$ of lagged values, allowing the model to be written as ${\bm{Y}}={\bm{X}}{\bm{\beta}}+{\bm{\epsilon}}$, where ${\bm{Y}}$ is the vector of observed values and ${\bm{Y}}\in\mathbb{R}^{n}$. This formulation enables the least squares estimation of ${\bm{\beta}}$ and the corresponding error in an autoregressive framework. The least squares estimate of ${\bm{\beta}}$ and the associated error are defined as

$$ ${\bm{\beta}}_{\mathrm{opt}}=\left({\bm{X}}^{T}{\bm{X}}\right)^{-1}{\bm{X}}^{T} {\bm{Y}},\quad E_{\mathrm{orig}}=\|{\bm{Y}}-{\bm{X}}{\bm{\beta}}_{\mathrm{opt} }\|^{2}.$ (8) $$

Figure: Figure 4: Adding context improves the forecasting accuracy of an AR model. In this experiment, we vary the number of contextual features from 0 to 5 to demonstrate how the inclusion of these features reduces the MSE for a simple Autoregressive forecaster.
Refer to caption: extracted/5933447/figures/Experiments/context_aware_ar.png

Now, we shift to the context-aware case. Let ${\bm{C}}$ be an $n\times q$ matrix representing the contextual metadata corresponding to ${\bm{X}}$. Our key intuition is that a straightforward way to integrate this contextual information into an AR model without altering its structure is by performing an exogenous regression on the residuals. This approach preserves the original autoregressive framework, allowing the contextual information to account for the variance unexplained by the AR model. In this method, the AR model first captures the time dependencies, and the metadata refines the forecast by reducing the residual error, leading to improved accuracy. The new context-aware autoregressive model can be expressed as ${\bm{Y}}^{\prime}={\bm{C}}{\bm{\gamma}}+{\bm{\epsilon}}^{\prime}$, where ${\bm{Y}}^{\prime}={\bm{Y}}-{\bm{X}}{\bm{\beta}}_{\mathrm{opt}}$ represents the residuals from the AR model, ${\bm{\gamma}}\in\mathbb{R}^{q}$ is the vector of coefficients for the contextual metadata, and ${\bm{\epsilon}}^{\prime}$ is the new error term. For this model, the least squares estimate of ${\bm{\gamma}}$ and the regression error is represented by

$$ ${\bm{\gamma}}_{\mathrm{opt}}=({\bm{C}}^{T}{\bm{C}})^{-1}{\bm{C}}^{T}{\bm{Y}}^{ \prime},\quad\quad E_{\mathrm{new}}=\|{\bm{Y}}^{\prime}-{\bm{C}}{\bm{\gamma}}_ {\mathrm{opt}}\|^{2}.$ (9) $$

We can demonstrate that the error $E_{\mathrm{new}}$ for the context-aware model is less than or equal to the error $E_{\mathrm{orig}}$ of the original model. The error for this model can be expressed as

$$ $E_{\mathrm{new}}=\min_{{\bm{\gamma}}}\|{\bm{Y}}-{\bm{X}}{\bm{\beta}}_{\mathrm{ opt}}-{\bm{C}}{\bm{\gamma}}\|^{2}.$ (10) $$

Since $\mathbf{0}_{q}$, the zero vector in $\mathbb{R}^{q}$, is a feasible solution for ${\bm{\gamma}}$, we have

$$ $E_{\mathrm{new}}=\|{\bm{Y}}-{\bm{X}}{\bm{\beta}}_{\mathrm{opt}}-{\bm{C}}{\bm{ \gamma}}_{\mathrm{opt}}\|^{2}\leq\|{\bm{Y}}-{\bm{X}}{\bm{\beta}}_{\mathrm{opt} }\|^{2}=E_{\mathrm{orig}}.$ (11) $$

The inequality holds because ${\bm{\gamma}}_{\mathrm{opt}}$ minimizes $\|{\bm{Y}}-{\bm{X}}{\bm{\beta}}_{\mathrm{opt}}-{\bm{C}}{\bm{\gamma}}\|^{2}$. This shows that a context-aware autoregressive forecaster is guaranteed to match or exceed the performance of its base model. Additionally, our key intuition here is that a context-aware model with zeroed coefficients for the contextual features (${\bm{\gamma}}=\mathbf{0}_{q}$) will perform identically to its context-agnostic counterpart, ensuring no degradation in performance when the context is not useful.

To empirically support our theoretical justification, we conducted an experiment where samples were generated from variables dependent on their past values and underlying contextual factors. We modeled the sequences using an AR(10) process, varying the number of contextual variables $q$ from 1 to 5. Fig. 4 illustrates how incorporating contextual information improves forecasting accuracy of autoregressive model. Additional details about this experiment can be found in Appendix D.1.

### 5 Methodology

Figure: Figure 5: ContextFormer Architecture. The architecture incorporates the multimodal contextual information in the form of metadata and temporal embeddings through a cross-attention-based method to improve the performance of an existing forecaster. During the fine-tuning phase of ContextFormer, the base model remains frozen, with only the final layer and newly added components being trained on paired contextual information.
Refer to caption: x5.png

In this section, we propose our method, ContextFormer, to incorporate multimodal information into deep-learning forecasting models to enhance forecasting accuracy. Additionally, we propose a plug-and-play fine-tuning approach for this architecture to optimize its performance further.

#### 5.1 Model Structure

The ContextFormer architecture, illustrated in Fig. 5, consists of a metadata embedding module, a temporal embedding module, and multiple cross-attention blocks. These components complement the base model architecture. The working of these components and the required pre-processing steps for time series forecasting are described herein:

Base Model. The base model for the ContextFormer can be any forecasting model capable of processing input time series and generating a hidden state representation. Neural network-based forecasters usually have an input layer, some hidden layers, and a final projection layer. The input layer processes the time series to generate embeddings, which are passed through the hidden layers. The projection layer maps the final hidden state to the dimensionality of the output $\hat{\bm{X}}_{\mathrm{future}}$.

Metadata Embedding. The metadata embedding block embeds the paired metadata for a given time series sample $x^{i}\in\mathbb{R}^{F}$. As outlined in Sec. 3, the paired metadata ${\bm{c}}^{i}$ includes both categorical ${\bm{c}}^{i}_{\mathrm{cat}}$ and continuous ${{\bm{c}}^{i}_{\mathrm{cont}}}$ features, which are combined as ${\bm{c}}^{i}={\bm{c}}^{i}_{\mathrm{cat}}\oplus{{\bm{c}}^{i}_{\mathrm{cont}}}$. Categorical features are represented as one-hot encodings and concatenated with the continuous features. This combined input is passed through a transformer network (Vaswani et al., ), allowing the model to simultaneously learn and capture correlations within the categorical and continuous domains.

Temporal Embedding. Similar to the metadata embedding block, this module generates temporal embeddings from the timestamps of a given sample. Timestamps are first decomposed into components such as year, month, day, hour, and minute, depending on the dataset’s granularity. These components are processed through a transformer network to extract temporal embeddings in a way similar to metadata processing. These embeddings help the model capture long-range correlations and periodic patterns within the time series.

Cross-attention Layers. The cross-attention layers are transformer blocks that use the hidden state representations of the historical time series, along with either the temporal or metadata embeddings, to extract relevant contextual information for forecasting.

#### 5.2 Training

The ContextFormer architecture can either be fully trained from scratch along with the base model using paired contextual metadata or can be used to fine-tune a pre-trained base model. One potential fine-tuning strategy involves a plug-and-play approach, where the context-aware model builds on a pre-trained forecaster that has already been trained on historical time-series data. In this approach, the pre-trained base model (except for the final layer) is frozen, and the ContextFormer components are added with their weights initialized to zero. The zero-initialization approach is motivated by the AR example in Sec. 4.2, where the model with zero-initialized parameters performs identically to the context-agnostic model. Subsequently, the temporal and metadata embedding modules, along with the cross-attention blocks and the final layer, are trained for a specified number of epochs.

Why do we opt for fine-tuning rather than training the context-aware model from scratch?
The advantages of using the plug-and-play fine-tuning setup with a pre-trained forecaster, compared to training a context-aware model from scratch, are as follows:

- 1.
The fine-tuned model is guaranteed to perform at least as well as the context-agnostic base model, provided the test distribution matches the training distribution. However, this guarantee does not apply to a context-aware model trained from scratch.
- 2.
For datasets with irrelevant metadata, training a context-aware model from scratch can be unstable, potentially hindering the model from effectively learning the time series’ trend and seasonality. In contrast, fine-tuning can utilize a pre-trained model, which has already captured time series dependencies, allowing it to focus solely on learning the contextual information.
- 3.
If a time series dataset includes metadata for only some data points, training a context-aware model from scratch would either require ignoring data points without metadata or augmenting new metadata, both of which are undesirable. With our fine-tuning approach, the model can be pre-trained on the entire dataset and then fine-tuned using only the data points that have metadata.
- 4.
The plug-and-play design of our model allows the creation of multiple context-aware models from a single forecaster. This flexibility motivates the development of dataset-agnostic universal forecasting models, which can be fine-tuned to generate dataset-specific models.

### 6 Experiments

**Table 1: Context-aware forecasters outperform Context-agnostic models on the synthetic dataset. An average of 11.6% improvement in MSE over baseline is observed on the ContextFormer fine-tuning of both the transformer models on the synthetic data.**
| Model | Method | MAE | MSE |
| --- | --- | --- | --- |
| PatchTST | Context-agnostic | 0.749 | 1.076 |
| Context-aware (Ours) | 0.702 | 0.968 |  |
| iTransformer | Context-agnostic | 0.764 | 1.118 |
| Context-aware (Ours) | 0.704 | 0.971 |  |

We have extensively evaluated the ContextFormer framework using two state-of-the-art transformer-based forecasters, PatchTST (Nie et al., ) and iTransformer (Liu et al., ), across various forecasting applications and time horizons. These evaluations showcase the impact of incorporating contextual metadata to enhance forecast accuracy. Although transformer-based forecasters were utilized as the base models in our study, the ContextFormer method is highly versatile and not limited to any particular model architecture. Its flexible design allows it to be integrated with any pre-existing forecasting model, irrespective of its internal implementation.

Preliminary Experiment.
Before experimenting with real-world data, we validated our architectural implementation using a synthetic dataset. In the first experiment with the ContextFormer architecture, we generated a dataset from samples of ARMA(2,2) processes with randomly chosen coefficients and added Gaussian noise to perturb the sequences (see Appendix C.1 for more details). Though the time-domain variations were minor, ARMA decomposition of the perturbed sequences revealed a significant divergence between the estimated and original generating coefficients.

**Table 2: ContextFormer enhances forecasting accuracy. We compare the PatchTST and iTransformer models, with and without the ContextFormer additions for time series forecasting on the specified datasets, with a fixed lookback length of $L=96$ and forecast lengths of $T\in\{48,96\}$. The best results for each base model in each row are highlighted in bold. Notably, the context-aware models consistently outperform their context-agnostic counterparts across all rows and metrics. The Chronos zero-shot results are used to compare our models with a massive pre-trained forecaster.**
| Model | PatchTST | iTransformer | Chronos |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Context-agnostic | Context-aware | Context-agnostic | Context-aware | Zero-shot |  |  |  |  |  |  |
| Dataset | T | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE |
|  | 48 | 0.573 | 0.770 | 0.524 | 0.674 | 0.577 | 0.771 | 0.540 | 0.696 | 0.600 | 0.873 |
| Air Quality | 96 | 0.622 | 0.901 | 0.572 | 0.802 | 0.631 | 0.919 | 0.591 | 0.813 | 0.664 | 0.989 |
|  | 48 | 0.038 | 0.058 | 0.029 | 0.036 | 0.042 | 0.067 | 0.028 | 0.035 | 0.069 | 0.104 |
| Electricity | 96 | 0.031 | 0.040 | 0.024 | 0.027 | 0.038 | 0.055 | 0.024 | 0.028 | 0.058 | 0.078 |
|  | 48 | 1.101 | 3.527 | 0.865 | 2.922 | 1.022 | 3.265 | 0.848 | 2.868 | 0.838 | 4.845 |
| Traffic | 96 | 1.084 | 3.415 | 0.845 | 2.767 | 1.025 | 3.165 | 0.830 | 2.766 | 0.842 | 5.142 |
|  | 48 | 0.123 | 0.238 | 0.115 | 0.228 | 0.129 | 0.257 | 0.124 | 0.257 | 0.106 | 0.198 |
| Retail | 96 | 0.139 | 0.291 | 0.128 | 0.265 | 0.145 | 0.309 | 0.143 | 0.310 | 0.122 | 0.245 |
|  | 48 | 0.854 | 1.231 | 0.821 | 1.192 | 0.832 | 1.177 | 0.810 | 1.153 | 0.677 | 0.885 |
| Bitcoin | 96 | 0.971 | 1.561 | 0.948 | 1.537 | 0.992 | 1.650 | 0.951 | 1.547 | 0.898 | 1.501 |

The initial ARMA coefficients, treated as latent variables, represented the true underlying structure of each series but were unrecoverable from the noisy data. For our context-aware forecasting task, we utilized these latent coefficients as metadata. These coefficients were continuous and time-invariant for each series and provided critical information that could potentially enhance
forecasting accuracy.
Since the synthetic data lacked timestamps, this experiment did not employ temporal embeddings. As shown in Table 1, the preliminary results were promising, with the inclusion of contextual information significantly improving forecasting accuracy for both architectures. Encouraged by these findings, we extended our experiments to a wide range of real-world datasets.

Real-world Datasets. We have validated our proposed ContextFormer approach across five forecasting tasks, each from a different domain, using a diverse group of datasets. These include the ECL (Electricity Load) dataset from Trindade (), the PEMS-SF (Traffic) dataset from Wu et al. (), the Beijing AQ (Air Quality) dataset from Chen (), the Store Sales Competition (Retail) dataset from Kaggle (), and the Monash (Bitcoin) dataset from Godahewa et al. (). Some of these datasets, such as Monash and Store Sales, feature complex time-varying metadata, including online search trends and daily oil prices. On the other hand, datasets like ECL and PEMS-SF primarily contain discrete, time-invariant metadata. Additional details about the datasets and their metadata can be found in Appendix C.
xtbfBitcoin

Metrics and Baselines. The context-aware models were evaluated for forecasting on the aforementioned datasets, using a lookback length of $L=96$ and forecasting horizons of $T=48$ and $T=96$. Forecast accuracy was assessed using Mean Squared Error (MSE) and Mean Absolute Error (MAE) as performance metrics, with our approach compared to state-of-the-art transformer-based models, including iTransformer, PatchTST, and Chronos (Ansari et al., ), a large pre-trained forecaster. More details about the models and the inference procedure can be found in Appendix D.

Our experiments show that incorporating contextual information into pre-trained, context-agnostic forecasters substantially improves baseline models’ performance in forecasting time series data across various domains. Specifically, our experiments explore the following key questions:

Figure: Figure 6: Context-aware forecasts show significant qualitative improvements over context-agnostic forecasts. The fine-tuned ContextFormer models produce forecasts that more accurately align with the ground truth distribution, offering better performance compared to the context-agnostic models.
Refer to caption: x6.png

Does incorporating contextual metadata improve forecasting accuracy?
The context-aware models consistently outperform the context-agnostic baselines on both forecasting metrics across all datasets and forecasting horizons, as shown in Table 2, thereby validating our initial hypothesis. The consistent improvement across datasets highlights the ability of our approach to effectively utilize contextual information from diverse multimodal sources. The most significant gain is seen for electric load forecasting, where incorporating the metadata leads to an average improvement of 42.1% in MSE and 28.1% in MAE across models and forecasting horizons.

Is this improvement in forecasting independent of the underlying base architecture?
We used two of the most advanced transformer-based forecasting architectures for our analysis: PatchTST and iTransformer. Despite significant differences in their implementations, the addition of ContextFormer consistently improved the forecasting accuracy for both models across all experiments. This suggests that our technique has the potential to improve the performance of any transformer-based forecaster, regardless of its internal architecture. Incorporating ContextFormer modules improved MSE by an average of 14.8% for PatchTST and 15.2% for iTransformer.

**Table 3: Plug-and-play fine-tuning outperforms full-training. The results for retail forecasting with $T=96$ using the PatchTST base model show that a context-aware model trained from scratch performs worse than the context-agnostic model. In contrast, the ContextFormer model, fine-tuned in a plug-and-play manner, outperforms both.**
| Training Type | Context | MAE | MSE |
| --- | --- | --- | --- |
| Base Model | Agnostic | 0.139 | 0.291 |
| Full Training | Aware | 0.154 | 0.370 |
| Fine-Tuning (Ours) | Aware | 0.128 | 0.265 |

Is the plug-and-play fine-tuning more effective than training an entire model from scratch?
One of our initial hypotheses was that a fine-tuned model should perform at least as well as the base model (at least on the training set), whereas no such assurance could be made for a context-aware model trained from scratch. The first part of our claim is supported by the results in Table 2, while the second part is evidenced by the values in Table 3. Here, the fully trained context-aware model performs significantly worse than the context-agnostic model for retail. This may be caused by either unstable training or base architecture’s intrinsic limitations in simultaneously learning the time series features along with the contextual correlations. In contrast, our ContextFormer mechanism is not constrained by these limitations. A more comprehensive result comparing the training methods across multiple datasets is provided in Appendix E.2.

Can context-aware forecasting effectively capture both complex and simple metadata?
While the air quality, store sales, and Bitcoin datasets are rich in multimodal, continuous, time-varying metadata, the traffic and electricity datasets contain only one-dimensional, discrete metadata in the form of sensor IDs and user IDs, respectively (in addition to temporal information). Our method improves performance on both datasets, demonstrating its ability to capture complex, high-dimensional metadata while leveraging temporal information and basic contextual features to boost performance. The average improvement in MSE using the complex metadata for air quality forecasting was 11.1%, while the inclusion of temporal features and sensor IDs enhanced the average MSE for traffic forecasting by 15.2%.

Can a context-aware model outperform large pre-trained forecasters?
The rise of large language models (LLMs) with zero-shot learning capabilities has paved the way for foundation models in time series forecasting. These models claim strong zero-shot performance on new datasets, at par with the state-of-the-art
time series forecasting models that have been specifically trained on them (Ansari et al., ). Chronos, one such foundation model, was included in our comparisons as a benchmark. In our comparisons, Chronos demonstrated comparable performance while occasionally surpassing our models on specific datasets. For the case of traffic forecasting with $L=96$, although Chronos outperformed the iTransformer base architecture in MAE, our corresponding context-aware model was able to surpass Chronos. This suggests our approach may provide a parameter-efficient solution for enhancing the existing forecasters to outperform foundational models. It is important to note that our primary objective is to enhance the performance of existing forecasters, and therefore, the key comparison is solely against the corresponding base models. However, as demonstrated in Table 2, our context-aware models have the additional capability to outperform large-scale pre-trained forecasters on certain forecasting tasks.

**Table 4: Forecasting Bitcoin prices with textual news articles. An average of 11.5% improvement in MAE over baseline is observed on using news articles for bitcoin price forecasting.**
| Model | Method | MAE ($) | RMSE ($) |
| --- | --- | --- | --- |
| PatchTST | Context-agnostic | 627.41 | 913.57 |
| Context-aware (Ours) | 551.81 | 810.15 |  |
| iTransformer | Context-agnostic | 514.89 | 769.11 |
| Context-aware (Ours) | 467.19 | 703.05 |  |

What types of metadata modalities can be utilized by a context-aware forecaster?
In our experiments, we incorporated diverse metadata types, including variables such as temperature, oil prices, geographic location, and web search trends. To further assess ContextFormer’s ability to handle multimodal metadata, we conducted experiments forecasting Bitcoin prices using financial news articles as metadata. Since no existing dataset provided this particular combination of data, we curated a novel dataset comprising hourly Bitcoin prices from 2022 to 2024, alongside daily financial news articles scraped from the web. To incorporate the textual metadata into the forecasting model, the articles were represented as 1536-dimensional embeddings generated using the OpenAI Embeddings model (details in Appendix C). The forecasting task involved predicting Bitcoin prices one day ahead based on the previous four days of data and corresponding daily news ($L=96$, $T=24$). Results shown in Table 4 demonstrate that incorporating news articles as contextual information significantly improved forecasting metrics across both architectures, highlighting the model’s effectiveness in managing complex multimodal metadata. Unlike the results for the other experiments, which are presented on a normalized scale for cross-dataset comparison, Table 4 displays the results in their original scale to emphasize the real-world economic impact of the improved forecast.

Limitations. One of the key limitations of our approach is the lack of a principled method to find which metadata or contextual feature is important for forecasting. Identifying the key metadata features in advance can help limit the diversity of metadata provided as input to the model, thereby simplifying the learning process during the fine-tuning stage.

### 7 Conclusion

In this paper, we introduced ContextFormer, a novel technique for integrating contextual information into existing forecasting models. Through comprehensive evaluations on diverse real-world datasets, we demonstrated ContextFormer’s ability to effectively handle complex, multimodal metadata while consistently outperforming baseline models and even forecasting foundation models. In addition, we proposed a resource-efficient plug-and-play fine-tuning framework that offers significant improvements to forecasting accuracy over training context-aware models from scratch.

In future work, we aim to test our approach on other contextual modalities such as images, videos, etc. An interesting future work is to analyze the effects of forecasting metadata first; thereby, we propose a two-step forecasting pipeline where we first forecast metadata. We hypothesize that the forecasted metadata can be used to obtain better forecasts. Our key intuition is that metadata is more human-interpretable, and therefore forecasting metadata could be an easier task to solve.

## Appendix

### Appendix A MSE Loss and Mutual Information

Following the approach in Jing et al. (), we can demonstrate that minimizing the MSE loss between $\bm{X}_{\mathrm{future}}$ and $\hat{\bm{X}}_{\mathrm{future}}$ is equivalent to maximizing the mutual information $\mathcal{I}\left(\bm{X}_{\mathrm{future}};\hat{\bm{X}}_{\mathrm{future}}\right)$, assuming a Gaussian noise model between the two variables. Specifically, minimizing the MSE is shown to be equivalent to maximizing the log-likelihood, which, in turn, maximizes the mutual information.

Suppose that the relationship between $\bm{X}_{\mathrm{future}}$ and $\hat{\bm{X}}_{\mathrm{future}}$ is modeled as

$$ $\hat{\bm{X}}_{\mathrm{future}}=\bm{X}_{\mathrm{future}}+Z,$ $$

where $Z$ is Gaussian noise with zero mean and variance $\sigma^{2}$, i.e., $Z\sim\mathcal{N}(0,\sigma^{2})$. The log-likelihood of obtaining $\bm{X}_{\mathrm{future}}$ given $\hat{\bm{X}}_{\mathrm{future}}$ can be derived from the probability density function of the Gaussian distribution. The log-likelihood function is given by

$$ $\log p(\bm{X}_{\mathrm{future}}\mid\hat{\bm{X}}_{\mathrm{future}})=-\frac{1}{2 }\log(2\pi\sigma^{2})-\frac{\|\bm{X}_{\mathrm{future}}-\hat{\bm{X}}_{\mathrm{ future}}\|_{2}^{2}}{2\sigma^{2}}.$ $$

Since $\frac{1}{2}\log(2\pi\sigma^{2})$ is a constant with respect to $\bm{X}_{\mathrm{future}}$, the log-likelihood is maximized when the term $\|\bm{X}_{\mathrm{future}}-\hat{\bm{X}}_{\mathrm{future}}\|_{2}^{2}$ is minimized, which is the same as minimizing the MSE. Therefore, minimizing the MSE is equivalent to maximizing the log-likelihood.

Having shown the equivalence of MSE and the log-likelihood, now the mutual information $\mathcal{I}(\bm{X}_{\mathrm{future}};\hat{\bm{X}}_{\mathrm{future}})$ between $\bm{X}_{\mathrm{future}}$ and $\hat{\bm{X}}_{\mathrm{future}}$ can be expressed as

$$ $\mathcal{I}(\bm{X}_{\mathrm{future}};\hat{\bm{X}}_{\mathrm{future}})=\mathcal{ H}(\bm{X}_{\mathrm{future}})-\mathcal{H}(\bm{X}_{\mathrm{future}}\mid\hat{\bm{ X}}_{\mathrm{future}})$ $$

where $\mathcal{H}(\bm{X}_{\mathrm{future}})$ is the entropy of $\bm{X}_{\mathrm{future}}$ and $\mathcal{H}(\bm{X}_{\mathrm{future}}\mid\hat{\bm{X}}_{\mathrm{future}})$ is the conditional entropy of $\bm{X}_{\mathrm{future}}$ given $\hat{\bm{X}}_{\mathrm{future}}$. For Gaussian noise, $\mathcal{H}(\bm{X}_{\mathrm{future}}\mid\hat{\bm{X}}_{\mathrm{future}})$ is related to the conditional variance of $\bm{X}_{\mathrm{future}}$ given $\hat{\bm{X}}_{\mathrm{future}}$,

$$ $\mathcal{H}(\bm{X}_{\mathrm{future}}\mid\hat{\bm{X}}_{\mathrm{future}})=- \mathbb{E}\left[\log p(\bm{X}_{\mathrm{future}}\mid\hat{\bm{X}}_{\mathrm{ future}})\right].$ $$

Maximizing the likelihood of the observed data $\hat{\bm{X}}_{\mathrm{future}}$ given the model (in this case, $\bm{X}_{\mathrm{future}}$) reduces the uncertainty $\mathcal{H}(\bm{X}_{\mathrm{future}}\mid\hat{\bm{X}}_{\mathrm{future}})$, effectively increasing the mutual information. Now, we know that minimizing the MSE maximizes the log-likelihood. This corresponds to making the estimate $\hat{\bm{X}}_{\mathrm{future}}$ as close as possible to the true value $\bm{X}_{\mathrm{future}}$, which reduces the variance of the noise $Z$ (or the uncertainty in $\hat{\bm{X}}_{\mathrm{future}}$).

Since mutual information $\mathcal{I}(\bm{X}_{\mathrm{future}};\hat{\bm{X}}_{\mathrm{future}})$ is a measure of the reduction in uncertainty about $\bm{X}_{\mathrm{future}}$ given $\hat{\bm{X}}_{\mathrm{future}}$, minimizing the conditional variance (or equivalently, maximizing the log-likelihood) increases $\mathcal{I}(\bm{X}_{\mathrm{future}};\hat{\bm{X}}_{\mathrm{future}})$. Thus, minimizing MSE maximizes the log-likelihood, which in turn maximizes the mutual information $\mathcal{I}(\bm{X}_{\mathrm{future}};\hat{\bm{X}}_{\mathrm{future}})$,

$$ $\text{min }\mathbb{E}_{p_{\mathrm{data}}}\;{\|\bm{X}_{\mathrm{future}}-\hat{ \bm{X}}_{\mathrm{future}}\|}_{2}\iff\text{max }\mathcal{I}\left(\bm{X}_{ \mathrm{future}};\hat{\bm{X}}_{\mathrm{future}}\right).$ $$

### Appendix B TimesFM with Covariates

TimesFM (Das et al., ), developed by Google Research, is one of the latest foundational models for time-series forecasting. It boasts superior zero-shot performance compared to other foundational models across most of the commonly used benchmark datasets. A key feature of TimesFM is its ability to incorporate static and dynamic covariates during inference, making it a context-aware forecaster by our definition. As a multipurpose model, it is not trained on any dataset-specific covariates but at the time of dataset-specific inference, it treats them as exogenous regression variables and fits linear models onto them.

**Table 5: TimesFM’s inability to leverage historical metadata through in-context regression. Without access to future covariate values, the context-aware TimesFM model does not demonstrate any performance improvement across the datasets used in our main experiment using its simplistic linear regression approach.**
| Dataset | L | Context-Agnostic | Context-Aware |  |  |
| --- | --- | --- | --- | --- | --- |
| MAE | MSE | MAE | MSE |  |  |
| Air Quality | 48 | 0.571 | 0.807 | 0.611 | 0.847 |
| 96 | 0.638 | 0.986 | 0.659 | 1.01 |  |
| Electricity | 48 | 0.073 | 0.156 | 0.084 | 0.249 |
| 96 | 0.057 | 0.129 | 0.065 | 0.188 |  |
| Traffic | 48 | 0.702 | 2.172 | 0.795 | 2.516 |
| 96 | 0.679 | 2.143 | 0.758 | 2.428 |  |
| Retail | 48 | 0.108 | 0.185 | 0.114 | 0.215 |
| 96 | 0.133 | 0.236 | 0.131 | 0.274 |  |
| Bitcoin | 48 | 0.667 | 0.822 | 0.724 | 0.921 |
| 96 | 0.952 | 1.509 | 0.973 | 1.572 |  |

One limitation of such a simplistic batched in-context regression model is that it may be incapable of extracting complex correlations among the covariates and the time series. Moreover, the TimesFM implementation requires the presence of future values of dynamic covariates through the forecasting horizon; this kind of information is often unavailable in real-world scenarios. To address this, the TimesFM developers have proposed two stop-gap solutions: either shifting and repeating past dynamic covariates as delayed proxies for the future or “bootstrapping”, where TimesFM is first used to forecast these past covariates into the future and then called again using these forecasts as future covariates. We employ the earlier method to evaluate the model’s ability to perform using only historical covariates (or contextual information, in our terms). The results, given in table 5, highlight the model’s inability to improve its performance via its current context-aware implementation in the absence of future covariates, as for all of our datasets (except one), the context-agnostic TimesFM outperforms the context-aware one on both the metrics.

### Appendix C Dataset Description

We will now describe the datasets used for training, validation and evaluation of our proposed model.

#### C.1 Synthetic Data

We generated a dataset comprising 10000 time-series sequences, each containing 192 samples, based on ARMA(2,2) processes with randomly initialized coefficients, sampled from a uniform distribution, for the first preliminary experiment. The stability of each ARMA process was ensured by verifying that all the roots of the corresponding characteristic equations were within the unit circle. To cause some perturbation, Gaussian noise with 0.1 variance was added to all the sequences. The metadata for the dataset, consisting of the four ARMA coefficients, was continuous and time-invariant. These coefficients differed significantly from the ARMA coefficients obtained from the noisy data for most of the sequences. The dataset was split in a 7:1:2 ratio among the train, validation, and test splits.

#### C.2 Beijing AQ (Air Quality)

The dataset obtained from Chen () contains hourly air pollutant concentration data and corresponding meteorological data from 12 locations in Beijing. The task is to forecast a 6-channel multivariate time series using historical data and weather forecast metadata. Missing values in the dataset are handled by imputing continuous metadata and time series values with their mean, while missing categorical metadata (such as wind direction) are assigned an “unknown” label. The data, spanning from 2013 to 2017, is split into training, validation, and test sets in a 7:1:2 ratio. For each set, we first apply a sliding window of length 144 with a stride of 24, resulting in 9828 training, 1332 validation, and 2796 test time series samples. Then, we use a sliding window of length 192 with a stride of 24, yielding 9012 training, 1188 validation, and 2556 test time series.

#### C.3 Store Sales (Retail)

The dataset, sourced from a popular Kaggle competition (Kaggle, ), contains daily sales data from 2013 to 2017 for 34 product families sold across 55 Favorita stores in Ecuador. The dataset includes features such as store number, product family, promotional status, and sales figures. Additionally, supplementary information like store metadata and oil prices is provided, offering time-varying metadata that can be leveraged for forecasting. For the forecasting tasks, we consider the complete time series for each product in each store, which is univariate time series. Initially, we apply a sliding window of length 144 with a stride of 24, resulting in 53460 training, 17820 validation, and 24948 test time series samples. Next, we use a sliding window of length 192 with a stride of 24, yielding 49896 training, 14256 validation, and 21384 test time series samples.

#### C.4 PEMS-SF (Traffic)

The dataset, sourced from Wu et al. (), contains 15 months of daily data from the California Department of Transportation PEMS website. It captures the occupancy rate (ranging from 0 to 1) of various freeway car lanes in the San Francisco Bay area. The data spans from 2008 to 2009, with measurements sampled every 10 minutes. The forecasting task involves predicting the electricity demand pattern for a single sensor (selected from a total of 861 sensors), framed as a univariate time series forecasting problem. The data is split temporally into training, validation, and test sets in a 7:1:2 ratio. We then apply sliding windows of lengths 144 and 192, each with a stride of 24. The number of samples for all the sets is given in Table 7.

#### C.5 ECL (Electricity )

The dataset taken from Trindade () consists of power consumption data for 370 users in Portugal over a period of 4 years from 2011 to 2015. It is a commonly used dataset for time series forecasting (Wu et al., ; Liu et al., ; Ansari et al., ). The forecasting task with respect to this dataset is to predict the electricity demand pattern for a single user, which is framed as a univariate time series forecasting problem. In the absence of any innate metadata features, we consider the 370 user IDs to be the only metadata. The data is sampled every 15 minutes, resulting in a time series with 96 daily timesteps. We process the data to remove days with significant 0 values. The data has been split, and sliding windows with stride 24 are used to get approximately a 7:1:2 ratio for the number of samples in the training, validation, and test sets, with the exact numbers given in table 7.

#### C.6 Monash (Bitcoin)

The dataset, sourced from the Monash Time Series Forecasting Repository (Godahewa et al., ), contains daily Bitcoin closing prices from 2010 to 2021, along with 18 potential influencing factors. These include metrics like hash rate, block size, mining difficulty, and social indicators such as the number of tweets and Google searches related to the keyword “Bitcoin.” During preprocessing, we excluded the number of tweets due to its limited availability, leaving us with 17-dimensional continuous time-varying metadata for the univariate forecasting task. The dataset is divided into training, validation, and test sets in a 7:1:2 ratio. We apply sliding windows of lengths 144 and 192 with a stride of 24 to the data.

**Table 6: Dataset Summary. For our main experiments, we selected real-world datasets that encompass prevalent forecasting tasks across diverse domains, including environmental science, energy, finance, retail, and transportation.**
| Dataset | Time-series | Continuous | Categorical |
| --- | --- | --- | --- |
|  | Data | Metadata | Metadata |
| Air   Quality | CO, NO2, SO2, O3, PM2.5, and PM10 Concentration | Temperature, Humidity, Wind Speed, Pressure, Dew Point | Location, Wind Direction |
| Retail | Product Sales Volume | Promotional Offers, Oil Prices | Store ID, Location, Item category |
| Traffic | Traffic Volume | None | Sensor ID |
| Electricity Load | Electricity     consumption | None | User ID |
| Bitcoin | Bitcoin Prices | 17 factors | None |

#### C.7 Additional Experiment (Bitcoin-News)

In the absence of a dataset containing both Bitcoin prices and corresponding news articles, we created a new dataset with hourly Bitcoin prices and daily financial news articles scraped from online sources between 2022 and 2024. The articles were processed using the OpenAI Embeddings (OpenAI, ) model ‘text-embedding-1-small’, generating a 1536-dimensional embedding for each day’s news. These embeddings were treated as continuous time-varying metadata for the univariate forecasting of hourly Bitcoin closing prices. This experiment was conducted with a fixed lookback length of $L=96$ and a forecasting horizon of $T=24$, allowing us to forecast daily Bitcoin trends based on the previous four days’ history and news articles. As before, the data was split temporally into training, validation, and test sets in a 7:1:2 ratio, with a sliding window of length 120 and a stride of 24 applied to create the datasets.

**Table 7: Dataset Description. This table summarizes the main features of the datasets used in our experiments, such as dimensionality, sample size, frequency, and training batch size. The dataset sample size are given in the format of (training size, validation size, test size).**
| Dataset | Channels | Size ($L=48$) | Size ($L=96$) | Frequency | Batch Size |
| --- | --- | --- | --- | --- | --- |
| Synthetic | 1 | - | (7000,1000,2000) | N.A. | 128 |
| Air Quality | 6 | (9828,1332,2796) | (9012,1188,2556) | Hourly | 32 |
| Retail | 1 | (53460,17820,24948) | (49896,14256,21384) | Daily | 128 |
| Electricity | 1 | (50265,9102,13627) | (50262,8750,13626) | 15 Minute | 128 |
| Traffic | 1 | (435666, 58548, 121401) | (433944, 56826, 119679) | 10 Minute | 384 |
| Bitcoin | 1 | (104,10,26) | (102,8,24) | Daily | 1 |
| Bitcoin-News | 1 | - | (524,71,148) | Hourly | 4 |

### Appendix D Implementation Details

#### D.1 Context-aware Autoregression

To provide theoretical justification for the ideas in Subsection 4.2, we generated a dataset consisting of 500-length sequences formed by linear combinations of an autoregressive process with five latent variables, followed by added perturbations. For this scenario, we choose the latent variables as the contextual information. The resulting dataset, denoted as ${\mathcal{D}}$, is structured as $\left\{\left({\bm{X}}^{n},{\bm{C}}^{n},{\bm{Y}}^{n}\right)\right\}_{n=1}^{N}$, For our experiments, we set $N=1000$.

We initially start by modeling these sequences through vanilla AR(10) models of the form ${\bm{Y}}={\bm{X}}{\bm{\beta}}+{\bm{\epsilon}}$, where ${\bm{X}}$ is a $490\times 10$ matrix and ${\bm{Y}}$ is a $490$-length vector. The corresponding paired metadata for such a sequence can be represented as ${\bm{C}}$, which is a $490\times 5$ matrix.

Next, we incorporate contextual metadata into the existing AR models by fitting them as exogenous regressors for the residuals. To demonstrate the impact of increasing context on forecasting accuracy, we gradually increase the dimensionality of the regression model from $q=1$ to $5$. The results for this context-aware autoregressive forecaster are visualized in Fig. 4.

#### D.2 Context-aware Transformers

##### D.2.1 Base Architectures

The core architecture for both the ContextFormer-enhanced forecasters contains an input history embedding layer, six hidden layer blocks, and a final projection layer. Each of the hidden layer blocks consists of two parallel cross-attention layers and a self-attention layer. Each attention layer operates in a 256-dimensional representational space while employing an 8-head attention mechanism.

PatchTST (Nie et al., ) is a popular transformer-based architecture for time series forecasting. In contrast to the traditional models that treat each time step as an individual token, PatchTST divides the data into patches, similar to what Vision Transformers (Dosovitskiy et al., ) do for images. Each patch in this setup represents a sequence of time steps, which enables the model to focus on long-term temporal patterns. By applying self-attention to these patches, PatchTST captures long-range dependencies more efficiently, reducing the computational cost associated with traditional transformers. This patch-based approach enables the model to handle longer sequences and large-scale forecasting tasks more effectively. PatchTST outperforms standard transformers on various benchmarks and can handle both univariate and multivariate time series data, making it a versatile choice for various forecasting tasks.

**Table 8: Design parameters for our experiments. The hyperparameters for embedding dimensions, attention heads, and activation functions are consistent across all the transformers in the base model and ContextFormer additions. The number of self-attention layers is the same for both the base models. The learning rate and dropout remain fixed throughout all the forecasters across the experiments. The parameters for patch length and stride are specific to the PatchTST input layer.**
| Design Parameter | Value |
| --- | --- |
| Embedding Dimension | 256 |
| Self-attention Layers | 6 |
| Attention Heads | 8 |
| Activation | GeLU |
| Patch Length | 16 |
| Stride | 8 |
| Learning Rate | $3\times 10^{-5}$ |
| Dropout | 0.1 |

iTransformer (Liu et al., ) extends the patching concept introduced in PatchTST by applying the model to inverted dimensions of the time series. Rather than embedding time steps, iTransformer treats each variable or feature of the time series as separate tokens. This shifts the focus from temporal dependencies to relationships between features across time. Despite this inversion, iTransformer retains core Transformer components, including multi-head attention and positional feed-forward networks, but applies them in a way that fundamentally alters how dependencies are modeled. Although the original iTransformer architecture can handle timestamps, we intentionally excluded them for the context-agnostic variant of the model. The rest of the implementation for both the base models is the same as what is given in the official TimesNet (Wu et al., ) Github repository.

##### D.2.2 ContextFormer Additions

Metadata Embedding.
The metadata embedding module consists of two fully connected (FC) networks followed by a transformer encoder. Discrete metadata is one-hot encoded and processed through one FC network, while continuous metadata is passed through a separate FC network. The input size of each network corresponds to the number of discrete or continuous features in the dataset, with the output dimensions consistently set to 256 for all experiments. If both metadata types are present, then their outputs are concatenated and processed through a linear layer to reduce the dimensionality from 512 to 256; otherwise, the single output is used. The resulting output is then added to the positional encodings and passed through a transformer encoder, generating the metadata embedding. This implementation of metadata embedding follows the approach described in Narasimhan et al. () for constructing metadata encoders in conditional time series generation.

**Table 9: Hyperparameters for the embedding modules. Both metadata and temporal embedding blocks share the same design parameters.**
| Design Parameter | Value |
| --- | --- |
| Feed-forward Dimension | 256 |
| Embedding Dimension | 256 |
| Self-attention Layers | 2 |
| Attention Heads | 8 |
| Activation | GeLU |

Temporal Embedding. The temporal embedding block shares a similar architecture with the metadata embedding but is specifically designed for continuous temporal data. Timestamp information, such as the year, month, day, and hour, is decomposed based on the dataset’s granularity and treated as continuous contextual features. In essence, the temporal embedding functions like a metadata embedding module, where the temporal data is embedded as continuous metadata. Unlike the metadata encoder, the temporal encoder exclusively handles continuous features and focuses entirely on capturing the temporal characteristics of the input.

##### D.2.3 Other Details

Training Parameters. The learning rate was set to $3\times 10^{-5}$ for all experiments, and a dropout rate of 0.1 was applied throughout the training process. Each of the ContextFormer models was trained for 100 epochs: the first 50 epochs focused on training the base models using historical data, while the remaining 50 epochs were dedicated to the ContextFormer fine-tuning. In the experiments described in Appendix E.2, the fully-trained context-aware models were trained on the time series data and paired metadata until convergence. The model with the lowest validation loss was saved and used for inference.

Model Size. The context-aware models comprised an average of approximately 13 million parameters. Of these, an average of 3.5 million parameters were associated with the original model, while the remaining approximately 9.5 million parameters were introduced by the ContextFormer additions. The total number of parameters varied slightly depending on the specific base architecture, the dimensions of the time series, and the corresponding metadata.

Software and Hardware.
All experiments were conducted using Python 3.9.12 and PyTorch 2.0.0 (Paszke et al., ), running on Nvidia RTX A5000 GPUs.

#### D.3 Bechmark Forecaster

Chronos (Ansari et al., ), introduced by Amazon Sciences in 2024, is a collection of pre-trained foundational time series models that leverage language model architectures. This framework preprocesses time series sequences into a fixed vocabulary through scaling and quantization. These tokenized sequences are then used to train transformer-based language models using cross-entropy loss. The Chronos architecture is based on the T5 family of language models, having 8 million to 710 million parameters, and is pre-trained on a large set of real-world datasets and synthetic data. For our experiments, we specifically use the ‘chronos-t5-base’ variant with 200 million parameters, which is more than 14 times the number of parameters in our largest context-aware model. As per Ansari et al. (), Chronos models significantly outperform both the classical statistical methods as well as specialized deep-learning models on the datasets on which they have been trained on, while showing comparable zero-shot results on new datasets. Among the datasets we used, Chronos was only trained on the Electricity dataset. Thus, we compare our trained model with Chronos’ zero-shot forecasting performance for the rest of the dataset. Note that Chronos can handle only single-channel data, thus, for the evaluation of the multivariate datasets, all the channels were processed as separate time series sequences.

### Appendix E Ablation Study

#### E.1 Main Results with Standard Deviation

For robust experimental results, each experiment is repeated three times with different random seeds. Due to space constraints, the main text presents the results without standard deviations. The complete results, including standard deviations, are provided in Table 10.

**Table 10: Results with Standard Deviation. We compare the PatchTST and iTransformer with and without the ContextFormer additions on forecasting time series with a fixed lookback length $L=96$ and prediction lengths $T\in\{48,96\}$. The mean and standard deviation are calculated for each experiment with three different seeds.**
| Model | PatchTST | itransformer |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Context-Agnostic | Context-Aware | Context-Agnostic | Context-Aware |  |  |  |  |  |
| Dataset | T | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE |
| Air Quality | 48 | $0.573\pm 0.005$ | $0.770\pm 0.013$ | $0.524\pm 0.002$ | $0.674\pm 0.011$ | $0.577\pm 0.005$ | $0.771\pm 0.006$ | $0.540\pm 0.002$ | $0.696\pm 0.002$ |
| 96 | $0.622\pm 0.006$ | $0.901\pm 0.014$ | $0.572\pm 0.004$ | $0.802\pm 0.009$ | $0.631\pm 0.004$ | $0.919\pm 0.007$ | $0.591\pm 0.006$ | $0.813\pm 0.013$ |  |
| Electricity | 48 | $0.038\pm 0.001$ | $0.058\pm 0.003$ | $0.024\pm 0.001$ | $0.029\pm 0.001$ | $0.042\pm 0.002$ | $0.067\pm 0.008$ | $0.028\pm 0.001$ | $0.035\pm 0.001$ |
| 96 | $0.031\pm 0.005$ | $0.040\pm 0.009$ | $0.024\pm 0.001$ | $0.027\pm 0.002$ | $0.038\pm 0.001$ | $0.055\pm 0.002$ | $0.024\pm 0.001$ | $0.028\pm 0.001$ |  |
| Traffic | 48 | $1.101\pm 0.066$ | $3.527\pm 0.179$ | $0.865\pm 0.009$ | $2.922\pm 0.047$ | $1.022\pm 0.019$ | $3.265\pm 0.096$ | $0.848\pm 0.016$ | $2.868\pm 0.024$ |
| 96 | $1.084\pm 0.042$ | $3.415\pm 0.173$ | $0.845\pm 0.014$ | $2.767\pm 0.049$ | $1.025\pm 0.031$ | $3.165\pm 0.034$ | $0.830\pm 0.002$ | $2.766\pm 0.020$ |  |
| Retail | 48 | $0.123\pm 0.002$ | $0.238\pm 0.005$ | $0.115\pm 0.001$ | $0.228\pm 0.001$ | $0.129\pm 0.002$ | $0.257\pm 0.004$ | $0.124\pm 0.001$ | $0.257\pm 0.005$ |
| 96 | $0.139\pm 0.002$ | $0.291\pm 0.010$ | $0.128\pm 0.001$ | $0.265\pm 0.004$ | $0.145\pm 0.002$ | $0.309\pm 0.006$ | $0.143\pm 0.001$ | $0.310\pm 0.012$ |  |
| Bitcoin | 48 | $0.854\pm 0.045$ | $1.231\pm 0.120$ | $0.821\pm 0.038$ | $1.192\pm 0.101$ | $0.832\pm 0.020$ | $1.177\pm 0.053$ | $0.810\pm 0.021$ | $1.153\pm 0.078$ |
| 96 | $0.971\pm 0.004$ | $1.561\pm 0.029$ | $0.948\pm 0.003$ | $1.537\pm 0.009$ | $0.992\pm 0.010$ | $1.650\pm 0.029$ | $0.951\pm 0.005$ | $1.547\pm 0.022$ |  |

#### E.2 Full-training Vs. Fine-tuning of Context-aware Models

To empirically support the use of fine-tuning over training from scratch, we present the results for numerous datasets using both approaches. These results in Table 11 show that even when training the context-aware architecture from scratch, its performance often falls short of our fine-tuned models for most of the datasets. This discrepancy could be attributed to the factors discussed in Sec. 5.2 or other potential causes.

**Table 11: Full Training Vs. Fine-tuning. We compare the two training methods for the context-aware PatchTST and iTransformer for time series forecasting with a fixed lookback length $L=96$ and prediction lengths $T\in\{48,96\}$. For most of the datasets, our plug-and-play fine-tuning method outperforms its fully-trained counterpart.**
| Model | PatchTST | itransformer |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Training | Full | Fine-tune | Full | Fine-tune |  |  |  |  |  |
| Dataset | T | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE |
| Air Quality | 48 | 0.507 | 0.611 | 0.524 | 0.674 | 0.530 | 0.646 | 0.540 | 0.696 |
| 96 | 0.550 | 0.717 | 0.572 | 0.802 | 0.562 | 0.734 | 0.591 | 0.813 |  |
| Electricity | 48 | 0.030 | 0.038 | 0.029 | 0.036 | 0.029 | 0.035 | 0.028 | 0.035 |
| 96 | 0.022 | 0.025 | 0.024 | 0.027 | 0.025 | 0.029 | 0.024 | 0.028 |  |
| Traffic | 48 | 0.994 | 3.128 | 0.865 | 2.922 | 0.917 | 2.903 | 0.848 | 2.868 |
| 96 | 0.955 | 3.006 | 0.845 | 2.767 | 0.896 | 2.859 | 0.830 | 2.766 |  |
| Retail | 48 | 0.116 | 0.230 | 0.115 | 0.228 | 0.121 | 0.252 | 0.124 | 0.257 |
| 96 | 0.154 | 0.370 | 0.128 | 0.265 | 0.151 | 0.326 | 0.143 | 0.310 |  |
| Bitcoin | 48 | 0.849 | 1.226 | 0.821 | 1.192 | 1.038 | 1.796 | 0.810 | 1.153 |
| 96 | 0.983 | 1.656 | 0.948 | 1.537 | 0.962 | 1.604 | 0.951 | 1.547 |  |

#### E.3 Effect of Temporal Information vs Full Context

Throughout this text, we have used the term contextual information to encompass all static or time-varying, continuous or discrete data associated with the time series in question. This includes easily accessible temporal information such as timestamps (e.g., month or day), which can significantly impact forecasting accuracy, especially for datasets with long-range periodicity. While many models already leverage timestamp encoding, our primary focus is on incorporating paired metadata alongside temporal information for time-series forecasting. Our architecture uses separate embeddings and cross-attention layers for temporal and non-temporal data. In Table 12, we present results comparing the effects of using only temporal information (commonly available in time series) versus utilizing the full contextual information, which includes both timestamps and associated metadata.

**Table 12: Temporal Information Vs. Full Context. We compare the performance of the context-aware PatchTST and iTransformer models for time-series forecasting, using both temporal information alone and the full context, which includes timestamps and metadata, with a fixed lookback length of $L=96$ and prediction lengths of $T\in\{48,96\}$. Incorporating the full contextual information, including metadata, leads to significant performance improvements over using only temporal data across most datasets.**
| Model | PatchTST | itransformer |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Context | Temporal | Full | Temporal | Full |  |  |  |  |  |
| Dataset | T | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE |
| Air Quality | 48 | 0.537 | 0.702 | 0.524 | 0.674 | 0.559 | 0.730 | 0.540 | 0.696 |
| 96 | 0.587 | 0.829 | 0.572 | 0.802 | 0.607 | 0.858 | 0.591 | 0.813 |  |
| Electricity | 48 | 0.033 | 0.044 | 0.029 | 0.036 | 0.031 | 0.040 | 0.028 | 0.035 |
| 96 | 0.029 | 0.034 | 0.024 | 0.027 | 0.027 | 0.032 | 0.024 | 0.028 |  |
| Traffic | 48 | 0.912 | 2.933 | 0.865 | 2.922 | 0.882 | 2.856 | 0.848 | 2.868 |
| 96 | 0.901 | 2.819 | 0.845 | 2.767 | 0.882 | 2.799 | 0.830 | 2.766 |  |
| Retail | 48 | 0.119 | 0.232 | 0.115 | 0.228 | 0.127 | 0.260 | 0.124 | 0.257 |
| 96 | 0.133 | 0.276 | 0.128 | 0.265 | 0.144 | 0.307 | 0.143 | 0.310 |  |
| Bitcoin | 48 | 0.833 | 1.204 | 0.821 | 1.192 | 0.831 | 1.244 | 0.810 | 1.153 |
| 96 | 0.949 | 1.538 | 0.948 | 1.537 | 0.952 | 1.538 | 0.951 | 1.547 |  |

### Appendix F Additional Qualitative Plots

In this section, we highlight the top three examples of both improvement and degradation in forecast quality after incorporating ContextFormer across various base architectures and datasets. This provides an unbiased comparison between context-aware and context-agnostic forecasts. Note that all plots in this section correspond to a forecasting horizon of 96 steps.

Figure: Figure 7: Qualitative Plots for Air Quality Dataset. The plots showcase the top three examples with the highest degradation and the improvement in MSE for the ContextFormer forecasts compared to the baseline.
Refer to caption: x7.png

Figure: Figure 8: Qualitative Plots for Electricity Dataset. The plots showcase the top three examples with the highest degradation and the improvement in MSE for the ContextFormer forecasts compared to the baseline.
Refer to caption: x8.png

Figure: Figure 9: Qualitative Plots for Traffic Dataset. The plots showcase the top three examples with the highest degradation and the improvement in MSE for the ContextFormer forecasts compared to the baseline.
Refer to caption: x9.png

Figure: Figure 10: Qualitative Plots for Retail Dataset. The plots showcase the top three examples with the highest degradation and the improvement in MSE for the ContextFormer forecasts compared to the baseline.
Refer to caption: x10.png

Figure: Figure 11: Qualitative Plots for Bitcoin Dataset. The plots showcase the top three examples with the highest degradation and the improvement in MSE for the ContextFormer forecasts compared to the baseline.
Refer to caption: x11.png

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 5
presentation: 2
questions: See the weakness
soundness: 3
strengths: The writing is easy to follow
summary: This paper presents a work on aggregating contextual information into time series forecasting. Specifically, the authors propose to use a universal context encoder to encode the contextual information as embedding and boost the time series forecastor with the encoding. Experimental results suggest that the contextual information boost the performance.
weaknesses: 1. The contribution is limited. This paper mainly discuss about contextual information in time series forecasting. However, it is well known that the contextual information is helpful for time series. What really matters is how to make use of it. In this paper, the authors list three reasons about why it is hard to make use of the information, such as multi modality and non-uniformity across domains. However, when really talking about how to make use of the context, the authors just simply concatenate the category variables with continuous variables. This design is unrelevant with the challenges mentioned. What if some out-of-domain dataset contain some variables that are unseen on the training data? Also, the design is well-justified. Why not align the metadata as one modality (like text) and then convert it to embeddings with one shared encoder? Do we need to align the timely metadata (like news with timestamp) with the speific timestamp before forwarding them all into the cross attention?

2. The theoretic analysis is useless but takes 1.5 pages. All the theoretic results are trivial corollary of conclusions from introductory undergraduate-level text book of information theory and machine learning. I suggest that the authors are just trying to decorate their papers with some theoretic analysis

3. Experimental results are not comprehensive enough. We prefer experiments on more datasets (like ETT) and baselines (like TimesNet).

4. The authors list 4 reasons why they prefer fine-tuning in Sec 5.2. But there is lack of empirical supports. I did not see training curves that reflect the unstable training of all-from-scratch. 

5. The bitcoin-news dataset description is too short. This could be the most insightful part about how to collect contextual data. But there is no details. How and where did you get the data? Are the data filtered? Are the collection based on time or keywords? Some many details are missing.

---

confidence: 5
contribution: 1
initial_rating: 5
presentation: 3
questions: The presentation and problem setup are clear and easy to follow. However, I have two major concerns regarding the theoretical analysis of the benefits of adding exogenous variables and the overall technical contribution of the paper.
1. By definition, mutual information is non-negative. Even in the worst-case scenario where A and C are completely independent, the increase in mutual information is 0. A higher mutual information value does not necessarily translate to higher accuracy or better performance. Noisy or even misleading data can also increase mutual information as long as some degree of dependency exists. Could the authors elaborate on this issue?
2. Adding exogenous variables to time series forecasting models is not a novel concept. It is a natural extension of general time series forecasting models and is a dominant approach for work in some domains, such as financial market predictions. For example, older transformer-based models like Autoformer can be adapted to include exogenous variables and are often used as baselines in other papers. Moreover, newer transformer-based models, such as Timexer, also support this approach natively. There are also attempts to use large language models (LLMs) to incorporate exogenous market information for stock movement prediction, such as Plutos. Therefore, I believe the overall contribution of the work is somewhat weak, as it does not introduce a new concept.
3. It is expected that the authors should at least compare the proposed framework with other models that can accept the same input. Otherwise, it is difficult to justify the performance of the proposed framework.
soundness: 3
strengths: 1. The presentation is clear.
2. The problem studied is interesting.
summary: This paper presents a framework designed to incorporate exogenous variables into time series forecasting.
weaknesses: 1. The contribution is limited.
2. There is a lack of comparison with SOTA models.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Refer to "Strengths And Weaknesses".
soundness: 3
strengths: 1. The starting point of this article is good. Exogenous contextual features can indeed serve as key auxiliary information to influence time series forecasting.
2. The paper is mostly well-presented and easy to follow.
summary: This article proposes ContextFormer, a novel plug-and-play framework that utilizes cross-attention blocks to integrate multimodal metadata, including categorical, continuous, time-varying contexts, and even textual information, into existing any context-agnostic base forecasters. The author selects two SOTA models of PatchTST and iTransformer as the base forecasters and validates ContextFormer framework on seven real-world datasets spanning energy, traffic, environmental, and financial domains. Experimental results confirm the effectiveness of the framework.
weaknesses: 1. Introduction part summarizes three challenges in incorporating metadata into forecasting models, but this article does not provide a detailed explanation of why ContextFormer can solve these problems. For example, the first challenge mentions the importance of aligning time series history and multimodal metadata. However, there is a lack of evidence on how ContextFormer ensures that the two have been aligned, the alignment effect, and the impact of aligned representation on forecasting results. Although the effectiveness of ContextFormer is reflected in the final MSE/MAE metrics, I mainly focus on these intermediate results and suggest the author to supplement these explanations.

2. In related works part, the author lists some methods for forecasting with covariates. As far as I know, it also includes methods such as TFT, TSMixer, and TimeXer, etc., but this article does not compare with these methods. It is suggested that the author supplements these comparison experiments.
Some references:
TFT: https://doi.org/10.1016/j.ijforecast.2021.03.012,
TSMixer: https://arxiv.org/pdf/2303.06053,
TimeXer: https://arxiv.org/pdf/2402.19072.

3. In Table 2 caption, it is mentioned that "The best results for each base model in each row are highlighted in bold.". However, the results of the Retail dataset (horizon=96, MSE metric) do not match this statement. It is recommended to revise the wording.

4. Doubt about lines 491-492: 
    1) The explicit motivation for comparing with Chronos and whether the comparison is fair?  
    2) Are the results in Table 2 sufficient to support this conclusion? I agree with the example provided by the author (486~487), but for other datasets, I have the following questions: (a) Is it still necessary to compare the context-aware model with Chronos for the two datasets of Air quality and Electricity, as the context-agnostic model is already better than Chronos? (b) For the Retail and Bitcoin datasets, Chronos performs the best. I hope the author can provide detailed explanations to alleviate my concerns.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: The authors can refer to the weakness listed above.
soundness: 2
strengths: 1. As a plug-in solution, ContextFormer can be integrated with various predictive backbones for a diverse array of applications.  
2. The two-stage fine-tuning methodology guarantees that the lower bound of ContextFormer is at least equivalent to that of context-agnostic approaches.
3. The extensive experiments look hopeful.
summary: This paper introduces ContextFormer, a method designed to incorporate multimodal contextual information into existing time series forecasting models. Traditional models typically rely solely on historical data, neglecting external influences such as news, weather, and market trends that can significantly impact predictions. ContextFormer effectively addresses this limitation by integrating diverse types of metadata—including categorical, continuous, time-varying, and textual information—using cross-attention mechanisms. The experiments demonstrate that ContextFormer can enhance forecasting performance by up to 30% compared to state-of-the-art models across various datasets in fields such as energy, traffic, environment, and finance.
weaknesses: 1. **Inadequate Presentation.** The organization of the paper lacks rationality. Excessive emphasis is placed on the importance of external context, which is well-known. The methods section is overly succinct. The descriptions of Metadata Embedding and Temporal Embedding are overly concise, leaving the application dimensions and the output shape ambiguous. Furthermore, the embedded interaction in the Information Fusion component is similarly unclear, too. Its applicability to both a single embedding for one variable (iTransformer and PatchTS) and a single embedding for one timestamp (Informer and Autoformer) across two distinct architectures remains uncertain. In summary, the workflow of the algorithm is perplexing.

2. **Limited Innovativeness.** Although the author presents a case in the introduction regarding the influence of news on stock prices, the treatment of this unstructured external text information is only mentioned in Appendix C.7 cursory and lacks corresponding experimental results on BITCOIN-NEWS. The models and data discussed throughout the paper primarily rely on structured auxiliary information, despite incorporating both continuous and categorical variables. The prior research has focused on integrating structured external information to enhance prediction accuracy, including exogenous variables [1] and timestamps [2]. The paper's core contribution remains ambiguous. Moreover, The three challenges outlined in the introduction are unpersuasive.

3. **Unrigorous Experiments.** As stated in Weaknesses 2, there has been some prior work aimed at integrating structured external information to enhance prediction accuracy. This paper should be compared with these frameworks that utilize external information, rather than solely with models that lack auxiliary information.

**References**

[1] 2024, TimeXer: Empowering Transformers for Time Series Forecasting with Exogenous Variables

[2] 2024, Rethinking the Power of Timestamps for Robust Time Series Forecasting: A Global-Local Fusion Perspective

Meta-review: In this paper, the authors propose an approach to leveraging contextual features in time series prediction. Reviewers found the paper well-written but noted that its contribution is limited, the theoretical analysis lacks depth, and the experimental results are insufficient. They also commented that the work does not provide much insight into why the proposed approach effectively addresses the problem. Based on these concerns, I believe this work falls below the acceptance threshold.

Decision: Reject