SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Andreas Auer * Martin Gauch*,† Daniel Klotz* Sepp Hochreiter*
*ELLIS Unit Linz and LIT AI Lab, Institute for Machine Learning,
Johannes Kepler University Linz, Austria
$\dagger$  Google Research, Zurich, Switzerland
To quantify uncertainty, conformal prediction methods are gaining continuously more interest and have already been successfully applied to various domains. However, they are difficult to apply to time series as the autocorrelative structure of time series violates basic assumptions required by conformal prediction. We propose HopCPT, a novel conformal prediction approach for time series that not only copes with temporal structures but leverages them. We show that our approach is theoretically well justified for time series where temporal dependencies are present. In experiments, we demonstrate that our new approach outperforms state-of-the-art conformal prediction methods on multiple real-world time series datasets from four different domains.
Uncertainty estimates are imperative to make actionable predictions for complex time-dependent systems (e.g., Gneiting & Katzfuss, 2014; Zhu & Laptev, 2017). This is particularly evident for environmental phenomena such as flood forecasting (e.g., Krzysztofowicz, 2001), since they exhibit pronounced seasonality. Conformal Prediction (CP, Vovk et al., 1999) provides uncertainty estimates based on prediction intervals. It achieves finite-sample marginal coverage with almost no assumptions, except that the data is exchangeable (Vovk et al., 2005; Vovk, 2012). However, CP for time series is not trivial because temporal dependencies often violate the exchangeability assumption.
HopCPT. For time series models that predict a given system, the errors are typically specific to the input describing the current situation. To apply CP, HopCPT uses continuous Modern Hopfield Networks (MHNs). The MHN uses large weights for past situations that were similar to the current one and weights close to zero for past situations that were dissimilar to the current one (Figure 1). The CP procedure uses the weighted errors to construct strong uncertainty estimates close to the desired coverage level. This exploits that similar situations (which we call a regime) tend to follow the same error distribution. HopCPT achieves new state-of-the-art efficiency, even under non-exchangeability and on large datasets.
Our main contributions are:
1. We propose HopCPT, a CP method for time series, a domain where CP struggled so far.
2. We introduce the concept of error regimes to CP, which softens the excitability requirements and enables efficient CP for time series.
3. HopCPT uses MHN for a similarity-based sample reweighting. In contrast to existing approaches, HopCPT can learn from large datasets and predict intervals at arbitrary coverage levels without retraining.
4. HopCPT achieves state-of-the-art results for conformal time series prediction tasks from various real-world applications.
Figure 1: Schematic illustration of HopCPT. The Modern Hopfield Network (MHN) identifies regimes similar to the current one and up-weights them (colored lines). The weighted information enriches the conformal prediction (CP) procedure so that prediction intervals can be derived.
5. HopCPT is the first algorithm with coverage guarantees that was applied to hydrological prediction applications — a domain where uncertainty plays a key role in tasks such as flood forecasting and hydropower management.
Regimes. In a world with non-linear dynamics, different environmental conditions lead to different error characteristics of models that predict based on these conditions. If we do not account for these different conditions, temporal changes may lead to unnecessarily large prediction intervals, i.e., to high uncertainty. For example, solar energy production is high and stable on a sunny day, fluctuates during cloudy days, and is zero at night. Often, the current environmental condition was already observed at previous points in time. The error at these time steps is therefore assumed to have the same distribution as the current error. Following Quandt (1958) and Hamilton (1990), we call the sets of time steps with similar environmental conditions regimes. Although conditional CP is in general impossible (Foygel Barber et al., 2021), we show that conditioning on such regimes can lead to better prediction intervals while preserving the specified coverage. Hamilton (1990) models time series regimes as a discrete Markov process and conditions a classical autoregressive model on the regime states. Sanquer et al. (2012) use a smooth transition approach to model multi-regime time series. Tajeuna et al. (2021) propose an approach to discover and model regime shifts in an ecosystem that comprises multiple time series. Further, Masserano et al. (2022) handle distribution shifts by retraining a forecasting model with training data from a non-uniform adaptive sampling. Although these approaches are not in a CP setting, their work is similar in spirit, as they also follow the general idea to condition on parts of the time series with similar regimes.
CP and extensions. For thorough introductions to CP, we refer the reader to the foundational work of Vovk et al. (1999) and a recent introductory paper by Angelopoulos & Bates (2021). There exist a variety of extensions for CP that go "beyond exchangeability" (Vovk et al., 2005). For example, Papadopoulos & Haralambous (2011) apply CP to a nearest neighbor regression setting, Teng et al. (2022) apply CP to the feature space of models, Angelopoulos et al. (2020) use CP to generate uncertainty sets for image classification tasks, and Toccaceli et al. (2017) use a label-conditional variant to apply CP to biological activity prediction. Of specific interest to us is the research regarding non-exchangeable data of Tibshirani et al. (2019) and Foygel Barber et al. (2022). Both handle potential shifts between the calibration and test set by reweighting the data points. Tibshirani et al. (2019) restrict themselves to settings with full knowledge about the change in distribution; Foygel Barber et al. (2022) rely on fixed weights. In our work, we refrain from this assumption because such information is typically not available in time series prediction. Another important research direction is the work on normalized conformity scores (see Fontana et al., 2023, and references therein). In this setting, the goal is to adapt the conformal bounds through a scaling factor in the nonconformity function. The work on normalized conformity scores does not explicitly tailor their approaches to time series.
CP for time series. Gibbs & Candes (2021) and Zaffran et al. (2022) account for shifts in sequential data by continuously adapting an internal coverage target. Adaption-based approaches like these are orthogonal to HopCPT and can serve as an enhancement. Stankeviciūtē et al. (2021) use CP in conjunction with recurrent neural networks in a multi-step prediction setting, assuming that the series of observations is independent. Thus, no weighting of the scores is required. Sun & Yu (2022) introduce CopulaCPTS which applies CP to time series with multivariate targets. They conformalize their prediction based on a copula of the target variables and adapt their calibration set in each step. Jensen et al. (2022) use a bootstrap ensemble to enable CP on time series. NexCP (Foygel Barber et al., 2022) uses exponential decay as the weighting method, arguing that the recent past is more likely to be of the same error distribution. HopCPT can learn this strategy, but does not a priori commit to it. Xu & Xie (2022a) propose EnbPI, which uses quantiles of the  $k$  most recent errors for the prediction interval. Additionally, they introduce a novel leave-one-out ensembling technique. This is specifically geared to settings with scarce data and difficult to use for larger datasets, which is why we do not apply it in our experiments. EnbPI is designed around the notion that near-term errors are often independent and identically distributed and therefore exchangeable. SPCI (Xu & Xie, 2022b) softens this requirement by exploiting the autocorrelative structure with a random forest. However, it re-calculates the random forest model at each time step, which is a computational burden that prohibits its application to large datasets. Our approach relaxes the requirement even further, as we do not assume that the data for the interval computations pertains to the  $k$  most recent errors.
Non-CP methods Beside CP there exist a wide range of approaches for uncertainty-aware time series prediction. For example, Mixture Density Networks (Bishop, 1994) directly estimate the parameters of a mixture of distributions. However, they require a distribution assumption and do not provide any theoretical guarantees. Gaussian Processes (e.g., Zhu et al., 2023; Corani et al., 2021; Sun et al., 2022) model time series by calculating posterior functions based on the samples and a prior but are computationally limited for large and high dimensional datasets.
Continuous Modern Hopfield Networks. MHN are energy-based associative memory networks. They advance conventional Hopfield Networks (Hopfield, 1982) by introducing continuous queries and states via a new energy function. The new energy function leads to exponential storage capacity, while retrieval is possible with a one-step update (Ramsauer et al., 2021). Examples for successful applications of MHN are Widrich et al. (2020); Furst et al. (2022); Dong et al. (2022); Sanchez-Fernandez et al. (2022); Paischer et al. (2022); Schäfl et al. (2022); and Xu et al. (2022). MHN are related to Transformers (Vaswani et al., 2017) as their attention mechanism is closely related to the association mechanism in MHN. In fact, Ramsauer et al. (2021) show that Transformers attention is a special case of the MHN association, at which we arrive when the queries and states are mapped to an associative Hopfield space with the dimensions  $d_{k}$ , and the inverse softmax temperature is set to  $\beta = \frac{1}{\sqrt{d_k}}$ . However, we use the framework of MHN because we want to highlight the associative memory mechanism as HopCPT directly ingests encoded observations. This perspective further allows HopCPT to update the memory for each new observation. For more details, we refer to Appendix H in the supplementary material.
1.2 Setting
Our setting consists of a multivariate time series  $\{(\pmb{x}_t,y_t)\}$ ,  $t = 1,\dots,T$ , with a feature vector  $\pmb{x}_t\in \mathbb{R}^m$ , a target variable  $y_{t}\in \mathbb{R}$ , and a given black-box prediction model  $\hat{\mu}$  that generates a point prediction  $\hat{y}_t = \hat{\mu} (X_t)$ . The input feature matrix  $\pmb{X}_{t + 1}$  can include all previous and current feature vectors  $\{\pmb{x}_i\}_{i = 1}^{t + 1}$ , as well as all previous targets  $\{y_i\}_{i = 1}^t$ . Our goal is to construct a corresponding prediction interval  $\widehat{C}_t^\alpha (\pmb{Z}_{t + 1})$  — a set that includes  $y_{t + 1}$  with at least a specified probability  $1 - \alpha$ . In its basic form,  $\pmb{Z}_{t + 1}$  will only contain  $\hat{y}_{t + 1}$ , but it can also inherit  $\pmb{X}_{t + 1}$  or other useful features. Following Vovk et al. (2005), we define the coverage as
$$
\Pr \left\{Y _ {t + 1} \in \widehat {C} _ {t} ^ {\alpha} \left(\mathbf {Z} _ {t + 1}\right) \right\} \geq 1 - \alpha , \tag {1}
$$
where  $Y_{t + 1}$  is the random variable of the prediction. An infinitely wide prediction interval is  $100\%$  reliable, but not informative of the uncertainty. Thus, CP aims to minimize the width of the prediction interval  $\widehat{C}_t^\alpha$ , while preserving the coverage. A smaller prediction interval is called a more efficient interval (Vovk et al., 2005) and usually evaluated as the mean of the interval width over the prediction period (PI-Width).
Standard split conformal prediction takes a calibration set of size  $n$  which has not been used to train the prediction model  $\hat{\mu}$ . For each data sample, it calculates the so-called non-conformity score (Vovk et al., 2005). In a regression setting, this score often simply corresponds to the absolute error of the prediction (e.g., Foygel Barber et al., 2022). The prediction interval is then calculated based on the empirical  $1 - \alpha$  quantile  $Q_{1 - \alpha}$  of the calibration scores:
$$
\widehat {C} _ {n} ^ {\alpha} \left(\mathbf {Z} _ {t + 1}\right) = \hat {\mu} \left(\mathbf {X} _ {t + 1}\right) \pm \mathrm {Q} _ {1 - \alpha} \left(\left\{\left| y _ {i} - \hat {\mu} \left(\mathbf {X} _ {i}\right) \right| \right\} _ {i = 1} ^ {n}\right). \tag {2}
$$
If the data is exchangeable and  $\hat{\mu}$  treats the data points symmetrically, the errors on the test set follow the distribution from the calibration. Hence, the empirical quantiles on the calibration and test set will be approximately equal and it is guaranteed that the interval provides the desired coverage.
The actual marginal miscoverage  $\alpha^{\star}$  is based on the observed samples of a test set. If the specified miscoverage  $\alpha$  differs from the  $\alpha^{\star}$  in the evaluation, we denote the difference as the coverage gap  $\Delta \mathrm{Cov} = \alpha -\alpha^{\star}$ .
The remainder of this manuscript is structured as follows: In Section 2, we present HopCPT alongside a theoretical motivation and a synthetic example that demonstrates the advantages of the approach. In Section 3, we evaluate the performance against state-of-the-art CP approaches and discuss the results. Section 4 gives our conclusions and provides an outlook on potential future work.
2 HopCPT
HopCPT<sup>1</sup> combines conformal-style quantile estimation with learned similarity-based MHN retrieval.
2.1 Theoretical Motivation
The theoretical motivation for the MHN retrieval stems from Foygel Barber et al. (2022), who introduced CP with weighted quantiles. In the split conformal setting, the according prediction interval is calculated as
$$
\widehat {C} _ {t} ^ {\alpha} \left(\boldsymbol {X} _ {t + 1}\right) = \hat {\mu} \left(\boldsymbol {X} _ {t + 1}\right) \pm \mathrm {Q} _ {1 - \alpha} \left(\sum_ {i = 1} ^ {t} a _ {i} \delta_ {\epsilon_ {i}} + a _ {t + 1} \delta_ {+ \infty}\right), \tag {3}
$$
where  $\hat{\mu}$  represents an existing point prediction model,  $\mathbf{Q}_{\tau}$  is the  $\tau$ -quantile of a distribution, and  $\delta_{\epsilon_i}$  is a point mass at  $|\epsilon_i|$  (i.e., a probability distribution with all its mass at  $|\epsilon_i|$ ), where  $\epsilon_i$  are the errors of the existing prediction model defined by  $\epsilon_i = y_i - \hat{\mu}(\mathbf{X}_i)$ . The normalized weight  $a_i$  of data sample  $i$  is
$$
a _ {i} = \left\{ \begin{array}{l l} \frac {1}{\omega_ {1} + \dots + \omega_ {t} + 1} & \text {i f} i = t + 1, \\ \frac {\omega_ {i}}{\omega_ {1} + \dots + \omega_ {t} + 1} & \text {e l s e}, \end{array} \right. \tag {4}
$$
where  $\omega_{i}$  are the un-normalized weights of the samples. In the case of  $\omega_{1} = \ldots = \omega_{t} = 1$ , this corresponds to standard split CP. Given this framework, Foygel Barber et al. (2022) show that  $\Delta$  Cov can be bounded in a non-exchangeable data setting: Let  $D = ((X_1,Y_1),\dots,(X_{t + 1},Y_{t + 1}))$  be a dataset where the last entry represents the test sample, and  $D^{i}$  be a permutation of  $D$  which exchanges the test sample at  $t + 1$  with the  $i$ -th sample. Then,  $\Delta$  Cov can be bounded from below by the weighted sum of the total variation distances  $d_{\mathrm{TV}}$  between these permutations:
$$
\Delta \operatorname {C o v} \geq - \sum_ {i = 1} ^ {t} a _ {i} \cdot \mathrm {d} _ {\mathrm {T V}} (D, D ^ {i}) \tag {5}
$$
If  $D$  is a composite of multiple regimes and the test sample is from the same regime as the calibration sample  $i$ , then the distance between  $D$  and  $D^i$  is small. Conversely, the distance might be big if the calibration sample is from a different regime. In HopCPT, the MHN association resembles direct estimates of  $a_i$  — dynamically assigning high values to samples from similar regimes.
Appendix B provides an extended theoretical discussion that relates HopCPT to the work of Foygel Barber et al. (2022) and Xu & Xie (2022a), who provide the basis for our theoretical
'https://github.com/ml-jku/HopCPT
analyses of the CP interval. The latter compute individual quantiles for the upper and lower bound of the prediction interval on the errors themselves, in contrast to standard CP which uses only the highest absolute error quantile. This can provide more efficient intervals in case  $E[\epsilon] \neq 0$  within the corresponding error distribution. Applying this principle to HopCPT where the error distribution is conditional to the error regime and assuming that HopCPT can successfully identify these regimes (Appendix: Assumption B.3), we arrive at a conditional asymptotic coverage bound (Appendix: Theorem B.8) and an asymptotic marginal coverage bound (Appendix: Theorem B.9).
2.2 Associative Soft-selection for CP
We use a MHN to identify parts of the time series where the conditional error distribution is similar: For time step  $t + 1$ , we query the memory of the past and look for matching patterns. The MHN then provides an association vector  $\pmb{a}_{t + 1}$  that allows to soft-select the relevant periods of the memory. The selection procedure is analogous to a  $k$ -nearest neighbor classifier for hard selection, but it has the advantage that the similarity measure can be learned. Formally, the soft-selection is defined as:
$$
\boldsymbol {a} _ {t + 1} = \operatorname {s o f t m a x} \left(\beta m \left(\boldsymbol {Z} _ {t + 1}\right) \boldsymbol {W} _ {q} \boldsymbol {W} _ {k} m \left(\boldsymbol {Z} _ {1: t}\right)\right), \tag {6}
$$
where  $m$  is an encoding network (Section 2.3) that transforms the raw time series features of the current step  $Z_{t + 1}$  to the query pattern and the memory  $Z_{1:t}$  to the stored key patterns;  $W_{q}$  and  $W_{k}$  are the learned transformations which are applied before associating the query with the memory;  $\beta$  is a hyperparameter that controls the softmax temperature. As mentioned above, HopCPT uses the softmax to amplify the impact of the data samples that are likely to follow a similar error distribution and to reduce the impact of samples that follow different distributions (see Section 2.1). This error weighting leads not only to more efficient prediction intervals in our experiments, but can also reduce the miscoverage (Section 3.2).
With the soft-selected time steps we can derive the CP interval using the observed errors  $\epsilon$ . Following Xu & Xie (2022a), we use individual quantiles for the upper and lower bound of the prediction interval, calculated from the errors themselves. HopCPT computes the prediction interval  $\widehat{C}_t^\alpha$  for time step  $t + 1$  in the following way:
$$
\widehat {C} _ {t} ^ {\alpha} \left(\boldsymbol {Z} _ {t + 1}\right) = \left[ \hat {\mu} \left(\boldsymbol {X} _ {t + 1}\right) + q \left(\frac {\alpha}{2}, \boldsymbol {Z} _ {t + 1}\right), \hat {\mu} \left(\boldsymbol {X} _ {t + 1}\right) + q \left(1 - \frac {\alpha}{2}, \boldsymbol {Z} _ {t + 1}\right) \right], \tag {7}
$$
where  $q(\tau, \mathbf{Z}_{t+1}) = \mathsf{Q}_{\tau}(E_{t+1})$  and  $E_{t+1}$  is a multiset created by drawing  $n$  times from  $[\epsilon_i]_{i=1}^t$  with corresponding probabilities  $[a_{t+1,i}]_{i=1}^t$ .
2.3 Encoding Network
We embed  $Z_{t}$  using a 2-layer fully connected network  $m^{L}$  with ReLU activations and enhance the representation by temporal encoding features  $z_{T,t}^{\mathrm{time}}$ . The full encoding network can be represented as  $m(Z_{t}) = [m^{L}(Z_{t})||z_{T,t}^{\mathrm{time}}]$ , where  $z_{T,t}^{\mathrm{time}}$  is a simple temporal encoding that makes time dependent notions of similarity learnable (e.g., the windowed approach of EnbPI or the exponentially decaying weighting scheme of NexCP). Specifically, we use  $z_{T,t}^{\mathrm{time}} = \frac{t}{T}$ . In general, we find that our method is not very sensitive to the exact encoding strategy (e.g., number of layers), which is why we kept to a simple choice throughout all experiments.
2.4 Training Procedure
We partition the split conformal calibration data into training and validation sets. Training MHN with quantiles is difficult, which is why we use an auxiliary task: Instead of applying the association mechanism from Equation 6 directly, we use the absolute errors as the value patterns of the MHN. This way, the MHN learns to align errors from time steps with similar regime properties. Intuitively, the observed errors from these time steps should work best to predict the current absolute error. We use the mean squared error as loss function (Equation 8). To allow for efficient training, we simultaneously calculate the association of all  $T$  time steps within a training split with each other. We mask the association from a time step to itself. The resulting association from each step to each step is  $A_{1:T,1:T}$ , and the loss function  $\mathcal{L}$  is
$$
\mathcal {L} = T ^ {- 1} * \left\| \left(\left| \epsilon_ {1: T} \right| - \boldsymbol {A} _ {1: T, 1: T} \mid \epsilon_ {1: T} \right|\right) ^ {2} \| _ {1}. \tag {8}
$$
Figure 2: Synthetic example evaluation. HopCPT has the smallest prediction interval width (PI-width), while maintaining the warranted coverage (i.e., a  $\Delta$  Cov that is positive and close to zero).
This incentivizes the network to learn a representation such that the resulting soft-selection focuses on time steps with a similar error distribution. Alternatively, one could use a loss based on sampling from the softmax. This would correspond more closely to the inference procedure. However, it makes training less efficient because each training step only carries information about a single value of  $\epsilon_{i}$ . In contrast,  $\mathcal{L}$  leads to more sample-efficient training. Choosing  $\mathcal{L}$  assumes that it leads to a retrieval of errors from appropriate regimes instead of mixing unrelated errors. Section 3.2 and Appendix A.3 provide evidence that this holds empirically.
2.5 Synthetic Example
The following synthetic example illustrates the advantages of the HopCPT association mechanism. We model a bivariate time series  $D = \{(x_{t},y_{t})\}_{t = 1}^{T}$ . While  $y_{t}$  serves as the target variable,  $x_{t}$  represents the feature in our prediction. The series is composed of two different regimes: target values are generated by  $y_{t} = 10 + x_{t} + \mathcal{N}(0,\frac{x_{t}}{2})$  or  $y_{t} = 10 + x_{t} + U(-x_{t},x_{t})$ .  $x_{t}$  is constant within a regime ( $x = 3$  and  $x = 21$ , respectively). The regimes alternate. For each regime, we sample the number of time steps from the discrete uniform distribution  $\mathcal{U}(1,25)$ . We create 1,000 time steps and split the data equally into training, calibration, and test sets. We use a ridge regression model as prediction model  $\hat{\mu}$ . HopCPT can identify the time steps of relevant regimes and therefore creates efficient prediction intervals while still preserving the coverage (Figure 2). EnbPI, SPCI, and NexCP focus only on the recent time steps and thus fail to base their intervals on information from the correct regime. Whenever the regime changes from small to high errors, EnbPI propagates the small error signal and therefore loses coverage. Similarly, its prediction intervals for the small error regime are inefficient (Figure 2, row 3). SPCI, NexCP, and standard CP cannot properly select the relevant time steps, either. They do not lose coverage, but produce wide intervals for all time steps (Figure 2, rows 4 and 5). Lastly, if we replace the MHN with a kNN, it can retrieve information from similar regimes. However, its naive retrieval mechanism fails to focus on the informative features because it cannot learn them (Figure 2, row 2).
2.6 Limitations