Title: Conformal Prediction for Time Series with Modern Hopfield Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: time series

uncertainty

prediction interval

conformal prediction

modern hopfield networks
Abstract: To quantify uncertainty, conformal prediction methods are gaining continuously more interest and have already been successfully applied to various domains. However, they are difficult to apply to time series as the autocorrelative structure of time series violates basic assumptions required by conformal prediction. We propose HopCPT, a novel conformal prediction approach for time series that not only copes with temporal structures but leverages them. We show that our approach is theoretically well justified for time series where temporal dependencies are present. In experiments, we demonstrate that our new approach outperforms state-of-the-art conformal prediction methods on multiple real-world time series datasets from four different domains.

Conformal Prediction for Time Series with Modern Hopfield Networks

Abstract

To quantify uncertainty, conformal prediction methods are gaining continuously more interest and have already been successfully applied to various domains. However, they are difficult to apply to time series as the autocorrelative structure of time series violates basic assumptions required by conformal prediction. We propose HopCPT, a novel conformal prediction approach for time series that not only copes with temporal structures but leverages them. We show that our approach is theoretically well justified for time series where temporal dependencies are present. In experiments, we demonstrate that our new approach outperforms stateof-the-art conformal prediction methods on multiple real-world time series datasets from four different domains.

1 Introduction

Uncertainty estimates are imperative to make actionable predictions for complex time-dependent systems (e.g., Gneiting & Katzfuss, 2014; Zhu & Laptev, 2017). This is particularly evident for environmental phenomena such as flood forecasting (e.g., Krzysztofowicz, 2001), since they exhibit pronounced seasonality. Conformal Prediction (CP, Vovk et al., 1999) provides uncertainty estimates based on prediction intervals. It achieves finite-sample marginal coverage with almost no assumptions, except that the data is exchangeable (Vovk et al., 2005; Vovk, 2012). However, CP for time series is not trivial because temporal dependencies often violate the exchangeability assumption.

HopCPT. For time series models that predict a given system, the errors are typically specific to the input describing the current situation. To apply CP, HopCPT uses continuous Modern Hopfield Networks (MHNs). The MHN uses large weights for past situations that were similar to the current one and weights close to zero for past situations that were dissimilar to the current one (Figure 1). The CP procedure uses the weighted errors to construct strong uncertainty estimates close to the desired coverage level. This exploits that similar situations (which we call a regime) tend to follow the same error distribution. HopCPT achieves new state-of-the-art efficiency, even under non-exchangeability and on large datasets.

Our main contributions are:

We propose HopCPT, a CP method for time series, a domain where CP struggled so far.

We introduce the concept of error regimes to CP, which softens the excitability requirements and enables efficient CP for time series.

HopCPT uses MHN for a similarity-based sample reweighting. In contrast to existing approaches, HopCPT can learn from large datasets and predict intervals at arbitrary coverage levels without retraining.

HopCPT achieves state-of-the-art results for conformal time series prediction tasks from various real-world applications.

value

value

MHN

t

CP

prediction

observation

prediction

CP interval

observation

HopCPT is the first algorithm with coverage guarantees that was applied to hydrological prediction applications - a domain where uncertainty plays a key role in tasks such as flood forecasting and hydropower management.

1.1 Related Work

Regimes. In a world with non-linear dynamics, different environmental conditions lead to different error characteristics of models that predict based on these conditions. If we do not account for these different conditions, temporal changes may lead to unnecessarily large prediction intervals, i.e., to high uncertainty. For example, solar energy production is high and stable on a sunny day, fluctuates during cloudy days, and is zero at night. Often, the current environmental condition was already observed at previous points in time. The error at these time steps is therefore assumed to have the same distribution as the current error. Following Quandt (1958) and Hamilton (1990), we call the sets of time steps with similar environmental conditions regimes . Although conditional CP is in general impossible (Foygel Barber et al., 2021), we show that conditioning on such regimes can lead to better prediction intervals while preserving the specified coverage. Hamilton (1990) models time series regimes as a discrete Markov process and conditions a classical autoregressive model on the regime states. Sanquer et al. (2012) use a smooth transition approach to model multi-regime time series. Tajeuna et al. (2021) propose an approach to discover and model regime shifts in an ecosystem that comprises multiple time series. Further, Masserano et al. (2022) handle distribution shifts by retraining a forecasting model with training data from a non-uniform adaptive sampling. Although these approaches are not in a CP setting, their work is similar in spirit, as they also follow the general idea to condition on parts of the time series with similar regimes.

CP and extensions. For thorough introductions to CP, we refer the reader to the foundational work of Vovk et al. (1999) and a recent introductory paper by Angelopoulos & Bates (2021). There exist a variety of extensions for CP that go 'beyond exchangeability' (Vovk et al., 2005). For example, Papadopoulos & Haralambous (2011) apply CP to a nearest neighbor regression setting, Teng et al. (2022) apply CP to the feature space of models, Angelopoulos et al. (2020) use CP to generate uncertainty sets for image classification tasks, and Toccaceli et al. (2017) use a labelconditional variant to apply CP to biological activity prediction. Of specific interest to us is the research regarding non-exchangeable data of Tibshirani et al. (2019) and Foygel Barber et al. (2022). Both handle potential shifts between the calibration and test set by reweighting the data points. Tibshirani et al. (2019) restrict themselves to settings with full knowledge about the change in distribution; Foygel Barber et al. (2022) rely on fixed weights. In our work, we refrain from this assumption because such information is typically not available in time series prediction. Another important research direction is the work on normalized conformity scores (see Fontana et al., 2023, and references therein). In this setting, the goal is to adapt the conformal bounds through a scaling factor in the nonconformity function. The work on normalized conformity scores does not explicitly tailor their approaches to time series.

CP for time series. Gibbs & Candes (2021) and Zaffran et al. (2022) account for shifts in sequential data by continuously adapting an internal coverage target. Adaption-based approaches like these are orthogonal to HopCPT and can serve as an enhancement. Stankeviˇ ci¯ ut˙ e et al. (2021) use CP in conjunction with recurrent neural networks in a multi-step prediction setting, assuming that the series of observations is independent. Thus, no weighting of the scores is required. Sun & Yu (2022) introduce CopulaCPTS which applies CP to time series with multivariate targets. They conformalize their prediction based on a copula of the target variables and adapt their calibration set in each step. Jensen et al. (2022) use a bootstrap ensemble to enable CP on time series. NexCP (Foygel Barber et al., 2022) uses exponential decay as the weighting method, arguing that the recent past is more likely to be of the same error distribution. HopCPT can learn this strategy, but does not a priori commit to it. Xu & Xie (2022a) propose EnbPI, which uses quantiles of the k most recent errors for the prediction interval. Additionally, they introduce a novel leave-one-out ensembling technique. This is specifically geared to settings with scarce data and difficult to use for larger datasets, which is why we do not apply it in our experiments. EnbPI is designed around the notion that near-term errors are often independent and identically distributed and therefore exchangeable. SPCI (Xu & Xie, 2022b) softens this requirement by exploiting the autocorrelative structure with a random forest. However, it re-calculates the random forest model at each time step, which is a computational burden that prohibits its application to large datasets. Our approach relaxes the requirement even further, as we do not assume that the data for the interval computations pertains to the k most recent errors.

Non-CP methods Beside CP there exist a wide range of approaches for uncertainty-aware time series prediction. For example, Mixture Density Networks (Bishop, 1994) directly estimate the parameters of a mixture of distributions. However, they require a distribution assumption and do not provide any theoretical guarantees. Gaussian Processes (e.g., Zhu et al., 2023; Corani et al., 2021; Sun et al., 2022) model time series by calculating posterior functions based on the samples and a prior but are computationally limited for large and high dimensional datasets.

Continuous Modern Hopfield Networks. MHNare energy-based associative memory networks. They advance conventional Hopfield Networks (Hopfield, 1982) by introducing continuous queries and states via a new energy function. The new energy function leads to exponential storage capacity, while retrieval is possible with a one-step update (Ramsauer et al., 2021). Examples for successful applications of MHN are Widrich et al. (2020); F¨ urst et al. (2022); Dong et al. (2022); SanchezFernandez et al. (2022); Paischer et al. (2022); Sch¨ afl et al. (2022); and Xu et al. (2022). MHN are related to Transformers (Vaswani et al., 2017) as their attention mechanism is closely related to the association mechanism in MHN. In fact, Ramsauer et al. (2021) show that Transformers attention is a special case of the MHN association, at which we arrive when the queries and states are mapped to an associative Hopfield space with the dimensions d k , and the inverse softmax temperature is set to β = 1 √ d k . However, we use the framework of MHN because we want to highlight the associative memory mechanism as HopCPT directly ingests encoded observations. This perspective further allows HopCPT to update the memory for each new observation. For more details, we refer to Appendix H in the supplementary material.

1.2 Setting

Our setting consists of a multivariate time series { ( x t , y t ) } , t = 1 , . . . , T , with a feature vector x t ∈ R m , a target variable y t ∈ R , and a given black-box prediction model ˆ µ that generates a point prediction ˆ y t = ˆ µ ( X t ) . The input feature matrix X t +1 can include all previous and current feature vectors { x i } t +1 i =1 , as well as all previous targets { y i } t i =1 . Our goal is to construct a corresponding prediction interval ̂ C α t ( Z t +1 ) -aset that includes y t +1 with at least a specified probability 1 -α . In its basic form, Z t +1 will only contain ˆ y t +1 , but it can also inherit X t +1 or other useful features. Following Vovk et al. (2005), we define the coverage as

where Y t +1 is the random variable of the prediction. An infinitely wide prediction interval is 100% reliable, but not informative of the uncertainty. Thus, CP aims to minimize the width of the prediction interval ̂ C α t , while preserving the coverage. A smaller prediction interval is called a more efficient interval (V ovk et al., 2005) and usually evaluated as the mean of the interval width over the prediction period ( PI-Width ).

Standard split conformal prediction takes a calibration set of size n which has not been used to train the prediction model ˆ µ . For each data sample, it calculates the so-called non-conformity score (Vovk et al., 2005). In a regression setting, this score often simply corresponds to the absolute error of the prediction (e.g., Foygel Barber et al., 2022). The prediction interval is then calculated based on the empirical 1 -α quantile Q 1 -α of the calibration scores:

If the data is exchangeable and ˆ µ treats the data points symmetrically, the errors on the test set follow the distribution from the calibration. Hence, the empirical quantiles on the calibration and test set will be approximately equal and it is guaranteed that the interval provides the desired coverage.

The actual marginal miscoverage α ⋆ is based on the observed samples of a test set. If the specified miscoverage α differs from the α ⋆ in the evaluation, we denote the difference as the coverage gap ∆ Cov = α -α ⋆ .

The remainder of this manuscript is structured as follows: In Section 2, we present HopCPT alongside a theoretical motivation and a synthetic example that demonstrates the advantages of the approach. In Section 3, we evaluate the performance against state-of-the-art CP approaches and discuss the results. Section 4 gives our conclusions and provides an outlook on potential future work.

2 HopCPT

HopCPT 1 combines conformal-style quantile estimation with learned similarity-based MHN retrieval.

2.1 Theoretical Motivation

The theoretical motivation for the MHN retrieval stems from Foygel Barber et al. (2022), who introduced CP with weighted quantiles. In the split conformal setting, the according prediction interval is calculated as

where ˆ µ represents an existing point prediction model, Q τ is the τ -quantile of a distribution, and δ ϵ i is a point mass at | ϵ i | (i.e., a probability distribution with all its mass at | ϵ i | ), where ϵ i are the errors of the existing prediction model defined by ϵ i = y i -ˆ µ ( X i ) . The normalized weight a i of data sample i is

where ω i are the un-normalized weights of the samples. In the case of ω 1 = . . . = ω t = 1 , this corresponds to standard split CP. Given this framework, Foygel Barber et al. (2022) show that ∆ Cov can be bounded in a non-exchangeable data setting: Let D = (( X 1 , Y 1 ) , . . . , ( X t +1 , Y t +1 )) be a dataset where the last entry represents the test sample, and D i be a permutation of D which exchanges the test sample at t +1 with the i -th sample. Then, ∆ Cov can be bounded from below by the weighted sum of the total variation distances d TV between these permutations:

If D is a composite of multiple regimes and the test sample is from the same regime as the calibration sample i , then the distance between D and D i is small. Conversely, the distance might be big if the calibration sample is from a different regime. In HopCPT, the MHN association resembles direct estimates of a i -dynamically assigning high values to samples from similar regimes.

Appendix B provides an extended theoretical discussion that relates HopCPT to the work of Foygel Barber et al. (2022) and Xu & Xie (2022a), who provide the basis for our theoretical

̸

analyses of the CP interval. The latter compute individual quantiles for the upper and lower bound of the prediction interval on the errors themselves, in contrast to standard CP which uses only the highest absolute error quantile. This can provide more efficient intervals in case E [ ϵ ] = 0 within the corresponding error distribution. Applying this principle to HopCPT where the error distribution is conditional to the error regime and assuming that HopCPT can successfully identify these regimes (Appendix: Assumption B.3), we arrive at a conditional asymptotic coverage bound (Appendix: Theorem B.8) and an asymptotic marginal coverage bound (Appendix: Theorem B.9).

2.2 Associative Soft-selection for CP

We use a MHN to identify parts of the time series where the conditional error distribution is similar: For time step t +1 , we query the memory of the past and look for matching patterns. The MHN then provides an association vector a t +1 that allows to soft-select the relevant periods of the memory. The selection procedure is analogous to a k -nearest neighbor classifier for hard selection, but it has the advantage that the similarity measure can be learned. Formally, the soft-selection is defined as:

where m is an encoding network (Section 2.3) that transforms the raw time series features of the current step Z t +1 to the query pattern and the memory Z 1: t to the stored key patterns; W q and W k are the learned transformations which are applied before associating the query with the memory; β is a hyperparameter that controls the softmax temperature. As mentioned above, HopCPT uses the softmax to amplify the impact of the data samples that are likely to follow a similar error distribution and to reduce the impact of samples that follow different distributions (see Section 2.1). This error weighting leads not only to more efficient prediction intervals in our experiments, but can also reduce the miscoverage (Section 3.2).

With the soft-selected time steps we can derive the CP interval using the observed errors ϵ . Following Xu & Xie (2022a), we use individual quantiles for the upper and lower bound of the prediction interval, calculated from the errors themselves. HopCPT computes the prediction interval ̂ C α t for time step t +1 in the following way:

where q ( τ, Z t +1 ) = Q τ ( E t +1 ) and E t +1 is a multiset created by drawing n times from [ ϵ i ] t i =1 with corresponding probabilities [ a t +1 ,i ] t i =1 .

2.3 Encoding Network

We embed Z t using a 2-layer fully connected network m L with ReLU activations and enhance the representation by temporal encoding features z time T,t . The full encoding network can be represented as m ( Z t ) = [ m L ( Z t ) || z time T,t ] , where z time T,t is a simple temporal encoding that makes time dependent notions of similarity learnable (e.g., the windowed approach of EnbPI or the exponentially decaying weighting scheme of NexCP). Specifically, we use z time T,t = t T . In general, we find that our method is not very sensitive to the exact encoding strategy (e.g., number of layers), which is why we kept to a simple choice throughout all experiments.

2.4 Training Procedure

We partition the split conformal calibration data into training and validation sets. Training MHN with quantiles is difficult, which is why we use an auxiliary task: Instead of applying the association mechanism from Equation 6 directly, we use the absolute errors as the value patterns of the MHN. This way, the MHN learns to align errors from time steps with similar regime properties. Intuitively, the observed errors from these time steps should work best to predict the current absolute error. We use the mean squared error as loss function (Equation 8). To allow for efficient training, we simultaneously calculate the association of all T time steps within a training split with each other. We mask the association from a time step to itself. The resulting association from each step to each step is A 1: T, 1: T , and the loss function L is

0

20

40

Value

HopCPT  (

Cov: 0.005  PI-Width: 20.16)

KNN  (

Cov:

0.001  PI-Width: 25.51)

0

20

40

Value

EnbPI  (

Cov:

0.106  PI-Width: 21.17)

SPCI  (

Cov: 0.013  PI-Width: 32.23)

0

20

40

60

80

100

t

0

20

40

Value

NexCP  (

Cov: 0.006  PI-Width: 33.6)

0

20

40

60

80

100

t

CP  (

Cov: 0.0424  PI-Width: 35.28)

yt

yt

This incentivizes the network to learn a representation such that the resulting soft-selection focuses on time steps with a similar error distribution. Alternatively, one could use a loss based on sampling from the softmax. This would correspond more closely to the inference procedure. However, it makes training less efficient because each training step only carries information about a single value of ϵ i . In contrast, L leads to more sample-efficient training. Choosing L assumes that it leads to a retrieval of errors from appropriate regimes instead of mixing unrelated errors. Section 3.2 and Appendix A.3 provide evidence that this holds empirically.

2.5 Synthetic Example

The following synthetic example illustrates the advantages of the HopCPT association mechanism. We model a bivariate time series D = { ( x t , y t ) } T t =1 . While y t serves as the target variable, x t represents the feature in our prediction. The series is composed of two different regimes: target values are generated by y t = 10 + x t + N (0 , x t 2 ) or y t = 10 + x t + U ( -x t , x t ) . x t is constant within a regime ( x = 3 and x = 21 , respectively). The regimes alternate. For each regime, we sample the number of time steps from the discrete uniform distribution U (1 , 25) . We create 1,000 time steps and split the data equally into training, calibration, and test sets. We use a ridge regression model as prediction model ˆ µ . HopCPT can identify the time steps of relevant regimes and therefore creates efficient prediction intervals while still preserving the coverage (Figure 2). EnbPI, SPCI, and NexCP focus only on the recent time steps and thus fail to base their intervals on information from the correct regime. Whenever the regime changes from small to high errors, EnbPI propagates the small error signal and therefore loses coverage. Similarly, its prediction intervals for the small error regime are inefficient (Figure 2, row 3). SPCI, NexCP, and standard CP cannot properly select the relevant time steps, either. They do not lose coverage, but produce wide intervals for all time steps (Figure 2, rows 4 and 5). Lastly, if we replace the MHN with a kNN, it can retrieve information from similar regimes. However, its naive retrieval mechanism fails to focus on the informative features because it cannot learn them (Figure 2, row 2).

2.6 Limitations

HopCPT builds upon the formal guarantees of CP, but it is able to relax the data exchangeability assumption of CP which are problematic for time series data. That said, HopCPT still relies on assumptions on how it learns to identify the respective error regimes and exchangablity within regimes (see Section 2.1 and Appendix B). Our extensive empirical evaluation suggests that these assumptions hold in practice. HopCPT is generally well-suited for very long time series because the memory requirement in the inference scales only linearly with the memory size. Nevertheless, for extremely large datasets it can be the case that not all historical time steps may be kept in the Hopfield memory

anymore. HopCPT can, similar to existing methods, disregard time steps by removing the oldest entries from the memory, or alternatively use a sub-sampling strategy. On the other hand, for datasets with very scarce data it might be difficult to learn a useful embedding of the time steps. In that case, it could be better to use kNN rather than learning the similarity with MHN (Appendix D). Additional considerations regarding the potential social impact of the method are summarized in Appendix I.

3 Experiments

This section provides a comparative evaluation of HopCPT and analyzes its association mechanism.

3.1 Setup

Datasets. We use datasets from four different domains: (a) Three solar radiation datasets from the US National Solar Radiation Database (Sengupta et al., 2018). The smallest one consists of 8 time series from different locations over a period of 84 days. This dataset is also used in Xu & Xie (2022a,b). In addition, we evaluate on a 1-year and a 3-year dataset, with 50 time series each. (b) An air quality dataset from Beijing, China (Zhang et al., 2017). It consists of 12 time series, each from a different measurement station, over a period of 4 years. The dataset has two prediction targets, the PM10 (as in Xu & Xie, 2022a,b) and PM2.5 concentrations, which we evaluate separately. (c) Sap flow 2 measurements from the Sapfluxnet data project (Poyatos et al., 2021). Since the individual measurement series are considerably heterogeneous in length, we use a subset of 24 time series, each with between 15,000 and 20,000 data points and varying sampling rates. (d) Streamflow, a dataset of water flow measurements and corresponding meteorologic observations from 531 rivers across the continental United States (Newman et al., 2015; Addor et al., 2017). The measurements span 28 years at a daily time scale. For more detailed information about the datasets see Appendix C.

Prediction models. Weuse four prediction models for the solar radiation, the air quality, and the sap flux datasets to ensure that our results generalize: a random forest, a LightGBM, a ridge regression, and a Long Short-Term Memory (LSTM) (LSTM; Hochreiter & Schmidhuber, 1997) model. For the former three models we follow the related work (Xu & Xie, 2022a,b; Foygel Barber et al., 2022) and train a separate prediction model for each individual time series. The random forest and LightGBM models are implemented with the darts library (Herzen et al., 2022), the ridge regression model with sklearn (Pedregosa et al., 2011). For the LSTM model, we instead train a global model on all time series of a dataset, as is standard for state-of-the-art deep learning models (e.g., Oreshkin et al., 2020; Salinas et al., 2020; Smyl, 2020). The LSTM is implemented with PyTorch (Paszke et al., 2019). For the streamflow dataset, we deviate from this scheme and instead only use the state-of-the-art model, which is also an LSTM network (Kratzert et al., 2021, see Appendix C).

Compared approaches. We compare HopCPT to different state-of-the-art CP approaches for time series data: EnbPI (Xu & Xie, 2022a), SPCI (Xu & Xie, 2022b), NexCP (Foygel Barber et al., 2022), CopulaCPTS (Sun & Yu, 2022), and AdaptiveCI 3 (Gibbs & Candes, 2021). In addition, the results of standard split CP (CP) serve as a baseline, which, for the LSTM base predictor, corresponds to CFRNN (Stankeviˇ ci¯ ut˙ e et al., 2021) in our setting (one-step, univariate target variable). Appendix A.1 describes our hyperparameter search for each method. For SPCI, an adaption of the original algorithm was necessary to provide scalability to larger datasets. Appendix A.2 contains more details and an empirical justification. Appendix G evaluates the addition of AdaptiveCI (Gibbs & Candes, 2021) as an enhancement to HopCPT and the other time series CP methods. Appendix A.3 gives a comparison to non-CP methods. Lastly, Appendix D presents a supplemental comparison to kNN that shows the superiority of the learned similarity representation in HopCPT.

Metrics. In our analyses, we compute ∆ Cov, PI-Width, and the Winkler score (Winkler, 1972) per time series and miscoverage level. The Winkler score jointly elicitates miscoverage and interval width in a single metric:

The score is calculated per time step t and miscoverage level α . It corresponds to the interval width IW α t = ̂ C α,u t -̂ C α,l t whenever the observed value y t is between the upper bound ̂ C α,u t ( Z t +1 ) and the lower bound ̂ C α,l t ( Z t +1 ) of ̂ C α t ( Z t +1 ) . If y t +1 is outside these bounds, a penalty is added to the interval width. We evaluate the mean Winkler score over all time steps.

Werepeated each experiment with 12 different seeds. For brevity, we only show the mean performance of one dataset per domain for α = 0 . 1 in the main paper (which is the most commonly reported value in the CP literature; e.g., Xu & Xie, 2022b; Foygel Barber et al., 2022; Gibbs & Candes, 2021). Appendix A.3 presents additional results for all datasets and more α levels.

3.2 Results & Discussion

HopCPT has the most efficient prediction intervals for each domain - with only one exception (Table 1; significance tested with a Mann-Whitney U test at p < 0 . 005 ) for the evaluated miscoverage level ( α = 0 . 1 ). In multiple experiments (Solar (3Y), Solar (1Y)), the HopCPT prediction intervals are less than half as wide as those of the approach with the second-smallest PI-Width. The secondmost efficient intervals are predicted most often by SPCI. This ranking also holds for the Winkler score, where HopCPT achieves the best (i.e., lowest) Winkler scores and SPCI ranks second in most experiments. Notably, these results reflect the increasing requirements posed by each uncertainty estimation method on the dataset (Section 1.1).

Furthermore, HopCPT outperforms the other methods regarding both Winkler score and PI-Width when we evaluate over additional datasets and at different miscoverage levels (see Appendix A.3). HopCPT is the best-performing approach in the vast majority of cases. The variation in size of the different solar datasets (results for Solar (1Y) and Solar (3M) in Appendix A.3) shows that HopCPT especially increases its lead when more data is available. We hypothesize that this is because the MHNcan learn more generalizable retrieval patterns with more data. However, data scarcity is not a limitation of similarity-based CP in general. In fact, a simplified and almost parameter-free variation of HopCPT, which replaces the MHN with kNN retrieval, performs best on the smallest solar dataset, as we demonstrate in Appendix D.

Most approaches have a coverage gap close to zero in almost all experiments - they approximately achieve the specified marginal coverage level. This is also reflected by the fact that the ranking of Winkler scores and PI-Widths agree in most evaluations. Appendix A.4 provides a supplementary analysis of the local coverage. The results for non-CP methods show a different picture (see Appendix A.3 and Table 7). Here, the non-CP models most often exhibit very narrow prediction intervals at the cost of high miscoverage.

Interestingly, standard CP also achieves good coverage for all experiments at the cost of inefficient prediction intervals. There is one notable exception: for the Sap flow dataset with ridge regression, we find ∆ Cov = -0 . 358 . In this case, we argue that the bad performance of standard CP is driven by a strong violation of the (required) exchangeability assumption. Specifically, a trend in the errors leads to a distribution shift over time (as illustrated in Appendix A.3, Figure 4). HopCPT and NexCP handle this shift without substantial loss in coverage. The inferior coverage of SPCI is likely influenced by the modification to larger datasets (see Appendix A.2).

Performance gaps between the approaches differ considerably across datasets, but are generally consistent across prediction models. The biggest differences between the best and worst methods exist in Solar (3Y) and Sap flow, likely due to the distinctive regimes in these datasets. The smallest (but still significant) differences are visible in the Streamflow data. On this dataset, we also evaluated the state-of-the-art non-CP uncertainty model in the domain (Klotz et al., 2022, see Appendix A.3) and found that HopCPT outperforms it with respect to efficiency.

t

0

200

400

Value

End of calibration period

yt

yt

low

high

Association weight

In terms of computational resources, SPCI is most demanding followed by HopCPT, as both methods require a learning step in contrast to the other methods. A detailed overview can be found in Appendix A.6.

To assess whether HopCPT learns meaningful associations within regimes, we conducted a qualitative study on the Solar (3Y) dataset. Figure 3 shows that HopCPT retrieves the most highly weighted errors from time steps with similar regimes. The illustrated weighting at the time step with a low prediction value retrieves previous time steps which are also in low-valued regimes. Similarly, Figure 5 (Appendix A.3) suggests that the learned distinction corresponds to the error regimes, which is crucial for HopCPT.

4 Conclusions

We have introduced HopCPT, a novel CP approach for time series tasks. HopCPT uses continuous Modern Hopfield Networks to construct prediction intervals based on previously seen events with similar error distribution regimes. We exploit that similar features lead to similar errors. Associating features with errors identifies regimes with similar error distributions. HopCPT learns this association in the Modern Hopfield Network, which dynamically adjusts its focus on stored previous features according to the current regime.

Our experiments with established and novel datasets show that HopCPT achieves state of the art: It generates more efficient prediction intervals than existing CP methods and approximately preserves coverage, even in non-exchangeable scenarios like time series. HopCPT comes with formal guarantees within the CP framework for uncertainty estimation in real-world applications such as streamflow prediction. Furthermore, HopCPT scales well to large datasets, provides multiple coverage levels after calibration, and shares information across individual time series within a dataset.

Future work comprises: (a) Drawing from multiple time series during the inference phase. HopCPT is already trained on the whole dataset at once, but it might be advantageous to leverage more information during inference as well. (b) Investigating direct training objectives from which learningbased CP might benefit. (c) Using HopCPT beyond time series, as non-exchangeability is also an issue in other domains which limits the applicability of existing CP methods.

References

Abbott, L. F. and Arian, Y. Storage capacity of generalized networks. Physical Review A , 36: 5091-5094, 1987. doi: 10.1103/PhysRevA.36.5091.

Addor, N., Newman, A. J., Mizukami, N., and Clark, M. P. The CAMELS data set: catchment attributes and meteorology for large-sample studies. Hydrology and Earth System Sciences (HESS) , 21(10):5293-5313, 2017.

Angelopoulos, A., Bates, S., Malik, J., and Jordan, M. I. Uncertainty sets for image classifiers using conformal prediction. arXiv preprint arXiv:2009.14193 , 2020.

Angelopoulos, A. N. and Bates, S. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511 , 2021.

Baldi, P. and Venkatesh, S. S. Number of stable points for spin-glasses and neural networks of higher orders. Physical Review Letters , 58:913-916, 1987. doi: 10.1103/PhysRevLett.58.913.

Bishop, C. M. Mixture density networks. Technical report, Neural Computing Research Group, 1994.

Caputo, B. and Niemann, H. Storage capacity of kernel associative memories. In Proceedings of the International Conference on Artificial Neural Networks (ICANN) , pp. 51-56, Berlin, Heidelberg, 2002. Springer-Verlag.

Chen, H. H., Lee, Y. C., Sun, G. Z., Lee, H. Y., Maxwell, T., and Giles, C. L. High order correlation model for associative memory. AIP Conference Proceedings , 151(1):86-99, 1986. doi: 10.1063/1. 36224.

Corani, G., Benavoli, A., and Zaffalon, M. Time series forecasting with gaussian processes needs priors. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pp. 103-117. Springer, 2021.

Dong, Z., Chen, Z., and Wang, Q. Retrosynthesis prediction based on graph relation network. In 2022 15th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI) , pp. 1-5. IEEE, 2022.

Fontana, M., Zeni, G., and Vantini, S. Conformal prediction: a unified review of theory and new challenges. Bernoulli , 29(1):1-23, 2023.

Fox, M. and Rubin, H. Admissibility of quantile estimates of a single location parameter. The Annals of Mathematical Statistics , pp. 1019-1030, 1964.

Foygel Barber, R., Candes, E. J., Ramdas, A., and Tibshirani, R. J. The limits of distribution-free conditional predictive inference. Information and Inference: A Journal of the IMA , 10(2):455-482, 2021.

Foygel Barber, R., Candes, E. J., Ramdas, A., and Tibshirani, R. J. Conformal prediction beyond exchangeability. arXiv preprint arXiv:2202.13415 , 2022.

F¨ urst, A., Rumetshofer, E., Lehner, J., Tran, V . T., Tang, F., Ramsauer, H., Kreil, D. P., Kopp, M. K., Klambauer, G., Bitto-Nemling, A., and Hochreiter, S. CLOOB: Modern Hopfield Networks with infoLOOB outperform CLIP. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems , 2022.

Gal, Y. and Ghahramani, Z. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning , pp. 1050-1059. PMLR, 2016.

Gardner, E. Multiconnected neural network models. Journal of Physics A , 20(11):3453-3464, 1987. doi: 10.1088/0305-4470/20/11/046.

Gibbs, I. and Candes, E. J. Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems , 34:1660-1672, 2021.

Gneiting, T. and Katzfuss, M. Probabilistic forecasting. Annual Review of Statistics and Its Application , 1(1):125-151, 2014. doi: 10.1146/annurev-statistics-062713-085831.

Hamilton, J. D. Analysis of time series subject to changes in regime. Journal of econometrics , 45 (1-2):39-70, 1990.

Herzen, J., L¨ assig, F., Piazzetta, S. G., Neuer, T., Tafti, L., Raille, G., Van Pottelbergh, T., Pasieka, M., Skrodzki, A., Huguenin, N., et al. Darts: User-friendly modern machine learning for time series. Journal of Machine Learning Research , 23(124):1-6, 2022.

Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural Comput. , 9(8):1735-1780, 1997.

Hopfield, J. J. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences , 79(8):2554-2558, 1982.

Hopfield, J. J. Neurons with graded response have collective computational properties like those of two-state neurons. Proceedings of the National Academy of Sciences , 81(10):3088-3092, 1984. doi: 10.1073/pnas.81.10.3088.

Horn, D. and Usher, M. Capacities of multiconnected memory models. Journal of Physics France , 49(3):389-395, 1988. doi: 10.1051/jphys:01988004903038900.

Jensen, V., Bianchi, F. M., and Anfinsen, S. N. Ensemble conformalized quantile regression for probabilistic time series forecasting. arXiv preprint arXiv:2202.08756 , 2022.

Klotz, D., Kratzert, F., Gauch, M., Keefe Sampson, A., Brandstetter, J., Klambauer, G., Hochreiter, S., and Nearing, G. Uncertainty estimation with deep learning for rainfall-runoff modeling. Hydrology and Earth System Sciences , 26(6):1673-1693, 2022. doi: 10.5194/hess-26-1673-2022.

Kratzert, F., Klotz, D., Shalev, G., Klambauer, G., Hochreiter, S., and Nearing, G. Towards learning universal, regional, and local hydrological behaviors via machine learning applied to large-sample datasets. Hydrology and Earth System Sciences , 23(12):5089-5110, 2019. doi: 10.5194/hess-23-5089-2019.

Kratzert, F., Klotz, D., Hochreiter, S., and Nearing, G. S. A note on leveraging synergy in multiple meteorological data sets with deep learning for rainfall-runoff modeling. Hydrology and Earth System Sciences , 25(5):2685-2703, 2021.

Kratzert, F., Gauch, M., Nearing, G., and Klotz, D. NeuralHydrology - a Python library for deep learning research in hydrology. Journal of Open Source Software , 7(71):4050, 2022. doi: 10.21105/joss.04050.

Krotov, D. and Hopfield, J. J. Dense associative memory for pattern recognition. In Lee, D. D., Sugiyama, M., Luxburg, U. V., Guyon, I., and Garnett, R. (eds.), Advances in Neural Information Processing Systems , pp. 1172-1180. Curran Associates, Inc., 2016.

Krzysztofowicz, R. The case for probabilistic forecasting in hydrology. Journal of hydrology , 249 (1-4):2-9, 2001.

Lei, J. and Wasserman, L. A. Distribution-free prediction bands for non-parametric regression. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 76, 2014.

Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019.

Masserano, L., Rangapuram, S. S., Kapoor, S., Nirwan, R. S., Park, Y., and Bohlke-Schneider, M. Adaptive sampling for probabilistic forecasting under distribution shift. In NeurIPS 2022 Workshop on Distribution Shifts: Connecting Methods and Applications , 2022.

Maurer, E. P., Wood, A. W., Adam, J. C., Lettenmaier, D. P., and Nijssen, B. A long-term hydrologically based dataset of land surface fluxes and states for the conterminous United States. Journal of climate , 15(22):3237-3251, 2002.

Newman, A. J., Clark, M. P., Sampson, K., Wood, A., Hay, L. E., Bock, A., Viger, R. J., Blodgett, D., Brekke, L., Arnold, J. R., Hopson, T., and Duan, Q. Development of a large-sample watershedscale hydrometeorological data set for the contiguous USA: data set characteristics and assessment of regional variability in hydrologic model performance. Hydrology and Earth System Sciences , 19(1):209-223, 2015. doi: 10.5194/hess-19-209-2015.

Newman, A. J., Mizukami, N., Clark, M. P., Wood, A. W., Nijssen, B., and Nearing, G. Benchmarking of a physically based hydrologic model. Journal of Hydrometeorology , 18(8):2215-2225, 2017. doi: 10.1175/JHM-D-16-0284.1.

Oreshkin, B. N., Carpov, D., Chapados, N., and Bengio, Y. N-beats: Neural basis expansion analysis for interpretable time series forecasting. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id=r1ecqn4YwB .

Paischer, F., Adler, T., Patil, V ., Bitto-Nemling, A., Holzleitner, M., Lehner, S., Eghbal-zadeh, H., and Hochreiter, S. History compression via language models in reinforcement learning. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pp. 17156-17185. PMLR, 17-23 Jul 2022.

Papadopoulos, H. and Haralambous, H. Reliable prediction intervals with regression neural networks. Neural Networks , 24(8):842-851, 2011.

Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

Pinsker, M. S. Information and information stability of random variables and processes . Holden-Day, 1964.

Poyatos, R., Granda, V., Flo, V., Adams, M. A., Adorj´ an, B., Aguad´ e, D., Aidar, M. P., Allen, S., Alvarado-Barrientos, M. S., Anderson-Teixeira, K. J., et al. Global transpiration data from sap flow measurements: the sapfluxnet database. Earth system science data , 13(6):2607-2649, 2021.

Psaltis, D. and Cheol, H. P. Nonlinear discriminant functions and associative memories. AIP Conference Proceedings , 151(1):370-375, 1986. doi: 10.1063/1.36241.

Quandt, R. E. The estimation of the parameters of a linear regression system obeying two separate regimes. Journal of the american statistical association , 53(284):873-880, 1958.

Ramsauer, H., Sch¨ afl, B., Lehner, J., Seidl, P., Widrich, M., Gruber, L., Holzleitner, M., Pavlovi´ c, M., Sandve, G. K., Greiff, V., Kreil, D., Kopp, M., Klambauer, G., Brandstetter, J., and Hochreiter, S. Hopfield networks is all you need. In 9th International Conference on Learning Representations (ICLR) , 2021.

Salinas, D., Flunkert, V., Gasthaus, J., and Januschowski, T. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting , 36(3):1181-1191, 2020.

Sanchez-Fernandez, A., Rumetshofer, E., Hochreiter, S., and Klambauer, G. CLOOME: contrastive learning unlocks bioimaging databases for queries with chemical structures. In NeurIPS 2022 Women in Machine Learning Workshop , 2022.

Sanquer, M., Chatelain, F., El-Guedri, M., and Martin, N. A smooth transition model for multipleregime time series. IEEE transactions on signal processing , 61(7):1835-1847, 2012.

Sch¨ afl, B., Gruber, L., Bitto-Nemling, A., and Hochreiter, S. Hopular: Modern Hopfield Networks for tabular data. arXiv preprint arXiv:2206.00664 , 2022.

Sengupta, M., Xie, Y., Lopez, A., Habte, A., Maclaurin, G., and Shelby, J. The national solar radiation data base (NSRDB). Renewable and sustainable energy reviews , 89:51-60, 2018.

Smyl, S. A hybrid method of exponential smoothing and recurrent neural networks for time series forecasting. International Journal of Forecasting , 36(1):75-85, 2020.

Stankeviˇ ci¯ ut˙ e, K., M Alaa, A., and van der Schaar, M. Conformal time-series forecasting. Advances in Neural Information Processing Systems , 34:6216-6228, 2021.

Sun, S. and Yu, R. Copula conformal prediction for multi-step time series forecasting. ArXiv , abs/2212.03281, 2022.

Sun, X., Kim, S., and Choi, J.-I. Recurrent neural network-induced gaussian process. Neurocomputing , 509:75-84, 2022.

Tagasovska, N. and Lopez-Paz, D. Single-model uncertainties for deep learning. Advances in Neural Information Processing Systems , 32, 2019.

Tajeuna, E. G., Bouguessa, M., and Wang, S. Modeling regime shifts in multiple time series. arXiv preprint arXiv:2109.09692 , 2021.

Teng, J., Wen, C., Zhang, D., Bengio, Y., Gao, Y., and Yuan, Y. Predictive inference with feature conformal prediction. arXiv preprint arXiv:2210.00173 , 2022.

Thornton, P. E., Running, S. W., White, M. A., et al. Generating surfaces of daily meteorological variables over large regions of complex terrain. Journal of hydrology , 190(3-4):214-251, 1997.

Tibshirani, R. J., Foygel Barber, R., Candes, E. J., and Ramdas, A. Conformal prediction under covariate shift. Advances in Neural Information Processing Systems , 32, 2019.

Toccaceli, P., Nouretdinov, I., and Gammerman, A. Conformal prediction of biological activity of chemical compounds. Annals of Mathematics and Artificial Intelligence , 81(1):105-123, 2017.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 30 , pp. 5998-6008. Curran Associates, Inc., 2017.

Vovk, V. Conditional validity of inductive conformal predictors. In Asian conference on machine learning , pp. 475-490. PMLR, 2012.

Vovk, V., Gammerman, A., and Saunders, C. Machine-learning applications of algorithmic randomness. In Bratko, I. and Dzeroski, S. (eds.), Proceedings of the Sixteenth International Conference on Machine Learning (ICML 1999), Bled, Slovenia, June 27 - 30, 1999 , pp. 444-453. Morgan Kaufmann, 1999.

Vovk, V., Gammerman, A., and Shafer, G. Algorithmic learning in a random world . Springer Science &Business Media, 2005.

Widrich, M., Sch¨ afl, B., Pavlovi´ c, M., Ramsauer, H., Gruber, L., Holzleitner, M., Brandstetter, J., Sandve, G. K., Greiff, V., Hochreiter, S., and Klambauer, G. Modern Hopfield Networks and attention for immune repertoire classification. In Advances in Neural Information Processing Systems . Curran Associates, Inc., 2020.

Winkler, R. L. A decision-theoretic approach to interval estimation. Journal of the American Statistical Association , 67(337):187-191, 1972.

Xia, Y., Mitchell, K., Ek, M., Sheffield, J., Cosgrove, B., Wood, E., Luo, L., Alonge, C., Wei, H., Meng, J., et al. Continental-scale water and energy flux analysis and validation for the North American Land Data Assimilation System project phase 2 (NLDAS-2): 1. intercomparison and application of model products. Journal of Geophysical Research: Atmospheres , 117(D3), 2012.

Xu, C. and Xie, Y. Conformal prediction for time series. arXiv preprint arXiv:2010.09107 , 2022a.

Xu, C. and Xie, Y. Sequential predictive conformal inference for time series. arXiv preprint arXiv:2212.03463 , 2022b.

Xu, Y., Yu, W., Ghamisi, P., Kopp, M., and Hochreiter, S. Txt2Img-MHN: Remote sensing image generation from text using Modern Hopfield Networks. arXiv preprint arXiv:2208.04441 , 2022.

Zaffran, M., F´ eron, O., Goude, Y., Josse, J., and Dieuleveut, A. Adaptive conformal predictions for time series. In International Conference on Machine Learning , pp. 25834-25866. PMLR, 2022.

Zhang, S., Guo, B., Dong, A., He, J., Xu, Z., and Chen, S. X. Cautionary tales on air-quality improvement in Beijing. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences , 473(2205):20170457, 2017.

Zhu, L. and Laptev, N. Deep and confident prediction for time series at Uber. In 2017 IEEE International Conference on Data Mining Workshops (ICDMW) , pp. 103-110. IEEE, 2017.

Zhu, X., Huang, L., Lee, E. H., Ibrahim, C. A., and Bindel, D. Bayesian transformed gaussian processes. Transactions on Machine Learning Research , 2023.

A Extended Experimental Setup & Results

A.1 Hyperparameter Search

We conducted an individual hyperparameter grid search for each predictor-dataset combination. For methods that require calibration data (HopCPT, AdaptiveCI, NexCP), each calibration set was split in half: One part served as actual calibration data while the other half was used for validation. 4 As EnbPI requires only the k past points, we used the full calibration set minus these k points for validation, so that it could fully exploit the available data. Table 2 shows all sets of hyperparameters used for the search.

Model selection. Model selection was done uniformly for all algorithms: As a first step, all models with a negative ∆ Cov on the validation set were excluded. From the remaining models, the model with the smallest PI-Width was selected. In cases where no model achieved non-negative ∆ Cov, the model with the highest ∆ Cov was selected.

HopCPT. HopCPT was trained for 3,000 epochs in each experiment. We chose this number to make sure that the loss and model selection metric curves are already converged. Throughout training, we validated every 5 epochs and selected the model that best fulfilled the model selection criteria described above. AdamW (Loshchilov & Hutter, 2019) with standard parameters ( β 1 = 0 . 9 , β 2 = 0 . 999 , δ = 0 . 01 ) was used as optimizer. The learning rate was part of the hyperparameter search. Depending on the dataset size, the batch size was set to 2 or 4 , where a batch size of n would mean that the full training part of the calibration set of n time series is used.

SPCI. The computational demand of SPCI did not allow to conduct a full hyperparameter search for the window length parameter (see more in Section A.2). Since SPCI applies a random forest on the window, one can assume that it is capable to find the relevant parts of the window. On top of that, a longer window has less risk than a window that is too short and (potentially) cuts off relevant information. Hence, we set the window length to 100 for all experiments, which corresponds to the longest setting in the original paper. To check our reasoning, we evaluated the performance of SPCI with window length 25 on all but the largest two datasets (Table 4 and Table 5). We found hardly any differences except for the smallest datasets Solar (3M). In that case we reason that the result is due to the limited calibration data in this setting.

MCD/MDN/Gauss. In Appendix A.3 we compare HopCPT to non-CP architectures that can provide prediction interval estimates. Specifically, we use Monte Carlo Dropout (MCD; Gal & Ghahramani, 2016), a Mixture Density Network (MDN; Bishop, 1994), and a Gauss model (i.e., an MDN with a single Gaussian density). An LSTM serves as backbone for all models. Therefore, there is no distinction between the predictor model and the uncertainty model. To allow for a fair comparison to CP approaches, the models are trained on both the predictor training data and the calibration data combined. None of these approaches are CP methods, and MCD was specifically devised to estimate only epistemic uncertainty. Nevertheless, we consider these methods as useful comparisons, since they are widely used for uncertainty quantification. The models were trained for 150 epochs with AdamW (Loshchilov & Hutter, 2019) (using its standard parameters: β 1 = 0 . 9 , β 2 = 0 . 999 , δ = 0 . 001 ). We chose this number to ensure that the loss and model selection metrics already converged. Throughout the training, we validated after every epoch and selected the model that best fulfilled the model selection criteria described above. The learning rate was part of the hyperparameter search.

Global LSTM. We conducted an individual hyperparameter tuning of the global LSTM prediction model for each dataset. Table 3 shows the hyperparameters used for the grid search. Similar to the LSTM uncertainty models, we trained for 150 epochs with an AdamW optimizer (Loshchilov & Hutter, 2019) and validated after each epoch. We optimized for the mean squared error and selected the model with the smallest validation mean squared error.

Table 3: Parameters used in the hyperparameter search of the global LSTM prediction model.

A.2 SPCI Retraining

As our results show, SPCI is very competitive (see, for example, Section 3.2). Xu & Xie (2022b) did, however, design SPCI so that its random forest regressor is retrained after each prediction step. This design choice is computationally prohibitive for larger datasets. To nevertheless allow a comparison against SPCI on the larger datasets, we modified the algorithm so that the retraining is skipped. Experiments on the smallest dataset (Table 6) show only small performance decrease with the modified algorithm. One limitation of the adapted algorithm is, however, that a strong shift in the error distribution(s) would potentially require a retraining on the new distribution. A viable proxy to detect such a change in our setting is the coverage performance of standard CP. The reason for this is that standard CP predictions are solely based on the calibration data and can thus not account for shifts. In the experiments (Section 3 and Appendix A.3), standard CP achieves good coverage with only one exception (see Section 3). Hence, we decided to include SPCI (in the modified version) to enable a comparison to it on larger datasets.

6000

4000

2000

0

2000

4000

6000

t

-2

-1

0

1

(10 4

)

End of calibration period

t

200

100

0

100

200

End of calibration period

low

high

Association weight

A.3 Additional Results

Tables 17-23 show the results for miscoverage levels α ∈ { 0 . 05 , 0 . 10 , 0 . 15 } for all evaluated combinations of datasets and predictors. Results for individual time series of the datasets are uploaded to the code repository (Appendix J).

CMAL. For the Streamflow dataset, we additionally compare to CMAL, which is the state-of-theart non-CP uncertainty estimation technique in the respective domain (Klotz et al., 2022). CMAL is a mixture density network (Bishop, 1994) based on an LSTM that predicts the parameters of asymmetric Laplacian distributions. As our experiments use the same dataset, we adopt the hyperparameters from Klotz et al. (2022) but lower the learning rate to 0 . 0001 because we train CMAL on more training samples (18 years, i.e., the training and calibration period combined, which allows for a fair comparison against the CP methods). Despite the fact that CMAL is not based on the CP paradigm, it achieves good coverage results, however, at the cost of wide prediction intervals. We argue that the Winkler score is low because CMAL considers all (and therefore also uncovered) samples during the optimization while CP methods do not consider them at all - therefore, the distance of these points to the interval might be smaller (see Table 23).

Error trend. Figure 4 shows the prediction errors of the ridge regression model on the Sap flow dataset. The errors exhibit a strong trend and shift towards a negative expectation value.

Association weights. Figure 5 investigates the association patterns of HopCPT and shows its capabilities to focus on time steps from similar error regimes. The depicted time step in a regime with negative errors retrieves time steps with primarily negative errors and, likewise, the time step in a regime with positive errors retrieves time steps with primarily positive errors of the same magnitude.

Non-CP methods We also compare HopCPT to non-CP methods to evaluate its performance within the broader field of uncertainty estimation for time series. Specifically, we compare against a Monte Carlo Dropout (MCD); a Mixture Density Network (MDN), and a Gauss model (i.e., an MDN with a single Gaussian component), with an LSTM backbone (see Appendix A.1 for more details about the models and their hyperparameters). This comparison shows that, although the non-CP methods often lead to very narrow prediction intervals, they fail to provide approximate coverage on most experiments (Table 7).

A.4 Local Coverage

Standard CP only provides marginal coverage guarantees and a constant prediction interval width (Vovk et al., 2005). In time series prediction tasks, this can lead to bad local coverage (Lei & Wasserman, 2014). To evaluate whether the coverage approximately holds also locally, we evaluated ∆ Cov on windows of size k . To avoid compensation of negative ∆ Cov in some windows by other windows with positive ∆ Cov, we upper-bounded each window's ∆ Cov by zero before averaging over the bounded coverage gaps - i.e., we calculated 1 W ∑ W w =1 -∆ Cov ⊤ 0 w , with

where ∆ Cov w is the ∆ Cov within window w .

Table 8 shows the results of this evaluation for window sizes k ∈ { 10 , 20 , 50 } and miscoverage level α = 0 . 1 . Depending on the dataset, most often either HopCPT or SPCI perform best. The overall rankings are only partly similar to the evaluation of the marginal coverage (Table 1 and Appendix A.3). Especially standard CP, which achieves competitive results in marginal coverage, falls short in this comparison. Only for Solar (3M), where the approach achieves a high marginal ∆ Cov, it preserves the local coverage best. Note that this comes with the drawback of very wide prediction intervals, i.e., bad efficiency. Overall, the results show that HopCPT and other time series CP methods improve the local coverage in non-exchangeable time series settings, compared to standard CP.

A.5 Negative Results

We also tested training with more involved training procedures than directly using the mean squared error. However, based on our empirical findings, the vanilla method with mean squared error outperforms more complex approaches. In the following, we briefly describe those additional configurations (so that potential future research can avoid these paths):

Pinball Loss . We tried to train the MHN with a pinball loss (the original use seems to stem from Fox & Rubin, 1964, we are however not aware of the naming origin) in many different variations (see points that follow), but consistently got worse results than with the mean squared error.

-Inspired by the ideas lined out by Tagasovska & Lopez-Paz (2019) we tried to use the miscoverage level as an additional input to the network, while also parameterizing the loss with it.

-We tried to use the pinball loss to predict multiple miscoverage values at the same time to get a more informed representation of the distribution we want to approximate.

Softmax Probabilities . We tried to use the softmax output of the MHN to directly estimate probabilities and use a maximum likelihood procedure. This did train, but it did not produce good results.

A.6 Computational Resources

We used different hardware setups in our experiments, however, most of them were executed on a machine with an Nvidia P100 GPU and a Xeon E5-2698 CPU. The runtime differs greatly between different dataset sizes. We report the approximate run times for a single experiment (i.e., one seed, all evaluated coverage levels, not including training the prediction model ˆ µ ):

HopCPT : Solar (3Y) 5h, Solar (1Y) 1h, Solar (3M) 7min, Air Quality (10PM) 3h, Air Quality (25PM) 3h, Sap flow 2.5h, Streamflow 12-20h

SPCI : (adapted): Solar (3Y) 5-10 days, Solar (1Y) 10-30h, Solar (3M) 45min, Air Quality (10PM) 13-30h, Air Quality (25PM) 13-30h, Sap flow 20-50h, Streamflow 12-17 days

EnbPI : all datasets under 6h

NexCP : all datasets under 45min

AdaptiveCI : all datasets under 45min

Standard CP : all datasets under 45min

B Theoretical Background

We denote the full dataset as D = ( D 1 , . . . , D t +1 ) , where D i = ( X i , Y i ) is the i -th datapoint. Since we are in the split conformal setting (Vovk et al., 2005), the full dataset is subdivided into two parts: the calibration and the test data. The prediction model ˆ µ is already given in this setting and not selected based on D . The prediction interval ̂ C α t for split conformal prediction is given by

where R i = | Y i -ˆ µ ( X i ) | , δ c is a point mass at c , and Q τ is the τ -quantile of a distribution. Vovk et al. (2005) show that this interval provides a marginal coverage guarantee in case of exchangeable data:

Theorem B.1 (Split Conformal Prediction (Vovk et al., 2005)) .

If the data points ( X 1 , Y 1 ) , . . . , ( X t , Y t ) , ( X t +1 , Y t +1 ) are exchangeable then the split conformal prediction set defined in (11) satisfies

This seminal result cannot be directly applied to our setting, because time series are often nonexchangeable (as we discuss in the introduction) and models often exhibit locally specific errors.

The remainder of this section is split into two parts. First, we discuss the theoretical connections between the MHN retrieval and CP through the lens of Foygel Barber et al. (2022). Second, we adapt the theory of Xu & Xie (2022a) to be applicable to our regime-based setting.

For the MHN retrieval, we evoke the results of Foygel Barber et al. (2022), who introduce the concept of non-exchangeable split conformal prediction. Instead of treating each calibration point equally, they assign an individual weight for each point, to account for non-exchangeability. Formally, we denote the weight for the i -th point by w i ∈ [0 , 1] , i = 1 , . . . , t ; and the respective normalized weights are defined as

The prediction interval for non-exchangeable split conformal setting ̂ C α t is given by

Foygel Barber et al. (2022) show that non-exchangeable split conformal prediction satisfies a similar bound as exchangeable split conformal prediction, which is extended by a term that accounts for a potential coverage gap ∆ Cov. Formally we define ∆ Cov = α -α ⋆ , where α is the specified miscoverage and α ⋆ is the observed miscoverage.

To specify the coverage gap they use the total variation distance d TV , which is a distance metric between two probability distributions. It is defined as

where P 1 and P 2 are arbitrary distributions defined on the space (Ω , F ) .

Intuitively, d TV measures the largest distance between two distributions. It is connected to the widely used Kullback-Leibler divergence D KL through Pinsker's inequality (Pinsker, 1964):

Given this measure, Foygel Barber et al. (2022) bound the coverage of non-exchangeable split conformal prediction as follows:

Theorem B.2 (Non-exchangeable split conformal prediction (Foygel Barber et al., 2022)) . The non-exchangeable split conformal method defined in (13) satisfies

where R ( D ) = ( R 1 , R 2 , . . . , R t +1 ) denotes the sequence of absolute residuals on D and D i denotes a sequence where the test point D t +1 is swapped with the i -th calibration point.

The coverage gap is represented by the rightmost term. Therefore,

Foygel Barber et al. (2022) prove this for the full-conformal setting, which also directly applies to the split conformal setting (as they note, too). For the sake of completeness, we state the proof of Theorem B.2 explicitly for split conformal prediction (adapted from Foygel Barber et al., 2022):

By the definition of the non-exchangeable split conformal prediction interval (13), it follows that

Next, we show that, for a given K ∈ { 1 , . . . , t +1 } ,

by considering that

̸

̸

̸

In the case of K = t +1 , this holds trivially since both sides are equivalent. In the case of K ≤ t , we can rewrite the left-hand side of inequality (17) as

̸

The right-hand side can be similarly represented as

̸

By the definition of the normalized weights (12) and the assumption that w K ∈ [0 , 1] , it holds that ˜ w t +1 ≥ ˜ w K . Applying this to the rewritten forms of the left- and right-hand sides, we can see that inequality (17) holds.

Substituting the right hand side of (16) by using (17), we see that

or equivalently by using (18):

For the next step we use the definition of the set of so-called strange points S :

A strange point s ∈ S therefore refers to a data point with residuals that are outside the prediction interval. Given the definition of S , we can directly see that

Combining (19) and (20) we see that non-coverage of Y t +1 implies strangeness of point K :

Given this, we arrive at the final derivation of the coverage bound:

where we use (21) from the penultimate to the last line. The first to the second line follows from K ⊥ ⊥ D i , which holds because K ⊥ ⊥ D and D i is a permutation of D that does not depend on K.

Foygel Barber et al. (2022) show two additional variations of the coverage gap bound based on Theorem B.2. We do not replicate them here. One notable observation of theirs, however, is that instead of using the total variation distance based on the residual vectors also the distance of the raw data vector can be used:

This coverage bound suggests that low weights for time steps corresponding to high total variation distance ensure small coverage gaps. However, lower weights weaken the estimation quality of the prediction intervals. Hence, all relevant time steps should receive high weights, while all others should receive weights close to zero.

HopCPT leverages these results by using the association weights a i from Equation 6 to reflect the weights ˜ w i of non-exchangeable split conformal prediction. HopCPT aims to assign high association weights to data points that are from the same error distribution as the current point, and low weights to data points in the opposite case. By Theorem B.2, association weights that follow this pattern lead to a small coverage gap.

An asymptotic (conditional) coverage bound If we assume clearly distinguished error distributions instead of gradually shifting ones, hard weight assignment, i.e., association weights that are either zero or one, are a meaningful choice. Given this simplification we can directly link HopCPT to the work of Xu & Xie (2022a).

Xu & Xie (2022a) introduce a variant of (split) conformal prediction with asymmetric prediction intervals:

The approach taken by Xu & Xie (2022a) additionally adjusts the share of α between the upper and lower quantiles such that the width of the prediction interval is minimized. In contrast, HopCPT adopts a simpler strategy and uses α 2 on both sides, which mitigates the need for excessive recalculation of the quantile function.

Based on the introduced interval calculation procedure, they also provide bounds regarding its coverage properties, in particular about the possible coverage gap. While Xu & Xie (2022a) base their assumption on error properties only in temporal proximity, we generalize this to proximity in the learned feature representation (which can include but is not limited to information about temporal proximity). Based on this proximity we assume:

Assumption B.3 (Error regimes) . Given { ( Z i , y i , ˆ y i ) } t +1 i =1 which represents the (encoded) dataset and the predictions ˆ y i of ˆ µ , there exists a partition T = { T i } i ∈ I of the time steps { 1 , . . . , t } , so that

(i) ∀ i ∈ I : { ϵ t = y t -ˆ y t | t ∈ T i } ∼ E i ,

(ii) ∃ i ∈ I : { ϵ t +1 } ∼ E i ,

(iii) the subset T i corresponding to t +1 can be identified based on Z t +1 ,

̸

(iv) ∀ i, j ∈ I : i = j ∨ E i = E j ,

where each E i is an arbitrary error distribution.

Properties (i) and (ii) imply that any given time step t +1 can be associated to a subset T i where all errors follow a common error distribution E i ; property (iii) ensures that the partition can be found or approximated from a set of features. Property (iv) is not necessary for the discussion at hand but avoids degenerate solutions, since tight coverage bounds call for assigning all errors from the same error distribution to one large partition. If the errors of the associated subset were sampled i.i.d. from E i , we could use standard split conformal methods (Vovk et al., 2005), and the variation distance between permutations of the errors in properties (i) and (ii) is zero. In practice, error noise can make it challenging to find and delineate a given partition. Thus, HopCPT uses a soft-selection approach to approximate the discrete association of a partition and the current features.

To align HopCPT with the results of Xu & Xie (2022a), and thus make use of their proposed coverage bounds for asymmetric prediction intervals, we first introduce their decomposition of the error: ϵ t = ˇ ϵ t +( µ ( x t ) -ˆ µ ( x t )) , where f is the true generating function, ˆ µ the prediction model, and ˇ ϵ a noise term associated with the label. The notation µ -t ( x ) stems from Xu & Xie (2022a) and denotes an evaluation of µ , selected without using t .

Next, we adopt their i.i.d. assumption about the noise with regard to our approach:

Assumption B.4 (Noise is approximately i.i.d. within an error regime; adaptation of Xu & Xie (2022a), Assumption 1) . Assume ∃ i ∈ I : { ˇ ϵ t | t ∈ T i } ∪ { ˇ ϵ t +1 } iid ∼ E i and that the CDF ˇ F i of E i is Lipschitz continuous with constant L i > 0 .

This i.i.d. assumption within each error regime only concerns the noise of the true generating function. Xu & Xie (2022a) note that the generating process itself can exhibit arbitrary dependence and be highly non-stationary while still having i.i.d. noise. The assumption is thus relatively weak.

Assumption B.5 (Estimation quality; adaptation of Xu & Xie (2022a), Assumption 2) . There exist real sequences { δ i, | T i | } i, | T i |≥ 1 such that

(i) ∀ i ∈ I : 1 | T i | ∑ t ∈ T i (ˆ µ -t ( x t ) -µ ( x t )) 2 ≤ δ 2 i, | T i |

(ii) ∃ i ∈ I : | ˆ µ -( t +1) ( x t +1 ) -µ ( x t +1 ) | ≤ δ i, | T i | , where the i for a given t +1 is the same i as in assumption B.4.

We want to bound the coverage of (24), which is based on the prediction error ϵ . However, Assumption B.4 only considers the noise ˇ ϵ of the true generating function. Xu & Xie (2022a) bind the difference between the CDF of the true generation function ˇ F i , the empirical analogue ˜ F i ( x ) := 1 | T i | ∑ j ∈ T i 1 { ˇ ϵ j ≤ x } , and the empirical CDF of the prediction error F i ( x ) := 1 | T i | ∑ j ∈ T i 1 { ϵ j ≤ x } .

In particular, they state two lemmas:

Lemma B.6 (Adaptation of Xu & Xie (2022a), Lemma 1) . Under Assumption B.4, for any subset size | T i | , there is an event A T which occurs with probability at least 1 -√ log(16 | T i | ) / | T i | , such that conditioning on A T ,

Lemma B.7 (Adaptation of Xu & Xie (2022a), Lemma 2) . Under Assumptions B.4 and B.5, we have

With this result we arrive at a bound for a conditional and a marginal coverage:

Theorem B.8 (Conditional coverage gap; adaptation of Xu & Xie (2022a), Theorem 1) . Given the prediction interval ̂ C α t defined in (24). Under Assumption B.4 and B.5, for any subset size | T i | , α ∈ (0 , 1) , and β ∈ [0 , α ] , we have:

Furthermore, if { δ i, | T i | } | T i |≥ 1 converges to zero, the upper bound in (25) converges to 0 when | T i |→∞ , and thus the conditional coverage is asymptotically valid.

Theorem B.9 (Marginal coverage gap; adaptation of Xu & Xie (2022a), Theorem 2) . Given the prediction interval ̂ C α t defined in (24). Under Assumption B.4 and B.5, for any subset size | T i | , α ∈ (0 , 1) , and β ∈ [0 , α ] , we have:

The proofs of the above theorems follow from Xu & Xie (2022a). To apply our adaptions to their poofs, one has to select the error regime matching to the test time step t +1 , i.e., select the right subset T i from the partitioning of time steps. T i then corresponds to the overall calibration data in the original assumptions, lemmas, theorems, and the respective proofs. This is valid since Assumption B.5-(i) encompasses all subsets and the existing subset in Assumption B.4 and Assumption B.5-(ii) is, as noted, the same.

Xu & Xie (2022a) provide additional bounds for two variations of Assumption B.4, imposing only the requirement of strongly mixing noise or noise following a linear process. These bounds apply also to our adaption but are omitted here for brevity.

Notes on Exchangeability, Stationarity, Seasonality, and Drifts Comparing HopCPT to standard CP we see that HopCPT assumes exchangeability over the weighted time steps within regimes. Hence, we require that the relationship between inputs and outputs remains stationary within regimes, which is weaker than requiring that the time series as such is stationary. For example, it allows that time series values decrease or grow over time as long as these changes are reflected in the inputs of the MHNand correspond to a regime. HopCPT softens the exchangeability requirements of standard CP in two ways:

The CP mechanism assumes that exchangeability is given within a regime that is selected by the MHN association.

Even if (1) is not the case, the memory of the MHN (and therefore the calibration data) is updated with each new observation. This memory update allows HopCPT to account for shifts - empirically, the benefit is visible in our experiments on the sap flow dataset, as we discuss briefly in Section 3.2 (lines 283-289) and in more detail in Appendix A.3 (Error Trend & Figure 4).

Regarding seasonality: HopCPT uses covariate features to describe the current time step. Hence, HopCPT implicitly considers seasonality by comparing a given time step with the time steps from the memory. Whenever a time step of a certain season is encountered, it focuses on time steps from the same season. In fact, our experiments already comprise both regular and irregular seasonality patterns. For example, the streamflow dataset contains yearly reoccurring patterns (spring, summer, fall, winter) as well as irregularly reoccurring similar flood events. In both cases, HopCPT can identify related time steps based on the covariates.

Regarding data drifts: an advantage of HopCPT is that the memory is updated with every new observation. Therefore, HopCPT can somewhat incorporate drifts in the target value as long as the covariates remain so that the learned representation is able to focus on the relevant observations. Empirically, the robustness against drifts is visible in our experiments on the sap flow dataset, as we discuss briefly in Section 3.2 (lines 283-289) and in more detail in Appendix A.3 (Error Trend & Figure 4).

C Dataset Details

Datsets from four different domains were used in the experiments. The following paragraphs summarize the details. Table 9 provides a quantitative overview.

Solar. We conducted experiments on three solar radiation datasets: Solar (3M), Solar (1Y), and Solar (3Y). All three datasets are based on data from the US National Solar Radiation Database (NSDB; Sengupta et al., 2018). Besides solar radiation as the target variable, the datasets include 8 other environmental features. Solar (3M) is the same dataset as used by Xu & Xie (2022a,b) and focuses on data from 8 cities in California. To show the scalability of HopCPT we additionally generated two larger datasets Solar (1Y) and Solar (3Y), which include data from 50 cities from different parts of the US. We selected the cities by ordering the areas provided by NSDB according to the population density and picked the top 50 under the condition that no city appears twice in the dataset.

Air quality. The datasets Air Quality (10PM) and Air Quality (25PM) are air quality measurements from 12 monitoring sites in Beijing, China (Zhang et al., 2017). The datasets provide two target variables (10PM and 2.5PM measurements) which we use separately in our experiments (the variables are used mutually exclusively in the datasets, e.g., when predicting 10PM we do not use 2.5PM as a feature). We encode the wind direction, given as a categorical variable, by mapping the north-south and the east-west components each to a scalar from -1 to 1 .

Sap flow. The Sap flow dataset is a subset of the Sapflux data project (Poyatos et al., 2021). This dataset includes sap flow measurements which we use as a target variable, as well as a set of environmental variables. The available features, the length of the measurements, and the sampling vary between the individual time series. To get a set of comparable time series we processed the data as follows: (a) We removed all time series where not all of the 10 environmental features are available. (b) If there was a single missing value between two existing ones, we filled the value with the value before (forward fill). (c) We cut out all sequences without any missing target or feature values (after step b). (d) From the resulting set of sequences we removed all that have less than 15,000 or more than 20,000 time steps.

Streamflow. For our experiments on the streamflow data, we use the Catchment Attributes and Meteorology for Large-sample Studies dataset (CAMELS; Newman et al., 2015; Addor et al., 2017). It provides meteorological time series from different data products, corresponding streamflow measurements, and static catchment attributes for catchments across the continental United States. We ran our experiments on the subset of 531 catchments that were used in previous hydrological benchmarking efforts (e.g., Newman et al., 2017; Kratzert et al., 2019, 2021; Klotz et al., 2022). Specifically, we trained the LSTM prediction model with the NeuralHydrology Python library (Kratzert et al., 2022) on precipitation, solar radiation, minimum and maximum temperature, and vapor pressure from the NLDAS (Xia et al., 2012), Maurer (Maurer et al., 2002), and Daymet (Thornton et al., 1997) meteorological data products. Further, the LSTM received 26 static catchment attributes at each time step that identify the catchment properties. Table 4 in Kratzert et al. (2019) provides a full list of these attributes. We trained the prediction model on data from the period Oct 1981 - Sep 1990 (note that for some catchments, this period starts later due to missing data in the beginning), calibrated the uncertainty models on Oct 1990 - Sep 1999, and tested them on the period Oct 1999 - Sep 2008.

D kNN vs. Learned Representation

As mentioned in the main paper, we designed HopCPT with large datasets in mind. It is only in such a setting that the learned part of our approach can truly play to its strengths and take advantage of nuanced interrelationships in the data. kNN provides us with a natural 'fallback' option for settings where not enough data is available to infer these relationships. Our comparisons in Table 10 and Table 11 substantiate this argument: kNN provides competitive results for the small dataset Solar (3M) (this can also be seen by contrasting the performance from Table 10 to the respective results in Appendix A.3), but is outperformed by HopCPT for the larger datasets.

E Quantile of Sample vs. Quantile of Weighted ECDF

HopCPT constructs prediction intervals by calculating a quantile on the set of weight-sampled errors (see Equation 7). An alternative approach is to calculate the quantile over a weighted empirical CDF of the errors. This approach would define q ( τ, Z t +1 ) in Equation 7 as

where δ ϵ i is a point mass at | ϵ i | . Empirically, we find little differences in the performance when comparing the two approaches (Tables 12 and 14).

F Asymmetrical Error Quantiles

HopCPT calculates the prediction interval by using equally large quantiles (in terms of distribution mass, not width) for the upper and lower bound (see Equation 7). Xu & Xie (2022a) instead search for a ˆ β ∈ [0 , α ] and calculate the lower quantile with ˆ β and the upper quantile with 1 -α + ˆ β , which minimizes the overall interval with. Applying this principle to Equation 7 one would arrive at

Practically, the search is done by evaluating different β values within the potential value range and selecting the one which produces the lowest interval width. Theoretically, this could lead to more efficient intervals at the cost of additional computation and potential asymmetry in regard to uncovered samples that are higher and lower than the prediction interval. Empirically, we do not find a notable improvement when applying this search to HopCPT (see Table 13).

G AdaptiveCI and HopCPT

As noted in Section 3, AdaptiveCI is orthogonal to HopCPT, SPCI, EnbPI, and NexCP. We therefore also evaluated the combined application of the models. Nevertheless, AdaptiveCI is an independent model on top of an existing model, which is reflected in the way we select the hyperparameters of the combined model: First, the hyperparameters are selected for each model without adaption through AdaptiveCI (see Appendix A.1) - hence, we use the same hyperparameters as in the main evaluation. Second, we conduct another hyperparameter search, given the model parameters from the first search, where we only search for the parameter of the adaptive component.

Tables 15 and 16 show the results of these experiments. Overall, the results are slightly better, as the Winkler score (which considers both width and coverage) slightly improves in most experiments. The ranking between the different models stays similar to the non-adaptive comparison (see Section 3.2) with HopCPT performing best on all but the smallest dataset.

H Details on Continuous Modern Hopfield Networks

Modern Hopfield Networks are associative memory networks. They learn to associate entries of a memory with a given 'query' sample in order to produce a prediction. In HopCPT, the memory corresponds to representations of past time steps and the query corresponds to a representation of the current time step. The association happens by means of weights on every memory entry, which are called 'association weights'. Query-memory sample pairs that have similar representations are assigned high association weights. In summary, the association weights indicate which samples are closely related - in HopCPT, this corresponds to finding the time steps which are part of the same time series regimes and therefore useful to estimate the current uncertainty. The association mechanism is in that sense closely related to the attention mechanism of Transformers (Vaswani et al., 2017). However, compared to other trainable networks, the memory-based architecture has the advantage that it does not need to encode all knowledge in network parameters but can dynamically adapt when given new memory entries.

We adapt the following arguments from F¨ urst et al. (2022) and Ramsauer et al. (2021) for a more formal overview. Associative memory networks have been designed to store and retrieve samples. Hopfield networks are energy-based, binary associative memories, which were popularized as artificial neural network architectures in the 1980s (Hopfield, 1982, 1984). Their storage capacity can be considerably increased by polynomial terms in the energy function (Chen et al., 1986; Psaltis & Cheol, 1986; Baldi & Venkatesh, 1987; Gardner, 1987; Abbott & Arian, 1987; Horn & Usher, 1988; Caputo & Niemann, 2002; Krotov & Hopfield, 2016). In contrast to these binary memory networks, we use continuous associative memory networks with far higher storage capacity. These networks are continuous and differentiable, retrieve with a single update, and have exponential storage capacity (and are therefore scalable, i.e., able tackle large problems; Ramsauer et al., 2021).

Formally, we denote a set of patterns { x 1 , . . . , x N } ⊂ R d that are stacked as columns to the matrix X = ( x 1 , . . . , x N ) and a state pattern (query) ξ ∈ R d that represents the current state. The largest norm of a stored pattern is M = max i ∥ x i ∥ . Then, the energy E of continuous Modern Hopfield Networks with state ξ is defined as (Ramsauer et al., 2021)

where C = β -1 log N + 1 2 M 2 . For energy E and state ξ , Ramsauer et al. (2021) proved that the update rule

converges globally to stationary points of the energy E and coincides with the attention mechanisms of Transformers (Vaswani et al., 2017; Ramsauer et al., 2021).

The separation ∆ i of a pattern x i is its minimal dot product difference to any of the other patterns:

̸

Apattern is well-separated from the data if ∆ i is above a given threshold (specified in Ramsauer et al., 2021). If the patterns x i are well-separated, the update rule Equation 29 converges to a fixed point close to a stored pattern. If some patterns are similar to one another and, therefore, not well-separated, the update rule converges to a fixed point close to the mean of the similar patterns.

The update rule of a Hopfield network thus identifies sample-sample relations between stored patterns. This enables similarity-based learning methods like nearest neighbor search (see Sch¨ afl et al., 2022), which HopCPT leverages to learn a retrieval of samples from similar error regimes.

I Potential Social Impact

Reliable uncertainty estimates are crucial, especially for complex time-dependent environmental phenomena. However, overreliance on these estimates can be dangerous. For example, unseen regimes might not be properly predicted. A changing climate that evolves the environment beyond already seen conditions can cause new forms of error regimes which cannot be predicted reliably. As most machine learning approaches, our method requires accurately labeled training data. Incorrect labels may lead to unexpected biases and prediction errors.

J Code and Data

The code and data to reproduce all of our experiments are available at https://github.com/ ml-jku/HopCPT .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel method for estimating uncertainty in prediction of time series. Modern Hopfield Networks is used as a method of measuring uncertainty based on the existence of similar historical data. This paper presents its construction method and shows that it works for multiple data and forecasting methods\n\nSTRENGTHS:\nThe authors have clarified the issues with CP for time series and proposed a method to improve this point by applying MHNs, and experiments have shown very good results.\n\nWEAKNESSES:\nMy four points of concern are as follows:\n- This paper consists only of a method proposal and an experimental evaluation, and does not include theoretical effects. In this structure, is an evaluation on four data sets sufficient to demonstrate the effectiveness of the proposed method? \n- The task setting of this paper seems to be related to Gaussian processes and Bayesian optimization as well. This point seems to be a natural one for many people, and while it may not be directly comparable, would it be appropriate to make no mention of it?\n- The proposed method incorporates MHNs, a function that calculates the reliability of forecasts from past information, into CP. The results shown in the experiment indicate the validity of the CP issue and the effectiveness of the improvement direction. However, the experiments did not provide justification for introducing HMNs to solve the problem.\n- Identification of issues and presentation of solutions is an important contribution, but it is a combination of existing methods, and I feel it is weak in terms of novelty.\n\nQUESTIONS:\n- Regarding the first point of Weakness, can you state why these data are sufficient, although they are a limit to the number of experiments? Or can they be increased?\n- For the second point, for example, can you compare it to or give reasons why it is not comparable to the following advanced methods of Gaussian processes\n\t- X. Zhu et. al, BayesianTransformedGaussianProcesses, TMLR (4/2023)\n\t- G. Corani et. al, Time Series Forecasting with Gaussian Processes Needs Priors, ECML PKDD 2021\n\t- X. Sun et. al, Recurrent neural network-induced Gaussian process, Neurocomputing 509(2022)\n- For the third point, can you specify why HMNs is more effective than other methods that can assume similar effects?\n\nLIMITATIONS:\nThe authors do not explicitly address Limitation. On the other hand, the study is oriented toward reducing the social impact on the limitations of conventional methods and does not promote adverse effects.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work presents HopCPT, a conformal prediction algorithm for Modern Hopfield Networks in the setting of time series forecasting. The work uses a novel (attention-like) weighted quantile approach to leverage similar time steps from the past, improving both coverage and sharpness compared to SOTA CP methods. The proof mainly extends from  (Foygel Barber et al., 2022) and (Xu and Xie 2022). The method is shown to perform well on both synthetic and real world datasets.\n\nSTRENGTHS:\n- Novel and sound CP algorithm on a widely-used architecture for forecasting. \n- Excellent efficiency performance on time series datasets, with through comparisons to existing methods. \n- I also appreciate the use of proper scoring rules for evaluation (Winkler score).\n\nWEAKNESSES:\n- From the paper alone, it's difficult to tell what exactly are the theoretical guarantees of your algorithm. While the authors did refer to Appendix B, I think it is important to briefly describe the assumptions and state the main bound (B.8 and B.9) of your theoretical analysis in the paper.\n\nQUESTIONS:\n1. Does your method assumes stationarity of the time series?  because the $a_i$s are fixed, \n\n2. Equation 7. Would searching for a $\\beta$ as in SPCI (Xu & Xie 2023)  instead of using $\\alpha/2$ using further improve efficiency?\n\n3. I'm not sure point 4 in your main contributions is worded in the best way - your guarantees are not specific to hydrological prediction. Maybe a better way to say it is that HopCPT is the first algorithm with marginal coverage guarantees that can be applied to  hydrological prediction applications?\n\n4. Given the current soft-selection algorithm, since the $a_{t+1}$ s are static, would HopCPT still perform well when the seasonality in data is not regular?  What if there are drifts in the distribution?\n\nLIMITATIONS:\n- It might be good to stress that the guarantee is marginal and asymptotic, which can be achieved by rather simple algorithms (see Gibbs & Candes 2021, 2022). This is of course due to the properties of time series data, not a weakness of the approach.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper deals with providing conformal prediction (CP) sets for the time series prediction problem or, more generally, the cases where the exchangeability assumption is violated. Similar to the prior work of Foygel Barber et al. (2022), they also try to assign weights to the points in the calibration set with higher (lower) values for those who are coming from a more (less) similar region (aka regimes) as the test point. For such a weighting mechanism, they use Modern Hopfield Networks (MHN) that are capable of finding regions similar to the point for which a prediction set is sought. Using numerical experiments, they showed the superiority of their proposed framework.\n\nSTRENGTHS:\nThe paper is indeed well-structured, nicely written, and easy to comprehend. \nThe extensive amount of numerical experiments supports their claim about the (empirical) superiority of their proposed framework over the vast range of competitors.\n\nWEAKNESSES:\nThe contribution of this paper seems to be relatively marginal. Indeed, the authors introduced a new way of weighting the non-conformity scores (i.e., reforming the distribution of non-conformity scores) on top of the NexSCP framework proposed by Foygel Barber et al. (2022) while adopting the structure proposed by Xu & Xie (2022a) for interval constructions.  \n\nWhile the numerical results show their framework's (empirical) superiority in terms of efficiency and coverage, one should keep in mind that it adds relatively high computational complexity to the model. This is, of course, also reported in Appendix A.6.\n\nQUESTIONS:\n1. What is the justification behind the choice of encoding network (section 2.3)? Apparently, for the current experiments, it is working adequately. However, why should one stick to such a structure for every other dataset?\n\n2. I still do not understand the role of $W_q$ and $W_k$. How are they learned? What is the difference between them?\n\nLIMITATIONS:\nThe authors mention a limitation of the proposed framework: the need for large data for HopCPT to perform well. However, there is still room for elaboration on other limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a conformal prediction approach for black box time series forecasting models. The method is based on learning to predict the model’s errors using a small neural network structured similarly to the softmax computation in an attention mechanism. The prediction interval of the proposed method utilizes the softmax scores in the error sampling procedure, drawing each error with the probability given by the softmax score. Both in synthetic and benchmark experiments, the proposed method outperforms other CP approaches in terms of PI-width and coverage gap.\n\nSTRENGTHS:\n* The manuscript is extremely well structured and very easy to follow. The authors provided an excellent overview of the conformal prediction task and relevant work. \n* The experiments provide a thorough insight into how the proposed method functions in a synthetic and more realistic setting. Figure 2 contrasts the proposed with competing methods in a comprehensible and straightforward way. \n* In addition to a solid theoretical motivation, the proposed method performs outstandingly and convincingly on a wide range of prediction models and several data sets.\n\nWEAKNESSES:\n* Without a background on Hopfield networks, the method seems an application of the attention idea to conformal prediction. It would be helpful to lay out the similarities and differences between MHNs and the attention mechanism. \n* I am missing a section on the limitations of the proposed method. For instance, how does the method scale to very long time series?\n\nQUESTIONS:\n* How is the actual marginal miscoverage $\\alpha^{*}$ computed?\n* Could you elaborate on the meaning of $\\delta_{+\\infty}$ in Eq. 3 and how are weights $w_i$ computed? Is it model-dependent?\n* Does the bound in Equation 5 apply to your method?\n* Is it possible to make Figure 3 less dense and make the connection to the prediction interval more comprehensible?\n* I am slightly confused about the notation in Eq. 3. The quantile should be computed on a set, but I am not sure from which values it is computed. Can you rewrite this in set-notation like Eq. 2?\n\nLIMITATIONS:\nWhile there is no dedicated limitation section, the authors provide “negative results” in the appendix. A concise paragraph about the method’s limitations would be appreciated.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}