Conformal Prediction for Time Series with Modern Hopfield Networks
Authors: Andreas Auer
Martin Gauch
Daniel Klotz
Sepp Hochreiter
Venue: NeurIPS 2023 poster (2023)
Keywords: time series
uncertainty
prediction interval
conformal prediction
modern hopfield networks
TL;DR: We propose HopCPT, a novel uncertainty estimation approach designed for time serie which is based on conformal prediction and uses continuous Modern Hopfield Networks for similarity-based sample reweighting.

Abstract: To quantify uncertainty, conformal prediction methods are gaining continuously more interest and have already been successfully applied to various domains. However, they are difficult to apply to time series as the autocorrelative structure of time series violates basic assumptions required by conformal prediction. We propose HopCPT, a novel conformal prediction approach for time series that not only copes with temporal structures but leverages them. We show that our approach is theoretically well justified for time series where temporal dependencies are present. In experiments, we demonstrate that our new approach outperforms state-of-the-art conformal prediction methods on multiple real-world time series datasets from four different domains.

Conformal Prediction for Time Series with Modern Hopfield Networks
Andreas Auer
Martin Gauch" Daniel Klotz" Sepp Hochreiter' "ELLIS Unit Linz and LIT AI Lab, Institute for Machine Learning,
Johannes Kepler University Linz, Austria Google Research, Zürich, Switzerland
Abstract
To quantify uncertainty, conformal prediction methods gaining continuously more interest and have already been successfully applied to various domains. However, they difficult to apply to time series the autocorrelative structure of time series violates basic assumptions required by conformal prediction. We propose HopCPT, novel conformal prediction approach for time series that not only copes with temporal structures but leverages them. We show that our approach is theoretically well justified for time series where temporal dependencies present. In experiments, we demonstrate that our new approach outperforms state- of-the-art conformal prediction methods on multiple real-world time series datasets from four different domains.
1 Introduction
Uncertainty estimates are imperative to make actionable predictions for complex time-dependent systems (e.g., .neiiing & Katzfuss 2014 Zhu & Laptev 2017). This is particularly evident for environmental phenomena such as flood forecasting (e.g.. Krzysztofowicc (000), since they exhibit pronounced seasonality. Conformal Prediction (CP, Vovk et al. 1999) provides uncertainty estimates based on prediction intervals. It achieves finite-sample marginal coverage with almost no assumptions, except that the data is exchangeable (Vovk et al. 2005 Vovk 2012). However, CP for time series is not trivial because temporal dependencies often violate the exchangeability assumption.
HopCPT. For time series models that predict given system, the errors are typically specific to the input describing the current situation. To apply CP, HopCPT uses continuous Modern Hopfield Networks (MHNs). The MHN uses large weights for past situations that similar the current one and weights close to zero for past situations that were dissimilar to the current one (Figure T The CP procedure the weighted errors construct strong uncertainty estimates close the desired coverage level. This exploits that similar situations (which we call a regime) tend to follow the same error distribution. HopCPT achieves new state-of-the-art efficiency, even under non-exchangeabilit and on large datasets.
Our main contributions are:
1. We propose HopCPT, CP method for time series, domain where CP struggled so far.
2. We introduce the concept of error regimes to CP, which softens the excitability requirements and enables efficient CP for time series.
3. HopCPT uses MHN for similarity based sample reweighting. In contrast existing approaches, HopCPT can learn from large datasets and predict intervals at arbitrary coverage levels without retraining.
4. HopCPT achieves state-of- the-art results for conformal time series prediction tasks from various real-world applications.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

observation
prediction
observation
prediction
value
value
CP interval
MHN
CP
Figure 1: Schematic illustration of HopCPT. The Modern Hopfield Network (MHN) identifies regimes similar to the current one and up- up-weights them (colored lines). The weighted information enriches the conformal prediction (CP) procedure so that prediction intervals can be derived.
5. HopCPT is the first algorithm with coverage guarantees that was applied to hydrological prediction applications domain where uncertainty plays key role in tasks such flood forecasting and hydropower management.
1.1 Related Work
Regimes. In a world with non-linear dynamics, different environmental conditions lead to different error characteristics of models that predict based on these conditions. If we do not account for these different conditions, temporal changes may lead to unnecessarily large prediction intervals, i.c., to high uncertainty. For example, solar energy production is high and stable on a sunny day, fluctuates during cloudy days, and is zero at night. Often, the current environmental condition was already observed at previous points time. The error these time steps is therefore assumed to have the same distribution the current error. Following Quandt (1958) and Hamilton (1990), we call the sets of time steps with similar environmental conditions regimes. Although conditional CP is in general impossible (Foygel Barber al. 2021). we show that conditioning on such regimes can lead to better prediction intervals while preserving the specified coverage. Hammltoon) 1990)) models time series regimes discrete Markov process and conditions classical autoregressive model on the regime states. [Sanquer et al. 2012) use a smooth transition approach to model multi-regime time series. Tajeuna al. (2021) propose an approach to discover and model regime shifts an ecosystem that comprises multiple time series. Further, Masserano et al. (2022) handle distribution shifts by retraining forecasting model with training data from a non-uniform adaptive sampling. Although these approaches are not in a CP setting, their work is similar in spirit, as they also follow the general idea to condition on parts of the time series with similar regimes.
CP and extensions. For thorough introductions to CP, refer the reader the foundational work of Vovk et (1999) and recent introductory paper by [Angelopoulos & Bates (2021) There exist variety of extensions for CP that go "beyond exchangeability" (Vovk et al. 20055). For example, Papadopoulos & Haralambous (001)) apply CP to a nearest neighbor regression setting, Teng al. (2022) apply CP the feature space of models, Angelopoulos to generate uncertainty sets for image classification tasks, and Toccaceli et al. (2017) 7) use a label- conditional variant apply CP biological activity prediction. Of specific interest to us is the research regarding non-exchangeable data of Tibshirani et al. (2019) and Foygel Barber et al. (2022). Both handle potential shifts between the calibration and test by reweighting the data points. Tibshirani et al.] (2019) restrict themselves to settings with full knowledge about the change in distribution; Foygel Barber al. (2022) rely on fixed weights. In our work, we refrain from this assumption because such information is typically not available in time series prediction. Another important research direction is the work on normalized confornity scores (see Fontaan et al., 2023 and references therein). In this setting, the goal is to adapt the conformal bounds through scaling factor in the nonconformity function. The work on normalized conformity scores does not explicitly tailor their approaches to time series.
al. (2020) use CP

---

CP for time series. Gibbs & Candes (2021) and Zaftran et al. (2022) account for shifts in sequential data by continuously adapting an internal coverage target. Adaption based approaches like these are orthogonal to HopCPT and can serve as an enhancement. (Stankevičiüté et al. (2021) use CP in conjunction with recurrent neural networks a multi-step prediction setting, assuming that the series of observations is independent. Thus, no weighting of the scores is required. [Sun & Yu (2022) introduce CopulaCPTS which applies CP to time series with multivariate targets. They conformalize their prediction based on a copula of the target variables and adapt their calibration set in each step. Jensen et al.] (2022) use a bootstrap ensemble to enable CP on time series. NexCP (Foygel Barber et [ uses exponential decay the weighting method, arguing that the recent past is more likely to be of the same error distribution. HopCPT can learn this strategy, but does not a priori commit to it. Xu & Xie (2022a) propose EnbPI, which uses quantiles of the k most recent errors for the prediction interval. Additionally, they introduce a novel leave-one-out ensembling technique. This is specifically geared to setrings with scarce data and difficult use for larger datasets, which is why we do not apply it in our experiments. EnbPI is designed around the notion that near-term errors are often independent and identically distributed and therefore exchangeable. SPCI (Xu & Xie| (022b) softens this requirement by exploiting the autocorrelative structure with a random forest. However, it re-calculates the random forest model each time step, which is computational burden that prohibits its application to large datasets. Our approach relaxes the requirement even further, as we do not assume that the data for the interval computations pertains to the most recent errors.
Non-CP methods Beside CP there exist a wide range of approaches for uncertainty- time series prediction. For example, Mixture Density Networks (Bishop 1994) directly estimate the parameters of mixture of distributions. However, they require a distribution assumption and do not provide any theoretical guarantees. Gaussian Processes (e.g.. et al. 2023 Corani et al., 202T Sun et al. 2022) model time series by calculating posterior functions based on the samples and a prior but computationally limited for large and high dimensional datasets.
Continuous Modern Hopfield Networks. MHN are energy-based associative memory networks. They advance conventional Hopfield Networks (Hopñeld 1982) by introducing continuous queries and states via a new energy function. The new energy function leads to exponential storage capacity, while retrieval is possible with one-step update (Ramsauer et al. (021). Examples for successful
applications of MHN are Widrich al. (2020); |Fürst et al. (2022); ) Dong et al. (2022); [Sanchez (2022); Schäfl et al. (022); and Xu et al. (022).. MHN are
Fernandez et al. (2022); Paischer et al.
related to Transformers Vaswani et al.
2017 as their attention mechanism is closely related to the
association mechanism in MHN. In fact. Ramsauer et al. (022)) show that Transformers attention is special case of the MHN association, at which we arrive when the queries and states are mapped to an associative Hopfield space with the dimensions dg, and the inverse softmax temperature is set to = VTte However, we use the framework of MHN because we want to highlight the associative
memory mechanism as HopCPT directly ingests encoded observations. This perspective further allows HopCPT to update the memory for each new observation. For more details, we refer to Appendix H in the supplementary material.
1.2 Setting
Our setting consists of multivariate time series {E J:)]. t =,.,,,,,7 He E RT a target variable y E R, and a given black-box prediction model j that generates a point prediction ft- j( X,). The input feature matrix X,41 can include all previous and current feature vectors { as well as all previous targets (v.) Our goal is to construct a corresponding prediction interval Ō: (Z++1) - set that includes yt+1 with at least a specified probability a. In its basic form, Z,-1 will only contain Ōt+1, but it can also inherit X or other useful features.
T, with feature vector
Following Vovk et al. (2005), we define the coverage as
Pr (YY+1 Cj (Ze+1) 21-a,
where Y is the random variable of the prediction. An infinitely wide prediction interval is 100% reliable, but not informative of the uncertainty. Thus, CP aims to minimize the width of the prediction interval Co, while preserving the coverage. A smaller prediction interval is called a more efficient interval (Vovk al. 2005) and usually evaluated as the mean of the interval width over the prediction period (PI- Pl-hi.th).
3

---

Standard split conformal prediction takes a calibration set of size which has not been used to train the prediction model ft. For each data sample, calculates the so-called non-conformity score (Vovk) et al. 2005). In regression setting, this score often simply corresponds to the absolute error of the prediction (e.g., Foygel Barber al. 2022). The prediction interval is then calculated based on the empirical 1 - a quantile Qı-a of the calibration scores:
(Z+11) j X ++1) + Q1-o ((1 M (()))))
the data is exchangeable and ji treats the data points symmetrically, the errors on the test set follow the distribution from the calibration. Hence, the empirical quantiles on the calibration and test set will be approximately equal and is guaranteed that the interval provides the desired coverage.
The actual marginal miscoverage ax is based on the observed samples of a test set. If the specified miscoverage differs from the a* in the evaluation, denote the difference as the coverage gap A Cov = a².
The remainder of this manuscript is structured follows: In Section we present HopCPT alongside a theoretical motivation and a synthetic example that demonstrates the advantages of the approach. In Section we evaluate the performance against ofate-of-the-ar CP approaches and discuss the results. Section gives our conclusions and provides an outlook on potential future work.
2
HopCPT
HopCPI copCPiecombbnes combines conformal-style quantile estimation with learned similarity- bassed MHN retrieval.
2.1
Theoretical Motivation
The theoretical motivation for the MHN retrieval stems from Foygel Barber et al. (2022), who introduced CP with weighted quantiles. In the split conformal setting, the according prediction interval is calculated as
1 (X141) = ((X++1) Qi-a Σ apde, + act10+00
where jt represents an existing point prediction model, Q is the r-quantile of a distribution, and 52, is a point mass at |es] (i.c., a probability distribution with all its mass at (e/).. where & are the errors of the existing prediction model defined by e; y/ (.). The normalized weight a, of data sample i is
if 1, else,
a,
w1
+w
where w; are the un-normalized weights of the samples. In the case of wi = wi = 1, this corresponds to standard split CP. Given this framework, Fooyel Barber et al. (2022) show that A Cov
can be bounded in non-exchangeable data setting: Let D) = ((X,, Yi). (X+11, Yz+1)) be dataset where the last entry represents the test sample, and D be permutation D which exchanges
the test sample at t+1 with the i-th sample. Then, A Cov can be bounded from below by the weighted sum of the total variation distances dry between these permutations:
A Cov
a; drvddvvDD D*)
If D is a composite of multiple regimes and the test sample is from the same regime as the calibration sample i, then the distance between D and D' is small. Conversely, the distance might be big if the calibration sample from a different regime. In HopCPT, the MHN association resembles direct estimates of a; dynamically assigning high values to samples from similar regimes.
Appendix B provides an extended theoretical discussion that relates HopCPT to the work of Foygel Barber et al. (2022) and Xu & Xie (2022a), who provide the basis for our theoretical

---

analyses of the CP interval. The latter compute individual quantiles for the upper and lower bound of the prediction interval on the enrors themselves, in contrast standard CP which uses only the highest absolute error quantile. This can provide more efficient intervals in case Elcc şf 0 within the corresponding error distribution. Applying this prineiple to HopCPT where the error distribution conditional to the error regime and assuming that HopCPT can successfully identify these regimes (Appendix: Assumption [B3), we arrive at a conditional asymptotic coverage bound (Appendix: Theorem B.8)
and an asymptotic marginal coverage bound (Appendix: Theorem B.9)
2.2 Associative Soft-selection for CP
We MHH to identify parts of the time series where the conditional emror distribution is similar: For time step - 1, we query the memory of the past and look for matching patterns. The MHN then provides an association vector ar+1 that allows to soft-select the relevant periods of the memory. The selection procedure is analogous to a k-nearest neighbor classifier for hard selection, but it has the advantage that the similarity measure can be learned. Formally, the soft-selection is defined as:
at-1 = softmax (3 m( Zz+1) Wa Wv ml m(Zi:)),
(6)
where m is an encoding network (Section 2.3) that transforms the raw time series features of the current step Z++1 to the query pattern and the memory i: to the stored key patterns; W, and W are the learned transformations which are applied before associating the query with the memory; / is a hyperparameter that controls the softmax temperature. As mentioned above, HopCPT uses the softmax to amplify the impact of the data samples that are likely to follow a similar error distribution and to reduce the impact of samples that follow different distributions (see Section 2.1 This error weighting leads not only to more efficient prediction intervals in our experiments, but can also reduce the miscoverage (Section 3.2.
With the soft-selected time steps we can derive the CP interval using the observed errors c. Following Xu & Xie (2022a), we use individual quantiles for the upper and lower bound of the prediction interval, calculated from the errors themselves. HopCPT computes the prediction interval Oj for time step + in the following way:
(2 ((x++1) + 2 Z(+1), ((+1),((Xx-))) + a(1
a Z+ti)]
(7)
2
where q(r, Z++1) = Q- 1F 1) and F4+1 is a multiset created by drawing n times from [s]]== with corresponding probabilities (4141,1ii11.
2.3 Encoding Network
We embed Z. using a 2-layer fully connected network m² with ReLU activations and enhance the representation by temporal encoding features zyme The full encoding network can be represented as m(Z,) Im"(Z)) || =eimm], where zzime is simple temporal encoding that makes time dependent notions of similarity learnable (e.g.. the windowed approach of EnbPI or the exponentially decaying weighting scheme of NexCP). Specifically, we use ztime = 4. In general, we find that our method is not very sensitive to the exact encoding strategy (e.g., number of layers), which why we kept to a simple choice throughout all experiments.
2.4 Training Procedure
We partition the split conformal calibration data into training and validation sets. Training MHN with quantiles is difficult, which is why we use an auxiliary task: Instead of applying the association mechanism from E directly we the absolute errors as the value patterns of the MHN. This way, the MHN learns to align errors from time steps with similar regime properties. Intuitively. the observed errors from these time steps should work best to predict the current absolute error. We use the mean squared error as loss function (Equation 8) To allow for efficient training, we simultaneously calculate the association of all T time steps within a training split with each other. We mask the association from time step to itself. The resulting association from each step to each step AITTI: ..P,aaa and the loss function Lis
L = T * |[(ELT]| ALTurleur
(8)
5

---

HopCPT la Cov: 0.005 Pi Width:: 20.16)
KNN (a Cov: 0.001 PI PidWidhh: 25.51)
Value
40
20
0-
EnbPI (á Cov: 10.106 Pi-Width: 21.17)
SPC 0.013 Pi-Wisth: 32
Value
40
20
0-
NexCP lA Cov: 0.006 Pi-Width: 33.6)
CP Cov: 0.0424 PI-Width: 28)
Value
40
20
0-
60
100
80
Figure 2: Synthetic example evaluation. HopCPT has the smallest prediction interval width (PI- width), while maintaining the warranted coverage (i.e., A Cov that is positive and close to zero).
This incentivizes the network learn representation such that the resulting soft-selection focuses on time steps with similar enror distribution. Alternatively, one could use loss based on sampling from the softmax. This would correspond more closely to the inference procedure. However, it makes training less efficient because each training step only carries information about single value of eq. In contrast, L leads to more sample-cfficient training. Choosing L assumes that leads to retrieval of errors from appropriate regimes instead of mixing unrelated errors. Section 3.2 and Appendix A.3 provide evidence that this holds empirically.
2.5 Synthetic Example
The following synthetic example illustrates the advantages of the HopCPT association mechanism. We model a bivariate time series D = {(,, yt) )t) While y, serves as the target variable, St represents the feature in prediction. The series is composed of two different regimes: target values are generated by y/2 = 10 + It + NO, dt or y = 10 + I4 + U- 24). 2g is constant within a regime (x and number of time steps from the discrete uniform distribution (1, 25). We create 1,000 time steps and split the data equally into training, calibration, and test sets. We prediction model f.. HopCPT can identify the time steps of relevant regimes and therefore creates efficient prediction intervals while still preserving the coverage (Figure EnbPI, SPCI, and NexCP focus only on the recent time steps and thus fail to base their intervals on information from the correct regime. Whenever the regime changes from small high errors, EnbPI propagates the small error signal and therefore loses coverage. Similarly, its prediction intervals for the small error regime are inefficient (Figure 2 row 3). SPCI, NexCP, and standard CP cannot properly select the relevant time steps, either. They do not lose coverage, but produce wide intervals for all time steps (Figure 2 rows 4 and 5). Lastly, if we replace the MHN with kNN, it can retrieve information from similar regimes. However, its naive retrieval mechanism fails to focus on the informative features because it cannot learn them (Figure 2 row 2).
21, respectively). The regimes alternate. For each regime, sample the
ridge regression model as
2.6 Limitations
HopCPT builds upon the formal guarantees of CP, but it is able to relax the data exchangeability assumption of CP which are problematic for time series data. That said, HopCPT still relies on assumptions on how it learns to identify the respective eтror regimes and exchangablity within regimes (see Section 2. and Appendix B). Our extensive empirical evaluation suggests that these assumptions hold in practice. HopCPT is generally well-suited for very long time series because the memory requirement in the inference scales only linearly with the memory size, Nevertheless, for extremely large datasets it can be the case that not all historical time steps may be kept in the Hopfield memory

---

anymore. HopCPT can, similar to existing methods, disregard time steps by removing the oldest entries from the memory, or alternatively use sub-sampling strategy, On the other hand, for datasets with very scarce data it might be difficult to learn a useful embedding of the time steps. In that case, it could be better to use kNN rather than learning the similarity with MHN (Appendix D). Additional considerations regarding the potential social impact of the method are summarized in Appendix
3
Experiments
This section provides a comparative evaluation of HopCPT and analyzes its association mechanism.
3.1
Setup
Datasets. We use datasets from four different domains: (a) Three solar radiation datasets from the US National Solar Radiation Database (Sengupta et al. 2018). The smallest one consists of time series from different locations over a period of 84 days. This dataset is also used in Xu & Xie (2022a]b). In addition, we evaluate on a 1-year and a 3-year dataset, with 50 time series each. (b) An air quality dataset from Beijing, China (Zhang et al. (017). It consists of 12 time series, each from a different measurement station, over a period of 4 years. The dataset has two prediction targets, the PM10 (as in [uuu& & Xie 2022a b) and PM2.5 concentrations. which evaluate separately. (c) Sap flow measurements from the Sapfluxnet data project (Poyatos et al., 2021). Since the individual measurement series are considerably heterogeneous in length,
subset of 24 time series, each with between 15,000 and 20,000 data points and varying sampling rates. (d) Streamflow, a dataset of water flow measurements and corresponding meteorologic observations from 531 rivers across the continental United States (Newman et al. 2015 Addor et al. 2017). The measurements span 28 years at a daily time scale. For more detailed information about the datasets see Appendix C
Prediction models. We use four prediction models for the solar radiation, the air quality, and the sap flux datasets ensure that our results generalize: random forest, LightGBM, ridge regression, and Long Short-Tern Term Memory (LSTM) (LSTM; Hochreiter & Schmidhuber 1997) model. For the former three models we follow the related work (Xu & Xie| 2022a|b] Foygel Barber et al. 2022) and train a separate prediction model for each individual time series. The random forest and LightGBM models are implemented with the darts library (Herzen al. 2022). the ridge regression model with sklearn (Pedregosa et al., 2011). For the LSTT model, we instead train a global model on all time series of dataset, as is standard for state-of-the-ar deep learning models (e.g., Sr..Kkrsskin et al. 2020) Salinas et al.] 2020 [Smyl 2020). The LSTM is implemented with Py PyTorch (Paszke et al. 2019). For the streamflow dataset, we deviate from this scheme and instead only use the state-of- of-tthe-rt art model, see Appendix C
which is also an LSTM network (Kratzert et al. 2021
Compared approaches. We compare HopCPT to different state-of-the-art CP approaches for time series data: EnbPI (Xu & Xiel 2022a). SPCI (Xu & Xie (022b), NexCP (Foygel Barber et al. (2022). CopulaCPTS (un & Yu, 2022), and AdaptiveC[] Gibbs & Candes 202 lb. In addition, the results of standard split CP (CP) serve
baseline, which, for the LSTM base predictor, corresponds to CF- RNN (Stankevičiüte et al., (02T)) in our setting (one-step, univariate target variable). Appendix A. describes our hyperparameter search for each method. For SPCI, an adaption of the original algorithm was necessary to provide scalability to larger datasets. Appendix A.2 contains more details and an empirical justification. Appendix G evvaluates the addition of AdaptiveCI (Gibbs & Candes 2021) as an enhancement to HopCPT and the other time series CP methods. Appendix[A. gives a comparison [
non-CP methods. Lastly, Appendix presents supplemental comparison to kNN that shows the superiority of the learned similarity representation in HopCPT.
SSap flow refers to the movement of water within plant. In the environmental sciences, it is commonly used proxy for plant transpiration.
'AdaptiveCI works on top of an existing quantile prediction method. Hence, we exclusively make compar- isons based on the LightGBM models that can predict quantiles instead of point estimates. The approach is orthogonal to the remaining compared models and could be combined with HopCPT.
7

---

Table I: Performance of the evaluated CP algorithms for the Solar (3Y), Air Quality (10PM), Sap flow, and Streamflow (SF) datasets. The specified miscoverage level a = 0.1 for all experiments. The column FC specifies the prediction algorithm used for the experiment (Forest: Random Forest, LGBM: LightGBM, Ridge: Ridge Regression, LSTM: LSTM neural network). Bold numbers correspond the best result for the respective metric in the experiment (PI-Width and Winkler score), given that ACov 2 -0 -0.25x, i.e., the specific algorithm reached at least approximate coverage (the result grayed otherwise). The error term represents the standard deviation over repeated runs with different seeds (results without an error term are from deterministic models).
CopulaCPTS
CPICF-RNN
AdaptiveCI
HopCPT
NexCP
EnbPI
SPC
UC
Forest
A Cov
0.029
0.012#0.000
-0.031
-0.002
0.005
0.004
Fi-Wi
39.0 $6.2
#9.0
103.1 103.1+0.1 +0.1
131.1
166.6
174.9
174.6
Winkler
0.73 4..20
1.74
2.47
2.53
2.75
2.76
LGBM
A Cov PI-Width
0.001 4:0.003
0.014#0.000
-0.023
-0.002
0.006
0.006
0.001
Solar 3Y
37.7
102.2#0.1
133.6
159.9
169.8
170.2
67.1
Winkler
0.57#0.01
1.75
2.52
2.55
2.80
2.81
1.19
Ridge
A Cov
0.040=t0.001
0.00220.000
-0.074
-0.001
0.004
0.005
PI- PI-Width
44.9
108.240.0
131.1
171.0
166.0
167.7
Winkler
0.64
1.82
2.49
2.66
2.73
2.74
LSTM
A Cov
0.001 1:0,006
0.014
-0.018
-0.001
0.007
0.007
PI-Width
17.9#0.6
27.8#0.00
24.6
28.2
31.9
33.0
Winkler
0.30#0.01
0.62
0.64
0.63
0.68
0.70
Forest
A Cov
0.028
0.000=0.000
-0.066
-0.004
-0.019
-0.033
PI-Width
93,9#11.1
118.5=0.1
202.8
263.5
243.1
229.8
Winkler
1.50
2.23
4.16
4.03
4.94
4.98
Air 10 PM
LGBM
A
0.01740.016
0.02320.000
-0.057
-0.004
-0.017
-0.028
-0.001
PI-Width
85.6*7.4
113.2
178.3
224.8
206.7
196.5
186.4
Winkler
1.45
1.9.
3.69
3.64
4.33
4.36
3.00
Ridge
A Cov
0.010
0.024
-0.045
-0.002
0.010
0.012
Width
79.9
93,9
120.0
153.3
153.7
155.3
Winkler
1.35
L.520
2.60
2.68
2.95
2.96
LSTM
A Cov
0.00240.005
0.010=0.000
-0.025
-0.002
0.001
0.004
PI-Width
62.7#1.5
62.240.0
58.1
62.4
61.8
63.0
Winkler
1.33
1.21 tb.00
1.32
1.29
1.34
1.34
Forest
A Cov
00009
0.007
-0.042
0.000
0.014
0.005
PI-Width
1078.7+78.7
1741.8+2.4
3671.6
6137.1
7131.1
7201.5
Winkler
#0.01
0.000
1.24
1.56
1.76
1.80
LGBM
A Cov
-0.016#0.013
0.003=0.000 00000
-0.040
-0.003
0.006
-0.007
0.010
Sap How
PI-Width
875.5=449.8
1582.3#14
2924.1
4805.3
5588.5
5614.8
6273.5
Winkler
0,27
0.49
0.96
1.25
1.43
1.46
1.50
A Cov
-0.025
0.0225+0.023 F0.023
-0.241 #0,000
-0.041
-0.015
-0.251
-0.358
Ridge
PI-Width
1639.6 493.2
2060.5+1.4
3117.5
10628.9
8943.3
7148.7
Winkler
0.46 40.05
4..00
0.92
2.42
3.31
5.85
LSTM
A Cov
0.004 0.004 400.04
0.000 0,000
-0.019
-0.000
-0.022
-0.042
PI-Width
#94.3#7-7
628.22 200.
768.0
990.0
903.9
817.2
Winkler
0.19
0.24
0.28
0.32
0.35
0.36
LSTM
A Cov
40:04
0.027=0.000
-0.054
-0.000
0.005
0.009
SF
PI-Width
1.39 10.17
1.58 10000
1.55
1.94
1.99
2,08
Winkler
0.79
0.91 1:000
1.27
1.21
1.28
1.29

---

Metrics In our analyses, we compute A Cov, PI-Width, and the Winkler score (Winkler 1972) per time series and miscoverage level. The Winkler score jointly elicitates miscoverage and interval width in single metric:
(Z+1)) + [1+41 - Cg. (1+41)
if yc+1 > (z+4(Z))),
WS, 26+1) =
(Z + [ (Z(-) Mt+1))
if Mt+11
Go.l (Z+t))
((++)),
(9)
IW7 (Z(+1)
else.
The score is calculated per time step and miscoverage level a. It corresponds to the interval width IW; Č.a Craud whenever the observed value y/ is between the upper bound Cr. (Z(+1) and the lower bound Cr ( (Z++1) of C, 1. If y++1 is outside these bounds, a penalty is added the interval width. We evaluate the mean Winkler score over all time steps.
We repeated each experiment with 12 different seeds. For brevity, we only show the mean performance of one dataset per domain for a = 0.1 in the main paper (which is the most commonly reported
value in the CP literature; e.g., Xu & Xie, 2022b Foygel Barber al., 2022 Gibbs & Candes 2021). Appendix A.3
presents additional results for all datasets and more a levels.
3.2 Results & Discussion
HopCPT has the most efficient prediction intervals for each domain - with only one exception (Table[T significance tested with a Mann-Whitncy U test at p < 0.005) for the evaluated miscoverage level (a = 0.1). )). In multiple experiments (Solar (3Y), Solar (IY)), the HopCPT prediction intervals less than half wide those of the approach with the second-smallest PI-Width. The second- most efficient intervals are predicted most often by SPCI. This ranking also holds for the Winkler score, where HopCPT achieves the best (i.e., lowest) Winkler scores and SPCI ranks second in most experiments. Notably, these results reflect the increasing requirements posed by each uncertainty estimation method on the dataset (Section
Furthermore, HopCPT outperforms the other methods regarding both Winkler score and PI- Width when we evaluate over additional datasets and at different miscoverage levels (see Appendix AA.3) HopCPT is the best-performing approach in the vast majority of cases. The variation in size of the different solar datasets (results for Solar (1Y) and Solar (3M) in Appendix AA.) shows that HopCPT especially increases its lead when more data is available. We hypothesize that this is because the MHN can learn more generalizable retrieval patterns with more data. However, data scarcity is not limitation of similarity of HopCPT, which replaces the MHN with kNN retrieval, performs best on the smallest solar dataset, as we demonstrate in Appendix D
CP in general. In fact, a simplified and almost parameter- freee variation
Most approaches have coverage gap close zero in almost all experiments they approximately achieve the specified marginal coverage level. This is also reflected by the fact that the ranking of Winkler scores and PI-Widths agree in most evaluations. Appendix A.4 provides supplementary analysis of the local coverage. The results for non-CP methods show a different picture (see Appendix A.3 and Table 7}. Here, the non-CP models most often exhibit very narrow prediction intervals at the cost of high miscoverage.
Interestingly, standard CP also achieves good coverage for all experiments at the cost of inefficient prediction intervals. There one notable exception: for the Sap flow dataset with ridge regression, we find A Cov = -0. -0.358. In this case, we argue that the bad performance of standard CP is driven by strong violation of the (required) exchangcability assumption. Specifically, a trend in the errors leads to distribution shift over time (as illustrated in Appendix AA.3) Figure 4). HopCPT and NexCP handle this shift without substantial loss in coverage. The inferior coverage of SPCI is likely influenced by the modification to larger datasets (see Appendix A.2).
Performance gaps between the approaches differ considerably across datasets, but are generally consistent across prediction models. The biggest differences between the best and worst methods exist Solar (3Y) and Sap flow, likely due to the distinctive regimes in these datasets. The smallest (but still significant) differences are visible in the Streamflow data. On this dataset, we also evaluated the ofate-of-the-ar non-CP uncertainty model in the domain (Klotz et al. 2022, see Appendix (A.3 and found that HopCPT outperforms it with respect to efficiency.

---

End of calibration period
Value
high
Association weight
400
200
low
9
Figure 3: Exemplary visualization of the 30 highest association weights that the MHN places on previous time steps (some are before the visualized part of the memory). HopCPT retrieves similar peak values when estimating peak, and it retrieves similar small values when estimating at small value.
In terms of computational resources, SPCI is most demanding followed by HopCPT, as both methods require a learning step in contrast to the other methods. A detailed overview can be found in Appendix|A.6
To assess whether HopCPT learns meaningful associations within regimes, we conducted a qualitative study on the Solar (3Y) dataset. Figure shows that HopCPT retrieves the most highly weighted errors from time steps with similar regimes. The illustrated weighting at the time step with a low prediction value retrieves previous time steps which are also low-valued regimes. Similarly, Figure (Appendix A.3) suggests that the learned distinction corresponds to the error regimes, which crucial for HopCPT.
4
Conclusions
We have introduced HopCPT, a novel CP approach for time series tasks. HopCPT uses continuous Modern Hopfield Networks to construct prediction intervals based on previously seen events with similar error distribution regimes. We exploit that similar features lead to similar errors. Associating features with emrors identifies regimes with similar error distributions. HopCPT learns this association in the Modern Hopfield Network, which dynamically adjusts its focus on stored previous features according to the current regime.
Our experiments with established and novel datasets show that HopCPT achieves state of the art: It generates more efficient prediction intervals than existing CP methods and approximately preserves coverage, even in non-exchangeable scenarios like time series. HopCPT comes with formal guarantees within the CP framework for uncertainty estimation in real-world applications such as streamflow prediction. Furthermore, HopCPT scales well to large datasets, provides multiple coverage levels after calibration, and shares information across individual time series within a dataset.
Future work comprises: (a) Drawing from multiple time series during the inference phase. HopCPT already trained the whole dataset once, but it might be advantageous to leverage more information during inference as well. (b) Investigating direct training objectives from which learning- based CP might benefit. (c) Using HopCPT beyond time series, as non-exchangeability is also an issue in other domains which limits the applicability of existing CP methods.
Acknowledgements
We would like to thank Angela Bitto- Mito-Neeliig for discussions during the development of the method. The ELLIS Unit Linz, the LIT AI Lab, the Institute for Machine Learning, are supported by the Federal State Upper Austria. thank projects AI-MOTION (LIT-201 8-6-886--0-0 YOU-212). DeepFlood Medical Cognitive Computing Center (MC3), INCONTROL-RI (FFG-881064) PRIMAL (FFG-873979), S3AI (FFG-872172), DL for GranularFlow (FFG-871302), EPILEPSIA (FFG-892171), AIRI FG 9-N (FWF-36284, 899943), INTEGRATE (FFG-89 (FFG-892418), ELISE (H2020-ICT-2019--
Y-UU--11,,,
FWF-36235), AIAGreenHeeaiingGrrdtHHHHH ID: 951847), SatersWaters COORI/ON-C16&2221-C...MAMM
thank AudiJKU Deep Learning Center, TGW LOGISTICS GROUP GMBH, Silicon Austria Labs (SAL), FILL Gesellschaft mbH. Anyline
10

---

GmbH, Google, ZF Friedrichshafen AG, Robert Bosch GmbHI UCB Biopharma SRL, Merck Healthcare KGaA, Verbund AG, GLS (Univ. Waterloo) Software Competence Center Hagenberg GmbH, TÚV Austria. Frauscher Sensonic, TRUMPF and the NVIDIA Corporation. The computational results presented have been achieved in part using the Vienna Scientific Cluster (VSC) and the In-Memory Supercomputer MACH-2 (operated by the Scientific Computing Administration of JKU Linz).
References
Abbott, L. F. and Arian, Y. Storage capacity of generalized networks. Physical Review A, 36: 5091-5094, 1987. doi: (0.1009PhysRevA336.500
Addor, N., Newman, A. J., Mizukami, N., and Clark, M. P. The CAMELS data set: catchment attributes and meteorology for large-sample studies. Hydrology and Earth System Sciences (HESS), 21(10):5233-5313, 2017.
Angelopoulos, A., Bates, S., Malik, J., and Jordan, M. Uncertainty sets for image classifiers using conformal prediction. arXiv preprint (tt:::2009.149999 2020.
Angelopoulos, A. N. and Bates, gentle introduction conformal prediction and distribution-fies free uncertainty quantification. arXiv preprint arXiv:2107.075// 2021.
Baldi, P. and Venkatesh, S. Number of stable points for spin-glasses and neural networks of higher orders. Physical Review Letters, 58:913-916, 1987. doi: 10. Rev.00/
Bishop, C. M. Mixture density networks. Technical report, Neural Computing Research Group, 1994.
Caputo, B. and Niemann, H. Storage capacity of kernel associative memories. In Proceedings of the International Conference Artificial Neural Networks (ICANN), pp. 51-56, Berlin, Heidelberg, 2002. Springer-inneeeee Verlag.
Chen, H. H., Lee, Y. C., Sun, G. Z., Lee, H. Y., Maxwell, T., and Giles, C. L. High order correlation model for associative memory. AIP Conference Proceedings, 151(1):86-99 1986. doi: 10.106 36224.
Corani, G., Benavoli, A., and Zaffalon, M. Time series forecasting with gaussian processes needs priors. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 103-117. Springer, 2021.
Dong, 7., Chen, 7., and Wang. Q. Retrosynthesis prediction based on graph relation network. In 2022 15th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BME)), pp. 1-5. IEEE, 2022.
Fontana, M., Zeni, G., and Vantini, S. Conformal prediction: a unified review of theory and new challenges. Bernoulli, 29(1):1-23, 2023.
Fox, M. and Rubin, H. Admissibility of quantile estimates of a single location parameter. The Annals of Mathematical Statistics, pp. 1019-1030, 1964.
Foygel Barber, R., Candes, E. Ramdas, A., and Tibshirani, R. J. The limits of distribution-free conditional predictive inference. Information and Inference: A Journal of the IMA, 10(2):455-42,, 2021.
Foygel Barber, R., Candes, E. J., Ramdas, A., and Tibshirani, R. J. Conformal prediction beyond exchangeability. arXiv preprint arXiv:2202.134111 2022.
Fürst, A., Rumetshofer, E., Lehner, J., Tran, V. T., Tang, F., Ramsauer, H., Kreil, D. P., Kopp. M. K., Klambauer, G., Bitto-Nemling, A., and Hochreiter, S. CLOOB: Modern Hopfield Networks with infoLOOB outperform CLIP. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022.
Gal, Y. and Ghahramani, Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference machine learning, pp. 1050-1059, PMLR, 2016.
11

---

Gardner, E. Multiconnected neural network models. Journal of Physics A, 20(11):3453- 3464, 1987. doi: 10.10088/0:055444702001111111 1/046.
Gibbs, I. and Candes, E. J. Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems, 34: 1660-1677, 2021.
Gneiting, T. and Katzfuss, M. Probabilisticc forecasting, Annual Review of Statistics and Its Applica- lion, 1(1):125-151, 2014. doi: 10. 1466//mmueessaaiiiii-067773-33-3
Hamilton, J. D. Analysis of time series subject to changes in regime. Jourmal of econometriex, 45 (1-2):39-70, 1990.
Herzen, J., Lässig, F., Piazzetta, S. G., Neuer, T., Tafti, L., Raille, G., Van Pottelbergh, T., Pasicka, M., Skrodzki, A., Huguenin, N., et al. Darts: friendlyy modern machine learning for time series. Journal of Machine Learning Research, 23(124):1-6, 2022.
Hochreiter, S. and Schmidhuber, Long short-term memory. Neural Comput., 13): 1997.
Hopfield, J.J. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79(8):2554-2558 1982.
Hopfield, J. J. Neurons with graded response have collective computational properties like those of two-state neurons. Proceedings of the National Academy of Sciences, 81(10):3088-300 1984. doi: 10. 10.1173/pnas.81.11 81.10.388
Horn, D. and Usher, M. Capacities of multicooneected memory models. Journal of Physics France, 49(3):389-393 1988. doi: 10. 10.1151//phyx0011880499000000000
Jensen, V., Bianchi, F. M., and Anfinsen, probabilistic time series forecasting. arXiv preprint arXiv:2202.08875 2022.
Ensemble conformalized quantile regression for
Klotz, D., Kratzert, F., Gauch, M., Keefe Sampson, A., Brandstetter, J., Klambauer, G., Hochreiter, S., and Nearing, G. Uncertainty estimation with deep learning for rainfall-runoff modeling. Hydrology and Earth System Sciences, 26(6):1633-1633 2022. doi: 10.5194/hess-26- 1673-2022.
Kratzert, F,, Klotz, D., Shalev, G., Klambauer, G., Hochreiter, S., and Nearing, G. Towards learning universal, regional, and local hydrological behaviors via machine learning applied to large-sample datasets. Hydrology and Earth System Sciences, 23(12):5089-5110 2019. doi: 23094/hess-235009-9
Kratzert, F., Klotz, D., Hochreiter, S., and Nearing, G. S. A note leveraging synergy in multiple meteorological data sets with deep learning for rainfall-runofi modeling. Hydrology and Earth System Sciences, 25(5):2685 2703, 2021.
Kratzert, F,, Gauch, M., Nearing, G., and Klotz, D. NeuralHydrology - a Python library for deep learning research in hydrology. Journal of Open Source Software, 7(7 ()71)),050, 2022. doi: 10.21 105/00ss.04050.
Krotov, D. and Hopfield,
Dense associative memory for pattern recognition. In Lee, D. D., Sugiyama. M., Luxburg, U. V., Guyon, I., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, pp. 1 1172-1180. Curran Associates, Inc., 2016.
Krzysztofowicz, R. The case for probabilistic forecasting in hydrology. Journal of hydrology, 249 (1-4):2-9, 2001.
Lei, J. and Wasserman, L. A. Distribution-fra prediction bands for non parrmmaeecticc regression. Journal of the Royal Statistical Society: Series (Statistical Methodology). 76, 2014.
Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net 2019.
12

---

Masserano, L., Rangapuram, S. S., Kapoor, S., Nirwan, R. S., Park, Y., and Shhlke-Schneider, M. Adaptive sampling for probabilistic forecasting under distribution shift. In NewrIPS 2022 Workshop on Distribution Shifts: Connecting Methods and Applications, 2022.
Maurer, E. P., Wood, A. W., Adam, J. C., Lettenmaier, D. P., and Nijssen, B. A long-term hydrologi- cally based dataset land surface fluxes and states for the conterminous United States. Journal of climate, 15(22):327733251 2002.
Newman, A. Clark, M. P., Sampson, K., Wood, A., Hay, L. E., Bock, A., Viger, R. J., Blodgett, D., Brekke, L., Arnold, J. R., Hopson, T., and Duan, Q. Development of a large-sample watershed- scale hydrometeorrologica data set for the contiguous USA: data set characteristics and assessment of regional variability in hydrologic model performance. Hydrology and Earth System Sciences, 19(1):209-223 223, 2015. doi: 10.5 10.5194/hess-
Newman, A. J., Mizukami, N., Clark, M. P., Wood, A. W., Nijsee, B., and Nearing, G. Benchmarking of a physically based hydrologic model. Journal of Hydrometeorrology, 18(3):2215-2225,2222 2017 doi: 10.1175/JHM-DD 1-M-----1-028.1.1
Oreshkin, B. N.. Carpov, D., Chapados, N., and Bengio, Y. N-beats: Neural basis expansion analysis for interpretable time series forecasting. In International Conference on Learning Representations, 2020.
Paischer, F., Adler, T.. Patil, V., Bitto-Nemling, A., Holzleitner, M., Lehner, S.. Eghbal-zzachh, H., and Hochreiter, S. History compression via language models in reinforcement learning. In Chaudhuri, K., Jegelka, S., Song, I.., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 17156-17185 PMLR, 17-23 Jul 2022.
Papadopoulos, H. and Haralambous, H. Reliable prediction intervals with regression neural networks. Neural Networks, 24(8):842-851, 2011.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2888 2830, 2011.
Pinsker, M. Information and information stability of random variables and processes. Holden-Dag -Day 1964.
Poyatos, R., Granda, V., Flo, V., Adams, M. A., Adorján, B., Aguadé, D., Aidar, M. P., Allen, S., Elvarado-Barriente M. S., Anderson- Anderson-Teixeiaa, K. J., al. Global transpiration data from sap flow measurements: the sapfluxnet database. Earth system science data, 13(6))2607-266) 2021.
Psaltis, D. and Cheol, H. P. Nonlinear discriminant functions and associative memories. AIP Conference Proceedings, 151()):370-375 1986. doi: 10.1063/1.36241.
Quandt, R. E. The estimation of the parameters of linear regression system obeying two separate regimes. Journal of the american statistical association, 53(284):873-88) 880, 1958.
Ramsauer, H., Schäfl, B., Lehner, J.. Seidl, P., Widrich, M., Gruber, L.,, Holzleitner, M., Pavlović, M., Sandve, G. K., Greiff, V., Kreil, D., Kopp, M., Klambauer, G., Brandstetter, J., and Hochreiter, Hopfield networks is all you need. In 9th International Conference on Learning Representations (ICLR), 2021.
Salinas, D., Flunkert, V., Gasthaus, J., and Januschowski, T. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, (1(3):111-1199) 2020.
Fencade-Fermaneez, A., Rumetshofer, E., Hochreiter, S., and Klambauer, G. CLOOME: contrastive learning unlocks bioimaging databases for queries with chemical structures. In NewrIPS 2022 Women in Machine Learning Workshop,
13

---

Sanquer, M., Chatelain, F., El-Guedri, M., and Martin, N. A smooth transition model for multiple- regime time series. IEEE transactions on signal processing, 61(7):183-11877 2012.
Schäfl, B., Gruber, L., Bitto-Nemllngg A., and Hochreiter, S. Hopular: Modern Hopfield Networks for tabular data, arXiv preprint arXiv:2206. 00644, 2022.
Sengupta, M., Xie, Y., Lopez, A.. Habte. A., Maclaurin, G., and Shelby, J. The national solar radiation data base (NSRDB). Renewable and sustainable energy reviews, 89:51-60, 2018.
Smyl, S. A hybrid method of exponential smoothing and recurrent neural networks for time series forecasting. International Journal of Forecasting, 36(1):75-85, 2020.
Stankevičiüté, K., M Alaa, A., and der Schaar, M. Conformal time-series forecasting. Advances in Neural Information Processing Systems, 34:6216-6228, 2021.
Sun, S. and Yu, R. Copula conformal prediction for multi-step time series forecasting, ArXiv, abs/2212.03281, 2022.
Sun, X., Kim, S., and Choi, J.-I. Recurrent neural network-induced gaussian process. Neurocomputiigg 509:75-84, 2022.
Tagasovska, N. and Lopez-Paz, Single-model uncertainties for deep learning. Advances Neural Information Processing Systems, 32, 2019.
Tajeuna, E. G., Bouguessa, M., and Wang. S. Modeling regime shifts in multiple time series. arXiv preprint ar arXiv:2109.0966 2021.
Teng, J., Wen, C., Zhang, D., Bengio, Y., Gao, Y., and Yuan, Y. Predictive inference with feature conformal prediction. arXiv preprint arXiv:22/0.00177 2022.
Thornton, E., Running, S. W., White, M. A., et al. Generating surfaces of daily meteorological variables over large regions of complex terrain. Journal of hydrology, 190(3-4):214-2511 1997.
Tibshirani, R. J., Foygel Barber, R., Candes, E. J., and Ramdas, A. Conformal prediction under covariate shift. Advances in Neural Information Processing Systems, 32, 2019.
Toccaceli, P., Nouretdinov, I., and Gammerman, A. Conformal prediction of biological activity of chemical compounds. Annals of Mathematics and Artificial Intelligence, 81(1):105-122 123, 2017.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S.. and Garnett, R. (eds.), Advances in Neural Information Processing Systems 30, pp. 5998-6008. Curran Associates, Inc., 2017.
Vovk, V. Conditional validity of inductive conformal predictors. In Asian conference machine learning, pp. 475 490. PMLR, 2012.
Vovk, V., Gammerman, A., and Saunders, C. Machine-leaming applications of algorithmic random- ness. Bratko, I. and Dzeroski, (eds.), Proceedings of the Sixteenth International Conference on Machine Learning (ICML 1999), Bled, Slovenia, June 27 30, 1999, pp. 444-453. Morgan Kaufmann, 1999.
Vovk, V., Gammerman, A., and Shafer, G. Algorithmic learning in random world. Springer Science & Business Media, 2005.
Widrich, M., Schäfl, B., Pavlović, M., Ramsauer, H., Gruber, L., Holzleitner, M., Brandstetter, J., Sandve, G. K., Greiff, V.. Hochreiter, S., and Klambauer, G. Modern Hopfield Networks and attention for immune repertoire classification. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2020.
Winkler, decision-theoretto approach to interval estimation. Journal of the American Statistical Association, 177331) 1972.
14

---

Xia, Y., Mitchell, K., Ek, M., Sheffield. J., Cosgrove, B., Wood, E.. Luo. L., Alonge, C., Wei, H., Meng, J., al. Continental water and energy flux analysis and validation for the North American Land Data Assimilation System project phase 2 (NLDAS-2): 1. intercomparison and application of model products. Journal of Geophysical Research: Atmospheres, 117(D3), 2012. 2012.
Xu, C. and Xie, Y. Conformal prediction for time series. arXiv preprint arXiv:2010.09107, 2022a.
Xu, C. and Xie, Y. Sequential predictive conformal inference for time series. arXiv preprint arXiv:2212.03465 2022b.
Xu, Y., Yu, W., Ghamisi, P., Kopp, M., and Hochreiter, S. Tx12Img-MHN: Remote sensing image generation from text using Modern Hopfield Networks. arXiv preprint arXiv:2208, 22Xiv:2208004441,, 2022.
Zaffran, M., Féron, O., Goude, Y., Josse. J., and Dieuleveut, A. Adaptive conformal predictions for time series. In International Conference on Machine Learning, pp. 35834-25866 PMIR,, 2022.
Zhang. S., Guo, B., Dong, A., He, J., Xu, Z., and Chen, S. X. Cautionary tales on air-quality improve- ment in Beijing. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 47302205):2010455, 2017.
Zhu, L. and Laptev, N. Deep and confident prediction for time series at Uber. In 2017 IEEE International Conference Data Mining Workshops (ICDMW), pp. 103-110. IEEE, 2017.
Zhu, X., Huang, L., Lee, E. H., Ibrahim, C. A., and Bindel, D. Bayesian transformed gaussian processes. Transactions on Machine Learning Research, 2023.
15

---

A Extended Experimental Setup & Results
A.1 Hyperparameter Search
We conducted an individual hyperparameter grid search for each predictor-datase combination. For methods that require calibration data (HopCPT, AdaptiveCI, NexCP), each calibration was split in half: One part served as actual calibration data while the other half was used for validation As EnbPI requires only the k past points, we used the full calibration set minus these l points for validation, so that it could fully exploit the available data. used for the search.
all sets of hyperparameters
Table Parameters used in the hyperparameter search of the uncertainty models
Method
Parameter
Value
Learning Rate Dropout
0.01. 0.001 0, 3.0.25, 0.5 yes/no
HopCPT
Time Encode
Mode
simple, momentum 0.002, 0.005, 0.01, 0.02
AdaptiveCT
200, 150, 125, 100, 75, 50, 25, 10
EnbPI
Window Length
0.999, 0.995, 0.993, 0.99, 0.98, 0.95, 0.90
NexCP
0.025, 0.05, 0.10, 0.15,
KNN
k-Top Share
0.20, 0.25, 0.30, 0.35
Learning Rate
0,005, 0.001, 0.0001 0.1, 0.25, 0.5
Dropout
MCD
Batch Size
512, 256
Hidden Size
64, 128, 256
Learning Rate
0.005, 0.001, 0.0001
Dropout
0, 0.1, 0.25
Gaus
Batch Size
512, 256
Hidden Size
64, 128, 256
Learning Rate
0.005, 0.001, 0.0001
Dropout
0, 0.1, 0.25
MDN
Batch Size
512, 256
Hidden Size
64, 128, 256
# Components
3.
Model selection. Model selection was done uniformly for all algorithms: As first step, all models with a negative A Cov on the validation set were excluded. From the remaining models, the model with the smallest PI-Width model with the highest A Cov was selected.
selected. In cases where model achieved non-negative A Cov, the
HPpCPT. HopCPT trained for 3,000 epochs in each experiment. We chose this number to make sure that the loss and model selection metric curves are already converged. Throughout training, validated every 5 epochs and selected the model that best fulfilled the model selection criteria described above. AdamW (Loshchilov & Hutter] 2019) with standard parameters (31 = 0.9, 32 0.999, 0.01) used as optimizer. The learning rate was part of the hyperparameter search. Depending on the dataset size, the batch size was set to 2 or 4, where batch size of n would mean that the full training part of the calibration set of n time series used.
Note that this is only the case in the hyperparameter search. In the evaluation, the full calibration set was used for calibration.
16

---

The computational demand of SPCI did not allow to conduct a full hyperparameter search for the window length parameter (see more in Section A.2p. Since SPCI applies a random forest on the window, one can assume that it is capable to find the relevant parts of the window. On top of that, longer window has less risk than window that is too short and (potentially) cuts off relevant information. Hence, we set the window length to 100 for all experiments, which corresponds to the SPCI with window length 25 on all but the largest two datasets (Table and Table 5 We found bardly any differences except for the smallest datasets Solar (3M), In that case we reason that the result is due to the limited calibration data in this setting.
SPCI.
longest setting in the original paper. To check our reasoning, we evaluated the performance
MCD/MDN/Gauss. In Appendix A.33 we compare HopCPT to non-CP architectures that can provide prediction interval estimates. Specifically, we use Monte Carlo Dropout (MCD; Gal & Ghahramani 2016), a Mixture Density Network (MDN; Bishop 1994), and a Gauss model (i.c., an MDN with single Gaussian density). An LSTM serves backbone for all models. Therefore, there is no distinction between the predictor model and the uncertainty model. To allow for a fair comparison to CP approaches, the models are trained on both the predictor training data and the calibration data combined. None of these approaches are CP methods, and MCD was specifically devised estimate only epistemic uncertainty. Nevertheless, we consider these methods as useful comparisons, since they are widely used for uncertainty quantification. The models were trained for 150 epochs with AdamW (Loshchilov & Hutter, 2019) (using its standard parameters: 33 0.9, 32 = 0.999, 8 = 0.001). We chose this number to ensure that the loss and model selection metrics already converged. Throughout the training, we validated after every epoch and selected the model that best fulfilled the model selection criteria described above. The learning rate was part of the hyperparameter search.
Global LSTM. We conducted an individual hyperparameter tuning of the global LSTM prediction model for each dataset. T the hyperparameters used for the grid search. Similar the LSTM uncertainty models, we trained for 1550 epochs with an AdamW optimizer (Loshchilov & Hutter 2019) and validated after each epoch. We optimized for the mean squared error and selected the model with the smallest validation mean squared error.
Table 3: Parameters used in the hyperparameter search of the global LSTM prediction model.
Parameter
Value
Learning Rate
0.005, 0.001, 0.0001
Dropout
0, 0.1, 0.25
Batch Size
512, 256
Hidden Size
64, 128, 256
A.2 SPCI Retraining
As our results show, SPCI is very competitive (see, for example, Section 3.2 Xu & Xie (022b)
did, however, design SPCI so that its random forest regressor is retrained after each prediction step. This design choice is computationally prohibitive for larger datasets. To nevertheless allow comparison against SPCI on the larger datasets, we modified the algorithm so that the retraining is skipped. Experiments on the smallest dataset 6) show only small performance decrease with the modified algorithm. One limitation of the adapted algorithm is, however, that a strong shift in the error distribution(s) would potentially require a retraining the new distribution. A viable proxy to detect such change in our setting is the coverage performance of standard CP. The reason for this is that standard CP predictions are solely based on the calibration data and can thus not account for shifts. In the experiments (Section 3 and Appendix A.3). standard CP achieves good coverage with only one exception (see Section 3p. Hence, we decided to include SPCI (in the modified version) to enable a comparison to it on larger datasets.
17

---

Table 4: Performance of SPCI with an input window of length 100 and length 25 on the solar datasets. The differences in performance are small compared to the differences across methods (Section The error term represents the standard deviation over repeated runs.
Window
100
25
Data
FC
A Cov
0.045
0.047
Forest
PI-Width
97.240.1
97.8=00.3
Winkler
1.26#0.00
1.25#0.00
A Cov
0.045=0.000
0.047
Solar 1Y
LGBM
PI- Width
96.6-40.2
97.6.40.1
Winkler
1.26
1,26:10.00
A Cov
0.031#0.000
0.031
#0.000
0.030
Ridge
PI-Width
112.g:10.2
113.740.1
Winkler
1.40
1.42#0.000
Cov
0.018=0.000
0.01g:00000
LSTM
P1-Width
22.5+000
22,2#0.00
Winkler
10.00
0.40
Cov
06.0000002
0.00440..00
Forest
PI-Width
38.84004
43.2
Solar Small
Winkler
1.82
1.53+0.00
Cov
0.052#0.002
0.027
LGBM
P1-Width
37.4
43.3
Winkler
1.8420.01
1.6340.00
A Cov
-0.05520.002
-0.05740.001
Ridge
PI-Width
51.9
52.8=0.2
Winkler
1.91
1.83+0.00
101
End of calibration period
-1
-2
-6000
-400 -2000
0
2000
4000
6000
Figure 4: Time series of the prediction error e for the ridge regression model on the Sap flow dataset. The time series spans the calibration (t 0) and test > 0) data. The red line (fitted by the least squares method) shows a strong trend in the error distribution.
18

---

Table 5: Performance of SPCI with input window of length 100 and length 25 on the different non-solar datasets. The differences in performance are small compared to the differences across methods (Section 3.2 The error term represents the standard deviation over repeated runs.
Window
Data
FC
0.008 10.0000000
0.000:00.000
Cov
Forest
PI Wi-th
118.5400.0
118.540.1
Winkler
2.23#00.0
2.23#0.00
Air 10 PM
Cov
0.02/=0.000
0.023=00000
LGBM
PI- Fi-th
113,2
113.2 113.2
Winkler
1.9420.00
1.94
A Cov 0.02420.000
0.024
Ridge
P1-Width 93.9
93.8
Winkler 1.5220.00
1.5240.00
A Cov
0.010#0.000
0.010#0.000
LSTM
PI-Width
62.3
62.3
Winkler
1.21 +0.00
1.21
A Cov
0.009
0.000
Forest
PI-Width
81.540.0
81.4
Winkler
2.02#0.00
2.02#0.00
Air PM
A Cov
0.002#0.001
0.001
LGBM
PI- Fidth
73.3
73.3
Winkler 1.84 +0.00
1.8000.000
Cov
0.010#0.000
+0.000
Ridge
PI- PI-Width
65.5
65.4
Winkler
1.40
1.40
Cov
0.017=00000
0.01720.000
LSTM
PI-Wi P1-Width
32.4
32.340.0
Winkler
4.000
0.92
Cov
0.007
0.007
Forest
PI- PI-Width
1743.1
1742.7#1.2
Winkler
0.59
0.5g
Cov
0.003=0.000
0.003
Sap flow
LGBM
P1-Width
1581.9+0.7
1583,6#1.
Winkler
0.49
0.4g
Cov 0.241
0.242210.000
Ridge
PI-Width
2061.5:114
2066.8#1.2
Winkler
2.52#0.00
2.52
A Cov
0.00/
0.006
LSTM
PI Width
628.6tt0.8
631.9
Winkler
0.24
0.24
19

---

Table 6: Performance of the original SPCI algorithm (Retrain) and the modified version (No Retrain) on the Solar (3M) dataset. Performance in terms of the evaluation metrics is similar compared to the differences between methods (Section 3.2). The computational demand of the modified version is considerably lower. The error term represents the standard deviation over repeated runs.
UC
No Retrain
Retrain
FC
A Cov
0.074420.001
0.049#0.001
0.05
PI-Width 58.0
62.4
Winkler 2.62#0.03
2.28
Forest
A Cov 0.0644.0.002
0.045
0.10 PI-Width
38.8
41.6
Winkler 1.82=0.0
1.67=0.00
A Cov
0.050
0.038
0.15
PI-Width
26.8+0.2
28.9
Winkler 1.44=0.02
1.35=0.00
A Cov 0.06240.001
0.061 #0.001
0.05
PI-Width
56.1
58.2
Winkler 2.65
2.51 =0.01
LGBM
A Cov 0.052+t0.002
0.049
0.10 PI-Width
37.4
39.0
Winkler 1.84=0.01
1.78=0.01
A Cov 0.042
0.03744.001
0.15
PI-Width 26.9
28.3
Winkler 1.47=0.00
1.42#t.00
A Cov 0.063
0.056
0.05
PI-Width
67.3
68.7
Winkler 2.47#0.01
2.25=0.01
Ridge
A Cov 0.055+0.002
0.048
0.10 PI-Width
51.9#0.2
54.2
Winkler 1.91
1.77=0.01
A Cov
0.039 0.039
0.036
0.15
PI-Width
43.1 #0.3
45.0
Winkler
1.59=0.00
1.50=0.00
20

---

high
End calibration period
200-
Association weight
0
-200
Figure 5: Visualization of the 30 highest association weights that the Hopfield network places on previous time steps. HopCPT retrieves similar error values when predicting at time of high error, and it retrieves similar, previous small errors when predicting at a time step with small error.
A.3 Additional Results
Tables I7-2 23
show the results for miscoverage levels a E {0.05, 0.10, 0.15] for all evaluated combinations of datasets and predictors. Results for individual time series of the datasets are uploaded the code repository (Appendix
CMAL. For the Streamflow dataset, we additionally compare to CMAL, which is the statc-of-the art non-CP uncertainty estimation technique in the respective domain (Klotz al. 2022). CMAL is mixture density network (Bishop 1994) based an LSTM that predicts the parameters of asymmetric Laplacian distributions.
our experiments use the same dataset, we adopt the hyperparameters from Klotz et al. (2022) but lower the learning rate to 0.0001 because we train CMAL on more training samples (18 years, i.e., the training and calibration period combined, which allows for fair comparison against the CP methods). Despite the fact that CMAL is not based on the CP paradigm, it achieves good coverage results, however, the cost of wide prediction intervals. We argue that Winkler score is low because CMAL considers all (and therefore also uncovered) samples during the optimization while CP methods do not consider them at all therefore, the distance of these points to the interval might be smaller (see Table 23
Error trend. F shows the prediction errors the ridge regression model on the Sap flow dataset. The emrors exhibit strong trend and shift towards negative expectation value.
investigates the association patterns of HopCPT and shows its capabilities to focus time steps from similar error regimes. The depicted time step in regime with negative errors retrieves time steps with primarily negative errors and, likewise, the time step in regime with positive errors retrieves time steps with primarily positive errors of the same magnitude.
Association weights. Figure 3
Non-CP methods We also compare HopCPT to non-CI methods to evaluate its performance within the broader field of uncertainty estimation for time series. Specifically, compare against Monte Carlo Dropout (MCD); a Mixture Density Network (MDN), and a Gauss model (i.c., an MDN with single Gaussian component), with LSTM backbone (see Appendix A. for more details about the models and their hyperparameters) This comparison shows that, although the non-CP methods often lead to very narrow prediction intervals, they fail to provide approximate coverage most experiments (Table
21

---

A.4 Local Coverage
Standard CP only provides marginal coverage guarantees and a constant prediction interval width
(Vovk et al. 2005). In time series prediction tasks, this can lead to bad local coverage (Lei & Wasserman 2014). To evaluate whether the coverage approximately holds also locally, we evaluated A Cov on windows of size k. To avoid compensation of negative A Cov in some windows by other windows with positive A Cov, we upper-bounded each window's s A Cov by zero before averaging 1
over the bounded coverage gaps - i.e., we calculated
W _w=1 - Cov TO with
A Cov To = [A Cov if A Cov.. 0, 0
(10)
else,
where A .ov., is the A Cov within window w.
Table 8 shows the results of this evaluation for window sizes k E {10, 50) and miscoverage level
0.1. Depending on the dataset, most often either HopCPT or SPCI perform best. The overall rankings are only partly similar to the evaluation of the marginal coverage (Table ann Appendix (A.3). Especially standard CP. which achieves competitive results in marginal coverage, falls short in this comparison. Only for Solar (3M), where the approach achieves a high marginal Cov, it preserves the local coverage best. Note that this comes with the drawback of very wide prediction intervals, i.e., bad efficiency. Overall, the results show that HopCPT and other time series CP methods improve the local coverage in non-exchangeable time series settings, compared to standard CP.
A.5 Negative Results
We also tested training with more involved training procedures than directly using the mean squared error. However, based on our empirical findings, the vanilla method with mean squared error outperforms more complex approaches. In the following, we briefly describe those additional configurations (so that potential future research can avoid these paths):
Pinball Loss. We tried train the MHN with a pinball loss (the original use seems to stem from Fox & Rubin 1964 we are however not aware of the naming origin) in many different variations (see points that follow), but consistently got worse results than with the mean squared error.
Inspired by the ideas lined out by Tagasovska & Lopez- (019)) we tried to use the miscoverage level an additional input to the network, while also parameterizing the loss with it.
tried to use the pinball loss to predict multiple miscoverage values at the same time to get more informed representation of the distribution want approximate.
Softmax Probabilities. We tried to use the softmax output of the MHN to directly estimate probabilities and use maximum likelihood procedure. This did train, but it did not produce good results.
A.6
Computational Resources
We used different hardware setups in our experiments, however, most of them were executed on machine with an Nvidia P100 GPU and a Xeon E5-2698 CPU. The runtime differs greatly between different dataset sizes. We report the approximate run times for single experiment (i.e., one seed, all evaluated coverage levels, not including training the prediction model pa):
1. HopCPT: Solar (3Y) 5h, Solar (IY) 1h, Solar (3M) 7min, Air Quality (10PM) 3h, Air Quality (25PM) 3h, Sap flow 2.5h, Streamflow 12-20h
2. SPCI: (adapted): Solar (3Y) 5-10 days, Solar (1Y) 10-30h, Solar (3M) 45min, Air Quality
(10PM) 13-30h, Air Quality (25PM) 13-30h, Sap flow 20-50h, Streamflow 12-17 days
3. EnbPI: all datasets under 6h
4. NexCP: all datasets under 45min
5. AdaptiveCI: all datasets under 45min
6. Standard CP: all datasets under 45min
22

---

Table 7: Performance of the best two CP methods (HopCPT and SPCI) and tfate-o-thh-e of-toh--aaataee non-CP methods for the different datasets and levels of a. All approaches build upon an LSTM model. Bold numbers correspond the best result for the respective metric in the experiment (PI-Width and Winkler score), given that ACov 2 -0.25x, i.e., the specific algorithm reached at least approximate coverage (the result is grayed out otherwise). The error term represents the standard deviation repeated runs with different seeds (results without an error term are from deterministic models).
HopCPT
SPCI
MCD
Gauss
MDN
A Cov
-0.003#0.005
0.004#0.000
-0.05820.004
-0.331 20.143
-0.005=00008
0.05
PI-Width
22.740.8
47.740.0
17.5+0.1
25.9
16.2=1.0
Winkler
0.38
0.87
0.75
0.46#0.10
0.33 0.33 -:0.66
Solar 3Y
A Cov
0.001 +00000
4..1000.000
+0.030
-0.310+002338
-0.013
-0.013#0.044
800047
0.10
PI-Width
17.9
27.8+0.00
14.840.1
21.8
13.2
Winkler
0.30
0.62
0.49
0.37#0.06
0.26
Cov
t0.007
0.024
0.001 #0.007
0.320 166
00.053#0.141
0.15
PI-Width
15.0#0.4
18.1
12.9
19.1
11,4=0.8
Winkler
0.26 1.000
0.40
0.38 1..00
0.32
0,23
Cov
0.019 4..003
0.006:+0.000
-0.041 40.004
-0. 100
10.055
0.05
PI-Width
21.4
37.0+0.0
20.5
14.6
13.2=14
Winkler
0.27
0.58
0.58
0.22
0.20+0.03
Solar 1Y
Cov
0.028
0.018=00000
00.00
-0.13.44.255
-0.130+0.182
P1-Width
16.0#0.6
22.5+0.0
17.3
12.3
10.8=1.2
Winkler
0.22
0.41
0.40
0.1820.01
0.1740.02
A Cov
t0.017
0.032200.00
0.02220.006
148
-0.185
0.15
P1-Width
13.1 13.1
10.0
15.2
10.8
9.3=1.1
Winkler
0.19
0.32
0.32+0.00
0.16#0.01
0.15#0.02
A Cov
-0.001 t0.001
0.003
-0. -0.202
-0.191
-0.218
0.05
PI-Width
90.7#1.9
86.1 +0.0
42.3
45.
43.9=M.6
Air 10PM
Winkler
+.80s
1.63 1.63+0.00
2.36
2.08
2.22
A Cov
-0.002
0.010
-0.201 40.009
-0.211
-0.243
0.10
PI-Width
1..7
62.2
70.7
38.6
35.3=7.3
Winkler
1.33#0.01
1.21
1.49#0001
1.42#0.06
1.4840.30
A Cov
-0.002#0.010
0.017=0.000
-0.192#0.009
-0.21840.027
-0.252#0.122
0.15
PI-Width
49.4
+0.0
30.6
33.8
30.2=6.0
Winkler
+0000
1.01 +0.00
1.16
1.15
1.17
A Cov
0.005
0.005 t0.005
-0.015
+0.100
-0.227
+0.109
0.05
PI-Width
#5.1
44.9#0.0
31.7
27.1
26.9=6.1
Air 25 PM
Winkler
1.19
1.29#0.00
#0.05
1.58=0.08
1.57
Cov
0.00720.008
-0.017=0.000
-0.095#0.027
-0.24720.020
-0.2344001120
0.10
PI-Width
40.7444.3
32.4 40.0
26.6
22.g
21.6=4.9
Winkler
10005
4..00
4..03
1.004
1.05
Cov
0.005 40.011
-0.016+00.000
-0.08640.030
-0.25340.020
40.143
0.15
PI-Width
32.5
26.1
23.242.6
20.0
18.4=4.2
Winkler
0.74
0.760
0.76
0.85=0.02
0.81
Cov
0.001 #0.002
0.000=0.000
-0.150=0.007
-0.072#0.01
-0.059=0.003
0.05
PI-Width
7.3,5
89..7#0.9
421.6
494.7=26.0
494.6441.2
Winkler
0.25100.01
0.33
0.4910.01
0.29
0.2740.06
Sap flow
Cov
4..004
0.004400.000
10.199000007
-0.06740.022
-0.06740.037
0.10
PI-Width
594.347.7
028.2:00.9
244.3
416.5421.9
131.4431.4
Winkler
0.19
0.24
0.30
0.20#0.01
0.19
A Cov
t00000
0.006=0.000
-0.122t0.007
0.0559#0.024
-0.068
0.15
PI-Width
489.9
493,
240. 2
311.4388
364.9#19.2
25..8
Winkler
0.17
4.20
0.2340.01
0.17#0.01
0.1640.02
23

---

Table 8: Negative average of zero-upper-boundded A Cov of rolling windows with miscoverage level
= 0.1. We evaluate each dataset dreaset-predictor combination at the windows sizes & E {10, 20, 50).
HopCPT
Enbb?
NexCP
SPE.
Data
FC
Ridge LGBM Forest Ridge LGBM Forest
.030
.040
.081
955
.059
.059
.020
.022
.034
.040
50
.012
.016
2001
.021
.032
Solar 3Y
10
.051
.040
.076
.059
.058
20
.040
.022
.052
.035
.039
5015
.031
.038
.023
(40)
.099
.055
.056
20
.014
.033
.083
.099
040
50
.007
.075
.027
030
.023
.023
.070
.055
.019
.015
.012
.043
.010
50
.010
.008
.031
.017
.055
Solar 1Y
10
.059
.023
.071
.056
.018
20
.049
.012
.046
.032
.110
.007
.018
.005
.063
.030
.070
.055
066
20
.053
.021
.055
.038
052
50
.045
.016
.043
.027
0015
LGBM Forest
.045
.074
.075
083
.026
.074
.045
.055
Solar Small
50
.015
.067
.034
.030
.042
10
.048
.093
.074
.074
.07
20
.032
.061
.044
.044
.048
50
.020
.051
.030
.029
.035
Ridge
.051
100
.065
.058
0064
20
.039
.073
.049
.099
.045
50
.032
.066
.041
0033
038
LGBM Forest
.033
.049
.124
.075
106
.025
.067
Air 10 PM
50
.018
.035
.099
.053
.089
10
.038
.037
.115
.074
100
20
.030
.030
.104
.065
.993
50
.021
.023
.089
.052
.083
Ridge LGBM Forest Ridge
.041
.035
.099
.068
.064
20
.032
.028
.089
.059
.057
50
.024
.220
.074
.045
.048
10
.076
.061
.139
.080
114
20
.053
.129
.071
.108
A5r PM
50
.056
.044
.113
.056
.096
10
.073
.051
124
.800
113
20
.064
.044
.114
.072
105
50
.053
.035
.097
.059
.093
.057
.043
.107
.075
.079
20
.048
.035
.096
.066
.072
50
.039
.027
.081
.053
.061
Ridge LGBM Forest
10
.070
.058
.107
.076
.073
.062
.550
.097
.067
.051
.042
.082
.056
.059
Sap flow
10
.107
.060
105
.078
082
20
.098
.052
.094
.069
.075
50
.086
.045
.078
.056
.067
.097
.102
.086
20
.088
.279
.089
.077
397
50
.075
.271
.071
.063
.300
24

---

B Theoretical Background
We denote the full dataset as D (Di...., Dz+1). where D; (X,, Yi) is the i-th datapoint. Since we are in the split conformal setting (Vovk et al., 2005), the full dataset is subdivided into two parts: the calibration and the test data. The prediction model j is already given in this setting and not selected based on D. The prediction interval Cg for split conformal prediction given by
2 Σ
C. (Xe+1) = (((++1)) + Q1-a
++1 ón. +
1 141 -8+00
(11)
where R; |Y; - (xX;)), de is a point mass at c, and Q, is the r-quantile of a distribution. Vovk al. 2005) show that this interval provides marginal coverage guarantee in case of exchangeable data:
Theorem B.1 (Split Conformal Prediction (ovk et al. 2005)).
the data points (X1,Y) ........... (X,, Yi), (X++1, Y(+1) exchangeable then the split conformal prediction set defined in 177 satisfies
Pr Pr(Yet E (X 21-a.
This seminal result cannot be directly applied to our setting, because time series are often non- exchangeable (as we discuss in the introduction) and models often exhibit locally specific errors.
The remainder of this section is split into two parts. First, we discuss the thcoretical connections between the MHN retrieval and CP through the lens of Foyyel Barber et al. (2022). Second, we adapt the theory of Xu & Xie 2022a) to be applicable to our regime-based setting.
For the MHN retrieval, we evoke the results Foygel Barber al. (2022), who introduce the concept of non-exchangeable split conformal prediction. Instead of treating each calibration point equally, they assign individual weight for each point, to account for non-exchangeability, Formally, we denote the weight for the i-th point by w; E [0, 1]. = I,...,44 4 and the respective normalized weights are defined
= 1...., t, and w Wi + + we +
(12)
W1
W.
The prediction interval for non-exchangeable split conformal setting Öi is given by
C (X++1) (((++1)) Q1-a
Σ
Wr 8n, 0441 8400
(13)
Foygel Barber et al. 2202) show that non von-exxhangeeble split conformal prediction satisfies a similar bound as exchangeable split conformal prediction, which is extended by term that accounts for a potential coverage gap A Cov. Formally we define A Cov = a at, where a is the specified miscoverage and a* is the observed miscoverage.
To specify the coverage gap they use the total variation distance drv, which is distance metric between two probability distributions. It is defined as
(Pr, Pa) sup (A) P2(A)), AEF
(14)
where Pi and Pg are arbitrary distributions defined on the space FI.
Intuitively, dtv measures the largest distance between two distributions. It is connected to the widely used Kullback-Leibler divergence DKL through Pinsker's inequality (Pinsker 1964):
drv(P,,P)) S V DKL
(15)
Given this measure, Foygel Barber et al. 2022) bound the coverage of non-exchangeable split conformal prediction follows:
25

---

Theorem B.2 (Non-exces exchangeable split conformal prediction Foygel Barber et al. 2022
The non-exchangeable split conformal method defined in (13) satishes
Pr FrfYet (Xet1)) 2 (1
Σ i drv (((())
where R(D)- = (R1, R2,.... Re+1) denotes the sequence of absolute residuals on D and D denotes sequence where the test point Dz41 swapped with the i-th calibration point.
The coverage gap is represented by the rightmost term. Therefore,
A Cov 2
Wy drv((R()),
2=1
Foygel Barber et al. the split conformal setting (as they note, too). For the sake of completeness, we state the proof of Theorem B.2 explicitly for split conformal prediction (adapted from Foygel Barber et al. 2022):
(2022) prove this for the full-conformal setting, which also directly applies to
By the definition of the non-exchangeable split conformal prediction interval (13). it follows that
Σ Wy dR, + Wi+1 6+0c )
Yo-1 & (X++1)
Ro+1 > Q1-a
(16)
Next, we show that, for a given K (l,...,(++11
[+1
Qı-a Σ 1=1
ws on. + W+t1 d+xc
2 Qi-a
Σ i=1
(17)
by considering that
Ris
if + K and + +
(R(D*)),
=
Re+1, if
K,
(18)
Rk:
t
In the case of = +++ 1, this holds trivially since both sides are equivalent. the case of K S t, we rewrite the left-hand side of inequality (17
w. 5R. + Wz41 8400
Σ
w; ōR. + wK(ŌRK + d.oc)) + (1z41 xx )
i=l....t:iKK
The right-hand side can be similarly represented as
Σ Wi )(R(D))),
Σ ki
Wi òR. + wKŌR.. + 1i+18Rg
Wi 8R. + Ük(ōRK + (R.H)) + (w++1 - ~ )6
By the definition of the normalized weights 12 and the assumption that E [0, 1]. it holds that
u 2 wx. Applying this to the rewritten forms of the left- and right-hand sides, we can see that inequality I17 holds.
26

---

Substituting the right hand side of (66) by using 17 we see that
H+
Y41 & (X(41)
Rut1 > Q1-a
Σ 04 .
or equivalently by using (18)
Y141 C (X++1)
(R(D*)) Q a
Σ W )(R(DRO))
(19)
For the next step the definition of the set of so-called strange points S:
S(c) i E [E
Q1-a
By
des
(20)
A strange point 8 E S therefore refers to a data point with residuals that are outside the prediction interval. Given the definition of S, we can directly see that
Σ (ES(c)
के S a for all e RR+11
(21)
Combining (19) and 20
that non-coverage of Y(41 implies strangeness of point K:
Yeu & Of (X++1) K (R((K))
(22)
Given this, we arrive at the final derivation of the coverage bound:
Pr E S(R(D)))
Σ Pr(K = and i e S(R(D)))
w ((
E
W
dlv dIv(R())
(ES(R(D)
i (RV(R(
where we use (2 from the penultimate the last line. The first to the second line follows from K. D', which holds because D and a permutation of that does not depend on K.
Foygel Barber et al. (202)) show two additional variations of the coverage gap bound based on Theorem B.2 We do not replicate them here. One notable observation of theirs, however, that instead of using the total variation distance based on the residual vectors also the distance of the raw data vector can be used:
A Cov 2 - - w; dTv(D, D').
(23)
27

---

This coverage bound suggests that low weights for time steps corresponding to high total variation distance ensure small coverage gaps. However, lower weights weaken the estimation quality of the prediction intervals. Hence, all relevant time steps should receive high weights, while all others should receive weights close to zero.
HopCPT leverages these results by using the association weights from Equation 6 to reflect the weights W; of non-exchangeable split conformal prediction. HopCPT aims to assign high association weights to data points that are from the same error distribution as the current point, and low weights data points the opposite case. By Theorem B.2 association weights that follow this pattern lead to small coverage gap.
An asymptotic (conditional) coverage bound If we assume clearly distinguished error distribu- tions instead of gradually shifting ones, hard weight assignment, i.c., association weights that are either zero or one, are meaningful choice. Given this simplification we can directly link HopCPT to the work Xf & Xie 2022a).
Xu & Xie 2022a introduce variant of (split) conformal prediction with asymmetric prediction intervals:
(1e+1))= = | ((((4))((4))) + Qg (((:)) + Q1- ((6))=)1)
(24)
The approach taken by Xu & Xie (2022a) additionally adjusts the share of a between the upper and lower quantiles such that the width of the prediction interval is minimized. In contrast, HopCPT adopts simpler strategy and uses / on both sides, which mitigates the need for excessive recalculation of the quantile function.
Based on the introduced interval calculation procedure, they also provide bounds regarding its coverage properties, in particular about the possible coverage gap. While Xu & Xie (2022a) base their assumption on error properties only in temporal proximity, we generalize this to proximity in the learned feature representation (which can include but is not limited information about temporal proximity). Based on this proximity we assume:
Assumption B.3 (Error regimes). Given ((Z, yi. 0+)++ which represents the (encoded) dataset and the predictions ÿ, of j.. there exists a partition = (Te)er of the time steps (1....,
......,,) f), so that
ii Vi E I fEz y de lt E Ti}- ~ E,
(ii) 7i E I: ((41) ~ F.,
(iii) the subset T; corresponding to + 1 can be identified based on Ze41,
(iv) Vi,j E I: : j E Ej.
where each E; is an arbitrary error distribution.
Properties (i) and (ii) imply that any given time step + can be associated to subset T; where all errors follow a common error distribution Eg; property (iii) ensures that the partition can be found approximated from set of features. Property (iv) is not necessary for the discussion at hand but avoids degenerate solutions, since tight coverage bounds call for assigning all errors from the same error distribution to large partition. If the errors of the associated subset were sampled i.i.d. from E,, we could use standard split conformal methods (Vovk et al. 2005). and the variation distance between permutations of the errors in properties (i) and (ii) zero. In practice, error noise can make it challenging to find and delineate a given partition. Thus, HopCPT uses a soft-selection approach to approximate the discrete association of a partition and the current features.
To align HopCPT with the results of & Xie (2022a), and thus make use of their proposed coverage bounds for asymmetric prediction intervals, we first introduce their decomposition of the error: & = & + (μ((x) ((x;)), )), where f is the true generating function, ja the prediction model, and a noise term associated with the label. The notation (x) stems from Xu & Xie (202aa) and denotes an evaluation of μ., selected without using l.
Next, we adopt their i..dd assumption about the noise with regard to our approach:
28

---

Assumption B.4 (Noise is approximately i..d within an enror regime; adaptation of Xu & Xie jid E; and that the CDF F, of E; is (2022a), Assumption 1). Assume 3i I: fållt E Tr} U {t++1} Lipschitz continuous with constant L, 0.
This i..dd assumption within each error regime only concerns the noise of the true generating function. Xu & Xie (2022a) note that the generating process itself can exhibit arbitrary dependence and be highly non-stationary while still having i.i.d. noise. The assumption thus relatively weak.
Assumption B.5 (Estimation quality; adaptation of & Xie (022a). Assumption 2). There exist real sequences (0.,m1/b,/11121 such that
ii ViEl: 7 Eien, (A- u(xx)))² S [.1T.
(ii) Di E I: E (111)111111))11111 (x141) u( 81.12.1. where the i for given 1+ is the same as in assumption B.4
We want bound the coverage of (44. which is based on the prediction error How- ever, Assumption [B.4 only considers the noise é of the true generating function. Xu & Xie (2022a) bind the difference between the CDF of the true generation function F,, the empiri- cal analogue (a) [Td Ejen, 162 I z), and the empirical CDF of the prediction error F((x) 17l Ejer, 10g o)..
In particular, they state two lemmas: Lemma B.6 (Adaptation of Xu & Xie (0222).. Lemma 1). Under Assumption B.4 for any subset size [T)) there is an event Ar which occurs with probability at least 1 /log 16 / such that conditioning on Ar,
sup (x) - Fi()) < /lolis/oooo2111111 /oog(16/7.i//TT
x
Lemma B.7 (Adaptation of Xu & Xie 2022a Lemma 2). Under Assumptions B.4 and B.5
we have
sup F((x) F((x)) S (T4 + 1)6 [|/3] + 2sup [F((x)) - (t())
With this result we arrive bound for a conditional and a marginal coverage: Theorem B.8 (Conditional coverage gap; adaptation of & Xie 2022a), Theorem 1). Given the prediction interval Oū defined in 24). Under Assumption B.4 and B.5 for any subset size (Ty), E (0, 1), and B E [0, we have:
E DiEl:(Pr YOUr (X++ 2e+1 (1-a)
<12 11166//16 + 4(L: + 1) 3.TT. + 8s,1Ta)).
(25)
Furthermore, if (6,, T: 1T:121 converges to zero, the upper bound in converges to when [ZR] + 2o, and thus the conditional coverage asymptotically valid.
Theorem B.9 (Marginal coverage gap; adaptation of Xu & Xie 2022a)
Theorem 2). Given the for any subset size (T:),
prediction interval Oa defined in 24). Under Assumption B.4 and B.5
E (0, 1), and B E a), we have:
3i I: |P Y1t1 E C (Xet)) - (1- 12 12 + 4(L, + 1)(02/3 6,.m)))
The proof's of the above theorems follow from Xu & Xie (2022a). To apply adaptions their poofs, one has to select the error regime matching to the test time step t + I, i.c., select the right subset T; from the partitioning of time steps. 7; then corresponds to the overall calibration data in the original assumptions, lemmas, theorems, and the respective proofs. This is valid since Assumption B.5 encompasses all subsets and the existing subset in Assumption B.4 and Assumption B.5 is, as noted, the same.
Xu & Xie (202aa) provide additional bounds for two variations of Assumption B.4 imposing only the requirement of strongly mixing noise or noise following linear process. These bounds apply
also to our adaption but omitted here for brevity.
29

---

Notes on Exchangeability, Stationarity, Seasonality, and Drifts Comparing HopCPT to standard CP we see that HopCPT assumes exchangeability over the weighted time steps within regimes. Hence, we require that the relationship between inputs and outputs remains stationary within regimes, which weaker than requiring that the time series as such is stationary. For example, it allows that time series values decrease or grow over time as long as these changes are reflected in the inputs of the MHN and correspond to a regime. HopCPT softens the exchangeability requirements of standard CP in two ways:
1. The CP mechanism assumes that exchangeability is given within a regime that is selected by the MHN association.
2. Even if (I) is not the case, the memory of the MHN (and therefore the calibration data) is updated with each new observation. This memory update allows HopCPT to account for shifts empirically, the benefit is visible in our experiments on the sap flow dataset, as we discuss briefly in Section 3.2 (lines 283-289) and in more detail in Appendix A.3 (Error Trend & Figure 4).
Regarding seasonality: HopCPT uses covariate features describe the current time step. Hence, HopCPT implicitly considers seasonality by comparing given time step with the time steps from the memory. Whenever time step of certain season is encountered, it focuses on time steps from the same season. In fact, our experiments already comprise both regular and irregular seasonality patterns. For example, the streamflow dataset contains yearly reoccurring patterns (spring, summer, fall, winter) as well as irregularly reoccurring similar flood events. In both cases, HopCPT can identify related time steps based on the covariates.
Regarding data drifts: an advantage of HopCPT that the memory is updated with every new observation. Therefore, HopCPT somewhat incorporate drifts in the target value as long the covariates remain so that the learned representation is able to focus on the relevant observations. Empirically, the robustness against drifts visible in our experiments on the sap flow dataset, we discuss briefly in Section 3.2 (lines 283-289) and in more detail in Appendix A.3 (Error Trend & Figure 4).
30

---

C
Dataset Details
Datsets from four different domains were used in the experiments. The following paragraphs summarize the details. Table provides a quantitative overview.
Solar. We conducted experiments on three solar radiation datasets: Solar (3M), Solar (1Y), and Solar (3Y). All three datasets are based on data from the US National Solar Radiation Database (NSDB: Sengupta JNSnu:|Sengupta et al. 2018). Besides solar radiation as the target variable, the datasets include 8 other environmental features. Solar (3M) is the same dataset as used by Xu & Xiel (2022a b) and focuses on data from cities California. To show the scalability of HopCPT we additionally generated two larger datasets Solar (1Y) and Solar (3Y), which include data from 50 cities from different parts of the US. We selected the cities by ordering the areas provided by NSDB according to the population density and picked the top 50 under the condition that no city appears twice in the dataset.
Air quality. The datasets Air Quality (10PM) and Air Quality (25PM) are air quality measurements from 12 monitoring sites in Beijing, China (Zhang et al., 2017). The datasets provide two target variables (I0PM and 2.5PM measurements) which we use separately in our experiments (the variables are used mutually exclusively in the datasets, e.g., when predicting 10PM we do not use 2.5PM as a feature). We encode the wind direction, given a categorical variable, by mapping the north south and the wast components each to a scalar from -1 to 1.
Sap flow. The Sap flow dataset is subset of the Sapflux data project (Poyatos et al. 202 This dataset includes sap flow measurements which we use as a target variable, as well as a set of environmental variables. The available features, the length of the measurements, and the sampling vary between the individual time series. To get set of comparable time series we processed the data as follows: (a) We removed all time series where not all of the 10 environmental features are available. (b) If there was single missing value between two existing ones, we filled the value with the value before (forward fill). (c) We cut out all sequences without any missing target or feature values (after step b). (d) From the resulting set of sequences we removed all that have less than 15,000 or more than 20,000 time steps.
Streamflow. For our experiments on the streamflow data, we use the Catchment Attributes and Meteorology for Large-sample Studies dataset (CAMELS; Newman et al. 2015 Addor al.) 2017). It provides meteorological time series from different data products, corresponding streamflow measurements, and static catchment attributes for catchments across the continental United States. We ran our experiments on the subset of 531 catchments that were used in previous hydrological
benchmarking efforts (c.g., Newman et al., 2017; Kratzert et al. 2019 2021 Klotz et al. 2022).
Specifically, we trained the LSTM prediction model with the NeuralHydrology Python library (Kratzert et al. 2022) on precipitation, solar radiation, minimum and maximum temperature, and vapor pressure from the NLDAS (Xia et al., 2012). Maurer (Maurer et al. 2002). and Daymet (Thornton et al. (997) meteorological data products. Further, the LSTM received 26 static catchment attributes at each time step that identify the catchment properties. Table 4 in Kratzert et al. (2019) provides a full list of these attributes. We trained the prediction model on data from the period Oct 1981 Sep 1990 (note that for some catchments, this period starts later due to missing data in the beginning), calibrated the uncertainty models on Oct 1990- Sep 1999, and tested them on the period Oct 1999- Sep 2008
D kNN vs. Learned Representation
As mentioned in the main paper, we designed HopCPT with large datasets in mind. It is only in such setting that the learned part of our approach can truly play to its strengths and take advantage of nuanced interrelationships in the data. kNN provides us with a natural "fallback" option for settings where not enough data available infer these relationships. Our comparisons in Table I0 and Table [T substannite this argument: kNN provides competitive results for the small dataset Solar (3M) (this can also be seen by contrasting the performance from Table 10 to the respective results in Appendix A.3). but is outperformed by HopCPT for the larger datasets.
31

---

Table 9: Details of the evaluated datasets.
Number Series
Time Steps per Series
Number of Features
Period
Sampling
Data Split [%]
Solar (3M)
2,000
01-03
2018
60m
8
60/15/25
Solar (1Y)
50
8,760
2019
60m
8
60/15/25
Solar (3Y)
50
26,304
2018-20
60m
8
34/33/33
Air Quality (10PM)
35,064
2013-17
60m
11
34/33/33
Air Quality (25PM)
35,064
2013-17
60m
34/33/33
15,000-
Sap flow
24
2008-16
varying
10
34/33/33
20,000
Streamflow
531
9,862
1981-2008
24h
41
34/33/33
E
Quantile of Sample vs. Quantile of Weighted ECDF
HopCPT constructs prediction intervals by calculating a quantile on the set of weight-samples errors (see Equation 77. An alternative approach is calculate the quantile over weighted empirical CDF of the errors. This approach would define (TT, Zz+1) in Equation
as
d(T, Z(+1) Qr
Σ aett,
(26)
where deg is point mass at |ei]. Empirically, we find little differences in the performance when comparing the two approaches (Tables and 14)
Asymmetrical Error Quantiles
HopCPT calculates the prediction interval by using equally large quantiles (in terms of distribution
mass, not width) for the upper and lower bound (see Equation 7). Xu & Xie (202aa) instead search
for 3 E [0, a] and calculate the lower quantile with 8 and the upper quantile with minimizes the overall interval with. Applying this principle to Equation 7 one would arrive
+ 3, which
3 argmin ((X++1) + 9(3, Zz41). + q(1 - a B, 3ε(0,a]
(27)
(Z(+1) (xX++1) + (3, (( 9(1 + B, Z Z
Practically, the search is done by evaluating different 3 values within the potential value range and selecting the one which produces the lowest interval width. Theoretically, this could lead to more efficient intervals at the cost of additional computation and potential asymmetry in regard to uncovered samples that are higher and lower than the prediction interval. Empirically, we do not find notable improvement when applying this search to HopCPT (see Table 13).
G AdaptiveCI and HopCPT
As noted in Section AdaptiveCI is orthogonal to HopCPT, SPCI, EnbPI, and NexCP. We therefore also evaluated the combined application of the models. Nevertheless, AdaptiveCI is an independent model on top of an existing model, which is reflected in the way we select the hyperparameters of the combined model: First, the hyperparameters are selected for each model without adaption through AdaptiveCI (see Appendix A. hence, we use the same hyperparameters as in the main evaluation. Second, we conduct another hyperparameter search, given the model parameters from the first search, where we only search for the parameter of the adaptive component.
32

---

Tables L5 and show the results of these experiments. Overall, the results are slightly better, as the Winkler score (which considers both width and coverage) slightly improves in most experiments. The ranking between the different models stays similar to the non-adaptive comparison (see Section 3.2 with HopCPT performing best on all but the smallest dataset.
H
Details on Continuous Modern Hopfield Networks
Modern Hopfield Networks associative memory networks. They learn to associate entries of memory with a given "query" sample in order to produce a prediction. In HopCPT, the memory corresponds representations of past time steps and the query corresponds to representation of the current time step. The association happens by means of weights on every memory entry, which called "association weights". Query-memony sample pairs that have similar representations assigned high association weights. In summary, the association weights indicate which samples are closely related
HopCPT, this corresponds to finding the time steps which are part of the same time series regimes and therefore useful to estimate the current uncertainty. The association mechanism in that sense closely related the attention mechanism of Transformers (Vaswani et al., 2017). However, compared to other trainable networks, the memory- based architecture has the advantage that it does not need to encode all knowledge in network parameters but can dynamically adapt when given new memory entries.
We adapt the following arguments from Fürst al. 2022 and Ramsauer et al. 202 for more formal overview. Associative memory networks have been designed to store and retrieve samples. Hopfield networks are energy-based, binary associative memories. which were popularized as artificial neural network architectures in the 1980s (Hopfield 1982 1984). Their storage capacity can be considerably increased by polynomial terms the energy lunction (Chen et al. 1986; Psaltis & Cheol, 1986, Baldi & Venkatesh] 1987] Gardner 1987 Abbott & Arian 1987 Horn & Usher 1988 Caputo & Niemann 2002; Krotov Hopfield 2016). In contrast to these binary memory networks, we use continuous associative memory networks with far higher storage capacity. These networks are continuous and differentiable, retrieve with single update, and have exponential storage capacity (and are therefore scalable, i.e., able tackle large problems: Ramsauer et al. (021).
Formally, denote set of patterns (.1,...., x. } Rd that stacked as columns the matrix X = (#1,.... ÆNn) and a state pattern (query) & E IR that represents the current state. The largest norm of a stored pattern is M = max; ||,]|| Then, the energy E of continuous Modern Hopfield Networks with state & is defined as (Ramsauer et al. 2021)
N exp exp(BaTT))
E - -3-1 log Σ
2 5 + C,
(28)
M2. For energy E and state &. Ramsauer et al. (2021 proved that the where C 3- log N + 1 2 update rule
nnew
X softmax(3X
(29)
converges globally stationary points of the energy E and coincides with the attention mechanisms of Transformers (Vaswani et al. 2017 Ramsauer et al. 202I).
The separation A, of pattern æ; is its minimal dot product difference any of the other patterns:
A min [ x; - x. x.Tr)). Ju
(30)
A pattern is well-separated from the data if A, is above given threshold (specified in Ramsauer et al. 2021 If the patterns x; are well-separated, the update rule Equation 29 converges to a fixed point close to stored pattern. If some patterns are similar to one another and, therefore, not well- well-separated, the update rule converges to a fixed point close to the mean of the similar patterns.
The update rule of Hopfield network thus identifies cample-sample relations between stored patterns. This enables similarity- learning methods like nearest neighbor search (see [Schäfl et al. 2022). which HopCPT leverages to learn retrieval of samples from similar error regimes.
33

---

I Potential Social Impact
Reliable uncertainty estimates are crucial, especially for complex time-dependent environmental phenomena. However, overreliance on these estimates can be dangerous. For example, unseen regimes might not be properly predicted. changing climate that evolves the environment beyond already seen conditions can cause new forms of eгror regimes which cannot be predicted reliably. As most machine learning approaches, our method requires accurately labeled training data. Incorrect labels may lead to unexpected biases and prediction errors.
J Code and Data
The code and data to reproduce all of our experiments are available at https //gtth////t/bb/// com/ m1 juUpOPPH
34

---

Table 10: Performance of KNN compared to HopCPT for the miscoverage 0.10 on the solar datasets. The error term represents the standard deviation over repeated runs.
UC
kNN
HopCPT
Data
FC
Forest
A Cov
0.015
0.029=0.012
PI-Width
97.2
39.0#6.22
Winkler
1.59
0.73 0.73++0.2
LGBM
A Cov
0.012
0.001 #00003
Solar 3Y
PI-Width
108.4
37.7 40.7
Winkler
1.74
0.57#0.01
Ridge
A Cov
0.009
0.040#9.001
PI-Width
136.9
44.9
Winkler
1.99
0.64
LSTM
A Cov
0.002
0.001
#00006
PI-Width
22.9
17.9 17.9 1.066
Winkler
0.51
0.001
0.047=00004
Forest
A Cov
0.026
PI-Width
66.9
11.0
Winkler
1.00
0.40
LGBM
A Cov
0.003
0.003
Solar IY
PI-Width
72.2
40.74:2.8 1:2.8
Winkler
1.08
0.57
Ridge
A Cov
0.007
0.011
20.016
PI-Width
128.8
59.9
Winkler
1.63
0.92
LSTM
A Cov
0.000
0.028
PI-Width
16.1
16.0+:0.6
Winkler
0.33
0.22
Forest
A Cov
0.034
0.008=00000 008
PI-Width
35.0
38.4 +3.4
Solar Small
Winkler
0.93
1.09
t0.08 +0.99
LGBM
A Cov
0.022
0.007
PI-Width
47.2
48.2
Winkler
1.17
1.24
Ridge
A Cov
0.004
0.016
PI-Width
49.1
78.2
Winkler
1.08
1.78 40.59
35

---

Table 11: Performance of KNN compared to HopCPT for the miscoverage a = 0.10 on the different
non-solar datasets. The error term represents the standard deviation over repeated runs.
UC
kNN
HopCPT
Data
FC
Forest
A Cov
0.071
0.028-0.019
PI-Width
186.3
93.9+1111
Winkler
3.81
1.50
Air 10 PM
LGBM
À Cov
0.064
0.017=0.016
PI-Width
164.6
85.647.4
Winkler
3.42
1.40
Ridge
A Cov
0.054
0.010=0.005 0.007
PI-Width
114.8
79.9
Winkler
2.39
1.35
LSTM
A Cov
0.031
0.002-0.000
PI-Width
57.2
62.7#1.5
Winkler
1.30
1.33
Forest
A Cov
0.081
0.022400001
PI-Width
158.4
48. 1.5.6
Winkler
3.77
1.12
Air 25 PM
LGBM
A Cov
0.073
-0.021=0011
PI-Width
130.4
46.8
Winkler
3.10
1.10
Ridge
A Cov
0.059
0.005=0.028
PI-Width
97.5
110.6
Winkler
2.30
1.04
LSTM
A Cov
0.033
0.007=0.008
PI-Width
34.7
10.7
Winkler
0.94
0.88
Forest
A Cov
0.056
0.009=0.019
PI-Width
3246.1
1078.7473.7
Winkler
1.00
0.30
LGBM
A Cov
0.063
0.016=0.013
Sap flow
PI-Width
2647.5
875.5
Winkler
0.88
0.27
Ridge
A Cov 0.052
0.025-0.023
PI-Width
2808.4
1639.6 493.2
Winkler
0.82
#0.05
LSTM
A Cov
0.026
0.004=0.004
PI-Width
719.5
594.3#7.7
Winkler
0.24
0.19 +0.01
A Cov
0.108
0.001 =0004
Streamflow LSTM
PI-Width
1.70
1.39 +0.17
Winkler
1.39
0.79
36

---

Table 12: Performance of the weighted sample quantile (Sample) and the weighted empirical CDF (ECDF) quantile strategies that HopCPT uses for the miscoverage a 0.10 on the solar datasets. The error term represents the standard deviation over repeated runs.
UC
Sample
ECDF
Data
FC
Forest
Cov
0.029##0002
0.021 20.008
PI-Width
39.0#6.2
33.1
Winkler
0.73-0.20
0.62 0.62 +0.03
LGBM
A Cov
0.001 0.001 +0.003
0.006
0.006
#0.006
Solar 3Y
PI-Width
30,7
37.1 1-0.9
Winkler
0.57-0.01
0.58 0.58 0.001
Ridge
A Cov
0.040=0.001
0.041=0.001
PI-Width
44,9
45.0
Winkler
0.64=0.00
0.64
LSTM
A Cov
0.001#0.000 #0.006
0.001 10.000
P1-Width
17.9
10.4
Winkler
0.30=0.01
0.30 0.30 00.00
Forest
A Cov
0.047410.004
0.048
PI-Width
28.6
28.g
Winkler
0.40=0.04
0.40
LGBM
A Cov
0.003
0. 0.00000007
Solar IY
PI-Width
40.742.8
40.5:
Winkler
0.57=0.08
0.56
Ridge
A Cov
0..1140.016
000066000013
P1-Width
59.9
$7.5
Winkler
0.92
0
LSTM
A Cov
0.028-t0.010
0.032
Pl-Width
16.0
16.6
Winkler
0.22=0.01
0.22
Forest
A Cov
0.008#0.000 000
0.026#0.023
PI-Width
38.4
35.1
Solar Small
Winkler
1.09-0.08 0.08
0.99 0.99 0.99
LGBM
0.007
0.011 #0.013
PI-Width
48.7
46,3
Winkler
1.24=0.08
1.16#0.04
Ridge
A Cov
0.016=#0.022
0.012#0.011
PI-Width
78.2=24.5
109.0=18.6
Winkler
1.78
2.55 60.38
37

---

Table 13: Comparison of HopCPT (a) with applying the approach proposed by (Xu & Xie 2022a by to search for | that minimizes the prediction interval width (right column), and (b) without this approach which corresponds rate a = 0.1. The error term represents the standard deviation over repeated runs.
a/2 (left column). The experiments done with a miscoverage
UC
a/2
With / search
Data
FC
Forest
A Cov
0.029=0.012
0.010
PI-Width
39.0=6.2
41.0#7.6
Winkler
0.73=0.20
0.86#0.22
LGBM
A Cov
0.001
-0.013#0.008
Solar 3Y
PI-Width
70.7
36.8=62
Winkler
0.57
0.62
Ridge
A Cov
0.040=u.001
0.021 #0.010
PI-Width
44.9
50.0#26.3
Winkler
0.64=0.00
0.77#0.43
LSTM
A Cov
0.001=0.006
-0.013=0.007
PI-Width
17.9*t.0
16.2
Winkler
0.30 #0.01
0.31 +0,00
Forest
À Cov
0.028=0.019
0.007
PI-Width
93.9=11.1
83,4=11.2
Winkler
1.50=0.08
1,49=0.07
Air 10 PM
LGBM
A Cov
0.017=0.016
-0.00=000066
PI-Width
85.6*7.4
77.3
Winkler
1.45 0..08
1.41 #0.04
Ridge
A Cov Pl-Width
0.010=0.007
-0.001 #0.000
79.9#4.7
74.6
Winkler
1.35=0.06
1.32=0.02
LSTM
Cov
0.0022
-0.006=0.007
PI-Width
62.7=1.6
60,9=2.3
Winkler
1.33
1.33
Forest
Cov
0.009
#0.025
P1-Width
773.7
857.2-556.7
Winkler
0.30=0.01
0.30#0.01
LGBM
A Cov
0.011
0.095=#00007
Sap flow
PI-Width
$75.5=49.8
690,9t31.0
Winkler
0.27=0.00
0.28
Ridge
A Cov
0.023 023
-0.050=0.020
PI-Width
1639.6=93.2
1389.0
Winkler
0.16 0.16=00.0
0.42=0.03
LSTM
A Cov
0.00
0.003
PI-Width
594.3=7-7
$57.8
Winkler
0.19#0.01
0.19=0.00
38

---

Table 14: Performance of the weighted sample quantile (Sample) and the weighted empirical CDF (ECDF) quantile strategies that HopCPT uses for the miscoverage a = 0.10 on the different non-solar datasets. The error term represents the standard deviation over repeated runs.
UC
Sample
ECDF
Data
FC
Forest
À Cov
0.028-t0.019
0.027
PI-Width
93.9*11.1
93.4 93.4 +11.0
Winkler
1.50=0.00
1.50
Air 10 PM
LGBM
A Cov
0.01720.016
0.014
PI-Width
85.6#7.4
84,5#8.3
Winkler
1.45#0.08
1.43
Ridge
Cov
0.010
0.013#0.0010
PI-Width
7
79.3 4.66 4.46
Winkler
1.35
1..45
LSTM
A Cov
0.00220.005
-0.05500.010
PI-Width
62.7#1.5
65.6+20.1
Winkler
1.33#0.01
1.35
Forest
A Cov
0.024+0.017
0.024=0.019
PI-Width
48,1 4:5. 45.6
4:45
Winkler
1.12#0.05
1.11 10.01
Air 25 PM
LGBM
A Cov
0.021 40.019
0.023-00001
PI-Width
46.8
46.6
Winkler
1.10#0.05
t.10
Ridge
A Cov
0.005
-0007700.227
PI-Width
49.3
49.1
Winkler
1.04
1.04
LSTM
À Cov
0.00740.000
0.002=0.015
PI-Width
10.7+4.3
35.0
Winkler
0.88 0.88
t..00
Forest
A Cov
0.009
0.011 #0.119
Pl-Width
1078.7
1082.3
Winkler
0.30#0.01
0.30
LGBM
A Cov
0.016#40001
-0.01720.013
Sap flow
PI-Width
875.5=449.8
875.6
Winkler
0.27
0.27
LSTM Ridge
A
0.025
PI-Width
1639.6
1639.6493.2
+93.2
1634,0 4:9.2
Winkler
0.46#0.05
0.50
A Cov
0.004#0.004
0.005=0.003
PI-Width
594.347.7
592.0
Winkler
0.19
0.19 10.00
A Cov
0.001 40.041
0.028 000002
Streamflow
LSTM
PI-Width
10
1.49 4..02
Winkler
0.79#0.03
0.77
39

---

Table 15: Performance of the CP algorithms HopCPT, SPCI, EnbPI, and NexCP, each combined with AdaptiveCI, for the miscoverage a 0.10 on the solar datasets. Bold numbers correspond to the best result for the respective metric in the experiment (PI-Width and Winkler score). The error term represents the standard deviation over repeated runs. combined with AdaptiveCI.
CopulaCPTS
CP/CF-RNNY
HopCPT
Sabrri 1
NexCP
SPCI
UC
Forest
A Cov
0.004
0.002
0.001
-..000
0.000
0.000
PI-Width
29.8
97.6=0.1
155.7
170.8
168.6
168.5
Winkler
0.59
1.69#0.00
2.43
2.54
2.52
2.52
LGBM
A Cov
00.01
0.002
0.001
0.000
0.000
0.000
Solar 3Y
PI- P1-Width
39,947.0
95.8
155.0
164.8
162.3
162.4
Winkler
0.62
1.70
2.49
2.57
2.54
2.54
A Cov
0.00240.000
0.000
0.006
0.000
0.000
0.000
Ridge
PI-Width
36.0+
110.6=u.1
166.3
177.3
172.8
172.4
Winkler
0.62
1.77
2.28
2.63
2.61
2.58
LSTM
A Cov
+0000000000
0.001 +0.000
0.002
0.000
0.001
0.001
PI- Width
17.8 17.8+
25.5
30.6
30.6
28.8
28.8
Winkler
0.30
0.59
0.63
0.62
0.63
0.63
Forest
A Cov
0.007#0.001
0.011 20.000
0.002
0.001
0.002
0.003
PI-Width
1.2
72.6#0.1
109.5
126.7
128.9
129.5
Winkler
0.40 40.02
1.140
1.63
1.85
1.84
1.84
LGBM
A Cov
0.000 +0.001
0.005 0.005
to.000
0.001
0.000
0.002
0.003
Solar IY
PI-Width
41.6+1.9
69.2
106.2
117.1
119.1
119.4
Winkler
0.56
1.15
1.63
1.78
1.77
1.77
Ridge
A Cov
0.000
0.004
0.002
0.000
0.000
0..00
PI-Width
61.5#9.9
98.9=0.2
167.8
174.8
174.9
175.6
Winkler
0.87 +0.11
+.300
2.01
2.08
2.08
2.08
LSTM
A Cov
0.001 #0,000
0.003 40.000
0.001
0.001
0.001
0.001
PI-Width
14.3
18.9 18.9
21.8
20.4
19.9
20.0
Winkler
0.22 1.001
4.40
0.42
0.41
0.41
0.41
Forest
A Cov
0.006
-0.023+0.002
0.003
0.014
0..07
0.009
PI-Width
33.6#2.9
64.7=0.7
92.9
127.3
131.1
128.1
Solar Small
Winkler
1.01
1.95#0.0
2.38
3.27
LGBM
A Cov
0.001
-0.015
0.005
0.005
0.008
0.010
P1-Width
44.8
60.9 20.6
20.6
96.4
135.7
128.3
125.0
Winkler
1.18
1.95
2.54
3.32
3.33
3.31
Ridge
A Cov
40.004
00.11540.001
0.003
0.008
0.008
0.006
PI-Width
104,7419.8
-9.3
102.5
97.7
96.2
93.0
Winkler
2.44
1.92=t0.01
2.56
2.75
2.75
2.72
40

---

Table 16: Performance of the CP algorithms HopCPT, SPCI, EnbPI, and NexCP, each combined with AdaptiveCI, for the miscoverage = 0.10 on the different non-solar datasets. Bold numbers correspond the best result for the respective metric in the experiment (PI-Width and Winkler score). The eπrоr term represents the standard deviation over repeated runs. & combined with AdaptiveCI.
CopulaCPTSt
CPICF-RNNY
HopCPT
SPCI f
EnbPI
NexCP
UC
Forest
Cov
0.002=t0.001
0..002
#0.000000000
-0.002
0.000
-0.003
-0.005
PI-Width
80.7#2.6
112.6
250.4
270.5
228.7
228.3
Winkler
1.45=0006
1.96#0.00
3.70
3.92
3.37
Air 10 PM
LGBM
Cov
0.00140.001
0.001=0.000
0.001
0.000
0.002
0.003
PI-Width
78.5 22.2
102.5 Ho.2
220.0
233.2
196.3
196.3
Winkler
1.38
1.83+0.00
3.38
3.54
Ridge
A Cov
+0.0000
0.000
-0.001
0.001
0.000
0.000
PI-Width
75.3
20.1
147.2
159.5
142.7
142.3
Winkler
1.30=0.02
1.46#0.00
2.46
2.67
2.41
2.42
LSTM
A Cov
0.001
0.000
0.000
0.001
0.001
0.000
PI-Width
65.4
$9.2
69.2
65.7
62.9
62.5
Winkler
1.30
1.18 1.18-00.00
1.32
1.29
1.29
1.30
Forest
A Cov
0.00140.001
0.001 40.000
0.002
-0.03
0.001
0.004
0.008
PI-Width
t..4
86.5
221.7
245.1
205.2
200.8
Winkler
1.10=0.02
1.76#0.00
3.68
3.97
3.43
3.47
Air 25 PM
LGBM
A Cov
0.000
0.000=00000
0.001
0.000
0.003
0.007
PI-Width
52.5+1.7
76, t
180.6
191.7
151.1
147.2
Winkler
1.10 1.10=0.05
1.69 1..0000000
3.17
3.38
2.73
2.74
Ridge
A Cov
0.000
0.001
200000
-..01
0.001
0.001
0.000
PI-Width
40.4
63.2
130.0
143.6
130.3
129.5
Winkler
0.04
1.33
2.36
2.64
2.53
2.52
LSTM
Cov
0.001
#0.001
0.001
0.001#0.000
00000
0.000
0.001
0.001
0.000
PI-Width
1
36.3
44.6
40.8
38.8
38.9
Winkler
0.89
0.89 0.0000000
0.96
0.94
0.95
0.93
Forest
Cov
40.007
1.00440.000
-0.006
0.000
0.004
0.003
PI-Width
1006.9 24455
1651. 1651.0
4142.8
6151.7
6034.6
6022.0
Winkler
0.29
00.00
1.11
1.54
1.53
1.53
LGBM
Cov
0.001
40.003
0.007
0..05
0.000
0.006
0.008
Sap flow
PI-Width
919.1
#25.8
1601.7
3327.0
4845.1
4787.0
5016.1
Winkler
0.26 40.01
0.45 +0.90
0.88
1.23
1.22
1.28
Ridge
A Cov
0.006
0.121 0.121#0.000 #0.000
-0.003
-0.000
-0.004
0.126
PI-Width
1492.5=29.8
2974. 76699
3431.4
10665.6
10426.2
9521.9
Winkler
0.39=0.01
1.53
0.87
2.39
2.36
2.87
LSTM
A Cov
0.000=t0.000
0.002#0.000
0.001
0.000
0.001
0.002
PI-Width
595.0 477.5
639.5 639.5 +1.8
863.1
1019.1
980.1
992.6
Winkler
0.19 0.19#0.00
0.22
0.28
0.32
0.31
0.32
41

---

Table 17: Performance of the evaluated CP algorithms on the Solar (3Y) datasets for the miscoverage levels a E {00055 0.10, 0.15}. The column FC specifies the prediction algorithm used for the experiment (Forest: Random Forest, LGBM: LightGBM, Ridge: Ridge Regression, LSTM: LSTM neural network). Bold numbers correspond the best result for the respective metric in the experiment (PI-Width and Winkler score), given that ACov 2 -0.25a, i.e., the specific algorithm reached at least approximate coverage (the result is grayed out otherwise). The error term represents the standard deviation over repeated runs (results without an error term are from deterministic models).
CopulaCPTS
CPICF-RNN
AdaptiveCI
HopCPT
NexCP
EnbPI
SPCI
FC
A Cov
00006=#0.006
0.007
-0.030
-0.002
0.002
0.002
0.05
PI-Wis I--ih
47.849.5
149.2
173.0
216.5
237.2
236.7
Winkler
0.99
2.22#0.00
3.01
2.95
3.30
3.30
Forest
A Cov
0.029
0.01220.000
-0.031
-0.002
0.005
0.004
0.10
PI-Width
39.0
10.1
131.1
166.6
174.9
174.6
Winkler
0.73
1.74
2.47
2.53
2.75
2.76
À Cov
0.052
0.014 +0.000
-0.027
-0.002
0.006
0.006
0.15
PI-Width
33.6+
+0.3
101.4
129.1
132.2
132.0
Winkler
10.15
1.46=0.00
2.12
2.23
2.38
2.39
À Cov
0.0002
0.007=0.000
-0.022
-0.002
0.002
0.002
0.001
0.05
PI-Width
45.6
148.0#0.1
183.0
215.9
237.9
237.8
88.1
Winkler
0.72 0.002
2.25 +0.00
3.09
3.06
3.45
3.46
1.36
LGBM
A Cov
0.001 0.001 +0.003
0.014
-0.023
-0.002
0.006
0.006
0.001
0.10
Pl-Width
37.7
102.2
133.6
159.9
169.8
170.2
67.1
Winkler
0.57#0.01
1.75=0.00
2.52
2.55
2.80
2.81
1.19
Cov
0.009=0.003
0.017#0.000
-0.022
-0.002
0.008
0.009
0.001
0.15
PI-Width
32.7#0.7
75.5
100.6
121.3
126.6
127.2
55.1
Winkler
0.50+0.01
1.46=0.00
2.15
2,21
2.39
2.40
00.001
-0.003
40.000
-0.080
-0.002
0.003
0.003
0.05
PI-Width
42.7
146.3 40000
151.0
226.2
223.8
224.8
Winkler
0.78
0.78
2.31 2.31 00000
3.04
3.20
3.44
3.46
Ridge
A Cov
4.040-00.00
0.00240.000
-0.074
-0.001
0.004
0.005
0.10
PI-Width
44.9
108.2 108.2 20.0
131.1
171.0
166.0
167.7
Winkler
0.64
1.82=0.00
2.49
2.66
2.73
2.74
A Cov
0.070=0.c01
0.009
-0.069
-0,000
0.004
0.006
0.15
PI-Width
30.5
10.5
1.00
114.7
142.2
141.2
142.7
Winkler
0.56
1.56=0.00
2.21
2.34
2.37
2.37
A Cov
0.003
0.004=0.000
-0.018
-0.001
0.002
0.001
0.05
PI-Width
22.7
47.740.0
41.0
47.3
51.1
54.7
Winkler
0.38
0.87#0.00
0.90
0.87
0.96
0.97
LSTM
A Cov
0.001 0.001 #00000
0.014
-0.018
-0.001
0.007
0.007
0.10
P1-Width
17.9
27.8
24.6
28.2
31.9
33.0
Winkler
00.01
0.62=0.00
0.64
0.63
0.68
0.70
A Cov
0.002
0.021=0.000
-0.017
-0.001
0.010
0.009
0.15
PI-Width
15.0
18.1 1:.1 t0.00
16.2
18.6
20.5
21.3
Winkler
0.26 40000
0.48 +0.00
0.50
0.50
0.54
0.55
42

---

Table 18: Performance of the evaluated CP algorithms on the Solar (1Y) dataset for the miscoverage levels a E {0.05, 0.10, 0.15}. The column FC specifies the prediction algorithm used for the experiment (Forest: Random Forest, LGBM: LightGBM, Ridge: Ridge Regression, LSTM: LSTM neural network). Bold numbers correspond the best result for the respective metric in the experiment (PI-Width and Winkler score), given that ACov 2 -0.25a, i.e., the specific algorithm reached at least approximate coverage (the result is grayed out otherwise). The error term represents the standard deviation over repeated runs (results without an error term are from deterministic models).
CopulaCPTS
CPICF-RNN
AdaptiveCI
HopCPT
EnbPI
NexCP
SPCI
FC
A Cov
0.016=0.0022
0.029
-0.017
0.002
0.034
0.035
0.05
PI-Width
35.0
147.8
133.4
173.9
241.8
262.9
Winkler
0.005
1.71=0.00
1.97
2.18
2.61
2.82
Forest
A Cov
0.047=0.004
0.045+t0.000
-0.018
0.002
0.056
0.063
0.10
PI-Width
2..6
97.1
98.8
127.8
182.4
204.9
Winkler
0.40 t0.04
1.26#0.00
1.65
1.84
2.10
2.30
A Cov
0.078 00000
0.052 +0,000
-0.016
0.002
0.070
0.086
0.15
PI-Width
24.6 +0.8
67.7+0.1
73.1
138.5
161.2
Winkler
0.35 00.03
1.00=0.00
1.42
1.59
1.76
1.93
À Cov
-0.010=0.000
1.022=+0000
-0.019
0.001
0.034
0.036
-0.000
0.05
PI-Width
51.0##3.4
147.8#0.2
131.5
165.0
237.9
265.2
62.2
Winkler
0.71 +0010
1.72 #0.000000
2.05
2.16
2.60
2.86
0.78
LGBM
A Cov
-0.003 00.000
0.045 0.045++0.000 t0.000
-0.017
0.002
0.056
0.065
0.000
0.10
Pl-Width
40.7 +2.8
96.5
93.9
118.0
171.1
196.6
49.5
Winkler
0.57
1.26=0.00
1.65
1.78
2.02
2.24
0.69
Cov
0.001
0.053400000
-0.014
0.002
0.068
0.086
0.000
0.15
PI-Width
34.5
68.0#0.1
68.2
85.7
125.5
149.1
42.5
Winkler
0.50
1.01=0.00
1.40
1.66
1.83
0.67
Cov
00.022
0.018 40.000
-0.025
-0.004
0.008
0.011
0.05
PI-Width
40.3
151.0 4.02
178.0
196.3
207.9
219.8
Winkler
1.15
1.78
2.26
2.29
2.35
2.49
Ridge
A Cov
-0.011
0.031
1:0.000
-0.031
-0.006
-0.010
-0.015
0.10
PI-Width
59.9
112.8
158.4
171.7
171.7
172.0
Winkler
0.92
1.40=0.00
2.06
2.09
2.12
2.17
A Cov
0.000=0.02z
0.005
-0.034
-0.008
-0.024
-0.040
0.15
PI-Width
1:4.
92.5 10.1
144.1
154.6
150.2
146.6
Winkler
0.81 10
1.21
1.92
1.96
1.98
2.02
A Cov
0.019
0.006
-0.018
-0.001
0.010
0.013
0.05
PI-Width
20.7
37.0
29.5
33.6
39.4
42.2
Winkler
t.201
0.58=0.00
0.60
0.57
0.59
0.61
LSTM
A Cov
0.028
0.018
-0,018
-0.001
0.018
0.025
0.10
PI-Width
16.0
22.0
17.4
23.1
25.0
Winkler
0.22 00.01
0.41=0.00
0.42
0.41
0.43
0.43
A Cov
00.01
0.032:40.000 .000
-0.014
0.001
0.030
0.040
0.15
PI-Width
13.1 t0.0
15.3
11.1
12.6
15.2
16.9
Winkler
0.19 t0000
t..00
0.32 =.32
0.33
0.33
0.33
0.34
43

---

Table 19: Performance the evaluated CP algorithms on the Solar (3M) dataset for the miscoverage levels E {0.05, 0.10, 0.15). The column FC specifies the prediction algorithm used for the experiment (Forest: Random Forest, LGBM: LightGBM, Ridge: Ridge Regression). Bold numbers correspond the best result for the respective metric in the experiment (PI- (PI-Width and Winkler score), given that ACov 2 -0.25a, i.e., the specific algorithm reached at least approximate coverage (the result grayed out otherwise). The error represents the standard deviation over repeated runs (results without an error term are from deterministic models).
CopulaCPPS
CPICF-RNN
AdaptiveCI
HopCPT
NexCP
SPCI
EnbPI
FC
A Cov
-0.002-0.003
-0.002
0.000
40.074
-0.022
-0.013
-0.022
-0.028
0.05
PI-Width
4
57.9
110.7
160.2
162.0
155.0
Winkler
1.211
2.622
2.92
3.70
3.95
4.04
Forest
A Cov
0.008 0.006
-..06440.00
-0.022
-0.021
-0.027
-0.025
PI-Width
38.4
38.8
86.0
122.9
110.4
111.4
Winkler
1.09
1.82=0.01
2.54
3.28
3.47
3.48
Cov
0.020=0.010
-0.052#00002
-0.028
-0.019
-0.020
-0.017
0.15
PI-Width
32.6
26.7
64.8
91.0
77.5
79.6
Winkler
0.99
1.45#0.02
2.40
2.91
2.99
2.98
Cov
0.002-0.008
-0.06340.001
-0.024
-0.014
-0.018
-0.019
-0.000
0.05
PI-Width
59,7#3.4
56.3
111.7
161.7
164.5
162.1
61.8
Winkler
1.49
2.66#0.02
3.10
3.87
4.13
4.18
1.45
LGBM
A Cov
0.007=0.b12
-0.052#0.002
-0.022
-0.020
-0.022
-0.021
-0.004
PI-Width
48.2
37.5
84.3
119.9
107.6
106.9
49.2
Winkler
1.24
1.84=0.01
2.67
3.35
3.52
3.53
1.26
A Cov
0.005=0.016
-0.04340.002
-0.024
-0.019
-0.013
-0.011
-0.007
0,15
PI-Width
41.4
26.9
60.8
85.1
75.8
77.5
44.1
Winkler
1.11
1.47=0.01
2.48
2.93
2.98
2.97
1.22
A Cov
0.01170.011
-0.06.
-0.021
0.004
0.009
-0.002
0.05
PI-Width
92.7
67.2
110.1
144.8
156.1
133.6
Winkler
1.99
2.49#0.02
3.07
3.82
3.78
3.75
Ridge
A Cov
0.016=0.022
-0.056#0.00z
-0.020
0.007
0.014
-0.003
PI-Width
78.2
51,8
84.9
85.2
78.1
Winkler
1.78 +0.59
1.91 10911
2.66
2.85
2.83
2.81
A Cov
-0.009 -0.009-0.021 0.002
-0.039+0.00z
-0.017
0.003
0.014
0.003
PI-Width
67.4 +1.00
43.2
71.6
70.9
71.9
70.7
Winkler
1.61 #0.51
1.59=0.00
2.29
2.34
2.34
2.33
44

---

Table 20: Performance of the evaluated CP algorithms the Air Quality (10PM) dataset for the miscoverage levels a E {0.05, 0.10, 0.153. The column FC specifies the prediction algorithm used for the experiment (Forest: Random Forest, LGBM: LightGBM, Ridge: Ridge Regression, LSTM: LSTM neural network). Bold numbers correspond to the best result for the respective metric the experiment (PI-Width and Winkler score), given that ACov 2 -0.25a, i.e., the specific algorithm reached at least approximate coverage (the result grayed out otherwise). The error term represents the standard deviation over repeated runs (results without an error models).
from deterministic
CopulaCPTS
CPICF-RNN
AdaptiveCI
HopCPT
NexCP
SPCI
EnbPI
FC
A Cov
0.006=0.013
-0.001
#0.000
-0.054
-0.005
-0,016
-0.027
0.05
PI-Width
116.72144
152.740.1
242.9
321.5
322.5
297.1
Winkler
1.94
2.86=0.00
4.97
4.82
6.47
6.61
Forest
Cov
0.028=0.019
0.008 .000
-0.066
-0.004
-0.019
-0.033
PI-Width
93.9+11.11
118.5
202.8
263.5
243.1
229.8
Winkler
+..09
2.23+00.00
4.16
4.03
4.94
4.98
Cov
0.050=0.025
0.008
-0.073
-0.002
-0.021
-0.039
0.15
PI-Width
80.5
99.8
175.7
229.4
207.1
198.2
Winkler
1.28+0008
1.92=0.00
3.72
3.61
4.20
A Cov
-0.000=0.011
0.008=00000
-0.047
-0.005
-0.014
-0.022
-0.000
0.05
PI-Width
106.3
146.1 146.1+0.1 +0.1
219.5
285.7
281.1
264.6
228.0
Winkler
1.87
2.49*+0.00
4.56
4.46
5.79
3.61
LGBM
A Cov
0.017-0.016
0.023+00.000
-0.057
-0.004
-0.017
-0.028
-0.001
0.10
PI-Width
85,6
113.2#0.1
178.3
224.8
206.7
196.5
186.4
Winkler
1.45#0.06
1.94
3.69
3.64
4.33
4.36
3.00
A Cov
0.033=0.019
0.038:00000
-0.061
-0.004
-0.017
-0.029
-0.001
0.15
PI-Width
73.4
94.9*0.1
151.4
188.9
168.2
161.9
161.1
Winkler
1.24 1.24 10.06
1.66
3.23
3.20
3.63
3.65
2.71
A Cov
0.001 0.000
0.000
-0.037
-0.003
0.005
0.006
0.05
PI-Width
100.6 48.2
120.2
152.6
202.3
213.7
219.3
Winkler
1.70
1.93=0.00
3.32
3.42
3.96
3.98
Ridge
A Cov
0.010=0.007
0.024
-0.045
-0.002
0.010
0.012
0.10
PI-Width
79.9
93.9+0.1
120.0
153.3
153.7
155.3
Winkler
1.35
1.52
2,60
2.68
2.95
2.96
A Cov
0.016
0.038 00.00 0000
-0.049
-0.003
0.015
0.018
0.15
PI-Width
68.1 ::.1
79.4
100.5
126.9
125.7
127.0
Winkler
t.17
1.31 20.00
2.23
2.31
2.46
2.46
Cov
0.001=0.000
0.003
-0.021
-0.002
-0.002
-0.001
0.05
PI-Width
90.7+1.9
86.1
80.8
88.8
86.8
88.1
Winkler
1,83
1.63 -.6300000
1.81
1.77
1.86
1.86
ASSTM
Cov
0.002-0.00 0.005
0.000
-0.025
-0.002
0.001
0.004
0,10
PI-Width
62.7
22.2
58.1
62.4
61.8
63.0
Winkler
1.33
=02000.00
1.32
1.29
1.34
1.34
A Cov
-0.002-0:010
0.01740.000
-0.028
-0.002
0.005
0.009
0.15
PI-Width
11.7
+0.0
46.5
49.6
49.6
50.8
Winkler
+000
1.01 =0.00
1.08
1.07
1.10
1.10
45

---

Table 21: Performance of the evaluated CP algorithms the Air Quality (25PM) dataset for the miscoverage levels a E {0.05, 0.10, 0.153. The column FC specifies the prediction algorithm used for the experiment (Forest: Random Forest, LGBM: LightGBM, Ridge: Ridge Regression, LSTM: LSTM neural network). Bold numbers correspond to the best result for the respective metric the experiment (PI-Width and Winkler score), given that ACov 2 -0.25a, i.e., the specific algorithm reached at least approximate coverage (the result grayed out otherwise). The error term represents the standard deviation over repeated runs (results without an error models).
from deterministic
CopulaCPTS
CPICF-RNN
AdaptiveCI
HopCPT
NexCP
EnbPI
SPCT
FC
A Cov
-0.033=0.013
-0.015
-0.065
-0.009
-0.018
-0.031
0.05
PI-Width
58,746.9
102.640.1
211.3
283,9
271.4
249.8
Winkler
1.51 1.51 20.04
2.67=0.00
5.01
4.71
6.47
6.59
Forest
Cov
-0.021=0.017
-0.009#0.000
-0.079
-0.007
-0.025
-0.041
PI-Width
48.1
81.5
177.3
235.6
212.6
203.5
Winkler
t.12
2.02
4.31
4.05
4.94
4.98
Cov
-0.014=.0119
+0.000
-0.084
-0.005
-0.029
-0.045
0.15
PI-Width
41.4
70.8
153.0
206.9
185.5
179.0
Winkler
0.94
1.71=0.00
3.88
3.67
4.25
A Cov
-0.031=0.014
-0.007=0.000
-0.054
-0.006
-0.020
-0.032
-0.001
0.05
PI-Width
57.2
93.5 +..1
176.4
239.5
212.7
196.0
208.8
Winkler
+.48
2.45 $0.00
4.30
4.35
5.47
5.58
3.99
LGBM
A Cov
0..02 000019
0.002
-0.063
-0.007
-0.027
-0.042
-0.001
0.10
PI-Width
46.8
00.0
142.1
182.2
154.5
143.9
163.3
Winkler
1.10#0.05
1.83=0.00
3.54
3.55
4.09
4.14
3.35
A Cov
-0.010=0.022
0.010#0.001
-0.066
-0.006
-0.029
-0.045
-0.001
0.15
PI-Width
40.2
62.4
118.8
147.6
124.2
117.1
133.3
Winkler
0.92
1.55 1.55=00.00
3.07
3.08
3.41
3.43
2.99
A Cov
-0.1180.018
-0.001
40.000
-0.043
-0.005
-0.002
-0.004
0.05
PI-Width
41.1
84.0
132.5
177.6
180.5
177.1
Winkler
1.35
1.82=0.00
3.11
3.31
3.82
3.85
Ridge
A Cov
-0.0550.022
0.011
-0.051
-0.006
-0.002
-0.004
0.10
PI-Width
49.3#10.6
65.5+0.0
106.2
134.3
127.4
125.3
Winkler
1.04 00111
1.40-20.00
2.54
2.66
2.92
2.93
A Cov
0.008=00 0.032
0.021 110.000
-0,053
-0.007
-0.001
-0.003
0.15
PI-Width
42.6
55.6:0.00
88.7
109.7
102.5
101.6
Winkler
0.11
1.20=0.00
2.21
2.30
2.46
2.46
Cov
0.005=0.005
-0.015#0.000
-0.023
-0.003
-0.015
-0.021
0.05
PI-Width
$7.1
44.9
50.8
56.0
48.4
46.1
Winkler
1.19
1.29
1.33
1.27
1.39
1.40
ASST
Cov
0.007-0.00s
-0.01740.000
-0.028
-0.003
-0.019
-0.025
0,10
PI-Width
4..7
+0.0
35.9
38.6
34.0
32.8
Winkler
0.005
0.93=0.00
0.97
0.94
0.99
0.99
A Cov
0.005=0.011
-0.016=0.000
-0.029
-0.003
-0.019
-0.025
0.15
PI-Width
+3.
26.1 26.0 t0.00
28.4
30.2
26.8
26.1
Winkler
t.74
0.76
0.79
0.77
0.80
0.81
46

---

Table 22: Performance of the evaluated CP algorithms on the Sap flow dataset for the miscoverage levels a E {0.05, 0.10, 0.15}. The column FC specifies the prediction algorithm used for the experiment (Forest: Random Forest, LGBM: LightGBM, Ridge: Ridge Regression, LSTM: LSTM neural network). Bold numbers correspond the best result for the respective metric in the experiment (PI-Width and Winkler score), given that ACov 2 -5, i.e. the specific algorithm reached at least approximate coverage (the result grayed out otherwise). The error term represents the standard deviation over repeated runs (results without an error term are from deterministic models).
CopulaCPTS
CPICF-RNN
AdaptiveC1
HopCPT
EmbPI
NexCP
SPCT
FC
A Cov
00.01+900024
-00002#0.000
-0.047
-0.003
0.010
0.003
0.05
PI- Width
439.3
2193.8
4264.1
7290.8
8805.8
8970.9
Winkler
0.38#0.01
0.74
1.42
1.75
2.04
2.12
Forest
A Cov
0.009
0.007
-0.042
0.000
0.014
0.005
0.10
PI-Width
1078.7473.7
1741.8
3671.6
6137.1
7131.1
7201.5
Winkler
0.30
0.59
1.24
1.56
1.76
1.80
A Cov
0.017
0.014
-0.034
0.002
0.017
0.009
0.15
PI- Wi-th
921.7=61.9
1456.7+222
3200.5
5261.8
5980.4
6062.0
Winkler
0.26#t0.01
0.50
1.12
1.43
1.57
1.60
A Cov
0.027
0.000=00.00
-0.043
-0.006
0.006
-0.002
0.008
0.05
PI-Width
1072.4=05.6
1988,0#1.0
3453.3
5737.6
6929.6
7087.5
7160.4
Winkler
0.35
0.62 +0.00
1.12
1.41
1.66
1.72
1.64
LGBB
A Cov
+0.016
0.003 +0.000
-0.040
-0.003
0.006
-0.007
0.010
0.10
PI-Width
875.5 444.8
1582.3
2924.1
4805.3
5588.5
5614.8
6273.5
Winkler
0.27#0.00
0.49
0.96
1.25
1.43
1.46
1.50
A Cov
-.0006
0.010#0.001
-0.036
-0.002
0.005
-0.012
0.010
0.15
PI-Width
751.7=41.8
1347.1
2531.2
4147.4
4586.3
4544.3
5564.7
Winkler
0.23#0.00
0.42
0.87
1.14
1.27
1.30
1.40
A Cov
40.014
-0.231
00.000 +0,000
-0.040
-0.012
-0.174
-0.331
0.05
PI- Width
1971.3:4155.4
2539.3 41.6
3595.3
11318.6
10230.3
8462.1
Winkler
0.54
3.97
1.04
2.54
3.53
8.20
Ridge
A Cov
-0.025 -0.025:0.022
20.000
-0.041
-0.015
-0.251
-0.358
0.10
PI-Width
1639.6 993.2
21.
3117.5
10628.9
8943.3
7148.7
Winkler
0.46 0.46#0.00
2.52+0.00
0.92
2.42
3.31
5.85
A Cov
0.026
0.235+0.001
-0.042
-0.016
-0.292
-0.375
0.15
PI- Width
1427.1 180.77
1792.0
2775.8
10073.3
7959.3
6155.9
Winkler
0.42
1.93+0.00
0.85
2.34
3.17
4.89
A Cov
0.001 0.001 1:0.002
0.000=t0.00
-0.018
-0.001
-0.012
-0.024
0.05
Width
79..
897.7#0.9
1020.5
1338.9
1300.1
1194.5
Winkler
0.25
0.33
0.36
0.40
0.45
0.47
SIIM
A Cov
0.004
0.004
-0.019
-0.000
-0.022
-0.042
0.10
PI- Width
594.347.
628,240..
768.0
990.0
903.9
817.2
Winkler
0.19:t0.01
0.0000.00
0.28
0.32
0.35
0.36
A Cov
0.005
0.006=0.000 000
-0.019
0.002
-0.026
-0.048
0.I5
PI- Width
489.9 #77.0
49..
618.2
780.6
681.5
620.4
Winkler
0.17
0.20 0.20 t.00
0.24
0.27
0.29
0.30
47

---

Table 23: Performance of the evaluated CP algorithms and the CMAL baseline (see Appendix A.3 on the Streamflow dataset for the miscoverage levels a {0.05, 0.10, 0.15). Bold numbers correspond to the best result for the respective metric in the experiment (PI-Width and Winkler score), given that áCov 2 0.2 i.e. the specific algorithm reached at least approximate coverage (the result grayed out otherwise). The error term represents the standard deviation over repeated runs (results without an error term are from deterministic models).
CopulaCPTS
CPICF-RNN
HopCPT
SPCI
EnbPI
NexCP
CMAL
Cov
-0.002-0.022
0.013+0.000
-0.042
-0.001
0.003
0.006
0044000004
0.05
PI-Width
1.91
2.57
2.53
3.23
3.44
3.63
2.4
Winkler
+.05
1.38
1.91
1.80
1.93
1.94
0.78
Cov
0.001 #0.041
0.027
-0.054
-0.000
0.005
0.009
004000000
0,10
PI-Width
1.39
1.58:0.00
1.55
1.94
1.99
2.08
1.90#0.06
Winkler
0.003
0.91 40.00
1.27
1.21
1.28
1.29
0.63=0.01
Cov
20.008
0.038
-0.061
0.001
0.005
0.009
00.001
PI-Width
1.11 #0.15
1.17
1.12
1.39
1.39
1.45
1.60#0.05
Winkler
40.03
0.700
0.98
0.95
0.99
1.00
0.55 +0.00
48


Reviews:
[{"review_id": "GIDzZulEjj", "reviewer": "Reviewer_tZC3", "summary": "The authors propose a conformal prediction approach for black box time series forecasting models. The method is based on learning to predict the model’s errors using a small neural network structured similarly to the softmax computation in an attention mechanism. The prediction interval of the proposed method utilizes the softmax scores in the error sampling procedure, drawing each error with the probability given by the softmax score. Both in synthetic and benchmark experiments, the proposed method outperforms other CP approaches in terms of PI-width and coverage gap. ", "questions": "* How is the actual marginal miscoverage $\\alpha^{*}$ computed?\n* Could you elaborate on the meaning of $\\delta_{+\\infty}$ in Eq. 3 and how are weights $w_i$ computed? Is it model-dependent?\n* Does the bound in Equation 5 apply to your method?\n* Is it possible to make Figure 3 less dense and make the connection to the prediction interval more comprehensible?\n* I am slightly confused about the notation in Eq. 3. The quantile should be computed on a set, but I am not sure from which values it is computed. Can you rewrite this in set-notation like Eq. 2?    \n", "limitations": "While there is no dedicated limitation section, the authors provide “negative results” in the appendix. A concise paragraph about the method’s limitations would be appreciated. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The manuscript is extremely well structured and very easy to follow. The authors provided an excellent overview of the conformal prediction task and relevant work. \n* The experiments provide a thorough insight into how the proposed method functions in a synthetic and more realistic setting. Figure 2 contrasts the proposed with competing methods in a comprehensible and straightforward way. \n* In addition to a solid theoretical motivation, the proposed method performs outstandingly and convincingly on a wide range of prediction models and several data sets. \n", "weaknesses": "* Without a background on Hopfield networks, the method seems an application of the attention idea to conformal prediction. It would be helpful to lay out the similarities and differences between MHNs and the attention mechanism. \n* I am missing a section on the limitations of the proposed method. For instance, how does the method scale to very long time series? \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Without a background on Hopfield networks, the method seems an application of the attention idea to conformal prediction. It would be helpful to lay out the similarities and differences between MHNs and the attention mechanism.\n\nWe agree that the connection between MHN and attention was not clear from our descriptions in the submitted paper. We use the framing of MHN to emphasize the memory aspect of the architecture. One important difference between the MHN and typical Transformer attention is that the MHN allows to directly ingest the calibration data into the memory. We have already provided a general overview of MHN in Appendix G, but we modified the revised paper to expand more on the connection between MHN and attention.\n\n> I am missing a section on the limitations of the proposed method. For instance, how does the method scale to very long time series?\n\nOther reviewers have raised similar points regarding the limitations. So far, the limitations of our method were somewhat dispersed throughout the manuscript and supplemental material. We added a dedicated limitations section to the revised paper, discussing:\n- Long time series (see below)\n- Dataset size: on very small datasets, it might be better to use kNN rather than learning the similarity with MHN. In this context, note our discussion of kNN as a fallback option for MHN in data-scarce scenarios (Appendix D).\n- Theoretical guarantees: as is true for all CP methods, the theoretical coverage guarantees require certain assumptions (see Appendix B for details). Nevertheless, our empirical evaluation shows that, in practice, these guarantees translate into efficient prediction intervals in real-world settings.\n\nRegarding long time series:  \nHopCPT is well-suited for very long time series. In fact, the more historical information is available, the better our method is able to learn the association mechanism. However, at some point the memory size becomes challenging, as no longer all historical time steps may be kept in the Hopfield memory. Existing methods (e.g., the exponential decay in NexCP) would simply disregard time steps far in the past. HopCPT can also disregard time steps by removing the oldest entries from the memory. Fortunately, the HopCPT association mechanism gives us more freedom for cases where very long-term dependencies exist: here, one could alternatively use a sub-sampling strategy. That said, in unpublished experiments we successfully applied HopCPT models with memory sizes of almost 2 million (512-dimensional) entries on an A40-GPU. We added the discussion about long time series to the revised manuscript. \n\n> How is the actual marginal miscoverage $\\alpha^*$ computed?\n\nBy “actual marginal miscoverage”, we mean the miscoverage which we observe on the samples of a test set, as opposed to the specified miscoverage $\\alpha$. I.e., $\\alpha^*$ is the fraction of samples that fall outside the predicted intervals. Ideally, $\\alpha^*$ would be exactly equal to $\\alpha$, which results in a coverage gap of $0$. We clarified the definition in the revised manuscript.\n\n> Could you elaborate on the meaning of $\\delta_{+\\infty}$ in Eq. 3 and how are weights $w_i$ computed? Is it model-dependent?\n\nThe $\\delta_{+\\infty}$ is a dirac delta distribution at $\\+\\infty$. I.e., a distribution with a value of $0$ everywhere but at $\\+\\infty$. $\\delta_{+\\infty}$ serves as a stand-in for the sample that we have yet to obtain. As an intuition: we do not yet know the error for time step $t+1$, so $\\+\\infty$ acts as a sort of “worst-case” estimate for the unknown error (see also Foygel Barber et al. 2022).\n\nFoygel Barber et al. (2022) choose the weights $w_i$ (and hence the $a_i$) a priori. HopCPT does not explicitly calculate weights $w_i$. Instead, we directly estimate the values of $a_i$ by means of the MHN association mechanism.\n\n> Does the bound in Equation 5 apply to your method?\n\nYes. We made this more explicit in the revised manuscript.\n\n> Is it possible to make Figure 3 less dense and make the connection to the prediction interval more comprehensible?\n\nWe improved Figure 3 to show a shorter period in time, which helped with the density of the illustration. As a result, the figure is now less cluttered. In the revised manuscript we added a reference to  Figure 5 (Appendix A.3) in the caption, which better shows the connection to prediction intervals (as it illustrates the association of past errors to the current time step in a similar style as Figure 3).\n\n> I am slightly confused about the notation in Eq. 3. The quantile should be computed on a set, but I am not sure from which values it is computed. Can you rewrite this in set-notation like Eq. 2?\n\nWe denote Eq. 3 in terms of distributions rather than sets. I.e., the “true” quantile of a distribution, rather than the empirical quantile of a finite set of elements. We would prefer to keep the notation of the theoretical motivation aligned with Foygel Barber et al. (2022) and the proof in Appendix B, which operates on distributions.\n\n> Limitations:\n> While there is no dedicated limitation section, the authors provide “negative results” in the appendix. A concise paragraph about the method’s limitations would be appreciated.\n\nAs mentioned above, we added a dedicated limitations section in the revised paper.\n"}, {"review_id": "rltoBBYau3", "reviewer": "Reviewer_YCj6", "summary": "This paper deals with providing conformal prediction (CP) sets for the time series prediction problem or, more generally, the cases where the exchangeability assumption is violated. Similar to the prior work of Foygel Barber et al. (2022), they also try to assign weights to the points in the calibration set with higher (lower) values for those who are coming from a more (less) similar region (aka regimes) as the test point. For such a weighting mechanism, they use Modern Hopfield Networks (MHN) that are capable of finding regions similar to the point for which a prediction set is sought. Using numerical experiments, they showed the superiority of their proposed framework.", "questions": "1. What is the justification behind the choice of encoding network (section 2.3)? Apparently, for the current experiments, it is working adequately. However, why should one stick to such a structure for every other dataset?\n\n2. I still do not understand the role of $W_q$ and $W_k$. How are they learned? What is the difference between them?", "limitations": "The authors mention a limitation of the proposed framework: the need for large data for HopCPT to perform well. However, there is still room for elaboration on other limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is indeed well-structured, nicely written, and easy to comprehend. \nThe extensive amount of numerical experiments supports their claim about the (empirical) superiority of their proposed framework over the vast range of competitors.", "weaknesses": "The contribution of this paper seems to be relatively marginal. Indeed, the authors introduced a new way of weighting the non-conformity scores (i.e., reforming the distribution of non-conformity scores) on top of the NexSCP framework proposed by Foygel Barber et al. (2022) while adopting the structure proposed by Xu & Xie (2022a) for interval constructions.  \n\nWhile the numerical results show their framework's (empirical) superiority in terms of efficiency and coverage, one should keep in mind that it adds relatively high computational complexity to the model. This is, of course, also reported in Appendix A.6.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The contribution of this paper seems to be relatively marginal. Indeed, the authors introduced a new way of weighting the non-conformity scores (i.e., reforming the distribution of non-conformity scores) on top of the NexSCP framework proposed by Foygel Barber et al. (2022) while adopting the structure proposed by Xu & Xie (2022a) for interval constructions.\n\nOur framework and theoretical guarantees are indeed motivated by and built upon the prior work of Foygel Barber et al. (2022) and Xu & Xie (2022a). In fact, one of our contributions is to connect some of the work done by the two groups of authors. However, our work adds significant additional contributions: To the best of our knowledge, we are the first to use a learned, similarity-based retrieval mechanism for conformal prediction. Our approach incorporates the covariates of the time series, while still being able to (1) consider the temporal proximity and (2) work in an online fashion. Hence, HopCPT provides much broader capabilities than Foygel Barber et al. (2022) and Xu & Xie (2022a). Further, we compare our new approach to existing work in a suite of different benchmarks. To the best of our knowledge, our paper contains the largest evaluation conducted for time series CP yet. We see our strong empirical improvements over the previous state of the art as evidence that HopCPT provides meaningful contributions for practical applications.\n\n> While the numerical results show their framework's (empirical) superiority in terms of efficiency and coverage, one should keep in mind that it adds relatively high computational complexity to the model. This is, of course, also reported in Appendix A.6.\n\nIt is true that HopCPT is computationally more demanding than standard CP (which performs poorly in time series settings), but as Appendix A.6 also shows, HopCPT is by far not the most compute-intensive method in our comparison. The second-best method in most benchmarks, SPCI, takes up to an order of magnitude longer in runtime. Also, we note that most of the runtime is needed for the one-time training procedure. Once HopCPT is trained, the inference (which may be executed far more often in practical applications) is much faster than that. To make these considerations more clear to readers, we added this discussion to the results section of the main paper. \n\n> What is the justification behind the choice of encoding network (section 2.3)? Apparently, for the current experiments, it is working adequately. However, why should one stick to such a structure for every other dataset?\n\nOther choices for the encoding network are indeed possible within our framework. In fact, in the development of our method, we tried a number of different strategies for the encoding (e.g., more sophisticated temporal encodings as they are used in Transformer architectures, number of encoding layers). Overall, we found that our method is not very sensitive to the encoding strategy. Hence, we kept to a simple choice. Our interpretation is that the Hopfield retrieval mechanism is expressive enough to retrieve the relevant information by itself, which makes the overall quality of our method depend less on the precise structure of the encoding network. \nThese considerations did not become clear in the original paper, so in our revision we added these points to the introduction of the encoding network in section 2.3.\n\n> I still do not understand the role of $W_q$  and $W_k$. How are they learned? What is the difference between them?\n\nThe nomenclature of $W_q$ and $W_k$ is motivated by the similarity of the Hopfield association mechanism to the attention mechanism in a Transformer network, where a “query” is compared to the “keys” of a memory in order to retrieve associated “values” from the memory. $W_q$ is the matrix of weights that are applied to the query vector, and $W_k$ are the weights that are applied to the key vectors. This way, the network can learn to adapt its measure of similarity between keys and queries to the task at hand. Section 2.4 provides more details. Nevertheless, your comment relates to reviewer tZC3’s point on providing more details about the connection to attention mechanisms. We therefore revised the introduction of the MHN to explain the association mechanism in more detail.\n\n>Limitations:\n> The authors mention a limitation of the proposed framework: the need for large data for HopCPT to perform well. However, there is still room for elaboration on other limitations.\n\nWe agree with your assessment. In the original paper the limitations were somewhat spread out across the manuscript and supplemental material. We therefore added a dedicated limitations section to the revised version. See our response to reviewer tZC3 where we outline the discussed limitations."}, {"review_id": "ki91RMmosv", "reviewer": "Reviewer_yfWn", "summary": "The work presents HopCPT, a conformal prediction algorithm for Modern Hopfield Networks in the setting of time series forecasting. The work uses a novel (attention-like) weighted quantile approach to leverage similar time steps from the past, improving both coverage and sharpness compared to SOTA CP methods. The proof mainly extends from  (Foygel Barber et al., 2022) and (Xu and Xie 2022). The method is shown to perform well on both synthetic and real world datasets.", "questions": "1. Does your method assumes stationarity of the time series?  because the $a_i$s are fixed, \n\n2. Equation 7. Would searching for a $\\beta$ as in SPCI (Xu & Xie 2023)  instead of using $\\alpha/2$ using further improve efficiency?\n\n3. I'm not sure point 4 in your main contributions is worded in the best way - your guarantees are not specific to hydrological prediction. Maybe a better way to say it is that HopCPT is the first algorithm with marginal coverage guarantees that can be applied to  hydrological prediction applications?\n\n4. Given the current soft-selection algorithm, since the $a_{t+1}$ s are static, would HopCPT still perform well when the seasonality in data is not regular?  What if there are drifts in the distribution? \n", "limitations": "- It might be good to stress that the guarantee is marginal and asymptotic, which can be achieved by rather simple algorithms (see Gibbs & Candes 2021, 2022). This is of course due to the properties of time series data, not a weakness of the approach.\n\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Novel and sound CP algorithm on a widely-used architecture for forecasting. \n- Excellent efficiency performance on time series datasets, with through comparisons to existing methods. \n- I also appreciate the use of proper scoring rules for evaluation (Winkler score). ", "weaknesses": "- From the paper alone, it's difficult to tell what exactly are the theoretical guarantees of your algorithm. While the authors did refer to Appendix B, I think it is important to briefly describe the assumptions and state the main bound (B.8 and B.9) of your theoretical analysis in the paper. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> From the paper alone, it's difficult to tell what exactly are the theoretical guarantees of your algorithm. While the authors did refer to Appendix B, I think it is important to briefly describe the assumptions and state the main bound (B.8 and B.9) of your theoretical analysis in the paper.\n\nThank you for your suggestion. In the revised manuscript, we highlighted the main results of the theoretical analysis from the Appendix in the main paper.\n\n> Does your method assumes stationarity of the time series? because the $a_i$s are fixed,\n\nWe assume exchangeability over the weighted time steps within regimes. Hence, we require that the relationship between inputs and outputs remains stationary within regimes, which is weaker than requiring that the time series as such is stationary. For example, it allows that time series values decrease or grow over time as long as these changes are reflected in the inputs of the MHN and correspond to a regime.\n\nHopCPT softens the exchangeability requirements of standard CP in two ways:\n- (1) The CP mechanism assumes that exchangeability is given within a regime that is selected by the MHN association.\n- (2) Even if (1) is not the case, the memory of the MHN (and therefore the calibration data) is updated with each new observation. This memory update allows HopCPT to account for shifts — empirically, the benefit is visible in our experiments on the sap flow dataset, as we discuss briefly in Section 3.2 (lines 283-289) and in more detail in Appendix A.3 (Error Trend & Figure 4).\n\nWe added the above extended discussion of stationarity and exchangeability requirements to the revised manuscript to better explain the scope of our assumptions.\n\nIn regard to the $a_i$s we would like to note that in contrast to Foygel Barber et al. (2022), where the weights are fixed a priori, HopCPT dynamically re-estimates the $a_i$s at every given time step.\n\n> Equation 7. Would searching for a $\\beta$ as in SPCI (Xu & Xie 2023) instead of using $\\alpha/2$ using further improve efficiency?\n\nWe considered searching for $\\beta$ in early stages of our work but found hardly any performance improvements, so we disregarded it in favor of reduced runtime. However, we agree that the impact of an optimized $\\beta$ is worth investigating and therefore added a corresponding ablation to the Appendix of the paper. The results can be found in the pdf which is attached to the “global” response.\n\n> I'm not sure point 4 in your main contributions is worded in the best way - your guarantees are not specific to hydrological prediction. Maybe a better way to say it is that HopCPT is the first algorithm with marginal coverage guarantees that can be applied to hydrological prediction applications?\n\nThank you for the suggestion. We agree and adopted your proposal.\n\n> Given the current soft-selection algorithm, since the  $a_{t+1}$s are static, would HopCPT still perform well when the seasonality in data is not regular? What if there are drifts in the distribution?\n\nAs noted above, the $a_t$ are the association weights that are calculated by the Hopfield mechanism at every time step. Hence, these values can change over time.\n\nRegarding seasonality: HopCPT uses covariate features to describe the current time step. Hence, HopCPT implicitly considers seasonality by comparing a given time step with the time steps from the memory. Whenever a time step of a certain season is encountered, it focuses on time steps from the same season. In fact, our experiments already comprise both regular and irregular seasonality patterns. For example, the streamflow dataset contains yearly reoccurring patterns (spring, summer, fall, winter) as well as irregularly reoccurring similar flood events. In both cases, HopCPT can identify related time steps based on the covariates.\n\nRegarding data drifts: an advantage of HopCPT is that the memory is updated with every new observation. Therefore, HopCPT can somewhat incorporate drifts in the target value as long as the covariates remain so that the learned representation is able to focus on the relevant observations. Empirically, the robustness against drifts is visible in our experiments on the sap flow dataset, as we discuss briefly in Section 3.2 (lines 283-289) and in more detail in Appendix A.3 (Error Trend & Figure 4).\n\nWe included this discussion of seasonality in the discussion of the scope of assumptions in the revised manuscript.\n\n> Limitations:\n> It might be good to stress that the guarantee is marginal and asymptotic, which can be achieved by rather simple algorithms (see Gibbs & Candes 2021, 2022). This is of course due to the properties of time series data, not a weakness of the approach.\n\nWe added a dedicated limitations section to the revised paper. The new section includes a discussion of the theoretical guarantees of HopCPT. We would like to note that besides the marginal property, we provide an asymptotic conditional guarantee in Theorem B.9. For more details on the limitation section see our response to reviewer tZC3."}, {"review_id": "sZQjdg9Rby", "reviewer": "Reviewer_Fgat", "summary": "This paper proposes a novel method for estimating uncertainty in prediction of time series. Modern Hopfield Networks is used as a method of measuring uncertainty based on the existence of similar historical data. This paper presents its construction method and shows that it works for multiple data and forecasting methods", "questions": "- Regarding the first point of Weakness, can you state why these data are sufficient, although they are a limit to the number of experiments? Or can they be increased?\n- For the second point, for example, can you compare it to or give reasons why it is not comparable to the following advanced methods of Gaussian processes\n\t- X. Zhu et. al, BayesianTransformedGaussianProcesses, TMLR (4/2023)\n\t- G. Corani et. al, Time Series Forecasting with Gaussian Processes Needs Priors, ECML PKDD 2021\n\t- X. Sun et. al, Recurrent neural network-induced Gaussian process, Neurocomputing 509(2022)\n- For the third point, can you specify why HMNs is more effective than other methods that can assume similar effects?", "limitations": "The authors do not explicitly address Limitation. On the other hand, the study is oriented toward reducing the social impact on the limitations of conventional methods and does not promote adverse effects.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The authors have clarified the issues with CP for time series and proposed a method to improve this point by applying MHNs, and experiments have shown very good results.", "weaknesses": "My four points of concern are as follows:\n- This paper consists only of a method proposal and an experimental evaluation, and does not include theoretical effects. In this structure, is an evaluation on four data sets sufficient to demonstrate the effectiveness of the proposed method? \n- The task setting of this paper seems to be related to Gaussian processes and Bayesian optimization as well. This point seems to be a natural one for many people, and while it may not be directly comparable, would it be appropriate to make no mention of it?\n- The proposed method incorporates MHNs, a function that calculates the reliability of forecasts from past information, into CP. The results shown in the experiment indicate the validity of the CP issue and the effectiveness of the improvement direction. However, the experiments did not provide justification for introducing HMNs to solve the problem.\n- Identification of issues and presentation of solutions is an important contribution, but it is a combination of existing methods, and I feel it is weak in terms of novelty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> This paper consists only of a method proposal and an experimental evaluation, and does not include theoretical effects. In this structure, is an evaluation on four data sets sufficient to demonstrate the effectiveness of the proposed method?\n\nBesides a brief theoretical motivation in the main paper, there is a theoretical analysis in Appendix B (supplementary material). As mentioned in response to reviewer yfWn, we addded  the main conclusions from the theoretical analysis in Appendix B to the main paper.\n\nThe empirical evaluation was done on 7 datasets (from 4 different domains), each with 4 different base prediction models and 3 different coverage levels. Overall, our evaluation is much broader than in the existing CP literature. We are confident that the clear performance improvement in the vast majority of experiments indeed shows the effectiveness of HopCPT.\n\n>  The task setting of this paper seems to be related to Gaussian processes and Bayesian optimization as well. This point seems to be a natural one for many people, and while it may not be directly comparable, would it be appropriate to make no mention of it?\n\nWe agree with the reviewer’s observation that Bayesian optimization and GP are related and should receive attention in our paper. We therefore added a discussion in the revised manuscript.\n\nAs rightly noted by the reviewer, our setting — where we conformalize existing black box estimators — is not directly comparable to GP: We aim to provide an approach that works for large and long multivariate time series datasets. Scalability to large datasets with many dimensions is not straightforward with GP, as for example noted by two of the papers (Corani et al. 2021, Sun et al. 2022) highlighted in your question below.\n\n> The proposed method incorporates MHNs, a function that calculates the reliability of forecasts from past information, into CP. The results shown in the experiment indicate the validity of the CP issue and the effectiveness of the improvement direction. However, the experiments did not provide justification for introducing HMNs to solve the problem.\n\nTo us, our paper provides three main reasons to introduce MHN in our framework:\n\n- (1) MHN allow to go beyond simply considering temporal proximity (as in EnbPI and NexCP). Instead, MHN also consider covariates. The need for using covariates is illustrated in the synthetic example (Section 2.5) and visible empirically in the performance of HopCPT (Section 3.2).\n- (2) MHN provide a flexible memory mechanism, which we can update for each new observation. In contrast to standard CP, these updates allow to account for drifts in the data. Our experiments on the sap flow dataset corroborate the benefits (see Section 3.2 (l 283ff) and Appendix A.3).\n- (3) MHN learn a measure of similarity based on covariates. The ablation in Appendix D (where we replaced the learned MHN similarity with a simple kNN approach) shows that a learned representation is needed, since naive similarity fails to capture the important structure that constitutes existing regimes.\n\nWe updated the HopCPT section to make these points more explicit.\n\n> Identification of issues and presentation of solutions is an important contribution, but it is a combination of existing methods, and I feel it is weak in terms of novelty.\n\nWe disagree with the claim that HopCPT is only a combination of existing methods: While Foygel Barber et al. (2022) propose the general idea of weighted CP, they only consider temporal proximity to arrive at fixed weights. To the best of our knowledge, HopCPT is the first CP method that uses the covariates of the time series to derive dynamic weights—while still being able to consider temporal proximity. In that sense, HopCPT generalizes NexCP but provides much broader capabilities. See also the answer to reviewer YCj6 where we outline our contributions in more detail.\n\n> Regarding the first point of Weakness, can you state why these data are sufficient, although they are a limit to the number of experiments? Or can they be increased?\n\nWe addressed these questions in our response to your first point.\n\n> For the second point, for example, can you compare it to or give reasons why it is not comparable to the following advanced methods of Gaussian processes  \n> ...\n\nIn our response to your second point we addressed why our setting is not directly comparable to Gaussian Processes. Nevertheless, we also think that GP are related to our work and added a discussion about the differences between GP and CP to the paper (including your references). Apart from the general point mentioned above, we would like to give some specific reasons why a direct comparison to the mentioned papers is difficult:\n- Zhu et al. (2023): They propose a method to make BTG more scalable in terms of dimensions. However, their approach is mainly useful in low-data regimes, which is opposed to the setting we are aiming for. Their experiments use training sets of hundreds of samples, while our training data comprises up to millions of samples.\n- Corani et. al (2021): Their approach requires to define potential seasonalities a priori and manually, which is often not possible, especially in cases with irregularly repeating patterns (e.g., in streamflow prediction).\n- Sun et. al (2022):  Their method is limited in terms of training set size. This is reflected in the experiments, where they even subsample the number of training examples.\n\n> For the third point, can you specify why HMNs is more effective than other methods that can assume similar effects?\n\nWe addressed this question directly in our answer to the third point above.\n\n> Limitations:  \n> ... \n\nWe added a dedicated limitations section to the revised paper that discusses the limitations of our method. See also our response to reviewer tZC3 for more details."}]

Author rebuttal:
We would like to thank all reviewers for their thorough reviews, helpful suggestions, and constructive comments. To summarize, we made the following main changes to the manuscript to follow the suggestions and comments of the reviewers:
- We added a discussion paragraph that elaborates on the main results of our theoretical analysis (Supplemental Appendix B) to the theory section in the main paper.
- We added a section that is dedicated to the limitations of our approach (see answer to reviewer tZC3).
- We now report the results of an ablation study in the Appendix. There, we analyze the influence of “searching” for $\beta$ as in Xu & Xi (2022a, 2022b) on HopCPT, as suggested by reviewer yfWn (results are given in the attached PDF).
- In the new version, we now discuss Gaussian Process approaches and highlight their similarities and differences to our work.
- We expanded the description on the connection between MHN and attention.


Meta-review: By and large, this paper has been well received by the reviewers. In particular, they all agree that the paper is well-structured and clearly presented, and that the experimental results are convincing. In the beginning, two of the reviewers were nevertheless quite critical and raised a couple of concerns. These could mostly be clarified in the rebuttal and discussion phase. Eventually, all reviewers were on the positive side, although the novelty of the paper still appears to be somewhat limited, and the empirical findings are not backed up by theoretical results. It could also be seen a bit critical that the authors made quite substantial changes to the manuscript. Admittedly, these were requested by the reviewers, but actually couldn't be checked. So in the end, there are pros and cons, but the consensus is that the former outweigh the latter.
Decision: Accept (poster)