SlowGuess/NeurIPS_2023_OCR huggingface dataset:
HopCPT builds upon the formal guarantees of CP, but it is able to relax the data exchangeability assumption of CP which are problematic for time series data. That said, HopCPT still relies on assumptions on how it learns to identify the respective error regimes and exchangability within regimes (see Section 2.1 and Appendix B). Our extensive empirical evaluation suggests that these assumptions hold in practice. HopCPT is generally well-suited for very long time series because the memory requirement in the inference scales only linearly with the memory size. Nevertheless, for extremely large datasets it can be the case that not all historical time steps may be kept in the Hopfield memory
anymore. HopCPT can, similar to existing methods, disregard time steps by removing the oldest entries from the memory, or alternatively use a sub-sampling strategy. On the other hand, for datasets with very scarce data it might be difficult to learn a useful embedding of the time steps. In that case, it could be better to use kNN rather than learning the similarity with MHN (Appendix D). Additional considerations regarding the potential social impact of the method are summarized in Appendix I.
This section provides a comparative evaluation of HopCPT and analyzes its association mechanism.
Datasets. We use datasets from four different domains: (a) Three solar radiation datasets from the US National Solar Radiation Database (Sengupta et al., 2018). The smallest one consists of 8 time series from different locations over a period of 84 days. This dataset is also used in Xu & Xie (2022a,b). In addition, we evaluate on a 1-year and a 3-year dataset, with 50 time series each. (b) An air quality dataset from Beijing, China (Zhang et al., 2017). It consists of 12 time series, each from a different measurement station, over a period of 4 years. The dataset has two prediction targets, the PM10 (as in Xu & Xie, 2022a,b) and PM2.5 concentrations, which we evaluate separately. (c) Sap flow $^2$  measurements from the Sapfluxnet data project (Poyatos et al., 2021). Since the individual measurement series are considerably heterogeneous in length, we use a subset of 24 time series, each with between 15,000 and 20,000 data points and varying sampling rates. (d) Streamflow, a dataset of water flow measurements and corresponding meteorologic observations from 531 rivers across the continental United States (Newman et al., 2015; Addor et al., 2017). The measurements span 28 years at a daily time scale. For more detailed information about the datasets see Appendix C.
Prediction models. We use four prediction models for the solar radiation, the air quality, and the sap flux datasets to ensure that our results generalize: a random forest, a LightGBM, a ridge regression, and a Long Short-Term Memory (LSTM) (LSTM; Hochreiter & Schmidhuber, 1997) model. For the former three models we follow the related work (Xu & Xie, 2022a,b; Foygel Barber et al., 2022) and train a separate prediction model for each individual time series. The random forest and LightGBM models are implemented with the darts library (Herzen et al., 2022), the ridge regression model with sklearnn (Pedregosa et al., 2011). For the LSTM model, we instead train a global model on all time series of a dataset, as is standard for state-of-the-art deep learning models (e.g., Oreshkin et al., 2020; Salinas et al., 2020; Smyl, 2020). The LSTM is implemented with PyTorch (Paszke et al., 2019). For the streamflow dataset, we deviate from this scheme and instead only use the state-of-the-art model, which is also an LSTM network (Kratzert et al., 2021, see Appendix C).
Compared approaches. We compare HopCPT to different state-of-the-art CP approaches for time series data: EnbPI (Xu & Xie, 2022a), SPCI (Xu & Xie, 2022b), NexCP (Foygel Barber et al., 2022), CopulaCPTS (Sun & Yu, 2022), and AdaptiveCI³ (Gibbs & Candes, 2021). In addition, the results of standard split CP (CP) serve as a baseline, which, for the LSTM base predictor, corresponds to CF-RNN (Stankeviciúte et al., 2021) in our setting (one-step, univariate target variable). Appendix A.1 describes our hyperparameter search for each method. For SPCI, an adaption of the original algorithm was necessary to provide scalability to larger datasets. Appendix A.2 contains more details and an empirical justification. Appendix G evaluates the addition of AdaptiveCI (Gibbs & Candes, 2021) as an enhancement to HopCPT and the other time series CP methods. Appendix A.3 gives a comparison to non-CP methods. Lastly, Appendix D presents a supplemental comparison to kNN that shows the superiority of the learned similarity representation in HopCPT.
${}^{2}$  Sap flow refers to the movement of water within a plant. In the environmental sciences, it is commonly used as a proxy for plant transpiration.
AdaptiveCI works on top of an existing quantile prediction method. Hence, we exclusively make comparisons based on the LightGBM models that can predict quantiles instead of point estimates. The approach is orthogonal to the remaining compared models and could be combined with HopCPT.
Table 1: Performance of the evaluated CP algorithms for the Solar (3Y), Air Quality (10PM), Sap flow, and Streamflow (SF) datasets. The specified miscoverage level is  $\alpha = 0.1$  for all experiments. The column FC specifies the prediction algorithm used for the experiment (Forest: Random Forest, LGBM: LightGBM, Ridge: Ridge Regression, LSTM: LSTM neural network). Bold numbers correspond to the best result for the respective metric in the experiment (PI-Width and Winkler score), given that  $\Delta \mathrm{Cov} \geq -0.25\alpha$ , i.e., the specific algorithm reached at least approximate coverage (the result is grayed otherwise). The error term represents the standard deviation over repeated runs with different seeds (results without an error term are from deterministic models).
SF	LSTM	Sap flow	Air 10 PM	Solar 3Y	UC
LSTM	Ridge	LGBM	Forest	LSTM	Ridge	LGBM	Forest	LSTM	Ridge	LGBM	Forest	LSTM	Ridge	LGBM	Forest	HCPCT
Δ Cov	0.0029±0.012	0.012±0.000	-0.031	131.1	166.6	174.9	174.6
Δ Cov	0.001±0.003	0.014±0.000	-0.002	131.1	166.6	174.9	174.6
Δ Cov	0.001±0.003	0.014±0.000	-0.002	131.1	166.6	174.9	174.6	2.47	2.53	2.75	2.76
Δ Cov	0.001±0.003	0.014±0.000	-0.002	131.1	166.6	174.9	174.6
Δ Cov	0.001±0.003	0.014±0,000	-0.002	131.1	166.6	174.9	174.6
Δ Cov	0.001±0.003	0.014±0.000	-0.002	131.1	166.6	174.9
Δ Cov	0.001±0.003	0.014±0.000	-0.002	131.1	166.6	174.9
Δ Cov	0.001±0.003	0.018±0.000	-0.001	131.1	166.6	174.9
Δ Cov	0.001±0.003	0.018±0.000	-0.001	131.1	166.6	174.9
DeltaCov	0.001±0.000	0.018±0.000	-0.001	131.1	166.6	174.9
Δ Cov	0.001±0.000	0.018±0.000	-0.001	131.1	166.6	174.9
Δ Cov	0.001±0.000	0.018±0.000	-0.001	121.0	153.3	153.7
Δ Cov	0.001±0.000	0.018±0.000	-0.001	121.0	153.3	153.7
Δ Cov	0.001±0.000	0.021±0.000	-0.001	121.0	153.3	153.7
Δ Cov	0.001±0.000	0.021±0.000	-0.001	121.0	153.3	153.7	-0.001	0.014	-0.001	-0.007	-0.010
Δ Cov	0.001±0.000	0.021±0.000	-0.001	121.0	153.3	153.7	-0.001	0.014	-0.001	-0.007	-0.010
Δ Cov	0.001±0.000	0.021±0.000	-0.001	121.0	153.3	153.7	-0.001	0.014	-0.001	-0.007	-0.010
Metrics. In our analyses, we compute  $\Delta$  Cov, PI-Width, and the Winkler score (Winkler, 1972) per time series and miscoverage level. The Winkler score jointly elicits miscoverage and interval width in a single metric:
$$
\mathrm {W S} _ {\alpha} \left(\boldsymbol {Z} _ {t + 1}, y _ {t + 1}\right) = \left\{ \begin{array}{l l} \mathrm {I W} _ {t} ^ {\alpha} \left(\boldsymbol {Z} _ {t + 1}\right) + \frac {2}{\alpha} \left(y _ {t + 1} - \widehat {C} _ {t} ^ {\alpha , u} \left(\boldsymbol {Z} _ {t + 1}\right)\right) & \text {i f} y _ {t + 1} > \widehat {C} _ {t} ^ {\alpha , u} \left(\boldsymbol {Z} _ {t + 1}\right), \\ \mathrm {I W} _ {t} ^ {\alpha} \left(\boldsymbol {Z} _ {t + 1}\right) + \frac {2}{\alpha} \left(\widehat {C} _ {t} ^ {\alpha , l} \left(\boldsymbol {Z} _ {t + 1}\right) - y _ {t + 1}\right) & \text {i f} y _ {t + 1} <   \widehat {C} _ {t} ^ {\alpha , l} \left(\boldsymbol {Z} _ {t + 1}\right), \\ \mathrm {I W} _ {t} ^ {\alpha} \left(\boldsymbol {Z} _ {t + 1}\right) & \text {e l s e .} \end{array} \right. \tag {9}
$$
The score is calculated per time step  $t$  and miscoverage level  $\alpha$ . It corresponds to the interval width  $\mathrm{IW}_t^\alpha = \widehat{C}_t^{\alpha ,u} - \widehat{C}_t^{\alpha ,l}$  whenever the observed value  $y_{t}$  is between the upper bound  $\widehat{C}_t^{\alpha ,u}(Z_{t + 1})$  and the lower bound  $\widehat{C}_t^{\alpha ,l}(Z_{t + 1})$  of  $\widehat{C}_t^\alpha (Z_{t + 1})$ . If  $y_{t + 1}$  is outside these bounds, a penalty is added to the interval width. We evaluate the mean Winkler score over all time steps.
We repeated each experiment with 12 different seeds. For brevity, we only show the mean performance of one dataset per domain for  $\alpha = 0.1$  in the main paper (which is the most commonly reported value in the CP literature; e.g., Xu & Xie, 2022b; Foygel Barber et al., 2022; Gibbs & Candes, 2021). Appendix A.3 presents additional results for all datasets and more  $\alpha$  levels.
3.2 Results & Discussion
HopCPT has the most efficient prediction intervals for each domain — with only one exception (Table 1; significance tested with a Mann-Whitney  $U$  test at  $p < 0.005$ ) for the evaluated miscoverage level ( $\alpha = 0.1$ ). In multiple experiments (Solar (3Y), Solar (1Y)), the HopCPT prediction intervals are less than half as wide as those of the approach with the second-smallest PI-Width. The second-most efficient intervals are predicted most often by SPCI. This ranking also holds for the Winkler score, where HopCPT achieves the best (i.e., lowest) Winkler scores and SPCI ranks second in most experiments. Notably, these results reflect the increasing requirements posed by each uncertainty estimation method on the dataset (Section 1.1).
Furthermore, HopCPT outperforms the other methods regarding both Winkler score and PI-Width when we evaluate over additional datasets and at different miscoverage levels (see Appendix A.3). HopCPT is the best-performing approach in the vast majority of cases. The variation in size of the different solar datasets (results for Solar (1Y) and Solar (3M) in Appendix A.3) shows that HopCPT especially increases its lead when more data is available. We hypothesize that this is because the MHN can learn more generalizable retrieval patterns with more data. However, data scarcity is not a limitation of similarity-based CP in general. In fact, a simplified and almost parameter-free variation of HopCPT, which replaces the MHN with kNN retrieval, performs best on the smallest solar dataset, as we demonstrate in Appendix D.
Most approaches have a coverage gap close to zero in almost all experiments — they approximately achieve the specified marginal coverage level. This is also reflected by the fact that the ranking of Winkler scores and PI-Widths agree in most evaluations. Appendix A.4 provides a supplementary analysis of the local coverage. The results for non-CP methods show a different picture (see Appendix A.3 and Table 7). Here, the non-CP models most often exhibit very narrow prediction intervals at the cost of high miscoverage.
Interestingly, standard CP also achieves good coverage for all experiments at the cost of inefficient prediction intervals. There is one notable exception: for the Sap flow dataset with ridge regression, we find  $\Delta \mathrm{Cov} = -0.358$ . In this case, we argue that the bad performance of standard CP is driven by a strong violation of the (required) exchangeability assumption. Specifically, a trend in the errors leads to a distribution shift over time (as illustrated in Appendix A.3, Figure 4). HopCPT and NexCP handle this shift without substantial loss in coverage. The inferior coverage of SPCI is likely influenced by the modification to larger datasets (see Appendix A.2).
Performance gaps between the approaches differ considerably across datasets, but are generally consistent across prediction models. The biggest differences between the best and worst methods exist in Solar (3Y) and Sap flow, likely due to the distinctive regimes in these datasets. The smallest (but still significant) differences are visible in the Streamflow data. On this dataset, we also evaluated the state-of-the-art non-CP uncertainty model in the domain (Klotz et al., 2022, see Appendix A.3) and found that HopCPT outperforms it with respect to efficiency.
Figure 3: Exemplary visualization of the 30 highest association weights that the MHN places on previous time steps (some are before the visualized part of the memory). HopCPT retrieves similar peak values when estimating at a peak, and it retrieves similar small values when estimating at a small value.
In terms of computational resources, SPCI is most demanding followed by HopCPT, as both methods require a learning step in contrast to the other methods. A detailed overview can be found in Appendix A.6.
To assess whether HopCPT learns meaningful associations within regimes, we conducted a qualitative study on the Solar (3Y) dataset. Figure 3 shows that HopCPT retrieves the most highly weighted errors from time steps with similar regimes. The illustrated weighting at the time step with a low prediction value retrieves previous time steps which are also in low-valued regimes. Similarly, Figure 5 (Appendix A.3) suggests that the learned distinction corresponds to the error regimes, which is crucial for HopCPT.
We have introduced HopCPT, a novel CP approach for time series tasks. HopCPT uses continuous Modern Hopfield Networks to construct prediction intervals based on previously seen events with similar error distribution regimes. We exploit that similar features lead to similar errors. Associating features with errors identifies regimes with similar error distributions. HopCPT learns this association in the Modern Hopfield Network, which dynamically adjusts its focus on stored previous features according to the current regime.
Our experiments with established and novel datasets show that HopCPT achieves state of the art: It generates more efficient prediction intervals than existing CP methods and approximately preserves coverage, even in non-exchangeable scenarios like time series. HopCPT comes with formal guarantees within the CP framework for uncertainty estimation in real-world applications such as streamflow prediction. Furthermore, HopCPT scales well to large datasets, provides multiple coverage levels after calibration, and shares information across individual time series within a dataset.
Future work comprises: (a) Drawing from multiple time series during the inference phase. HopCPT is already trained on the whole dataset at once, but it might be advantageous to leverage more information during inference as well. (b) Investigating direct training objectives from which learning-based CP might benefit. (c) Using HopCPT beyond time series, as non-exchangeability is also an issue in other domains which limits the applicability of existing CP methods.
We would like to thank Angela Bitto-Nemling for discussions during the development of the method. The ELLIS Unit Linz, the LIT AI Lab, the Institute for Machine Learning, are supported by the Federal State Upper Austria. We thank the projects AI-MOTION (LIT-2018-6-YOU-212), DeepFlood (LIT-2019-8-YOU-213), Medical Cognitive Computing Center (MC3), INCONTROL-RL (FFG-881064), PRIMAL (FFG-873979), S3AI (FFG-872172), DL for GranularFlow (FFG-871302), EPILEPSIA (FFG-892171), AIRI FG 9-N (FWF-36284, FWF-36235), AI4GreenHeatingGrids(FFG- 899943), INTEGRATE (FFG-892418), ELISE (H2020-ICT-2019-3 ID: 951847), Stars4Waters (HORIZON-CL6-2021-CLIMATE-01-01). We thank Audi.JKU Deep Learning Center, TGW LOGISTICS GROUP GMBH, Silicon Austria Labs (SAL), FILL Gesellschaft mbH, Anyline
GmbH, Google, ZF Friedrichshafen AG, Robert Bosch GmbH, UCB Biopharma SRL, Merck Healthcare KGaA, Verbund AG, GLS (Univ. Waterloo) Software Competence Center Hagenberg GmbH, TÜV Austria, Frauscher Sensonic, TRUMPF and the NVIDIA Corporation. The computational results presented have been achieved in part using the Vienna Scientific Cluster (VSC) and the In-Memory Supercomputer MACH-2 (operated by the Scientific Computing Administration of JKU Linz).
Abbott, L. F. and Arian, Y. Storage capacity of generalized networks. Physical Review A, 36: 5091-5094, 1987. doi: 10.1103/PhysRevA.36.5091.
Addor, N., Newman, A. J., Mizukami, N., and Clark, M. P. The CAMELS data set: catchment attributes and meteorology for large-sample studies. Hydrology and Earth System Sciences (HESS), 21(10):5293-5313, 2017.
Angelopoulos, A., Bates, S., Malik, J., and Jordan, M. I. Uncertainty sets for image classifiers using conformal prediction. arXiv preprint arXiv:2009.14193, 2020.
Angelopoulos, A. N. and Bates, S. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021.
Baldi, P. and Venkatesh, S. S. Number of stable points for spin-glasses and neural networks of higher orders. Physical Review Letters, 58:913-916, 1987. doi: 10.1103/PhysRevLett.58.913.
Bishop, C. M. Mixture density networks. Technical report, Neural Computing Research Group, 1994.
Caputo, B. and Niemann, H. Storage capacity of kernel associative memories. In Proceedings of the International Conference on Artificial Neural Networks (ICANN), pp. 51-56, Berlin, Heidelberg, 2002. Springer-Verlag.
Chen, H. H., Lee, Y. C., Sun, G. Z., Lee, H. Y., Maxwell, T., and Giles, C. L. High order correlation model for associative memory. AIP Conference Proceedings, 151(1):86-99, 1986. doi: 10.1063/1.36224.
Corani, G., Benavoli, A., and Zaffalon, M. Time series forecasting with gaussian processes needs priors. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 103-117. Springer, 2021.
Dong, Z., Chen, Z., and Wang, Q. Retrosynthesis prediction based on graph relation network. In 2022 15th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), pp. 1-5. IEEE, 2022.
Fontana, M., Zeni, G., and Vantini, S. Conformal prediction: a unified review of theory and new challenges. Bernoulli, 29(1):1-23, 2023.
Fox, M. and Rubin, H. Admissibility of quantile estimates of a single location parameter. The Annals of Mathematical Statistics, pp. 1019-1030, 1964.
Foygel Barber, R., Candes, E. J., Ramdas, A., and Tibshirani, R. J. The limits of distribution-free conditional predictive inference. Information and Inference: A Journal of the IMA, 10(2):455-482, 2021.
Foygel Barber, R., Candes, E. J., Ramdas, A., and Tibshirani, R. J. Conformal prediction beyond exchangeability. arXiv preprint arXiv:2202.13415, 2022.
Fürst, A., Rumetshofer, E., Lehner, J., Tran, V. T., Tang, F., Ramsauer, H., Kreil, D. P., Kopp, M. K., Klambauer, G., Bitto-Nemling, A., and Hochreiter, S. CLOOB: Modern Hopfield Networks with infoLOOB outperform CLIP. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022.
Gal, Y. and Ghahramani, Z. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059. PMLR, 2016.
Xia, Y., Mitchell, K., Ek, M., Sheffield, J., Cosgrove, B., Wood, E., Luo, L., Alonge, C., Wei, H., Meng, J., et al. Continental-scale water and energy flux analysis and validation for the North American Land Data Assimilation System project phase 2 (NLDAS-2): 1. intercomparison and application of model products. Journal of Geophysical Research: Atmospheres, 117(D3), 2012.
Xu, C. and Xie, Y. Conformal prediction for time series. arXiv preprint arXiv:2010.09107, 2022a.
Xu, C. and Xie, Y. Sequential predictive conformal inference for time series. arXiv preprint arXiv:2212.03463, 2022b.
Xu, Y., Yu, W., Ghamisi, P., Kopp, M., and Hochreiter, S. Txt2Img-MHN: Remote sensing image generation from text using Modern Hopfield Networks. arXiv preprint arXiv:2208.04441, 2022.
Zaffran, M., Feron, O., Goude, Y., Josse, J., and Dieuleveut, A. Adaptive conformal predictions for time series. In International Conference on Machine Learning, pp. 25834-25866. PMLR, 2022.
Zhang, S., Guo, B., Dong, A., He, J., Xu, Z., and Chen, S. X. Cautionary tales on air-quality improvement in Beijing. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 473(2205):20170457, 2017.
Zhu, L. and Laptev, N. Deep and confident prediction for time series at Uber. In 2017 IEEE International Conference on Data Mining Workshops (ICDMW), pp. 103-110. IEEE, 2017.
Zhu, X., Huang, L., Lee, E. H., Ibrahim, C. A., and Bindel, D. Bayesian transformed gaussian processes. Transactions on Machine Learning Research, 2023.
A Extended Experimental Setup & Results
A.1 Hyperparameter Search
We conducted an individual hyperparameter grid search for each predictor-dataset combination. For methods that require calibration data (HopCPT, AdaptiveCI, NexCP), each calibration set was split in half: One part served as actual calibration data while the other half was used for validation. As EnbPI requires only the  $k$  past points, we used the full calibration set minus these  $k$  points for validation, so that it could fully exploit the available data. Table 2 shows all sets of hyperparameters used for the search.
Table 2: Parameters used in the hyperparameter search of the uncertainty models
Method	Parameter	Value
HopCPT	Learning Rate	0.01, 0.001
Dropout	0, 0.25, 0.5
Time Encode	yes/no
AdaptiveCI	Mode	simple, momentum
γ	0.002, 0.005, 0.01, 0.02
EnbPI	Window Length	200, 150, 125, 100, 75, 50, 25, 10
NexCP	ρ	0.999, 0.995, 0.993, 0.99, 0.98, 0.95, 0.90
kNN	k-Top Share	0.025, 0.05, 0.10, 0.15, 0.20, 0.25, 0.30, 0.35
MCD	Learning Rate	0.005, 0.001, 0.0001
Dropout	0.1, 0.25, 0.5
Batch Size	512, 256
Hidden Size	64, 128, 256
Gaus	Learning Rate	0.005, 0.001, 0.0001
Dropout	0, 0.1, 0.25
Batch Size	512, 256
Hidden Size	64, 128, 256
MDN	Learning Rate	0.005, 0.001, 0.0001
Dropout	0, 0.1, 0.25
Batch Size	512, 256
Hidden Size	64, 128, 256
# Components	3, 5, 7
Model selection. Model selection was done uniformly for all algorithms: As a first step, all models with a negative  $\Delta$  Cov on the validation set were excluded. From the remaining models, the model with the smallest PI-Width was selected. In cases where no model achieved non-negative  $\Delta$  Cov, the model with the highest  $\Delta$  Cov was selected.
HopCPT. HopCPT was trained for 3,000 epochs in each experiment. We chose this number to make sure that the loss and model selection metric curves are already converged. Throughout training, we validated every 5 epochs and selected the model that best fulfilled the model selection criteria described above. AdamW (Loshchilov & Hutter, 2019) with standard parameters  $(\beta_{1} = 0.9, \beta_{2} = 0.999, \delta = 0.01)$  was used as optimizer. The learning rate was part of the hyperparameter search. Depending on the dataset size, the batch size was set to 2 or 4, where a batch size of  $n$  would mean that the full training part of the calibration set of  $n$  time series is used.
Note that this is only the case in the hyperparameter search. In the evaluation, the full calibration set was used for calibration.
SPCI. The computational demand of SPCI did not allow to conduct a full hyperparameter search for the window length parameter (see more in Section A.2). Since SPCI applies a random forest on the window, one can assume that it is capable to find the relevant parts of the window. On top of that, a longer window has less risk than a window that is too short and (potentially) cuts off relevant information. Hence, we set the window length to 100 for all experiments, which corresponds to the longest setting in the original paper. To check our reasoning, we evaluated the performance of SPCI with window length 25 on all but the largest two datasets (Table 4 and Table 5). We found hardly any differences except for the smallest datasets Solar (3M). In that case we reason that the result is due to the limited calibration data in this setting.