SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \Pr \left\{K \in \mathcal {S} \left(R (D ^ {K})\right) \right\} = \sum_ {i = 1} ^ {t + 1} \Pr \left\{K = i \text {a n d} i \in \mathcal {S} \left(R (D ^ {i})\right) \right\} \\ = \sum_ {i = 1} ^ {t + 1} \tilde {w} _ {i} \cdot \Pr \left\{i \in \mathcal {S} \left(R (D ^ {i})\right) \right\} \\ \leq \sum_ {i = 1} ^ {t + 1} \tilde {w} _ {i} \cdot \left(\Pr \left\{i \in \mathcal {S} (R (D)) \right\} + \mathrm {d} _ {\mathrm {T V}} (R (D), R (D ^ {i}))\right) \\ = \mathbb {E} \left[ \sum_ {i \in \mathcal {S} (R (D))} \tilde {w} _ {i} \right] + \sum_ {i = 1} ^ {t} \tilde {w} _ {i} \cdot \mathsf {d} _ {\mathsf {T V}} \big (R (D), R (D ^ {i}) \big) \\ \leq \alpha + \sum_ {i = 1} ^ {t} \tilde {w} _ {i} \cdot \mathrm {d} _ {\mathsf {T V}} \big (R (D), R (D ^ {i}) \big), \\ \end{array}
$$
where we use (21) from the penultimate to the last line. The first to the second line follows from  $K \perp D^i$ , which holds because  $K \perp D$  and  $D^i$  is a permutation of  $D$  that does not depend on  $\mathbf{K}$ .
Foygel Barber et al. (2022) show two additional variations of the coverage gap bound based on Theorem B.2. We do not replicate them here. One notable observation of theirs, however, is that instead of using the total variation distance based on the residual vectors also the distance of the raw data vector can be used:
$$
\Delta \operatorname {C o v} \geq - \sum_ {i} \tilde {w} _ {i} \cdot \mathrm {d} _ {\mathrm {T V}} (D, D ^ {i}). \tag {23}
$$
This coverage bound suggests that low weights for time steps corresponding to high total variation distance ensure small coverage gaps. However, lower weights weaken the estimation quality of the prediction intervals. Hence, all relevant time steps should receive high weights, while all others should receive weights close to zero.
HopCPT leverages these results by using the association weights  $a_i$  from Equation 6 to reflect the weights  $\tilde{w}_i$  of non-exchangeable split conformal prediction. HopCPT aims to assign high association weights to data points that are from the same error distribution as the current point, and low weights to data points in the opposite case. By Theorem B.2, association weights that follow this pattern lead to a small coverage gap.
An asymptotic (conditional) coverage bound If we assume clearly distinguished error distributions instead of gradually shifting ones, hard weight assignment, i.e., association weights that are either zero or one, are a meaningful choice. Given this simplification we can directly link HopCPT to the work of Xu & Xie (2022a).
Xu & Xie (2022a) introduce a variant of (split) conformal prediction with asymmetric prediction intervals:
$$
\widehat {C} _ {t} ^ {\alpha} \left(x _ {t + 1}\right) := \left[ \hat {\mu} _ {- (t + 1)} \left(x _ {t + 1}\right) + \mathrm {Q} _ {\frac {\alpha}{2}} \left(\left\{\hat {\epsilon} _ {i} \right\} _ {i = 1} ^ {t}\right), \hat {\mu} _ {- (t + 1)} \left(x _ {t + 1}\right) + \mathrm {Q} _ {1 - \frac {\alpha}{2}} \left(\left\{\hat {\epsilon} _ {i} \right\} _ {i = 1} ^ {t}\right) \right] \tag {24}
$$
The approach taken by Xu & Xie (2022a) additionally adjusts the share of  $\alpha$  between the upper and lower quantiles such that the width of the prediction interval is minimized. In contrast, HopCPT adopts a simpler strategy and uses  $\frac{\alpha}{2}$  on both sides, which mitigates the need for excessive recalculation of the quantile function.
Based on the introduced interval calculation procedure, they also provide bounds regarding its coverage properties, in particular about the possible coverage gap. While Xu & Xie (2022a) base their assumption on error properties only in temporal proximity, we generalize this to proximity in the learned feature representation (which can include but is not limited to information about temporal proximity). Based on this proximity we assume:
Assumption B.3 (Error regimes). Given  $\{(\mathbf{Z}_i, y_i, \hat{y}_i)\}_{i=1}^{t+1}$  which represents the (coded) dataset and the predictions  $\hat{y}_i$  of  $\hat{\mu}$ , there exists a partition  $\mathcal{T} = \{T_i\}_{i \in I}$  of the time steps  $\{1, \dots, t\}$ , so that
(i)  $\forall i\in I:\{\epsilon_t = y_t - \hat{y}_t\mid t\in T_i\} \sim E_i,$
(ii)  $\exists i\in I:\{\epsilon_{t + 1}\} \sim E_i$
(iii) the subset  $T_{i}$  corresponding to  $t + 1$  can be identified based on  $\mathbf{Z}_{t + 1}$
(iv)  $\forall i,j\in I:i = j\lor E_i\neq E_j,$
where each  $E_{i}$  is an arbitrary error distribution.
Properties (i) and (ii) imply that any given time step  $t + 1$  can be associated to a subset  $T_{i}$  where all errors follow a common error distribution  $E_{i}$ ; property (iii) ensures that the partition can be found or approximated from a set of features. Property (iv) is not necessary for the discussion at hand but avoids degenerate solutions, since tight coverage bounds call for assigning all errors from the same error distribution to one large partition. If the errors of the associated subset were sampled i.i.d. from  $E_{i}$ , we could use standard split conformal methods (Vovk et al., 2005), and the variation distance between permutations of the errors in properties (i) and (ii) is zero. In practice, error noise can make it challenging to find and delineate a given partition. Thus, HopCPT uses a soft-selection approach to approximate the discrete association of a partition and the current features.
To align HopCPT with the results of Xu & Xie (2022a), and thus make use of their proposed coverage bounds for asymmetric prediction intervals, we first introduce their decomposition of the error:  $\epsilon_{t} = \check{\epsilon}_{t} + (\mu (\mathbf{x}_{t}) - \hat{\mu} (\mathbf{x}_{t}))$ , where  $f$  is the true generating function,  $\hat{\mu}$  the prediction model, and  $\check{\epsilon}$  a noise term associated with the label. The notation  $\mu_{-t}(\mathbf{x})$  stems from Xu & Xie (2022a) and denotes an evaluation of  $\mu$ , selected without using  $t$ .
Next, we adopt their i.i.d. assumption about the noise with regard to our approach:
Assumption B.4 (Noise is approximately i.i.d. within an error regime; adaptation of Xu & Xie (2022a), Assumption 1). Assume  $\exists i\in I:\{\check{\epsilon}_t|t\in T_i\} \cup \{\check{\epsilon}_{t + 1}\} \stackrel {\mathrm{iid}}{\sim}E_i$  and that the CDF  $\tilde{F}_i$  of  $E_{i}$  is Lipschitz continuous with constant  $L_{i} > 0$
This i.i.d. assumption within each error regime only concerns the noise of the true generating function. Xu & Xie (2022a) note that the generating process itself can exhibit arbitrary dependence and be highly non-stationary while still having i.i.d. noise. The assumption is thus relatively weak.
Assumption B.5 (Estimation quality; adaptation of Xu & Xie (2022a), Assumption 2). There exist real sequences  $\{\delta_{i,|T_i|}\}_{i,|T_i|\geq 1}$  such that
(i)  $\forall i\in I:\frac{1}{|T_i|}\sum_{t\in T_i}(\hat{\mu}_{-t}(\pmb {x}_t) - \mu (\pmb {x}_t))^2\leq \delta_{i,|T_i|}^2$
(ii)  $\exists i\in I:|\hat{\mu}_{-(t + 1)}(\pmb{x}_{t + 1}) - \mu (\pmb{x}_{t + 1})|\leq \delta_{i,|T_i|}$ , where the  $i$  for a given  $t + 1$  is the same  $i$  as in assumption B.4.
We want to bound the coverage of (24), which is based on the prediction error  $\epsilon$ . However, Assumption B.4 only considers the noise  $\check{\epsilon}$  of the true generating function. Xu & Xie (2022a) bind the difference between the CDF of the true generation function  $\tilde{F}_i$ , the empirical analogue  $\tilde{F}_i(x) := \frac{1}{|T_i|} \sum_{j \in T_i} \mathbf{1}\{\check{\epsilon}_j \leq x\}$ , and the empirical CDF of the prediction error  $F_i(x) := \frac{1}{|T_i|} \sum_{j \in T_i} \mathbf{1}\{\epsilon_j \leq x\}$ .
In particular, they state two lemmas:
Lemma B.6 (Adaptation of Xu & Xie (2022a), Lemma 1). Under Assumption B.4, for any subset size  $|T_i|$ , there is an event  $A_T$  which occurs with probability at least  $1 - \sqrt{\log(16|T_i|) / |T_i|}$ , such that conditioning on  $A_T$ ,
$$
\sup  _ {x} | \tilde {F} _ {i} (x) - \tilde {F} _ {i} (x) | \leq \sqrt {\log (1 6 | T _ {i} |) / | T _ {i} |}.
$$
Lemma B.7 (Adaptation of Xu & Xie (2022a), Lemma 2). Under Assumptions B.4 and B.5, we have
$$
\sup  _ {x} | F _ {i} (x) - \tilde {F} _ {i} (x) | \leq (L _ {i} + 1) \delta_ {i, | T _ {i} |} ^ {2 / 3} + 2 \sup  _ {x} | \tilde {F} _ {i} (x) - \check {F} _ {i} (x) |.
$$
With this result we arrive at a bound for a conditional and a marginal coverage:
Theorem B.8 (Conditional coverage gap; adaptation of Xu & Xie (2022a), Theorem 1). Given the prediction interval  $\widehat{C}_t^\alpha$  defined in (24). Under Assumption B.4 and B.5, for any subset size  $|T_i|$ ,  $\alpha \in (0,1)$ , and  $\beta \in [0,\alpha]$ , we have:
$$
\begin{array}{l} \exists i \in I: \left| \Pr \left\{Y _ {t + 1} \in \widehat {C} _ {t} ^ {\alpha} \left(X _ {t + 1}\right) \mid X _ {t + 1} = \boldsymbol {x} _ {t + 1} \right\} - (1 - \alpha) \right| \\ \leq 1 2 \sqrt {\log (1 6 | T _ {i} |) / | T _ {i} |} + 4 \left(L _ {i} + 1\right) \left(\delta_ {i, | T _ {i} |} ^ {2 / 3} + \delta_ {i, | T _ {i} |}\right). \tag {25} \\ \end{array}
$$
Furthermore, if  $\{\delta_{i,|T_i|}\}_{|T_i|\geq 1}$  converges to zero, the upper bound in (25) converges to  $0$  when  $|T_{i}|\to \infty$ , and thus the conditional coverage is asymptotically valid.
Theorem B.9 (Marginal coverage gap; adaptation of Xu & Xie (2022a), Theorem 2). Given the prediction interval  $\widehat{C}_t^\alpha$  defined in (24). Under Assumption B.4 and B.5, for any subset size  $|T_i|$ ,  $\alpha \in (0,1)$ , and  $\beta \in [0,\alpha]$ , we have:
$$
\exists i \in I: \left| \Pr \left\{Y _ {t + 1} \in \widehat {C} _ {t} ^ {\alpha} \left(X _ {t + 1}\right) \right\} - (1 - \alpha) \right| \leq 1 2 \sqrt {\log \left(1 6 \left| T _ {i} \right|\right) / \left| T _ {i} \right|} + 4 \left(L _ {i} + 1\right) \left(\delta_ {i, \left| T _ {i} \right|} ^ {2 / 3} + \delta_ {i, \left| T _ {i} \right|}\right).
$$
The proofs of the above theorems follow from Xu & Xie (2022a). To apply our adoptions to their proofs, one has to select the error regime matching to the test time step  $t + 1$ , i.e., select the right subset  $T_{i}$  from the partitioning of time steps.  $T_{i}$  then corresponds to the overall calibration data in the original assumptions, lemmas, theorems, and the respective proofs. This is valid since Assumption B.5-(i) encompasses all subsets and the existing subset in Assumption B.4 and Assumption B.5-(ii) is, as noted, the same.
Xu & Xie (2022a) provide additional bounds for two variations of Assumption B.4, imposing only the requirement of strongly mixing noise or noise following a linear process. These bounds apply also to our adaption but are omitted here for brevity.
Notes on Exchangeability, Stationarity, Seasonality, and Drifts Comparing HopCPT to standard CP we see that HopCPT assumes exchangeability over the weighted time steps within regimes. Hence, we require that the relationship between inputs and outputs remains stationary within regimes, which is weaker than requiring that the time series as such is stationary. For example, it allows that time series values decrease or grow over time as long as these changes are reflected in the inputs of the MHN and correspond to a regime. HopCPT softens the exchangeability requirements of standard CP in two ways:
1. The CP mechanism assumes that exchangeability is given within a regime that is selected by the MHN association.
2. Even if (1) is not the case, the memory of the MHN (and therefore the calibration data) is updated with each new observation. This memory update allows HopCPT to account for shifts — empirically, the benefit is visible in our experiments on the sap flow dataset, as we discuss briefly in Section 3.2 (lines 283-289) and in more detail in Appendix A.3 (Error Trend & Figure 4).
Regarding seasonality: HopCPT uses covariate features to describe the current time step. Hence, HopCPT implicitly considers seasonality by comparing a given time step with the time steps from the memory. Whenever a time step of a certain season is encountered, it focuses on time steps from the same season. In fact, our experiments already comprise both regular and irregular seasonality patterns. For example, the streamflow dataset contains yearly reoccurring patterns (spring, summer, fall, winter) as well as irregularly reoccurring similar flood events. In both cases, HopCPT can identify related time steps based on the covariates.
Regarding data drifts: an advantage of HopCPT is that the memory is updated with every new observation. Therefore, HopCPT can somewhat incorporate drifts in the target value as long as the covariates remain so that the learned representation is able to focus on the relevant observations. Empirically, the robustness against drifts is visible in our experiments on the sap flow dataset, as we discuss briefly in Section 3.2 (lines 283-289) and in more detail in Appendix A.3 (Error Trend & Figure 4).
Datsets from four different domains were used in the experiments. The following paragraphs summarize the details. Table 9 provides a quantitative overview.
Solar. We conducted experiments on three solar radiation datasets: Solar (3M), Solar (1Y), and Solar (3Y). All three datasets are based on data from the US National Solar Radiation Database (NSDB; Sengupta et al., 2018). Besides solar radiation as the target variable, the datasets include 8 other environmental features. Solar (3M) is the same dataset as used by Xu & Xie (2022a,b) and focuses on data from 8 cities in California. To show the scalability of HopCPT we additionally generated two larger datasets Solar (1Y) and Solar (3Y), which include data from 50 cities from different parts of the US. We selected the cities by ordering the areas provided by NSDB according to the population density and picked the top 50 under the condition that no city appears twice in the dataset.
Air quality. The datasets Air Quality (10PM) and Air Quality (25PM) are air quality measurements from 12 monitoring sites in Beijing, China (Zhang et al., 2017). The datasets provide two target variables (10PM and 2.5PM measurements) which we use separately in our experiments (the variables are used mutually exclusively in the datasets, e.g., when predicting 10PM we do not use 2.5PM as a feature). We encode the wind direction, given as a categorical variable, by mapping the north-south and the east-west components each to a scalar from  $-1$  to  $1$ .
Sap flow. The Sap flow dataset is a subset of the Sapflux data project (Poyatos et al., 2021). This dataset includes sap flow measurements which we use as a target variable, as well as a set of environmental variables. The available features, the length of the measurements, and the sampling vary between the individual time series. To get a set of comparable time series we processed the data as follows: (a) We removed all time series where not all of the 10 environmental features are available. (b) If there was a single missing value between two existing ones, we filled the value with the value before (forward fill). (c) We cut out all sequences without any missing target or feature values (after step b). (d) From the resulting set of sequences we removed all that have less than 15,000 or more than 20,000 time steps.
Streamflow. For our experiments on the streamflow data, we use the Catchment Attributes and Meteorology for Large-sample Studies dataset (CAMELS; Newman et al., 2015; Addor et al., 2017). It provides meteorological time series from different data products, corresponding streamflow measurements, and static catchment attributes for catchments across the continental United States. We ran our experiments on the subset of 531 catchments that were used in previous hydrological benchmarking efforts (e.g., Newman et al., 2017; Kratzert et al., 2019, 2021; Klotz et al., 2022). Specifically, we trained the LSTM prediction model with the NeuralHydrology Python library (Kratzert et al., 2022) on precipitation, solar radiation, minimum and maximum temperature, and vapor pressure from the NLDAS (Xia et al., 2012), Maurer (Maurer et al., 2002), and Daymet (Thornton et al., 1997) meteorological data products. Further, the LSTM received 26 static catchment attributes at each time step that identify the catchment properties. Table 4 in Kratzert et al. (2019) provides a full list of these attributes. We trained the prediction model on data from the period Oct 1981 - Sep 1990 (note that for some catchments, this period starts later due to missing data in the beginning), calibrated the uncertainty models on Oct 1990 - Sep 1999, and tested them on the period Oct 1999 - Sep 2008.
D kNN vs. Learned Representation
As mentioned in the main paper, we designed HopCPT with large datasets in mind. It is only in such a setting that the learned part of our approach can truly play to its strengths and take advantage of nuanced interrelationships in the data. kNN provides us with a natural "fallback" option for settings where not enough data is available to infer these relationships. Our comparisons in Table 10 and Table 11 substantiate this argument: kNN provides competitive results for the small dataset Solar (3M) (this can also be seen by contrasting the performance from Table 10 to the respective results in Appendix A.3), but is outperformed by HopCPT for the larger datasets.
Table 9: Details of the evaluated datasets.
Number of Series	Time Steps per Series	Period	Sampling	Number of Features	Data Split [%]
Solar (3M)	8	2,000	01–03 2018	60m	8	60/15/25
Solar (1Y)	50	8,760	2019	60m	8	60/15/25
Solar (3Y)	50	26,304	2018–20	60m	8	34/33/33
Air Quality (10PM)	12	35,064	2013–17	60m	11	34/33/33
Air Quality (25PM)	12	35,064	2013–17	60m	11	34/33/33
Sap flow	24	15,000–20,000	2008–16	varying	10	34/33/33
Streamflow	531	9,862	1981–2008	24h	41	34/33/33
E Quantile of Sample vs. Quantile of Weighted ECDF
HopCPT constructs prediction intervals by calculating a quantile on the set of weight-sampled errors (see Equation 7). An alternative approach is to calculate the quantile over a weighted empirical CDF of the errors. This approach would define  $q(\tau, Z_{t+1})$  in Equation 7 as
$$
q (\tau , \mathbf {Z} _ {t + 1}) = \mathrm {Q} _ {\tau} \left(\sum_ {i = 1} ^ {t} a _ {t + 1, i} \delta_ {\epsilon_ {i}}\right), \tag {26}
$$
where  $\delta_{\epsilon_i}$  is a point mass at  $|\epsilon_i|$ . Empirically, we find little differences in the performance when comparing the two approaches (Tables 12 and 14).
F Asymmetrical Error Quantiles
HopCPT calculates the prediction interval by using equally large quantiles (in terms of distribution mass, not width) for the upper and lower bound (see Equation 7). Xu & Xie (2022a) instead search for a  $\hat{\beta} \in [0, \alpha]$  and calculate the lower quantile with  $\hat{\beta}$  and the upper quantile with  $1 - \alpha + \hat{\beta}$ , which minimizes the overall interval with. Applying this principle to Equation 7 one would arrive at
$$
\hat {\beta} = \underset {\beta \in [ 0, \alpha ]} {\operatorname {a r g m i n}} \left[ \hat {\mu} \left(\boldsymbol {X} _ {t + 1}\right) + q (\beta , \boldsymbol {Z} _ {t + 1}), \hat {\mu} \left(\boldsymbol {X} _ {t + 1}\right) + q (1 - \alpha + \beta , \boldsymbol {Z} _ {t + 1}) \right] \tag {27}
$$
$$
\widehat {C} _ {t} ^ {\alpha} (\boldsymbol {Z} _ {t + 1}) = \left[ \hat {\mu} (\boldsymbol {X} _ {t + 1}) + q (\hat {\beta}, \boldsymbol {Z} _ {t + 1}), \hat {\mu} (\boldsymbol {X} _ {t + 1}) + q (1 - \alpha + \hat {\beta}, \boldsymbol {Z} _ {t + 1}) \right],
$$
Practically, the search is done by evaluating different  $\beta$  values within the potential value range and selecting the one which produces the lowest interval width. Theoretically, this could lead to more efficient intervals at the cost of additional computation and potential asymmetry in regard to uncovered samples that are higher and lower than the prediction interval. Empirically, we do not find a notable improvement when applying this search to HopCPT (see Table 13).
G AdaptiveCI and HopCPT
As noted in Section 3, AdaptiveCI is orthogonal to HopCPT, SPCI, EnbPI, and NexCP. We therefore also evaluated the combined application of the models. Nevertheless, AdaptiveCI is an independent model on top of an existing model, which is reflected in the way we select the hyperparameters of the combined model: First, the hyperparameters are selected for each model without adaption through AdaptiveCI (see Appendix A.1) — hence, we use the same hyperparameters as in the main evaluation. Second, we conduct another hyperparameter search, given the model parameters from the first search, where we only search for the parameter of the adaptive component.
Tables 15 and 16 show the results of these experiments. Overall, the results are slightly better, as the Winkler score (which considers both width and coverage) slightly improves in most experiments. The ranking between the different models stays similar to the non-adaptive comparison (see Section 3.2) with HopCPT performing best on all but the smallest dataset.
H Details on Continuous Modern Hopfield Networks
Modern Hopfield Networks are associative memory networks. They learn to associate entries of a memory with a given "query" sample in order to produce a prediction. In HopCPT, the memory corresponds to representations of past time steps and the query corresponds to a representation of the current time step. The association happens by means of weights on every memory entry, which are called "association weights". Query-memory sample pairs that have similar representations are assigned high association weights. In summary, the association weights indicate which samples are closely related — in HopCPT, this corresponds to finding the time steps which are part of the same time series regimes and therefore useful to estimate the current uncertainty. The association mechanism is in that sense closely related to the attention mechanism of Transformers (Vaswani et al., 2017). However, compared to other trainable networks, the memory-based architecture has the advantage that it does not need to encode all knowledge in network parameters but can dynamically adapt when given new memory entries.
We adapt the following arguments from Furst et al. (2022) and Ramsauer et al. (2021) for a more formal overview. Associative memory networks have been designed to store and retrieve samples. Hopfield networks are energy-based, binary associative memories, which were popularized as artificial neural network architectures in the 1980s (Hopfield, 1982, 1984). Their storage capacity can be considerably increased by polynomial terms in the energy function (Chen et al., 1986; Psaltis & Cheol, 1986; Baldi & Venkatesh, 1987; Gardner, 1987; Abbott & Arian, 1987; Horn & Usher, 1988; Caputo & Niemann, 2002; Krotov & Hopfield, 2016). In contrast to these binary memory networks, we use continuous associative memory networks with far higher storage capacity. These networks are continuous and differentiable, retrieve with a single update, and have exponential storage capacity (and are therefore scalable, i.e., able tackle large problems; Ramsauer et al., 2021).
Formally, we denote a set of patterns  $\{\pmb{x}_1,\dots ,\pmb{x}_N\} \subset \mathbb{R}^d$  that are stacked as columns to the matrix  $\pmb {X} = (x_{1},\ldots ,x_{N})$  and a state pattern (query)  $\pmb {\xi}\in \mathbb{R}^{d}$  that represents the current state. The largest norm of a stored pattern is  $M = \max_{i}\| \pmb{x}_{i}\|$ . Then, the energy E of continuous Modern Hopfield Networks with state  $\pmb{\xi}$  is defined as (Ramsauer et al., 2021)
$$
\mathrm {E} = - \beta^ {- 1} \log \left(\sum_ {i = 1} ^ {N} \exp \left(\beta \boldsymbol {x} _ {i} ^ {T} \boldsymbol {\xi}\right)\right) + \frac {1}{2} \boldsymbol {\xi} ^ {T} \boldsymbol {\xi} + \mathrm {C}, \tag {28}
$$
where  $\mathrm{C} = \beta^{-1}\log N + \frac{1}{2} M^2$ . For energy E and state  $\xi$ , Ramsauer et al. (2021) proved that the update rule
$$
\boldsymbol {\xi} ^ {\text {n e w}} = \boldsymbol {X} \operatorname {s o f t m a x} \left(\beta \boldsymbol {X} ^ {T} \boldsymbol {\xi}\right) \tag {29}
$$
converges globally to stationary points of the energy  $\mathrm{E}$  and coincides with the attention mechanisms of Transformers (Vaswani et al., 2017; Ramsauer et al., 2021).