Energy-based Hopfield Boosting for Out-of-Distribution Detection

Claus Hofmann 1 2 Simon Schmid 3 Bernhard Lehner 2 4 Daniel Klotz 5 Sepp Hochreiter 1 2
summary, our contributions are as follows:

Abstract
Out-of-distribution (OOD) detection is criticalwhen deploying machine learning models in thereal world. Outlier exposure (OE) methods, which
incorporate auxiliary outlier data (AUX) in the
training process, can drastically improve OOD
detection performance. We introduce Hopfield
Boosting, a boosting approach, which leverages
modern Hopfield energy to sharpen the decisionboundary between the in-distribution (ID) andOOD data. Hopfield Boosting encourages themodel to focus on hard-to-distinguish auxiliary
outlier examples that lie close to the decisionboundary between ID and AUX data. Our method
achieves a new state-of-the-art in OOD detection
with OE, improving the FPR95 from 2.28 to 0.92
on CIFAR-10, from 11.24 to 7.94 on CIFAR-100,
and from 50.74 to 36.60 on ImageNet-1K.

1. We propose Hopfield Boosting, an OOD detection approach that samples weak learners by using the modern
Hopfield energy (MHE; Ramsauer et al., 2021).
2. Hopfield Boosting achieves a new state-of-the-art in
OOD detection. It improves the average false positive
rate at 95% true positives (FPR95) from 2.28 to 0.92
on CIFAR-10, from 11.24 to 7.94 on CIFAR-100, andfrom 50.74 to 36.60 on ImageNet-1K.

2. Related Work
Post-hoc OOD detection. A common OOD detection
approach is to use a post-hoc strategy, where one employs
statistics obtained from a classifier. The perhaps most wellknown approach in this class is the Maximum Softmax
Probability (MSP; Hendrycks & Gimpel, 2017), where one
utilizes maxy p( y | x ) to estimate whether a sample is
OOD. Despite good empirical performance, this view isintrinsically limited, since OOD detection should focus onp(x). A wide range of post-hoc OOD detection approaches
have been proposed to address the shortcomings of MSP(e.g., Liu et al., 2020; Sun et al., 2021; Djurisic et al., 2023).
Most related to Hopfield Boosting is the work of Zhang et al.
(2023a) – to our knowledge, they are the first to apply MHE
to do OOD detection.

1. Introduction
Out-of-distribution (OOD) detection is crucial when using
machine learning systems in the real world (Ruff et al.,2021). Deployed models will — sooner or later — encounter inputs that deviate from the training distribution.
For example, a system trained to recognize music genres
might also hear a sound clip of construction site noise. In thebest case, a naive deployment can then result in overly confident predictions. In the worst case, we will get erratic model
behavior and completely wrong predictions (Hendrycks &Gimpel, 2017). The purpose of OOD detection is to classify these inputs as OOD, such that the system can then,for instance, notify users that no prediction is possible. In

Auxiliary outlier data and outlier exposure. A second
group of OOD detection approaches are outlier exposure(OE) methods. Like Hopfield Boosting, they incorporate
AUX data in the training process to improve the detection of
OOD samples (e.g., Hendrycks et al., 2019b; Liu et al., 2020;
Ming et al., 2022). We provide more detailed discussions
on a range of OE methods in Appendix C.1. As far as
we know, all OE approaches optimize an objective (LOOD ),which aims at improving the model’s discriminative power
between in-distribution (ID) and OOD data using the AUX
data set as a stand-in for the OOD case. In general, OE
methods conceptualize the AUX data set as a large anddiverse data set (e.g., ImageNet for vision tasks). Recent
approaches therefore actively try to find informative samples
close to the decision boundary between ID and AUX data
for the training. The aim is to refine the decision boundary,*
Equal contribution 1 Johannes Kepler University Linz, Institutefor Machine Learning, JKU LIT SAL eSPML Lab, Austria 2 Silicon
Austria Labs, JKU LIT SAL eSPML Lab, Austria 3 Software Competence Center Hagenberg GmbH, Austria 4 Johannes Kepler University Linz, JKU LIT SAL eSPML Lab, Austria 5 Department ofComputational Hydrosystems, Helmholtz Centre for Environmental Research–UFZ, Leipzig, Germany. Correspondence to: Claus
Hofmann <hofmann@ml.jku.at>.

Published at the 2 nd Differentiable Almost Everything Workshop
at the 41 st International Conference on Machine Learning, Vienna, Austria. July 2024. Copyright 2024 by the author(s).

1Energy-based Hopfield Boosting for Out-of-Distribution Detection

Figure 1: The Hopfield Boosting concept. The first step (weight) creates weak learners by firstly choosing in-distribution
samples (ID, orange), and by secondly choosing auxiliary outlier samples (AUX, blue) according to their assigned
probabilities; the second step (evaluate) computes the losses for the resulting predictions (Section 3); and the third step
(update) assigns new probabilities to the AUX samples according to their position on the hypersphere (see Figure 2).
ensuring the ID data is more tightly encapsulated (e.g., Ming
et al., 2022). Hopfield Boosting also makes use of samples
close to the boundary by giving them higher weights for theboosting step.

Ruff et al. (2021), an observation is OOD if it is in the set
O = {ξ D ∈ RD | pID (ξ D ) < ϵ} where ϵ > 0.

(1)

Since the probability density pID is in general not known,
one needs to estimate pID (ξ D ). In practice, it is common
to define an outlier score s(ξ) that uses a threshold γ andan encoder ϕ, where ξ = ϕ(ξ D ). The outlier score should— in the best case — preserve the density ranking. Givens(ξ) and ϕ, OOD detection can be formulated as a binaryclassification task with the classes ID and OOD:(ID
if s(ϕ(ξ D )) ≥ γDB̂(ξ , γ) =.
(2)
OOD if s(ϕ(ξ D )) < γ

Continuous modern Hopfield networks. MHNs areenergy-based associative memory networks. They advanceconventional Hopfield networks (Hopfield, 1984) by introducing continuous queries and states with the MHE as a newenergy function. MHE leads to exponential storage capacity,while retrieval is possible with a one-step update (Ramsauer
et al., 2021). Section 3.2 gives an introduction to MHE for
OOD detection. For further details on MHNs, we refer toAppendix A.

3.2. Modern Hopfield Energy
The log-sum-exponential (lse) function is defined as!
NX−1
exp(βzi ) ,lse(β, z) = β
log

3. Method
This section presents Hopfield Boosting: First, we formalize
the OOD detection task. Second, we give an overview ofthe MHE. Finally, we introduce the boosting framework.
Figure 1 shows a summary of Hopfield Boosting.

(3)

i=1

where β is the inverse temperature and z ∈ RN is a vector.
The lse can be seen as a soft approximation to the maximum
function: As β → ∞, the lse approaches maxi zi .

3.1. Classification and OOD Detection

Given a set of N d-dimensional stored patterns(x1 , x2 , . . . , xN ) arranged in a data matrix X, and a ddimensional query ξ, the MHE is defined as

Consider a multi-class classification task denoted as(X D , Y D , Y), where X D ∈ RD×N represents a set of N
DD
D-dimensional feature vectors (xD
1 , x2 , . . . , xN ), whichDDN
are i.i.d. samples xi ∼ pID . Y ∈ Y denotes the labels
associated with these feature vectors, and Y is a set containing possible classes . We consider an observation ξ D ∈ RD
that deviates considerably from the data generation pID (ξ D )that defines the “normality” of our data as OOD. FollowingE(ξ; X) = − lse(β, X T ξ) +1 T
ξ ξ + C,2

(4)

where C = β −1 log N + 12 M 2 and M is the largest normof a pattern: M = maxi ||xi ||. X is also called the memoryof the MHN.
2

Energy-based Hopfield Boosting for Out-of-Distribution Detection

ID and AUX data and decreases with increasing distancefrom the decision boundary in both directions. We provide
a theoretical background on Eb in Appendix F.

To use the MHE for OOD detection, Hopfield Boosting acquires the memory patterns X by feeding raw data instancesDD(xD
1 , x2 , . . . , xN ) of the ID data set arranged in the dataDmatrix X ∈ RD×N to an encoder ϕ : RD → Rd (e.g.,
ResNet): xi = ϕ(xDi ). We denote the component-wise
application of ϕ to the patterns in X D as X = ϕ(X D ).
Similarly, a raw query ξ D ∈ RD is fed through the encoder
to obtain the query pattern: ξ = ϕ(ξ D ). One can now useE(ξ; X) to estimate whether ξ is ID or OOD: A low energy
indicates ξ is ID, and a high energy signifies that ξ is OOD.
The MHE can be viewed as a soft, continuously differentiable approximation to the nearest neighbor similarity: As
β → ∞, the lse term in the MHE returns the dot-productsimilarity of ξ to its nearest neighbor in X.

To calculate the weights wt+1 , we use a collection of AUX
patterns as a query matrix Ξ and compute the respectiveenergies Eb of those patterns. The resulting energy vector Eb (Ξ; X, O) is then normalized by a softmax. This
computation provides the updated weights:
wt+1 = softmax(βEb (Ξ; X, O)).

Training the model using MHE. In this section, we introduce how Hopfield Boosting uses the sampled weak learners
to improve the detection of patterns outside the training distribution. We follow the established training method for OE(e.g., Hendrycks et al., 2019b): Train a classifier on the IDdata using cross-entropy and add an OOD loss that uses the
AUX data set to sharpen the decision boundary between the
ID and OOD regions. This yields a composite loss

3.3. Boosting Framework
Sampling of informative outlier data. Similar to Ming
et al. (2022), Hopfield Boosting selects informative outliers
close to the ID-OOD decision boundary. For this selection,
Hopfield Boosting constructs a distribution over the AUX
data similar to AdaBoost (Freund & Schapire, 1995) by sampling data instances close to the decision boundary more
frequently. We refer to samples close to the decision boundary as weak learners — their nearest neighbors consist of
samples from their own class as well as from the foreignclass. The Hopfield Boosting mechanism thus results inthe special setup where a (projected) data point becomes aweak learner itself. That is, it is possible to use this data
point to construct a classifier that barely performs better
than random guessing (see Appendix E.5). Vice versa, astrong hypothesis can be created by forming an ensembleof a set of weak learners (Figure 2).

L = LCE + λLOOD ,Hopfield Boosting computes LOOD as follows:
LOOD =1 X
Eb (ξ; Xs , Os ),
2Ns

(8)

ξ

where the memories Xs and Os contain the embeddingsof the sampled data instances: Xs = ϕ(XsD ) and Os =ϕ(OsD ). The sum is taken over the observations ξ, which
are drawn from (Xs ∥ Os ). Hopfield Boosting computes
LOOD for each mini-batch individually. To the best of ourknowledge, Hopfield Boosting is the first method that uses
Hopfield networks in this way to train a deep neural network.
We note that there is a relation between Hopfield Boosting
and SVMs with an RBF kernel (see Appendix G.3). However, the optimization procedure of SVMs is in general notdifferentiable. In contrast, our novel energy function is
fully differentiable. This allows us to use it to train neuralnetworks. Rätsch et al. (2001) shows the relation between
highly weighted samples in AdaBoost and support vectors.

We introduce an MHE-based energy function which Hopfield Boosting uses to determine how weak a specific learnerξ is (with higher energy indicating a weaker learner).
+ lse(β, X T ξ) + lse(β, O T ξ),(7)

Hopfield Boosting explicitly minimizes Eb . Given theweight vector wt , and the data sets X D and O D , we obtaina mini-batch XsD containing Ns samples from X D by uniform sampling, and a mini-batch of Ns weak learners OsD
from O D by sampling according to wt with replacement.
We then feed the respective mini-batches into the neuralnetwork ϕ (e.g., ResNet) to create an embedding.

We denote the matrix with the raw AUX data instances
D
D×M
D
(oD
, and the memory containing
1 , . . . , oN ) as O ∈ R
the encoded AUX patterns as O = ϕ(O D ). The boostingproceeds as follows: There exists a weight (w1 , . . . , wM )for each data point in O D . The individual weights areaggregated into the weight vector wt . Hopfield Boostinguses these weights to draw mini-batches OsD from O D fortraining, where weak learners receive higher weights.

Eb (ξ; X, O) = − 2 lse(β, (X ∥ O)T ξ)

(6)

(5)

where (X ∥O) ∈ Rd×(N +M ) denotes the concatenated data
matrix containing the patterns from both X and O. Before
computing Eb , we normalize the feature vectors in X, O,
and ξ to unit length. Figure 3 displays the energy landscapeof Eb (ξ; X, O) using exemplary data on a 3-dimensional
sphere. Eb is maximal at the decision boundary between

Inference. At inference time, the OOD score s(ξ) iss(ξ) = lse(β, X T ξ) − lse(β, O T ξ).

(9)

For computing s(ξ), Hopfield Boosting uses the 50,000 random samples from the ID and AUX data sets, respectively.
3

Energy-based Hopfield Boosting for Out-of-Distribution Detection

Table 1: OOD detection performance on CIFAR-10. We
compare results from Hopfield Boosting, POEM (Minget al., 2022), EBO-OE (Liu et al., 2020), and MSP-OE(Hendrycks et al., 2019b) on ResNet-18. ↓ indicates “lower
is better” and ↑ “higher is better”. All values in %. Standard
deviations are estimated across five training runs.

Figure 2: Synthetic example of the adaptive resamplingmechanism. Hopfield Boosting forms a strong hypothesisby sampling and combining a set of weak hypotheses close
to the decision boundary. The heatmap on the backgroundshows exp(βEb (ξ, X, O)), where β is 60. Only the sampled (i.e., highlighted) points serve as memories X and O.

OOD Data
SVHN
LSUN-C
LSUN-R
TexturesiSUN
Places 365

4. Experiments

Mean

This section first presents a toy example to give the reader
an intuition of Hopfield Boosting, and then describes our
experiments and results on CIFAR-10. For the selection of
the hyperparameters and additional experiments, including
ablations, additional baselines, and results on CIFAR-100
and ImageNet-1K, we refer to Appendix H.

MetricHB (ours)

POEM

EBO-OEMSP-OE

FPR95 ↓
AUROC ↑FPR95 ↓
AUROC ↑FPR95 ↓
AUROC ↑FPR95 ↓
AUROC ↑FPR95 ↓
AUROC ↑FPR95 ↓
AUROC ↑

0.23±0.08
99.57±0.06
0.82±0.17
99.40±0.04
0.00±0.00
99.98±0.02
0.16±0.02
99.84±0.01
0.00±0.00
99.97±0.02
4.28±0.23
98.51±0.10

1.48±0.68
99.33±0.15
4.02±0.91
98.89±0.15
0.00±0.00
99.88±0.12
0.49±0.04
99.72±0.05
0.00±0.00
99.87±0.12
7.70±0.68
97.56±0.26

2.66±0.91
99.15±0.23
6.82±0.74
98.43±0.10
0.00±0.00
99.98±0.02
1.11±0.17
99.61±0.02
0.00±0.00
99.98±0.01
11.77±0.68
96.39±0.30

4.31±1.10
99.20±0.15
7.02±1.14
98.83±0.15
0.00±0.00
99.96±0.00
2.29±0.16
99.57±0.01
0.00±0.00
99.96±0.00
21.42±0.88
95.91±0.17

0.92
99.55

2.28
99.21

3.73
98.92

5.84
98.90

FPR95 ↓
AUROC ↑

Training. The network trains for 100 epochs. Each epoch,the model sees the entire ID data set and a selection of AUX
samples. We evaluate the composite loss from Equation (7)for each mini-batch and update the model accordingly. Afteran epoch, we update the sample weights, yielding wt+1 .
During the sample weight update, Hopfield Boosting does
not compute gradients. The update of the sample weights
wt+1 proceeds as follows: First, we fill the memories X
and O with 50,000 samples, respectively. Second, we use
the obtained X and O to get the energy Eb (Ξ; X, O) for
500,000 AUX samples and compute wt+1 according toEquation (6). In the following epoch, we sample the minibatches OsD according to wt+1 with replacement.

4.1. Toy Example
Figure 2 demonstrates how the weighting in Hopfield Boosting allows good estimations of the decision boundary, even
if Hopfield Boosting only samples a small number of weaklearners. This is desirable because the AUX data contains
a large number of data instances that are uninformative for
the OOD detection task. For small, low dimensional data,
one can always use all the data to compute Eb (Figure 2, a).
For large problems, this strategy is difficult, and the naive
solution of uniformly sampling N data points would also notwork. This will yield many uninformative points (Figure 2,b). When using Hopfield Boosting and sampling N weaklearners according to wt , the result better approximates thedecision boundary of the full data (Figure 2, c).

4.3. Results & Discussion
Table 1 summarizes the results for CIFAR-10. Hopfield
Boosting surpasses POEM, improving the mean FPR95
metric from 2.28 to 0.92.
We also explore the influence of boosting (Table 6): Theexperiment shows that sampling weak learners contributes
considerably to the performance of Hopfield Boosting. Although Hopfield Boosting shows superior performance compared to POEM even without boosting, outlier sampling canbeat this version on every dataset on the FPR95 metric.

4.2. Data & Setup
CIFAR-10 We train Hopfield Boosting with ResNet-18(He et al., 2016) on the CIFAR-10 data set (Krizhevsky,2009). We use ImageNet-RC (Chrabaszcz et al., 2017) (alow-resolution version of ImageNet) as the AUX data. For
testing the OOD detection performance, we use the data sets
SVHN (Netzer et al., 2011), Textures (Cimpoi et al., 2014),
iSUN (Xu et al., 2015), Places 365 (López-Cifuentes et al.,2020), and two versions of the LSUN data set (Yu et al.,2015) — one where the images are cropped, and one where
they are resized to match the resolution of the CIFAR data
sets (32x32 pixels). We evaluate the discriminative powerof s(ξ) between CIFAR and the respective OOD data set
using the FPR95 and the AUROC.

5. Conclusions
We introduce Hopfield Boosting: a method for OOD detection with OE. Hopfield Boosting uses an energy term toboost a classifier between inlier and outlier data by sampling weak learners that are close to the decision boundary.
We compare Hopfield Boosting to three OOD detection approaches. Overall, Hopfield Boosting shows the best results.
4

Energy-based Hopfield Boosting for Out-of-Distribution Detection

resentations. In International conference on machinelearning, pp. 1597–1607. PMLR, 2020.

Acknowledgements
We thank Christian Huber for helpful feedback and fruitful
discussions.

Chrabaszcz, P., Loshchilov, I., and Hutter, F. A downsampled variant of imagenet as an alternative to the CIFAR
datasets. arXiv preprint arXiv:1707.08819, 2017.

The ELLIS Unit Linz, the LIT AI Lab, the Institute for Machine Learning, are supported by the Federal State Upper
Austria. We thank the projects AI-MOTION (LIT-20186-YOU-212), DeepFlood (LIT-2019-8-YOU-213), Medical Cognitive Computing Center (MC3), INCONTROLRL (FFG-881064), PRIMAL (FFG-873979), S3AI (FFG872172), DL for GranularFlow (FFG-871302), EPILEPSIA (FFG-892171), AIRI FG 9-N (FWF-36284, FWF36235), AI4GreenHeatingGrids(FFG- 899943), INTEGRATE (FFG-892418), ELISE (H2020-ICT-2019-3 ID:
951847), Stars4Waters (HORIZON-CL6-2021-CLIMATE01-01). We thank Audi.JKU Deep Learning Center, TGW
LOGISTICS GROUP GMBH, Silicon Austria Labs (SAL),
University SAL Labs initiative, FILL Gesellschaft mbH,
Anyline GmbH, Google, ZF Friedrichshafen AG, Robert
Bosch GmbH, UCB Biopharma SRL, Merck Healthcare
KGaA, Verbund AG, GLS (Univ. Waterloo) Software Competence Center Hagenberg GmbH, TÜV Austria, Frauscher
Sensonic, Borealis AG, TRUMPF and the NVIDIA Corporation. This work has been supported by the "University SAL
Labs" initiative of Silicon Austria Labs (SAL) and its Austrian partner universities for applied fundamental researchfor electronic-based systems. We acknowledge EuroHPC
Joint Undertaking for awarding us access to Karolina at
IT4Innovations, Czech Republic and Leonardo at CINECA,
Italy.

Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., , andVedaldi, A. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern
Recognition (CVPR), 2014.
Cortes, C. and Vapnik, V. Support-vector networks. Machine learning, 20(3):273–297, 1995.
Djurisic, A., Bozanic, N., Ashok, A., and Liu, R. Extremely
simple activation shaping for out-of-distribution detection.
In The Eleventh International Conference on Learning
Representations, 2023.
Freund, Y. and Schapire, R. E. A decision-theoretic generalization of on-line learning and an application to boosting.
In Computational Learning Theory: Eurocolt ’95, pp.
23–37. Springer-Verlag, 1995.
Fürst, A., Rumetshofer, E., Lehner, J., Tran, V. T., Tang,F., Ramsauer, H., Kreil, D., Kopp, M., Klambauer, G.,
Bitto, A., et al. CLOOB: Modern Hopfield networks
with InfoLOOB outperform clip. Advances in neural
information processing systems, 35:20450–20468, 2022.
Gardner, E. Multiconnected neural network models. Journalof Physics A: Mathematical and General, 20(11):3453,
1987.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE
conference on computer vision and pattern recognition,
pp. 770–778, 2016.

References
Abbott, L. F. and Arian, Y. Storage capacity of generalized
networks. Physical review A, 36(10):5091, 1987.

He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Momentum contrast for unsupervised visual representation
learning. In Proceedings of the IEEE/CVF conference on
computer vision and pattern recognition, pp. 9729–9738,
2020.

Ahn, S., Korattikara, A., and Welling, M. Bayesian posterior
sampling via stochastic gradient Fisher scoring. In Proceedings of the 29th International Coference on International Conference on Machine Learning, pp. 1771–1778,
Madison, WI, USA, 2012. Omnipress.

Hendrycks, D. and Gimpel, K. A baseline for detecting
misclassified and out-of-distribution examples in neural networks. In International Conference on Learning
Representations, 2017. URL https://openreview.
net/forum?id=Hkg4TI9xl.

Baldi, P. and Venkatesh, S. S. Number of stable points
for spin-glasses and neural networks of higher orders.
Physical Review Letters, 58(9):913, 1987.
Caputo, B. and Niemann, H. Storage capacity of kernel associative memories. In Artificial Neural Networks—ICANN
2002: International Conference Madrid, Spain, August
28–30, 2002 Proceedings 12, pp. 51–56. Springer, 2002.

Hendrycks, D., Basart, S., Mazeika, M., Zou, A., Kwon, J.,
Mostajabi, M., Steinhardt, J., and Song, D. Scaling outof-distribution detection for real-world settings. arXiv
preprint arXiv:1911.11132, 2019a.

Chen, H., Lee, Y., Sun, G., Lee, H., Maxwell, T., and Giles,
C. L. High order correlation model for associative memory. In AIP Conference Proceedings, volume 151, pp.
86–99. American Institute of Physics, 1986.

Hendrycks, D., Mazeika, M., and Dietterich, T. Deepanomaly detection with outlier exposure.
In International Conference on Learning Representations,2019b. URL https://openreview.net/forum?
id=HyxCxhRcY7.

Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A
simple framework for contrastive learning of visual rep5

Energy-based Hopfield Boosting for Out-of-Distribution Detection

Hopfield, J. J. Neural networks and physical systems withemergent collective computational abilities. Proceedingsof the National Academy of Sciences, 79(8):2554–2558,
1982.

for out-of-distribution detection. In The Twelfth International Conference on Learning Representations,2024. URL https://openreview.net/forum?
id=uNkKaD3MCs.

Hopfield, J. J. Neurons with graded response have collective
computational properties like those of two-state neurons.
Proceedings of the National Academy of Sciences, 81(10):3088–3092, 1984. doi: 10.1073/pnas.81.10.3088.

McInnes, L., Healy, J., and Melville, J. Umap: Uniform
manifold approximation and projection for dimension
reduction. arXiv preprint arXiv:1802.03426, 2018.
Ming, Y., Fan, Y., and Li, Y. POEM: Out-of-distribution
detection with posterior sampling. In Chaudhuri, K.,
Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato,
S. (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedingsof Machine Learning Research, pp. 15650–15665. PMLR,
17–23 Jul 2022. URL https://proceedings.mlr.
press/v162/ming22a.html.

Horn, D. and Usher, M. Capacities of multiconnected memory models. Journal de Physique, 49(3):389–395, 1988.
Isola, P., Xiao, J., Torralba, A., and Oliva, A. What makes animage memorable? In CVPR 2011, pp. 145–152. IEEE,
2011.
Jiang, W., Cheng, H., Chen, M., Wang, C., and Wei, H. DOS:
Diverse outlier sampling for out-of-distribution detection.
In The Twelfth International Conference on Learning
Representations, 2024. URL https://openreview.
net/forum?id=iriEqxFB4y.

Moody, J. and Darken, C. J. Fast learning in networks oflocally-tuned processing units. Neural computation, 1(2):281–294, 1989.
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B.,
and Ng, A. Y. Reading digits in natural images with
unsupervised feature learning. 2011.

Krizhevsky, A. Learning multiple layers of features fromtiny images. Master’s thesis, Deptartment of Computer
Science, University of Toronto, 2009.

Oord, A. v. d., Li, Y., and Vinyals, O. Representation learning with contrastive predictive coding. arXiv preprint
arXiv:1807.03748, 2018.

Krotov, D. and Hopfield, J. J. Dense associative memory
for pattern recognition. In Lee, D. D., Sugiyama, M.,
Luxburg, U. V., Guyon, I., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, pp.
1172–1180. Curran Associates, Inc., 2016.

Park, G. Y., Kim, J., Kim, B., Lee, S. W., and Ye, J. C.
Energy-based cross attention for Bayesian context update in text-to-image diffusion models. In Thirty-seventh
Conference on Neural Information Processing Systems,
2023.

LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradientbased learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.

Psaltis, D. and Park, C. H. Nonlinear discriminant functions
Liu, W., Wang, X., Owens, J., and Li, Y. Energy-based
and associative memories. In AIP conference Proceedout-of-distribution detection. In Larochelle, H., Ranings, volume 151, pp. 370–375. American Institute ofzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.),
Physics, 1986.
Advances in Neural Information Processing Systems,
Ramsauer, H., Schäfl, B., Lehner, J., Seidl, P., Widrich,volume 33, pp. 21464–21475. Curran Associates,M., Gruber, L., Holzleitner, M., Pavlović, M., Sandve,
Inc., 2020.
URL https://proceedings.
G. K., Greiff, V., Kreil, D., Kopp, M., Klambauer, G.,neurips.cc/paper/2020/file/
Brandstetter, J., and Hochreiter, S. Hopfield networks
f5496252609c43eb8a3d147ab9b9c006-Paper.
is all you need. In 9th International Conference on
pdf.
Learning Representations (ICLR), 2021. URL https:
Liu, X., Lochman, Y., and Zach, C. Gen: Pushing the//openreview.net/forum?id=tL89RnzIiCd.
limits of softmax-based out-of-distribution detection. In
Rätsch, G., Onoda, T., and Müller, K.-R. Soft margins for
Proceedings of the IEEE/CVF Conference on Computer
AdaBoost. Machine learning, 42:287–320, 2001.
Vision and Pattern Recognition, pp. 23946–23955, 2023.
Ridnik, T., Ben-Baruch, E., Noy, A., and Zelnik-Manor, L.
López-Cifuentes, A., Escudero-Vinolo, M., Bescós, J., andImagenet-21k pretraining for the masses. In Thirty-fifth
García-Martín, Á. Semantic-aware scene recognition.
Conference on Neural Information Processing Systems
Pattern Recognition, 102:107256, 2020.
Datasets and Benchmarks Track (Round 1), 2021.
Loshchilov, I. and Hutter, F. Sgdr: Stochastic graRuff, L., Kauffmann, J. R., Vandermeulen, R. A., Montavon,
dient descent with warm restarts. arXiv preprintG., Samek, W., Kloft, M., Dietterich, T. G., and Müller,
arXiv:1608.03983, 2016.
K.-R. A unifying review of deep and shallow anomaly
Lu, H., Gong, D., Wang, S., Xue, J., Yao, L., andMoore, K.
Learning with mixture of prototypes

detection. Proceedings of the IEEE, 109(5):756–795,
2021.
6

Energy-based Hopfield Boosting for Out-of-Distribution Detection

Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh,S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. Imagenet large scale visual recognition
challenge. International journal of computer vision, 115:
211–252, 2015.

Wei, X.-S., Cui, Q., Yang, L., Wang, P., Liu, L., and Yang, J.
Rpc: a large-scale and fine-grained retail product checkout dataset, 2022. URL https://rpc-dataset.
github.io/.
Welling, M. and Teh, Y. W. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the
28th International Conference on Machine Learning, pp.
681–688, Madison, WI, USA, 2011. Omnipress.

Saleh, R. A. and Saleh, A. Statistical properties of the
log-cosh loss function used in machine learning. arXiv
preprint arXiv:2208.04564, 2022.

Xu, P., Ehinger, K. A., Zhang, Y., Finkelstein, A., Kulkarni, S. R., and Xiao, J. Turkergaze: Crowdsourcing
saliency with webcam based eye tracking. arXiv preprint
arXiv:1504.06755, 2015.

Schäfl, B., Gruber, L., Bitto-Nemling, A., and Hochreiter,
S. Hopular: Modern Hopfield networks for tabular data.
arXiv preprint arXiv:2206.00664, 2022.
Sehwag, V., Chiang, M., and Mittal, P. Ssd: A unified
framework for self-supervised outlier detection. arXiv
preprint arXiv:2103.12051, 2021.

Xu, P., Chen, J., Zou, D., and Gu, Q. Global convergenceof Langevin dynamics based algorithms for nonconvex
optimization. In Bengio, S., Wallach, H., Larochelle, H.,
Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.),
Advances in Neural Information Processing Systems, volume 31, pp. 3122–3133. Curran Associates, Inc., 2018.

smeschke. Four Shapes. https://www.kaggle.com/
2018.
datasets/smeschke/four-shapes/,
URL https://www.kaggle.com/datasets/
smeschke/four-shapes/.

Yang, J., Wang, P., Zou, D., Zhou, Z., Ding, K., Peng, W.,
Wang, H., Chen, G., Li, B., Sun, Y., et al. Openood:
Benchmarking generalized out-of-distribution detection.
Advances in Neural Information Processing Systems, 35:
32598–32611, 2022.

Sun, Y., Guo, C., and Li, Y. React: Out-of-distribution
detection with rectified activations. Advances in Neural
Information Processing Systems, 34:144–157, 2021.
Tao, L., Du, X., Zhu, X., and Li, Y. Non-parametric outlier
synthesis. arXiv preprint arXiv:2303.02966, 2023.
Teh, Y. W., Thiery, A. H., and Vollmer, S. J. Consistency andfluctuations for stochastic gradient Langevin dynamics. J.
Mach. Learn. Res., 17(1):193–225, 2016.

Yu, F., Seff, A., Zhang, Y., Song, S., Funkhouser, T., andXiao, J. LSUN: Construction of a large-scale image
dataset using deep learning with humans in the loop.
arXiv preprint arXiv:1506.03365, 2015.

Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C.,
Shepard, A., Adam, H., Perona, P., and Belongie, S. The
inaturalist species classification and detection dataset. In
Proceedings of the IEEE conference on computer visionand pattern recognition, pp. 8769–8778, 2018.

Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., and Yoo,
Y. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of theIEEE/CVF international conference on computer vision,
pp. 6023–6032, 2019.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones,
L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention
is all you need. In Guyon, I., Luxburg, U. V., Bengio, S.,
Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R.
(eds.), Advances in Neural Information Processing Systems 30, pp. 5998–6008. Curran Associates, Inc., 2017.

Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D.
mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Zhang, J., Fu, Q., Chen, X., Du, L., Li, Z., Wang, G., Han, S.,
Zhang, D., et al. Out-of-distribution detection based onin-distribution data patterns memorization with modern
Hopfield energy. In The Eleventh International Conference on Learning Representations, 2023a.

Wang, Q., Fang, Z., Zhang, Y., Liu, F., Li, Y., andHan, B. Learning to augment distributions for out-ofdistribution detection. In NeurIPS, 2023a. URL https://openreview.net/forum?id=OtU6VvXJue.
Wang, Q., Ye, J., Liu, F., Dai, Q., Kalander, M., Liu, T., Hao,J., and Han, B. Out-of-distribution detection with implicit
outlier transformation. arXiv preprint arXiv:2303.05033,
2023b.

Zhang, J., Inkawhich, N., Linderman, R., Chen, Y., and Li,
H. Mixture outlier exposure: Towards out-of-distribution
detection in fine-grained environments. In Proceedingsof the IEEE/CVF Winter Conference on Applicationsof Computer Vision (WACV), pp. 5531–5540, January2023b.

Wang, T. and Isola, P. Understanding contrastive representation learning through alignment and uniformity on thehypersphere. In International Conference on Machine
Learning, pp. 9929–9939. PMLR, 2020.

Zhang, R., Li, C., Zhang, J., Chen, C., and Wilson, A. G.
Cyclical stochastic gradient MCMC for Bayesian deeplearning. In International Conference on Learning Representations, 2020.
7

Energy-based Hopfield Boosting for Out-of-Distribution Detection

Zheng, Y., Zhao, Y., Ren, M., Yan, H., Lu, X., Liu, J., and
Li, J. Cartoon face recognition: A benchmark dataset. In
Proceedings of the 28th ACM International Conference
on Multimedia, pp. 2264–2272, 2020.
Zhu, J., Geng, Y., Yao, J., Liu, T., Niu, G., Sugiyama,M., and Han, B. Diversified outlier exposure for outof-distribution detection via informative extrapolation.
Advances in Neural Information Processing Systems, 36,
2023.

8

Energy-based Hopfield Boosting for Out-of-Distribution Detection

A. Details on Continuous Modern Hopfield Networks
The following arguments are adopted from Fürst et al. (2022) and Ramsauer et al. (2021). Associative memory networks
have been designed to store and retrieve samples. Hopfield networks are energy-based, binary associative memories, whichwere popularized as artificial neural network architectures in the 1980s (Hopfield, 1982, 1984). Their storage capacity canbe considerably increased by polynomial terms in the energy function (Chen et al., 1986; Psaltis & Park, 1986; Baldi &Venkatesh, 1987; Gardner, 1987; Abbott & Arian, 1987; Horn & Usher, 1988; Caputo & Niemann, 2002; Krotov & Hopfield,2016). In contrast to these binary memory networks, we use continuous associative memory networks with far higher storage
capacity. These networks are continuous and differentiable, retrieve with a single update, and have exponential storage
capacity (and are therefore scalable, i.e., able to tackle large problems; Ramsauer et al., 2021).
Formally, we denote a set of patterns {x1 , . . . , xN } ⊂ Rd that are stacked as columns to the matrix X = (x1 , . . . , xN )and a state pattern (query) ξ ∈ Rd that represents the current state. The largest norm of a stored pattern is M = maxi ∥xi ∥.
Then, the energy E of continuous Modern Hopfield Networks with state ξ is defined as (Ramsauer et al., 2021)

E = −β

−1

log

N
X

exp(βxTi ξ)

i=1

!

+

1 T
ξ ξ + C,2

(10)where C = β −1 log N + 12 M 2 . For energy E and state ξ, Ramsauer et al. (2021) proved that the update ruleξ new = X softmax(βX T ξ)

(11)converges globally to stationary points of the energy E and coincides with the attention mechanisms of Transformers(Vaswani et al., 2017; Ramsauer et al., 2021).
The separation ∆i of a pattern xi is its minimal dot product difference to any of the other patterns:∆i = min xTi xi − xTi xj .
j,j̸=i(12)

A pattern is well-separated from the data if ∆i is above a given threshold (specified in Ramsauer et al., 2021). If the patterns
xi are well-separated, the update rule Equation 11 converges to a fixed point close to a stored pattern. If some patterns are
similar to one another and, therefore, not well-separated, the update rule converges to a fixed point close to the mean of thesimilar patterns.
The update rule of a Hopfield network thus identifies sample–sample relations between stored patterns. This enables
similarity-based learning methods like nearest neighbor search (see Schäfl et al., 2022), which Hopfield Boosting leverages
to detect samples outside the distribution of the training data.

B. Notes on Langevin Sampling
Another method that is appropriate for earlier acquired models is to sample the posterior via the Stochastic Gradient Langevin
Dynamics (SGLD) (Welling & Teh, 2011). This method is efficient since it iteratively learns from small mini-batches
Welling & Teh (2011); Ahn et al. (2012). See basic work on Langevin dynamics Welling & Teh (2011); Ahn et al. (2012);
Teh et al. (2016); Xu et al. (2018). A cyclical stepsize schedule for SGLD was very promising for uncertainty quantification
Zhang et al. (2020). Larger steps discover new modes, while smaller steps characterize each mode and perform the posterior
sampling.

C. Related work
C.1. Details on further OE approaches
This section gives details about related works from the area of OE in OOD detection. With OE, we refer to the usage of
AUX for training an OOD detector in general.
MSP-OE. Hendrycks et al. (2019b) were the first to introduce the term OE in the context of OOD detection. Specifically,
they improve an MSP-based OOD detection (Hendrycks & Gimpel, 2017): They train a classifier on the ID data set andmaximize the entropy of the predictive distribution of the classifier for the AUX data. The combined loss they employ is9Energy-based Hopfield Boosting for Out-of-Distribution Detection

L = LCE + λLOOD
LOOD =E [H(U , pθ (o))]

(13)
(14)

oD ∼pAUX

where H denotes the cross-entropy, U denotes the uniform distribution over K classes, and pθ is the model mapping thefeatures to the predictive distribution over the K classes.
EBO-OE. Liu et al. (2020) propose a post-hoc and an OE approach. Their post-hoc approach (EBO) is to use theclassifier’s energy to perform OOD detection:E(ξ D ) = − β −1 lse(β, fθ (ξ D ))DDs(ξ ) = −E(ξ , fθ )

(15)
(16)

where fθ outputs the model’s logits as a vector. Their OE approach (EBO-OE) promotes a low energy on ID samples and ahigh energy on AUX samples:

LOOD =

E
xD ∼pID[(max(0, E(xD ) − mID ))2 ] +EoD ∼pAUX[(max(0, mAUX − E(oD )))2 ](17)

where mID and mAUX are margin hyperparameters.
POEM. Ming et al. (2022) propose to incorporate Thompson sampling into the OE process. More specifically, theysample a linear decision boundary in embedding space between the ID and AUX data using Bayesian linear regression andthen select those samples from the AUX data set that are closest to the sampled decision boundary. In the following epoch,
they sample the AUX data uniformly from the selected data instances without replacement and optimize the model with the
EBO-OE loss (Equation (17)).
MixOE. Zhang et al. (2023b) employ mixup (Zhang et al., 2018) between the ID and AUX samples to augment the OE
task. Formally, this results in the following:x̃ = λxD + (1 − λ)oDỹ = λy + (1 − λ)U

LOOD =

E[H(ỹ, x̃)]

(18)
(19)
(20)

(xD ,y)∼pID
oD ∼pAUX

Alternatively, they also propose to employ CutMix (Yun et al., 2019) instead of mixup (which would change the mixing
operation in Equation (18)).
DAL. Wang et al. (2023a) augment the AUX data by defining a Wasserstein-1 ball around the AUX data and performing
OE using this Wasserstein ball. DAL is motivated by the concept of distribution discrepancy: The distribution of the real
OOD data will in general be different from the distribution of the AUX data. The authors argue that their approach can makeOOD detection more reliable if the distribution discrepancy is large.
DivOE. Zhu et al. (2023) pose the question of how to utilize the given outliers from the AUX data set if the auxiliary
outliers are not informative enough to represent the unseen OOD distribution. They suggest solving this problem by
diversifying the AUX data using extrapolation, which should result in better coverage of the OOD space of the resultant
extrapolated distribution. Formally, they employ a loss using a synthesized distribution with a manipulation ∆:10

Energy-based Hopfield Boosting for Out-of-Distribution Detection

LOOD =EoD ∼pAUX[(1 − γ)H(U , pθ (oD )) + γ max[H(U , pθ (oD + ∆)) − H(U , pθ (oD ))]]∆

(21)

DOE. Wang et al. (2023b) implicitly synthesize auxiliary outlier data using a transformation of the model weights. They
argue that perturbing the model parameters has the same effect as transforming the data.
DOS. Jiang et al. (2024) apply K-means clustering to the features of the AUX data set. They then employ a balanced
sampling from the K obtained clusters by selecting the same number of samples from each cluster for training. Morespecifically, they select those n samples from each cluster which are closest to the decision boundary between the ID and
OOD regions.

D. Societal Impact
This section discusses the potential positive and negative societal impacts of our work. As our work aims improves thestate-of-the-art in OOD detection, we focus on potential societal impact of OOD detection in general.
• Postive Impacts– Improved model reliability: OOD detection aims to detect unfamiliar inputs that have little support in the model’s
training distribution. When these samples are detected, one can, for example, notify the user that no prediction ispossible, or trigger a manual intervention. This can lead to an increase in a model’s reliability.
– Abstain from doing uncertain predictions: When a model with appropriate OOD detection recognizes that aquery sample has limited support in the training distribution, it can abstain from performing a prediction. Thiscan, for example, increase trust in ML models, as they will rather tell the user they are uncertain than report a
confidently wrong prediction.
• Negative Impacts– Wrong sense of safety: Having OOD detection in place could cause users to wrongly assume that all OOD inputs
will be detected. However, like most systems, also OOD detection methods can make errors. It is important toconsider that certain OOD examples could remain undetected.
– Potential for misinterpretation: As with many other ML systems, the outcomes of OOD detection methods areprone to misinterpretation. It is important to acquaint oneself with the respective method before applying it inpractice.

11

Energy-based Hopfield Boosting for Out-of-Distribution Detection

E. Toy Examples
E.1. 3D Visualizations of Eb on a hypersphere(a) Out-of-Distribution energy function(b) Exponentiated Out-of-Distribution energy function

Figure 3: Depiction of the energy function Eb (ξ; X, O) on a hypersphere. (a) shows Eb (ξ, X, O) with exemplary inlier(orange) and outlier (blue) points; and (b) shows exp(βEb (ξ, X, O)). β was set to 128. Both, (a) and (b), rotate the sphereby 0, 90, 180, and 270 degrees around the vertical axis.
This example depicts how inliers and outliers shape the energy surface (Figure 3). We generated patterns so that X clusters
around a pole and the outliers populate the remaining perimeter of the sphere. This is analogous to the idea that one hasaccess to a large AUX data set, where some data points are more and some less informative for OOD detection (e.g., asconceptualized in Ming et al., 2022).
E.2. Dynamics of LOOD on Patterns in Euclidean SpaceIn this example, we applied our out-of-distribution loss LOOD on a simple binary classification problem. As we are working in
Euclidean space and not on a sphere, we use a modified version of MHE, which uses the negative squared Euclidean distanceinstead of the dot-product-similarity. For the formal relation between Equation (22) and MHE, we refer to Appendix G.1:

E(ξ; X) = − β

−1

log

N
X

β
exp(− ||ξ − xi ||22 )
2
i=1

!

(22)

Figure 4a shows the initial state of the patterns and the decision boundary exp(βEb (ξ; X, O)). We store the samples of
the two classes as stored patterns in X and O, respectively, and compute LOOD for all samples. We then set the learning
rate to 0.1 and perform gradient descent with LOOD on the data points. Figure 4b shows that after 25 steps, the distance
between the data points and the decision boundary has increased, especially for samples that had previously been close
to the decision boundary. After 100 steps, as shown in Figure 4d, the variability orthogonal to the decision boundary has
almost completely vanished, while the variability parallel to the decision boundary is maintained.
12Energy-based Hopfield Boosting for Out-of-Distribution Detection(a) After 0 steps(b) After 25 steps(c) After 50 steps(d) After 100 steps

Figure 4: LOOD applied to exemplary data points on euclidean space. Gradient updates are applied to the data points directly.
We observe that the variance orthogonal to the decision boundary shrinks while the variance parallel to the decision boundary
does not change to this extent. β is set to 2.

13

Energy-based Hopfield Boosting for Out-of-Distribution Detection

E.3. Dynamics of LOOD on Patterns on the Sphere(a) After 0 steps(b)
After 500 steps(c)After 2500 steps

Figure 5: LOOD applied to exemplary data points on a sphere. Gradients are applied to the data points directly. We observethat the geometry of the space forces the patterns to opposing poles of the sphere.
E.4. Learning Dynamics of Hopfield Boosting on Patterns on a Sphere - VideoThe example video1 demonstrates the learning dynamics of Hopfield Boosting on a 3-dimensional sphere. We randomly
generate ID patterns X clustering around one of the sphere’s poles and AUX patterns O on the remaining surface of thesphere. We then apply Hopfield Boosting on this data set. First, we sample the weak learners close to the decision boundary
for both classes, X and O. Then, we perform 2000 steps of gradient descent with LOOD on the sampled weak learners. We
apply the gradient updates to the patterns directly and do not propagate any gradients to an encoder. Every 50 gradient steps,
we re-sample the weak learners. For this example, the initial learning rate is set to 0.02 and increased after every gradientstep by 0.1%.
1

https://youtu.be/4AB3tILdrvQ

14

Energy-based Hopfield Boosting for Out-of-Distribution Detection

E.5. Location of Weak Learners near the Decision Boundary
A weak learner can be constructed from a sample close to the decision boundary as follows: Given the sample, we create aradial decision boundary centered on the sample. We now optimize one parameter (the radius of the decision boundary) tominimize the error on the training set. Our classifier is a weak learner because no matter how the radius is selected, theresulting hypothesis will likely only perform slightly better than random guessing.

Figure 6: A prototypical classifier (red circle) that is constructed with a sample close to the decision boundary. Classifiers
like this one will only perform slightly better than random guessing (as indicated by the radial decision boundaries) and are,therefore, well-suited for weak learners.

15

Energy-based Hopfield Boosting for Out-of-Distribution Detection

F. Notes on Eb
F.1. Probabilistic Interpretation of Eb
We model the class-conditional densities of the in-distribution data and auxiliary data as mixtures of Gaussians with thepatterns as the component means and tied, diagonal covariance matrices with β −1 in the main diagonal.

Np( ξ | ID ) =
1 XN ξ; xi , β −1 I
N i=1

(23)

M

p( ξ | AUX ) =
1 X
N ξ; oi , β −1 I
M i=1

(24)

Further, we assume the distribution p(ξ) as a mixture of p( ξ | ID ) and p( ξ | AUX ) with equal prior probabilities (mixture
weights):p(ξ) = p(ID) p( ξ | ID ) + p(AUX) p( ξ | AUX )11=p( ξ | ID ) + p( ξ | AUX )
2
2

(25)
(26)

The probability of an unknown sample ξ being an AUX sample is given byp( ξ | AUX ) p(AUX)p(ξ)
p( ξ | AUX )=2 p(ξ)p( ξ | AUX )=p( ξ | AUX ) + p( ξ | ID )1=p( ξ | ID )1 + p( ξ | AUX )p( AUX | ξ ) ==1
1 + exp(log(p( ξ | ID )) − log(p( ξ | AUX )))

(27)
(28)
(29)
(30)
(31)

where in line (30) we have used that p( ξ | AUX ) > 0 for all ξ ∈ Rd . The probability of ξ being an ID sample is given byp( ID | ξ ) =p( ξ | ID )2 p(ξ)(32)

1
1 + exp(log(p( ξ | AUX )) − log(p( ξ | ID )))= 1 − p( AUX | ξ )

=

(33)
(34)

Consider the function
fb (ξ) = p( AUX | ξ) · p( ID | ξ )=p( ξ | AUX ) · p( ξ | ID )4p(ξ)2
16

(35)
(36)

Energy-based Hopfield Boosting for Out-of-Distribution Detection
C

By taking the log of Equation (36) we obtain the following. We use = to denote equality up to an additive constant that
does not depend on ξ.
C

β −1 log (fb (ξ)) = − 2 β −1 log (p(ξ)) + β −1 log (p( ξ | ID )) + β −1 log (p( ξ | AUX ))(37)

Pre-multiplication by β −1 is equivalent to a change of base of the log. The term − β −1 log(p(ξ)) is equivalent to the MHE(Ramsauer et al., 2021) (up to an additive constant) when assuming normalized patterns, i.e. ||xi ||2 = 1 and ||oi ||2 = 1,
and an equal number of patterns M = N in the two Gaussian mixtures p( ξ | ID ) and p( ξ | AUX ):−β−1log(p(ξ)) = − β

−1

log

C



1
1
p( ξ | ID ) + p( ξ | AUX )22= − β −1 log (p( ξ | ID ) + p( ξ | AUX ))
N
X



(39)
N
X



1
1
N ξ; xi , β −1 I +N ξ; oi , β −1 I
N i=1
N i=1
!
N
N
X
X


C
−1
−1
−1
N ξ; oi , β I= − β logN ξ; xi , β I += − β −1 log

=−β

−1

log

!

(41)

N
X
β
β
exp(− ||ξ − oi ||22 )exp(− ||ξ − xi ||22 ) +
2
2
i=1
i=1

N
X

N
X

!

(42)

!

(43)

N
X
β
β
exp(βxTi ξ − ξ T ξ) += − β −1 log
exp(βoTi ξ − ξ T ξ)
2
2
i=1
i=1
!
N
N
X
X
1
exp(βoTi ξ) + ξ T ξ
exp(βxTi ξ) += − β −1 log
2
i=1
i=1
C

(40)

i=1

i=1

C

(38)

C

= − lse(β, (X ∥ O)T ξ) +1 T1ξ ξ + β −1 log N + M 2
2
2

(44)
(45)

Analogously, β −1 log(p( ξ | ID )) and β −1 log(p( ξ | AUX )) also yield MHE terms. Therefore, Eb is equivalent toβ −1 log(fb (ξ)) under the assumption that ||xi ||2 = 1 and ||oi ||2 = 1 and M = N . The 12 ξ T ξ terms that are contained in
the three MHEs cancel out.
C

β −1 log (fb (ξ)) = − 2 lse(β, (X ∥ O)T ξ) + lse(β, X T ξ) + lse(β, O T ξ) = Eb (ξ; X, O)(46)

fb (ξ) can also be interpreted as the variance of a Bernoulli distribution with outcomes ID and AUX:
fb (ξ) = p( AUX | ξ ) p( ID | ξ ) = p( ID | ξ )(1 − p( ID | ξ )) = p( AUX | ξ )(1 − p( AUX | ξ ))(47)

In other words, minimizing Eb means to drive a Bernoulli-distributed random variable with the outcomes ID and AUX
towards minimum variance, i.e., p( ID | ξ ) is driven towards 1 if p( ID | ξ ) > 0.5 and towards 0 if p( ID | ξ ) < 0.5.
Conversely, the same is true for p( AUX | ξ ).
From Equation (31), under the assumptions that ||xi ||2 = 1 and ||oi ||2 = 1 and M = N , the conditional probabilityp( AUX | ξ ) can be computed as follows:p( AUX | ξ ) = σ(log(p( ξ | AUX )) − log(p( ξ | ID )))TT= σ(β (lse(β, O ξ) − lse(β, X ξ)))
17

(48)
(49)

Energy-based Hopfield Boosting for Out-of-Distribution Detection

where σ denotes the logistic sigmoid function. Similarly, p( ID | ξ ) can be computed usingp( ID | ξ ) = σ(β (lse(β, X T ξ) − lse(β, O T ξ)))= 1 − p( AUX | ξ )

(50)
(51)

F.2. Alternative Formulations of Eb and fb
Eb can be rewritten as follows.
Eb (ξ; X, O) = − 2 lse(β, (X ∥ O)T ξ) + lse(β, X T ξ) + lse(β, O T ξ)β= − 2β −1 log cosh(lse(β, X T ξ) − lse(β, O T ξ)) − 2β −1 log(2)
2

(52)
(53)

To prove this, we first show the following:− β −1 log exp( β lse(β, X T ξ) ) + exp( β lse(β, O T ξ) )!!
NX
= − β −1 log exp β β −1 log
exp(βxTi ξ)
+ exp



(54)
β β −1 log

i=1

= − β

−1

log

N
X

exp(βxTi ξ) +NX

exp(βoTi ξ)

i=1

i=1

= − lse(β, (X ∥ O)T ξ)NX

exp(βoTi ξ)

i=1

!

!!!

(55)
(56)
(57)

Let EX = −lse(β, X T ξ) and EO = −lse(β, O T ξ).
Eb (ξ; X, O) = − 2 lse(β, (X ∥ O)T ξ) + lse(β, X T ξ) + lse(β, O T ξ)
= − 2β(58)−1log ( exp( − β EX ) + exp( − β EO ) ) − EX − EOββ
= − 2β −1 log exp( − EX ) + exp( − β EO + EX ) − EO
2
2


β
β
β
β
EO ) + exp( − EO + EX )= − 2β −1 log exp( − EX +
2
2
2
2


β
= − 2β −1 log cosh
( − EX + EO ) − 2β −1 log(2)2β(lse(β, X T ξ) − lse(β, O T ξ)) − 2β −1 log(2)= − 2β −1 log cosh2β
= − 2β −1 log cosh(lse(β, O T ξ) − lse(β, X T ξ)) − 2β −1 log(2)
2

(59)
(60)
(61)
(62)
(63)
(64)

By exponentiation of the above result we obtainfb (ξ) ∝ exp(βEb (ξ; X, O)) =1
4 cosh2βTT2 (lse(β, X ξ) − lse(β, O ξ))



(65)

The function log cosh(x) is related to the negative log-likelihood of the hyperbolic secant distribution (see e.g. Saleh &2
Saleh, 2022). For values of x close to 0, log cosh can be approximated by x2 , and for values far from 0, the function behaves
as |x| − log(2).
18Energy-based Hopfield Boosting for Out-of-Distribution Detection(a) Probabilities(b) Log-probabilities

Figure 7: The product of two logistic sigmoids yields fb (a); the sum of two log-sigmoids yields log(fb ) = Eb (b).
F.3. Derivatives of Eb
In this section, we investigate the derivatives of the energy function Eb . The derivative of the lse is:∇z lse(β, z) = ∇z β

−1

log

N
X
i=1

exp(βzi )!

(66)= softmax(β z)Thus, the derivative of the MHE E(ξ; X) w.r.t. ξ is:∇ξ E(ξ; X) = ∇ξ (−lse(β, X T ξ) +1 T
ξ ξ + C) = − Xsoftmax(βX T ξ) + ξ
2(67)

The update rule of the MHN

ξ t+1 = Xsoftmax(βX T ξ t )(68)

is derived via the concave-convex procedure. It coincides with the attention mechanisms of Transformers and has been
proven to converge globally to stationary points of the energy E(ξ; X) (Ramsauer et al., 2021). It can also be shown thatthe update rule emerges when performing gradient descent on E(ξ; X) with step size η = 1 Park et al. (2023):

ξ t+1 = ξ t − η ∇ξ E(ξ t ; X)(69)

ξ t+1 = Xsoftmax(βX T ξ t )(70)

From Equation (67), we can see that the gradient of Eb (ξ; X, O) w.r.t. ξ is:∇ξ Eb (ξ; X, O) = ∇ξ (− 2 lse(β, (X ∥ O)T ξ) + lse(β, X T ξ) + lse(β, O T ξ))
T

T

(71)
T

= − 2 (X ∥ O) softmax(β(X ∥ O) ξ) + Xsoftmax(βX ξ) + Osoftmax(βO ξ)
19

(72)

Energy-based Hopfield Boosting for Out-of-Distribution Detection

When Xsoftmax(βX T ξ), Osoftmax(βO T ξ), lse(β, X T ξ) and lse(β, O T ξ) are available, one can efficiently compute(X ∥ O) softmax(β(X ∥ O)T ξ) as follows:(X ∥ O) softmax(β(X ∥ O)T ξ)

(73)

T

= ∇ξ lse(β, (X ∥ O) ξ)= ∇ξ β

−1

(74)
T

T



log exp(βlse(β, X ξ)) + exp(βlse(β, O ξ)) lse(β, X T ξ)
T
T
= Xsoftmax(βX ξ) Osoftmax(βO ξ) softmax βlse(β, O T ξ)

(75)
(76)

We can also compute the gradient of Eb (ξ; X, O) w.r.t. ξ via the log cosh-representation of Eb (see Equation (64)). The
derivative of the log cosh function isd −1
β log cosh(βx) = tanh(βx)
dx

(77)

Therefore, we can compute the gradient of Eb (ξ; X, O) as∇ξ Eb (ξ; X, O)

(78)




β
= ∇ξ − 2β −1 log cosh(lse(β, O T ξ) − lse(β, X T ξ))
2



β
(lse(β, O T ξ) − lse(β, X T ξ)) Osoftmax(βO T ξ) − Xsoftmax(βX T ξ)
= − tanh2β= − tanh(lse(β, X T ξ) − lse(β, O T ξ)) Xsoftmax(βX T ξ) − Osoftmax(βO T ξ)
2

(79)
(80)
(81)

Next, we would like to compute the gradient of Eb (ξ; X, O) w.r.t. the memory matrices X and O. For this, let us first lookat the gradient of the MHE E(ξ; X) w.r.t. a single stored pattern xi (where X is the matrix of concatenated stored patterns(x1 , x2 , . . . , xN )):∇xi E(ξ; X) = − ξsoftmax(βX T ξ)i(82)

Thus, the gradient w.r.t. the full memory matrix X is

∇X E(ξ; X) = −ξsoftmax(βX T ξ)T(83)

We can now also use the log cosh formulation of Eb (ξ; X, O) to compute the gradient of Eb (ξ; X, O), w.r.t X and O:βT
T(lse(β, X ξ) − lse(β, O ξ))∇X Eb (ξ; X, O) = ∇X − 2βlog cosh
2


β
T
T
= − tanh(lse(β, X ξ) − lse(β, O ξ)) ξsoftmax(βX T ξ)T
2
−1



20

(84)
(85)
(86)

Energy-based Hopfield Boosting for Out-of-Distribution Detection

Analogously, the gradient w.r.t O is

∇O Eb (ξ; X, O) = − tanh




β
T
T
(lse(β, O ξ) − lse(β, X ξ)) ξsoftmax(βO T ξ)T
2

21

(87)

Energy-based Hopfield Boosting for Out-of-Distribution Detection

G. Notes on the Relationship between Hopfield Boosting and other methods
G.1. Relation to Radial Basis Function Networks
This section shows the relation between radial basis function networks (RBF networks; Moody & Darken, 1989) and modern
Hopfield energy (following Schäfl et al., 2022). Consider an RBF network with normalized linear weights:

φ(ξ) =NX

β
ωi exp(− ||ξ − µi ||22 )
2
i=1

(88)

where β denotes the inverse tied variance β = σ12 , and the ωi are normalized using the softmax function:exp(βai )ωi = softmax(βa)i = PN
j=1 exp(βaj )(89)

An energy can be obtained by taking the negative log of φ(ξ):E(ξ) = − β −1 log (φ(ξ))= − β −1 log

N
X

(90)

ωi exp(−i=1β||ξ − µi ||22 ))
2

!

!
1
2
−1
= −β
log
exp( β(− ||ξ − µi ||2 + β log softmax(βa)i ) )
2
i=1
!
N
X
1
−1
2
exp( β(− ||ξ − µi ||2 + ai − lse(β, a) )
= −β
log
2
i=1
!
N
X
1 T
1 T
T
−1
= −β
log
exp( β(− ξ ξ + µi ξ − µi µi + ai ) ) + lse(β, a)
2
2
i=1
−1

N
X

(91)
(92)
(93)
(94)

Next, we define ai = 21 µTi µi

E(ξ) = − β

−1

log

N
X
i=1

!

1
+ ξ T ξ + lse(β, a)
2

(95)

1
1
+ ξ T ξ + β −1 log N + M 2
2
2

(96)

exp(βµTi ξ)

Finally, we use the fact that lse(β, a) ≤ maxi ai + β −1 log N

E(ξ) = − β −1 log

NX

exp(βµTi ξ)

i=1!

where M = maxi ||µi ||2
G.2. Contrastive Representation Learning
A commonly used loss function in contrastive representation learning (e.g., Chen et al., 2020; He et al., 2020) is the InfoNCE
loss (Oord et al., 2018):
22

Energy-based Hopfield Boosting for Out-of-Distribution Detection

LNCE =

E(x,y)∼pposM{x−i }i=1 ∼pdata"Tef (x) f (y)/τ
− log
P
− T
ef (x)T f (y)/τ + i ef (xi ) f (y)/τ

#

(97)

Wang & Isola (2020) show that LNCE optimizes two objectives:

LNCE =

E(x,y)∼ppos



|

T

−f (x) f (y)/τ{z

Alignment



+

}

|

E

(x,y)∼pposM{x−i }i=1 ∼pdata

"

log e

f (x)T f (y)/τ

+

X
i

{z

Uniformity

e

T
f (x−
i ) f (x)/τ

!#

(98)

}

Alignment enforces that features from positive pairs are similar, while uniformity encourages a uniform distribution of thesamples over the hypersphere.
In comparison, our proposed loss, LOOD , does not visibly enforce alignment between samples within the same class. Instead,
we can observe that it promotes uniformity to the instances of the foreign class. Due to the constraints that are imposed by
the geometry of the space the optimization is performed on, that is, ||f (x)|| = 1 when the samples move on a hypersphere,
the loss encourages the patterns in the ID data have maximum distance to the samples of the AUX data, i.e., they concentrateon opposing poles of the hypersphere. A demonstration of this mechanism can be found in Appendix E.2 and E.3
G.3. Support Vector Machines
In the following, we will show the relation of Hopfield Boosting to support vector machines (SVMs; Cortes & Vapnik, 1995)
with RBF kernel. We adopt and expand the arguments of Schäfl et al. (2022).
Assume we apply an SVM with RBF kernel to model the decision boundary between ID and AUX data. We train on thefeatures Z = (x1 , . . . , xN , o1 , . . . , oM ) and assume that the patterns are normalized, i.e., ||xi ||2 = 1 and ||oi ||2 = 1. We
define the targets (y1 , . . . , y(N +M ) ) as 1 for ID and −1 for AUX data. The decision rule of the SVM equates toB̂(ξ) =(

ID
if s(ξ) ≥ 0
OOD if s(ξ) < 0

(99)

where

s(ξ) =NX+M

αi yi k(zi , ξ)

(100)



(101)

i=1



β
k(zi , ξ) = exp − ||ξ − zi ||222

We assume that there is at least one support vector for both ID and AUX data, i.e., there exists at least one index i s.t.
αi yi > 0 and at least one index j s.t. αj yj < 0. We now split the samples zi in s(ξ) according to their label:s(ξ) =NX
i=1

αi k(xi , ξ) −We define an alternative score:
23

M
X
i=1

αN +i k(oi , ξ)(102)

Energy-based Hopfield Boosting for Out-of-Distribution Detection

PN
αi k(xi , ξ)sfrac (ξ) = PMi=1i=1 αN +i k(oi , ξ)

(103)
(104)

Because we assumed there is at least one support vector for both ID and AUX data and as the αi are constrained to be
non-negative and because k(·, ·) > 0, the numerator and denominator are strictly positive. We can, therefore, specify a newdecision rule B̂frac (ξ).

(

B̂frac (ξ) =ID
if sfrac (ξ) ≥ 1
OOD if sfrac (ξ) < 1(105)

Although the functions s(ξ) and sfrac (ξ) are different, the decision rules B̂(ξ) and B̂frac (ξ) are equivalent. Another possible
pair of score and decision rule is the following:slog (ξ) = β−1log(sfrac (ξ)) = β

−1

log

N
X

αi k(xi , ξ)i=1(

B̂log (ξ) =Let us more closely examine the term β −1 log

β

−1

log

N
X
i=1

αi k(xi , ξ)

!

P

= β

−1

!

− β

ID
if slog (ξ) ≥ 0
OOD if slog (ξ) < 0N
i=1 αi k(xi , ξ)



−1

log

M
X

αN +i k(oi , ξ)

i=1

!

!
β
2
exp(βai ) exp − ||ξ − xi ||2
2
i=1
!

N
X
β T
β T
T
exp(βai ) exp − ξ ξ + βxi ξ − xi xi
2
2
i=1

!
N
X
β T
β
exp − ξ ξ + βxTi ξ − xTi xi + βai
2
2
i=1
!
N
X

1
1
exp βxTi ξ + βai− ξT ξ −22i=1= β −1 log= β −1 log= β −1 log(107). We define ai = β −1 log(αi ).

N
X

log

(106)



(108)
(109)
(110)
(111)

We now construct a memory XH and query ξH such that we can compute (111) using the MHE (Equation (4)):

. . . xN. . . aN
 
ξ
ξH =
1

XH =

x1
a1

We obtain
24



(112)
(113)

Energy-based Hopfield Boosting for Out-of-Distribution Detection

1 Tξ ξH + C
2 H

T
E(ξH ; XH ) = − lse(β, XHξH ) +

= −β

−1

= −β

−1

= −β

−1

log

N
X

exp βxTi ξ + 1βai

i=1

log

N
X

βxTi ξ + βai

exp

i=1

log

N
X

αi k(xi , ξ)

i=1

!



(114)


!

!

+

1 T
1
ξ ξ + · 12 + C
2
2

(115)

+

1
1 T
ξ ξ + + C
2
2

(116)

+ C

(117)

We construct OH analogously to Equation (112) and thus can computeTTslog (ξ) = E(ξH ; OH ) − E(ξH ; XH ) = lse(β, XHξH ) − lse(β, OH
ξH )

(118)

which is exactly the score Hopfield Boosting uses for determining whether a sample is OOD (Equation (9)). In contrast to
SVMs, Hopfield Boosting uses a uniform weighting of the patterns in the memory when computing the score. However,
Hopfield Boosting can emulate a weighting of the patterns by more frequently sampling patterns with high weights into thememory.
G.4. HE and SHE
Zhang et al. (2023a) introduce two post-hoc methods for OOD detection using MHE, which are called “Hopfield Energy”(HE) and “Simplified Hopfield Energy” (SHE). Like Hopfield Boosting, HE and SHE both employ the MHE to determine
whether a sample is ID or OOD. However, unlike Hopfield Boosting, HE and SHE offer no possibility to include AUX datain the training process to improve the OOD detection performance of their method. The rest of this section is structured asfollows: First, we briefly introduce the methods HE and SHE, second, we formally analyze the two methods, and third, we
relate them to Hopfield Boosting.
Hopfield Energy (HE)The method HE (Zhang et al., 2023a) computes the OOD score sHE (ξ) as follows:sHE (ξ) = lse(β, XcT ξ)(119)where Xc ∈ Rd×Nc denotes the memory (xc1 , . . . , xcNc ) containing Nc encoded data instances of class c. HE uses theprediction of the ID classification head to determine which patterns to store in the Hopfield memory:c = argmax p( y | ξ D )

(120)

y

Simplified Hopfield Energy (SHE) The method SHE (Zhang et al., 2023a) employs a simplified score sSHE (ξ):

sSHE (ξ) = mTc ξ(121)

where mc ∈ Rd denotes the mean of the patterns in memory Xc :
N

mc =

c
1 X
xci
Nc i=1

25(122)

Energy-based Hopfield Boosting for Out-of-Distribution Detection

Relation between HE and SHE In the following, we show a simple yet enlightening relation between the scores sHE andsSHE . For mathematical convenience, we first slightly modify the score sHE :sHE (ξ) = lse(β, XcT ξ) − β −1 log Nc(123)

All data sets which were employed in the experiments of Zhang et al. (2023a) (CIFAR-10 and CIFAR-100) are class-balanced.
Therefore, the additional term β −1 log Nc does not change the result of the OOD detection on those data sets, as it only
amounts to the same constant offset for all classes.
The functionN

1 X
exp(βzi )N i=1

lse(β, z) − β −1 log N = β −1 log

!

(124)

converges to the mean function as β → 0:Nlim (lse(β, z) − β −1 log N ) =

β→0

1 X
zi
N i=1

(125)

We now investigate the behavior of sHE in this limit:

N

1 X T
(x ξ)
N i=1 ci
!T
N
1 X
=
ξ
xci
N i=1

lim (lse(β, XcT ξ) − β −1 log N ) =β→0= mTc ξ

(126)

(127)
(128)

where

N

mc =

1 X
xci
N i=1

(129)

Therefore, we have shown thatlim sHE (ξ) = sSHE (ξ)

β→0

(130)

Relation of HE and SHE to Hopfield Boosting. In contrast to HE and SHE, Hopfield Boosting uses an AUX data set tolearn a decision boundary between the ID and OOD regions during the training process. To do this, our work introduces
a novel MHE-based energy function, Eb (ξ; X, O), to determine how close a sample is to the learned decision boundary.
Hopfield Boosting uses this energy function to frequently sample weak learners into the Hopfield memory and for computing
a novel Hopfield-based OOD loss LOOD . To the best our knowledge, we are the first to use MHE in this way to train a neuralnetwork.
26

Energy-based Hopfield Boosting for Out-of-Distribution Detection

The OOD detection score of Hopfield Boosting iss(ξ) = lse(β, X T ξ) − lse(β, O T ξ).

(131)

where X ∈ Rd×N contains the full encoded training set (x1 , . . . , xN ) of all classes and O ∈ Rd×M contains AUX samples.
While certainly similar to sHE , the Hopfield Boosting score s differs from sHE in three crucial aspects:
1. Hopfield Boosting uses AUX data samples in the OOD detection score in order to create a sharper decision boundary
between the ID and OOD regions.
2. Hopfield Boosting normalizes the patterns in the memories X and O and the query ξ to unit length, while HE andSHE use unnormalized patterns to construct their memories Xc and their query pattern ξ.
3. The score of Hopfield Boosting, s(ξ), contains the full encoded training data set, while sHE only contains the patternsof a single class. Therefore Hopfield Boosting computes the similarities of a query sample ξ to the entire ID data set.
In Appendix H.10, we show that this process only incurs a moderate overhead of 7.5% compared to the forward passof the ResNet-18.
The selection of the score function s(ξ) is only a small aspect of Hopfield Boosting. Hopfield Boosting additionally samples
informative AUX data close to the decision boundary, optimizes an MHE-based loss function, and thereby learns a sharp
decision boundary between ID and OOD regions. Those three aspects are novel contributions of Hopfield Boosting. In
contrast, the work of Zhang et al. (2023a) solely focuses on the selection of a suitable Hopfield-based OOD detection score
for post-hoc OOD detection.

27

Energy-based Hopfield Boosting for Out-of-Distribution Detection

Table 2: OOD detection performance on CIFAR-10. We compare results from Hopfield Boosting, DOS (Jiang et al., 2024),
DOE (Wang et al., 2023b), DivOE (Zhu et al., 2023), DAL (Wang et al., 2023a), MixOE (Zhang et al., 2023b), POEM
(Ming et al., 2022), EBO-OE (Liu et al., 2020), and MSP-OE (Hendrycks et al., 2019b) on ResNet-18. ↓ indicates “lower isbetter” and ↑ “higher is better”. All values in %. Standard deviations are estimated across five training runs.
OOD Dataset
SVHN
LSUN-Crop
LSUN-Resize
TexturesiSUN
Places 365
Mean

MetricHB (ours)

DOS

DOE

DivOE

DAL

MixOE

POEM

EBO-OEMSP-OE

FPR95 ↓
AUROC ↑FPR95 ↓
AUROC ↑FPR95 ↓
AUROC ↑FPR95 ↓
AUROC ↑FPR95 ↓
AUROC ↑FPR95 ↓
AUROC ↑

0.23±0.08
99.57±0.06
0.82±0.17
99.40±0.04
0.00±0.00
99.98±0.02
0.16±0.02
99.84±0.01
0.00±0.00
99.97±0.02
4.28±0.23
98.51±0.10

3.09±0.75
99.15±0.22
3.66±0.98
99.04±0.20
0.00±0.00
99.99±0.01
1.28±0.20
99.63±0.04
0.00±0.00
99.99±0.01
12.26±0.97
96.63±0.43

1.97±0.58
99.60±0.13
3.22±0.45
99.30±0.12
0.00±0.00
100.00±0.00
2.75±0.57
99.35±0.12
0.00±0.00
100.00±0.00
19.72±2.39
95.06±0.72

6.21±0.84
98.53±0.08
1.88±0.25
99.50±0.02
0.00±0.00
99.89±0.05
1.20±0.11
99.59±0.02
0.00±0.00
99.88±0.05
13.70±0.50
96.95±0.09

1.25±0.62
99.61±0.15
4.17±0.27
99.13±0.02
0.00±0.00
99.92±0.05
0.95±0.13
99.74±0.01
0.00±0.00
99.93±0.04
14.22±0.51
96.77±0.07

27.54±2.46
95.37±0.44
0.14±0.07
99.61±0.11
0.16±0.17
99.89±0.06
4.68±0.22
98.91±0.07
0.17±0.12
99.87±0.05
16.30±1.09
96.92±0.22

1.48±0.68
99.33±0.15
4.02±0.91
98.89±0.15
0.00±0.00
99.88±0.12
0.49±0.04
99.72±0.05
0.00±0.00
99.87±0.12
7.70±0.68
97.56±0.26

2.66±0.91
99.15±0.23
6.82±0.74
98.43±0.10
0.00±0.00
99.98±0.02
1.11±0.17
99.61±0.02
0.00±0.00
99.98±0.01
11.77±0.68
96.39±0.30

4.31±1.10
99.20±0.15
7.02±1.14
98.83±0.15
0.00±0.00
99.96±0.00
2.29±0.16
99.57±0.01
0.00±0.00
99.96±0.00
21.42±0.88
95.91±0.17

0.92
99.55

3.38
99.07

4.61
98.88

3.83
99.06

3.43
99.18

8.17
98.43

2.28
99.21

3.73
98.92

5.84
98.90

FPR95 ↓
AUROC ↑

H. Additional Experiments & Experimental Details
H.1. Hyperparameter Setup
Like Yang et al. (2022), we use SGD with an initial learning rate of 0.1 and a weight decay of 5 · 10−4 . We decrease thelearning rate during the training process with a cosine schedule (Loshchilov & Hutter, 2016). We sample mini-batchesof size 128 per data set For a fair comparison, we apply these settings to all OOD detection methods that we test. For
training Hopfield Boosting, we use a single value for β throughout the training and evaluation process and for all OOD data
sets. We tune the value of β for each ID data set separately by selecting the value of β from the set {2, 4, 8, 16, 32} that
performs best in the validation process. We select λ — the weight for the OOD loss LOOD — from {0.1, 0.25, 0.5, 1.0}. In
our experiments, β = 4 and λ = 0.5 yields the best results for CIFAR-10 and CIFAR-100. For ImageNet-1K, we set β = 32
and λ = 0.25. To tune the hyperparameters, we use a validation process with different OOD data for model selection.
Specifically, we validate the model on MNIST (LeCun et al., 1998), and ImageNet-RC with different pre-processing than intraining (resize to 32x32 pixels instead of crop to 32x32 pixels), as well as Gaussian and uniform noise. We conducted ourexperiments on various NVIDIA GPUs (e.g., Titan V, A100) on an internal cluster.
H.2. Results on CIFAR-10
Table 2 shows the results from Section 4 with five additional baselines.
H.3. Results on CIFAR-100
We conduct experiments on CIFAR-100 as ID data set. In this setting, we closely follow the experimental setup for
CIFAR-10 described in Section 4.2. Table 3 shows that Hopfield Boosting surpasses POEM (the previously best method),
improving the mean FPR95 from 11.76 to 7.95. On the SVHN data set, Hopfield Boosting improves the FPR95 metric themost, decreasing it from 33.59 to 13.27.
H.4. Results on ImageNet-1K
Following Wang et al. (2023b), we evaluate Hopfield Boosting on the large-scale benchmark: We use ImageNet-1K(Russakovsky et al., 2015) as ID data set and ImageNet-21K (Ridnik et al., 2021) as AUX data set. The OOD test data
sets are Textures (Cimpoi et al., 2014), SUN (Xu et al., 2015), Places 365 (López-Cifuentes et al., 2020), and iNaturalist(Van Horn et al., 2018). In this setting, we fine-tune a pre-trained ResNet-50 using Hopfield Boosting. As we show inTable 4, Hopfield Boosting surpasses all methods in our comparison in terms of both mean FPR95 and mean AUROC.
Compared to POEM (the previously best method) Hopfield Boosting improves the mean FPR95 from 50.74 to 36.60. This
demonstrates that Hopfield Boosting scales very favourably to large-scale settings.
28

Energy-based Hopfield Boosting for Out-of-Distribution Detection

Table 3: OOD detection performance on CIFAR-100. We compare results from Hopfield Boosting, DOS (Jiang et al., 2024),
DOE (Wang et al., 2023b), DivOE (Zhu et al., 2023), DAL (Wang et al., 2023a), MixOE (Zhang et al., 2023b), POEM
(Ming et al., 2022), EBO-OE (Liu et al., 2020), and MSP-OE (Hendrycks et al., 2019b) on ResNet-18. ↓ indicates “lower isbetter” and ↑ “higher is better”. All values in %. Standard deviations are estimated across five training runs.
OOD Dataset
SVHN
LSUN-Crop
LSUN-Resize
TexturesiSUN
Places 365
Mean

MetricHB (ours)

DOS

DOE

DivOE

DAL

MixOE

POEM

EBO-OEMSP-OE

FPR95 ↓
AUROC ↑FPR95 ↓
AUROC ↑FPR95 ↓
AUROC ↑FPR95 ↓
AUROC ↑FPR95 ↓
AUROC ↑FPR95 ↓
AUROC ↑

13.27±5.46
97.07±0.81
12.68±2.38
96.54±0.65
0.00±0.00
99.98±0.01
2.35±0.13
99.22±0.02
0.00±0.00
99.98±0.01
19.36±1.02
95.85±0.37

9.84±2.75
97.64±0.39
19.40±2.45
96.42±0.35
0.01±0.00
99.96±0.02
6.02±0.52
98.33±0.11
0.03±0.01
99.95±0.02
32.13±1.55
91.73±0.39

19.38±4.60
95.72±1.1228.23±2.69
93.79±0.88
0.05±0.04
99.99±0.01
19.42±1.58
94.93±0.48
0.01±0.02
99.99±0.00
58.68±4.15
83.47±1.55

28.77±5.42
94.25±0.98
35.10±4.2392.45±0.94
0.01±0.00
99.99±0.00
11.52±0.49
97.02±0.08
0.06±0.01
99.97±0.00
44.20±0.95
88.28±0.26

19.95±2.34
95.69±0.66
24.24±2.12
95.04±0.43
0.00±0.00
99.94±0.02
5.22±0.39
98.50±0.16
0.01±0.02
99.93±0.02
33.43±1.11
91.10±0.29

41.54±13.16
92.27±2.71
23.10±7.39
96.11±1.09
10.27±10.72
97.99±1.92
28.99±6.79
94.24±1.2114.40±13.48
97.23±2.5947.01±6.41
89.20±1.8633.59±4.12
94.06±0.51
15.72±3.4696.85±0.60
0.00±0.00
99.57±0.09
2.89±0.32
98.97±0.08
0.00±0.00
99.59±0.09
18.39±0.68
95.03±0.71

36.33±2.95
92.93±0.72
21.06±3.1295.79±0.62
0.00±0.00
99.57±0.03
5.07±0.54
98.15±0.16
0.00±0.00
99.57±0.03
26.68±2.18
91.35±0.70

19.86±6.90
95.74±1.60
32.88±1.2892.85±0.33
0.03±0.01
99.97±0.00
10.34±0.40
97.42±0.08
0.08±0.02
99.96±0.01
45.96±0.85
87.77±0.15

7.94
98.11

11.24
97.34

20.96
94.65

19.94
95.33

13.81
96.70

27.55
94.51

11.76
97.34

14.86
96.23

18.19
95.62

FPR95 ↓
AUROC ↑

Table 4: OOD detection performance on ImageNet-1K. We compare results from Hopfield Boosting, DOS (Jiang et al.,2024), DOE (Wang et al., 2023b), DivOE (Zhu et al., 2023), DAL (Wang et al., 2023a), MixOE (Zhang et al., 2023b),
POEM (Ming et al., 2022), EBO-OE (Liu et al., 2020), and MSP-OE (Hendrycks et al., 2019b) on ResNet-50. ↓ indicates“lower is better” and ↑ “higher is better”. All values in %. Standard deviations are estimated across five training runs.
HB (ours)
Textures
SUN
Places 365
iNaturalist
Mean

FPR95 ↓
AUROC ↑FPR95 ↓
AUROC ↑FPR95 ↓
AUROC ↑FPR95 ↓
AUROC ↑FPR95 ↓
AUROC ↑

±1.05

DOS

DOE
±0.93

DivOE
±7.19

±0.74

DAL

MixOE
±0.66

±4.91

POEM
±0.67

EBO-OE±1.26

MSP-OE

44.59
88.01±0.57
37.37±1.84
91.24±0.5253.31±2.05
87.10±0.52
11.11±0.66
97.65±0.20

40.29
89.88±0.18
59.29±0.9684.30±0.2169.72±1.01
81.62±0.2249.55±1.41
90.49±0.38

83.83
64.22±9.25
83.73±8.78
72.95±7.9486.30±6.69
70.37±7.1770.82±13.89
83.82±5.75

42.80
88.18±0.06
61.00±0.57
83.64±0.30
71.09±0.60
80.35±0.33
30.51±0.4293.81±0.10

43.8887.39±0.15
65.31±0.61
81.47±0.22
74.46±0.75
78.72±0.28
51.92±0.74
88.33±0.21

41.05
88.51±1.29
65.14±2.5382.20±0.72
71.34±1.4980.31±0.4247.28±1.55
90.19±0.35

31.26
92.22±0.14
57.46±0.9085.38±0.35
68.87±1.05
81.79±0.40
45.37±1.79
92.01±0.33

29.67
92.40±0.23
57.69±1.61
85.83±0.60
70.03±1.8381.35±0.63
49.02±4.40
91.44±0.79

48.38±0.87
86.25±0.25
66.01±0.26
81.45±0.20
74.58±0.44
78.89±0.19
51.73±1.35
88.51±0.30

36.60
91.00

54.71
86.57

81.17
72.84

51.35
86.49

58.90
83.98

56.20
85.30

50.74
87.85

51.60
87.75

60.17
83.78

H.5. Comparison HE/SHE
Since Hopfield Boosting shares similarities with the MHE-based methods HE and SHE (Zhang et al., 2023a), we also lookedat the approach as used for their methods. We use the same ResNet-18 as a backbone network as we used in the experiments
for Hopfield Boosting, but train it on CIFAR-10 without OE. We modify the approach of Zhang et al. (2023a) to not only use
the penultimate layer, but perform a search over all layer activation combinations of the backbone for the best-performing
combination. We also do not use the classifier to separate by class. From the search, we see that the concatenated activationsof layers 3 and 5 give the best performance on average, so we use this setting. We experience a quite noticeable drop in
performance compared to their results (Table 5). Since the computation of the MHE is the same, we assume the reason forthe performance drop is the different training of the ResNet-18 backbone network, where (Zhang et al., 2023a) used strong
augmentations.
H.6. Ablations
In the first ablation we look at the impact of the boosting itself, specifically the sampling of weak learners. The comparison isdone by training the same network architecture (Resnet-18) with and without weak learner sampling. The experiment showsthat boosting has a noticeable effect on the performance: Hopfield Boosting performs better with weak learner sampling.
We investigate the impact of different encoder backbone architectures on OOD detection performance with Hopfield
Boosting. The baseline uses a ResNet-18 as the encoder architecture. For the ablation, the following architectures are29

Energy-based Hopfield Boosting for Out-of-Distribution Detection

Table 5: Comparison between HE, SHE and our version. ↓ indicates “lower is better” and ↑ indicates “higher is better”.

OOD Dataset
SVHN
LSUN-Crop
LSUN-Resize
TexturesiSUN
Places 365
Mean

Ours
FPR95 ↓ AUROC ↑FPR95 ↓HE
AUROC ↑SHE
FPR95 ↓ AUROC ↑

36.79
13.10
16.65
44.54
19.20
39.02

93.18
97.25
96.84
89.38
96.08
90.63

35.81
17.74
20.69
46.29
22.52
41.56

92.35
95.96
95.87
86.67
95.08
88.41

35.07
18.19
21.66
46.19
23.25
42.57

92.81
96.10
95.85
87.44
95.06
88.38

28.21

93.89

30.77

92.39

31.66

92.60

Table 6: OOD detection performance on CIFAR-10. We compare Hopfield Boosting trained with weighted sampling andrandom sampling on ResNet-18. ↓ indicates “lower is better” and ↑ indicates “higher is better”. All values in %. Standard
deviations are estimated across five training runs.
OOD Dataset
SVHN
LSUN-Crop
LSUN-Resize
TexturesiSUN
Places 365
Mean

Weighted Sampling
FPR95 ↓
AUROC ↑
0.23±0.08
0.82±0.20
0.00±0.00
0.16±0.02
0.00±0.00
4.28±0.26

99.57±0.06
99.40±0.05
99.98±0.02
99.85±0.01
99.97±0.02
98.51±0.11

Random Sampling
FPR95 ↓
AUROC ↑
0.70±0.13
1.58±0.31
0.00±0.00
0.26±0.06
0.00±0.00
6.20±0.21

99.55±0.08
99.24±0.10
99.98±0.01
99.81±0.02
99.99±0.00
97.68±0.21

0.92

99.55

1.46

99.38

used as a comparison: ResNet-34, ResNet-50, and Densenet-100. It can be observed, that the larger architectures lead to aslight increase in OOD performance (Table 7). We also see that a change in architecture from ResNet to Densenet leads toa different OOD behavior: The result on the Places365 data set is greatly improved, while the performance on SVHN isnoticeably worse than on the ResNet architectures. The FPR95 of Densenet on SVHN also shows a high variance, which isdue to one of the five independent training runs performing very badly at detecting SVHN samples as OOD: The worst runscores an FPR95 5.59, while the best run achieves an FPR95 of 0.24.
H.7. Effect on Learned Representation
In order to analyze the impact of Hopfield Boosting on learned representations, we utilize the output of our model’s
embedding layer (see 4.2) as the input for a manifold learning-based visualization. Uniform Manifold Approximation
and Projection (UMAP) McInnes et al. (2018) is a non-linear dimensionality reduction technique known for its ability topreserve both global and local structure in high-dimensional data.
First, we train two models – with and without Hopfield Boosting– and extract the embeddings of both ID and OOD data setsfrom them. This results in a 512-dimensional vector representation for each data point, which we further reduce to twodimensions with UMAP. The training data for UMAP always corresponds to the training data of the respective method.
That is, the model trained without Hopfield Boosting is solely trained on CIFAR-10 data, and the model trained with
Hopfield Boosting is presented with CIFAR-10 and AUX data during training, respectively. We then compare the learned
representations concerning ID and OOD data.
Figure 8 shows the UMAP embeddings of ID (CIFAR-10) and OOD (AUX and SVHN) data based on our model trained
without (a) and with Hopfield Boosting (b). Without Hopfield Boosting, OOD data points typically overlap with ID data
points, with just a few exceptions, making it difficult to differentiate between them. Conversely, Hopfield Boosting allows to
distinctly separate ID and OOD data in the embedding.
30

Energy-based Hopfield Boosting for Out-of-Distribution Detection

Table 7: Comparison of OOD detection performance on CIFAR-10 of Hopfield Boosting on different encoders. ↓ indicates“lower is better” and ↑ indicates “higher is better”. Standard deviations are estimated across five independent training runs.

OOD Dataset
SVHN
LSUN-Crop
LSUN-Resize
TexturesiSUN
Places 365
Mean

ResNet-18
FPR95 ↓
AUROC ↑

ResNet-34
FPR95 ↓
AUROC ↑

ResNet-50
FPR95 ↓
AUROC ↑

Densenet-100
FPR95 ↓
AUROC ↑

0.23±0.08
0.82±0.20
0.00±0.00
0.16±0.02
0.00±0.00
4.28±0.26

99.57±0.06
99.40±0.05
99.98±0.02
99.85±0.01
99.97±0.02
98.51±0.11

0.33±0.25
0.65±0.14
0.00±0.00
0.15±0.07
0.00±0.00
4.13±0.54

99.63±0.07
99.54±0.07
99.89±0.04
99.89±0.04
99.98±0.02
98.46±0.22

0.19±0.09
0.69±0.15
0.00±0.00
0.16±0.07
0.00±0.00
4.75±0.45

99.64±0.1199.47±0.09
99.93±0.10
99.83±0.01
99.98±0.02
98.71±0.05

2.11±2.76
0.40±0.23
0.00±0.00
0.08±0.03
0.00±0.00
2.56±0.20

99.31±0.35
99.52±0.09
100.0±0.00
99.88±0.01
99.99±0.01
99.26±0.03

0.92

99.55

0.88

99.57

0.97

99.59

0.86

99.66

(a) without Hopfield Boosting(b) with Hopfield Boosting

Figure 8: UMAP embeddings of ID (CIFAR-10) and OOD (AUX and SVHN) data based on our model trained without (a)and with Hopfield Boosting (b). Clearly, without Hopfield Boosting, the embedded OOD data points tend to overlap with
the ID data points, making it impossible to distinguish between ID and OOD. On the other hand, Hopfield Boosting shows aclear separation of ID and OOD data in the embedding.

H.8. OOD Examples from the Places 365 Data Set with High Semantic Similarity to CIFAR-10
We observe that Hopfield Boosting and all competing methods struggle with correctly classifying the samples from thePlaces 365 data set as OOD the most. Table 1 shows that for Hopfield Boosting, the FPR95 for the Places 365 data set with
CIFAR-10 as the ID data set is at 4.28. The second worst FPR95 for Hopfield Boosting was measured on the LSUN-Cropdata set at 0.82.
We inspect the 100 images from Places 365 that perform worst (i.e., that achieve the highest score s(ξ)) on a model trainedwith Hopfield Boosting on the CIFAR-10 data set as the in-distribution data set. Figure 9 shows that within those 100
images, the Places 365 data set contains a non-negligible amount of data instances that show objects from semantic classes
contained in CIFAR-10 (e.g., horses, automobiles, dogs, trucks, and airplanes). We argue that data instances that clearlyshow objects of semantic classes contained in CIFAR-10 should be considered as in-distribution, which Hopfield Boostingcorrectly recognizes. Therefore, a certain amount of error can be anticipated on the Places 365 data set for all OOD detection
methods. We leave a closer evaluation of the amount of the anticipated error up to future work.
For comparison, Figure 10 shows the 100 images from Places 365 with the lowest score s(ξ), as evaluated by a model trainedwith Hopfield Boosting on CIFAR-10. There are no objects visible that have clear semantic overlap with the CIFAR-10classes.
31

Energy-based Hopfield Boosting for Out-of-Distribution Detection

Figure 9: The set of top-100 images from the Places 365 data set which Hopfield Boosting recognized as in-distribution.
The image captions show s(ξ) of the respective image below the caption.

32

Energy-based Hopfield Boosting for Out-of-Distribution Detection

Figure 10: The set of top-100 images from the Places 365 data set which Hopfield Boosting recognized as out-of-distribution.
The image captions show s(ξ) of the respective image below the caption.

H.9. Results on Noticeably Different Data Sets
The choice of additional data sets should not be driven by a desire to showcase good performance; rather, we suggest opting
for data that highlights weaknesses, as it holds the potential to drive investigations and uncover novel insights. Simple toy
data is preferable due to its typically clearer and more intuitive characteristics compared to complex natural image data. Inalignment with these considerations, the following data sets captivated our interest: iCartoonFace (Zheng et al., 2020), Four
Shapes (smeschke, 2018), and Retail Product Checkout (RPC) (Wei et al., 2022). In Figure 11, we show random samples
from these data sets to demonstrate the noticeable differences compared to CIFAR-10.
33

Energy-based Hopfield Boosting for Out-of-Distribution Detection

Table 8: Comparison between EBO-OE (Liu et al., 2020) and our version. ↓ indicates “lower is better” and ↑ indicates“higher is better”.

OOD Dataset
iCartoonFace
Four Shapes
RPC

Hopfield Boosting
FPR95 ↓ AUROC ↑EBO-OE
FPR95 ↓ AUROC ↑

0.60
40.81
4.07

4.01
62.55
18.51

99.57
90.53
98.65

98.94
75.34
96.10

Figure 11: Random samples from three data sets, each noticeably different from CIFAR-10. First row: iCartoonFace;
Second row: Four shapes; Third row: RPC.

In Table 8, we present some preliminary results using models trained with the respective method on CIFAR-10 as ID data set(as in Table 1). Results for comparison are presented for EBO-OE only, as time constraints prevented experimenting withadditional baseline methods. Although one would expect near-perfect results due to the evident disparities with CIFAR-10,
Four Shapes (smeschke, 2018) and RPC (Wei et al., 2022) seem to defy that expectation. Their results indicate a weakness
in the capability to identify outliers robustly since many samples are classified as inliers. Only iCartoonFace (Zheng et al.,2020) is correctly detected as OOD, at least to a large degree. Interestingly, the weakness uncovered by this data is present
in both methods, although more pronounced in EBO-OE. Therefore, we suspect that this specific behavior may be a general
weakness when training OOD detectors using OE, an aspect we plan to investigate further in our future work.
H.10. Runtime Considerations for Inference
When using Hopfield Boosting in inference, an additional inference step is needed to check whether a given sample is ID
or OOD. Namely, to obtain the score (Equation (9)) of a query sample ξ D , Hopfield Boosting computes the dot productsimilarity of the embedding obtained from ξ = ϕ(ξ D ) to all samples in the Hopfield memories X and O. In our
experiments, X contains the full in-distribution data set (50,000 samples) and O contains a subset of the AUX data setof equal size. We investigate the computational overhead of computing the dot-product similarity to 100,000 samples inrelation to the computational load of the encoder. For this, we feed 100 batches of size 1024 to an encoder (1) without
using the score and (2) with using the score, measure the runtimes per batch, and compute the mean and standard deviation.
We conduct this experiment with four different encoders on an NVIDIA Titan V GPU. The results are shown in Figure 12
and Table 9. One can see that, especially for larger models, the computational overhead of determining the score is very
moderate in comparison.
34

Energy-based Hopfield Boosting for Out-of-Distribution Detection

Figure 12: Mean inference runtimes for Hopfield Boosting on four different encoders on an NVIDIA Titan V GPU. We plot
the contributions to the total runtime of the encoder and the MHE-based score (Equation (9)) separately. The evaluationshows that the score computation adds a negligible amount of computational overhead to the total runtime.

Table 9: Inference runtimes for Hopfield Boosting with four different encoders on an NVIDIA Titan V GPU. We compare
the runtime of the encoder only and the runtime of the encoder with the MHE-based score computation (Equation (9))
combined.

Encoder

Time encoder (ms / batch)

ResNet-18
ResNet-34
ResNet-50
Densenet-100

Time encoder + score (ms / batch)

±0.24

±0.19

100.93209.80±0.40
360.93±1.51251.24±1.36108.50217.33±0.51
368.17±0.62258.82±0.84

Rel. overhead (%)
7.50
3.59
2.01
3.02

H.11. Compute Ressources
Our experiments were conducted on an internal cluster equipped with a variety of different GPU types (ranging from the
NVIDIA Titan V to the NVIDIA A100-SXM-80GB). For our experiments on ImageNet-1K, we additionally used resourcesof an external cluster that is equipped with NVIDIA A100-SXM-64GB GPUs.
For our experiments with Hopfield Boosting on CIFAR-10 and CIFAR-100, one run (100 epochs) of Hopfield Boosting
trained for about 8.0 hours on a single NVIDIA RTX 2080 Ti GPU and required 4.3 GB of VRAM. Fnding the hyperparameters required 160h of compute for CIFAR-10 and CIFAR-100, respectively. These were divided across four RTX 2080 Ti.
Estimating the standard deviation required 40 hours of compute on a single RTX 2080 Ti for CIFAR-10 and CIFAR-100
respectively.
For ImageNet-1K, one run (4 epochs) of Hopfield Boosting trained for about 4.4 hours on a single NVIDIA A-100SXM64GB GPU and required 26.9 GB of VRAM. Finding the optimal hyperparameters required a total of 86h of compute,
divided across 20 NVIDIA A-100-SXM64GB GPUs. Estimating the standard deviation required 22 hours of compute,
divided across 5 NVIDIA A-100-SXM64GB GPUs.
The amount of resources reported above cover the compute for obtaining the results of Hopfield Boosting reported in thepaper. The total amount of compute resources for the project is substantially higher. Notable additional compute expenses are
preliminary training runs during the development of Hopfield Boosting, and the training runs for tuning the hyperparametersand evaluating the results of the methods we compare Hopfield Boosting to.

35

Energy-based Hopfield Boosting for Out-of-Distribution Detection

H.12. Data Sets and Licenses
We provide a list of the data sets we used in our experiments and, where applicable, specify their licenses:• CIFAR-10 (Krizhevsky, 2009): License unknown• CIFAR-100 (Krizhevsky, 2009): License unknown• ImageNet-RC (Chrabaszcz et al., 2017): Custom License2• SVHN (Netzer et al., 2011): Creative Commons (CC)• Textures (Cimpoi et al., 2014): Custom License3• iSUN (Xu et al., 2015): License unknown• Places 365 (López-Cifuentes et al., 2020): License unknown• LSUN (Yu et al., 2015): License unknown• ImageNet-1K (Russakovsky et al., 2015): Custom License2• ImagetNet-21K (Ridnik et al., 2021): Custom License2• SUN (Isola et al., 2011): License unknown• iNaturalist (Van Horn et al., 2018): Custom License42

https://image-net.org/download.php
https://www.robots.ox.ac.uk/~vgg/data/dtd/index.html
4https://github.com/visipedia/inat_comp/tree/master/2017
336

Energy-based Hopfield Boosting for Out-of-Distribution Detection

Table 10: OOD detection performance on CIFAR-10. We compare results from Hopfield Boosting, PALM (Lu et al., 2024),
NPOS (Tao et al., 2023), SSD+ (Sehwag et al., 2021), ASH (Djurisic et al., 2023), GEN (Liu et al., 2023), EBO (Liu et al.,2020), MaxLogit (Hendrycks et al., 2019a), and MSP (Hendrycks & Gimpel, 2017) on ResNet-18. ↓ indicates “lower isbetter” and ↑ “higher is better”. All values in %. Standard deviations are estimated across five training runs.
SVHN
LSUN-Crop
LSUN-Resize
TexturesiSUN
Places 365
Mean

FPR95 ↓
AUROC ↑FPR95 ↓
AUROC ↑FPR95 ↓
AUROC ↑FPR95 ↓
AUROC ↑FPR95 ↓
AUROC ↑FPR95 ↓
AUROC ↑FPR95 ↓
AUROC ↑

Method type
Augmentations
Auxiliary outlier data

HB (ours)

PALM

NPOS

SSD+

ASH

GEN

EBO

MaxLogit

MSP

0.23±0.08
99.57±0.06
0.82±0.17
99.40±0.04
0.00±0.00
99.98±0.02
0.16±0.02
99.84±0.01
0.00±0.00
99.97±0.02
4.28±0.23
98.51±0.10

1.24±0.49
99.70±0.12
1.21±0.27
99.65±0.05
27.01±5.82
95.41±0.74
17.32±2.50
96.82±0.71
25.71±4.83
95.60±0.65
22.97±2.1794.95±0.53

9.04±1.13
98.37±0.23
5.52±0.50
98.97±0.04
26.85±3.1495.68±0.36
27.72±2.55
95.36±0.35
26.90±3.52
95.74±0.38
32.62±0.13
93.76±0.12

3.05±0.22
99.41±0.06
2.83±1.10
99.37±0.1634.30±2.1794.78±0.25
21.20±2.20
96.46±0.35
35.71±2.27
94.49±0.25
24.99±1.21
94.93±0.2225.17±9.55
94.86±2.09
13.13±1.81
97.33±0.36
38.18±5.78
90.39±2.00
46.08±6.2288.32±2.08
42.41±6.28
89.06±2.26
48.03±2.04
85.65±0.7733.26±5.99
93.53±1.4219.40±2.2296.48±0.46
31.50±3.92
94.04±0.84
44.62±4.1490.12±1.3235.85±4.05
93.05±0.84
45.82±1.07
88.68±0.28

32.10±6.41
93.43±1.60
17.25±2.30
96.73±0.46
30.69±4.03
94.02±0.86
44.67±4.46
89.61±1.50
34.99±4.33
92.99±0.90
44.87±1.1188.53±0.30

33.27±6.18
93.29±1.57
18.50±2.24
96.52±0.4731.64±4.01
93.90±0.86
44.97±4.4489.56±1.48
36.02±4.18
92.88±0.90
45.63±1.2688.42±0.29

49.41±3.77
92.48±0.9338.32±2.61
94.37±0.5345.82±3.48
92.84±0.80
55.04±2.8690.10±0.9249.10±3.06
91.99±0.74
57.58±0.97
88.06±0.25

0.92
99.55

15.91
97.02

21.44
96.31

20.35
96.57

35.50
90.94

35.07
92.65

34.09
92.55

35.00
92.43

49.21
91.64

OE
Weak
✓

Training
Strong✗Training
Strong✗Training
Strong✗Post-hoc
Weak✗

Post-hoc
Weak✗

Post-hoc
Weak✗

Post-hoc
Weak✗

Post-hoc
Weak✗

H.13. Non-OE Baselines
To confirm the prevailing notion that OE methods can improve the OOD detection capability in general, we compare
Hopfield Boosting to 3 training methods (Sehwag et al., 2021; Tao et al., 2023; Lu et al., 2024) and 5 post-hoc methods(Hendrycks & Gimpel, 2017; Hendrycks et al., 2019b; Liu et al., 2020, 2023; Djurisic et al., 2023). For all methods, wetrain a ResNet-18 on CIFAR-10. For Hopfield Boosting, we use the same training setup as described in section 4.2. For
the post-hoc methods, we do not use the auxiliary outlier data. For the training methods, we use the training procedures
described in the respective publications for 100 epochs. Notably, all training methods employ stronger augmentations thanthe OE or the post-hoc methods. The OE and post-hoc methods use the following augmentations (denoted as “Weak”):
1. RandomCrop (32x32), padding 4
2. RandomHorizontalFlip
The training methods use the following augmentations (denoted as “Strong”):
1. RandomResizedCrop (32x32), scale 0.2-1
2. RandomHorizontalFlip
3. ColorJitter applied with probability 0.8
4. RandomGrayscale applied with probability 0.2
Table 10 shows the results of the comparison of Hopfield Boosting to the post-hoc and training methods. Hopfield Boosting
is better at OOD detection than all non-OE baselines on CIFAR-10 in terms of both mean AUROC and mean FPR95 by alarge margin. Further, Hopfield Boosting achieves the best OOD detection on all OOD data sets in terms of FPR95 and
AUROC, except for SVHN and LSUN-Crop, where PALM (Lu et al., 2024) shows better AUROC results. An interesting
avenue for future work is to combine one of the non-OE based training methods with the OE method Hopfield Boosting.

37Energy-based Hopfield Boosting for Out-of-Distribution Detection

I. Informativeness of Sampling with High Boundary ScoresThis section adopts and expands the arguments of Ming et al. (2022) on sampling with high boundary scores.
We assume the extracted features of a trained deep neural network to approximately equal a Gaussian mixture model withequal class priors:11
N (ξ; µ, σ 2 I) + N (ξ; −µ, σ 2 I)
22pID (ξ) = p(ξ|ID) = N (ξ; µ, σ 2 I)p(ξ) =

2

pAUX (ξ) = p(ξ|AUX) = N (ξ; −µ, σ I)

(132)
(133)
(134)

Using the MHE and sufficient data from those distributions, we can estimate the densities p(ξ), p(ξ|ID) and p(ξ|AUX).
Lemma I.1. (see Lemma E.1 in Ming et al. (2022)) Assume the M sampled data points oi ∼ pAUX satisfy the following
constraint on high boundary scores Eb (ξ)−

PM

i=1 Eb (oi )

≤ϵ

(135)

|2µT oi | ≤ M ϵσ 2

(136)

M

Then they have

M
X
i=1

Proof. They first obtain the expression for Eb (ξ) under the Gaussian mixture model described above and can expressp(AUX|ξ) asp(ξ|AUX)p(AUX)p(ξ)
1
2 p(ξ|AUX)= 1
p(ξ|ID)+ 21 p(ξ|AUX)
2

(137)

p(AUX|ξ) =

=
=

(138)

(2πσ 2 )−d/2 exp(− 2σ1 2 ||ξ − µ||22 )2−d/2(2πσ )exp(− 2σ1 2 ||ξ + µ||22 ) + (2πβ −1 )−d/2 exp(− 2σ1 2 ||ξ − µ||22 )1
1 + exp(− 2σ1 2 (||ξ − µ||22 − ||ξ + µ||22 ))

(139)
(140)

1
When defining fAUX (ξ) = 2σ1 2 (||ξ − µ||22 − ||ξ + µ||22 ) such that p(AUX|ξ) = σ(fAUX (ξ)) = 1 + exp(−f, theyAUX (ξ))define Eb as follows:

Eb (ξ) = −|fAUX (ξ)|
1
= − 2 | ||ξ − µ||22 − ||ξ + µ||22 |2σ1= − 2 | ξ T ξ − 2µT ξ + µT µ − (ξ T ξ + 2µT ξ + µT µ)|
2σ|2µT ξ|
=−
σ2
38

(141)
(142)
(143)
(144)

Energy-based Hopfield Boosting for Out-of-Distribution Detection

Therefore, the constraint in Equation (136) is translated to

M
X
i=1

|2µT oi | ≤ M ϵσ 2

(145)

PM
As maxi∈M |µT oi | ≤ i=1 |µT oi | given a fixed M , the selected samples can be seen as generated from pAUX with theconstraint that all samples lie within the two hyperplanes in Equation (145).
Parameter estimation. Now they show the benefit of such constraint in controlling the sample complexity. Assume thesignal/noise ratio is large: ||µ||σ = r ≫ 1, and ϵ ≤ 1 is some constant.

Assume the classifier is given by

θ =

N
M
X
X
1
oi )
xi −
(
N + M i=1
i=1

(146)

where oi ∼ pAUX and xi ∼ pID . One can decompose θ. Assuming M = N :11θ =µ+ η+ ω
2
2
N
1 X
xi ) − µ
η =
(
N i=1

(147)
(148)

M

1 X
− oi ) − µ
(
N i=1

ω =

(149)

We would now like to determine the distributions of the random variables ||η||22 and µT η||η||22 =

√

d
X

ηi2

(150)

i=1

ηi ∼ N (0,

σ2
)
N

N
ηi ∼ N (0, 1)
σ
√
N 2
(
ηi ) ∼ χ21
σ

(151)
(152)
(153)

Therefore, for ||η||22 we have
d √
X
N
N 2
2
(
||η||2 =ηi ) ∼ χ2d
2
σ
σ
i=1

Now we would like to determine the distribution of µT η:
39

(154)

Energy-based Hopfield Boosting for Out-of-Distribution DetectionµT η =

d
X

(155)

µi ηi

i=1

µi ηi ∼ N (0,
d
X
i=1

d
X
i=1

µi ηi ∼ N (0,σ 2 µ2i)NdX
σ 2 µ2
i

i=1

N

(156)
)

(157)

d

µi ηi ∼ N (0,σ2 X 2µ )N i=1 iµT ησ2
∼ N (0, )
||µ||
N

(158)
(159)

Concentration bounds. They now develop concentration bounds for ||η||22 and µT η. First, we look at ||η||22 . A
concentration bound for χ2d is:√P(X − d ≥ 2 dx + 2x) ≤ exp(−x)(160)

By assuming x = 8σd 2 we obtainrddd+ 2 2 ) ≤ exp(− 2 )
8σ 2
8σ
8σ
d
d
d
P(X ≥ √ + 2 + d) ≤ exp(− 2 )
4σ
8σ
2σ
N
d
d
d
2
P( 2 ||η||2 ≥ √ + 2 + d) ≤ exp(− 2 )
σ
8σ
2σ 4σ
σ2 d
d
d
P(||η||22 ≥( √ + 2 + d)) ≤ exp(− 2 )
N
8σ
2σ 4σ
P(X − d ≥ 2

d

(161)
(162)
(163)
(164)

If d ≥ 2 we have that51dd√ + 2 >
4σ
σ
2σ

(165)

and thus, the above bound can be simplified when assuming d ≥ 2 as follows:
P(||η||22 ≥σ2 1d( + d)) ≤ exp(− 2 )
N σ
8σ

(166)

For ||ω||22 , since all oi is drawn i.i.d. from pAUX , under the constraint in Equation (145), the distribution of ω can be seen asa truncated distribution of η. Thus, with some finite positive constant c, we have

P(||ω||22 ≥5

Strictly, the bound is valid for d >

1
σ2
1
d
σ2
(d + )) ≤ cP(||η||22 ≥(d + )) ≤ c exp(− 2 )
N
σ
N
σ
8σ

√

2

40

(167)

Energy-based Hopfield Boosting for Out-of-Distribution Detection

Now, we develop a bound for µT η. A concentration bound for N (µ, σ 2 ) is
−t2
)
2σ 2

(168)

µT η
−t2 N
≥ t) ≤ exp(
)
||µ||
2σ 2

(169)

P(X − µ ≥ t) ≤ exp(
T

2

η
σ
By applying µ||µ|| ∼ N (0, N ) to the above bound we obtain

P(

Assuming t = (σ||µ||)1/2 we obtain−(σ||µ||)NµT η≥ (σ||µ||)1/2 ) ≤ exp(
)
||µ||
2σ 2
−||µ||NµT η≥ (σ||µ||)1/2 ) ≤ exp(
)
P(
||µ||
2σ
P(

(170)
(171)

Due to symmetry, we have−||µ||NµT η≤ −(σ||µ||)1/2 ) ≤ exp(
)
||µ||
2σ
µT η
µT η
−||µ||NP(−≤ −(σ||µ||)1/2 ) + P(
≥ (σ||µ||)1/2 ) ≤ 2 exp(
)
||µ||
||µ||
2σ
P(−

(172)
(173)

We can rewrite the above bound using the absolute value function.

P(−||µ||N|µT η|≥ (σ||µ||)1/2 ) ≤ 2 exp(
)
||µ||
2σ

(174)

Benefit of high boundary scores. We will now show why sampling with high boundary scores is beneficial. Recall the
results from Equations (145) and (149):

M
X
i=1

|2µT oi | ≤ M ϵσ 2
ω =

M
X
1
(−
oi ) − µ
M
i=1

(175)
(176)

The triangle inequality is|a + b| ≤ |a| + |b||a + (−b)| ≤ |a| + |b|
41

(177)
(178)

Energy-based Hopfield Boosting for Out-of-Distribution Detection

Using the two facts above and the triangle inequality we can bound |µT ω|:
M

1 X T
σ2 ϵ
µ oi | ≤
|
M i=1
2

(179)

M
X
1
σ2 ϵ
|−
µT oi | ≤
M
2
i=1

(180)

M
X
1
σ2 ϵ
µT oi | + ||µ||22 ≤|−+ ||µ||22
M
2
i=1
M
X
σ2 ϵ
1
|−
+ ||µ||22µT oi − µT µ| ≤M
2i=1|µT ω| ≤ ||µ||22 +Developing a lower bound.

σ2 ϵ
2

(181)
(182)
(183)

Let
1
σ2
(d + )
Nσσ2
1||ω||22 ≤(d + )Nσ|µT η|≤ (σ||µ||)1/2||µ||||η||22 ≤

(184)
(185)
(186)

hold simultaneously. The probability of this happening can be bounded as follows: We define T and its complement T̄ :1
σ2
1|µT η|
σ2(d + )} ∩ {||ω||22 ≤(d + )} ∩ {≤ (σ||µ||)1/2 }
N
σ
N
σ
||µ||
1
σ2
1
|µT η|
σ2(d + )} ∪ {||ω||22 >(d + )} ∪ {> (σ||µ||)1/2 }T̄ = {||η||22 >
N
σ
N
σ
||µ||
T = {||η||22 ≤

(187)
(188)

With P(T ) + P(T̄ ) = 1. The probability P(T̄ ) can be bounded using Boole’s inequality and the results in Equations (166),(167) and (174):

P(T̄ ) ≤ exp(−d/8σ 2 ) + c exp(−d/8σ 2 ) + 2 exp(
P(T̄ ) ≤ (1 + c) exp(−d/8σ 2 ) + 2 exp(−||µ||N)
2σ−||µ||N
)
2σ

(189)
(190)

Further, we can bound the probability P(T ):−||µ||N)
2σ−||µ||N)
1 − P(T ) ≤ (1 + c) exp(−d/8σ 2 ) + 2 exp(
2σ−||µ||N)
P(T ) ≥ 1 − (1 + c) exp(−d/8σ 2 ) − 2 exp(
2σP(T̄ ) ≤ (1 + c) exp(−d/8σ 2 ) + 2 exp(

42

(191)
(192)
(193)

Energy-based Hopfield Boosting for Out-of-Distribution Detection

Therefore, the probability of the assumptions in Equations (184), (185), and (186) occuring simultneously is at least
1 − (1 + c) exp(−d/8σ 2 ) − 2 exp( −||µ||N).
2σ
By using the triangle inequality, Equation (147) and the Assumptions (184) and (185) they can bound ||θ||22 :11
η + ω||22
2
2
1
1
2
2
||θ||2 ≤ ||µ||2 + || η||22 + || ω||22
2
2
1
1
2
2
2
||θ||2 ≤ ||µ||2 + ||η||2 + ||ω||22
4
4
1
1 σ2
2
2
(d + )||θ||2 ≤ ||µ||2 +
2N
σ
2
1
σ
(d + )||θ||22 ≤ ||µ||22 +Nσ||θ||22 = || µ +

(194)
(195)
(196)
(197)
(198)

The reverse triangle inequality is defined as|x − y| ≥ |x| − |y||x − (−y)| ≥ |x| − |y|

(199)
(200)

Using the reverse triangle inequality, Equations (147), (183) and Assumption (186) we have that1 T
1µ η + µT ω|
2
2
1
1 T
T
T
|µ θ| ≥ |µ µ| − | µ η| − | µT ω|
2
2
1 1/2
1
1 σ2 ϵ2
3/2
T− ||µ||22 −|µ θ| ≥ ||µ||2 − σ ||µ||
2
2
2 2
2
1
1
1
σ
ϵ
|µT θ| ≥ ||µ||22 − σ 1/2 ||µ||3/2 −
2
2
2 2
2
σ
ϵ
1
)
|µT θ| ≥ (||µ||22 − σ 1/2 ||µ||3/2 −22|µT θ| = |µT µ +

(201)
(202)
(203)
(204)
(205)

They have assumed that the signal/noise ratio is large: ||µ||σ = r ≫ 1. Thus, we can drop the absolute value, because we
assume that the term inside the || is larger than zero:
1
1||µ||22 ϵ(||µ||22 − ||µ||1/2 ||µ||3/2 −
)
2
r
2r2
ϵ 1
1
|µT θ| ≥ (1 − − 2 ) (||µ||22 )r
2r 2|µT θ| ≥

(206)
(207)

We have(1 −1ϵ− 2) ≥ 0r
2r

if r ≥ 1.36602540378443 . . . and ϵ ≤ 1, and therefore
43

(208)

Energy-based Hopfield Boosting for Out-of-Distribution Detection|µT θ| ≥1σ2 ϵ(||µ||22 − σ 1/2 ||µ||3/2 −
)
2
2

(209)

Because of Equation (198) and the fact that if x ≤ y and sgn (x) = sgn (y) then x−1 ≥ y −1 we have
1
1
≥q
2
||θ||
||µ||2 + σ (d + 1 )
2

N

(210)

σ

We can combine the Equations (209) and (210) to give a single bound:2||µ||22 − σ 1/2 ||µ||3/2 − σ2 ϵ|µT θ|q≥2||θ||2 ||µ||2 + σ (d + 1 )
2

N

(211)

σ

we define θ such that µT θ > 0 and thus2||µ||22 − σ 1/2 ||µ||3/2 − σ2 ϵ
µT θ
q
≥
2
||θ||
2 ||µ||2 + σ (d + 1 )
2

N

(212)

σ

The false negative rate FNR(θ) and false positive rate FPR(θ) are

FNR(θ) =FPR(θ) =

Z 0

−∞
Z ∞
0

N (x;

N (x;

µT θ 2, σ ) dx||θ||−µT θ 2, σ ) dx
||θ||

(213)
(214)

As N (x; µ, σ 2 ) = N (−x; −µ, σ 2 ), we have FNR(θ) = FPR(θ). From Equation (212) we can see that as ϵ decreases, theTlower bound of µ||θ||θ will increase. Thus, the mean of the Gaussian distribution in Equation (213) will increase and therefore,
the false negative rate will decrease, which shows the benefit of sampling with high boundary scores. This completes theextended proof adapted from (Ming et al., 2022).

44