Energy-based Hopfield Boosting for Out-of-Distribution Detection
Authors: Claus Hofmann
Simon Lucas Schmid
Bernhard Lehner
Daniel Klotz
Sepp Hochreiter
Venue: NeurIPS 2024 poster (2024)
Keywords: ood
out-of-distribution
boosting
deep learning
outlier exposure
Primary area: safety_in_machine_learning

Abstract: Out-of-distribution (OOD) detection is critical when deploying machine learning models in the real world. Outlier exposure methods, which incorporate auxiliary outlier data in the training process, can drastically improve OOD detection performance compared to approaches without advanced training strategies. We introduce Hopfield Boosting, a boosting approach, which leverages modern Hopfield energy to sharpen the decision boundary between the in-distribution and OOD data. Hopfield Boosting encourages the model to focus on hard-to-distinguish auxiliary outlier examples that lie close to the decision boundary between in-distribution and auxiliary outlier data. Our method achieves a new state-of-the-art in OOD detection with outlier exposure, improving the FPR95 from 2.28 to 0.92 on CIFAR-10, from 11.76 to 7.94 on CIFAR-100, and from 50.74 to 36.60 on ImageNet-1K.

Energy-based Hopfield Boosting for Out-of-Distribution Detection
Claus Hofmann
Simon Schmid
Bernhard Lehner
Daniel Klotz
Sepp Hochreiter
Institute for Machine Learning. JKU LIT SAL IWS Lab, Johannes Kepler University, Linz, Austria Software Competence Center Hagenberg GmbH, Austria Silicon Austria I Labs, JKU LIT SAL IWSS Lab, Linz, Austria
Department of Computational Hydrosystems, Helmholtz Centre for Environmental Research- Leipzig, Germany
hofmann@ml, j.k...ml...... j.k.jkutt
Abstract
cut-of-distribution (OOD) detection is critical when deploying machine learning models in the real world. Outlier exposure methods, which incorporate auxiliary outlier data in the training process, can drastically improve OOD detection perfor- mance compared approaches without advanced training strategies. We introduce Hopfield Boosting, a boosting approach, which leverages modern Hopfield en- ergy to sharpen the decision boundary between the in-distribution and OOD data. Hopfield Boosting encourages the model to focus on hard-to-distingee outlier examples that lie close the decision boundary between in-distribution and auxiliary outlier data. Our method achieves a new state-of-the of-te-of-hh-art in OOD detection with outlier exposure, improving the FPR95 from 2.28 to 0.92 on CIFAR-1O, from 11.76 to 7. 7.94 on CIFAR-100, and from 50.74 to 36.60 on ImageNet-IK.
auxiliary
Introduction
Out-of-distribution (OOD) detection is crucial when using machine learning systems in the real world (Ruff et al., 2021; Yang et al., 2021; Liu et al., 2021). Deployed models will - sooner or later encounter inputs that deviate from the training distribution. For example, system trained to recognize music genres might also hear a sound clip construction site noise. In the best case, naive deployment can then result in overly confident predictions. In the worst case, will get erratic model behavior and completely wrong predictions (Hendrycks & Gimpel, 2017). The purpose of OOD detection is classify these inputs OOD, such that the system can then, for instance, notify users that no prediction is possible. In this paper we propose Hopfield Boosting, novel OOD detection method that leverages the energy component modern Hopfield networks (MHNs; Ramsauer et al., 2021) and advances the state-of-the-ard of OOD detection. This energy represents measure dissimilarity between a data instances X and query instance &. therefore natural fit for doing OOD detection (as shown in Zhang et al., 2023a).
auxiliary outlier data set (AUX) boost the model's OOD detection capacity. This allows the training process to learn a boundary around the in-distribution (ID) data, improving the performance the OOD detection task. In summary, our contributions follows:
Hopfield Boosting
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

weight
evaluate
AUX
ID
Neural Network
update
Figure 1: The Hopfield Boosting concept. The first step (weight) creates weak learners by firstly choosing in-distribution samples (ID, orange), and by secondly choosing auxiliary outlier samples (AUX, blue) according their assigned probabilities; the second step (evaluate) computes the losses for the resulting predictions (Section 3); and the third step (update) assigns new probabilities to the AUX samples according to their position on the hypersphere (see Figure 2).
1. We propose Hopfield Boosting, OOD detection approach that samples weak learners by using the MHE (Ramsauer et al., 2021).
2. Hopfield Boosting achieves a new state-of-the-art in OOD detection. It improves the average false positive rate 95% true positives (FPR95) from 2.28 to 0.92 on CIFAR-10, from 11 1.38 to 7.94 on CIFAR-100, and from 50.74 to 36.60 on ImageNet-IK.
3. We provide theoretical background that motivates Hopfield Boosting for OOD detection.
Related Work
Some authors (e.g., Bishop. 1994; Roth et al., 2022; Yang et al., 2022) distinguish between anomalies, outliers, and novelties. These distinctions reflect different goals within applications (Ruff al., 2021). For example, when an anomaly is found, it will usually be removed from the training pipeline. However, when novelty found should be studied. We focus on the detection of samples that are not part of the training distribution and consider sample categorization a downstream task.
Post-hoc OOD detection. A common and straightforward OOD detection approach is to use post-hoe strategy, where one employs statistics obtained from classifier. The perhaps most well known and simplest approach in this class is the Maximum Softmax Probability (MSP; Hendrycks Gimpel, 2017), where one utilizes p( y æ the most likely class y given a feature vector R2D to estimate whether sample is OOD. Despite good empirical performances, this view is intrinsically limited, since OOD detection should focus on p(æ) (Morteza & Li, 2022), A wide range of post-hoc OOD detection approaches have been proposed to address the shortcomings of MSP (e.g.. Lee et al., 2018; Hendrycks et al., 2019a; Liu et al., 2020; Sun al., 2021, 2022: Wang al., 2022: Zhang et al., 2023a; Djurisic et al., 2023; Liu et al., 2023; Xu et al., 2024). Most related to Hopfield Boosting is work of Zhang al. (2023a) - to our knowledge, they are the first to apply the MHE to OOD detection. Specifically, they use the ID data set to produce stored patterns and then use modified version of MHE as the OOD score. While post-hoe approaches can deployed out of the box on any model, crucial limitation is that their performance heavily depends the employed model itself.

---

Training methods. In contrast to post-hoe strategies, training- based methods modify the training process to improve the model's OOD detection capability (e.g.. Hendrycks et al., 2019c; Tack et al.. 2020; Sehwag et al., 2021; Du et al., 2022; Hendrycks et al., 2022; Wei et al., 2022a; Ming et al., 2023; Tao et al., 2023; Lu et al., 2024). For example, Seff-Supereied Outlier Detection (SSD; Sehwag et al., 2021) leverages contrastive self-supervised learning to train model for OOD detection.
Auxiliary outlier data and outlier exposure. A third group of OOD detection approaches are outlier exposure (OE) methods. Like Hopfield Boosting, they incorporate AUX data in the training process to improve the detection of OOD samples (e.g. Hendrycks et al., 2019b; Liu et al., 2020; Ming et al., 2022; Zhang et al., 2023b; Wang et al., 2023a; Zhu et al., 2023; Jiang et al., 2024). We provide more detailed discussions on a range of OE methods in Appendix C.1. As far as we know, all OE approaches optimize an objective (CoOD), which aims at improving the model's '' discriminative power between ID and OOD data using the AUX data set as a stand-in for the OOD case. Hendrycks et al. (2019b) were the first to the term OE to describe a more restrictive OE concept. Since their approach uses the MSP for incorporating the AUX data we refer to it as MSP-OE. Further, we refer to the OE approach introduced in Liu al. (2020) as EBO-OE (to differentiate from EBO, their post-hoc approach). In general, OE methods conceptualize the AUX data set as large and diverse data (e.g., ImageNet for vision tasks). As a consequence, usually, only small subset of the samples bear semantic similarity to the ID data set most data points are easily distinguishable from the ID data. Recent approaches therefore actively try find informative samples for the training. The aim is to refine the decision boundary, ensuring the ID data is more tightly encapsulated (e.g.. Chen et al.. 2021; Ming et al., 2022). For example, Posterior Sampling-basee Ming et al., 2022) selects samples close to the decision boundary using Thompson sampling: They first sample linear decision boundary between ID and AUX data and then select those data instances which are closest to the sampled decision boundary. Hopfield Boosting also makes use of samples close to the boundary by giving them higher weights for the boosting step.
Outlier Mining (POEM;
Continuous modern Hopfield networks. MHNs are energy-based associative memory networks. They advance conventional Hopfield networks (Hopfield, 1984) by introducing continuous queries and states with the MHE as new energy function. MHE leads to exponential storage capacity, while retrieval is possible with a one-step update (Ramsauer et al., 2021). The update rule of MHNs coincides with attention as it is used in the Transformer (Vaswani et al., 2017). Examples for successful applications of MHNs are Widrich et al. (2020); Fürst et al. (2022); Sanchez-Fernandez et al. (2022); Paischer et al. (2022); Schäfl et al. (2022); Schimunek et al. (2023) and Auer al. (2023). Section 3.2 gives an introduction to MHE for OOD detection. For further details on MHNs, we refer to Appendix A.
Boosting for classification. Boosting, in particular, AdaBoost (Freund & Schapire, 1995), is an ensemble learning technique for classification. It is designed to focus ensemble members toward data instances that are hard to classify by assigning them higher weights. These challenging instances often lie near the maximum margin hyperplane (Rätsch et al., 2001). akin to support vectors in support vector machines (SVMs; Cortes & Vapnik, 1995). Popular boosting methods include Gradient boosting (Breiman, 1997), LogitBoost (Friedman et al., 2000), and I.Boost (Demiriz et al., 2002).
Radial basis function networks. Radial basis function networks (RBF networks; Moody & Darken, 1989) are function approximators of the form
N
| - p4 20²
((E)= = Σ
Wi exp
1=1
where are linear weights, L4; are the component means and of are the component variances. RBF networks can be described as a weighted linear superposition of N radial basis functions and have previously been used as hypotheses for boosting (Rätsch et al., 2001). If the linear weights are strictly positive, RBF networks can be viewed as an unnormalized weighted mixture of Gaussian distributions p)(E) (V( μ4; of I) with = {I,.... NY. Appendix H.1 explores the connection between RBF networks and MHNs via Gaussian mixtures in more depth. We refer to Bishop (1995) and Müller et al. (1997) for more general information on RBF networks.
3

---

3 Method
This section presents Hopfield Boosting: First, formalize the OOD detection task. Second, give an overview of the MHE and why it is suitable for OOD detection. Finally, we introduce the AUX-based boosting framework. Figure I shows summary of the Hopfield Boosting concept.
3.1 Classification and OOD Detection
Consider multi-class classification task denoted as (X Y )), where X P represents set of N D-dimensional feature vectors (a 222 ................................ RR), which are i..d. samples x; PID- Y E yN denotes the labels associated with these feature vectors, and ) is a set containing possible classes (|| K signifies the number of distinct classes), We consider observations 5d RD that deviate considerably from the data generation (I that defines the "normality" of our data as OOD. Following Ruff al. (2021), an observation OOD if it pertains the
O = E RRP PID < e) where 2 0.
(2)
Since the probability density of the data generation Pro is in general not known, needs to estimate ()D(5"), In practice, is common to define an outlier score s(E) that uses an encoder d, where & = 6($2). The outlier score should in the best case preserve the density ranking. In contrast to a density estimation, the score s{) does not have to fulfill requirements a probability density (like proper normalization non-negativitt). Given s(E) and é, OOD detection can be formulated as a binary classification task with the classes ID and OOD:
[ID
if (()(() 27
B(EP,)
OOD s( <9
is common to choose the threshold y so that a portion of 95% of ID samples from a previously unseen validation set are correctly classified as ID. However, metrics like the area under the receiver operating characteristic (AUROC) can be directly computed on s(E) without specifying y since the AUROC computation sweeps over the threshold.
3.2
Modern Hopfield Energy
The log-sum-exponential (lse) function is defined as
Ise( (0, z) 3-1 log Σ exp(32z)
where 3 is the inverse temperature and 2 E R N is a vector. The Ise can be seen soft approximation to the maximum function: As B + the Ise approaches max; Ri.
a.) arranged in data matrix X, and Given set of N d-dimensional stored patterns (æ1, 22, ........... ,., d-dimensional query &, the MHE is defined as
E(E: X) = - 1ss xx 2 5" + C,
(5)
where B- N + M and .M the largest norm of pattern: M max; ||>:]]. X is also called the memory of the MHN. Intuitively, Equation (5) can be explained as follows: The dot-product within the Ise computes a similarity for given & E IRd to all patterns in the memory X E dx The Ise function aggregates the similarities to form single value, where the 3 parameterizes the aggregation operation: If oo, the maximum similarity of & to the patterns X returned.
To use the MHE for OOD detection, Hopfield Boosting acquires the memory patterns X by feeding xx R) ) of the IDD data arranged in the data matrix X DDD to an encoder 1 - Rdd (e.g., ResNet): I, d(x?). We denote the component wise application of & to the patterns in X as X
raw data instances (z:P. .
( Similarly, a raw query gD E RD is fed through the encoder to obtain the query pattern: & = di( P) One can now use E(g; X) to estimate whether ID or OOD: A low energy indicates E is ID. and a high energy signifies that is OOD.

---

3.3 Boosting Framework
Sampling of informative outlier data. Hopfield Boosting uses AUX data to learn a decision boundary between the ID and OOD region during the training. Similar to Chen et al. (2021) and Ming et al. (2022), Hopfield Boosting selects informative outliers close to the ID-OOD decision boundary. For this selection, Hopfield Boosting weights the AUX data similar to AdaBoost (Freund & Schapire, 1995) by sampling data instances close to the decision boundary more frequently. We consider samples close to the decision boundary as weak learners their nearest neighbors consist of samples from their own class as well as from the foreign class. An individual weak learner represents classifier that is only slightly better than random guessing (Figure 6). Vice versa, a strong learner can be created by forming an ensemble of a set of weak learners (Figure 2).
We denote the matrix containing the raw AUX data instances (op of ......ON) as O E R PxM and the memory containing the encoded AUX patterns as O = p(0)) The boosting process proceeds follows: There exists a weight (W1, W2, WN) for each data point in O and the individual weights are aggregated into the weight vector wt Hopfield Boosting uses these weights to draw mini-batches O from OD for training, where weak learners are sampled more frequently.
We introduce an MHEEbased energy function which Hopfield Boosting uses to determine how weak specific learner & is (with higher energy indicating a weaker learner):
EE X, = --2 1se(3, (X E) + 1se(3, (T + (se(B,0"s) (TTE).
E),
(6)
where X E R contains ID patterns, O E Rdx. contains AUX patterns, and (X || O) E (N++M denotes the concatenated data matrix containing the patterns from both X and O. Before computing Eg, we normalize the feature vectors in X. O, and & to unit length. Figure 3 displays the energy landscape of (E: X,O) O) using exemplary data on 3-dimensional sphere. F., is maximal at the decision boundary between II) and AUX data and decreases with increasing distance from the decision boundary in both directions.
As we show in our theoretical discussion in Appendix G, when modeling the class-conditional densities of the ID and AUX data set as mixtures of Gaussian distributions
N
|
ID = N
Σ
(7)
i=1
N
p( E AUX)
N Σ
N(E: (,,:0,,
(8)
with equal class priors p(ID) P(UX)) 1/2 and normalized patterns ||>æ|| = and ||o,|| = 1.
we obtain (5: X, 0) - 9-1 log(p( ID ) ) AUX }), where S denotes equality up to an irrelevant additive constant. The exponential F., is the variance of Bernoulli random variable with the outcomes {ID, AUX) conditioned on &. Thus, according to Fg, the weak learners are situated at locations where the model defined in Equations (7) and (8) is uncertain.
Given a set of query values (EI, E2...., Ex. assembled in query matrix E E Rddx", we denote vector of energies e E R with e; = Eb X) as
e Fu, Ex,
To calculate the weights w.+1, we use the memory of AUX patterns as a query matrix = and compute the respective energies E, of those patterns. The resulting energy vector E,, X, 0) is then normalized by a softmax. This computation provides the updated weights:
Wi+1 =
(10)
Appendix J provides theoretical background on how informative samples close to the decision boundary are beneficial for training OOD detector.
5

---

Overview
(a) All Data
(b) Random Sampling
(c) Hopfield Boosting
2.0
2.0
2.0
inlinr
oullier
-10
-1.5
-1.5
2 -1.5
0
Figure 2: Synthetic example of the adaptive resampling mechanism. Hopfield Boosting forms strong learner by sampling and combining set of weak learners close to the decision boundary. The heatmap on the background shows (xp( ()), where 60. Only the sampled (i.e., highlighted) points serve memories X and O.
Training the model with MHE. In this section, introduce how Hopfield Boosting the sampled weak learners to improve the detection of patterns outside the training distribution. We follow the established training method for OE (Hendrycks al., 2019b; Liu et al., 2020; Ming et al., 2022): Train classifier on the ID data using the standard cross-entropy loss and add an OOD loss sharpen the decision boundary between the ID and OOD regions.
that uses the AUX data Formally, this yields the loss
C LCE + XLOOD.
(11)
where À is a hyperparamter indicating the relative importance of LOOD. Hopfield Boosting explicitly minimizes Eg (which is also the energy function Hopfield Boosting uses to sample weak learners). Given the weight vector w;, and the data sets X and OP, we obtain a mini-batch X containing samples from X by uniform sampling, and a mini-batch of N weak learners O from O by sampling according to with replacement. We then feed the respective mini-batches into the neural network Prws to create latent feature (in our experiments, we always use the feature of the penultimate layer of a ResNet). Our proposed approach then uses two heads:
A linear classification head that maps the latent feature to the class logits for LCE.
2. A 2-layer MIP Prooj maps the features from the penultimate layer the output for LOOD-
Hopfield Boosting computes LOOD on the representations it obtains from Φ Oproj Obase as follows:
LOOD
Es EE, O.),
(12)
2N
where the memories X, and O, contain the encodings of the sampled data instances: X, o( 6(
and O, = P...). The sum is taken over the observations &. which are drawn from (X, O.). Hopfield Boosting computes LOOD for each mini-batch by first calculating the pairwise similarity matrix between the patterns in the mini-batch, followed by determining the Eg, values of the individual observations &. and, finally a mean reduction. To the best of our knowledge, Hopfield Boosting the first method that uses Hopfield networks in this way to train a deep neural network. We note that there is relation between Hopfield Boosting and SVMs with an RBF kernel (see Appendix H.3). However, the optimization procedure of SVMs is in general not differentiable. In contrast, our novel energy function fully differentiable. This allows us to use it to train neural networks.
Summary. Algorithm provides an outline of Hopfield Boosting. Each iteration consists of three main steps: 1. weight, 2. evaluate, and 3. update. First, Hopfield Boosting samples mini-batch from the ID data and weights the AUX data by sampling mini-batch according to w;. Second, Hopfield Boosting evaluates the composite loss the sampled mini-batch Third, Hopfield Boosting updates the model parameters and, every N-th step, also the sampling weights for the AUX data set w.+1-
Inference.
At inference time, the OOD score s(E) is
s(E)
1so(3, 3, X {) - 1sc(B, O²
(13)

---

Algorithm 1 Hopfield Boosting
Require: N. X, O, Y. LeF. Es, 3
Set all weights w1 to 1/|O|
for = 1 to do
1. Weight. Get hypothesis X, - AUX::
L.a. Mini-batch sampling X, from X, and
I.b. Sub-sampling of weak learners O, from O according to the weighting wt.
2. Evaluate. Compute loss from Equation (11) on X, and Os-
3. Update. Update model for the next iteration:
3.a. At every step, update the full model (backbone, classification head, and MHE)
3.b. At every I * N step calculate new weights for O with W = softmax( 0))
end for
For computing s()), Hopfield Boosting uses the 50,000 random samples from the ID and AUX data sets, respectively. As we show in Appendix L8, this step entails only very moderate computational overhead in relation to a complete forward pass (c.g., an overhead of 7.5% for ResNet-18 on an NVIDIA Titan V GPU with 50,000 patterns stored in each of the memories X and O). We additionally experimented with using only 1se xT as a score, which also gives reasonable results. However, the approach in Equation (13) has turned out to be superior. Equation (13) uses information from both ID and AUX samples. This can, for example, be beneficial for handling query patterns & that are dissimilar from both the memory patterns in X well from the memory patterns in O.
3.4 Comparison of Hopfield Boosting to HE and SHE
Zhang al. (2023a) propose two post-hoc methods for OOD detection with Hoppiio Energy" (HE) and "Simppified Hopfield Energy" (SHE). In contrast to Hopfield Boosting, HE and SHE do not use AUX data to get better boundary between ID and OOD data. Rather, their methods evaluate the MHE on ID patterns only to determine whether a sample is ID or OOD. Additional differences include the selection of patterns stored in the memory or the normalization of the patterns. The OE process of Hopfield Boosting drastically improves the OOD detection performance compared to HE and SHE. We verify that the unique contributions of Hopfield Boosting (like the energy-based loss and the boosting process) are responsible for the superior performance with two extensions of HE that include AUX data (the comparison can be found in Appendix 1.9). For further information on the differences to HE and SHE, we refer to Appendix H.4.
4 Experiments
4.1 Toy Example
This section presents toy example illustrating the main intuitions behind Hopfield Boosting, For the sake of clarity, the toy example does not consider the inlier classification task that would induce secondary processes, which would obscure the explanations. Formally, we do not consider the first term on the right-hand side of Equation (11). For further toy examples, refer to Appendix F.
Figure 2 demonstrates how the weighting in Hopfield Boosting allows good estimations of the decision boundary, even if Hopfield Boosting only samples small number of weak learners. This is advantageous because the AUX data set contains large number of data instances that are uninforma- tive for the OOD detection task. For small, low dimensional data, one can always use all the data to compute E, (Figure 2, a). For large problems (like in Ming et al., 2022), this strategy difficult, and the naive solution of uniformly sampling N data points would also not work. This will yield many uninformative points (Figure 2, b). When using Hopfield Boosting and sampling N weak learners according to w., the result better approximates the decision boundary of the full data (Figure 2, c).
7

---

Table 1: OOD detection performance on CIFAR-10. We compare results from Hopfield Boosting, DOS (Jiang et al., 2024), DOE (Wang et al., 2023b), DivOE (Zhu et al., 2023), DAL (Wang et al., 2023a), MixOE (Zhang et al., 2023b), POEM (Ming et al., 2022), EBO-OE (Liu et al., 2020), and MSP-OE (Hendrycks et al., 2019b) on ResNer-18. & indicates "lower is better" and 1 "higher is better". All values in %. Standard deviations are estimated across five training runs.
Metric
(ound)
DOS
DOE
DivOE
DAL
MixOE
POEM
EBO-OE
MSP-OE
FPR95
0 23
3,08
1,0750054
6,213084
1,25
27.54
L,ggaas
4.31
SVHN
AUROCT 1
90 90.57::uu
98. 18.15
90 00.60=9-13
96. 53
99.4 99.61
95.377114
90.35am 15
90 15ee2ss
90. 90.20
LSUN-Crop
FPR05
ALROCT
0.825111
99.40290
98.0 08.0458
3.66
99.3925
3,2250
18840
18.13-400
4.17
0. 0.14-007
99 09.63-10 29 11
98,89 15
4.0250 o1
6,229 TC
58.4356 t0
98.8355
7:02
0.00
0.00 0.00
0.002000
0.00
0.36
0.00
0.00 0.00
0.00 1.000m0.000 -9.00
LSUN- Resizee
AUROC1
99. 19.9g
90. ao.g.i...
100
99 2g
99.90
99.88
99.98
99
Tettures
FPR95
AUROC1
89,84 84
0. 0.:69.82
90 63-00044
1 22693.39
00.55444122
2,75
99. 00.5048000
1.2220:11
0.96 0.9699523
00. 70:70:
98 98.01
4 68 22.2
90. 90.724m06
0,00
00 61
1.11
2. 2,20-98.6
06: 08.5778048
0.00:1..00
0.00 0.00
0.00%.o0 10.00..0
0.00
0.00 0.00
0.17
0.00
0.00
0.00 10.00
ISUN
AUROC1
00 17777
90. 00..0g
100 00:+0.00
99 sg-
90 00.03-3.04
87
00.8729-12
90 90.98=0.00
90.
FPROS
4 4.26:9
12.2 12.26
10.7242
13 70734.00
14.22 14.22=1::
16.3043.00
7.70
11: 11.777
21. 21.1125.45
Places 365
AUROC1 1
98.51
96.60 00.63=9.44
25,0g40
96 06.05-49.0
06.77=900
96,9241023
97. 97.5649 30
96.30 96.30=90.0 30
96.88 06.91
Mcan
FPR95
AUROC1
0.92
3.38
98.07
98
4.81
3.83
99.06
08.18
3.40
8.17
2,28
3.73
5.84
Accuracy
94,0240000
94. 94.7455M
94.0g4n
94. 7240
96. 11
96 00.0001..0 90
89.20 10
01 325000000
94.838971
Table 2: OOD detection performance on ImageNet- IK. We compare results from Hopfield Boosting, DOS (Jiang et al., 2024), DOE ( (Wang et al., 2023b), DivOE (Zhu et al., 2023), DAL (Wang et al., 2023a), MixOE (Zhang et al., 2023b), POEM (Ming et al., 2022). EBO-OE (Liu et al., 2020), and MSP-OE (Hendrycks et al., 2019b) on ResNet-50. & indicates "lower is better" and "higher is better". All values in %. Standard deviations are estimated across five training runs.
Metric
HB (ours)
DOS
DOB
Dis08
DAL
MixOE
POEM
EBO-DE
MSP-OE
FRR95
41.5g
20. 21.2ga
83,
83.83a7.4
42 201000.74
43 88.88
11.05=491
SL. 26
29 29:677122
08.35
Textures
AUROC1
50.0
$0.86-0.1s
24
18
87.30=
$8.51 2.
92 22-004 14
SUN
FFR95
1
91. 91.24
1..4
$4,30=0
50.29 10.00
72.95
69.73 18.75
83 83.64450-30
61.00
65.91 85.0
$1.47=00000
65.14
$2,20
86 86.3840
57.46
57.69-116
85.833000
81.45-100
66.01 an:3r
FPRO5
63.31 53.31
60.72 8.00
86.39
71 h0:00
74.4645-5
71.34=
68 45
74.
AUROCA
87. 10
$1.02:50.2
70.37
80.35 tr..3
78.7222000 08
50.31:9
81. 11.77929.0 as
$1,3520
8g:00 19
INnturalist
FPR95
AUROC
95.65 10.20
11.11 11.11
49.55#1441
90-10-40.58
70. 70..88
63.62
30. 553.4153.42
93.81 $0.50
51.02=0074
47,28
90.19:40.35
46. 16.3771179
92.01
91,41
40.02*s
51, 51-73-1355
88.51
Mean
AUROC1
91:00
80.57
54.71
81.17
72.81
86.49
$3.98
55.30
56.20
50.
87.85
87.75
60.17
83.78
ID
Accuracy
70.30
70.0g
61.3616.4
74.80
75.00
75.07:0020
75.66 75.6649.03
75.6g40.0
75.71
4.2 Data & Setup
CIEAR-10 & CIFAR-100. Our training and evaluation proceeds as follows: We train Hopfield Boosting with R8 (He et al., 2016) the CIFAR-10 and CIFAR- 100 data sets (Krizhevsky, 2009), respectively. In these settings, we use ImageNet-RC: (Chrabaszez et al., 2017) (a low-resolution version of ImageNet) as the AUX data set. For testing the OOD detection performance, we the data sets SVHN (Street View House Numbers) (Netzer et al., 2011), Textures (Cimpoi et al., 2014), iSUN (Xu et al., 2015), Places 365 (López-Cifuentes et al., 2020), and two versions of the LSUN data set (Yu et al., 2015)-a - one where the images are cropped, and one where they are resized to match the resolution of the CIFAR data sets (32x32 pixels). We refer the two LSUN data sets as LSUN-Crop Crop and LSUN- respectively. We compute the scores s()) as described in Equation
(13) and then evaluate the discriminative power of s{E) between CIFAR and the respective OOD data set using the FPR95 and the AUROC. We use a validation process with different OOD data for model selection. Specifically, we validate the model on MNIST (LeCun et al., 1998), and ImageNet-RC with different pre- pe-pproesssing than in training (resize to 32x32 pixels instead of crop to 32x32 pixels), as well as Gaussian and uniform noise.
ImageNet- We evaluate Hopfield Boosting on the large-scale benchmark: We use ImageNet-1K 1K (Russakovsky et al., 2015) as ID data set and ImageNet-21K (Ridnik et al., 2021) as AUX data set. The OOD test data sets are Textures (Cimpoi et al., 2014), SUN (Xu et al., 2015), Places 365 (López- Cifuentes et al., 2020), and iNaturalist (Van Horn et al., 2018). this setting, all images are scaled to a resolution of 224x224. To keep our method comparable to other OE methods, we closely follow the training and evaluation protocol of (Zhu et al., 2023). This implies htat we fine-tune a Nes that was pre-trained on the ImageNet- IK ID classification task (as provided by TorchVision Vision, 2016).
8

---

Table 3: Ablated training procedures CIFAR-10. We compare the result of Hopfield Boosting to the results of our method when not using weighted sampling. the projection head, or the OOD loss. + indicates "lower is better" and "higher is better". All values in %. Standard deviations are estimated across five training runs.
Weighted Sampling Projection Head
LoOD
EPR95
0 0.:30000
0.70
L012900
45.05418601
SVHN
AUROC
50.000
99 53.55 10.0
89.21
90.99 12.0
FPR55
0.28
1.58
2:22210.6
28.59 28.5994431
LSUN-Crop
AUROC
10.000
98 18:22-1011
us 28 10.35 10.25
LSUX- Resize
FPR95
AU AUROCT 1
99. 98-0002
0.00
99 9g
0.00
99 00.9000000
0.0000.00
80.15
50,30=730
FPR95
0..611..1
0.26
0.39 20.10
40.30 1.1.0
Textures
AUROC
99. 99.85
90 00.81
09.70
89.58
EPR95
0.00
0.00
0.00=0000
51.08ttii
ISUN
AUROCT
99.97
99 09.99
89 .3.99 t0.02
88, 15.89
FPR95
4.2
8.20#0021
5.73=100
77.44
Places 365
AUROC
98.5120.1
07.6820.21
94.7750.51
78.30
Mean
FPRO5
AUROC
99.55
0.92
99.38
1.46
98.
2.06
88.50
50.40
Baselines. As mentioned earlier, previous works offer vast experimental evidence that OE methods offer superior OOD detection compared to methods without OE (see e.g., Ming et al., 2022; Wang et al., 2023a). Our experiments in Appendix 1.14 confirm this. Thus, we focus on comprehensive comparison of Hopfield Boosting to eight OE methods: MSP-OE (Hendrycks et al., 2019b), EBO-OE (Liu et al., 2020), POEM (Ming et al., 2022), MixOE (Zhang et al., 2023b), DAL (Wang et al., 2023a), DivOE (Zhu et al., 2023), DOE (Wang et al., 2023b) and DOS (Jiang et al., 2024).
Training setup. The network trains for 100 epochs (CIFAR-10/100 or 4 epochs (ImageNet-18 1 respectively. In each epoch, the model processes the entire ID data set and a selection of AUX samples (sampled according to wz). We sample mini-batches of size 128 per data set, resulting in combined batch size of 256. We evaluate the composite loss from Equation (11) for each resulting mini-batch and update the model accordingly. After epoch, we update the sample weights, yielding Wt+1. For efficiency reasons, we only compute the weights for 500,000 AUX data instances (~-40% of ImageNet), which we denote as E. The weights of the remaining samples are set to 0. During the sample weight update, Hopfield Boosting does not compute gradients or update model parameters. The update of the sample weights Wa+1 proceeds as follows: First, we fill the memories X and O with 50,000 ID samples and 50,000 AUX samples, respectively. Second, we use the obtained X and O get the energy F,g (E; X, X,O) O) for each of the 500,000 AUX samples in E and compute w according to Equation (10). In the following epoch, Hopfield Boosting samples the mini-batches O according to w with replacement. To allow the storage of even more patterns in the Hopfield memory during the weight update process, one could incorporate a vector similarity engine (e.g., Douze et al., 2024) into the process. This would potentially allow a less noisy estimate of the sample weights. For the sake of simplicity, we did not opt to do this in our implementation of Hopfield Boosting. As we show in section 4.3, Hopfield Boosting achieves state-of-the the-har-t OOD detection results and can scale to large datasets (ImageNet- IK) even without access to a similarity engine.
Hyperparameters & Model Selection. Like Yang al. (2022), we SGD with an initial learning rate of 0.1 and a weight decay of 5- 10-4. We decrease the learning rate during the training process with a cosine schedule (Loshchilov & Hutter, 2016). Appendix I.2 describes the image transformations and pre-processing. We apply optimizer, weight decay, learning rate, scheduler, and transformations consistently all OOD detection methods of the comparison. For training Hopfield Boosting, we use single value for 3 throughout the training and evaluation process and for all OOD data sets. For model selection, we use grid search with X, chosen from the set (0.1, 0.25, 0.5, 1.0), and B3, chosen from the set {2, 4, 8, 16, 32). From these hyperparameter configurations, we select the model with the lowest mean FPR95 metric (where the mean is taken over the validation OOD data sets) and do not consider the ID classification accuracy for model selection. In our experiments, and = 0.5 yields the best results for CIFAR-10 10 and CIFAR- 100. For ImageNet- K, we set. 3 32 and À 0.25.

---

4.3 Results & Discussion
Table I summarizes the results for CIFAR-10. Hopfield Boosting achieves equal or better performance compared to the other methods regarding the FPR95 metric for all OOD data sets. It surpasses POEM (the previously best OOD detection approach with OE in our comparison), improving the mean FPR95 metric from 2.28 to 0.92. On CIFAR- 100 (Appendix L.1,, Hopfield Boosting improves the mean FPR95 metric from 11.76 7.94. It is notable that all methods achieve perfect FPR95 results on the LSUN-ResiEn and iSUN data sets. This is somewhat problematic since there exists evidence that the -SUN-ResizZ data set can give misleading results due to image artifacts resulting from the resizing procedure (Tack et al., 2020; Yang et al., 2022). We hypothesize that a similar issue exists with the iSUN data set, in our experiments, LSUN-RRsizz and iSUN behave very similarly.
On ImageNet- IK (Table 2), Hopfield Boosting surpasses all methods our comparison terms of both mean FPR95 and mean AUROC. Compared to POEM (the previously best method) Hopfield Boosting improves the mean FPR95 from 50.74 to 36.60. This demonstrates that Hopfield Boosting scales very favourably to large-scale settings.
We observe that all methods tested perform worst on the Places 365 data set. To gain more insights regarding this behavior, we look at the data instances from the Places 365 data set that Hopfield Boosting trained on CIFAR-10 most confidently classifies as in-distribution (i.e., which receive the highest scores s{)). Visual inspection shows that among those images, a large portion contains objects from semantic classes included in CIFAR- 10 (e.g., airplanes, horses, automobiles). We refer to Appendix 1.6 for more details.
We evaluate the performance of the following 3 ablated training procedures the CIFAR-10 benchmark to gauge the importance of the individual contributions of Hopfield Boosting: (a) Random sampling instead of weighted sampling, (b) Random sampling instead of weighted sampling and no projection head, (c) No application of COOD. The results (Table 3) show that all contributions (i.e, weighted sampling, the projection head, and COOD) are important factors for Hopfield Boosting performance. For additional ablations, we refer to Appendix I.
When subjecting Hopfield Boosting to data sets that were designed to show the weakness of OOD detection approaches (Appendix 1.7), we identify instances where a substantial number of outliers are wrongly classified as inliers. Testing with EBO-OE yields comparable outcomes, indicating that this phenomenon extends beyond Hopfield Boosting.
5 Limitations
Lastly, we would like to discuss two limitations that we found: First, we see opportunity to improve the evaluation procedure for OOD detection. Specifically, it remains unclear how reliably the performance on specific data sets can indicate the general ability to detect OOD inputs. Our results from iSUN and LSUN-Resize (Section 4.3) indicate that issues like image artifacts in data sets greatly influence model evaluation. Second, although OE-based approaches improve the OOD detection capability, their reliance on AUX data can limit their applicability. For one, the selection of the AUX data is crucial (since it determines the characteristics of the decision boundary surrounding the inlier data). Furthermore, the use of AUX data can be prohibitive in domains where only a few or no outliers at all are available for training the model.
Conclusions
We introduce Hopfield Boosting: an approach for OOD detection with OE, Hopfield Boosting uses an energy term to boost a classifier between inlier and outlier data by sampling weak learners that are close to the decision boundary. We illustrate how Hopfield Boosting shapes the energy surface to form a decision boundary. Additionally, we demonstrate how the boosting mechanism creates a sharper decision boundary than with random sampling. We compare Hopfield Boosting to eight modern OOD detection approaches using OE. Overall, Hopfield Boosting shows the best results.
10

---

Acknowledgements
We thank Christian Huber for helpful feedback and fruitful discussions.
The ELLIS Unit Linz, the LIT La the Institute for Machine Learning, are supported by the Federal State Upper Austria. We thank the projects AI-MOTION
2-22018---YO--12,, DeepFlood (LIT- 2019-8- YOU-213), Medical Cognitive Computing Center (MC3), INCONTROL-RL, (FFG-88 1064), PRIMAL (FFG-873979) S3AI (FFG-872172) DL for GranularFlow (FFG-871302), EPILEPSIA (FFG-892171), AIRI FG 9-N (FWF-36284, FWF-36235), A14Grren/Hecatigrriiddd INTEGRATE (FFG-892418), ELISE (H2020-ICT-20119 ID: 951847), Staer (HORIZON- We thank Audi.JKU Deep Learning Center, TGW LOGISTICS GROUP GMBH. Silicon Austria Labs (SAL), University SAL Labs initiative, FILL Gesellschaft mbH, Anyline GmbH, Google, ZF Friedrichshafen AG, Robert Bosch GmbH, UCB Biopharma SRL, Merck Healthcare KGaA, Verbund AG, GLS (Univ. Waterloo) Software Competence Center Hagenberg GmbH, TÜV Austria, Frauscher Sensonic, Borealis AG, TRUMPF and the NVIDIA Corporation. This work has been supported by the "University SAL Labs" initiative of Silicon Austria Labs (SAL) and its Austrian partner universities for applied fundamental research for electronic- based systems. Daniel Klotz acknowledges funding from the Helmholtz Initiative and Networking Fund (Young Investigator Group COMPOUNDX, grant agreement no. VH-NG-1537) We acknowledge EuroHPC Joint Undertaking for awarding us access to Karolina IT44Innvvations, Czech Republic and Leonardo at CINECA, Italy.
899943),
References
Abbott, I.. F. and Arian, Y. Storage capacity of generalized networks. Physical review A, 36(10): 5091, 1987.
Ahn, S., Korattikara, A., and Welling, M. Bayesian posterior sampling via stochastic gradient Fisher scoring. In Proceedings of the 29th International Coference on International Conference on Machine Learning, pp. 1771-1778, Madison, WI, USA, 2012. Omnipress.
Anderson, B. G. and Sojoudi, S. Certified robustness via locally biased randomized smoothing. In Learning for Dynamics and Control Conference, pp. 207-220. PMLR, 2022.
Auer, A., Gauch, M., Klotz, D., and Hochreiter, S. Conformal prediction for time series with modern hopfield networks. Advances in Neural Information Processing Systems, 36:56027-56074, 2023,
Baldi, P. and Venkatesh, S. S. Number of stable points for spin- and neural networks of higher orders. Physical Review Letters, 58(9):913, 1987.
Bishop, C. Neural Networks for Pattern Recognition. Oxford University Press, 1995.
Bishop, C. M. Novelty detection and neural network validation. IEE Proceedings- Vision, Image and Signal processing, 141(4):277-22,, 1994.
Breiman, Arcing the edge. Technical report, Citeseer, 1997.
Caputo, B. and Niemann, Storage capacity of kernel associative memories. In Artificial Neural Net- Wookks-ICANe 2002: International Conference Madrid, Spain, August 28-30, 2002 Proceedings 12, pp. 51-56. Springer, 2002.
Chen, H., Lee, Y., Sun, G., Lee, H., Maxwell, T., and Giles, C. L. High order correlation model for associative memory. In AIP Conference Proceedings, volume 151, pp. 86-99, American Institute of Physics, 1986.
Chen, J., Li, Y., Wu, X., Liang, Y., and Jha, S. Atom: Robustifying outst---istiitutiion detection using outlier mining. In Machine Learning and Knowledge Discovery in Databases. Research Track: European Conference, ECML PKDD 2021, Bilbao, Spain, September 13-17, 2021, Proceedings, Part III 21, pp. 430-445. Springer, 2021.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1559711077. PMLR, 2020.
11

---

Chrabaszez, P., Loshchilov, I., and Hutter, F. A downsampled variant of imagenet as an alternative to the CIFAR datasets. arXiv preprint (rXiv:1707.0881, 2017.
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
Cortes, C. and Vapnik, V. Support-vector networks. Machine learning, 20(3):73-2277, 1995.
Cubuk, F., D., Zoph, B., Shlens, J., and Le, Q. V. Randaugment: Practical automated data augmenta- tion with reduced search space. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition workshops, pp. 702-703, 2020.
Demiriz, A., Bennett, K. P., and Shawe- Shawe-Taylor, J. Linear programming boosting via column generation. Machine Learning, 46:225-254 2002.
Djurisic, A., Bozanic, N., Ashok. A., and Liu, Extremely simple activation shaping for out-of- distribution detection. In The Eleventh International Conference on Learning Representations, 2023.
Douze, M., Guzhva, A., Deng, C., Johnson, J., Szilvasy, G., Mazaré, P.-E., Lomeli, M., Hosseini, L., and Jégou, H. The faiss library. 2024.
Du, X., Wang, Z., Cai, M., and Li, Y. Vos: Learning what you don't know by virtual outlier synthesis. arXiv preprint arXiv:2202.01 197, 2022.
Freund, Y. and Schapire, R. E. A decision-theoretcc generalization of on-line learning and an application to boosting. In Computational Learning Theory: Eurocolt 195, pp. 23-37. Springer- Verlag, 1995.
Friedman, J., Hastie, T,, and Tibshirani, R. Additive logistic regression: statistical view of boosting (with discussion and a rejoinder by the authors). The annals of statistics, 28(2):337-407, 2000.
Fürst, A., Rumetshofer, E., Lehner, J.. Tran, V. T.. Tang, F., Ramsauer, H., Kreil, D., Kopp, M.. Klambauer, G., Bitto, A., et al. CLOOB: Modern Hopfield networks with InfoLOOB outperform clip. Advances in neural information processing systems, 35:20450-20468, 2022.
Gardner, E. Multiconnected neural network models. Journal of Physics A: Mathematical and General, 20(1 10)3333,,,, 1987.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on compuler vision and pattern recognition, pp. 770-778, 2016.
He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Hendrycks, D. and Gimpel, K. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference Learning Representations, 2017. URL wtw.pperrrrriieeeeeeoroorrraaaaa
Hendrycks, D., Basart, S., Mazeika, M., Zou, A., Kwon, J., Mostajabi, M., Steinhardt, J., and Song, D. Scaling ouitt--disttibution detection for real-world settings. arXiv preprint arXiv: 131..113,, 2019a.
Hendrycks, D., Mazeika, M. and Dietterich, T. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations, 2019b. URL https://openreviie: htt.fforumiid-HyyCchheee
Hendrycks, D., Mazeika, M., Kadavath, S., and Song, D. Using self-supervised learning can improve model robustness and uncertainty. Advances in neural information processing systems, 32, 2019c.
Hendrycks, D., Zou, A., Mazeika, M., Tang, L., Li, B., Song, D., and Steinhardt, J. Pixmix: Dreamlike pictures comprehensively improve safety measures. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 16783-16792, 2022.
12

---

Hopfield, J. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79(8):2554-2555 1982.
Hopfield, J. J. Neurons with graded response have collective computational properties like those of two-state neurons. Proceedings of the National Academy of Sciences, 81(10):30883300 -3092, 1984. doi: 10. 10.1073/pnas.8 81. 10.3088.
Horn, D. and Usher, M. Capacities of multiconnected memory models. Journal de Physique, 49(3): 389-395, 1988.
Hu, J. Y.-C.. Chang, P.-... Luo, H., Chen, H.-Y., Li, W., Wang, W.-P. and Liu, H. Outlier-effiieent hopfield layers for large transformer-base models. In Forty-first International Conference on Machine Learning, 2024.
Isola, P. Xiao, J., Torralba, A., and Oliva, A. What makes an image memorable? In CVPR 2011, pp- 145-152. IERE, 2011.
Jiang, W., Cheng, H., Chen, M., Wang, C., and Wei, H. DOS: Diverse outlier sampling for out-of- distribution detection, The Twelfth International Conference on Learning Representations, 2024. URL https: : wpwpreveeeeeeeeeeeeeeee wwrrriiiwwne///rurreeeeee http://oomnnndrriieeeeee
Krizhevsky, A. Learning multiple layers of features from tiny images. Master's thesis, Deptartment of Computer Science, University of Toronto, 2009.
Krotov, D. and Hopfield,
Dense associative memory for pattern recognition. Lee, D. D., Sugiyama, M., Luxburg, U. V., Guyon, I., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, pp. 1172-118 180. Curran Associates, Inc., 2016.
LeCun, Y., Botton, L.,, Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(12))
1998.
Lee, K., Lee, K., Lee, H., and Shin, J. A simple unified framework for detecting nut-of-distribution samples and adversarial attacks. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa- Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https: http::::///cceeeeeeeeeee neurg.. pew.arppp.cc/ppppr//////////e pdf.
Liu, J., Shen, Z., He, Y., Zhang, X., Xu, R., Yu, H., and Cui, P. Towards out-of-distribution generalization: A survey, arXiv preprint wx..v:2218.... 13624, 2021.
Liu, W., Wang, X., Owens, J. and Li, Y. Energy-based out-of-distribution detection. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. and Lin, H. (eds.), Ad- vances Neural Information Processing Systems, volume 33, pp. 21464-21475. Curran Associates, Inc., 2020. URL https: setps://proceeding: www.:ppp..ccpppppr//0000.....
pp. 4449222200044888844444a1a0933009
Paper pdf.
Liu, X., Lochman, Y., and Zach, C. Gen: Pushing the limits of softmax-base based out-of-distributton de- tection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
López-Cifuenter A., Escudero- Fscudero-Vinolo, M., Bescós, J., and Gareía-Martin, Á. Semantic-aware scene recognition. Pattern Recognition, 102:107256, 2020.
Loshchilov, I. and Hutter, F. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 1608. 03983, 2016.
Lu, H., Gong, D., Wang, S., Xue, J., Yao, L.,, and Moore, K. Learning with mixture of prototypes for out-of-distribution detection. In The Twelfth International Conference on Learning Representations, 2024. URL https: //tp:::///pp/e //tp////eee//eeeee///eeeeee wwt.:fernnut////// 2erumm?dduWlkkaDDMMG..
McInnes, I.,, Healy, J., and Melville, Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint 10X.v:1802.03426. 2018.
13

---

Ming, Y. Fan, Y., and Li, Y. POEM:
detection with posterior sampling. In Chaudhuri, K., Jegelka, S., Song. L., Szepesvari, C., Niu, G., and Sabato. S. (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 15650-15665. PMLR, 17-23 Jul 2022. URI. utt.attss:/prroeeedin alr.press. watss//v2/////// v162/ming2222 html.
Ming, Y., Sun, Y., Dia, O., Li, How to exploit hyperspherical embeddings for out-of-distributtinn detection? In The Eleventh International Conference Learning Representations, 2023.
Moody, and Darken, C. Fast learning networks of locally-tuned processing units. Neural computation, 1(2):21-224, 1989.
Morteza, and Li, Y. Provable guarantees for understanding out-of-distribution detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 7831-7840, 2022.
Müller, K.-R., Smola, A. J., Rätsch, G., Schölkopf, B., Kohlmorgen, J., and Vapnik, V. Predicting time series with support vector machines. In International conference artificial neural networks, pp. 199-1004, Springer, 1997.
Netzer, Y., Wang, T., Coates, A.. Bissacco, A., Wu, B., and Ng, A. Y. Reading digits in natural images with unsupervised feature learning. 2011.
Oord, A. v. d., Li, Y., and Vinyals, O. Representation learning with contrastive predictive coding. arXiv preprint arXiv:800..00 807700778, 2018.
Paischer, F., Adler, T., Patil, V., Bitto-Nemling, A., Holzleitner, M., Lehner, S., ZZdeh, H., and Hochreiter, S. History compression via language models in reinforcement learning. In International Conference Machine Learning, pp. 17156-17185. PMLR, 2022.
Park, G. Y., Kim, J., Kim, B., Lee, S. W., and Ye, J. C. Energy-based cross attention for Bayesian con- text update in text-to-image diffusion models. In Thirty-seventh Conference Neural Information Processing Systems, 2023.
Psaltis, D. and Park, C. H. Nonlinear discriminant functions and associative memories. In AIP conference Proceedings, volume 151, pp. 370-375. American Institute of Physics, 1986.
Ramsauer, H., Schäff, B., Lehner, J., Seidl, P., Widrich, M., Gruber, L., Holzleitner, M., Pavlović, M., Sandve, G. K., Greiff, V., Kreil, D., Kopp, M., Klambauer, G., Brandstetter, J., and Hochreiter, S. Hopfield networks all you need. In 9th International Conference on Learning Representations (ICLR), 2021. URL https: http//ppeereeeeeeeeeeeeeeeee wwwerrewwwwwweeeeeeeeeeeeeeeeeee wwwt/eeuummmmee wid.ttttntttiic.....c
Rätsch, G., Onoda, T., and Müller, K.-.. Soft margins for AdaBoost. Machine learning, 42:287-320. 2001.
Ridnik, T., Ben-Baruch, E., Noy, A., and Zelnik-Manor, L. Imagenet-21k pretraining for the masses. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
Roth, K., Pemula, L., Zepeda, J., Schölkopf, B., Brox, T., and Gehler, P. Towards total recall in industrial anomaly detection. In Proceedings the the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14318 11438,1 2022.
Ruff, L., Kauffmann, J. R., Vandermeulen, R. A., Montavon, G., Samek, W., Kloft, M., Dietterich. T. G., and Müller, K.-R. A unifying review of deep and shallow anomaly detection. Proceedings of the IEEE, 109(5):756-79, 2021.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z.,, Karpathy, A., Khosla, A., Bernstein, M., et Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211- 1522, 2015.
Saleh, R. A. and Saleh, Statistical properties of the log-cosh loss function used machine learning. arXiv preprint arXiv:2208.04555 2022.
14

---

Manchez-Fermanderr A., Rumetshofer, E., Hochreiter, S., and Klambauer, G. CLOOME: a new search engine unlocks bioimaging databases for queries with chemical structures. bioRxiv, pp. 2022-11, 2022.
Schäft, B., Gruber, L., Bitto-Nemling, A., and Hochreiter, S. Hopular: Modern Hopfield networks for tabular data. arXiv preprini urXiv:2206.00664, 2022.
Schimunek, J., Seidl, P., Friedrich, L., Kuhn, D., Rippmann, F., Hochreiter, S., and Klambauer, G. Context-enriched molecule representations improve few-shot drug discovery. In The Eleventh International Conference Learning Representations, 2023.
Sehwag, V., Chiang, M., and Mittal, P. Ssd: unified framework for self-supervised outlier detection. arXiv preprint arXiv: 10..x:2103./20000 2021.
smeschke. Four Shapes.
2018. URL
Sun, Y., Guo, C., and Li, Y. React: Out-of-distribution detection with rectified activations. Advances in Neural Information Processing Systems, 34:144 2021.
Sun, Y., Ming, Y., Zhu, X., and Li, Y. Out-of-distributiion detection with deep nearest neighbors, In International Conference on Machine Learning, pp. 20827-20840. PMLR, 2022.
Tack, J. Mo, S., Jeong, J., and Shin, J. Csi: Novelty detection via contrastive learning on distribu- tionally shifted instances. Advances neural information processing systems, 13:11189-1188 2020.
Tao, L., Du, X., Zhu, X., and Li, Y. Non-parametric outlier synthesis. arXiv:2303.02966, 2023.
arXiv preprint
Teh, Y. W., Thiery, A. H., and Vollmer, S. J. Consistency and fluctuations for stochastic gradient Langevin dynamics. J. Mach. Learn. Res., 17()):193-225, 2016.
Torch Viscon Torchvision: Pytorch's computer vision library. tttps://githube com/ com/pytorch/ vision, 2016.
Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., and Belongie, The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8769-8778, 2018.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 30, pp. 5998-6008 Curran Associates, Inc., 2017.
Wang. H., Li, 7., Feng, I.,, and Zhang, W. Vim: Out-of-distribution with virtual-logit matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4921-4930, 2022.
Wang, Q., Fang, Z., Zhang, Y., Liu, F., Li. Y.. and Han, B. Learning to augment distributions for out-of-distribution detection. In NeurIPS, 2023a. URL btsttps://ppenrrree..eeeee//// OtU6VvXJue.
Wang, Q., Ye, J., Liu, F., Dai, Q. Kalander, M., Liu, T., Hao, J., and Han, B. Out-of-distribttion detection with implicit outlier transformation. arXiv preprint arXiv:2303.0500 2023b.
Wang, T. and Isola, Understanding contrastive representation learning through alignment and uniformity the hypersphere. In International Conference on Machine Learning, pp. 9929-9929999 PMLR, 2020.
Wei, H., Xie, R., Cheng, H., Feng, L., An, B., and Li, Y. Mitigating neural network overconfidence with logit normalization. International conference on machine learning, pp. 23631-23644 PMLR, 2022a.
15

---

Wei, X.-S., Cui, Q. Yang, L.. Wang. P., Liu, L., and Yang. J. Rpe: large-scale and fine- fine-grained retail product checkout dataset, 2022b. URL https://rppcc ratte//
dataset seaeet.githnn github. io/.
Welling, M. and Teh, Y. W. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th International Conference Machine Learning, pp. 681-688, Madison, WI, USA, 2011. Omnipress.
Widrich, M., Schäfl, B., Pavlović, M., Ramsauer, H., Gruber, L.., Holzleitner, M., Brandstetter, J., Sandve, G. K., Greiff, V., Hochreiter, S., and Klambauer, G. Modern Hopfield networks and attention for immune repertoire classification. ArXiv, 2007.13503, 2020.
Xu, K., Chen, R., Franchi, G., and Yao, A. Scaling for training time and post-hoc out-of-distribution detection enhancement. In The Twelfth International Conference on Learning Representations. 2024.
Xu, P. Ehinger, K. A., Zhang, Y., Finkelstein, A., Kulkarni, S. R., and Xiao, J. Turkergaze: Crowdsourcing saliency with webcam based eye tracking. arXiv preprint arXiv: 5504 (67555, 2015.
Xu, P., Chen, J., Zou, D., and Gu. Q. Global convergence of Langevin dynamics based algorithms for nonconvex optimization. In Bengio. S., Wallach, H., Larochelle, H., Grauman, K., Cesa- Cesa-Bianche Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 31, pp. 1122-3133 Curran Associates, Inc., 2018.
Yang, J., Zhou, K., Li, Y., and Liu, Z. Generalized out-of-distribution detection: A survey. arXiv preprint a 10. 1334, 2021.
Yang, J., Wang, P., Zou, D., Zhou, Z., Ding, K., Peng, W., Wang, H., Chen, G., Li, B., Sun, Y., et Openood: Benchmarking generalized out-of- distriiiin detection. Advances in Neural Information Processing Systems, 11332598-32611 2022.
Yu, F., Seff, A., Zhang, Y., Song, S., Funkhouser, T., and Xiao, J. LSUN: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint 1rXiv:1506. 03365, 2015.
Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., and Yoo, Y. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6023-6032, 2019.
Zhang. H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Zhang, J., Fu, Q., Chen, X., Du, I.., Li, 7., Wang, G., Han, S., Zhang, D., et al. Out-of-distribution detection based on in-distribution data patterns memorization with modern Hopfield energy. In The Eleventh International Conference on Learning Representations, 2023a.
Zhang, J., Inkawhich, N., Linderman, R., Chen, Y., and Li, H. Mixture outlier exposure: Towards -iitt--iiittiittiion detection in fine- geee-gaaned environments. In Proceedings of the IEEE/CVF Winter Conference Applications Computer Vision (WACV), pp. 5531-5540, January 2023b.
Zhang, R., Li, C., Zhang, J., Chen, C., and Wilson, A. G. Cyclical stochastic gradient MCMC for Bayesian deep learning. In International Conference on Learning Representations, 2020.
Zheng, Y., Zhao, Y., Ren, M., Yan, H., Lu, X., Liu, J., and Li, J. Cartoon face recognition: A benchmark dataset. In Proceedings of the 28th ACM International Conference on Multimedia, pp- 2020.
Zhu, J., Geng, Y., Yao, J., Liu, T., Niu, G., Sugiyama, M., and Han, B. Diversified outlier exposure for -fttt--distiibution detection via informative extrapolation. Advances in Neural Information Processing Systems, 36, 2023.
16

---

A Details on Continuous Modern Hopfield Networks
The following arguments are adopted from Fürst al. (2022) and Ramsauer et al. (2021). Associative memory networks have been designed to store and retrieve samples, Hopfield networks are energy- based, binary associative memories, which were popularized as artificial neural network architectures in the 1980s (Hopfield, 1982, 1984). Their storage capacity can be considerably increased by polynomial terms in the energy function (Chen et al., 1986; Psaltis & Park, 1986; Baldi & Venkatesh, 1987; Gardner, 1987; Abbott & Arian, 1987; Horn & Usher, 1988; Caputo & Niemann, 2002; Krotov contrast these binary memory networks, we use continuous associative memory networks with far higher storage capacity. These networks are continuous and differentiable, retrieve with a single update, and have exponential storage capacity (and are therefore scalable, i.e., able to tackle large problems; Ramsauer et al., 2021).
& Hopfield, 2016).
Formally, denote
of patterns (.,,..., n E.d that are stacked as columns to the matrix X = (I1, ................................ ZN) and state pattern (query) & E IR that represents the current state. The largest norm of a stored pattern is M = max; ||=|||- Then, the energy E of continuous Modern Hopfield Networks with state & is defined as (Ramsauer et al., 2021)
N exp((axf E)
E = -g- log Σ
+ 2 5" + C,
(14)
where - - log N + 1 M22 For energy E and state 5, Ramsauer et al. (2021) proved that the 2
update rule
<new
X softmax(3X"
(15)
converges globally to stationary points of the energy E and coincides with the attention mechanisms of Transformers (Vaswani et al., 2017; Ramsauer et al., 2021).
The separation A, of pattern x; is its minimal dot product difference any of the other patterns:
A min (x. a; xI x[;;))
(16)
joデi
pattern is well-separated from the data if A; above given threshold (specified in Ramsauer et al., 2021). If the patterns æ; are well-separated, the update rule Equation 15 converges to a fixed point close to stored pattern. If some patterns are similar the update rule converges to a fixed point close to the mean of the similar patterns.
another and, therefore, not well-separated,
The update rule of Hopfield network thus identifies sample-saple relations between stored patterns. This enables similarity- -based learning methods like nearest neighbor search (see Schäft et al., 2022), which Hopfield Boosting leverages detect samples outside the distribution of the training data.
Hopfield networks have recently been used for OOD detection (Zhang al., 2023a). Hu al. (2024) introduces Hopfield layers for outlier-efficient memory update.
B Notes on Langevin Sampling
Another method that is appropriate for earlier acquired models is to sample the posterior via the Stochastic Gradient Langevin Dynamics (SGLD) (Welling & Teh, 2011). This method is efficient since iteratively learns from small mini-batches Welling Teh (201 I); Ahn al. (2012). See basic work on Langevin dynamics Welling & Teh (201 I); Ahn et al. (2012); Teh et al. (2016): Xu et al. (2018). cyclical stepsize schedule for SGLD was very promising for uncertainty quantification Zhang et al. (2020). Larger steps discover new modes, while smaller steps characterize each mode and perform the posterior sampling.
C Related work
C.1 Details further OE approaches
This section gives details about related works from the area of OE in OOD detection. With OE, we refer to the usage of AUX for training OOD detector in general.
17

---

MSP-OE. Hendrycks et al. (2019b) were the first to introduce the term OE in the context of OOD detection. Specifically, they improve They train classifier on the ID data set and maximize the entropy of the predictive distribution of the classifier for the AUX data. The combined loss they employ
MSP-based OOD detection (Hendrycks & Gimpel, 2017):
XLOOD E [H(U) <P-pAUX
LCE
(17)
LOOD
(18)
where denotes the cross-entropy. 24 denotes the uniform distribution over classes, and pe is the model mapping the features to the predictive distribution over the K classes.
EBO-OE. Liu et al. (2020) propose a post-hoc and an OE approach. Their post-hoc approach (EBO) is to use the classifier's energy to perform OOD detection:
B(ED) - 1so ( s(P)) -E(E" .o)
(19)
(20)
where Sa outputs the model's logits as a vector. Their OE approach (EBO-OE) promotes a low energy on ID) samples and a high energy on AUX samples:
LOOD
E (max(00,E(z)) - mmn)) +
[[max(0,, MAUx E(o²)))²
(21)
xr TPPPD
a MPPAx
where mip and MAX are margin hyperparameters.
Ming et al. (2022) propose to incorporate Thompson sampling into the OE process. More specifically, they sample a linear decision boundary in embedding space between the ID and AUX data using Bayesian linear regression and then select those samples from the AUX data set that are closest to the sampled decision boundary. In the following epoch, they sample the AUX data uniformly from the selected data instances without replacement and optimize the model with the EBO-OE loss (Equation (21)).
POEM.
MixOE. Zhang al. (2023b) employ mixup (Zhang et al., 2018) between the ID and AUX samples to augment the OE task. Formally, this results the following:
à A + -1- (1=
(22)
= Ay + (1 - AYU
(23)
LOOD
[ x)
(24)
wPD Ppaax
Alternatively, they also propose to employ CutMix (Yun et al., 2019) instead of mixup (which would change the mixing operation in Equation (22)).
DAL. Wang (2023a) augment the AUX data by defining a Wasserstein-1 ball around the AUX data and performing OF using this Wasserstein ball. DAL is motivated by the concept of distribution discrepancy: The distribution the real OOD data will in general be different from the distribution of the AUX data. The authors argue that their approach can make OOD detection more reliable if the distribution discrepancy is large.
DivOE. Zhu et al. (2023) pose the question of how to utilize the given outliers from the AUX data set if the auxiliary outliers are not informative enough to represent the unseen OOD distribution. They suggest solving this problem by diversifying the AUX data using extrapolation, which should
18

---

result in better coverage of the OOD space of the resultant extrapolated distribution. Formally, they employ loss using synthesized distribution with manipulation A:
LOOD
E 101 H)H((U + y max + + Po (oo)) 0 ~xpuU
(25)
DOE, Wang et al. (2023b) implicitly synthesize auxiliary outlier data using a transformation of the model weights. They argue that perturbing the model parameters has the same effect as transforming the data.
DOS. Jiang al. (2024) apply K-means clustering the features of the AUX data set. They then employ a balanced sampling from the K obtained clusters by selecting the same number of samples from each cluster for training. More specifically, they select those samples from each cluster which are closest to the decision boundary between the ID and OOD regions.
D Future Work
D.1 Smooth and Sharp Decision Boundaries
Our work treats samples close to the decision boundary as weak learners. However, the wider ramifications of this behavior remain unclear. One can also view the sampling data instances close to the decision boundary as a form of adversarial training in that we search for something like "natural adversarial examples": Loosely speaking, the usual adversarial example case starts with given ID sample and corrupts it in a specific way to get the classifier to output the wrong class probabilities. In our case, we start with large set of potential adversarial instances (the AUX data) and search for the ones that could be either ID or OOD samples. That is, the sampling process will more likely select data instances that are hard to discriminate for the model - for example, if the model is uncertain whether leaf in an auxiliary outlier sample is frog or not.
This process be viewed "sharpening" the decision boundary: The boosting process frequently samples data instances with high uncertainty. LOOD encourages the model to assign less uncertainty the sampled data instances. After training, few training data instances will have high uncertainty. Nevertheless, a closer systematic evaluation of the sharpened decision boundary of Hopfield Boosting important to fully understand the potential implications w.r.t. adversarial examples. We view such an investigation out-of-scope for this work. However, we consider it an interesting avenue for future work.
Anderson & Sojoudi (2022) show that a smooth decision boundary helps with "classical" adversarial examples. In this framing, our approach would produce different adversarial examples that not based on noise but are more akin to "natural adversarial examples". For example, it is perfectly fine for that OOD sample close to the boundary does not correspond to any of the ID classes. Furthermore, the noise based smoothing leads to adversarial robustness at the (potential) cost of degrading classification performance. Similarly, our sharpening of the boundaries leads to better discrimination between ID and OOD region at the (potential) cost of degrading ID classification performance.
E
Societal Impact
This section discusses the potential positive and negative societal impacts of our work. As our work aims improves the state-on of off-the-art art in OOD detection, we focus on potential societal impact of OOD detection in general.
Postive Impacts
Improved model reliability: OOD detection aims to detect unfamiliar inputs that have little support in the model's training distribution. When these samples are detected, one can, for example, notify the user that no prediction possible, or trigger manual intervention. This can lead to increase in model's reliability.
19

---

Abstain from doing uncertain predictions: When a model with appropriate OOD detection recognizes that query sample has limited support in the training distribution, it can abstain from performing prediction, This can, for example, increase trust in MI. models, as they will rather tell the user they are uncertain than report confidently wrong prediction.
Negative Impacts
Wrong sense of safety: Having OOD detection in place could cause users to wrongly assume that all OOD inputs will be detected. However, like most systems, also OOD detection methods can make errors. is important to consider that certain OOD examples could remain undetected.
Potential for misinterpretation: As with many other ML systems, the outcomes of OOD detection methods are prone to misinterpretation. It is important to acquaint oneself with the respective method before applying it in practice.
20

---

Toy Examples
F.1
3D Visualizations of E, on hypersphere
(a) Out-offDistribution energy function
035
-125
(b) Exponentiated Out-of Diitt--Distribution energy function
0.20
0.05
Figure 3: Depiction of the energy function Eg(E: X, X on hypersphere. (a) shows E, X, O) with exemplary inlier (orange) and outlier (blue) points; and (b) shows cxp (E. X, 0)). was set to 128. Both, (a) and (b), rotate the sphere by 0, 90, 180, and 270 degrees around the vertical axis
This example depicts how inliers and outliers shape the energy surface (Figure 3). We generated patterns so that X clusters around a pole and the outliers populate the remaining perimeter of the sphere. This analogous to the idea that one has access to large AUX data set, where some data points are more and some less informative for OOD detection (e.g., conceptualized in Ming et al., 2022).
F22 Dynamics of LOOD Patterns in Euclidean Space
In this example, applied our distri--ittiittoonn loss LOOD problem. As we are working in Euclidean space and not on a sphere, we use a modified version of MHE, which the negative squared Euclidean distance instead of the not-product-ssmilarity For the formal relation between Equation (26) and MHE, we refer to Appendix H.1:
simple binary classification
N
B ||E 2
E(E: X) = - -B-1 log Σ
exp
22113)
(26)
Figure 4a shows the initial state of the patterns and the decision boundary exp(3E O)). We store the samples of the two classes as stored patterns in X and O, respectively, and compute LOOD for all samples. We then set the learning rate to 0.1 and perform gradient descent with LOOD on data points. Figure 4b shows that after 25 steps, the distance between the data points and the decision boundary has increased, especially for samples that had previously been close to the decision boundary. After 100 steps, as shown Figure 4d, the variability orthogonal the decision boundary has almost completely vanished, while the variability parallel to the decision boundary is maintained.
21

---

(a) After 0 steps
(b) After 25 steps
3-
0.29
0.22
0.15
apt8 Coodl
6018 Lonef
0.05
0.03
(c) After 50 steps
(d) After 100 steps
0.20
0.20
axp- Loxds
0.10
0.30
-1
-1
0.00
0.00
Figure 4: LOOD applied to exemplary data points on euclidean space. Gradient updates are applied to the data points directly. We observe that the variance orthogonal to the decision boundary shrinks while the variance parallel to the decision boundary does not change to this extent. 3 is set to 2.
22

---

E3 Dynamics of LOOD on Patterns on the Sphere
(a) After 0 steps
(b) After 500 steps
(c) After 2500 steps
Figure 5: LOOD applied to exemplary data points on sphere. Gradients are applied to the data points directly. We observe that the geometry of the space forces the patterns to opposing poles of the sphere.
F.4 Learning Dynamics of Hopfield Boosting on Patterns on a Sphere - Video
The example video' demonstrates the learning dynamics of Hopfield Boosting on 3-dimensional sphere. We randomly generate ID patterns X clustering around one of the sphere's 's poles and AUX patterns O the remaining surface of the sphere. We then apply Hopfield Boosting this data set. First, we sample the weak learners close to the decision boundary for both classes, X and O. Then, we perform 2000 steps of gradient descent with LOOD on the sampled weak learners. We apply the gradient updates to the patterns directly and do not propagate any gradients to an encoder. Every 50 gradient steps, we sampee the weak learners. For this example, the initial learning rate is set to 0.02 and increased after every gradient step by 0.1%.
hnttps://youtu. bel.ttdd-OttttttOftto Ofok
23

---

E5
Location of Weak Learners near the Decision Boundary
20
05
-05
-15
Twetare
Figure 6: A prototypical classifier (red circle) that is constructed with a sample close to the decision boundary. Classifiers like this one will only perform slightly better than random guessing (as indicated by the radial decision boundaries) and are, therefore, well-suited for weak learners.
E.6 Interaction between ID and OOD losses
Figure 7: Synthetic example of Hopfield Boosting training dynamics. The ID data is split in two classes (shown
and orange); the AUX data (blue) is sampled uniformly. We minimize the loss
L LCe + LOOD, and apply the gradient updates the patterns directly. The left Figure shows the initial pattern positions, the rightmost Figure shows the positions after 1000 gradient updates: The classes are well-separated; Eg forms tight decision boundary around the ID data.
To demonstrate how Hopfield Boosting can interact in a classification task we created toy example that resembles the ID classification setting (Figure 7): The example shows the decision boundary and the inlier samples organized in two classes (shown in red and orange). We sample uniformly distributed auxiliary outliers. Then, we minimize the compound objective (applying the gradient updates on the patterns directly). This shows that Hopfield Boosting able separate the classes well and that still forms tight decision boundary around the ID data.
24

---

G
Notes on Eg
G.1 Probabilistic Interpretation of E.
We model the class-conditional densities the in-distribution data and auxiliary data as mixtures of Gaussians with the patterns the component means and tied, diagonal covariance matrices with /3-1 in the main diagonal.
1
p( E
(1ID)
N
(27)
N
pl
E AUX)
1
Σ
N 05,
(28)
M
Further, we assume the distribution p(E) as a mixture of p( E ID) and p( E AUX) with equal prior probabilities (mixture weights):
p(E) = p(ID) p( ( + p(AUX) pl & AUX)
(29)
p(E ( ID) P AUX)
(30)
The probability an unknown sample & being an AUX sample is given by
P( & AUX ((UUX)
p( AUX E)
(31)
P(E)
P( & AUX )
(32)
2 P(E)
p( & AUX)
(33)
P( AUX) + E ID)
(34)
ID + PL ET AUX)
(35)
+ exp(log(p( E ID )) log(p( & AUX )))
where in line (34) we have used that p( & AUX for all Rd. The probability of & being ID sample given by
P(E ID)
p( ID {
(36)
2 2(E)
(37)
1 exp(log(p( & AUX )) log(p( & ID ID))
pl AUX )
(38)
Consider the function
fo(E) p( AUX| E) p( ID (
(39)
P(E AUX p( & ID)
(40)
4p(E)2
By taking the log of Equation (40) we obtain the following. We use additive constant that does not depend on
C
to denote equality up to an
25

---

g-! log (fo(E) - -2,311 log (p(E)) + 8=1 log (p( (IID)) + B-1 log (p( E AUX )) (41)
Multiplication by 8 is equivalent to a change of base of the log. The term - . log(p( ()) is
equivalent to the MHE (Ramsauer et al., 2021) (up an additive constant) when assuming normalized patterns, i.e. ||=; ||2 = 1 and ||o:|12 l22 = 1, and an equal number of patterns M = N in the two Gaussian
mixtures p( E ID) and p( E AUX) ):
-00 = 3- log 2 E (IID) + 1 2 El ap(EIAUX, AUX))
-
(42)
_ 3- log (p(
& ID) + p((5 AUX))
(43)
N
N
8 log
N (5:
1
Σ
N
(5:01,
(44)
N
N
N
C 3 log
Σ
N' (E:00,8-11) 111 Σ
N (E: 9-111
(45)
N
C 3 log
exp( exp(--11 -21le - =x:1)) +
Σ
exp(
2 - -
(46)
2
N
C 8 log
exp( (3x, T E
_
+
exp(3o, g - _
-2 _
(47)
2
1=1
N
1
8-1 log
Σ exp(Bx,f E) + Σ
exp(3of E)
+
5T
(48)
2
_ 1se(3, (X E)
+
E + g- log N + 2
1
2 MP
1
(49)
Analogously, 8-1 log(p( & ID)) )) and 9-1 log(p( & AUX) )) also yield MHE terms. Therefore, E5g
equivalent to 3-1 (f(( under the assumption that |ag|122 and ||o:l22 = 1 and M N The _ terms that are contained in the three MHEs cancel out.
g-1 log (f)(E) C (x, () + Be(33X"' + Iso (3) = E5: X
(50)
fo(E) can also be interpreted as the variance of a Bernoulli distribution with outcomes ID and AUX:
fa(E) = pl AUX (E) pl LDIDEE ( = p( ID - p( ID ()) pl AUX| & (1 - pl AUX E (1
(51)
In other words, minimizing Es means to drive Bernoulli-distrbbee outcomes ID and AUX towards minimum variance, i... p( ID E is driven towards if p( ID { > 0.5 and towards 0 if p( ID ( < 0.5. Conversely, the same is true for p( AUX g
random variable with the
From Equation (35), under the assumptions that |> = and ||o.112 and M N, the conditional probability p( AUX $ can be computed as follows:
p( AUX | { = olog(p( & |AU )) - log(p( E ID )))
(52)
= oB (Ise(8, O² g) Isc( x²) E))
(53)
26

---

where a denotes the logistic sigmoid function. Similarly, p( ID | & can be computed using
p ID ID|E) = o(3 (Ise(3, x² - -sse o² E))) (,,
(54)
1 - p( AUX ()
(55)
G.2 Alternative Formulations of Eg and fb
E, can be rewritten follows.
Ex(E:X,0) Ise [3, (X {)) + xe( + Ise(3, O²
(56)
--23 log cosh
3
(Ise x² -se()3 2,8 log(2) (57)
2
To prove this, we first show the following:
-3-1
log exp( B 1se xTE) ) exp( B3 (se(3,0TE)))
(58)
3 33- log Σ exp((Bo ))
N
3-1 log
exp
exp
3 log
Σ exp(3ofE)
(59)
N
3-1 log Σ
exp(3an 5) +
Σ exp(3of
(60)
= - 1se(3, (X (O)TE)
(61)
Let Ex
-Isc(3, x and Eo (s((3.0T))
Es(E: X,0) 2 1se (X || [)) + Ise(B, x² + Ise(3, O²
(62)
28- log ( exp(
Fxx expl - 33 Fo )) F.x
Fo
(63)
B
8
28- log exp(
Ex +exp(( Eo +
Eo
(64)
2
2
B
B -oo) expp(
8
8
-28-1
log expl
Ex
Eo
Ex
(65)
2
2
2
B
+ Eo)) -2,31 log(2)
28
log cosh
Ex
(66)
3 (Ise xT Ise(B,
28-1 log cosh
OTE)) (e(B,OTTE)
23-11 log(2)
(67)
28 log cosh
(1se OTE) - Ise(3, xT&)))
2.3-1 log(2)
(68)
By exponentiation of the above result we obtain
1
fo(E) exp(3E. X,))
(69)
4 cosh?
( (Ise XT - 1se O*E)
The function log cæ))( is related to the negative log-likelihood of the hyperbolic secant distribution (see e.g. Saleh & Saleh, 2022). For values of x close to 0. log cosh can be approximated by 2 and for values far from 0, the function behaves as |x|| - log(2).
27

---

MID M(D
1-p(1D (8)
0.4
0.2
lugi1 - P(D
kugla(1 p)) log(e) + lugh p] butlh)
0.0
In
AUX)
Inglst loglo AUX))
(a) Probabilities
(b) Log-probabilities
Figure 8: The product of two logistic sigmoids yields fo (a); the sum two log-sigmoids yields
(og(f)) = E, (b).
G.3
Derivatives of E,
In this section, we investigate the derivatives of the energy function Eg. The derivative of the Ise is:
V 15) V 3-1 log Σ Cex((Bz)
softmax( z)
(70)
Thus, the derivative of the MHE E(E: X) w.r.t. & is:
V X)E:X) Ve (=1sse(3,
2 -=- + +C 15ofmmax333"0 +
(71)
The update rule of the MHN
5² Xsoftmax(3x"e""
(72)
is derived via the concave-conver procedure. It coincides with the attention mechanisms of Transform- ers and has been proven to converge globally to stationary points of the energy E(g: X) (Ramsauer et al., 2021). It can also be shown that the update rule emerges when performing gradient descent on E(E: X) with step size Park et (2023):
gl+1 g¹ 17 VeE(E";X) X)
(73)
g+ = Xsoftmax(3XX* g¹)
(74)
From Equation (71), we can see that the gradient of Eg X, O) w.r.t. & is:
VeEo(;:X.0) = V (- 2 Ise || () + 1seeB,X xT) + Ise(3, OTEE) E))
(75)
[X 0) softmax(3(XX (O) + Xsoftmax(3x7+ E) + Osoftmax(30" {
(76)
When Xsostmax(3 (33 (3 E), Osoftmax(330 Ise( XT and (e(3.0")) are available, one efficiently compute (X O) ( O)T {) as follows:
28

---

(X O) softmax(3(X ())
(77)
Ve 1se(B, (X [0)*
(78)
= Ve 3 log (exp( x² + exp(AIse(3, O²
(79)
33
(Iso X E)
)
(Xsoftmax(3X" Osoftmax(30* {)) softmax
(110 1sc O ()
(80)
We also compute the gradient of (E: X, O) w.r.t. via the log coss-veprreenntation of E (see Equation (68)). The derivative of the log cosh function is
d 8- log cosh(B2) = tan)(Bx)
(81)
dz
Therefore, can compute the gradient of (5 x
,0)
Ve (,(5:X,0)
(82)
B (3) 1se(3, x"E))
Ve 23- log cosh
(83)
2
tanh (ls0 O² - 1so(3. xx())) (O))ftmax(30" X {))
(84)
= - -tanh (
[ [1se( 1,se(3,X') x²)
|se(3, (3)0 (X Softmax(33x' ( Osoftmax(3000 (B00 {))
(85)
Next, would like to compute the gradient of (E; X, O) w.r.t. the memory matrices X and O. For this, let us first look at the gradient of the MHE E(; w.r.. a single stored pattern æ; (where X is the matrix of concatenated stored patterns (I1, ......... xN))):
Ve, X)
-Esoftmax(3X E)
(86)
Thus, the gradient w.r.t the full memory matrix X is
Vx Ex; X) = -Esoftmax(3x
(87)
We can now also use the log cosh formulation of Eg(E: X, O) to compute the gradient of EE(E: X, O), w.r. X and O:
g (3se (Iee(3,xT)) - -,s((3, o²
XxE((:XXO) = V 23-1 log cosh
(88)
2
tanh ((se(( (3, (3, xT - 1se OTE)) Esoftmax(3x7
(89)
(90)
Analogously, the gradient w.r.t O
Vo X,0) = - =ttanh (2 (1se(3, (TTE - -Ise(3, XT E)) xTE))
(30ftmax(30"
(91)
29

---

H Notes on the Relationship between Hopfield Boosting and other methods
H.1 Relation to Radial Basis Function Networks
This section shows the relation between radial basis function networks (RBF networks; Moody & Darken, 1989) and modern Hopfield energy (following Schäfl et al., 2022). Consider an RBF network with normalized linear weights:
3 " 14: mill
(E
Σ
wi exp(
(92)
where 3 denotes the inverse tied variance 3 = gz, and the w; are normalized using the softmax function:
exp(Ba,)
w; softmax((Ba) Σ
(93)
exp(Bag) E}-1ex((aa})
An energy can be obtained by taking the negative log of (E))
E(E) - log (sp(E)
(94)
3
8- log
Wi exp(
115 m.1l3))
(95)
2
8-1 log
exp( B( 1 muil3 3-3 log softmax(8a))))
(96)
= 3-1 log
exp( B( 2 - -m112 + +a; - =s((3.
(97)
8-1 log
exp( B( TT g + u1 z pe + aai) Ise(B, a)
1
(98)
Next, we define 1 u4i
N E(E) 33-1 log Σ exp(33uT E)
+ 1
22 S Ise(3, a)
(99)
Finally, we use the fact that Ise(3, a) S max, a; + 3-1 log
1 ST 3-
1 3-- log N
E(g) 3-1 log Σ exp((p(T()
2
+ M2
(100)
where M = max; ||10//2
H.2
Contrastive Representation Learning
commonly used loss function in contrastive representation learning (e.g., Chen al., 2020; He et al., 2020) is the InfoNCE loss (Oord et al., 2018):
(t)) f(y)/T
LXCE
E (x.g))-Pps (x 3ud Puaa
log
(101)
f(w)// + E, ftm)
30

---

Wang & Isola (2020) show that LNCE optimizes two objectives:
)
log elx) ( rUolT
Σ (ea (elor))too)))
LNCB
E (2-3)~-Ppa
(pox) {or 1Md ropdata
Alignment
Uniformity
(102)
Alignment enforces that features from positive pairs are similar, while uniformity encourages uniform distribution of the samples over the hypersphere.
In comparison, our proposed loss, COOD, does not visibly enforce alignment between samples within the same class. Instead, we can observe that it promotes uniformity to the instances of the foreign class. Due the constraints that are imposed by the geometry of the space the optimization is performed on, that is, Ilf(z))] = when the samples move on a hypersphere, the loss encourages the patterns in the ID data have maximum distance to the samples the AUX data, i.e., they concentrate on opposing poles of the hypersphere. A demonstration of this mechanism can be found in Appendix F.2 and F.3
H.3 Support Vector Machines
the following, we will show the relation of Hopfield Boosting to support vector machines (SVMs: Cortes & Vapnik, 1995) with RBF kernel. We adopt and expand the arguments of Schäfl et al. (2022).
Assume apply SVM with RBF kernel model the decision boundary between ID and AUX data. We train on the features Z = (I1,... NN O1..... оM and assume that the patterns are normalized, i.e., ||æ/|12 1 and |oo la We define the targets (y1...., -(N+NM) for ID and -1 for AUX data. The decision rule of the SVM equates to
if x 2 0 B(E) = OODD
ID
(103)
if sf 0
where
N
s(E)
Σ
ogyik(z;,
E)
(104)
i=1
3
k(z,, { = exp
(-2
|1E
zz
(105)
We assume that there is at least one support vector for both ID and AUX data, i.e., there exists least one index s.t. a,y, >> O and at least one index j s.t. 0rU3 < 0. We now split the samples in s(E) according to their label:
s(E)
Σ a2 {)
Σ aN+.k(o.
(106)
We define altermative score:
v ank(at E)
Strae (E)
(107)
M Σ
i aN4sk((o
(108)
31

---

Because we assumed there is at least one support vector for both ID and AUX data and as the a; are constrained be non-negative and because k(-,)) 0, the numerator and denominator are strictly positive, can, therefore, specify new decision rule (E).
[ID
if Strme (E) 2 <
(109)
OOD
if Strne (E)
Although the functions s(E) and Stac (E) are different, the decision rules B(E) and (E) are equivalent. Another possible pair of score and decision rule is the following:
Slog Slog(E) =33-log(( lSgg (E))
8 log Σ
(ak(
8- log
Σ an k(o,
(110)
Bing (E)
ID
if Slog (5)
(111)
OOOD
if slog (E) <)<0
Let us more closely examine the term 8 log EN osk(zz;)
We define a, 3- log(a))
N
3 -- 1-116 - 2;113]
log
Σ (2)(
= 1-8-100p
exp(Ba;) exp
(112)
B 5" & + Bux
3
9-1 log
exp(Baa) exp
a ai
2
2
(113)
8 log Σ exp (- ET + Saet E
B 2 x + Bas)
(114)
3aa) )
1 5"
1
3-11og
Σ exp (3x
(115)
2
We now construct a memory X and query Eu such that we can compute (115) using the MHE (Equation (5)):
X
[x1
EN
(116)
aN
{
En
(117)
We obtain
+ 1 Shik + C
E(ÉH: Xx) - Iso(B, X XHEEH)
(118)
2
exp (Bxe & + 1Bai) )
1
1
-- log
+
2
12 + c
(119)
N
1
- -B 3-- log
Σ
exp (Bxef' & + sass))
+ 5 +
ブ + + c
(120)
2
i=1
N
- 1-8-110p log
Σ
ark((z
+ C
(121)
32

---

We construct Oμ analogously to Equation (116) and thus can compute
Slog (4) = E(EH; OH) - E(EH; x (seeB,,X_,_ ( EH
(122)
which is exactly the score Hopfield Boosting uses for determining whether a sample is OOD (Equation (13)). In contrast to SVMs, Hopfield Boosting uses uniform weighting of the patterns in the memory when computing the score. However, Hopfield Boosting can emulate a weighting the patterns by more frequently sampling patterns with high weights into the memory.
H.4 HE and SHE
Zhang et al. (2023a) introduce two post-hoc methods for OOD detection using MHE, which are called "Hopfield Energy" (HE) and "Simplified Hopfield Energy" (SHE). Like Hopfield Boosting, HE and SHE both employ the MHE to determine whether sample is ID or OOD. However, unlike Hopfield Boosting, HE and SHE offer possibility to include AUX data in the training process to improve the OOD detection performance of their method. The rest of this section structured as follows: First, briefly introduce the methods HE and SHE, second, formally analyze the two methods, and third, we relate them to Hopfield Boosting.
Hopfield Energy (HE) follows:
The method HE (Zhang al., 2023a) computes the OOD score SHE as
STIE (E) 1se( X
(123)
where X, TRdx... denotes the memory (Zel..... N.N.) containing No encoded data instances of class c. HE uses the prediction the ID classification head to determine which patterns to store in the Hopfield memory:
c
argmax pl y E
(124)
Simplified Hopfield Energy (SHE) The method SHE (Zhang et al., 2023a) employs a simplified score (S):
(SHE(E) =m² g
(125)
where m. Rdd denotes the mean of the patterns in memory Xx:
m.
Tei
(126)
Relation between HE and SHE In the following, we show simple yet enlightening relation between the scores She and SsmE For mathematical convenience, we first slightly modify the score SHEt
SHE 1se(3, (3) 9 log N.
(127)
All data sets which were employed in the experiments Zhang et al. (2023a) (CIFAR-10 10 and CIFAR-100) are class-balanced. Therefore, the additional term 3-1 log N, does change the result of the OOD detection on those data sets, as it only amounts to the same constant offset for all classes.
The function
33

---

N
1s, 9-1 llog N 3-1 3-+log
Σ
expl(2z)
(128)
converges the mean function B 0:
1
lim (Ise(B, x) 3-
(129)
N
i=1
We now investigate the behavior of SHE in this limit:
N
lim 8-0
- -8-1 "oogNN
Σ
x
-
(130)
N
i=1
N
T
Tei
E
(131)
N
i=1
m
E
(132)
where
m
cii
(133)
Therefore, have shown that
lim SHE (E) SSHE B-00
(134)
Relation of HE and SHE to Hopfield Boosting. In contrast to HE and SHE, Hopfield Boosting uses AUX data training process. To do this, our work introduces a novel MHE-based energy function, E, X O). determine how close a sample is the learnt decision boundary. Hopfield Boosting uses this energy function to frequently sample weak learners into the Hopfield memory and for computing novel Hopfield-basee OOD loss LOOD- To the best our knowledge, we the first use MHE in this way to train neural network.
learn a decision boundary between the ID and OOD regions during the
The OOD detection score of Hopfield Boosting is
s(E) Ise( (T) (e,
(135)
where E |BdxN contains the full encoded training set ( ØN) of all classes and E R contains AUX samples. While certainly similar SHe, the Hopfield Boosting score differs from SHe in three crucial aspects:
Hopfield Boosting uses AUX data samples in the OOD detection score in order to create a sharper decision boundary between the ID and OOD regions.
2. Hopfield Boosting normalizes the patterns in the memories X and O and the query & to unit length, while HE and SHE use unnormalized patterns construct their memories X, and their query pattern E
34

---

3. The score of Hopfield Boosting, s(E), contains the full encoded training data set, while SHE only contains the patterns of single class. Therefore Hopfield Boosting computes the similarities of a query sample & to the entire ID data set. In Appendix 1.8, we show that this process only incurs a moderate overhead of 7.5% compared to the forward pass of the ResNet-18.
The selection of the score function s(E) is only small aspect of Hopfield Boosting. Hopfield Boosting additionally samples informative AUX data close to the decision boundary, optimizes an MHE-based loss function, and thereby learns a sharp decision boundary between ID and OOD regions. Those three aspects are novel contributions of Hopfield Boosting, In contrast, work of Zhang et al. (2023a) solely focuses on the selection of suitable Hopfield-based OOD detection score for post-hoc OOD detection.
35

---

Table 4: OOD detection performance on CIFAR-100. We compare results from Hopfield Boosting.
DOS (Jiang et al., 2024), DOE (Wang al., 2023b), DivOE (Zhu al., 2023), DÄL (Wang et al.,
2023a), MixOE (Zhang et al., 2023b), POEM (Ming et al., 2022), EBO-OE (Liu et al., 2020), and MSP-OE OE (Hendrycks et al., 2019b) on ResNet-18. + indicates "lower is better" and "higher is better". All values in %. Standard deviations are estimated across five training runs.
Metric
HB (ouns)
DOS
DOB
DivOE
DAL
MixOB
POEM
EBO-OE
MSP-OE
SVIN
FPR95 1
AUROCT
97.0710
18,27450
D B.sqess
07.64-0.30
96.7261
28.77a5-48
84.25
50.001
19 19.00
02.274877
41.54
94.00
33 33.ggenii
36,33
19. 19.80t0
96.7g
2.54
28. 22.2
4.23
2212
23.10 4:35
15. 15.72-43.45
-112
32.88 11
LSUN-Crop
AUROC
96. 08.54=008
53. 60.7050.00
02.
96 36.0459944
06. 06.11
96 08..8800.00
05.70
92,8544
LSUN- Resior
FPR95 1
AUROC
09.9840.1
0.0020.00
99 49.5
99, 10.09
0.01 BM0
99. 99
0.00
50 90.94
0T. 07.99*1.00
10.27:3072
99 99.0714908
0.00=8000
09,5740.0
0.00
90.97
[TR95
2.35 10.15
6 4.32
10.4241.0
5 5.22-55.3
28, 46.79
2
5.07-935
10.5g
Textues
AUROC
20:002
08,3330031
$4.03 as
50
06.2
9y:shis
98.15
17.4225000
as
0.0gesto0g
ISUN
AUROC
90.08
90
07, 07.2342.0
99.5galica
09.5744.0
90
FDR95 1
10.36
52. 13-1.35
58. 68.88
44 2..23
33 33.43-1.11
47 (7.0
18. 16.39-0
26.68
45 06
Places 365
AUROC
95, 87188-000 37
91.73 -1.733
63 63-47
88.25 88.25
91. 10
80.20=1.54
95 05-03-
81.35
67. 77.M
Mess
BPK95
AUROC
98.
7.44
94.05
19.94
27.65
11.76
Accuracy
75.0g do
75.7240.36
76.91 10.00
77
22000
19.85
$9.00
I Additional Experiments & Experimental Details
L1 Results CIFAR-100
When applying Hopfield Boosting on CIFAR-100 (Table 4), Hopfield Boosting surpasses POEM (the previously best method), improving the mean FPR95 from 11.76 to 7.95. On the SVHN data set, Hopfield Boosting improves the FPR95 metric the most, decreasing it from 33.59 to 13.27. For the LSUN Resize and iSUN data sets, we observe a similar behavior as we saw in our CIFAR-10 10 evaluation almost all methods achieve perfect result with regard to the FPR95 metric.
1.2 Pre- Pre-Processing and Transformations
For evaluating OOD detection methods, consistent pre-processing and image transformation is crucial. An inconsistent application of image transformations will skew results when comparing different OOD detection methods. We, therefore, apply the same pre-processing steps and transformations to all OOD detection methods compare.
CIFAR-10 & CIFAR-100. For CIFAR-10 10 and CIFAR-100, we apply the following transformations:
1. RandomCrop (32x32, padding 4)
2. Random/HorizonaalFiip
ImageNet-RC. For ImageNet-RC (used as AUX data set for the ID data sets CIFAR-10 and CIFAR-100), we apply the following transformations:
1. RandomCrop (32x32)
2. RandomCrop (32x32, padding 4)
3. RandomHorizoottlFFiip
ImageNet-IK. For ImageNet-IK, we apply the following transformations. We closely follow the transformations used in the experiments of Zhu al. (2023):
1. Resize (224x224)
2. RandomCrop (224x224, padding
3. RandomHHriizomtalFiip
36

---

Table 5: Comparison between HE, SHE and version. + indicates "lower is better" and indicates "higher better".
Ours
HE
SHE
OOD Dataset
FPR95 +
AUROC 1
FPR95 4
AUROC t
FPR95
AUROC
SVHN
36.79
93.18
35.81
92.35
35.07
92.81
LSUN-Crop -Cop
13.10
97.25
17.74
95.96
18.19
96.10
-SUN-Resize
16.65
96.84
20.69
95.87
21.66
95.85
Textures
44.54
89.38
46.29
86.67
46.19
87.44
iSUN
19.20
96.08
22.52
95.08
23.25
95.06
Places 365
39.02
90.63
41.56
88.41
42.57
88.38
Mean
28.21
93.89
30.77
92.39
31.66
92.60
ImageNet-2IK, For ImageNet-21K (used as AUX data set for the ID data sets ImageNet-IK] we apply the following transformations. We closely follow the transformations used in the experiments of Zhu et al. (2023):
1. RandAugment (Cubuk et al., 2020)
2. Resize (224x224)
3. RandomCrop (224x224, padding 4)
4. RandomiHorizonallFiip
1.3 Comparison HE/SHE
Since Hopfield Boosting shares similarities with the MHE-based methods HE and SHE (Zhang et al., 2023a), we also looked the approach as used for their methods. We use the same ResNet-18 as backbone network as we used in the experiments for Hopfield Boosting, but train it on CIFAR-10 without OF We modify the approach of Zhang et al. (2023a) to not only use the penultimate layer, but perform search over all layer activation combinations of the backbone for the best pert-pperorming combination. We also do not use the classifier to separate by class. From the search, we see that the concatenated activations of layers and give the best performance on average, setting. We experience a quite noticeable drop in performance compared to their results (Table 5). Since the computation of the MHE is the same, we assume the reason for the performance drop is the different training the ResNet- 18 backbone network, where (Zhang et al., 2023a) used strong augmentations.
use this
1.4 Ablations
We investigate the impact of different encoder backbone architectures OOD) detection performance with Hopfield Boosting. The baseline uses a ResNet-18 as the encoder architecture. For the ablation, the following architectures can be observed, that the larger architectures lead to a slight increase in OOD performance (Table 6). We also see that change architecture from ResNet Densenet leads different OOD behavior: The result on the Places365 data set is greatly improved, while the performance on SVHN is noticeably
used comparison: ResNet-34, ResNet-50, and Densenet- 100.
than on the ResNet architectures. The FPR95 of Densenet SVHN also shows high variance, which is due to one of the five independent training runs performing very badly detecting SVHN samples as OOD: The worst run scores an FPR95 5.59, while the best run achieves an FPR95 of 0.24.
I.5 Effect on Learned Representation
In order to analyze the impact of Hopfield Boosting on learned representations, we utilize the output of our model's s embedding layer (see 4.2) as the input for a manifold learning-based visualization. Uniform Manifold Approximation and Projection (UMAP) McInnes et al. (2018) is non-linear
37

---

Table 6: Comparison of OOD detection performance CIFAR-10 of Hopfield Boosting different encoders. + indicates "lower is better" and f indicates "higher is better". Standard deviations are estimated across five independent training runs.
ResNet-18
ResNet-34
ResNet-50
Densenet-100
OOD Dataset
FPR95 L
AUROC 1
FPR95 4
AUROC
FPR95 J
AUROC f
PPR95
AUROC1 1
SVHN
0.23
99. 99.5720..0
0 0.33#40.2
99. 90.63=0.07
0,19:00.09
99 00.6440.11
2.11
99.31
LSUN-Crop Crop
0.82
99. 99.10
0.65
99. 39.5410.00
0.6g
99. 99.17#0000
0.40=20.28
99. 09.52 09
LSUN- Resize
0.00#0000
99. 99.98 09.98=0.02
0. 0.00=+0.00
99.89=0.04
0.00#+0.00
99.93 99.93+0.10
0.00=0.00
100.0 100.0=0.00 00
Textures
0.16+
99. 19..85#0.01
0.15+00.07
99.89 04.8900.44
0.16##0.07
99.8 89.83+0.01
0.08=0.02
99.88=000
ISUN
0.00
99.97#0.02
0.000:0.00
99. 90.9800002
0.00
991.98t00002
0.00=0000
99.99
Places 365
4 4.28:
98.51
4.13
98. 98.46-0.22
4.75:t0.5
98.71 #9.05
2. 2.56
99. 99.26**0 03
Mean
0.92
99.55
0.88
99.57
0.97
99.59
0.86
99.66
(a) without Hopfield Boosting
(b) with Hopfield Boosting
CIFAR-10 AUX SVHN
CIFAR-I0 AUX
SVHN
25
20
20
15
10
-5
-15
-2
20
10
3
10
15
Figure 9: UMAP embeddings of ID (CIFAR-I0) and OOD (AUX and SVHN) data based on our model trained without (a) and with Hopfield Boosting (b). Clearly, without Hopfield Boosting, the embedded OOD data points tend to overlap with the ID data points, making it impossible to distinguish between ID and OOD. On the other hand, Hopfield Boosting shows clear separation of ID and OOD data in the embedding.
dimensionality reduction technique known for its ability to preserve both global and local structure in high-dimensional data.
First, we train two models with and without Hopfield Boosting- and extract the embeddings of both ID and OOD data sets from them. This results in 512-dimensional vector representation for each data point, which we further reduce two dimensions with UMAP. The training data for UMAP always corresponds to the training data of the respective method. That is, the model trained without Hopfield Boosting solely trained on CIFAR- 10 data, and the model trained with Hopfield Boosting is presented with CIFAR- 10 and AUX data during training, respectively. We then compare the learned representations concerning ID and OOD data.
Figure shows the UMAP embeddings of ID (CIFAR-10) and OOD (AUX and SVHN) data based on our model trained without (a) and with Hopfield Boosting (b). Without Hopfield Boosting, OOD data points typically overlap with ID data points, with just few exceptions, making it difficult to differentiate between them. Conversely, Hopfield Boosting allows to distinctly separate ID and OOD data in the embedding.
1.6 OOD Examples from the Places 365 Data Set with High Semantic Similarity to CIFAR-10
We observe that Hopfield Boosting and all competing methods struggle with correctly classifying the samples from the Places 365 data set as OOD the most. Table shows that for Hopfield Boosting, the FPR95 for the Places 365 data set with CIFAR- 10 as the ID data set is at 4.28. The second worst FPR95 for Hopfield Boosting was measured on the LSUN-Crop data set at 0.82.
38

---

We inspect the 100 images from Places 365 that perform worst (i.e., that achieve the highest score s())) on model trained with Hopfield Boosting on the CIFAR-10 data set the in-distribution data set. Figure 10 shows that within those 100 images, the Places 365 data set contains non-negligible amount of data instances that show objects from semantic classes contained in CIFAR-10 (e.g.. horses, automobiles, dogs, trucks, and airplanes). We argue that data instances that clearly show objects of semantic classes contained in CIFAR-IU should be considered as in-distribution, which Hopfield Boosting correctly recognizes. Therefore, certain amount of error can be anticipated on the Places 365 data set for all OOD detection methods. We leave closer evaluation of the amount of the anticipated error up to future work.
For comparison, Figure 11 shows the 100 images from Places 365 with the lowest score s(E), as evaluated by a model trained with Hopfield Boosting on CIFAR- 10. There are no objects visible that have clear semantic overlap with the CIFAR-10 classes.
1.955
1.945
1.955
3.944
3.908
1960
1944
1.944
1.944
1.9
1.944
1.944
3.944
3.544
1.944
3.44
1:44
1.944
1.94
3.944
1.944
1968
1943
1.043
L.43
1:943
1945
2.343
2,543
19433
3:943
1.912
1.367
1912
1.022
1.02
1.022
3,832
1.412
3.442
1442
1,02
1841
3.441
3,943
1943
Figure 10: The set of top-100 100 images from the Places 365 data set which Hopfield Boosting recognized as in-distribution. The image captions show s({) of the respective image below the caption.
39

---

-1925
-1,925
1.925
4.925
-1,25
-1925
+,,23
-1.25
-1.25
2.925
3.975
-1,923
3.975
1.925
3,925
3.95
-,423
3. -.995
4 975
5,922
+.323
1.992
-3,925
1.923
1.925
1.925
1.923
1.925
1,25
-1.22
1.925
1.925
1.955
11925
3.522
1.925
4.,92
3,915
1.355
4.925
1.95
-1.923
-1925
-1.925
3.922
-1.975
+1925
4.,22
4.925
4.975
-3.022
Figure 11: The of top-100 images from the Places 365 data set which Hopfield Boosting recognized as out-of disttiibbiu...onn. The image captions show s(E) of the respective image below the caption.
1.7 Results on Noticeably Different Data Sets
The choice of additional data should be driven by desire showcase good performance; rather, we suggest opting for data that highlights weaknesses, as it holds the potential to drive investigations and uncover novel insights. Simple toy data preferable due its typically clearer and more intuitive characteristics compared to complex natural image data. In alignment with these considerations, the following data sets captivated our interest: iCartoonFace (Zheng et al., 2020), Four Shapes (smeschke, 2018), and Retail Product Checkout (RPC) (Wei et al., 2022b). In Figure 12, we show random samples from these data sets to demonstrate the noticeable differences compared to CIFAR- 10.
40

---

Table 7: Comparison between EBO-OE (Liu al., 2020) and version. + indicates "lower is better" and indicates "higher is better".
Hopfield Boosting
EBO-OE
OOD Dataset
FPR95 4
AUROC
FPR95 +
AUROC 1
iCartoonFace
0.60
99.57
4.01
98.94
Four Shapes
40.81
90.53
62.55
75.34
RPC
4.07
98.65
18.51
96.10
Random samples from three data sets, each noticeably different from CIFAR-10. 10. First row: iCartoonFace; Second row: Four shapes; Third row: RPC.
Figure 12:
Table we present some preliminary results using models trained with the respective method CIFAR-10 10 as ID data set (as in Table 1). Results for comparison presented for EBO-OE only, as time constraints prevented experimenting with additional baseline methods. Although would expect near-perfee results due to the evident disparities with CIFAR- 10. Four Shapes (smeschke, 2018) and RPC (Wei et al., 2022b) seem to defy that expectation. Their results indicate weakness in the capability identify outliers robustly since many samples inliers. Only iCartoonFace (Zheng et al., 2020) is correctly detected as OOD, at least to a large degree. Interestingly, the weakness uncovered by this data is present in both methods, although more pronounced in EBO-OE. Therefore, we suspect that this specific behavior may be general weakness when training OOD detectors using OE, aspect plan investigate further in our future work.
classified as
L.8 Runtime Considerations for Inference
When using Hopfield Boosting in inference, additional inference step is needed to check whether a given sample is ID or OOD. Namely, to obtain the score (Equation (13)) of a query sample eD Hopfield Boosting computes the dot product similarity of the embedding obtained from to all samples in the Hopfield memories X and O. In our experiments, X contains the full in- distribution data (50,000 samples) and O contains a subset of the AUX data set of equal size. We investigate the computational overhead of computing the dot- product similarity to 100,000 samples relation the computational load of the encoder, For this, we feed 100 batches of size 1024 an encoder (1) without using the score and (2) with using the score, measure the runtimes compute the mean and standard deviation. We conduct this experiment with four different encoders on an NVIDIA Titan V GPU. The results are shown in Figure 13 and Table 8. One can see that, especially for larger models, the computational overhead of determining the score very moderate comparison.
(E
batch, and
41

---

Table 9: OOD detection performance on CIFAR-10. We compare results from Hopfield Boosting with two extensions of HE (Zhang al., 2023a) ResNet- 18: HE+AUX includes AUX data in the OOD score. HE+OE applies OE (Hendrycks et al., 2019b) during the training process. + indicates "lower better" and "higher is better". All values in %.
OOD Dataset
Metric
HB (ours)
HE+AUX
HE+OR
FPR95
0.23
25.02
2.38
SVHN
AUROC
99.57
94.90
99.30
PPR95
0.82
LSUN-Crop
AUROC f
99.40
98.67
99.22
FPR95
0.00
13.69
0.00
ISUN--Resize
AUROC
99.98
97.68
99.95
FPR95
0.16
17.42
0.70
Textures
AUROC
99.84
97.08
99.72
FPR95
0.00
14.76
0.00
iSUN
AUROC
99.97
97.68
99.95
FPR95
4.28
41.24
10.84
Places 365
AUROC T
98.51
91.16
96.83
FPR95
0.92
19.91
2.72
Mean
AUROC
99.55
96.15
99.16
Incoder
MHt MHtt-based score
300
Runtime (ms. (bach)
250
150
ResNeb-18
ResNiet-34
ResNet-50
DenseNet-100
Eicoder
Figure 13: Mean inference runtimes for Hopfield Boosting on four different encoders on an NVIDIA Titan V GPU. We plot the contributions to the total runtime of the encoder and the MHEE-ased score (Equation (13)) separately. The evaluation shows that the score computation adds a negligible amount of computational overhead to the total runtime.
Table 8: Inference runtimes for Hopfield Boosting with four different encoders on an NVIDIA Titan V GPU. We compare the runtime of the encoder only and the runtime of the encoder with the MHE-based score computation (Equation (13)) combined.
Encoder
Time encoder (ms batch)
Time encoder score (ms / batch)
Rel. overhead (%)
ResNet-18
100.93=0.21
108.50#0.19
7.50
ResNet-34
209.8010...
217 217.33
3,59
ResNet-50
360.93 1.51
068.1740..86
2.01
Densenet-100
251.24+1.30 1.36
25.882
3.02
I.9 HE and SHE Extensions with AUX Data
To show that the unique contributions of Hopfield Boosting (like the energy-based loss LOOD and the boosting process) are responsible for the superior performance of Hopfield Boosting, we devise two extensions of HE that include AUX data and compare them to Hopfield Boosting.
42

---

Classification Error
Mean OOD FPR95
35
30
Messurement
10
5
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Figure 14: Tradcoff between classification error and Mean OOD FPR95 for different values of A. Decreasing the value of À to 0.1 improves the classification error while maintaining low OOD FPR95. (i.e., training only the ID classifier) achieves low classification error but dramatically increases the OOD FPR95.
The first extension (HE+AUX) uses model trained only the ID data and adapts HE to include an MHE term that measures the energy of & on the AUX data O:
Smod (E) SHE(E) 1se(3,
(136)
The second extension (HE+OE) applies OE (Hendrycks al., 2019b) while training the model. then uses SHE to estimate whether a sample is ID or OOD. For both extensions, we select 3 by minimizing the mean FPR95 on the OOD test data sets to obtain an upper bound of the possible performance of these extensions. The B we selected for HE+AUX is 0.001, and for HE+OE is 0.01.
Our results (Table 9) show that Hopfield Boosting superior to both extensions. HE+AUX results in mean FPR95 of 19.91, HE+OE achieves a mean FPR95 of 2.72. Hopfield Boosting improves on both extensions, achieving a mean FPR95 0.92.
I.10 Ablation on the Hyperparameter À
There is usually an inherent tradcoff between ID accuracy and OOD detection performance when employing OE methods. In practice one always improve the tradeoff by using models with more capacity- - in the extreme case practitioners can even train a separate ID network. Hence, the model selection process employed only considered the OOD detection performance and did not take the ID accuracy into account. To investigate and how this tradeoff can controlled by changing the hyperparameters of Hopfield Boosting, conduct the following experiment:
We (1) ablate the hyperparameter (the weight of the out-of-distributtinn loss) and run Hopfield Boosting the CIFAR-10 10 benchmark; (2) select À from the range [0, 1] with step size of 0.1; and (3) record the OOD detection performance (the mean FPR95 where the mean is taken over the OOD test data sets) and the ID classification error for the individual settings of X.
The results indicate that decreasing the hyperparameter À improves the ID classification accuracy of Hopfield Boosting (Figure 14). At the same time, the mean OOD AUROC is only moderately influenced: When setting, the hyperparameter setting reported in the original manuscript, the mean ID classification error is 5.98%, and the mean FPR95 is 0.92%. When decreasing À to 0.1, the mean ID classification error improves to 5.02%. Similarly, the FPR95 only slightly increases to 1.08% (which is still substantially better than the second- beot outlier exposed method, POEM, which achieves a mean FPR95 of 2.2.%). Hence, practitioners can control the tradeoff between ID classification accuracy and OOD detection performance.
43

---

1,000
0.990
0,998
0.996
SWN OnD AUROC
0.994
anD
0.994
0.992
0.992
3,990
0,968
0.990
8
100 1000
3000
10000 30000
10
50
500
1000
5000 10000 50000
No. patberns Hopheld memory
patterns Hophetld
(a) Balanced Hopfield networks
(h) Imbalanced Hopfield networks
Figure 15: Ablating the number of patterns stored in the Hopfield memories during inference. AUROC on SVHN based on the number of patterns in the Hopfield memory. In (a), X and contain the same number of patterns; in (b) X contains 50, 000 patterns, and we vary the number of patterns in O. The variability of the AUROC is reduced when X and contain 50,000 patterns, respectively.
L11 Ablation the Number of Patterns Stored in the Memories during Inference
In our implementation of Hopfield Boosting, we fill the memories and O with N = 50, 000 patterns to compute the score s()), respectively. To investigate the robustness of Hopfield Boosting when changing the number of patterns N, we conduct the following experiments:
1. We train Hopfield Boosting CIFAR-10 (ID data) and ImageNet (AUX data). During the weight update process, we store 50,000 patterns in the memories X and O, and then ablate the number of patterns stored in the memories for computing the score s(E) at inference time. We evaluate the discriminative power of s{{) on SVHN with 1, 5, 10, 50, 100, 500, 1,000, 5,000, 10,000, and 50,000 patterns stored in the memories X and O. To investigate the influence of the stochastic process of sampling N patterns from the ID and AUX data sets, conduct 50 runs for all of the and create boxplots of the runs. The results (Figure 15a) show that sampling 50, 000 patterns has the lowest variability of the individual trials. We argue that the reason for this is that by this time the entire ID) data is stored in the Hopfield memory - which effectively eliminates stochasticity from randomly selecting N patterns from the ID data.
2. To verify that we can use s(E) when the number of patterns in X and O is imbalanced, we fill X with all 50,000 data instances of CIFAR- 10 and fill O with 1, 5, 10, 50, 100, 500, 1000, 5000, 10,000, and 50,000 data instances of the AUX data set. Then, we evaluate the discriminative power of ({) for the different instances. Our results (Figure 15b) show that Hopfield Boosting is robust to an imbalance in the number of samples X and O. The setting with 50,000 samples in both memories (which the setting we use the experiments
in our original manuscript) incurs the least variability.
L.12 Compute Ressources
Our experiments were conducted internal cluster equipped with variety of different GPU types (ranging from the NVIDIA Titan V to the NVIDIA (0000-XM-80GBB)) For our experiments on ImageNet-IK, we additionally used resources of an external cluster that is equipped with NVIDIA A100-SXM-6400 GPUs.
For our experiments with Hopfield Boosting on CIFAR-10 10 and CIFAR- 100, one run (100 epochs) of Hopfield Boosting trained for about 8.0 hours on single NVIDIA RTX 2080 Ti GPU and required 4.3 GB of VRAM. Fnding the hyperparameters required 160h of compute for CIFAR-10 10 and CIFAR-100, respectively. These were divided across four RTX 2080 Ti. Estimating the standard deviation required 40 hours of compute single RTX 2080 Ti for CIFAR-10 and CIFAR- 100 respectively.
44

---

For ImageNet- IK. one run (4 epochs) of Hopfield Boosting trained for about 4.4 hours on a single NVIDIA A-100-XXM64GB GPU and required 26.9 GB of VRAM. Finding the optimal hyperpa- rameters required a total of 86h of compute, divided across 20 NVIDIA A-100-XXM64GB GPUs. Estimating the standard deviation required 22 hours of compute, divided across NVIDIA A-100- SXM64GB GPUs.
The amount of resources reported above cover the compute for obtaining the results of Hopfield Boosting reported in the paper. The total amount of compute resources for the project is substantially higher. Notable additional compute expenses are preliminary training runs during the development of Hopfield Boosting, and the training runs for tuning the hyperparameters and evaluating the results of the methods we compare Hopfield Boosting to.
45

---

I.13 Data Sets and Licenses
We provide a list of the data sets we used in our experiments and, where applicable, specify their licenses:
CIFAR-10 (Krizhevsky, 2009): License unknown
CIFAR-100 (Krizhevsky, 2009): License unknown
ImageNet-RC (Chrabaszez et al., 2017): Custom License
SVHN (Netzer et al., 2011): Creative Commons (CC)
Textures (Cimpoi et al., 2014): Custom License"
iSUN (Xu et al., 2015): License unknown
Places 365 (Lépez-Cifuuntes et al., 2020): License unknown
LSUN (Yu et al., 2015): License unknown
ImageNet-1K K (Russakovsky al., 2015): Custom License
ImagetNet-21K (Ridnik et al., 2021): Custom License?
SUN (Isola et al., 2011) License unknown
iNaturalist (Van Horn et al., 2018): Custom License
paccpe://mage--netorr// wwwp:o/w...ooo/ooa.opopp.p
Pnttpe://uw robots. ac. wttt:////aa/a//aa/////////// hitieddd//dx..eeee html
46

---

Table 10: OOD detection performance on CIFAR-10. We compare results from Hopfield Boosting. PALM (Lu al., 2024), NPOS (Tao et al., 2023), SSD+ (Sehwag al., 2021), ASH (Djurisic et al., 2023), GEN (Liu et al., 2023), EBO (Liu et al., 2020), MaxLogit (Hendrycks et al., 2019a), and MSP (Hendrycks & Gimpel, 2017) ResNet-18. + indicates "lower is better" and 1 "higher is better".
All values in %. Standard deviations are estimated across five training runs.
HB (owrs)
PALM
NPOS
SSD+
ASH
GEN
EBO
MaxLagit
MSP
FPR95
0. 0.235000
1:240005
B.pgpttia
3.00400:2
25, 25.1718
20.2646
32.10
3022ptt or
48.4yako 1o
AUROCT
90.575000
88:44 2000
04.8652000
98.63e1
82.4geht
FPR35
0.8258
1.215977
5.5239950
2,831
13.1341
19.4049
17.25-
18.50 s1
38.324
LSUN-Crop
AUROC1
90.405004
99 65.6500000 06
98.977200
30.37
07. 97.334
96 16.48-40.00
96.73-MM.. 4.
06.5244447
94.3772
FPR95
0.000
27.01
31 31.50aam
30.69
3L647
45.8243
LSUN-Resize
AUROC1
96 95.41
04.78
00.3972.0
04.04
84.02240
03.002
922 02288421.0
FPR95
0.16 0.16:18.0z
17.32-2250
21.72
21,20/1
d0.08 1n
44.2244
41.07
44,97
56.044 1.
Textures
AUROC1
89.84-0.00
96.8240.22
40.3
96. 40.35
88, 28.32
10.12241.00
80.56
90.10
0.00 on
25.71a
26.904a.0
36.718999
a
36.02 to
ISUN
AUROCT
90. 89.97-20.z
96 05
86. 742444
94.4949
88.06422
92.50-
92, 12.88M0
91. woggttr
FPR95
4 2.28
22 12.975217
6g
24,90=
08, of
15, 8241
11,87:11
45, 63643111
57 55.55
Places 365
98.51-2030
84.90=9022
85 65.6571777
68 28
68.53-1130
28.42240.2
88, (6..66tt..
Mean
FPRO5
AUROCT
99.55
0.92
97.02
13.91
96.31
21.44
96.57
20.35
35, 35.50
90.04
35,
92.65
34.09
92.55
92.43
81.0 91.64
48:21
Method type
OF
Training
Triining Strong
Training Sarong
Post-hoe Weak
Prot-Booe Weak
Pout-hoe Weak
Post-hoe Weak
Post-hoe Weak
Angmentations
Weak
Strong
Audliary outlie
L14 Non-OE Baselines
To confirm the prevailing notion that OE methods can improve the OOD detection capability in general, we compare Hopfield Boosting 3 training methods (Sehwag al., 2021; Tao al., 2023; Lu et al., 2024) and post-hoc methods (Hendrycks & Gimpel, 2017: Hendrycks et al.. 2019b; Liu al., 2020, 2023; Djurisic al., 2023). For all methods, we train a ResNet- on CIFAR-10. For Hopfield Boosting, we use the same training setup as described in section 4.2. For the post-hoc methods, we do not use the auxiliary outlier data. For the training methods, we use the training procedures described in the respective publications for 100 epochs. Notably, all training methods employ stronger augmentations than the OE or the post-hoc methods. The OE and post-hoc methods use the following augmentations (denoted as "Weak"):
1. RandomCrop (32x32), padding 4
2. Random/HorizonalFFip
The training methods use the following augmentations (denoted as "Strong"):
RandomResizedCrop (32x32), scale 0.2-1
2. RandomHorizoottllFlip
3. ColorJitter applied with probability 0.8
4. RandomGrayscale applied with probability 0.2
Table 10 shows the results of the comparison of Hopfield Boosting to the post-hoc and training methods. Hopfield Boosting is better OOD detection than all non-OE baselines on CIFAR-10 in terms of both mean AUROC and mean FPR95 by a large margin. Further, Hopfield Boosting achieves the best OOD detection on all OOD data sets in terms of FPR95 and AUROC, except for SVHN and L.SUN-rro. where PALM (Lu et al., 2024) shows better AUROC results. An interesting avenue for future work to combine one of the non-OE based training methods with the OE method Hopfield Boosting.
47

---

Informativeness of Sampling with High Boundary Scores
This section adopts and expands the arguments of Ming al. (2022) on sampling with high boundary scores.
We assume the extracted features of trained deep neural network to approximately equal Gaussian mixture model with equal class priors:
1 -E: P(E) 2 (
(137)
2
(ID ID) -N(E:H,O
(138)
PAUX (E) - P(E\AUX) -N
(139)
Using the MHE and sufficient data from those distributions, estimate the densities p(E).
P(E) ID) and P(E]AUX).
Lemma J.1. (see Lemma E.1 in Ming et al. (2022)) Assume the M sampled data points o, ~ PAUX satisfy the following constraint high boundary scores Ey(E)
M Disl
Es(o,)
S
(140)
Then they have
22 0.l S Meo²
(141)
Proof. They first obtain the expression for EE(E) under the Gaussian mixture model described above and can express p(AUXlE) as
p(AUX|E) P(EIAUX)p(AUX))
(142)
P(E)
{U(EIAUX)
(143)
P(EID + AUX)
(2TO²) d/2 exp(
20z 115 ml/22
(144)
(2TO2) d/2 exp(
20 115 14 (22
(2 1) d/2
expl
20 11
(145)
exp( - 20 (1
14
||E 141l3)
When defining (E) - (II 1 |g + 1413) such that p(AUX(E) = =o(sxux(E))
" they define Es, follows:
(E -f)
(146)
= 20
||E 141/2 - ||5 + 1a1221
(147)
E" 2p & + 1² 4 (E" 2 & n² p)]
(148)
2
2
(149)
o²
48

---

Therefore, the constraint in Equation (141) is translated to
M
2u Pu"oo - Meo²
(150)
As maxieM oz S EM 1 o.] given fixed M, the selected samples can be seen as generated from PAUX with the constraint that all samples lie within the two hyperplanes in Equation (150).
Parameter estimation. Now they show the benefit of such constraint in controlling the sample complexity. Assume the signal/noise ratio is large: ||||| r 1, and I 1 some constant.
Assume the classifier given by
M N
0
oi)
(151)
M
where o, ~ PAUX and Ø; ~ PID. One can decompose Assuming N:
1
0 μ + 17
w
(152)
N
=
x3i)
(153)
1=1
M
0:)
(154)
1=1
We would now like to determine the distributions of the random variables |nl1 and fz.
|l Σ v
(155)
Th ~ NNO σ² )
(156)
VN
m
1()1
(157)
σ
NN
n1) x²
(158)
Therefore, for || we have
N
7:)²
Xa
(159)
Now would like to determine the distribution of μ² n';;
49

---

1² n
Mi q
(160)
i=1
14 11
NOO
(161)
d NOO, Σ
?
P4 Th
(162)
i=1
0²
d
Σ P4 1J N(0,
Σ
(163)
N 1=1
1= |
NO0 N)
(164)
Concentration bounds. They now develop concentration bounds for |17122 and μ² n.n. First, we look at ||n]] 2. A concentration bound for x² is:
P(X - d dx + 2x) s exp(-2)
(165)
By assuming x
we obtain
852
P(X d2 RDX-d2211
802
+
2
80
< exp(
802
(166)
d
d
d
P(X >
+
+ + < expl
(167)
20
4
80
N
d
d
IP
1 2
+
+ d) 2 exp(
(168)
/20
2
80
0²
d
P([n
3
+ d))
- exp(
(169)
N
/20
4
80
If d 2 2 we have that
d
(170)
V20
4g2
and thus, the above bound can simplified when assuming 2 as follows:
d² N ( d)) < exp( 80
P([n
(171)
For || since all o, is drawn i.i.d. from PAUX. under the constraint Equation (150), the distribution of w be seen as truncated distribution of m7. Thus, with some finite positive constant we have
o² (d )) <
22 N (d+ ( s cexp(- d
P(| z
<P([n
(172)
N
802
Now, we develop a bound for f='n n7. A concentration bound for N(,, 0²) is
SStrictly, the bound is valid for d 2
50

---

2 P(X p 2 S expl
(173)
2n²
By applying till ~ NO0 o²)
to the above bound we obtain
T
-t2
π 2 t) S
P
exp(
(174)
2
Assuming (d]|||||) we obtain
4 7
(
(0112(1)1/2)
exp(
(175)
20²
4 n
-111011AV
P(
2
(ollm(1)
v exp(
(176)
|A||
2a
Due to symmetry, we have
nº n
-111111N
< (olloal))) exp(
(177)
|
20
P n 7 S (o)lln1))) + P nT n
-1|Ke1.NV
(2)
S 2
(178)
||l||
20
We rewrite the above bound using the absolute value function.
m
1
P
(allml)) exxp( |||
(179)
2a
Benefit of high boundary scores. We will now show why sampling with high boundary scores is beneficial. Recall the results from Equations (150) and (154):
M
22 Th o,1 s Meo"
(180)
M
w
0:
(181)
M
t=1
The triangle inequality is
a bl x
|a|
+ [6]
(182)
+
V (-b)
[a] [b]
(183)
Using the two facts above and the triangle inequality can bound Jp2" w|:
51

---

1 M
o²t
# T 0:l S
(184)
M
2
1=1
O²E
1 0 $
(185)
M
1 M
o²c
Σ
To 0,1
1|11|1
(186)
2
1
Σ
M
0: 11 μb
0²c
+
1112
(187)
M
2
1=1
02e
11
w/
1 |
(188)
2
Developing lower bound. Let
a²
|| -
N (d
(189)
||م| v N (d
(190)
| m |
(ا|| 1/2
(191)
hold simultaneously. The probability of this happening can be bounded follows: We define T and its complement T:
o² (d+
1 n
T 2
(d (d+
S (allm()))
(192)
||l|||
a²
o²
1
I m
7 (11ml)
>
(2)l
(193)
||]|||
With P(T) + P(T)
= 1. The probability PPT)) can be bounded using Boole's inequality and the results in Equations (171), (172) and (179):
1|
P(T) I exp(-d/8o²) + cexp(-d/8o") + + exxp(
(194)
20
-1mu|IAV
P(T) < (1 + c) cxp( + 2exp(
(195)
20
Further, can bound the probability P(T)):
||p|||N
P(T) < +1+ (1 2exp(
(196)
20
||p1||N
P(T) < (1 + c) exp( d/80 +2+22250 eexp(
(197)
22
|||||/
P(T) > (1+ exp( d 2 exp(
(198)
2a
Therefore, the probability of the assumptions in Equations (189), (190), and (191) occuring simult- ||||| N
neously is at least 1 (1 + c) exp(-d/8o exp(
26
52

---

By using the triangle inequality, Equation (152) and the Assumptions (189) and (190) they can bound |
|001
"
wl1
(199)
|011 < || + 11 nl2
+ 1 12
(200)
1001 & | + 4 1nl1
+ 4 1w11
(201)
l
|101 & | + 2 (d
(202)
+ N (d+ 1
0²
|101 S |
(203)
The reverse triangle inequality defined as
y 2 |2| x (-w)1
(204)
2 ||a||
|y]
(205)
Using the reverse triangle inequality, Equations (152), (188) and Assumption (191) we have that
0 = 1u² u +
2u T
1
+
wl
(206)
1 N 1 1
1
Tu
1/2 TTTlll
1
(207)
To
N
||
1
1/2
||/1//2
3/2
2 |
1
1
(208)
2
2 2
1 10
1
M
||
(209)
2
2 2
02E)
1NT0
(1
2/2 |/|1//2
(210)
They have assumed that the signal/noise ratio is large: ||l|| = r >>>>>> 1. Thus, we can drop the absolute value, because assume that the term inside the || is larger than zero:
1 2 55 (11e0112 - 1
(211)
2,
1 _
0=
2, 1 (11mm
(212)
We have
(1
2r 2
(213)
if r 2 1.36602550338444.....
and e < 1, and therefore
03
1 0
(1111 02/21120113/2
(214)
2
Because of Equation (203) and the fact that if and sgn (z) sgn (3) then x 2 y- we have
53

---

(215)
|ull + x2 (d+
We can combine the Equations (214) and (215) to give a single bound:
|A" 0
1 12 /²/2 |1|| d²c
2
(216)
|||
2 | + (d+ )
we define such that 0² > 0 and thus
n 0
14 2
02/2
p2|3/22
o²c
(217)
2v |1112 (d+ 1
The false negative rate FNR(O) and false positive rate FPR(O) are
FN = [ N(:
1²
o
dx
(218)
|00|
-H² 0
FPR(O) N(x;
dx
(219)
||0||
As Nz; Ni-=; we have FNR(())=PPR(CC "²e
= FPR(O)). From Equation (217) can will increase. Thus, the mean of the Gaussian distribution
that as c decreases, the lower bound of [ell in Equation (218) will increase and therefore, the false negative rate will decrease, which shows the benefit of sampling with high boundary scores. This completes the extended proof adapted from (Ming et al., 2022).
54

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction regarding improvement in OOD detection capabilities are backed by extensive experiments in Section 4. Our theoretical results in the Appendix (most notably Sections G.1, H.I, and J) support the applicability of our method for OOD detection with OE.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section 4.3 and Appendix 1.7 show that when subjecting Hopfield Boosting to data sets that have been designed to show the weaknesses of OE methods, we identify instances where substantial number of outliers are wrongly classified as inliers by Hopfield Boosting and EBO-OE.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattio, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
55

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Our theoretical results include the probabilistic interpretation of Equation (6) in Appendix G.I, the suitability of Hopfield Boosting for OOD detection in Appendix J, and the connection of Hopfield Boosting to RBF networks (Appendix H.1) and SVM (Appendix H.3). The proofs of our claims contained in the respective Appendices.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The details of the experiments we conducted are explained in section 4.2, and the data sets are publicly available. The code of Hopfield Boosting is included in the submission.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
56

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the code to reproduce the experimental results of Hopfield Boosting in the submission. All data sets used are publicly available. Descriptions how to run the code and how to obtain the data sets come with the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
Sttpi///guies//oodeeeeee wtt.subiissssoooiiooooooooooooee for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nips. cttpcc/pppcccccccc gwwp//g///////ee///e////
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the training and OOD test data sets, the optimizer, and the hyperparameters we tested and selected, as well as the selection procedure Section 4.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
57

---

Justification: We provide estimates of the standard deviation for the main results of our paper on Hopfield Boosting and the compared methods in the Tables 1, 4, 2, 10. The standard deviations were estimated across five training runs, which is stated in the captions of the respective tables.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide detailed description of the compute resources we used as well as the amount of compute our experiments required in Appendix 12.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: / http///eurrp..eeee cc htc/public//ee d lines?
Answer: [Yes]
Justification: We have read the NeurIPS Code of Ethics and found that our work conforms with the code of ethics. More specifically, did not include any human subjects in our work. We excluded deprecated data sets from our work.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
58

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide a discussion of positive and negative societal impacts in Appendix
E.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Our work concerned with the improvement of out-of-distribution (OOD)
detection algorithms on small- to mid-scale vision data sets, and is therefore not considered to pose a high risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
59

---

Answer: [Yes]
Justification: We provide a list of the data sets we use (and cite the original authors) in Section 4.2. We provide a list of the respective licenses (where applicable) in Appendix I.13.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code that is included in the submission comes with READE.md file, which contains all instructions how to run the code to reproduce the experiments.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
60

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
61


Reviews:
[{"review_id": "jVaawnVaDN", "reviewer": "Reviewer_qRFa", "summary": "This paper addresses the crucial task of out-of-distribution (OOD) detection, essential for the safe deployment of machine learning models in real-world scenarios. Within the domain of OOD detection, outlier exposure methods, which use auxiliary outlier data during training, have shown significant improvements in OOD detection performance. In this paper, the authors introduce Hopfield Boosting, a boosting approach that utilizes modern Hopfield energy together with the OOD data to refine the decision boundary between in-distribution (ID) and OOD data. By focusing on hard-to-distinguish auxiliary outlier examples near the decision boundary, Hopfield Boosting enhances the model's ability to differentiate between ID and OOD data. The authors of the paper demonstrates the effectiveness of the proposed method empirically with several benchmark datasets.", "questions": "- The paper introduces the MHE-based energy function to combine OOD data with Hopfield Energy. Would it be possible to leverage OOD data in other ways so that it is more similar to HE or SHE in the prior work [1]? If so, how does the proposed method compare to such a naive way of combining HE/SHE with outlier exposure? I think an ablation study like this would help strengthen the novelty of the paper. \n- It was stated that 50k samples were used during inference to compute the OOD score. How much better/worse does the proposed method get when we vary the number of samples used? \n- What if IN and OOD data are highly imbalanced? \n\n[1] \"Out-of-distribution detection based on in-distribution data patterns memorization with modern Hopfield energy\"", "limitations": "Yes the authors of the paper adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is mostly well written and easy to follow. \n- The proposed method is technically sound. \n- Authors of the paper carry out ample experiments. \n- The proposed method achieves good performance.", "weaknesses": "- A major complaint I have for the paper is the lack of novelty. As mentioned by the authors of the paper in Section 3.4, the proposed method is an incremental extension of the prior work which also proposed to use Hopfield Energy for OOD detection. The main difference lies in the fact that the proposed method also utilizes auxiliary OOD data. Despite the effectiveness of the proposed method, outlier exposure is a well understood method to enhance the effectiveness of OOD detection, and it is not surprising to me that utilizing OOD data with Hopfield Energy enhances the effectiveness of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Weaknesses:**\n\n- We respectfully disagree with the claim that Hopfield Boosting is an incremental extension. We take it as a hint that we did not emphasize the novelty enough. Hopfield Boosting not only combines existing concepts in a novel, principled, and theoretically well-motivated way, but also includes specific innovations that have not been used in previous work: The novel energy function that, to our knowledge, is first introduced in our paper, fits a nonlinear decision boundary into the embedding space of a neural network. Hopfield Boosting samples patterns close to the decision boundary and computes the OOD loss using the same energy function. Our theoretical results are also novel. They show that the novel energy function is well-motivated from a probabilistic perspective. We will make sure that the revised manuscript states this more clearly.\n\n**Response to Questions:**\n- Adapting the setting of Zhang et al. (2023) to leverage OOD data in the way we do is not trivial. Probably, the most straightforward avenues to include AUX data in their setting are: (a) Use an existing OOD loss (e.g., the one from Hendrycks et al., 2019) to perform outlier exposure during training and use HE/SHE on the resulting model. (b) Use a model trained only on the ID data and adapt HE or SHE to include an MHE term that measures the energy of $\\boldsymbol{\\xi}$ on the AUX data. Due to time constraints, we only conducted experiment (b) for the rebuttal, but we will include both HE/SHE extensions in the camera-ready version of the manuscript.\n\n  In specific, experiment (b) consists of two steps: First, we take CIFAR-10 as a benchmark and use the following score: $s_{\\mathrm{mod}}(\\boldsymbol{\\xi}) \\ = \\ s_{\\mathrm{HE}}(\\boldsymbol{\\xi}) - \\mathrm{lse}(\\beta, \\boldsymbol{O}^T\\boldsymbol{\\xi})$, where $\\boldsymbol{O}$ contains 50,000 randomly selected encoded patterns from the auxiliary outlier data set. Second, we tune $\\beta$ such that $s_{\\mathrm{mod}}$ performs well on the OOD Test data to obtain an upper bound on the possible performance. The $\\beta$ we selected is $0.001$. The results are as follows (all numbers in %):\n\n  |             | FPR95 | AUROC |\n  |-------------|-------|-------|\n  | SVHN        | 25.02 | 94.90 |\n  | Textures    | 17.42 | 97.08 |\n  | Places365   | 41.24 | 91.16 |\n  | LSUN-Crop   | 7.35  | 98.67 |\n  | LSUN-Resize | 13.69 | 97.68 |\n  | iSUN        | 14.76 | 97.42 |\n  | **Mean**    | 19.91 | 96.15 |\n\n  For comparison, Hopfield Boosting obtains a mean FPR95 of 0.92%. This indicates that the innovations introduced in our work are the main factors for the effectiveness of Hopfield Boosting.\n\n- This question is best answered with an experiment that shows the effect of the memory size:\n\n  We train Hopfield Boosting on CIFAR-10 (ID data) and ImageNet (AUX data). During the weight update process, we store 50,000 patterns in the memories $\\boldsymbol{X}$ and $\\boldsymbol{O}$, and then ablate the number of patterns stored in the memories for computing the score $s(\\boldsymbol{\\xi})$ at inference time. We evaluate the discriminative power of $s(\\boldsymbol{\\xi})$ on SVHN with 1, 5, 10, 50, 100, 500, 1,000, 5,000, 10,000, and 50,000 patterns stored in the memories $\\boldsymbol{X}$ and $\\boldsymbol{O}$. To investigate the influence of the stochastic process of sampling $N$ patterns from the ID and AUX data sets, we conduct 50 runs for all of the $N$ and create boxplots of the runs.\n\n  The results (attached PDF) show that sampling $N=50,000$ patterns has the lowest variability of the individual trials. We argue that the reason for this is that by the time we reach $N=50,000$ the entire ID data set is stored in the Hopfield memory — which effectively eliminates stochasticity from randomly selecting $N$ patterns from the ID data.\n\n  These are interesting results and we thank the reviewer for bringing the point up. We will include the results in the Appendix of our final paper.\n\n- We see two notions of imbalance that could be relevant here:\n  1. Imbalance w.r.t. the number of samples contained in the ID and AUX data sets: The experiments we conducted in the manuscript show that Hopfield Boosting can handle this notion of imbalance. CIFAR-10 (the ID data set) contains 50,000 training samples, while ImageNet (the AUX data set) contains 1,281,167 training samples. Hopfield Boosting uses all training data from CIFAR-10, but for ImageNet, we sample 50,000 data instances to match the memory size of the ID data set. This is a simple way to ensure that the score $s(\\boldsymbol{\\xi})$ is not influenced by the number of data instances contained in the ID and AUX data sets.\n  2. Imbalance w.r.t. the number of samples stored in the Hopfield memories: This imbalance is indeed not tackled in the original manuscript, albeit it could be beneficial to store an imbalanced number of patterns in the memories $\\boldsymbol{X}$ and $\\boldsymbol{O}$. Hence, we do the following experiment to verify that we can use $s(\\boldsymbol{\\xi})$ when the number of patterns in $\\boldsymbol{X}$ and $\\boldsymbol{O}$ is imbalanced: We fill $\\boldsymbol{X}$ with all 50,000 data instances of CIFAR-10 and fill $\\boldsymbol{O}$ with 1, 5, 10, 50, 100, 500, 1000, 5000, 10,000, and 50,000 data instances. Then, we evaluate the discriminative power of $s(\\boldsymbol{\\xi})$ for the different instances. Our results (attached PDF) show that Hopfield Boosting is robust to an imbalance in the number of samples in $\\boldsymbol{X}$ and $\\boldsymbol{O}$. The setting with 50,000 samples in both memories (which is the setting we use in the experiments in our original manuscript) incurs the least variability.\nWe will also include this experiment in the Appendix of the updated manuscript.\n\n**References:**\n\nHendrycks et al., Deep Anomaly Detection with Outlier Exposure, ICLR 2019\n\nZhang et al., Out-of-Distribution Detection based on In-Distribution Data Patterns Memorization with Modern Hopfield Energy, ICLR 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sRzBULYIW9", "reviewer": "Reviewer_DQmw", "summary": "- The paper introduces a novel approach to improve OOD detection by leveraging modern Hopfield energy. The proposed method, called Hopfield Boosting, utilizes auxiliary outlier data to refine the decision boundary between ID and OOD data. By focusing on hard-to-distinguish auxiliary outlier samples near the decision boundary, the method enhances the ability to detect OOD samples.", "questions": "The questions related to the methodology and experimental results have been addressed in the Weaknesses section.", "limitations": "The authors have adequately discussed the limitations of their work.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The incorporation of modern Hopfield energy and auxiliary outlier data for OOD detection is interesting. The method effectively sharpens the decision boundary between ID and OOD data by utilizing the energy measure, which quantifies dissimilarity between data instances and stored patterns. By sampling informative outliers close to the decision boundary, Hopfield Boosting ensures that the model learns a more precise boundary, leading to better OOD detection performance.\n- The proposed method achieves significant improvements in OOD detection metrics compared to existing methods.", "weaknesses": "- The experimental results in Table 3 show that the effect of weighted sampling is not very significant. Despite this, the proposed method still outperforms existing approaches. It is unclear whether these performance gains are primarily due to the model's structure (two heads for classification and energy computation), the OOD score defined in Equation (13), or the proposed $L\\_{OOD}$. This ambiguity makes it difficult to pinpoint the key factors contributing to the improved performance and understand the true efficacy of each component of the proposed method. More detailed ablation studies are needed to disentangle and evaluate the contributions of these individual components.\n- The paper lacks a detailed explanation of how the hyperparameter $\\beta$ is determined. While it is mentioned that $\\beta$ is selected through a validation process, the specific metrics or criteria used to decide the optimal value are not discussed. Providing a clear methodology for $\\beta$ selection, including the validation metrics used, would enhance the reproducibility and transparency of the proposed approach.\n- There is a need for discussion regarding the impact of the proposed method on in-distribution (ID) accuracy. The proposed method could potentially negatively affect the performance of the classification head, which would undermine its overall utility. Addressing how the method influences ID accuracy and providing empirical evidence to show that it does not significantly degrade classification performance is crucial for establishing the method's efficacy and practical value.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Weaknesses:**\n\n- Because of the highly competitive nature of the CIFAR-10 benchmark, we view every improvement as important (e.g., the previous methods' AUROC were already above 99%; e.g., POEM achieves an AUROC of 99.21%). That said, we agree that more evaluations w.r.t. the individual components introduced in Hopfield Boosting will shed more light on which factors influence the good performance of Hopfield Boosting, and thank the reviewer for this suggestion. To find the contribution of the individual elements of Hopfield Boosting, we evaluate the performance of the following ablated training procedures on the CIFAR-10 benchmark:\n\n  1. Random sampling instead of weighted sampling\n  2. Random sampling instead of weighted sampling and no projection head\n  3. No application of $\\\\mathcal{L}_{OOD}$\n\n  The results in the PDF (Table 2) attached to the answer on top of the page show that all of weighted sampling, the projection head, and $\\\\mathcal{L}_{OOD}$ are important factors that contribute to Hopfield Boosting’s performance.\n  We will include these experimental results in the Appendix.\n\n- We agree with the reviewer that the manuscript will benefit from a more detailed description of the validation process. We will expand the experimental section with additional information regarding the validation and model selection process, as follows:\n\n  For validation and model selection, we evaluate the model on the OOD data sets MNIST and ImageNet-RC with different preprocessing than in training (resize to 32x32 pixels instead of crop to 32x32 pixels), as well as Gaussian and uniform noise. We train models using a grid search with $\\lambda$ selected from the set $\\\\{0.1, 0.25, 0.5, 1.0\\\\}$, and $\\beta$ selected from the set $\\\\{2, 4, 8, 16, 32\\\\}$. From said hyperparameter configurations, we select the model with the lowest mean FPR95 metric (where the mean is taken over the validation OOD data sets) and do not consider the ID classification accuracy for model selection.\n\n- We agree with the reviewer that reporting the ID accuracy will help the reader to better understand the influence between outlier exposure methods and the ID classifier’s performance. Therefore, we have compiled the ID accuracies achieved by the individual outlier exposure methods and included results in the PDF attached.\n\n  There is usually an inherent tradeoff between ID accuracy and OOD detection performance when employing outlier exposure methods. We do however not view it as a crucial aspect of our work, since in practice one can always improve the tradeoff by using models with more capacity — in the extreme case practitioners can even train a separate ID network. Hence, the model selection process we employed so far only considered the OOD detection performance and did not take the ID accuracy into account. That said, we still agree with the general sentiment of the reviewer. Thus, we decided to examine the tradeoff by conducting the following experiment:\n\n  We (1) ablate the $\\\\lambda$ hyperparameter (the weight of the out-of-distribution loss $\\\\mathcal{L}_{OOD}$) and run Hopfield Boosting on the CIFAR-10 benchmark; (2) select $\\\\lambda$ from the range $[0, 1]$ with a step size of $0.1$; and (3) record the OOD detection performance (the mean FPR95 where the mean is taken over the OOD test data sets) and the ID classification error for the individual settings of $\\\\lambda$. \n\n  The results indicate that decreasing the $\\\\lambda$ hyperparameter improves the ID classification accuracy of Hopfield Boosting (Figure 1). At the same time, the mean OOD AUROC is only moderately influenced: When setting $\\\\lambda = 0.5$, the hyperparameter setting reported in the original manuscript, the mean ID classification error is 5.98%, and the mean FPR95 is 0.92%. When decreasing $\\\\lambda$ to 0.1, the mean ID classification error improves to 5.02%. Similarly, the FPR95 only slightly increases to 1.08% (which is still substantially better than the second-best outlier exposed method, POEM, which achieves a mean FPR95 of 2.28%). Hence, practitioners can control the tradeoff between ID classification accuracy and OOD detection performance.\n\n  Our update will include the ID accuracy results and the tradeoff Figure.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fnu1DnM2vD", "reviewer": "Reviewer_UgW1", "summary": "A novel method for identifying OOD samples is presented using Hopfield boosting.  A classifier is trained using a loss function which sharpens the decision boundary between inlier and outlier data samples, where outlier data samples are purposefully drawn from an auxiliary dataset.  By sampling examples close to the decision boundary and determining the modern hopfield energy of the sample, the decision boundary is adjusted to exclude OOD samples from the auxilliary dataset.  The method is demonstrated twice using a low resolution and high resolution example.  In both examples, the Hopfield boosting method outperforms state of the art on average.  An extensive appendix includes additional background, theoretical results, and experimental results.", "questions": "1. How do these results compliment/contrast with results suggesting that a smooth decision boundary is more robust to adversarial samples? See for example [Anderson (2022)](https://proceedings.mlr.press/v168/anderson22a)\n\n2. Most computer vision datasets are expected to have some features in common, which motivates transfer learning techniques.  What justification was used to determine that the test datasets would be strictly OOD w.r.t. the training datasets CIFAR or ImageNet?\n\n3. Is there any analysis which shows that the decision boundaries sharpened during Hopfield Boosting are semantically meaningful, and not based on noise of the dataset?\n\n4. The ResNet-50 model is fine-tuned rather than trained from scratch; what dataset was used to train the pre-initialized weights? Why was the model not trained from scratch?  Because the feature space was already formed (4 epochs is not enough to significantly shift the model), does this imply that the model classification layer is likely identifying noise?", "limitations": "Please see the questions above.  The remaining limitations are thoroughly addressed in the work.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The method is original, theoretically sound, and useful.  The results are clearly presented, to the point where a reader may be able to implement the algorithm without referencing the provided code.  The research question is impactful, and the result presented in this paper is a theoretical and experimental advancement in the field.", "weaknesses": "The primary weakness is that the paper does not conclusively show that the method does not lead to overfitting.  The models are trained with a classification loss, but the authors state \"the toy example does not consider the inlier classification task that would induce secondary processes, which would obscure the explanations.\"  However, failure to report these results in the context of the decision boundary refinements shown in Figure 2, 3b and 6 opens the work to questions of model overfitting.  \n\nThis concern is only partially lifted by near perfect results on the test datasets.  Because the false positive rate (FPR) is reported with respect to an OOD test dataset, there is a question of whether the model is losing in-distribution detections.   Robustness to OOD samples of the original dataset, robustness to adversarial instances, and catastrophic forgetting are all implicated in the sharpening of decision boundaries; this work avoids acknowledging these cases and prevents inference about the implications by omitting the classifier performance for the trained models.  This can be resolved by reporting the model F1 scores in the appendix for the ID classification task. If the models were trained using the Hopfield Boosting alone, then this would not be necessary.  But the presence of the classifier loss means the secondary effects have implications for the semantic meaning of the decision boundary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Weaknesses:**\n- We agree with this. To demonstrate how Hopfield Boosting can interact in a classification task we created a toy example that resembles the ID classification setting (Figure 4 in the PDF attached to the response on the top of the page): The example shows the decision boundary and the inlier samples organized in two classes. We sample uniformly distributed auxiliary outliers. Then, we minimize the compound objective $\\mathcal{L} = \\mathcal{L}\\_{CE}+\\mathcal{L}\\_{OOD}$ (applying the gradient updates on the patterns directly). This shows that $\\mathcal{L}_{CE}$ is able to separate the two classes well and that $E_b$ still forms a tight decision boundary around the ID data.\n- We agree with the reviewer. The application of Hopfield Boosting influences the ID decision boundary: It decreases the ID classification accuracy on CIFAR-10 on average from 94.80% without Hopfield Boosting to 94.02% with Hopfield Boosting.\nOur work treats samples close to the decision boundary as weak learners. However, the wider ramifications of this behavior remain unclear. For example, the reviewer rightfully points towards a connection to adversarial examples. And indeed, one can also view the sampling of data instances close to the decision boundary as a form of adversarial training in that we search for something like “natural adversarial examples”: Loosely speaking, the usual adversarial example case starts with a given ID sample and corrupts it in a specific way to get the classifier to output the wrong class probabilities. In our case, we start with a large set of potential adversarial instances (the AUX data) and search for the ones that could be either ID or OOD samples. That is, the sampling process will more likely select data instances that are hard to discriminate for the model — for example, if the model is uncertain whether a leaf in an auxiliary outlier sample is a frog or not. Nonetheless, a closer systematic evaluation of the sharpened decision boundary of Hopfield Boosting is important to fully understand the potential implications w.r.t. adversarial examples. We view such an investigation as out-of-scope for this work. However, we consider it as an interesting avenue for future work and will mention so explicitly in the revised manuscript.\n\n  We include the classification results (with the F1 scores) of a model trained with Hopfield Boosting on the CIFAR-10 benchmark in the official comment below. Table 1 in the PDF posted on top of the page also includes the ID accuracies of all compared methods. Figure 1 in the PDF shows the performance when ablating on $\\lambda$, the weight of our OOD loss. We will add all those results in the Appendix of the updated manuscript.\n\n**Response to Questions:**\n1. We would like to thank the reviewer for this very interesting reference. The dichotomy between sharp and smooth decision boundaries is highly relevant to our work. We will discuss it in the updated manuscript. The smooth boundaries discussed in Anderson et al. (2022) help with “classical” adversarial examples. In this framing, our approach would produce different adversarial examples that are not based on noise, but are more akin to “natural adversarial examples” (see our answer to the weakness above). For example, it is perfectly fine for us that an OOD sample close to the boundary does not correspond to any of the ID classes. Furthermore, the noise based smoothing leads to adversarial robustness at the (potential) cost of  degrading classification performance. Similarly, our sharpening of the boundaries leads to better discrimination between ID and OOD region at the (potential) cost of degrading classification performance. \n2. The test data sets we employ for OOD (Tables 1 and 2 in the original manuscript) can be seen as a de-facto standard benchmark. They have been used in a wide range of other publications (e.g., Hendrycks et al., 2019; Ming et al., 2022; Zhu et al., 2023). We opted to follow the established work for the sake of comparability.\nHowever, we believe that the reviewer raises an excellent point in that the choice of OOD data sets needs to be more rigorously discussed by the community. Indeed, albeit the selection is not a focus of our work, we did take up upon the resulting problems with regard to benchmarking: In Appendix H.5 (original manuscript), we investigate the Places 365 data set and show that a substantial amount of instances contained in this data set contains semantic overlap with the ID classes of CIFAR-10. And, in Appendix H.6, we evaluate Hopfield Boosting for additional, noticeably different data sets (iCartoonFaces, 4 Shapes, RPC).\n3. Yes, in Appendix H.5 of the original manuscript, we show that Hopfield Boosting identifies data instances from the OOD test data set Places 365 that contain classes of the ID data set CIFAR-10 (e.g., automobiles, dogs, trucks, and airplanes) as ID. This indicates that Hopfield Boosting learns semantically meaningful features.\n4. This information is indeed missing in our manuscript and we will add it. In short: We utilize a ResNet-50 that was pre-trained on ImageNet-1K and is provided by Torchvision. We wanted to keep our method comparable to other outlier exposure methods and therefore closely follow the training protocol of Zhu et al. (2023) (hence, we fine-tune the pre-trained ResNet-50 for four epochs). We mainly chose the model because it is well-documented. The model achieves an ID accuracy of 76.3% after fine-tuning with Hopfield Boosting for four epochs. The pre-trained model achieves an ID accuracy of 76.1%.\n\n**References:**\n\nAnderson et al., Certified Robustness via Locally Biased Randomized Smoothing, PMLR 2022\n\nHendrycks et al., Deep Anomaly Detection with Outlier Exposure, ICLR 2019\n\nMing et al., POEM: Out-of-Distribution Detection with Posterior Sampling, ICML 2022\n\nZhu et al., Diversified Outlier Exposure for Out-of-Distribution Detection via Informative Extrapolation, NeurIPS 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hJfxkBioC3", "reviewer": "Reviewer_BR8F", "summary": "The authors introduce Hopfield Boosting for addressing the out-of-distribution detection task. This algorithm trains a model with two heads: one for normal classification and the other for assigning an out-of-distribution score. The OOD score head maintains a list of in-distribution samples and out-of-distribution samples. It uses a modern Hopfield network to leverage the similarity between the inference sample and stored samples to assign the OOD score.\nThey further improve their performance by first identifying OOD samples that are near the border of the decision boundary using an MHE-based energy function. Then, they use these samples as OOD samples for the OOD score head instead of all auxiliary OOD samples. They evaluated their method on CIFAR-10 and ImageNet-1k and achieved state-of-the-art results.", "questions": "1. What is the relation between ImageNet-21k, which is used as auxiliary OOD data, and the OOD data used for testing? How many semantically similar classes does ImageNet-21k share with each of them? If it shares a considerable number of classes, a test dataset that shares fewer semantically similar classes would be helpful.\n2. ImageNet augmentation: Please explain the augmentation used on in-distribution data for ImageNet. Is it the same as mentioned in line 970? Random cropping to 32x32 for images that are very large, like those in ImageNet-1k, seems unreasonable as 32x32 parts of the image 3. may not contain any meaningful object. Could you provide some examples of in-distribution images post-augmentation?\n4. Can you provide the accuracy of classification head of the network?\n5. What is the reason for cropping images to 32x32 in the ImageNet-1k case? Why not use the more common 224x224 image size?", "limitations": "The authors presented the limitations of their work in section H.6, which could suggest that their method lacks generalization in at least some areas.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Large-scale experiment on ImageNet.\n2. Novel use of Hopfield Boosting.\n3. Good toy examples that clearly illustrate their point.\n4. Low computation requirement.", "weaknesses": "1. Only 32x32 images were used.\n2. OE methods generally require auxiliary OOD data. As a result, evaluation on a dataset that does not have common classes with the auxiliary OOD dataset is crucial to assess the method's generalization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Weaknesses:**\n1. For the experiments comprising ImageNet-1K as the ID data set, we use images with size 224x224. This misunderstanding arises because we missed to report the resolutions we use for the ImageNet benchmark. We apologize for this error. We will make sure that the updated version of the manuscript contains this information.\n2. The reviewer’s analysis is correct and we agree that generalization beyond the classes contained in the auxiliary outlier data set is a highly desirable property for OOD detection methods with OE. It is hard to say with absolute certainty that an OOD test data set does not contain any class overlap with the auxiliary outlier data set, but as far as we know, ImageNet-1K (the auxiliary outlier data set in the CIFAR-10 setting) and SVHN (an OOD test set in the CIFAR-10 setting) do not share common classes.\n\n**Response to Questions:**\n\n1. For our experiments, we selected auxiliary outlier data sets that cover a large and diverse region of the feature space. Optimally, one would have highly specific auxiliary data that is 100%  indicative of the OOD samples. However, since the OOD samples are generally not known a-priori there is the option to fall back on such large and diverse auxiliary data sets. Such datasets then have the advantage that they likely contain samples that share some similarity with the OOD samples used for testing. This is a realistic setting for many applications since it is often possible to easily mine such auxiliary datasets. It is, however, unrealistic to expect to obtain an auxiliary outlier data set that will completely cover the entire OOD region. Therefore, Hopfield Boosting frequently samples data instances close to the decision boundary between the ID and OOD region. This allows Hopfield Boosting to learn a decision boundary that more tightly encapsulates the ID data.\n  To verify the effectiveness of Hopfield Boosting trained on the ImageNet-1K benchmark with ImageNet-21K as auxiliary outlier data on noticeably different data sets, we evaluated the model on three additional OOD data sets: iCartoonFaces, RPC, and FourShapes (we refer to Appendix H.6 of the original manuscript for examples from these data sets). The results are as follows (all numbers in %):\n\n    |               | FPR95 | AUROC |\n    |---------------|-------|-------|\n    | iCartoonFaces | 3.7   | 98.76 |\n    | RPC           | 79.73 | 92.64 |\n    | FourShapes    | 0.0   | 99.84 |\n\n2. We apologize for not reporting that we use images with size 224x224 for the ImageNet-1K benchmark. And, indeed, we also did miss to describe the used transformations, which follow Zhu et al. (2023) for comparability. Namely:\n\n    - Resize to 224x224\n    - RandomCrop 224x224, padding 4\n    - RandomHorizontalFlip\n\n    We will include this list of transformations in the Appendix of the final manuscript.\n\n3. The mean ID accuracy for Hopfield Boosting is 94.02% for CIFAR-10 (ResNet-18), 75.08% for CIFAR-100 (ResNet-18), and 76.3% for ImageNet-1K (ResNet-50). In the PDF attached to the response on the top of this page, we also include an ablation on $\\lambda$, the weight of our energy-based OOD loss. The experiment shows that decreasing $\\lambda$ to $0.1$ will improve the ID accuracy when using Hopfield Boosting (from 94.02% to 94.98%), while only moderately influencing OOD detection performance (the FPR95 metric increases from 0.92% to 1.08%).\n4. We will communicate more clearly that we use a resolution of 224x224 in the ImageNet-1K benchmark.\n\n**References:**\n\nZhu et al., Diversified Outlier Exposure for Out-of-Distribution Detection via Informative Extrapolation, NeurIPS 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for taking the time to provide high-quality feedback. It allowed us to significantly enhance our paper.
In summary, the reviewers appreciated the technical quality and the theoretical depth of our contribution. They also positively acknowledged that Hopfield Boosting obtains good results in OOD detection, achieving substantial improvements to existing methods on three benchmarks (including the large-scale ImageNet-1K benchmark). In terms of questions, the reviewers `DQmw`, `UgW1`, and `BR8F` were interested in the accuracy of the ID classification of Hopfield Boosting. We now report the ID accuracy for Hopfield Boosting and the other methods in our comparison in Table 1 in the document attached to this comment. In short, Hopfield Boosting achieves an ID classification accuracy of 94.02%, while a model trained without Hopfield Boosting obtains 94.80% (averaged over 5 independent training runs). Thanks to the feedback we received from the reviewers, we could also substantially strengthen the experimental evaluation of Hopfield Boosting: In summary, we conducted the following additional experiments:

- We investigate the tradeoff between ID classification accuracy and OOD detection performance of Hopfield Boosting (Figure 1): In the CIFAR-10 benchmark, we ablate on the hyperparameter $\lambda$, which controls the influence of our novel energy-based loss, $\mathcal{L}_{OOD}$, on the total loss. The experiment shows that decreasing $\lambda$ from 0.5 to 0.1 improves the classification error from 5.98% to 5.02%, while only slightly influencing the OOD detection results, increasing the mean FPR95 from 0.92% to 1.08%.

- To measure the contribution of the individual components of Hopfield Boosting (energy-based OOD loss, projection head, weighted sampling) on the total performance, we train models on CIFAR-10 by partially or completely removing said components. The results in Table 2 show that all components contribute considerably to the total performance of Hopfield Boosting, and the mean FPR95 metric worsens from 0.92% to 50.40% when eliminating all components.

- An additional toy experiment visualizes the training process of Hopfield Boosting while also considering the inlier classification task. Figure 4 shows that $\mathcal{L}_{CE}$ is able to separate the two classes well and that $E_b$ still forms a tight decision boundary around the ID data.

- We conduct experiments on how the number of samples stored in the Hopfield memories $\boldsymbol{X}$ and $\boldsymbol{O}$ at inference time influences the OOD detection result on CIFAR-10. Specifically: (a) we measure the performance with 1, 5, 10, 50, 100, 500, 1000, 5000, 10,000, and 50,000 samples in the memory of $\boldsymbol{X}$ and $\boldsymbol{O}$, respectively, and (b) keep the size of $\boldsymbol{X}$ constant at 50,000 while ablating on the number of samples in $\boldsymbol{O}$ (with the same sample sizes as in (a)). The results in Figures 2 and 3 show that Hopfield Boosting is rather robust to the number of samples in the memory, while the setting with 50,000 samples in both memories (which is the setting we use in the experiments in our original manuscript) incurs the least variability.

- We investigate a way to incorporate auxiliary outlier data in the OOD detection methods HE/SHE (Zhang et al., 2023). The results on CIFAR-10 show that extending the OOD detection score of HE, $s_\mathrm{HE}$, by a modern Hopfield energy term containing the auxiliary outlier data is insufficient to obtain OOD detection results that are competitive to Hopfield Boosting (see the response to reviewer `qRFa`).

- We show results of Hopfield Boosting trained on the ImageNet-1K benchmark when evaluated on noticeably different data sets: iCartoonFaces, RPC, and FourShapes (see the response to reviewer `BR8F`; we refer to Appendix H.6 of the original manuscript for examples from these data sets).

The Tables and Figures with the detailed results of the experiments are included in the PDF attached to this comment.

**References:**

Zhang et al., Out-of-Distribution Detection based on In-Distribution Data Patterns Memorization with Modern Hopfield Energy, ICLR 2023

Meta-review: The authors introduce Hopfield Boosting, leveraging modern Hopfield energy to refine the decision boundaries effectively, which enhances the model's ability to differentiate between in-distribution (ID) and OOD data. The method's empirical success is evidenced by substantial improvements in OOD detection metrics across several benchmarks.

Nevertheless, concerns were raised about the paper’s novelty, the impact on ID classification accuracy, and the proposed method's theoretical underpinning. The authors have provided substantial rebuttals, improving the theoretical explanations and demonstrating robustness through additional experiments. The final decision leans towards acceptance, contingent on the authors addressing these remaining concerns in their final revision.
Decision: Accept (poster)