Title: Energy-based Hopfield Boosting for Out-of-Distribution Detection
Venue: NeurIPS 2024
Accepted: yes
Keywords: ood

out-of-distribution

boosting

deep learning

outlier exposure
Abstract: Out-of-distribution (OOD) detection is critical when deploying machine learning models in the real world. Outlier exposure methods, which incorporate auxiliary outlier data in the training process, can drastically improve OOD detection performance compared to approaches without advanced training strategies. We introduce Hopfield Boosting, a boosting approach, which leverages modern Hopfield energy to sharpen the decision boundary between the in-distribution and OOD data. Hopfield Boosting encourages the model to focus on hard-to-distinguish auxiliary outlier examples that lie close to the decision boundary between in-distribution and auxiliary outlier data. Our method achieves a new state-of-the-art in OOD detection with outlier exposure, improving the FPR95 from 2.28 to 0.92 on CIFAR-10, from 11.76 to 7.94 on CIFAR-100, and from 50.74 to 36.60 on ImageNet-1K.

Energy-based Hopfield Boosting for Out-of-Distribution Detection

Abstract

Out-of-distribution (OOD) detection is critical when deploying machine learning models in the real world. Outlier exposure methods, which incorporate auxiliary outlier data in the training process, can drastically improve OOD detection performance compared to approaches without advanced training strategies. We introduce Hopfield Boosting, a boosting approach, which leverages modern Hopfield energy to sharpen the decision boundary between the in-distribution and OOD data. Hopfield Boosting encourages the model to focus on hard-to-distinguish auxiliary outlier examples that lie close to the decision boundary between in-distribution and auxiliary outlier data. Our method achieves a new state-of-the-art in OOD detection with outlier exposure, improving the FPR95 from 2.28 to 0.92 on CIFAR-10, from 11.76 to 7.94 on CIFAR-100, and from 50.74 to 36.60 on ImageNet-1K.

1 Introduction

Out-of-distribution (OOD) detection is crucial when using machine learning systems in the real world (Ruff et al., 2021; Yang et al., 2021; Liu et al., 2021). Deployed models will - sooner or later - encounter inputs that deviate from the training distribution. For example, a system trained to recognize music genres might also hear a sound clip of construction site noise. In the best case, a naive deployment can then result in overly confident predictions. In the worst case, we will get erratic model behavior and completely wrong predictions (Hendrycks & Gimpel, 2017). The purpose of OOD detection is to classify these inputs as OOD, such that the system can then, for instance, notify users that no prediction is possible. In this paper we propose Hopfield Boosting, a novel OOD detection method that leverages the energy component of modern Hopfield networks (MHNs; Ramsauer et al., 2021) and advances the state-of-the-art of OOD detection. This energy represents a measure of dissimilarity between a set of data instances X and a query instance ξ . It is therefore a natural fit for doing OOD detection (as shown in Zhang et al., 2023a).

Hopfield Boosting uses an auxiliary outlier data set (AUX) to boost the model's OOD detection capacity. This allows the training process to learn a boundary around the in-distribution (ID) data, improving the performance at the OOD detection task. In summary, our contributions are as follows:

weight

evaluate

AUX

update

We propose Hopfield Boosting, an OOD detection approach that samples weak learners by using the MHE (Ramsauer et al., 2021).

Hopfield Boosting achieves a new state-of-the-art in OOD detection. It improves the average false positive rate at 95% true positives (FPR95) from 2.28 to 0.92 on CIFAR-10, from 11.38 to 7.94 on CIFAR-100, and from 50.74 to 36.60 on ImageNet-1K.

We provide theoretical background that motivates Hopfield Boosting for OOD detection.

2 Related Work

Some authors (e.g., Bishop, 1994; Roth et al., 2022; Yang et al., 2022) distinguish between anomalies, outliers, and novelties. These distinctions reflect different goals within applications (Ruff et al., 2021). For example, when an anomaly is found, it will usually be removed from the training pipeline. However, when a novelty is found it should be studied. We focus on the detection of samples that are not part of the training distribution and consider sample categorization as a downstream task.

Post-hoc OOD detection. A common and straightforward OOD detection approach is to use a post-hoc strategy, where one employs statistics obtained from a classifier. The perhaps most well known and simplest approach in this class is the Maximum Softmax Probability (MSP; Hendrycks & Gimpel, 2017), where one utilizes p ( y | x ) of the most likely class y given a feature vector x ∈ R D to estimate whether a sample is OOD. Despite good empirical performances, this view is intrinsically limited, since OOD detection should focus on p ( x ) (Morteza & Li, 2022). A wide range of post-hoc OOD detection approaches have been proposed to address the shortcomings of MSP (e.g., Lee et al., 2018; Hendrycks et al., 2019a; Liu et al., 2020; Sun et al., 2021, 2022; Wang et al., 2022; Zhang et al., 2023a; Djurisic et al., 2023; Liu et al., 2023; Xu et al., 2024). Most related to Hopfield Boosting is the work of Zhang et al. (2023a) - to our knowledge, they are the first to apply the MHE to OOD detection. Specifically, they use the ID data set to produce stored patterns and then use a modified version of MHE as the OOD score. While post-hoc approaches can be deployed out of the box on any model, a crucial limitation is that their performance heavily depends on the employed model itself.

Training methods. In contrast to post-hoc strategies, training-based methods modify the training process to improve the model's OOD detection capability (e.g., Hendrycks et al., 2019c; Tack et al., 2020; Sehwag et al., 2021; Du et al., 2022; Hendrycks et al., 2022; Wei et al., 2022a; Ming et al., 2023; Tao et al., 2023; Lu et al., 2024). For example, Self-Supervised Outlier Detection (SSD; Sehwag et al., 2021) leverages contrastive self-supervised learning to train a model for OOD detection.

Auxiliary outlier data and outlier exposure. A third group of OOD detection approaches are outlier exposure (OE) methods. Like Hopfield Boosting, they incorporate AUX data in the training process to improve the detection of OOD samples (e.g., Hendrycks et al., 2019b; Liu et al., 2020; Ming et al., 2022; Zhang et al., 2023b; Wang et al., 2023a; Zhu et al., 2023; Jiang et al., 2024). We provide more detailed discussions on a range of OE methods in Appendix C.1. As far as we know, all OE approaches optimize an objective ( L OOD), which aims at improving the model's discriminative power between ID and OOD data using the AUX data set as a stand-in for the OOD case. Hendrycks et al. (2019b) were the first to use the term OE to describe a more restrictive OE concept. Since their approach uses the MSP for incorporating the AUX data we refer to it as MSP-OE. Further, we refer to the OE approach introduced in Liu et al. (2020) as EBO-OE (to differentiate it from EBO, their post-hoc approach). In general, OE methods conceptualize the AUX data set as a large and diverse data set (e.g., ImageNet for vision tasks). As a consequence, usually, only a small subset of the samples bear semantic similarity to the ID data set - most data points are easily distinguishable from the ID data. Recent approaches therefore actively try to find informative samples for the training. The aim is to refine the decision boundary, ensuring the ID data is more tightly encapsulated (e.g., Chen et al., 2021; Ming et al., 2022). For example, Posterior Sampling-based Outlier Mining (POEM; Ming et al., 2022) selects samples close to the decision boundary using Thompson sampling: They first sample a linear decision boundary between ID and AUX data and then select those data instances which are closest to the sampled decision boundary. Hopfield Boosting also makes use of samples close to the boundary by giving them higher weights for the boosting step.

Continuous modern Hopfield networks. MHNs are energy-based associative memory networks. They advance conventional Hopfield networks (Hopfield, 1984) by introducing continuous queries and states with the MHE as a new energy function. MHE leads to exponential storage capacity, while retrieval is possible with a one-step update (Ramsauer et al., 2021). The update rule of MHNs coincides with attention as it is used in the Transformer (Vaswani et al., 2017). Examples for successful applications of MHNs are Widrich et al. (2020); Fürst et al. (2022); Sanchez-Fernandez et al. (2022); Paischer et al. (2022); Schäfl et al. (2022); Schimunek et al. (2023) and Auer et al. (2023). Section 3.2 gives an introduction to MHE for OOD detection. For further details on MHNs, we refer to Appendix A.

Boosting for classification. Boosting, in particular, AdaBoost (Freund & Schapire, 1995), is an ensemble learning technique for classification. It is designed to focus ensemble members toward data instances that are hard to classify by assigning them higher weights. These challenging instances often lie near the maximum margin hyperplane (Rätsch et al., 2001), akin to support vectors in support vector machines (SVMs; Cortes & Vapnik, 1995). Popular boosting methods include Gradient boosting (Breiman, 1997), LogitBoost (Friedman et al., 2000), and LPBoost (Demiriz et al., 2002).

Radial basis function networks. Radial basis function networks (RBF networks; Moody & Darken, 1989) are function approximators of the form

where ω i are linear weights, µ i are the component means and σ 2 i are the component variances. RBF networks can be described as a weighted linear superposition of N radial basis functions and have previously been used as hypotheses for boosting (Rätsch et al., 2001). If the linear weights are strictly positive, RBF networks can be viewed as an unnormalized weighted mixture of Gaussian distributions p i ( ξ ) = N ( ξ ; µ i , σ 2 i I ) with i = { 1 , . . . , N } . Appendix H.1 explores the connection between RBF networks and MHNs via Gaussian mixtures in more depth. We refer to Bishop (1995) and Müller et al. (1997) for more general information on RBF networks.

3 Method

This section presents Hopfield Boosting: First, we formalize the OOD detection task. Second, we give an overview of the MHE and why it is suitable for OOD detection. Finally, we introduce the AUX-based boosting framework. Figure 1 shows a summary of the Hopfield Boosting concept.

3.1 Classification and OOD Detection

Consider a multi-class classification task denoted as ( X D , Y D , Y ) , where X D ∈ R D × N represents a set of N D -dimensional feature vectors ( x D 1 , x D 2 , . . . , x D N ), which are i.i.d. samples x D i ∼ p ID. Y D ∈ Y N denotes the labels associated with these feature vectors, and Y is a set containing possible classes ( ||Y|| = K signifies the number of distinct classes). We consider observations ξ D ∈ R D that deviate considerably from the data generation p ID ( ξ D ) that defines the 'normality' of our data as OOD. Following Ruff et al. (2021), an observation is OOD if it pertains to the set

Since the probability density of the data generation p ID is in general not known, one needs to estimate p ID ( ξ D ) . In practice, it is common to define an outlier score s ( ξ ) that uses an encoder ϕ , where ξ = ϕ ( ξ D ) . The outlier score should - in the best case - preserve the density ranking. In contrast to a density estimation, the score s ( ξ ) does not have to fulfill all requirements of a probability density (like proper normalization or non-negativity). Given s ( ξ ) and ϕ , OOD detection can be formulated as a binary classification task with the classes ID and OOD:

It is common to choose the threshold γ so that a portion of 95% of ID samples from a previously unseen validation set are correctly classified as ID. However, metrics like the area under the receiver operating characteristic (AUROC) can be directly computed on s ( ξ ) without specifying γ since the AUROC computation sweeps over the threshold.

3.2 Modern Hopfield Energy

The log-sum-exponential ( lse ) function is defined as

where β is the inverse temperature and z ∈ R N is a vector. The lse can be seen as a soft approximation to the maximum function: As β →∞ , the lse approaches max i z i .

Given a set of N d -dimensional stored patterns ( x 1 , x 2 , . . . , x N ) arranged in a data matrix X , and a d -dimensional query ξ , the MHE is defined as

where C = β -1 log N + 1 2 M 2 and M is the largest norm of a pattern: M = max i || x i || . X is also called the memory of the MHN. Intuitively, Equation (5) can be explained as follows: The dot-product within the lse computes a similarity for a given ξ ∈ R d to all patterns in the memory X ∈ R d × N . The lse function aggregates the similarities to form a single value, where the β parameterizes the aggregation operation: If β →∞ , the maximum similarity of ξ to the patterns in X is returned.

To use the MHE for OOD detection, Hopfield Boosting acquires the memory patterns X by feeding raw data instances ( x D 1 , x D 2 , . . . , x D N ) of the ID data set arranged in the data matrix X D ∈ R D × N to an encoder ϕ : R D → R d (e.g., ResNet): x i = ϕ ( x D i ) . We denote the component-wise application of ϕ to the patterns in X D as X = ϕ ( X D ) . Similarly, a raw query ξ D ∈ R D is fed through the encoder to obtain the query pattern: ξ = ϕ ( ξ D ) . One can now use E( ξ ; X ) to estimate whether ξ is ID or OOD: A low energy indicates ξ is ID, and a high energy signifies that ξ is OOD.

3.3 Boosting Framework

Sampling of informative outlier data. Hopfield Boosting uses AUX data to learn a decision boundary between the ID and OOD region during the training. Similar to Chen et al. (2021) and Ming et al. (2022), Hopfield Boosting selects informative outliers close to the ID-OOD decision boundary. For this selection, Hopfield Boosting weights the AUX data similar to AdaBoost (Freund &Schapire, 1995) by sampling data instances close to the decision boundary more frequently. We consider samples close to the decision boundary as weak learners - their nearest neighbors consist of samples from their own class as well as from the foreign class. An individual weak learner represents a classifier that is only slightly better than random guessing (Figure 6). Vice versa, a strong learner can be created by forming an ensemble of a set of weak learners (Figure 2).

We denote the matrix containing the raw AUX data instances ( o D 1 , o D 2 , . . . , o D N ) as O D ∈ R D × M , and the memory containing the encoded AUX patterns as O = ϕ ( O D ) . The boosting process proceeds as follows: There exists a weight ( w 1 , w 2 , . . . , w N ) for each data point in O D and the individual weights are aggregated into the weight vector w t . Hopfield Boosting uses these weights to draw mini-batches O D s from O D for training, where weak learners are sampled more frequently.

We introduce an MHE-based energy function which Hopfield Boosting uses to determine how weak a specific learner ξ is (with higher energy indicating a weaker learner):

where X ∈ R d × N contains ID patterns, O ∈ R d × M contains AUX patterns, and ( X ∥ O ) ∈ R d × ( N + M ) denotes the concatenated data matrix containing the patterns from both X and O . Before computing E b , we normalize the feature vectors in X , O , and ξ to unit length. Figure 3 displays the energy landscape of E b ( ξ ; X , O ) using exemplary data on a 3-dimensional sphere. E b is maximal at the decision boundary between ID and AUX data and decreases with increasing distance from the decision boundary in both directions.

As we show in our theoretical discussion in Appendix G, when modeling the class-conditional densities of the ID and AUX data set as mixtures of Gaussian distributions

with equal class priors p ( ID ) = p ( AUX ) = 1 / 2 and normalized patterns || x i || = 1 and || o i || = 1 , we obtain E b ( ξ ; X , O ) C = β -1 log( p ( ID | ξ ) · p ( AUX | ξ )) , where C = denotes equality up to an irrelevant additive constant. The exponential of E b is the variance of a Bernoulli random variable with the outcomes { ID , AUX } conditioned on ξ . Thus, according to E b , the weak learners are situated at locations where the model defined in Equations (7) and (8) is uncertain.

Given a set of query values ( ξ 1 , ξ 2 , . . . , ξ n ) assembled in a query matrix Ξ ∈ R d × n , we denote a vector of energies e ∈ R n with e i = E b ( ξ i ; X , O ) as

To calculate the weights w t +1 , we use the memory of AUX patterns as a query matrix Ξ = O and compute the respective energies E b of those patterns. The resulting energy vector E b ( Ξ ; X , O ) is then normalized by a softmax . This computation provides the updated weights:

Appendix J provides theoretical background on how informative samples close to the decision boundary are beneficial for training an OOD detector.

Overview

(a) All Data

(c)Hopfield Boosting

2.0

2.0

2.0

10

inlier

outlier

-10

-1.5

-1.5

-1.5

2

Q

1

2

-1

0

1

2

-1

0

1

2

Training the model with MHE. In this section, we introduce how Hopfield Boosting uses the sampled weak learners to improve the detection of patterns outside the training distribution. We follow the established training method for OE (Hendrycks et al., 2019b; Liu et al., 2020; Ming et al., 2022): Train a classifier on the ID data using the standard cross-entropy loss and add an OOD loss that uses the AUX data set to sharpen the decision boundary between the ID and OOD regions. Formally, this yields the loss

where λ is a hyperparamter indicating the relative importance of L OOD. Hopfield Boosting explicitly minimizes E b (which is also the energy function Hopfield Boosting uses to sample weak learners). Given the weight vector w t , and the data sets X D and O D , we obtain a mini-batch X D s containing N samples from X D by uniform sampling, and a mini-batch of N weak learners O D s from O D by sampling according to w t with replacement. We then feed the respective mini-batches into the neural network ϕ base to create a latent feature (in our experiments, we always use the feature of the penultimate layer of a ResNet). Our proposed approach then uses two heads:

A linear classification head that maps the latent feature to the class logits for L CE.

A 2-layer MLP ϕ proj maps the features from the penultimate layer to the output for L OOD.

Hopfield Boosting computes L OOD on the representations it obtains from ϕ = ϕ proj ◦ ϕ base as follows:

where the memories X s and O s contain the encodings of the sampled data instances: X s = ϕ ( X D s ) and O s = ϕ ( O D s ) . The sum is taken over the observations ξ , which are drawn from ( X s ∥ O s ) . Hopfield Boosting computes L OOD for each mini-batch by first calculating the pairwise similarity matrix between the patterns in the mini-batch, followed by determining the E b values of the individual observations ξ , and, finally a mean reduction. To the best of our knowledge, Hopfield Boosting is the first method that uses Hopfield networks in this way to train a deep neural network. We note that there is a relation between Hopfield Boosting and SVMs with an RBF kernel (see Appendix H.3). However, the optimization procedure of SVMs is in general not differentiable. In contrast, our novel energy function is fully differentiable. This allows us to use it to train neural networks.

Summary. Algorithm 1 provides an outline of Hopfield Boosting. Each iteration t consists of three main steps: 1. weight, 2. evaluate, and 3. update. First, Hopfield Boosting samples a mini-batch from the ID data and weights the AUX data by sampling a mini-batch according to w t . Second, Hopfield Boosting evaluates the composite loss on the sampled mini-batch. Third, Hopfield Boosting updates the model parameters and, every N -th step, also the sampling weights for the AUX data set w t +1 .

Inference. At inference time, the OOD score s ( ξ ) is

For computing s ( ξ ) , Hopfield Boosting uses the 50,000 random samples from the ID and AUX data sets, respectively. As we show in Appendix I.8, this step entails only a very moderate computational overhead in relation to a complete forward pass (e.g., an overhead of 7.5% for ResNet-18 on an NVIDIA Titan V GPU with 50,000 patterns stored in each of the memories X and O ). Weadditionally experimented with using only lse( β, X T ξ ) as a score, which also gives reasonable results. However, the approach in Equation (13) has turned out to be superior. Equation (13) uses information from both ID and AUX samples. This can, for example, be beneficial for handling query patterns ξ that are dissimilar from both the memory patterns in X as well as from the memory patterns in O .

3.4 Comparison of Hopfield Boosting to HE and SHE

Zhang et al. (2023a) propose two post-hoc methods for OOD detection with MHE:'Hopfield Energy' (HE) and 'Simplified Hopfield Energy' (SHE). In contrast to Hopfield Boosting, HE and SHE do not use AUX data to get a better boundary between ID and OOD data. Rather, their methods evaluate the MHE on ID patterns only to determine whether a sample is ID or OOD. Additional differences include the selection of patterns stored in the memory or the normalization of the patterns. The OE process of Hopfield Boosting drastically improves the OOD detection performance compared to HE and SHE. We verify that the unique contributions of Hopfield Boosting (like the energy-based loss and the boosting process) are responsible for the superior performance with two extensions of HE that include AUX data (the comparison can be found in Appendix I.9). For further information on the differences to HE and SHE, we refer to Appendix H.4.

4 Experiments

4.1 Toy Example

This section presents a toy example illustrating the main intuitions behind Hopfield Boosting. For the sake of clarity, the toy example does not consider the inlier classification task that would induce secondary processes, which would obscure the explanations. Formally, we do not consider the first term on the right-hand side of Equation (11). For further toy examples, we refer to Appendix F.

Figure 2 demonstrates how the weighting in Hopfield Boosting allows good estimations of the decision boundary, even if Hopfield Boosting only samples a small number of weak learners. This is advantageous because the AUX data set contains a large number of data instances that are uninformative for the OOD detection task. For small, low dimensional data, one can always use all the data to compute E b (Figure 2, a). For large problems (like in Ming et al., 2022), this strategy is difficult, and the naive solution of uniformly sampling N data points would also not work. This will yield many uninformative points (Figure 2, b). When using Hopfield Boosting and sampling N weak learners according to w t , the result better approximates the decision boundary of the full data (Figure 2, c).

Metric

HB (ours)

DOS

DOE

DivOE

DAL

MixOE

POEM

EBO-OE

MSP-OE

FPR95

↓

0

.

23

±

0

.

08

3

.

09

±

0

.

75

1

.

97

±

0

.

58

6

.

21

±

0

.

84

1

.

25

±

0

.

62

27

.

54

±

2

.

46

1

.

48

±

0

.

68

2

.

66

±

0

.

91

4

.

31

±

1

.

10

SVHN

AUROC

↑

99

.

57

±

0

.

06

99

.

15

±

0

.

22

99

.

60

±

0

.

13

98

.

53

±

0

.

08

99

.

61

±

0

.

15

95

.

37

±

0

.

44

99

.

33

±

0

.

15

99

.

15

±

0

.

23

99

.

20

±

0

.

15

FPR95

↓

0

.

82

±

0

.

17

3

.

66

±

0

.

98

3

.

22

±

0

.

45

1

.

88

±

0

.

25

4

.

17

±

0

.

27

0

.

14

±

0

.

07

4

.

02

±

0

.

91

6

.

82

±

0

.

74

7

.

02

±

1

.

14

LSUN-Crop

AUROC

↑

99

.

40

±

0

.

04

99

.

04

±

0

.

20

99

.

30

±

0

.

12

99

.

50

±

0

.

02

99

.

13

±

0

.

02

99

.

61

±

0

.

11

98

.

89

±

0

.

15

98

.

43

±

0

.

10

98

.

83

±

0

.

15

FPR95

↓

0

.

00

±

0

.

00

0

.

00

±

0

.

00

0

.

00

±

0

.

00

0

.

00

±

0

.

00

0

.

00

±

0

.

00

0

.

16

±

0

.

17

0

.

00

±

0

.

00

0

.

00

±

0

.

00

0

.

00

±

0

.

00

LSUN-Resize

AUROC

↑

99

.

98

±

0

.

02

99

.

99

±

0

.

01

100

.

00

±

0

.

00

99

.

89

±

0

.

05

99

.

92

±

0

.

05

99

.

89

±

0

.

06

99

.

88

±

0

.

12

99

.

98

±

0

.

02

99

.

96

±

0

.

00

FPR95

↓

0

.

16

±

0

.

02

1

.

28

±

0

.

20

2

.

75

±

0

.

57

1

.

20

±

0

.

11

0

.

95

±

0

.

13

4

.

68

±

0

.

22

0

.

49

±

0

.

04

1

.

11

±

0

.

17

2

.

29

±

0

.

16

Textures

AUROC

↑

99

.

84

±

0

.

01

99

.

63

±

0

.

04

99

.

35

±

0

.

12

99

.

59

±

0

.

02

99

.

74

±

0

.

01

98

.

91

±

0

.

07

99

.

72

±

0

.

05

99

.

61

±

0

.

02

99

.

57

±

0

.

01

FPR95

↓

0

.

00

±

0

.

00

0

.

00

±

0

.

00

0

.

00

±

0

.

00

0

.

00

±

0

.

00

0

.

00

±

0

.

00

0

.

17

±

0

.

12

0

.

00

±

0

.

00

0

.

00

±

0

.

00

0

.

00

±

0

.

00

iSUN

AUROC

↑

99

.

97

±

0

.

02

99

.

99

±

0

.

01

100

.

00

±

0

.

00

99

.

88

±

0

.

05

99

.

93

±

0

.

04

99

.

87

±

0

.

05

99

.

87

±

0

.

12

99

.

98

±

0

.

01

99

.

96

±

0

.

00

FPR95

↓

4

.

28

±

0

.

23

12

.

26

±

0

.

97

19

.

72

±

2

.

39

13

.

70

±

0

.

50

14

.

22

±

0

.

51

16

.

30

±

1

.

09

7

.

70

±

0

.

68

11

.

77

±

0

.

68

21

.

42

±

0

.

88

Places 365

AUROC

↑

98

.

51

±

0

.

10

96

.

63

±

0

.

43

95

.

06

±

0

.

72

96

.

95

±

0

.

09

96

.

77

±

0

.

07

96

.

92

±

0

.

22

97

.

56

±

0

.

26

96

.

39

±

0

.

30

95

.

91

±

0

.

17

FPR95

↓

0

.

92

3

.

38

4

.

61

3

.

83

3

.

43

8

.

17

2

.

28

3

.

73

5

.

84

Mean

AUROC

↑

99

.

55

99

.

07

98

.

88

99

.

06

99

.

18

98

.

43

99

.

21

98

.

92

98

.

90

±

0

.

09

±

0

.

13

±

0

.

12

±

0

.

17

±

0

.

05

±

1

.

50

±

1

.

30

±

0

.

35

±

0

.

23

ID

↑

94

.

02

94

.

74

94

.

93

94

.

72

95

.

11

96

.

60

89

.

20

91

.

32

94

.

83

4.2 Data & Setup

CIFAR-10 & CIFAR-100. Our training and evaluation proceeds as follows: We train Hopfield Boosting with ResNet-18 (He et al., 2016) on the CIFAR-10 and CIFAR-100 data sets (Krizhevsky, 2009), respectively. In these settings, we use ImageNet-RC (Chrabaszcz et al., 2017) (a low-resolution version of ImageNet) as the AUX data set. For testing the OOD detection performance, we use the data sets SVHN (Street View House Numbers) (Netzer et al., 2011), Textures (Cimpoi et al., 2014), iSUN (Xu et al., 2015), Places 365 (López-Cifuentes et al., 2020), and two versions of the LSUN data set (Yu et al., 2015) - one where the images are cropped, and one where they are resized to match the resolution of the CIFAR data sets (32x32 pixels). We refer to the two LSUN data sets as LSUN-Crop and LSUN-Resize, respectively. We compute the scores s ( ξ ) as described in Equation (13) and then evaluate the discriminative power of s ( ξ ) between CIFAR and the respective OOD data set using the FPR95 and the AUROC. We use a validation process with different OOD data for model selection. Specifically, we validate the model on MNIST (LeCun et al., 1998), and ImageNet-RC with different pre-processing than in training (resize to 32x32 pixels instead of crop to 32x32 pixels), as well as Gaussian and uniform noise.

ImageNet-1K. We evaluate Hopfield Boosting on the large-scale benchmark: We use ImageNet-1K (Russakovsky et al., 2015) as ID data set and ImageNet-21K (Ridnik et al., 2021) as AUX data set. The OOD test data sets are Textures (Cimpoi et al., 2014), SUN (Xu et al., 2015), Places 365 (LópezCifuentes et al., 2020), and iNaturalist (Van Horn et al., 2018). In this setting, all images are scaled to a resolution of 224x224. To keep our method comparable to other OE methods, we closely follow the training and evaluation protocol of (Zhu et al., 2023). This implies htat we fine-tune a ResNet-50 that was pre-trained on the ImageNet-1K ID classification task (as provided by TorchVision, 2016).

Weighted Sampling

✓

✗

✗

✗

Projection Head

✓

✓

✗

✗

L

OOD

✓

✓

✓

✗

FPR95

↓

0

.

23

±

0

.

06

0

.

70

±

0

.

13

1

.

01

±

0

.

27

45

.

65

±

13

.

51

SVHN

AUROC

↑

99

.

57

±

0

.

06

99

.

55

±

0

.

08

99

.

21

±

0

.

21

90

.

99

±

2

.

68

FPR95

↓

0

.

28

±

0

.

05

1

.

58

±

0

.

31

2

.

22

±

0

.

36

28

.

59

±

4

.

33

LSUN-Crop

AUROC

↑

99

.

40

±

0

.

05

99

.

24

±

0

.

10

98

.

28

±

0

.

25

94

.

08

±

0

.

86

FPR95

↓

0

.

00

±

0

.

02

0

.

00

±

0

.

00

0

.

00

±

0

.

00

50

.

30

±

7

.

16

LSUN-Resize

AUROC

↑

99

.

98

±

0

.

02

99

.

98

±

0

.

01

99

.

98

±

0

.

01

89

.

15

±

1

.

89

FPR95

↓

0

.

16

±

0

.

01

0

.

26

±

0

.

06

0

.

38

±

0

.

10

49

.

36

±

1

.

63

Textures

AUROC

↑

99

.

85

±

0

.

01

99

.

81

±

0

.

02

99

.

70

±

0

.

06

89

.

58

±

0

.

52

FPR95

↓

0

.

00

±

0

.

02

0

.

00

±

0

.

00

0

.

00

±

0

.

00

51

.

08

±

6

.

38

iSUN

AUROC

↑

99

.

97

±

0

.

02

99

.

99

±

0

.

00

99

.

98

±

0

.

01

88

.

89

±

1

.

74

FPR95

↓

4

.

28

±

0

.

11

6

.

20

±

0

.

21

8

.

73

±

1

.

06

77

.

44

±

0

.

81

Places 365

AUROC

↑

98

.

51

±

0

.

11

97

.

68

±

0

.

21

94

.

77

±

0

.

81

78

.

30

±

0

.

83

FPR95

↓

0

.

92

1

.

46

2

.

06

50

.

40

Mean

AUROC

↑

99

.

55

99

.

38

98

.

65

88

.

50

Baselines. As mentioned earlier, previous works offer vast experimental evidence that OE methods offer superior OOD detection compared to methods without OE (see e.g., Ming et al., 2022; Wang et al., 2023a). Our experiments in Appendix I.14 confirm this. Thus, we focus on a comprehensive comparison of Hopfield Boosting to eight OE methods: MSP-OE (Hendrycks et al., 2019b), EBO-OE (Liu et al., 2020), POEM (Ming et al., 2022), MixOE (Zhang et al., 2023b), DAL (Wang et al., 2023a), DivOE (Zhu et al., 2023), DOE (Wang et al., 2023b) and DOS (Jiang et al., 2024).

Training setup. The network trains for 100 epochs (CIFAR-10/100) or 4 epochs (ImageNet-1K), respectively. In each epoch, the model processes the entire ID data set and a selection of AUX samples (sampled according to w t ). We sample mini-batches of size 128 per data set, resulting in a combined batch size of 256. We evaluate the composite loss from Equation (11) for each resulting mini-batch and update the model accordingly. After an epoch, we update the sample weights, yielding w t +1 . For efficiency reasons, we only compute the weights for 500,000 AUX data instances ( ∼ 40% of ImageNet), which we denote as Ξ . The weights of the remaining samples are set to 0. During the sample weight update, Hopfield Boosting does not compute gradients or update model parameters. The update of the sample weights w t +1 proceeds as follows: First, we fill the memories X and O with 50,000 ID samples and 50,000 AUX samples, respectively. Second, we use the obtained X and O to get the energy E b ( Ξ ; X , O ) for each of the 500,000 AUX samples in Ξ and compute w t +1 according to Equation (10). In the following epoch, Hopfield Boosting samples the mini-batches O D s according to w t +1 with replacement. To allow the storage of even more patterns in the Hopfield memory during the weight update process, one could incorporate a vector similarity engine (e.g., Douze et al., 2024) into the process. This would potentially allow a less noisy estimate of the sample weights. For the sake of simplicity, we did not opt to do this in our implementation of Hopfield Boosting. As we show in section 4.3, Hopfield Boosting achieves state-of-the-art OOD detection results and can scale to large datasets (ImageNet-1K) even without access to a similarity engine.

Hyperparameters & Model Selection. Like Yang et al. (2022), we use SGD with an initial learning rate of 0 . 1 and a weight decay of 5 · 10 -4 . We decrease the learning rate during the training process with a cosine schedule (Loshchilov & Hutter, 2016). Appendix I.2 describes the image transformations and pre-processing. We apply optimizer, weight decay, learning rate, scheduler, and transformations consistently to all OOD detection methods of the comparison. For training Hopfield Boosting, we use a single value for β throughout the training and evaluation process and for all OOD data sets. For model selection, we use a grid search with λ , chosen from the set { 0 . 1 , 0 . 25 , 0 . 5 , 1 . 0 } , and β , chosen from the set { 2 , 4 , 8 , 16 , 32 } . From these hyperparameter configurations, we select the model with the lowest mean FPR95 metric (where the mean is taken over the validation OOD data sets) and do not consider the ID classification accuracy for model selection. In our experiments, β = 4 and λ = 0 . 5 yields the best results for CIFAR-10 and CIFAR-100. For ImageNet-1K, we set β = 32 and λ = 0 . 25 .

4.3 Results & Discussion

Table 1 summarizes the results for CIFAR-10. Hopfield Boosting achieves equal or better performance compared to the other methods regarding the FPR95 metric for all OOD data sets. It surpasses POEM (the previously best OOD detection approach with OE in our comparison), improving the mean FPR95 metric from 2.28 to 0.92. On CIFAR-100 (Appendix I.1), Hopfield Boosting improves the mean FPR95 metric from 11.76 to 7.94. It is notable that all methods achieve perfect FPR95 results on the LSUN-Resize and iSUN data sets. This is somewhat problematic since there exists evidence that the LSUN-Resize data set can give misleading results due to image artifacts resulting from the resizing procedure (Tack et al., 2020; Yang et al., 2022). We hypothesize that a similar issue exists with the iSUN data set, as in our experiments, LSUN-Resize and iSUN behave very similarly.

On ImageNet-1K (Table 2), Hopfield Boosting surpasses all methods in our comparison in terms of both mean FPR95 and mean AUROC. Compared to POEM (the previously best method) Hopfield Boosting improves the mean FPR95 from 50.74 to 36.60. This demonstrates that Hopfield Boosting scales very favourably to large-scale settings.

We observe that all methods tested perform worst on the Places 365 data set. To gain more insights regarding this behavior, we look at the data instances from the Places 365 data set that Hopfield Boosting trained on CIFAR-10 most confidently classifies as in-distribution (i.e., which receive the highest scores s ( ξ ) ). Visual inspection shows that among those images, a large portion contains objects from semantic classes included in CIFAR-10 (e.g., airplanes, horses, automobiles). We refer to Appendix I.6 for more details.

We evaluate the performance of the following 3 ablated training procedures on the CIFAR-10 benchmark to gauge the importance of the individual contributions of Hopfield Boosting: (a) Random sampling instead of weighted sampling, (b) Random sampling instead of weighted sampling and no projection head, (c) No application of L OOD. The results (Table 3) show that all contributions (i.e, weighted sampling, the projection head, and L OOD) are important factors for Hopfield Boosting 's performance. For additional ablations, we refer to Appendix I.

When subjecting Hopfield Boosting to data sets that were designed to show the weakness of OOD detection approaches (Appendix I.7), we identify instances where a substantial number of outliers are wrongly classified as inliers. Testing with EBO-OE yields comparable outcomes, indicating that this phenomenon extends beyond Hopfield Boosting.

5 Limitations

Lastly, we would like to discuss two limitations that we found: First, we see an opportunity to improve the evaluation procedure for OOD detection. Specifically, it remains unclear how reliably the performance on specific data sets can indicate the general ability to detect OOD inputs. Our results from iSUN and LSUN-Resize (Section 4.3) indicate that issues like image artifacts in data sets greatly influence model evaluation. Second, although OE-based approaches improve the OOD detection capability, their reliance on AUX data can limit their applicability. For one, the selection of the AUX data is crucial (since it determines the characteristics of the decision boundary surrounding the inlier data). Furthermore, the use of AUX data can be prohibitive in domains where only a few or no outliers at all are available for training the model.

6 Conclusions

We introduce Hopfield Boosting: an approach for OOD detection with OE. Hopfield Boosting uses an energy term to boost a classifier between inlier and outlier data by sampling weak learners that are close to the decision boundary. We illustrate how Hopfield Boosting shapes the energy surface to form a decision boundary. Additionally, we demonstrate how the boosting mechanism creates a sharper decision boundary than with random sampling. We compare Hopfield Boosting to eight modern OOD detection approaches using OE. Overall, Hopfield Boosting shows the best results.

References

Abbott, L. F. and Arian, Y. Storage capacity of generalized networks. Physical review A , 36(10): 5091, 1987.

Ahn, S., Korattikara, A., and Welling, M. Bayesian posterior sampling via stochastic gradient Fisher scoring. In Proceedings of the 29th International Coference on International Conference on Machine Learning , pp. 1771-1778, Madison, WI, USA, 2012. Omnipress.

Anderson, B. G. and Sojoudi, S. Certified robustness via locally biased randomized smoothing. In Learning for Dynamics and Control Conference , pp. 207-220. PMLR, 2022.

Auer, A., Gauch, M., Klotz, D., and Hochreiter, S. Conformal prediction for time series with modern hopfield networks. Advances in Neural Information Processing Systems , 36:56027-56074, 2023.

Baldi, P. and Venkatesh, S. S. Number of stable points for spin-glasses and neural networks of higher orders. Physical Review Letters , 58(9):913, 1987.

Bishop, C. Neural Networks for Pattern Recognition . Oxford University Press, 1995.

Bishop, C. M. Novelty detection and neural network validation. IEE Proceedings-Vision, Image and Signal processing , 141(4):217-222, 1994.

Breiman, L. Arcing the edge. Technical report, Citeseer, 1997.

Caputo, B. and Niemann, H. Storage capacity of kernel associative memories. In Artificial Neural Networks-ICANN 2002: International Conference Madrid, Spain, August 28-30, 2002 Proceedings 12 , pp. 51-56. Springer, 2002.

Chen, H., Lee, Y., Sun, G., Lee, H., Maxwell, T., and Giles, C. L. High order correlation model for associative memory. In AIP Conference Proceedings , volume 151, pp. 86-99. American Institute of Physics, 1986.

Chen, J., Li, Y ., Wu, X., Liang, Y ., and Jha, S. Atom: Robustifying out-of-distribution detection using outlier mining. In Machine Learning and Knowledge Discovery in Databases. Research Track: European Conference, ECML PKDD 2021, Bilbao, Spain, September 13-17, 2021, Proceedings, Part III 21 , pp. 430-445. Springer, 2021.

Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pp. 1597-1607. PMLR, 2020.

Chrabaszcz, P., Loshchilov, I., and Hutter, F. A downsampled variant of imagenet as an alternative to the CIFAR datasets. arXiv preprint arXiv:1707.08819 , 2017.

Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., , and Vedaldi, A. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , 2014.

Cortes, C. and Vapnik, V. Support-vector networks. Machine learning , 20(3):273-297, 1995.

Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q. V. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops , pp. 702-703, 2020.

Demiriz, A., Bennett, K. P., and Shawe-Taylor, J. Linear programming boosting via column generation. Machine Learning , 46:225-254, 2002.

Djurisic, A., Bozanic, N., Ashok, A., and Liu, R. Extremely simple activation shaping for out-ofdistribution detection. In The Eleventh International Conference on Learning Representations , 2023.

Douze, M., Guzhva, A., Deng, C., Johnson, J., Szilvasy, G., Mazaré, P.-E., Lomeli, M., Hosseini, L., and Jégou, H. The faiss library. 2024.

Du, X., Wang, Z., Cai, M., and Li, Y. Vos: Learning what you don't know by virtual outlier synthesis. arXiv preprint arXiv:2202.01197 , 2022.

Freund, Y. and Schapire, R. E. A decision-theoretic generalization of on-line learning and an application to boosting. In Computational Learning Theory: Eurocolt '95 , pp. 23-37. SpringerVerlag, 1995.

Friedman, J., Hastie, T., and Tibshirani, R. Additive logistic regression: a statistical view of boosting (with discussion and a rejoinder by the authors). The annals of statistics , 28(2):337-407, 2000.

Fürst, A., Rumetshofer, E., Lehner, J., Tran, V. T., Tang, F., Ramsauer, H., Kreil, D., Kopp, M., Klambauer, G., Bitto, A., et al. CLOOB: Modern Hopfield networks with InfoLOOB outperform clip. Advances in neural information processing systems , 35:20450-20468, 2022.

Gardner, E. Multiconnected neural network models. Journal of Physics A: Mathematical and General , 20(11):3453, 1987.

He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 770-778, 2016.

He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 9729-9738, 2020.

Hendrycks, D. and Gimpel, K. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations , 2017. URL https://openreview.net/forum?id=Hkg4TI9xl .

Hendrycks, D., Basart, S., Mazeika, M., Zou, A., Kwon, J., Mostajabi, M., Steinhardt, J., and Song, D. Scaling out-of-distribution detection for real-world settings. arXiv preprint arXiv:1911.11132 , 2019a.

Hendrycks, D., Mazeika, M., and Dietterich, T. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations , 2019b. URL https://openreview. net/forum?id=HyxCxhRcY7 .

Hendrycks, D., Mazeika, M., Kadavath, S., and Song, D. Using self-supervised learning can improve model robustness and uncertainty. Advances in neural information processing systems , 32, 2019c.

Hendrycks, D., Zou, A., Mazeika, M., Tang, L., Li, B., Song, D., and Steinhardt, J. Pixmix: Dreamlike pictures comprehensively improve safety measures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 16783-16792, 2022.

Hopfield, J. J. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences , 79(8):2554-2558, 1982.

Hopfield, J. J. Neurons with graded response have collective computational properties like those of two-state neurons. Proceedings of the National Academy of Sciences , 81(10):3088-3092, 1984. doi: 10.1073/pnas.81.10.3088.

Horn, D. and Usher, M. Capacities of multiconnected memory models. Journal de Physique , 49(3): 389-395, 1988.

Hu, J. Y.-C., Chang, P.-H., Luo, H., Chen, H.-Y., Li, W., Wang, W.-P., and Liu, H. Outlier-efficient hopfield layers for large transformer-based models. In Forty-first International Conference on Machine Learning , 2024.

Isola, P., Xiao, J., Torralba, A., and Oliva, A. What makes an image memorable? In CVPR 2011 , pp. 145-152. IEEE, 2011.

Jiang, W., Cheng, H., Chen, M., Wang, C., and Wei, H. DOS: Diverse outlier sampling for out-ofdistribution detection. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=iriEqxFB4y .

Krizhevsky, A. Learning multiple layers of features from tiny images. Master's thesis, Deptartment of Computer Science, University of Toronto, 2009.

Krotov, D. and Hopfield, J. J. Dense associative memory for pattern recognition. In Lee, D. D., Sugiyama, M., Luxburg, U. V., Guyon, I., and Garnett, R. (eds.), Advances in Neural Information Processing Systems , pp. 1172-1180. Curran Associates, Inc., 2016.

LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Lee, K., Lee, K., Lee, H., and Shin, J. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., CesaBianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper/2018/file/ abdeb6f575ac5c6676b747bca8d09cc2-Paper.pdf .

Liu, J., Shen, Z., He, Y., Zhang, X., Xu, R., Yu, H., and Cui, P. Towards out-of-distribution generalization: A survey. arXiv preprint arXiv:2108.13624 , 2021.

Liu, W., Wang, X., Owens, J., and Li, Y. Energy-based out-of-distribution detection. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems , volume 33, pp. 21464-21475. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/ f5496252609c43eb8a3d147ab9b9c006-Paper.pdf .

Liu, X., Lochman, Y., and Zach, C. Gen: Pushing the limits of softmax-based out-of-distribution detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 23946-23955, 2023.

López-Cifuentes, A., Escudero-Vinolo, M., Bescós, J., and García-Martín, Á. Semantic-aware scene recognition. Pattern Recognition , 102:107256, 2020.

Loshchilov, I. and Hutter, F. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 , 2016.

Lu, H., Gong, D., Wang, S., Xue, J., Yao, L., and Moore, K. Learning with mixture of prototypes for out-of-distribution detection. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=uNkKaD3MCs .

McInnes, L., Healy, J., and Melville, J. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 , 2018.

Ming, Y., Fan, Y., and Li, Y. POEM: Out-of-distribution detection with posterior sampling. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pp. 15650-15665. PMLR, 17-23 Jul 2022. URL https://proceedings. mlr.press/v162/ming22a.html .

Ming, Y., Sun, Y., Dia, O., and Li, Y . How to exploit hyperspherical embeddings for out-of-distribution detection? In The Eleventh International Conference on Learning Representations , 2023.

Moody, J. and Darken, C. J. Fast learning in networks of locally-tuned processing units. Neural computation , 1(2):281-294, 1989.

Morteza, P. and Li, Y. Provable guarantees for understanding out-of-distribution detection. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pp. 7831-7840, 2022.

Müller, K.-R., Smola, A. J., Rätsch, G., Schölkopf, B., Kohlmorgen, J., and Vapnik, V. Predicting time series with support vector machines. In International conference on artificial neural networks , pp. 999-1004. Springer, 1997.

Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y. Reading digits in natural images with unsupervised feature learning. 2011.

Oord, A. v. d., Li, Y., and Vinyals, O. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Paischer, F., Adler, T., Patil, V., Bitto-Nemling, A., Holzleitner, M., Lehner, S., Eghbal-Zadeh, H., and Hochreiter, S. History compression via language models in reinforcement learning. In International Conference on Machine Learning , pp. 17156-17185. PMLR, 2022.

Park, G. Y., Kim, J., Kim, B., Lee, S. W., and Ye, J. C. Energy-based cross attention for Bayesian context update in text-to-image diffusion models. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Psaltis, D. and Park, C. H. Nonlinear discriminant functions and associative memories. In AIP conference Proceedings , volume 151, pp. 370-375. American Institute of Physics, 1986.

Ramsauer, H., Schäfl, B., Lehner, J., Seidl, P., Widrich, M., Gruber, L., Holzleitner, M., Pavlovi´ c, M., Sandve, G. K., Greiff, V., Kreil, D., Kopp, M., Klambauer, G., Brandstetter, J., and Hochreiter, S. Hopfield networks is all you need. In 9th International Conference on Learning Representations (ICLR) , 2021. URL https://openreview.net/forum?id=tL89RnzIiCd .

Rätsch, G., Onoda, T., and Müller, K.-R. Soft margins for AdaBoost. Machine learning , 42:287-320, 2001.

Ridnik, T., Ben-Baruch, E., Noy, A., and Zelnik-Manor, L. Imagenet-21k pretraining for the masses. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1) , 2021.

Roth, K., Pemula, L., Zepeda, J., Schölkopf, B., Brox, T., and Gehler, P. Towards total recall in industrial anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 14318-14328, 2022.

Ruff, L., Kauffmann, J. R., Vandermeulen, R. A., Montavon, G., Samek, W., Kloft, M., Dietterich, T. G., and Müller, K.-R. A unifying review of deep and shallow anomaly detection. Proceedings of the IEEE , 109(5):756-795, 2021.

Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. Imagenet large scale visual recognition challenge. International journal of computer vision , 115:211-252, 2015.

Saleh, R. A. and Saleh, A. Statistical properties of the log-cosh loss function used in machine learning. arXiv preprint arXiv:2208.04564 , 2022.

Sanchez-Fernandez, A., Rumetshofer, E., Hochreiter, S., and Klambauer, G. CLOOME: a new search engine unlocks bioimaging databases for queries with chemical structures. bioRxiv , pp. 2022-11, 2022.

Schäfl, B., Gruber, L., Bitto-Nemling, A., and Hochreiter, S. Hopular: Modern Hopfield networks for tabular data. arXiv preprint arXiv:2206.00664 , 2022.

Schimunek, J., Seidl, P., Friedrich, L., Kuhn, D., Rippmann, F., Hochreiter, S., and Klambauer, G. Context-enriched molecule representations improve few-shot drug discovery. In The Eleventh International Conference on Learning Representations , 2023.

Sehwag, V., Chiang, M., and Mittal, P. Ssd: A unified framework for self-supervised outlier detection. arXiv preprint arXiv:2103.12051 , 2021.

smeschke. Four Shapes. https://www.kaggle.com/datasets/smeschke/four-shapes/ , 2018. URL https://www.kaggle.com/datasets/smeschke/four-shapes/ .

Sun, Y., Guo, C., and Li, Y. React: Out-of-distribution detection with rectified activations. Advances in Neural Information Processing Systems , 34:144-157, 2021.

Sun, Y., Ming, Y., Zhu, X., and Li, Y. Out-of-distribution detection with deep nearest neighbors. In International Conference on Machine Learning , pp. 20827-20840. PMLR, 2022.

Tack, J., Mo, S., Jeong, J., and Shin, J. Csi: Novelty detection via contrastive learning on distributionally shifted instances. Advances in neural information processing systems , 33:11839-11852, 2020.

Tao, L., Du, X., Zhu, X., and Li, Y. Non-parametric outlier synthesis. arXiv preprint arXiv:2303.02966 , 2023.

Teh, Y. W., Thiery, A. H., and Vollmer, S. J. Consistency and fluctuations for stochastic gradient Langevin dynamics. J. Mach. Learn. Res. , 17(1):193-225, 2016.

TorchVision. Torchvision: Pytorch's computer vision library. https://github.com/pytorch/ vision , 2016.

Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., and Belongie, S. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 8769-8778, 2018.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 30 , pp. 5998-6008. Curran Associates, Inc., 2017.

Wang, H., Li, Z., Feng, L., and Zhang, W. Vim: Out-of-distribution with virtual-logit matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 4921-4930, 2022.

Wang, Q., Fang, Z., Zhang, Y., Liu, F., Li, Y., and Han, B. Learning to augment distributions for out-of-distribution detection. In NeurIPS , 2023a. URL https://openreview.net/forum?id= OtU6VvXJue .

Wang, Q., Ye, J., Liu, F., Dai, Q., Kalander, M., Liu, T., Hao, J., and Han, B. Out-of-distribution detection with implicit outlier transformation. arXiv preprint arXiv:2303.05033 , 2023b.

Wang, T. and Isola, P. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning , pp. 9929-9939. PMLR, 2020.

Wei, H., Xie, R., Cheng, H., Feng, L., An, B., and Li, Y. Mitigating neural network overconfidence with logit normalization. In International conference on machine learning , pp. 23631-23644. PMLR, 2022a.

Wei, X.-S., Cui, Q., Yang, L., Wang, P., Liu, L., and Yang, J. Rpc: a large-scale and fine-grained retail product checkout dataset, 2022b. URL https://rpc-dataset.github.io/ .

Welling, M. and Teh, Y. W. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning , pp. 681-688, Madison, WI, USA, 2011. Omnipress.

Widrich, M., Schäfl, B., Pavlovi´ c, M., Ramsauer, H., Gruber, L., Holzleitner, M., Brandstetter, J., Sandve, G. K., Greiff, V., Hochreiter, S., and Klambauer, G. Modern Hopfield networks and attention for immune repertoire classification. ArXiv , 2007.13505, 2020.

Xu, K., Chen, R., Franchi, G., and Yao, A. Scaling for training time and post-hoc out-of-distribution detection enhancement. In The Twelfth International Conference on Learning Representations , 2024.

Xu, P., Ehinger, K. A., Zhang, Y., Finkelstein, A., Kulkarni, S. R., and Xiao, J. Turkergaze: Crowdsourcing saliency with webcam based eye tracking. arXiv preprint arXiv:1504.06755 , 2015.

Xu, P., Chen, J., Zou, D., and Gu, Q. Global convergence of Langevin dynamics based algorithms for nonconvex optimization. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems , volume 31, pp. 3122-3133. Curran Associates, Inc., 2018.

Yang, J., Zhou, K., Li, Y., and Liu, Z. Generalized out-of-distribution detection: A survey. arXiv preprint arXiv:2110.11334 , 2021.

Yang, J., Wang, P., Zou, D., Zhou, Z., Ding, K., Peng, W., Wang, H., Chen, G., Li, B., Sun, Y., et al. Openood: Benchmarking generalized out-of-distribution detection. Advances in Neural Information Processing Systems , 35:32598-32611, 2022.

Yu, F., Seff, A., Zhang, Y ., Song, S., Funkhouser, T., and Xiao, J. LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365 , 2015.

Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., and Yoo, Y. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision , pp. 6023-6032, 2019.

Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations , 2018.

Zhang, J., Fu, Q., Chen, X., Du, L., Li, Z., Wang, G., Han, S., Zhang, D., et al. Out-of-distribution detection based on in-distribution data patterns memorization with modern Hopfield energy. In The Eleventh International Conference on Learning Representations , 2023a.

Zhang, J., Inkawhich, N., Linderman, R., Chen, Y., and Li, H. Mixture outlier exposure: Towards out-of-distribution detection in fine-grained environments. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pp. 5531-5540, January 2023b.

Zhang, R., Li, C., Zhang, J., Chen, C., and Wilson, A. G. Cyclical stochastic gradient MCMC for Bayesian deep learning. In International Conference on Learning Representations , 2020.

Zheng, Y., Zhao, Y., Ren, M., Yan, H., Lu, X., Liu, J., and Li, J. Cartoon face recognition: A benchmark dataset. In Proceedings of the 28th ACM International Conference on Multimedia , pp. 2264-2272, 2020.

Zhu, J., Geng, Y., Yao, J., Liu, T., Niu, G., Sugiyama, M., and Han, B. Diversified outlier exposure for out-of-distribution detection via informative extrapolation. Advances in Neural Information Processing Systems , 36, 2023.

A Details on Continuous Modern Hopfield Networks

The following arguments are adopted from Fürst et al. (2022) and Ramsauer et al. (2021). Associative memory networks have been designed to store and retrieve samples. Hopfield networks are energybased, binary associative memories, which were popularized as artificial neural network architectures in the 1980s (Hopfield, 1982, 1984). Their storage capacity can be considerably increased by polynomial terms in the energy function (Chen et al., 1986; Psaltis & Park, 1986; Baldi & Venkatesh, 1987; Gardner, 1987; Abbott & Arian, 1987; Horn & Usher, 1988; Caputo & Niemann, 2002; Krotov & Hopfield, 2016). In contrast to these binary memory networks, we use continuous associative memory networks with far higher storage capacity. These networks are continuous and differentiable, retrieve with a single update, and have exponential storage capacity (and are therefore scalable, i.e., able to tackle large problems; Ramsauer et al., 2021).

Formally, we denote a set of patterns { x 1 , . . . , x N } ⊂ R d that are stacked as columns to the matrix X = ( x 1 , . . . , x N ) and a state pattern (query) ξ ∈ R d that represents the current state. The largest norm of a stored pattern is M = max i ∥ x i ∥ . Then, the energy E of continuous Modern Hopfield Networks with state ξ is defined as (Ramsauer et al., 2021)

where C = β -1 log N + 1 2 M 2 . For energy E and state ξ , Ramsauer et al. (2021) proved that the update rule

converges globally to stationary points of the energy E and coincides with the attention mechanisms of Transformers (Vaswani et al., 2017; Ramsauer et al., 2021).

The separation ∆ i of a pattern x i is its minimal dot product difference to any of the other patterns:

̸

Apattern is well-separated from the data if ∆ i is above a given threshold (specified in Ramsauer et al., 2021). If the patterns x i are well-separated, the update rule Equation 15 converges to a fixed point close to a stored pattern. If some patterns are similar to one another and, therefore, not well-separated, the update rule converges to a fixed point close to the mean of the similar patterns.

The update rule of a Hopfield network thus identifies sample-sample relations between stored patterns. This enables similarity-based learning methods like nearest neighbor search (see Schäfl et al., 2022), which Hopfield Boosting leverages to detect samples outside the distribution of the training data.

Hopfield networks have recently been used for OOD detection (Zhang et al., 2023a). Hu et al. (2024) introduces Hopfield layers for outlier-efficient memory update.

B Notes on Langevin Sampling

Another method that is appropriate for earlier acquired models is to sample the posterior via the Stochastic Gradient Langevin Dynamics (SGLD) (Welling & Teh, 2011). This method is efficient since it iteratively learns from small mini-batches Welling & Teh (2011); Ahn et al. (2012). See basic work on Langevin dynamics Welling & Teh (2011); Ahn et al. (2012); Teh et al. (2016); Xu et al. (2018). A cyclical stepsize schedule for SGLD was very promising for uncertainty quantification Zhang et al. (2020). Larger steps discover new modes, while smaller steps characterize each mode and perform the posterior sampling.

C Related work

C.1 Details on further OE approaches

This section gives details about related works from the area of OE in OOD detection. With OE, we refer to the usage of AUX for training an OOD detector in general.

MSP-OE. Hendrycks et al. (2019b) were the first to introduce the term OE in the context of OOD detection. Specifically, they improve an MSP-based OOD detection (Hendrycks & Gimpel, 2017): They train a classifier on the ID data set and maximize the entropy of the predictive distribution of the classifier for the AUX data. The combined loss they employ is

where H denotes the cross-entropy, U denotes the uniform distribution over K classes, and p θ is the model mapping the features to the predictive distribution over the K classes.

EBO-OE. Liu et al. (2020) propose a post-hoc and an OE approach. Their post-hoc approach (EBO) is to use the classifier's energy to perform OOD detection:

where f θ outputs the model's logits as a vector. Their OE approach (EBO-OE) promotes a low energy on ID samples and a high energy on AUX samples:

where m ID and m AUX are margin hyperparameters.

POEM. Ming et al. (2022) propose to incorporate Thompson sampling into the OE process. More specifically, they sample a linear decision boundary in embedding space between the ID and AUX data using Bayesian linear regression and then select those samples from the AUX data set that are closest to the sampled decision boundary. In the following epoch, they sample the AUX data uniformly from the selected data instances without replacement and optimize the model with the EBO-OE loss (Equation (21)).

MixOE. Zhang et al. (2023b) employ mixup (Zhang et al., 2018) between the ID and AUX samples to augment the OE task. Formally, this results in the following:

Alternatively, they also propose to employ CutMix (Yun et al., 2019) instead of mixup (which would change the mixing operation in Equation (22)).

DAL. Wang et al. (2023a) augment the AUX data by defining a Wasserstein-1 ball around the AUX data and performing OE using this Wasserstein ball. DAL is motivated by the concept of distribution discrepancy: The distribution of the real OOD data will in general be different from the distribution of the AUX data. The authors argue that their approach can make OOD detection more reliable if the distribution discrepancy is large.

DivOE. Zhu et al. (2023) pose the question of how to utilize the given outliers from the AUX data set if the auxiliary outliers are not informative enough to represent the unseen OOD distribution. They suggest solving this problem by diversifying the AUX data using extrapolation, which should

result in better coverage of the OOD space of the resultant extrapolated distribution. Formally, they employ a loss using a synthesized distribution with a manipulation ∆ :

DOE. Wang et al. (2023b) implicitly synthesize auxiliary outlier data using a transformation of the model weights. They argue that perturbing the model parameters has the same effect as transforming the data.

DOS. Jiang et al. (2024) apply K-means clustering to the features of the AUX data set. They then employ a balanced sampling from the K obtained clusters by selecting the same number of samples from each cluster for training. More specifically, they select those n samples from each cluster which are closest to the decision boundary between the ID and OOD regions.

D Future Work

D.1 Smooth and Sharp Decision Boundaries

Our work treats samples close to the decision boundary as weak learners. However, the wider ramifications of this behavior remain unclear. One can also view the sampling of data instances close to the decision boundary as a form of adversarial training in that we search for something like 'natural adversarial examples': Loosely speaking, the usual adversarial example case starts with a given ID sample and corrupts it in a specific way to get the classifier to output the wrong class probabilities. In our case, we start with a large set of potential adversarial instances (the AUX data) and search for the ones that could be either ID or OOD samples. That is, the sampling process will more likely select data instances that are hard to discriminate for the model - for example, if the model is uncertain whether a leaf in an auxiliary outlier sample is a frog or not.

This process can be viewed as 'sharpening' the decision boundary: The boosting process frequently samples data instances with high uncertainty. L OOD encourages the model to assign less uncertainty to the sampled data instances. After training, few training data instances will have high uncertainty. Nevertheless, a closer systematic evaluation of the sharpened decision boundary of Hopfield Boosting is important to fully understand the potential implications w.r.t. adversarial examples. We view such an investigation as out-of-scope for this work. However, we consider it an interesting avenue for future work.

Anderson & Sojoudi (2022) show that a smooth decision boundary helps with 'classical' adversarial examples. In this framing, our approach would produce different adversarial examples that are not based on noise but are more akin to 'natural adversarial examples'. For example, it is perfectly fine for us that an OOD sample close to the boundary does not correspond to any of the ID classes. Furthermore, the noise based smoothing leads to adversarial robustness at the (potential) cost of degrading classification performance. Similarly, our sharpening of the boundaries leads to better discrimination between ID and OOD region at the (potential) cost of degrading ID classification performance.

E Societal Impact

This section discusses the potential positive and negative societal impacts of our work. As our work aims improves the state-of-the-art in OOD detection, we focus on potential societal impact of OOD detection in general.

· Postive Impacts

-Improved model reliability : OOD detection aims to detect unfamiliar inputs that have little support in the model's training distribution. When these samples are detected, one can, for example, notify the user that no prediction is possible, or trigger a manual intervention. This can lead to an increase in a model's reliability.

-Abstain from doing uncertain predictions : When a model with appropriate OOD detection recognizes that a query sample has limited support in the training distribution, it can abstain from performing a prediction. This can, for example, increase trust in ML models, as they will rather tell the user they are uncertain than report a confidently wrong prediction.

· Negative Impacts

-Wrong sense of safety : Having OOD detection in place could cause users to wrongly assume that all OOD inputs will be detected. However, like most systems, also OOD detection methods can make errors. It is important to consider that certain OOD examples could remain undetected.

-Potential for misinterpretation : As with many other ML systems, the outcomes of OOD detection methods are prone to misinterpretation. It is important to acquaint oneself with the respective method before applying it in practice.

F Toy Examples

F.1 3D Visualizations of E b on a hypersphere

(a) Out-of-Distribution energy function

(b) Exponentiated Out-of-Distribution energy function

0.25

0.50

0.75

-1.00

1.25

1.50

1.75

0.20

0.15

0.10

0.05

0.00

This example depicts how inliers and outliers shape the energy surface (Figure 3). We generated patterns so that X clusters around a pole and the outliers populate the remaining perimeter of the sphere. This is analogous to the idea that one has access to a large AUX data set, where some data points are more and some less informative for OOD detection (e.g., as conceptualized in Ming et al., 2022).

F.2 Dynamics of L OOD on Patterns in Euclidean Space

In this example, we applied our out-of-distribution loss L OOD on a simple binary classification problem. As we are working in Euclidean space and not on a sphere, we use a modified version of MHE, which uses the negative squared Euclidean distance instead of the dot-product-similarity. For the formal relation between Equation (26) and MHE, we refer to Appendix H.1:

Figure 4a shows the initial state of the patterns and the decision boundary exp( βE b ( ξ ; X , O )) . We store the samples of the two classes as stored patterns in X and O , respectively, and compute L OOD for all samples. We then set the learning rate to 0.1 and perform gradient descent with L OOD on the data points. Figure 4b shows that after 25 steps, the distance between the data points and the decision boundary has increased, especially for samples that had previously been close to the decision boundary. After 100 steps, as shown in Figure 4d, the variability orthogonal to the decision boundary has almost completely vanished, while the variability parallel to the decision boundary is maintained.

(a) After 0 steps

(b) After 25 steps

(c) After 50 steps

(d) After 100 steps

4

4

0.20

0.20

3

3

2

0.15

(2)007 g)dxa

2

0.15

(2)007 g)dxa

0.10

0.10

0

0.05

0.05

0

1

2

3

4

5

6

/

0

1

2

3

4

5

6

0.00

0.00

4

4

0.20

0.20

3

3

2

0.15

(2)0007 8

2

0.15

dxa

1

0.10

0.10

0

0

0.05

-1

0.05

0

1

2

3

4

5

6

0

1

2

3

4

5

6

0.00

0.00

F.3 Dynamics of L OOD on Patterns on the Sphere

(a) After 0 steps

(b) After 500 steps

(c) After 2500 steps

：

F.4 Learning Dynamics of Hopfield Boosting on Patterns on a Sphere - Video

The example video 1 demonstrates the learning dynamics of Hopfield Boosting on a 3-dimensional sphere. We randomly generate ID patterns X clustering around one of the sphere's poles and AUX patterns O on the remaining surface of the sphere. We then apply Hopfield Boosting on this data set. First, we sample the weak learners close to the decision boundary for both classes, X and O . Then, we perform 2000 steps of gradient descent with L OOD on the sampled weak learners. We apply the gradient updates to the patterns directly and do not propagate any gradients to an encoder. Every 50 gradient steps, we re-sample the weak learners. For this example, the initial learning rate is set to 0 . 02 and increased after every gradient step by 0 . 1% .

F.5 Location of Weak Learners near the Decision Boundary

2.5

2.0

1.5

1.0

Feature 2

inlier

0.5

outlier

0.0

0.5

1.0

1.5

Feature 1

F.6 Interaction between ID and OOD losses

0.25

0.25

0.25

0.20

0.20

0.20

0.20

0.15

90001 g)dxa

2

0.15

)0001 (f)dxa

0.15

gj0001 gf)dxa

0.15

0001 (h/dka

0.10

0.10

0.10

0.10

0.05

0.05

D

0.05

0.05

7

0.00

0.00

0.00

0.00

Z

E

8

2

3

To demonstrate how Hopfield Boosting can interact in a classification task we created a toy example that resembles the ID classification setting (Figure 7): The example shows the decision boundary and the inlier samples organized in two classes (shown in red and orange). We sample uniformly distributed auxiliary outliers. Then, we minimize the compound objective (applying the gradient updates on the patterns directly). This shows that Hopfield Boosting is able to separate the two classes well and that still forms a tight decision boundary around the ID data.

G Notes on E b

G.1 Probabilistic Interpretation of E b

We model the class-conditional densities of the in-distribution data and auxiliary data as mixtures of Gaussians with the patterns as the component means and tied, diagonal covariance matrices with β -1 in the main diagonal.

Further, we assume the distribution p ( ξ ) as a mixture of p ( ξ | ID ) and p ( ξ | AUX ) with equal prior probabilities (mixture weights):

The probability of an unknown sample ξ being an AUX sample is given by

where in line (34) we have used that p ( ξ | AUX ) > 0 for all ξ ∈ R d . The probability of ξ being an ID sample is given by

By taking the log of Equation (40) we obtain the following. We use C = to denote equality up to an additive constant that does not depend on ξ .

Consider the function

Multiplication by β -1 is equivalent to a change of base of the log . The term -β -1 log( p ( ξ )) is equivalent to the MHE (Ramsauer et al., 2021) (up to an additive constant) when assuming normalized patterns, i.e. || x i || 2 = 1 and || o i || 2 = 1 , and an equal number of patterns M = N in the two Gaussian mixtures p ( ξ | ID ) and p ( ξ | AUX ) :

Analogously, β -1 log( p ( ξ | ID )) and β -1 log( p ( ξ | AUX )) also yield MHE terms. Therefore, E b is equivalent to β -1 log( f b ( ξ )) under the assumption that || x i || 2 = 1 and || o i || 2 = 1 and M = N . The 1 2 ξ T ξ terms that are contained in the three MHEs cancel out.

f b ( ξ ) can also be interpreted as the variance of a Bernoulli distribution with outcomes ID and AUX:

In other words, minimizing E b means to drive a Bernoulli-distributed random variable with the outcomes ID and AUX towards minimum variance, i.e., p ( ID | ξ ) is driven towards 1 if p ( ID | ξ ) > 0 . 5 and towards 0 if p ( ID | ξ ) < 0 . 5 . Conversely, the same is true for p ( AUX | ξ ) .

From Equation (35), under the assumptions that || x i || 2 = 1 and || o i || 2 = 1 and M = N , the conditional probability p ( AUX | ξ ) can be computed as follows:

where σ denotes the logistic sigmoid function. Similarly, p ( ID | ξ ) can be computed using

G.2 Alternative Formulations of E b and f b

E b can be rewritten as follows.

To prove this, we first show the following:

Let E X = -lse( β, X T ξ ) and E O = -lse( β, O T ξ ) .

By exponentiation of the above result we obtain

The function log cosh( x ) is related to the negative log-likelihood of the hyperbolic secant distribution (see e.g. Saleh & Saleh, 2022). For values of x close to 0 , log cosh can be approximated by x 2 2 , and for values far from 0 , the function behaves as | x | -log(2) .

(a) Probabilities

(b) Log-probabilities

1.0

0

0.8

1

0.6

p(ID|)

-2

1 - p(ID | )

p(1-p)=fb

0.4

3

0.2

log(p(ID |≤))

log(1-p(ID|ε))

log（p(1-p）)=log(p)+ log（1-p)=log(fb)

0.0

5

-2

0

2

-2

0

2

4

log(p(≤ | ID)) - log(p( | AUX))

log(p(≤ | ID)) - log(p(≤ | AUX))

G.3 Derivatives of E b

In this section, we investigate the derivatives of the energy function E b . The derivative of the lse is:

Thus, the derivative of the MHE E( ξ ; X ) w.r.t. ξ is:

The update rule of the MHN

is derived via the concave-convex procedure. It coincides with the attention mechanisms of Transformers and has been proven to converge globally to stationary points of the energy E( ξ ; X ) (Ramsauer et al., 2021). It can also be shown that the update rule emerges when performing gradient descent on E( ξ ; X ) with step size η = 1 Park et al. (2023):

From Equation (71), we can see that the gradient of E b ( ξ ; X , O ) w.r.t. ξ is:

When X softmax( β X T ξ ) , O softmax( β O T ξ ) , lse( β, X T ξ ) and lse( β, O T ξ ) are available, one can efficiently compute ( X ∥ O ) softmax( β ( X ∥ O ) T ξ ) as follows:

We can also compute the gradient of E b ( ξ ; X , O ) w.r.t. ξ via the log cosh -representation of E b (see Equation (68)). The derivative of the log cosh function is

Therefore, we can compute the gradient of E b ( ξ ; X , O ) as

Next, we would like to compute the gradient of E b ( ξ ; X , O ) w.r.t. the memory matrices X and O . For this, let us first look at the gradient of the MHE E( ξ ; X ) w.r.t. a single stored pattern x i (where X is the matrix of concatenated stored patterns ( x 1 , x 2 , . . . , x N ) ):

Thus, the gradient w.r.t. the full memory matrix X is

We can now also use the log cosh formulation of E b ( ξ ; X , O ) to compute the gradient of E b ( ξ ; X , O ) , w.r.t X and O :

(90)

Analogously, the gradient w.r.t O is

H Notes on the Relationship between Hopfield Boosting and other methods

H.1 Relation to Radial Basis Function Networks

This section shows the relation between radial basis function networks (RBF networks; Moody & Darken, 1989) and modern Hopfield energy (following Schäfl et al., 2022). Consider an RBF network with normalized linear weights:

where β denotes the inverse tied variance β = 1 σ 2 , and the ω i are normalized using the softmax function:

An energy can be obtained by taking the negative log of φ ( ξ ) :

Next, we define a i = 1 2 µ T i µ i

Finally, we use the fact that lse( β, a ) ≤ max i a i + β -1 log N

where M = max i || µ i || 2

H.2 Contrastive Representation Learning

A commonly used loss function in contrastive representation learning (e.g., Chen et al., 2020; He et al., 2020) is the InfoNCE loss (Oord et al., 2018):

Wang & Isola (2020) show that L NCE optimizes two objectives:

Alignment enforces that features from positive pairs are similar, while uniformity encourages a uniform distribution of the samples over the hypersphere.

In comparison, our proposed loss, L OOD, does not visibly enforce alignment between samples within the same class. Instead, we can observe that it promotes uniformity to the instances of the foreign class. Due to the constraints that are imposed by the geometry of the space the optimization is performed on, that is, || f ( x ) || = 1 when the samples move on a hypersphere, the loss encourages the patterns in the ID data have maximum distance to the samples of the AUX data, i.e., they concentrate on opposing poles of the hypersphere. A demonstration of this mechanism can be found in Appendix F.2 and F.3

H.3 Support Vector Machines

In the following, we will show the relation of Hopfield Boosting to support vector machines (SVMs; Cortes & Vapnik, 1995) with RBF kernel. We adopt and expand the arguments of Schäfl et al. (2022).

Assume we apply an SVM with RBF kernel to model the decision boundary between ID and AUX data. We train on the features Z = ( x 1 , . . . , x N , o 1 , . . . , o M ) and assume that the patterns are normalized, i.e., || x i || 2 = 1 and || o i || 2 = 1 . We define the targets ( y 1 , . . . , y ( N + M ) ) as 1 for ID and -1 for AUX data. The decision rule of the SVM equates to

We assume that there is at least one support vector for both ID and AUX data, i.e., there exists at least one index i s.t. α i y i > 0 and at least one index j s.t. α j y j < 0 . We now split the samples z i in s ( ξ ) according to their label:

We define an alternative score:

where

(108)

Because we assumed there is at least one support vector for both ID and AUX data and as the α i are constrained to be non-negative and because k ( · , · ) > 0 , the numerator and denominator are strictly positive. We can, therefore, specify a new decision rule ˆ B frac ( ξ ) .

Although the functions s ( ξ ) and s frac ( ξ ) are different, the decision rules ˆ B ( ξ ) and ˆ B frac ( ξ ) are equivalent. Another possible pair of score and decision rule is the following:

Let us more closely examine the term β -1 log ( ∑ N i =1 α i k ( x i , ξ ) ) . We define a i = β -1 log( α i ) .

We now construct a memory X H and query ξ H such that we can compute (115) using the MHE (Equation (5)):

We obtain

We construct O H analogously to Equation (116) and thus can compute

which is exactly the score Hopfield Boosting uses for determining whether a sample is OOD (Equation (13)). In contrast to SVMs, Hopfield Boosting uses a uniform weighting of the patterns in the memory when computing the score. However, Hopfield Boosting can emulate a weighting of the patterns by more frequently sampling patterns with high weights into the memory.

H.4 HE and SHE

Zhang et al. (2023a) introduce two post-hoc methods for OOD detection using MHE, which are called 'Hopfield Energy' (HE) and 'Simplified Hopfield Energy' (SHE). Like Hopfield Boosting, HE and SHE both employ the MHE to determine whether a sample is ID or OOD. However, unlike Hopfield Boosting, HE and SHE offer no possibility to include AUX data in the training process to improve the OOD detection performance of their method. The rest of this section is structured as follows: First, we briefly introduce the methods HE and SHE, second, we formally analyze the two methods, and third, we relate them to Hopfield Boosting.

Hopfield Energy (HE) The method HE (Zhang et al., 2023a) computes the OOD score s HE ( ξ ) as follows:

where X c ∈ R d × N c denotes the memory ( x c 1 , . . . , x cN c ) containing N c encoded data instances of class c . HE uses the prediction of the ID classification head to determine which patterns to store in the Hopfield memory:

Simplified Hopfield Energy (SHE) The method SHE (Zhang et al., 2023a) employs a simplified score s SHE ( ξ ) :

where m c ∈ R d denotes the mean of the patterns in memory X c :

Relation between HE and SHE In the following, we show a simple yet enlightening relation between the scores s HE and s SHE. For mathematical convenience, we first slightly modify the score s HE:

All data sets which were employed in the experiments of Zhang et al. (2023a) (CIFAR-10 and CIFAR-100) are class-balanced. Therefore, the additional term β -1 log N c does not change the result of the OOD detection on those data sets, as it only amounts to the same constant offset for all classes.

The function

where

Therefore, we have shown that

converges to the mean function as β → 0 :

We now investigate the behavior of s HE in this limit:

Relation of HE and SHE to Hopfield Boosting. In contrast to HE and SHE, Hopfield Boosting uses an AUX data set to learn a decision boundary between the ID and OOD regions during the training process. To do this, our work introduces a novel MHE-based energy function, E b ( ξ ; X , O ) , to determine how close a sample is to the learnt decision boundary. Hopfield Boosting uses this energy function to frequently sample weak learners into the Hopfield memory and for computing a novel Hopfield-based OOD loss L OOD. To the best our knowledge, we are the first to use MHE in this way to train a neural network.

The OOD detection score of Hopfield Boosting is

where X ∈ R d × N contains the full encoded training set ( x 1 , . . . , x N ) of all classes and O ∈ R d × M contains AUX samples. While certainly similar to s HE, the Hopfield Boosting score s differs from s HE in three crucial aspects:

Hopfield Boosting uses AUX data samples in the OOD detection score in order to create a sharper decision boundary between the ID and OOD regions.

Hopfield Boosting normalizes the patterns in the memories X and O and the query ξ to unit length, while HE and SHE use unnormalized patterns to construct their memories X c and their query pattern ξ .

The score of Hopfield Boosting, s ( ξ ) , contains the full encoded training data set, while s HE only contains the patterns of a single class. Therefore Hopfield Boosting computes the similarities of a query sample ξ to the entire ID data set. In Appendix I.8, we show that this process only incurs a moderate overhead of 7 . 5% compared to the forward pass of the ResNet-18.

The selection of the score function s ( ξ ) is only a small aspect of Hopfield Boosting. Hopfield Boosting additionally samples informative AUX data close to the decision boundary, optimizes an MHE-based loss function, and thereby learns a sharp decision boundary between ID and OOD regions. Those three aspects are novel contributions of Hopfield Boosting. In contrast, the work of Zhang et al. (2023a) solely focuses on the selection of a suitable Hopfield-based OOD detection score for post-hoc OOD detection.

Metric

HB (ours)

DOS

DOE

DivOE

DAL

MixOE

POEM

EBO-OE

MSP-OE

FPR95

↓

13

.

27

±

5

.

46

9

.

84

±

2

.

75

19

.

38

±

4

.

60

28

.

77

±

5

.

42

19

.

95

±

2

.

34

41

.

54

±

13

.

16

33

.

59

±

4

.

12

36

.

33

±

2

.

95

19

.

86

±

6

.

90

SVHN

AUROC

↑

97

.

07

±

0

.

81

97

.

64

±

0

.

39

95

.

72

±

1

.

12

94

.

25

±

0

.

98

95

.

69

±

0

.

66

92

.

27

±

2

.

71

94

.

06

±

0

.

51

92

.

93

±

0

.

72

95

.

74

±

1

.

60

FPR95

↓

12

.

68

±

2

.

38

19

.

40

±

2

.

45

28

.

23

±

2

.

69

35

.

10

±

4

.

23

24

.

24

±

2

.

12

23

.

10

±

7

.

39

15

.

72

±

3

.

46

21

.

06

±

3

.

12

32

.

88

±

1

.

28

LSUN-Crop

AUROC

↑

96

.

54

±

0

.

65

96

.

42

±

0

.

35

93

.

79

±

0

.

88

92

.

45

±

0

.

94

95

.

04

±

0

.

43

96

.

11

±

1

.

09

96

.

85

±

0

.

60

95

.

79

±

0

.

62

92

.

85

±

0

.

33

FPR95

↓

0

.

00

±

0

.

00

0

.

01

±

0

.

00

0

.

05

±

0

.

04

0

.

01

±

0

.

00

0

.

00

±

0

.

00

10

.

27

±

10

.

72

0

.

00

±

0

.

00

0

.

00

±

0

.

00

0

.

03

±

0

.

01

LSUN-Resize

AUROC

↑

99

.

98

±

0

.

01

99

.

96

±

0

.

02

99

.

99

±

0

.

01

99

.

99

±

0

.

00

99

.

94

±

0

.

02

97

.

99

±

1

.

92

99

.

57

±

0

.

09

99

.

57

±

0

.

03

99

.

97

±

0

.

00

FPR95

↓

2

.

35

±

0

.

13

6

.

02

±

0

.

52

19

.

42

±

1

.

58

11

.

52

±

0

.

49

5

.

22

±

0

.

39

28

.

99

±

6

.

79

2

.

89

±

0

.

32

5

.

07

±

0

.

54

10

.

34

±

0

.

40

Textures

AUROC

↑

99

.

22

±

0

.

02

98

.

33

±

0

.

11

94

.

93

±

0

.

48

97

.

02

±

0

.

08

98

.

50

±

0

.

16

94

.

24

±

1

.

21

98

.

97

±

0

.

08

98

.

15

±

0

.

16

97

.

42

±

0

.

08

FPR95

↓

0

.

00

±

0

.

00

0

.

03

±

0

.

01

0

.

01

±

0

.

02

0

.

06

±

0

.

01

0

.

01

±

0

.

02

14

.

40

±

13

.

48

0

.

00

±

0

.

00

0

.

00

±

0

.

00

0

.

08

±

0

.

02

iSUN

AUROC

↑

99

.

98

±

0

.

01

99

.

95

±

0

.

02

99

.

99

±

0

.

00

99

.

97

±

0

.

00

99

.

93

±

0

.

02

97

.

23

±

2

.

59

99

.

59

±

0

.

09

99

.

57

±

0

.

03

99

.

96

±

0

.

01

FPR95

↓

19

.

36

±

1

.

02

32

.

13

±

1

.

55

58

.

68

±

4

.

15

44

.

20

±

0

.

95

33

.

43

±

1

.

11

47

.

01

±

6

.

41

18

.

39

±

0

.

68

26

.

68

±

2

.

18

45

.

96

±

0

.

85

Places 365

AUROC

↑

95

.

85

±

0

.

37

91

.

73

±

0

.

39

83

.

47

±

1

.

55

88

.

28

±

0

.

26

91

.

10

±

0

.

29

89

.

20

±

1

.

86

95

.

03

±

0

.

71

91

.

35

±

0

.

70

87

.

77

±

0

.

15

FPR95

↓

7

.

94

11

.

24

20

.

96

19

.

94

13

.

81

27

.

55

11

.

76

14

.

86

18

.

19

Mean

AUROC

↑

98

.

11

97

.

34

94

.

65

95

.

33

96

.

70

94

.

51

97

.

34

96

.

23

95

.

62

ID

↑

75

.

08

±

0

.

46

75

.

72

±

0

.

26

76

.

96

±

0

.

33

76

.

91

±

0

.

30

77

.

29

±

0

.

14

79

.

20

±

2

.

99

66

.

38

±

0

.

85

69

.

07

±

0

.

32

76

.

87

±

0

.

39

I Additional Experiments & Experimental Details

I.1 Results on CIFAR-100

When applying Hopfield Boosting on CIFAR-100 (Table 4), Hopfield Boosting surpasses POEM (the previously best method), improving the mean FPR95 from 11.76 to 7.95. On the SVHN data set, Hopfield Boosting improves the FPR95 metric the most, decreasing it from 33.59 to 13.27. For the LSUN-Resize and iSUN data sets, we observe a similar behavior as we saw in our CIFAR-10 evaluation - almost all methods achieve a perfect result with regard to the FPR95 metric.

I.2 Pre-Processing and Transformations

For evaluating OOD detection methods, consistent pre-processing and image transformation is crucial. An inconsistent application of image transformations will skew results when comparing different OOD detection methods. We, therefore, apply the same pre-processing steps and transformations to all OOD detection methods we compare.

CIFAR-10 & CIFAR-100. For CIFAR-10 and CIFAR-100, we apply the following transformations:

RandomCrop (32x32, padding 4)

RandomHorizontalFlip

ImageNet-RC. For ImageNet-RC (used as AUX data set for the ID data sets CIFAR-10 and CIFAR-100), we apply the following transformations:

RandomCrop (32x32)

RandomCrop (32x32, padding 4)

RandomHorizontalFlip

ImageNet-1K. For ImageNet-1K, we apply the following transformations. We closely follow the transformations used in the experiments of Zhu et al. (2023):

Resize (224x224)

RandomCrop (224x224, padding 4)

RandomHorizontalFlip

ImageNet-21K. For ImageNet-21K (used as AUX data set for the ID data sets ImageNet-1K), we apply the following transformations. We closely follow the transformations used in the experiments of Zhu et al. (2023):

RandAugment (Cubuk et al., 2020)

Resize (224x224)

RandomCrop (224x224, padding 4)

RandomHorizontalFlip

I.3 Comparison HE/SHE

Since Hopfield Boosting shares similarities with the MHE-based methods HE and SHE (Zhang et al., 2023a), we also looked at the approach as used for their methods. We use the same ResNet-18 as a backbone network as we used in the experiments for Hopfield Boosting, but train it on CIFAR-10 without OE. We modify the approach of Zhang et al. (2023a) to not only use the penultimate layer, but perform a search over all layer activation combinations of the backbone for the best-performing combination. We also do not use the classifier to separate by class. From the search, we see that the concatenated activations of layers 3 and 5 give the best performance on average, so we use this setting. We experience a quite noticeable drop in performance compared to their results (Table 5). Since the computation of the MHE is the same, we assume the reason for the performance drop is the different training of the ResNet-18 backbone network, where (Zhang et al., 2023a) used strong augmentations.

I.4 Ablations

We investigate the impact of different encoder backbone architectures on OOD detection performance with Hopfield Boosting. The baseline uses a ResNet-18 as the encoder architecture. For the ablation, the following architectures are used as a comparison: ResNet-34, ResNet-50, and Densenet-100. It can be observed, that the larger architectures lead to a slight increase in OOD performance (Table 6). We also see that a change in architecture from ResNet to Densenet leads to a different OOD behavior: The result on the Places365 data set is greatly improved, while the performance on SVHN is noticeably worse than on the ResNet architectures. The FPR95 of Densenet on SVHN also shows a high variance, which is due to one of the five independent training runs performing very badly at detecting SVHN samples as OOD: The worst run scores an FPR95 5.59, while the best run achieves an FPR95 of 0.24.

I.5 Effect on Learned Representation

In order to analyze the impact of Hopfield Boosting on learned representations, we utilize the output of our model's embedding layer (see 4.2) as the input for a manifold learning-based visualization. Uniform Manifold Approximation and Projection (UMAP) McInnes et al. (2018) is a non-linear

(a) without Hopfield Boosting

(b) with Hopfield Boosting

CIFAR-10·AUXSVHN

CIFAR-10AUXSVHN

25

20

20

15

10

10-

5

0

0

-5

-10-

-10

-15-

-20-

-5

0

5

10

15

20

25

-10

-5

0

5

10

15

20

25

dimensionality reduction technique known for its ability to preserve both global and local structure in high-dimensional data.

First, we train two models - with and without Hopfield Boosting- and extract the embeddings of both ID and OOD data sets from them. This results in a 512-dimensional vector representation for each data point, which we further reduce to two dimensions with UMAP. The training data for UMAP always corresponds to the training data of the respective method. That is, the model trained without Hopfield Boosting is solely trained on CIFAR-10 data, and the model trained with Hopfield Boosting is presented with CIFAR-10 and AUX data during training, respectively. We then compare the learned representations concerning ID and OOD data.

Figure 9 shows the UMAP embeddings of ID (CIFAR-10) and OOD (AUX and SVHN) data based on our model trained without (a) and with Hopfield Boosting (b). Without Hopfield Boosting, OOD data points typically overlap with ID data points, with just a few exceptions, making it difficult to differentiate between them. Conversely, Hopfield Boosting allows to distinctly separate ID and OOD data in the embedding.

I.6 OOD Examples from the Places 365 Data Set with High Semantic Similarity to CIFAR-10

We observe that Hopfield Boosting and all competing methods struggle with correctly classifying the samples from the Places 365 data set as OOD the most. Table 1 shows that for Hopfield Boosting, the FPR95 for the Places 365 data set with CIFAR-10 as the ID data set is at 4.28. The second worst FPR95 for Hopfield Boosting was measured on the LSUN-Crop data set at 0.82.

We inspect the 100 images from Places 365 that perform worst (i.e., that achieve the highest score s ( ξ ) ) on a model trained with Hopfield Boosting on the CIFAR-10 data set as the in-distribution data set. Figure 10 shows that within those 100 images, the Places 365 data set contains a non-negligible amount of data instances that show objects from semantic classes contained in CIFAR-10 (e.g., horses, automobiles, dogs, trucks, and airplanes). We argue that data instances that clearly show objects of semantic classes contained in CIFAR-10 should be considered as in-distribution, which Hopfield Boosting correctly recognizes. Therefore, a certain amount of error can be anticipated on the Places 365 data set for all OOD detection methods. We leave a closer evaluation of the amount of the anticipated error up to future work.

For comparison, Figure 11 shows the 100 images from Places 365 with the lowest score s ( ξ ) , as evaluated by a model trained with Hopfield Boosting on CIFAR-10. There are no objects visible that have clear semantic overlap with the CIFAR-10 classes.

1.945

1.945

1.945

1.945

1.945

1.944

1.944

1.944

1.944

1.944

1.944

1.944

1.944

1.944

1.944

1.944

1.944

1.944

1.944

1.944

1.944

1.944

1.944

1.944

1.944

1.944

1.944

1.944

1.944

1.944

1.944

1.944

1.944

1.944

1.944

1.943

1.943

1.943

1.943

1.943

1.943

1.943

1.943

1.943

1.943

1.943

1.943

1.943

1.943

1.943

1.943

1.943

1.943

1.943

1.943

1.943

1.943

1.943

1.943

1.942

1.942

1.942

1.942

1.942

1.942

1.942

1.942

1.942

1.942

1.942

1.942

1.942

1.942

1.942

1.942

1.942

1.942

1.941

1.941

1.941

1.941

1.941

1.941

1.941

1.941

1.941

1.941

1.941

1.941

1.941

1.941

1.941

1.941

1.941

1.941

1.941

1.941

1.941

1.941

1.941

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

-1.925

I.7 Results on Noticeably Different Data Sets

The choice of additional data sets should not be driven by a desire to showcase good performance; rather, we suggest opting for data that highlights weaknesses, as it holds the potential to drive investigations and uncover novel insights. Simple toy data is preferable due to its typically clearer and more intuitive characteristics compared to complex natural image data. In alignment with these considerations, the following data sets captivated our interest: iCartoonFace (Zheng et al., 2020), Four Shapes (smeschke, 2018), and Retail Product Checkout (RPC) (Wei et al., 2022b). In Figure 12, we show random samples from these data sets to demonstrate the noticeable differences compared to CIFAR-10.

In Table 7, we present some preliminary results using models trained with the respective method on CIFAR-10 as ID data set (as in Table 1). Results for comparison are presented for EBO-OE only, as time constraints prevented experimenting with additional baseline methods. Although one would expect near-perfect results due to the evident disparities with CIFAR-10, Four Shapes (smeschke, 2018) and RPC (Wei et al., 2022b) seem to defy that expectation. Their results indicate a weakness in the capability to identify outliers robustly since many samples are classified as inliers. Only iCartoonFace (Zheng et al., 2020) is correctly detected as OOD, at least to a large degree. Interestingly, the weakness uncovered by this data is present in both methods, although more pronounced in EBO-OE. Therefore, we suspect that this specific behavior may be a general weakness when training OOD detectors using OE, an aspect we plan to investigate further in our future work.

I.8 Runtime Considerations for Inference

When using Hopfield Boosting in inference, an additional inference step is needed to check whether a given sample is ID or OOD. Namely, to obtain the score (Equation (13)) of a query sample ξ D , Hopfield Boosting computes the dot product similarity of the embedding obtained from ξ = ϕ ( ξ D ) to all samples in the Hopfield memories X and O . In our experiments, X contains the full indistribution data set (50,000 samples) and O contains a subset of the AUX data set of equal size. We investigate the computational overhead of computing the dot-product similarity to 100,000 samples in relation to the computational load of the encoder. For this, we feed 100 batches of size 1024 to an encoder (1) without using the score and (2) with using the score, measure the runtimes per batch, and compute the mean and standard deviation. We conduct this experiment with four different encoders on an NVIDIA Titan V GPU. The results are shown in Figure 13 and Table 8. One can see that, especially for larger models, the computational overhead of determining the score is very moderate in comparison.

Encoder

350

MHE-based score

300

Runtime (ms / batch)

250

200

150

100

50

0

ResNet-18

ResNet-34

ResNet-50

DenseNet-100

Encoder

I.9 HE and SHE Extensions with AUX Data

To show that the unique contributions of Hopfield Boosting (like the energy-based loss L OOD and the boosting process) are responsible for the superior performance of Hopfield Boosting, we devise two extensions of HE that include AUX data and compare them to Hopfield Boosting.

40

Classification Error

Mean OOD FPR95

35

Measurement(%)

25

20

15

10

5

0

0.0

0.1

0.2

0.3

0.6

0.7

0.8

0.9

1.0

入

The first extension (HE+AUX) uses a model trained only on the ID data and adapts HE to include an MHE term that measures the energy of ξ on the AUX data O :

The second extension (HE+OE) applies OE (Hendrycks et al., 2019b) while training the model. It then uses s HE to estimate whether a sample is ID or OOD. For both extensions, we select β by minimizing the mean FPR95 on the OOD test data sets to obtain an upper bound of the possible performance of these extensions. The β we selected for HE+AUX is 0 . 001 , and for HE+OE is 0 . 01 .

Our results (Table 9) show that Hopfield Boosting is superior to both extensions. HE+AUX results in a mean FPR95 of 19.91, HE+OE achieves a mean FPR95 of 2.72. Hopfield Boosting improves on both extensions, achieving a mean FPR95 of 0.92.

I.10 Ablation on the Hyperparameter λ

There is usually an inherent tradeoff between ID accuracy and OOD detection performance when employing OE methods. In practice one can always improve the tradeoff by using models with more capacity - in the extreme case practitioners can even train a separate ID network. Hence, the model selection process we employed only considered the OOD detection performance and did not take the ID accuracy into account. To investigate if and how this tradeoff can be controlled by changing the hyperparameters of Hopfield Boosting, we conduct the following experiment:

We (1) ablate the hyperparameter (the weight of the out-of-distribution loss) and run Hopfield Boosting on the CIFAR-10 benchmark; (2) select λ from the range [0 , 1] with a step size of 0 . 1 ; and (3) record the OOD detection performance (the mean FPR95 where the mean is taken over the OOD test data sets) and the ID classification error for the individual settings of λ .

The results indicate that decreasing the hyperparameter λ improves the ID classification accuracy of Hopfield Boosting (Figure 14). At the same time, the mean OOD AUROC is only moderately influenced: When setting, the hyperparameter setting reported in the original manuscript, the mean ID classification error is 5 . 98% , and the mean FPR95 is 0 . 92% . When decreasing λ to 0.1, the mean ID classification error improves to 5 . 02% . Similarly, the FPR95 only slightly increases to 1 . 08% (which is still substantially better than the second-best outlier exposed method, POEM, which achieves a mean FPR95 of 2 . 28% ). Hence, practitioners can control the tradeoff between ID classification accuracy and OOD detection performance.

1.000

O

0.998

0.998

0.996

O

0.996

AUROC

AUROC

o

0.994

SVHNOOD

0.994

SVHN

0.992

o

0.990

0.992

O

0.988

0.990

8

1

5

10

50

100

500

100050001000050000

1

5

10

50

100

500

100050001000050000

No.patterns inHopfield memory

No. patterns in Hopfield memory o

Figure 15: Ablating the number of patterns stored in the Hopfield memories during inference. AUROC on SVHN based on the number of patterns in the Hopfield memory. In (a), X and O contain the same number of patterns; in (b) X contains 50 , 000 patterns, and we vary the number of patterns in O . The variability of the AUROC is reduced when X and O contain 50,000 patterns, respectively.

I.11 Ablation on the Number of Patterns Stored in the Memories during Inference

In our implementation of Hopfield Boosting, we fill the memories X and O with N = 50 , 000 patterns to compute the score s ( ξ ) , respectively. To investigate the robustness of Hopfield Boosting when changing the number of patterns N , we conduct the following experiments:

We train Hopfield Boosting on CIFAR-10 (ID data) and ImageNet (AUX data). During the weight update process, we store 50,000 patterns in the memories X and O , and then ablate the number of patterns stored in the memories for computing the score s ( ξ ) at inference time. We evaluate the discriminative power of s ( ξ ) on SVHN with 1, 5, 10, 50, 100, 500, 1,000, 5,000, 10,000, and 50,000 patterns stored in the memories X and O . To investigate the influence of the stochastic process of sampling N patterns from the ID and AUX data sets, we conduct 50 runs for all of the and create boxplots of the runs. The results (Figure 15a) show that sampling 50 , 000 patterns has the lowest variability of the individual trials. We argue that the reason for this is that by this time the entire ID data set is stored in the Hopfield memory - which effectively eliminates stochasticity from randomly selecting N patterns from the ID data.

To verify that we can use s ( ξ ) when the number of patterns in X and O is imbalanced, we fill X with all 50,000 data instances of CIFAR-10 and fill O with 1, 5, 10, 50, 100, 500, 1000, 5000, 10,000, and 50,000 data instances of the AUX data set. Then, we evaluate the discriminative power of s ( ξ ) for the different instances. Our results (Figure 15b) show that Hopfield Boosting is robust to an imbalance in the number of samples in X and O . The setting with 50,000 samples in both memories (which is the setting we use in the experiments in our original manuscript) incurs the least variability.

I.12 Compute Ressources

Our experiments were conducted on an internal cluster equipped with a variety of different GPU types (ranging from the NVIDIA Titan V to the NVIDIA A100-SXM-80GB). For our experiments on ImageNet-1K, we additionally used resources of an external cluster that is equipped with NVIDIA A100-SXM-64GB GPUs.

For our experiments with Hopfield Boosting on CIFAR-10 and CIFAR-100, one run (100 epochs) of Hopfield Boosting trained for about 8.0 hours on a single NVIDIA RTX 2080 Ti GPU and required 4.3 GB of VRAM. Fnding the hyperparameters required 160h of compute for CIFAR-10 and CIFAR-100, respectively. These were divided across four RTX 2080 Ti. Estimating the standard deviation required 40 hours of compute on a single RTX 2080 Ti for CIFAR-10 and CIFAR-100 respectively.

For ImageNet-1K, one run (4 epochs) of Hopfield Boosting trained for about 4.4 hours on a single NVIDIA A-100-SXM64GB GPU and required 26.9 GB of VRAM. Finding the optimal hyperparameters required a total of 86h of compute, divided across 20 NVIDIA A-100-SXM64GB GPUs. Estimating the standard deviation required 22 hours of compute, divided across 5 NVIDIA A-100SXM64GB GPUs.

The amount of resources reported above cover the compute for obtaining the results of Hopfield Boosting reported in the paper. The total amount of compute resources for the project is substantially higher. Notable additional compute expenses are preliminary training runs during the development of Hopfield Boosting, and the training runs for tuning the hyperparameters and evaluating the results of the methods we compare Hopfield Boosting to.

I.13 Data Sets and Licenses

We provide a list of the data sets we used in our experiments and, where applicable, specify their licenses:

CIFAR-10 (Krizhevsky, 2009): License unknown

CIFAR-100 (Krizhevsky, 2009): License unknown

ImageNet-RC (Chrabaszcz et al., 2017): Custom License 2

SVHN (Netzer et al., 2011): Creative Commons (CC)

Textures (Cimpoi et al., 2014): Custom License 3

iSUN (Xu et al., 2015): License unknown

Places 365 (López-Cifuentes et al., 2020): License unknown

LSUN (Yu et al., 2015): License unknown

ImageNet-1K (Russakovsky et al., 2015): Custom License 2

ImagetNet-21K (Ridnik et al., 2021): Custom License 2

SUN (Isola et al., 2011): License unknown

iNaturalist (Van Horn et al., 2018): Custom License 4

Table 10: OOD detection performance on CIFAR-10. We compare results from Hopfield Boosting, PALM (Lu et al., 2024), NPOS (Tao et al., 2023), SSD+ (Sehwag et al., 2021), ASH (Djurisic et al., 2023), GEN (Liu et al., 2023), EBO (Liu et al., 2020), MaxLogit (Hendrycks et al., 2019a), and MSP (Hendrycks & Gimpel, 2017) on ResNet-18. ↓ indicates 'lower is better' and ↑ 'higher is better'. All values in % . Standard deviations are estimated across five training runs.

HB (ours)

PALM

NPOS

SSD+

ASH

GEN

EBO

MaxLogit

MSP

FPR95

↓

0

.

23

±

0

.

08

1

.

24

±

0

.

49

9

.

04

±

1

.

13

3

.

05

±

0

.

22

25

.

17

±

9

.

55

33

.

26

±

5

.

99

32

.

10

±

6

.

41

33

.

27

±

6

.

18

49

.

41

±

3

.

77

SVHN

AUROC

↑

99

.

57

±

0

.

06

99

.

70

±

0

.

12

98

.

37

±

0

.

23

99

.

41

±

0

.

06

94

.

86

±

2

.

09

93

.

53

±

1

.

42

93

.

43

±

1

.

60

93

.

29

±

1

.

57

92

.

48

±

0

.

93

FPR95

↓

0

.

82

±

0

.

17

1

.

21

±

0

.

27

5

.

52

±

0

.

50

2

.

83

±

1

.

10

13

.

13

±

1

.

81

19

.

40

±

2

.

22

17

.

25

±

2

.

30

18

.

50

±

2

.

24

38

.

32

±

2

.

61

LSUN-Crop

AUROC

↑

99

.

40

±

0

.

04

99

.

65

±

0

.

05

98

.

97

±

0

.

04

99

.

37

±

0

.

16

97

.

33

±

0

.

36

96

.

48

±

0

.

46

96

.

73

±

0

.

46

96

.

52

±

0

.

47

94

.

37

±

0

.

53

FPR95

↓

0

.

00

±

0

.

00

27

.

01

±

5

.

82

26

.

85

±

3

.

14

34

.

30

±

2

.

17

38

.

18

±

5

.

78

31

.

50

±

3

.

92

30

.

69

±

4

.

03

31

.

64

±

4

.

01

45

.

82

±

3

.

48

LSUN-Resize

AUROC

↑

99

.

98

±

0

.

02

95

.

41

±

0

.

74

95

.

68

±

0

.

36

94

.

78

±

0

.

25

90

.

39

±

2

.

00

94

.

04

±

0

.

84

94

.

02

±

0

.

86

93

.

90

±

0

.

86

92

.

84

±

0

.

80

FPR95

↓

0

.

16

±

0

.

02

17

.

32

±

2

.

50

27

.

72

±

2

.

55

21

.

20

±

2

.

20

46

.

08

±

6

.

22

44

.

62

±

4

.

14

44

.

67

±

4

.

46

44

.

97

±

4

.

44

55

.

04

±

2

.

86

Textures

AUROC

↑

99

.

84

±

0

.

01

96

.

82

±

0

.

71

95

.

36

±

0

.

35

96

.

46

±

0

.

35

88

.

32

±

2

.

08

90

.

12

±

1

.

32

89

.

61

±

1

.

50

89

.

56

±

1

.

48

90

.

10

±

0

.

92

FPR95

↓

0

.

00

±

0

.

00

25

.

71

±

4

.

83

26

.

90

±

3

.

52

35

.

71

±

2

.

27

42

.

41

±

6

.

28

35

.

85

±

4

.

05

34

.

99

±

4

.

33

36

.

02

±

4

.

18

49

.

10

±

3

.

06

iSUN

AUROC

↑

99

.

97

±

0

.

02

95

.

60

±

0

.

65

95

.

74

±

0

.

38

94

.

49

±

0

.

25

89

.

06

±

2

.

26

93

.

05

±

0

.

84

92

.

99

±

0

.

90

92

.

88

±

0

.

90

91

.

99

±

0

.

74

FPR95

↓

4

.

28

±

0

.

23

22

.

97

±

2

.

17

32

.

62

±

0

.

13

24

.

99

±

1

.

21

48

.

03

±

2

.

04

45

.

82

±

1

.

07

44

.

87

±

1

.

11

45

.

63

±

1

.

26

57

.

58

±

0

.

97

Places 365

AUROC

↑

98

.

51

±

0

.

10

94

.

95

±

0

.

53

93

.

76

±

0

.

12

94

.

93

±

0

.

22

85

.

65

±

0

.

77

88

.

68

±

0

.

28

88

.

53

±

0

.

30

88

.

42

±

0

.

29

88

.

06

±

0

.

25

FPR95

↓

0

.

92

15

.

91

21

.

44

20

.

35

35

.

50

35

.

07

34

.

09

35

.

00

49

.

21

Mean

AUROC

↑

99

.

55

97

.

02

96

.

31

96

.

57

90

.

94

92

.

65

92

.

55

92

.

43

91

.

64

Method type

OE

Training

Training

Training

Post-hoc

Post-hoc

Post-hoc

Post-hoc

Post-hoc

Augmentations

Weak

Strong

Strong

Strong

Weak

Weak

Weak

Weak

Weak

Auxiliary outlier data

✓

✗

✗

✗

✗

✗

✗

✗

✗

I.14 Non-OE Baselines

To confirm the prevailing notion that OE methods can improve the OOD detection capability in general, we compare Hopfield Boosting to 3 training methods (Sehwag et al., 2021; Tao et al., 2023; Lu et al., 2024) and 5 post-hoc methods (Hendrycks & Gimpel, 2017; Hendrycks et al., 2019b; Liu et al., 2020, 2023; Djurisic et al., 2023). For all methods, we train a ResNet-18 on CIFAR-10. For Hopfield Boosting, we use the same training setup as described in section 4.2. For the post-hoc methods, we do not use the auxiliary outlier data. For the training methods, we use the training procedures described in the respective publications for 100 epochs. Notably, all training methods employ stronger augmentations than the OE or the post-hoc methods. The OE and post-hoc methods use the following augmentations (denoted as 'Weak'):

RandomCrop (32x32), padding 4

RandomHorizontalFlip

The training methods use the following augmentations (denoted as 'Strong'):

RandomResizedCrop (32x32), scale 0.2-1

RandomHorizontalFlip

ColorJitter applied with probability 0.8

RandomGrayscale applied with probability 0.2

Table 10 shows the results of the comparison of Hopfield Boosting to the post-hoc and training methods. Hopfield Boosting is better at OOD detection than all non-OE baselines on CIFAR-10 in terms of both mean AUROC and mean FPR95 by a large margin. Further, Hopfield Boosting achieves the best OOD detection on all OOD data sets in terms of FPR95 and AUROC, except for SVHN and LSUN-Crop, where PALM (Lu et al., 2024) shows better AUROC results. An interesting avenue for future work is to combine one of the non-OE based training methods with the OE method Hopfield Boosting.

J Informativeness of Sampling with High Boundary Scores

This section adopts and expands the arguments of Ming et al. (2022) on sampling with high boundary scores.

We assume the extracted features of a trained deep neural network to approximately equal a Gaussian mixture model with equal class priors:

Using the MHE and sufficient data from those distributions, we can estimate the densities p ( ξ ) , p ( ξ | ID ) and p ( ξ | AUX ) .

Lemma J.1. (see Lemma E.1 in Ming et al. (2022)) Assume the M sampled data points o i ∼ p AUX satisfy the following constraint on high boundary scores E b ( ξ )

Proof. They first obtain the expression for E b ( ξ ) under the Gaussian mixture model described above and can express p ( AUX | ξ ) as

When defining f AUX ( ξ ) = 1 2 σ 2 ( || ξ -µ || 2 2 -|| ξ + µ || 2 2 ) such that p ( AUX | ξ ) = σ ( f AUX ( ξ )) = 1 1 + exp( -f AUX ( ξ )) , they define E b as follows:

Then they have

Therefore, the constraint in Equation (141) is translated to

As max i ∈ M | µ T o i | ≤ ∑ M i =1 | µ T o i | given a fixed M , the selected samples can be seen as generated from p AUX with the constraint that all samples lie within the two hyperplanes in Equation (150).

Parameter estimation. Now they show the benefit of such constraint in controlling the sample complexity. Assume the signal/noise ratio is large: || µ || σ = r ≫ 1 , and ϵ ≤ 1 is some constant. Assume the classifier is given by

where o i ∼ p AUX and x i ∼ p ID. One can decompose θ . Assuming M = N :

We would now like to determine the distributions of the random variables || η || 2 2 and µ T η

Now we would like to determine the distribution of µ T η :

Therefore, for || η || 2 2 we have

If d ≥ 2 we have that 5

Concentration bounds. They now develop concentration bounds for || η || 2 2 and µ T η . First, we look at || η || 2 2 . A concentration bound for χ 2 d is:

By assuming x = d 8 σ 2 we obtain

and thus, the above bound can be simplified when assuming d ≥ 2 as follows:

For || ω || 2 2 , since all o i is drawn i.i.d. from p AUX, under the constraint in Equation (150), the distribution of ω can be seen as a truncated distribution of η . Thus, with some finite positive constant c , we have

Now, we develop a bound for µ T η . A concentration bound for N ( µ, σ 2 ) is

Due to symmetry, we have

We can rewrite the above bound using the absolute value function.

Benefit of high boundary scores. We will now show why sampling with high boundary scores is beneficial. Recall the results from Equations (150) and (154):

Using the two facts above and the triangle inequality we can bound | µ T ω | :

The triangle inequality is

By applying µ T η || µ || ∼ N (0 , σ 2 N ) to the above bound we obtain

Assuming t = ( σ || µ || ) 1 / 2 we obtain

Developing a lower bound. Let

hold simultaneously. The probability of this happening can be bounded as follows: We define T and its complement ¯ T :

With P ( T ) + P ( ¯ T ) = 1 . The probability P ( ¯ T ) can be bounded using Boole's inequality and the results in Equations (171), (172) and (179):

Further, we can bound the probability P ( T ) :

Therefore, the probability of the assumptions in Equations (189), (190), and (191) occuring simultneously is at least 1 -(1 + c ) exp( -d/ 8 σ 2 ) -2 exp( -|| µ || N 2 σ ) .

By using the triangle inequality, Equation (152) and the Assumptions (189) and (190) they can bound || θ || 2 2 :

The reverse triangle inequality is defined as

Using the reverse triangle inequality, Equations (152), (188) and Assumption (191) we have that

They have assumed that the signal/noise ratio is large: || µ || σ = r ≫ 1 . Thus, we can drop the absolute value, because we assume that the term inside the || is larger than zero:

We have

if r ≥ 1 . 36602540378443 . . . and ϵ ≤ 1 , and therefore

Because of Equation (203) and the fact that if x ≤ y and sgn ( x ) = sgn ( y ) then x -1 ≥ y -1 we have

We can combine the Equations (214) and (215) to give a single bound:

we define θ such that µ T θ > 0 and thus

The false negative rate FNR( θ ) and false positive rate FPR( θ ) are

As N ( x ; µ, σ 2 ) = N ( -x ; -µ, σ 2 ) , we have FNR( θ ) = FPR( θ ) . From Equation (217) we can see that as ϵ decreases, the lower bound of µ T θ || θ || will increase. Thus, the mean of the Gaussian distribution in Equation (218) will increase and therefore, the false negative rate will decrease, which shows the benefit of sampling with high boundary scores. This completes the extended proof adapted from (Ming et al., 2022).

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims made in the abstract and introduction regarding improvement in OODdetection capabilities are backed by extensive experiments in Section 4. Our theoretical results in the Appendix (most notably Sections G.1, H.1, and J) support the applicability of our method for OOD detection with OE.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Section 4.3 and Appendix I.7 show that when subjecting Hopfield Boosting to data sets that have been designed to show the weaknesses of OE methods, we identify instances where a substantial number of outliers are wrongly classified as inliers by Hopfield Boosting and EBO-OE.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Our theoretical results include the probabilistic interpretation of Equation (6) in Appendix G.1, the suitability of Hopfield Boosting for OOD detection in Appendix J, and the connection of Hopfield Boosting to RBF networks (Appendix H.1) and SVM (Appendix H.3). The proofs of our claims are contained in the respective Appendices.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The details of the experiments we conducted are explained in section 4.2, and the data sets are publicly available. The code of Hopfield Boosting is included in the submission.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the code to reproduce the experimental results of Hopfield Boosting in the submission. All data sets used are publicly available. Descriptions how to run the code and how to obtain the data sets come with the code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide the training and OOD test data sets, the optimizer, and the hyperparameters we tested and selected, as well as the selection procedure in Section 4.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide estimates of the standard deviation for the main results of our paper on Hopfield Boosting and on the compared methods in the Tables 1, 4, 2, 3, 6, 10. The standard deviations were estimated across five training runs, which is stated in the captions of the respective tables.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide a detailed description of the compute resources we used as well as the amount of compute our experiments required in Appendix I.12.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read the NeurIPS Code of Ethics and found that our work conforms with the code of ethics. More specifically, we did not include any human subjects in our work. We excluded deprecated data sets from our work.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We provide a discussion of positive and negative societal impacts in Appendix E.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our work is concerned with the improvement of out-of-distribution (OOD) detection algorithms on small- to mid-scale vision data sets, and is therefore not considered to pose a high risk.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We provide a list of the data sets we use (and cite the original authors) in Section 4.2. We provide a list of the respective licenses (where applicable) in Appendix I.13.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The code that is included in the submission comes with a READE.md file, which contains all instructions how to run the code to reproduce the experiments.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce Hopfield Boosting for addressing the out-of-distribution detection task. This algorithm trains a model with two heads: one for normal classification and the other for assigning an out-of-distribution score. The OOD score head maintains a list of in-distribution samples and out-of-distribution samples. It uses a modern Hopfield network to leverage the similarity between the inference sample and stored samples to assign the OOD score.\nThey further improve their performance by first identifying OOD samples that are near the border of the decision boundary using an MHE-based energy function. Then, they use these samples as OOD samples for the OOD score head instead of all auxiliary OOD samples. They evaluated their method on CIFAR-10 and ImageNet-1k and achieved state-of-the-art results.\n\nSTRENGTHS:\n1. Large-scale experiment on ImageNet.\n2. Novel use of Hopfield Boosting.\n3. Good toy examples that clearly illustrate their point.\n4. Low computation requirement.\n\nWEAKNESSES:\n1. Only 32x32 images were used.\n2. OE methods generally require auxiliary OOD data. As a result, evaluation on a dataset that does not have common classes with the auxiliary OOD dataset is crucial to assess the method's generalization.\n\nQUESTIONS:\n1. What is the relation between ImageNet-21k, which is used as auxiliary OOD data, and the OOD data used for testing? How many semantically similar classes does ImageNet-21k share with each of them? If it shares a considerable number of classes, a test dataset that shares fewer semantically similar classes would be helpful.\n2. ImageNet augmentation: Please explain the augmentation used on in-distribution data for ImageNet. Is it the same as mentioned in line 970? Random cropping to 32x32 for images that are very large, like those in ImageNet-1k, seems unreasonable as 32x32 parts of the image 3. may not contain any meaningful object. Could you provide some examples of in-distribution images post-augmentation?\n4. Can you provide the accuracy of classification head of the network?\n5. What is the reason for cropping images to 32x32 in the ImageNet-1k case? Why not use the more common 224x224 image size?\n\nLIMITATIONS:\nThe authors presented the limitations of their work in section H.6, which could suggest that their method lacks generalization in at least some areas.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA novel method for identifying OOD samples is presented using Hopfield boosting.  A classifier is trained using a loss function which sharpens the decision boundary between inlier and outlier data samples, where outlier data samples are purposefully drawn from an auxiliary dataset.  By sampling examples close to the decision boundary and determining the modern hopfield energy of the sample, the decision boundary is adjusted to exclude OOD samples from the auxilliary dataset.  The method is demonstrated twice using a low resolution and high resolution example.  In both examples, the Hopfield boosting method outperforms state of the art on average.  An extensive appendix includes additional background, theoretical results, and experimental results.\n\nSTRENGTHS:\nThe method is original, theoretically sound, and useful.  The results are clearly presented, to the point where a reader may be able to implement the algorithm without referencing the provided code.  The research question is impactful, and the result presented in this paper is a theoretical and experimental advancement in the field.\n\nWEAKNESSES:\nThe primary weakness is that the paper does not conclusively show that the method does not lead to overfitting.  The models are trained with a classification loss, but the authors state \"the toy example does not consider the inlier classification task that would induce secondary processes, which would obscure the explanations.\"  However, failure to report these results in the context of the decision boundary refinements shown in Figure 2, 3b and 6 opens the work to questions of model overfitting.  \n\nThis concern is only partially lifted by near perfect results on the test datasets.  Because the false positive rate (FPR) is reported with respect to an OOD test dataset, there is a question of whether the model is losing in-distribution detections.   Robustness to OOD samples of the original dataset, robustness to adversarial instances, and catastrophic forgetting are all implicated in the sharpening of decision boundaries; this work avoids acknowledging these cases and prevents inference about the implications by omitting the classifier performance for the trained models.  This can be resolved by reporting the model F1 scores in the appendix for the ID classification task. If the models were trained using the Hopfield Boosting alone, then this would not be necessary.  But the presence of the classifier loss means the secondary effects have implications for the semantic meaning of the decision boundary.\n\nQUESTIONS:\n1. How do these results compliment/contrast with results suggesting that a smooth decision boundary is more robust to adversarial samples? See for example [Anderson (2022)](https://proceedings.mlr.press/v168/anderson22a)\n\n2. Most computer vision datasets are expected to have some features in common, which motivates transfer learning techniques.  What justification was used to determine that the test datasets would be strictly OOD w.r.t. the training datasets CIFAR or ImageNet?\n\n3. Is there any analysis which shows that the decision boundaries sharpened during Hopfield Boosting are semantically meaningful, and not based on noise of the dataset?\n\n4. The ResNet-50 model is fine-tuned rather than trained from scratch; what dataset was used to train the pre-initialized weights? Why was the model not trained from scratch?  Because the feature space was already formed (4 epochs is not enough to significantly shift the model), does this imply that the model classification layer is likely identifying noise?\n\nLIMITATIONS:\nPlease see the questions above.  The remaining limitations are thoroughly addressed in the work.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- The paper introduces a novel approach to improve OOD detection by leveraging modern Hopfield energy. The proposed method, called Hopfield Boosting, utilizes auxiliary outlier data to refine the decision boundary between ID and OOD data. By focusing on hard-to-distinguish auxiliary outlier samples near the decision boundary, the method enhances the ability to detect OOD samples.\n\nSTRENGTHS:\n- The incorporation of modern Hopfield energy and auxiliary outlier data for OOD detection is interesting. The method effectively sharpens the decision boundary between ID and OOD data by utilizing the energy measure, which quantifies dissimilarity between data instances and stored patterns. By sampling informative outliers close to the decision boundary, Hopfield Boosting ensures that the model learns a more precise boundary, leading to better OOD detection performance.\n- The proposed method achieves significant improvements in OOD detection metrics compared to existing methods.\n\nWEAKNESSES:\n- The experimental results in Table 3 show that the effect of weighted sampling is not very significant. Despite this, the proposed method still outperforms existing approaches. It is unclear whether these performance gains are primarily due to the model's structure (two heads for classification and energy computation), the OOD score defined in Equation (13), or the proposed $L\\_{OOD}$. This ambiguity makes it difficult to pinpoint the key factors contributing to the improved performance and understand the true efficacy of each component of the proposed method. More detailed ablation studies are needed to disentangle and evaluate the contributions of these individual components.\n- The paper lacks a detailed explanation of how the hyperparameter $\\beta$ is determined. While it is mentioned that $\\beta$ is selected through a validation process, the specific metrics or criteria used to decide the optimal value are not discussed. Providing a clear methodology for $\\beta$ selection, including the validation metrics used, would enhance the reproducibility and transparency of the proposed approach.\n- There is a need for discussion regarding the impact of the proposed method on in-distribution (ID) accuracy. The proposed method could potentially negatively affect the performance of the classification head, which would undermine its overall utility. Addressing how the method influences ID accuracy and providing empirical evidence to show that it does not significantly degrade classification performance is crucial for establishing the method's efficacy and practical value.\n\nQUESTIONS:\nThe questions related to the methodology and experimental results have been addressed in the Weaknesses section.\n\nLIMITATIONS:\nThe authors have adequately discussed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the crucial task of out-of-distribution (OOD) detection, essential for the safe deployment of machine learning models in real-world scenarios. Within the domain of OOD detection, outlier exposure methods, which use auxiliary outlier data during training, have shown significant improvements in OOD detection performance. In this paper, the authors introduce Hopfield Boosting, a boosting approach that utilizes modern Hopfield energy together with the OOD data to refine the decision boundary between in-distribution (ID) and OOD data. By focusing on hard-to-distinguish auxiliary outlier examples near the decision boundary, Hopfield Boosting enhances the model's ability to differentiate between ID and OOD data. The authors of the paper demonstrates the effectiveness of the proposed method empirically with several benchmark datasets.\n\nSTRENGTHS:\n- The paper is mostly well written and easy to follow. \n- The proposed method is technically sound. \n- Authors of the paper carry out ample experiments. \n- The proposed method achieves good performance.\n\nWEAKNESSES:\n- A major complaint I have for the paper is the lack of novelty. As mentioned by the authors of the paper in Section 3.4, the proposed method is an incremental extension of the prior work which also proposed to use Hopfield Energy for OOD detection. The main difference lies in the fact that the proposed method also utilizes auxiliary OOD data. Despite the effectiveness of the proposed method, outlier exposure is a well understood method to enhance the effectiveness of OOD detection, and it is not surprising to me that utilizing OOD data with Hopfield Energy enhances the effectiveness of the proposed method.\n\nQUESTIONS:\n- The paper introduces the MHE-based energy function to combine OOD data with Hopfield Energy. Would it be possible to leverage OOD data in other ways so that it is more similar to HE or SHE in the prior work [1]? If so, how does the proposed method compare to such a naive way of combining HE/SHE with outlier exposure? I think an ablation study like this would help strengthen the novelty of the paper. \n- It was stated that 50k samples were used during inference to compute the OOD score. How much better/worse does the proposed method get when we vary the number of samples used? \n- What if IN and OOD data are highly imbalanced? \n\n[1] \"Out-of-distribution detection based on in-distribution data patterns memorization with modern Hopfield energy\"\n\nLIMITATIONS:\nYes the authors of the paper adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}