Out-of-Distribution Detection based on In-Distribution Data Patterns Memorization with Modern Hopfield Energy
Authors: Jinsong Zhang
Qiang Fu
Xu Chen
Lun Du
Zelin Li
Gang Wang
xiaoguang Liu
Shi Han
Dongmei Zhang
Venue: ICLR 2023 poster (2023)
Keywords: Out-of-Distribution detection
Hopfield Energy
Hyperparameter-Free

Abstract: Out-of-Distribution (OOD) detection is essential for safety-critical applications of deep neural networks. OOD detection is challenging since DNN models may produce very high logits value even for OOD samples. Hence, it is of great difficulty to discriminate OOD data by directly adopting Softmax on output logits as the confidence score. Differently, we detect the OOD sample with Hopfield energy in a store-then-compare paradigm. In more detail, penultimate layer outputs on the training set are considered as the representations of in-distribution (ID) data. Thus they can be transformed into stored patterns that serve as anchors to measure the discrepancy of unseen data for OOD detection. Starting from the energy function defined in Modern Hopfield Network for the discrepancy score calculation, we derive a simplified version SHE with theoretical analysis. In SHE, we utilize only one stored pattern to present each class, and these patterns can be obtained by simply averaging the penultimate layer outputs of training samples within this class. SHE has the advantages of hyperparameterfree
and high computational efficiency. The evaluations of nine widely-used OOD datasets show the promising performance of such a simple yet effective approach and its superiority over State-of-the-Art models. Code is available at https://github.com/zjs975584714/SHE ood detection.

Published as a conference paper at ICLR 2023
OUT-OF-DISTRIBUTION DETECTION BASED ON IN-DISTRIBUTION DATA PATTERNS MEMO RIZATION WITH MODERN HOPFIELD ENERGY
Jinsong Zhang 1+1, Qiang Fu 2-1, Xu Chen 2 Lun Du 2, Zelin Li Gang Wang , XiaoGuang Liu
Shi Han 2, Dongmei Zhang?
College of Computer Science, Nankai University, Tianjin, China
2 Microsoft Corporation, Beijing, China
Izhangjs, s, wgzwp, libxg)enbji nankaaiili.a..i.. nankai. edu.
(qi fu, xu. chen, lun. du, v-zelinli, sh hhinan, 20m com
ABSTRACT
Out-of-Distributie (OOD) detection is essential for safety-critical applications of deep neural networks. OOD detection challenging since DNN models may produce very high logits value even for OOD samples. Hence, it is of great dif- ficulty to discriminate OOD data by directly adopting Softmax on output logits the confidence score. Differently, paradigm. In more detail, penultimate layer out-
detect the OOD sample with Hopfield
energy in a store-then-compiee puts on the training set are considered as the representations of in-distribution (ID) data. Thus they can be transformed into stored patterns that serve as an- chors to measure the discrepancy of unseen data for OOD detection. Starting from the energy function defined in Modern Hopfield Network for the discrep- ancy score calculation, we derive simplified version SHE with theoretical anal- ysis. In SHE, we utilize only one stored pattern to present each class, and these patterns can be obtained by simply averaging the penultimate layer outputs training samples within this class. SHE has the advantages of hyperparameter- free and high computational efficiency. The evaluations of nine widely-used OOD datasets show the promising performance of such simple yet effective approach and its superionity over State-of-the-AM models. Code is available at https://giinbbboomm////555444411
INTRODUCTION
Deep Neural Network (DNN) has yielded remarkable achievements in broad range of fields in recent years (He al.||220100 2016 Huang et al. 2017), and is extensively deployed in numerous real-world scenarios (Krizhevsky et al. 2017 Redmon & Farhadi [017]. One of its powerful capabilities lies in the promising generalization ability from training data to unseen in-distribution (ID) data. However, the finite training data cannot guarantee the completeness of data distribution. so it is inevitable to encounter -fitt--distribution (OOD) data. The Softmax-based prediction allows OOD samples to gain high confidence in specific classes, which is unacceptable in practice, especially for safety- related areas. It can lead to erroneous collisions in autonomous driving or irreparably large financial losses. Therefore, OOD detection is critical with respect to Al safety (Amodei et al. 2016).
Existing efforts on OOD detection for DNN can be roughly divided into two categories. The first group of approaches requires designing and retraining new auxiliary networks specifically for OOD detection rather than directly using already trained models (Denouden et al. 20188] DDeriss & Tay- [or 2018 Yu & Aizawa 2019 Zhang et al. 2020). The objective should be modified accordingly and OOD samples are sometimes introduced to train the new networks. However, it is almost im- possible to exhaust all kinds of OOD samples, and retraining can also be cumbersome. The methods
'Equal contribution
This work was done during interning at Microsoft.
1 Corresponniig author.

---

Published as a conference paper at ICLR 2023
of the second category elaborate the confidence design for the network outputs, e.g., the logits, the Softmax probability (Liang et al. 20177] Liu et al.) [0200] Sun et al. 2021) or embedding features (Lee et al., 2018: Schwag et al.) 2021 Sun et al. (2022). By these means, there is need to modify the backbone model and the objective, which motivates us to explore OOD detection in this manner.
In deep learning, the intermediate layer output can be regarded as the representation of input data in the latent space. Further, as shown in Figure. [] (left), guided by the training process, such repre- sentations of ID samples of the same category tend to present some common patterns for prediction accuracy. In contrast, these representations of OOD samples should deviate from such commonality since they are not considered during the training process. Based on this intuition, OOD detection can be formulated as a store-then-compare process: representations of ID samples within each category are maintained during the training procedure as stored patterns, and test pattern will be compared with the store patterns. If there is a noticeable discrepancy, then it can be judged as an OOD sample.
The critical question is how to measure the discrepancy between the OOD sample and the stored patterns under this setting? To accomplish this goal, we adopt the key idea of classic memory network, Hopfield Network. The Hopfield Network (binary state) was first introduced in (Hopfield 1982) and (Hopfield) 1984) proposed continuous state version. Modern Hopfield Network (both continuous and binary) was introduced in (Krotov & Hopfield 2016), and (Ramsauer et al. 2020) proposed a new energy function for continuous state Hopfield networks and point out the relationship with the transformer.
Hopfield Network targets recovering distorted test patterns so that the recovered patterns are as close to the stored patterns as possible. It achieves this goal by specific update rules that minimize the predefined energy function. The more the recovered pattern resembles the stored pattern, the lower the energy is. Therefore, the energy function serves a vital role as it indicates the gap between the recovered patterns and the stored patterns. For OOD detection tasks, the energy function of the Hopfield Network well-suited desirable measure of the discrepancy between the OOD sample and the stored patterns.
In this paper, we propose a new OOD detection method HE with memorization of ID data patterns and the Modern Hopfield Energy function (Ramsauer et al. 2020). In more detail, the representa- tions of training ID samples are stored as patterns for each category in advance, and OOD samples are detected under the energy function. As the intermediate results are more informative than the highly- compeesed final output logits, we preserve the outputs of the penultimate layer (i.c., the input of the final output layer) representations. Furthermore, to address the challenges of the memory cost of pattern memorization, we derive a Simplified Hopfield Energy function- method SHE. In SHE, only one pattern is required for each category and there is hyperparameter be tuned. Theoretical analysis proves the effectiveness of our design. The remarkable performances on nine widely-used OOD detection datasets on three different networks demonstrate the superiority of our proposed SHE (and HE) over state-of-the-ar methods. We summarize the main contributions of our paper as follows:
We propose a Modern Hopfield Energy-based method HE for ouistf-diitribution detection. It uses store-then-comee compare paradigm that compares test samples with pre-stored patterns to measure the discrepancy from in-distribution data according to Hopfield energy.
We derive a simplified version of HE, named as SHE, which greatly reduces the memory and the computation cost. In addition, SHE is hyperparameter-ffee Theoretical analysis is conducted illustrate the effectiveness of SHE.
Extensive experiments on nine OOD detection datasets of three prominent computer vi- sion backbone networks indicate both the effectiveness and the efficiency of our designed methods. Experiments on large-scale datasets (e.g.. ImageNet- lkk also show the superior our approach. In-depth analysis and ablation studies are also included to shed light on the mechanism behind it.
2 RELATED WORK
Network Redesign and Retrain. Given original network architectures are designed for target tasks like classification, straightforward paradigm of OOD detection is elaborate on the network
2

---

Published as a conference paper at ICLR 2023
20
1_00
10_0
0_3
ID_2
-20
1D_3
10.6
10
40-
0.7
-00
10_5
-60
10_9
-80
10,
OCC Data
10_9
-80
-80
-00
-40
-20 20
40
60
90
-80 -60
-40 -20 20
40
60
80
Figure 1: Visualization of the distribution of ID/OOD patterns (left) and ID/Stored patterns (right) by t-SNE (Van der Maaten & Hinton 2008), the ID/OOI) refers to CIFAR1O (Krizhevsky et al. 2009) and LSUN (Yu et al. 2015)). respectively. The backbone network is ResNet18 (He et al, 2016 In the left figure, the OOD patterns are away from the ID patterns. In the right figure, each pentagram denotes the stored pattern in SHE corresponding to each category, from the figure we can see the stored pattern can represent the corresponding ID patterns well.
chitecture and the objective, and retrain a new network for OOD detection. (Denouden et al. 2018) used ID data train AutoEncoder, and OOD samples were supposed to have high reconstruction errors. VDeisses & Taylor 2018)) expanded the original network with an extra branch to predict confidence score and obtained the final output with this confidence score. (Zhang et al.) 2020) pro- posed flow-based model that retrieved the feature map of test samples and fed it into an additional network. Such an additional network was designed only for forecasting whether the test sample is ID or not. (Yu & Aizawa 2019) firstly trained encoder in supervised way, and then two clas- sifiers for ID samples and OOD samples were trained with an unsupervised objective, respectively. All these methods aim to redesign or introduce new layers auxiliary networks with corresponding objectives for OOD detection. Retraining a network can be extremely time- conseminn,,nng, especially when the parameter scale is substantially large. The modified objective that considers OOD detec- tion in addition to the original task may also have side effects of degrading model performance on the original task. In addition, some methods require OOD samples input, which imposes addi- tional requirements on the datasets. A potential risk is that the model will achieve poor results on data beyond the distribution of the trained OOD samples. Unlike those approaches, our method does not need to make any changes (including both architecture and objective) to the original network and does not need to do any additional training, which is plausible property for real applications.
Network Output Transformation. Apart from adjusting network structure retraining the net- work with revised objectives, transforming network outputs to obtain the desired measure is the other classic OOD detection paradigm. The first study on network output transformation was pro- posed by (Hendrycks & Gimpel 2016). which used the Maximum Softmax Probability (MSP) to measure the confidence of test samples. Intuitively, ID data is more likely to obtain high confi- dence from Softmax measure than OOD samples. In (Liang et al. 2017). input data were perturbed with ID-sample- friendly perturbations and the Softmax probability parameter T; thus OOD data and ID data are more separable. (Lee et al. 2018) first generated class-conditional Gaussian distribution from middle layer outputs of the already trained network on training data and then calculated the confidence of test samples under the Mahalanobis distance measure. (Serrà al. 2019) made assumptions about the complexity of output and input images, and they advocated estimating the complexity of the input image to impact the output for efficient OOD detection. (Liu et al. 2020) detected the OOD samples by an energy-based based score function on the final output logits. Note that such energy function is different from ours as it calculates such score merely based on the output logits of the testing sample instead of comparing with stored patterns. Based on the observation that the mean activation of the OOD sample had larger varia- tions, (Sun et al., 202T) set a threshold to clip the output of the penultimate layer, thereby reducing the output magnitude of OOD samples in the last layer. These approaches can be directly applied to OOD detection tasks without additional training, which is more practical for real-world applications. (Sehwag et al. 2021) utilize the advantage of self-supervised training. (Sun al. 2022) calculate
re- scaled by temperature
3

---

Published as a conference paper at ICLR 2023
all the Euclidean distance with each pattern of train sample and the k-th sorted distance as the metric for OOD detection.
METHODOLOGY
In this section, we first introduce the preliminaries of the OOD detection task and the Hopfield Net- work. Then elaborate on how to leverage the critical concepts of the Hopfield Network on OOD detection. More precisely. the energy function defined in the Modern Hopfield Network (Ram-) sauer al. 2020) is introduced
the basis of store-then-compare OOD detection paradigm. A simplified energy function is further proposed to reduce the memory demand, which is of high computational efficiency and free of hyperparameters. Finally, we compare the difference in pattern choice, i.e., patterns derived from the penultimate layer outputs versus final logits. We also conduct in-depth theoretical analyses of our method.
3.1 OUT-OF- DIN-RIISTTIBIIOON DETECTION
A neural network f aims to learn mapping function from training sample æ to its corresponding label y as y = f(:; Ø) with parameter 0. Then a testing sample x' fed into the trained network for the prediction y¹. When x' and training sample æ from the same data distribution, then x' called an ID sample; otherwise, it is regarded as an OOD sample. Prediction results for OOD samples in turn fail be meaningful. More severely, blindly classifying OOD samples into any existing class may raise fatal risks in safety-critical scenarios. Thus, the OOD detection task is to design measure function x') that allows OOD samples be as clearly distinguishable from ID samples as possible. Eventually, OOD detection can be formulated as follows:
FOOD
if D(f:
x²'
ID
if D(f: =1 =1
3.2 HOPFIELD NETWORK
Hopfield Network (Hopfield 1984 Krotov & Hopfield 2016 Ramsauer et al. 2020) store and retrieve continuous patterns. By minimizing the predefined energy function, it can gradually update the input test pattern & < pax to a certain converged pattern that similar to one of the stored patterns. We denote all stored patterns as stored pattern set S E Idx with each column 8, [Rdx1 representing one specific stored pattern, and is the total number of the stored pattern. Here d is the dimension of patterns. Thus, the energy function aims to guide the updating procedure in the Modern Hopfield Network can be written as:
1 E Energy LSE (B,E*S) ( 5
LSE(S, -- llog Σ exp (Be,)
(3)
where LSE denotes the log-sum-exp function and is defined in Eq. 3 and are constant. The vector e denotes gd S, where ey represents the inner product of the input test pattern & and the j-th stored pattern 8j. The second term g on the right of Eq. 2 serves as a regularization on the magnitude of Revisiting the energy function, we can find that is essentially measure that depicts the similarity between training patterns and testing patterns.
3.3
HE: OOD DETECTION WITH MODERN HOPFIELD ENERGY
As described above, obvious that the energy function of Modern Hopfield Network can be an appropriate candidate for measuring the discrepancy between OOD instances and ID instances. We denote all stored patterns for class as a stored pattern set S, E R N. with each column $:y E Rddx representing one specific stored pattern, and N, is the total number of the stored patterns within class More precisely, we preserve the penultimate layer outputs of ID training samples as stored patterns: for each class a corresponding stored pattern set S, is derived from the penultimate layer
4

---

Published as a conference paper at ICLR 2023
outputs of ID training samples that are correctly classified by the network within this class. When testing a new sample, we can obtain its penultimate layer output & as well as its prediction result, e.g., class i, from the trained model. Then, just need to conduct the calculation between & and the corresponding stored pattern set S; to obtain the similarity score.
Notice that, there is the magnitude regularization item E in the energy function Eq. 2 Since is introduced to prevent the imput pattern from scaling-up during the pattern updating process of Modern Hopfield Network, it is not necessary when measuring the discrepancy between the input pattern and the stored patterns. Thus we omit such a term as the magnitude of the input pattern & can also provide some information for the ID and OOD discrimination. Also, we omit the constant e in Eq. 2 because the constant in the measure function does not change the OOD detection result. In summary, the Hopfield energy-based OOD detection measure can be denoted as denotes the classfication result of by the already trained model):
(4) Note that, Eq. measures the similarity instead of discrepancy, which means the higher score indicates ID data while lower score as OOD data. is worth mentioning that, different from most traditional OOD detection methods that only consider the information from the input test sample itself with the trained network, we leverage the information from all training samples of the predicted class to make the comparison for better OOD detection.
HE(()) LSE (3,5"S:).
3.4 SHE: OOD DETECTION WITH SIMPLIFIED HOPFIELD ENERGY
Although HE has the theoretical foundation from the Modern Hopfield network and is proven to be effective through empirical evaluation, the need to store patterns of all correctly classified training samples may prevent it from generalization to real-world applications. Particularly when the scale of the dataset is extremely large the latent representation is ultra-high dimensional, it will impose a considerable burden on the storage and the computation. During the evaluation, we discover that the hyperparameter / in Eq. 4 should be small enough in case that there is any element esy (denotes the inner product between the testing pattern and the j-th stored pattern of class i) with extra large value, which will degrade the robustness of OOD detection with large When 3 is relatively small, we can transform Eq. with Taylor series (here we use two Taylor series as exp(x) = 1 + x and log(1 + x) RJ x)) by:
N, exp (Ber))
N Σ j=1
LSE(S, e,)
log
Σ
log
(1 + Berz)
3
N:
1 E)= Eij
1 log
N:
Σ
Beri
1
log N;
(5)
3
3
N₂
j=1
log N; +
1 log
1+
1+
BET
===1=
3
N
All test samples that are predicted to be class share the same stored patterns size N,, so the first term 8 log N₂ remains the same for them and can be regarded as a constant. Thus, we ignore the first term, and the measure function Eq. 4 can be simplified from the LSE function (Eq. 3 to the inner product of g and 5; because LSF(3, e,)
( is positively related to g" Sy:
SHE(E) = =
(6)
Here S, is defined as:
N 5,
1
Sij-
N
We can interpret SHE from another perspective: S, can be viewed as the average of vectors from the stored pattern set S, In other words, stored pattern set S, is degraded to representation vector S, Considering the redundancy of patterns that frequently appears in deep learning and samples from
5

---

Published as a conference paper at ICLR 2023
the same class usually have similar patterns, it is reasonable to prune the stored pattern set into single average vector, which also shown in Figure. (right). enables us to eliminate the memory overhead of storing large amount of ID data patterns, and further reduces the computational cost as well. Besides, the only hyperparameter 3 also disappears, indicating that we do not need to tune any hyperparameter. In summary, SHE is highly efficient regarding both storage and computation and does not have any hyper parameter to tune. Such properties can indeed facilitate the deployment of SHE in practice.
3.5 PENULTIMATE LAYER VERSUS LOGITS LAYER FOR PATTERN STORAGE
In this section, we analyze the benefits of choosing outputs from the penultimate layer, compared with the logits layer, as the pattern.
Intuitive Explanation. Given that most OOD detection methods usually use the output logits, one interesting question is how to choose the layer output as the patterns, the penultimate layer output, or the output logits. Note that, when we calculate the energy function by Eq. o the stored pattern S; comes from the same category i as the testing pattern & is classified. Therefore, when we use the output logits as patterns, no matter & comes from an ID or OOD sample, its maximum value position of logits is always the same as the maximum value position of logits of S: by design, which the category index of g. We call such alignment of maximum value position of logits as "Peak Alignment" which leads to high energy function score for the OOD pattern more easily. It in turn raises the difficulty of discriminating ID and OOD samples. However, when we use the penultimate layer output as patterns, there is no such "Peak Alignment" effect between & and S; since the value of the penultimate layer output is not so concentrated distributed, promoting the similarity score calculated from energy function more separable for ID and OOD patterns shown in Figure. 2 Moreover, we provide the theoretical analysis in the Appendix
Energy Score from Penultimate Layer
Energy Score from Logits Layer
Energy Score for ID
25
Energy Score for ID Energy Score for 000
60
Energy Score for 000
Energy Score
50
Energy Score
15
40
30
10-
20
5-
10
250
1000 1500 1750 Samples from ID and OOD
250 750 1000 1250 1500 1750 2000 Samples from ID and 00D
Figure 2: Distribution of the Hopfield Energy Score calculated from 2,000 ID and 2,000 OOD samples, the pattern derived from penultimate layer (left) and output logits (right), respectively. When using penultimate layer, the score can be distinguished more for ID and OOD samples. The ID and OOD CIFARTO (Krizhevsky et 2009) and SVHN (Netzer et al. 201T respectively, and the backbone network is ResNet18.
4 EXPERIMENTS
In this section, we conduct experiments on nine OOD detection datasets with three backbone net- works and two ID datasets evaluate the performance of our methods.
4.1 DATASET
There are types of datasets in the experiments: The in-distribution (ID) dataset and the Out- of-distribution (OOD) dataset. The former only utilized during the training procedure, while the latter serves to test models and does not contain any IDD dataset sample.

---

Published as a conference paper at ICLR 2023
ID Dataset. CIFARIO, CIFAR100 and ImageNet- 1k are three ID datasets in experiments. CI- FARIO (Kiihhvssy et al. 5,000 training images and 1,000 testing images. CIFARI00 Krizhevsky et al. (2009) consists of 60,000 images with 100 categories with 500 training images and 100 testing images for each class. ImageNet- 1k (Deng et al. 2009) is composed of 1,350,000 images 1000 different object categories, each containing 1,300 training images and 50 testing images.
2009) is composed of 60,000 images with 10 categories, each containing
OOD Dataset. There are nine OOD dataset for evaluation, including SVHN (Netzer et al. 201
LSUN-C (Yu et al., 2013) (crop) and LSUN-R (Yu et al., 2015) (resize), iSUN (Xu et al. 2013
Places (Zhou et al. 2017). DTD (Cimpoi et al. 2014 4p. SUN (Xiao al. 2010), iNaturalist ( (Van Horn
et al. 2018). Imagenet (resize) (Deng et al. 2009). Details can be found in the original papers.
4.2 EXPERIMENT SETTINGS
Backbone Network. We choose ResNet18 (He et al. 2016), ResNet34 (He et al. 2016) and WRN40-2 (Zagoryyko & Komodakis 2016) our backbone networks, which are trained on the ID dataset CIFARTO and CIFARTOO, respectively. For ImageNet, we choose ResNet50 (He et al., 2016) as backhone network.
To evaluate the performance of our proposed design, We also conduct exper- iments on the Softmax-based approach "MSP" (Hendrycks & Gimpel 2016) and other excellent methods, "Energy" (Liu et al. 2020) "ODIN" (Liang et al.J (017). "Mahalanobis" (Lee et al.) 2018) and "ReAct" (Sun et al. 202T) the "ReAct" here is combined with Energy as described in (Sun et al. 2021) and is the state-of- the-r method before.
Baseline Methods.
Evaluation Metries. The evaluation metries are: (1) the False Positive Rate (FPR95) of OOD samples when the True Positive Rate of in-distribution samples is at 95%; (2) the area under the receiver operating characteristic curve (AUROC), Models with smaller FPR95 and higher AUROC results are regarded as effective. All experimental values are expressed as percentages (%) and the bolded numbers (sometimes colored with gray cell background) denote the best result.
4.3 EXPERIMENTAL RESULT
MSP (FPR95: 38.34)
Energy (FPR95: 20.80)
SHE (FPR95: 3.58)
000
000
000
Frequency
Score
Score
Score
Figure 3: Confidence distribution of ID data and OOD data calculated from ResNet18. ID/OOD refers to CIFAR10 and SUN (Xiao et al. 2010), performance is compared with MSP (Hendryeks & Gimpel 2016) and Energy (Liu et al. 2020).
4.3.1 OVERALL RESULTS OF SHE.
In this section, we demonstrate the effectiveness of our method "SHE" through extensive experi- ments. The experimental results on nine OOD datasets are organized in Table [ taking CIFARIO as ID training data. As shown in Table I MSP has the worse results, while Energy has lot of improvement, illustrating the potential of energy-based solutions for OOD detection. Our approach SHE obtains almost all (26/27) of the best performance on nine OOD datasets for three different backbones. More precisely, our approach reduces the average FPR95 by 16.81% for ResNet18, 6.29% for ResNet34, and 12% for WRN40-2 compared with the best baseline. Our approach performs well when CIFAR100 ImageNet- 1k is choosed as the ID training data while with some limitation, the detailed table 5 (CIFAR and table 7 (ImageNet- Ikk is put in the Appendix A We also illustrate Figure to demonstrate model performance directly. Under the peak of ID score
7

---

Published as a conference paper at ICLR 2023
Table I: OOD detection performance of SHE using CIFARIO as dataset.
Backhone
0OD
Methodk
Network
Datasets
FPR95 (1)
MSP
AUROC (1)
FPR95 (.)
Lieeryy
AUROC (1)
FPR95 (L)
ReAd
AUROC (f)
FPR95 (L)
SHE (Ours)
AUROC (1)
SVHN
74.99
85.83
51.81
91.05
54.35
90.52
5.87
98. 74
97.20
LSUN-R
38.92
94.75
14.98
97.49
6.67
98.42
ISUN
35.82
95.24
11.99
97.26
11.78
97. 80
4.16
98.85
ResNet18
Places
DTD
54.93
39.16
89.67
94.39
54.58
21.06
96.57
86.95
48.
17.36
97.14
91.42
32,02
6.31
89.11
98.70
Tiny Imagenei
SUN
38.34
94.65
20,80
96.80
15.08
97.44
3.58
99.24
INaturalist
68.40
83.80
65.16
89.59
52. 10
92.80
27.32
95.02
Average
48.65
92.38
31.45
94.36
28.55
95.29
11.74
97.15
LSUN-C
27.27
96.43
6.05
6.77
98.53
2.37
99.44
LSUUN.R
34.53
95.38
9.25
98.15
98.29
3.73
98.71
ISUN
33.15
95.56
8.69
98.21
8.06
98,
4.13
99.04
ResNer34
Places
DID
32.99
94.27
95.34
26.10
12.37
97.63
11.81
95.98
97.64
2.86
96.57
99.32
Tiny Imagerat
93.31
24.17
21.91
16.66
97.02
SUN
29. 86
95.77
10.03
98.02
9.85
97.99
1.47
99 63
iNaturalist
22.64
96.71
11.07
97.55
7,85
98.17
5.05
99 08
Average
33.42
95.34
13.62
97.34
12.31
97,59
6.02
98.
SVHN
30,80
95,96
12.58
97.70
LSUN-C
49.58
92.85
23.91
96.03
25
95.46
32.33
93.94
LSUN-R
42.86
94,04
15.91
97.34
14.91
97.42
9.19
98.10
ISUN
44.43
94,01
16.76
97.23
16.77
97.26
8.32
98.30
WRN40-2
Places
DTD
42.8K
94.01
16.98
97.06
20
96.44
5.27
98
Tiny Imugennet
55.44
91.37
38.77
93.87
38.22
94.34
24.20
95.56
SUN
41.04
94.49
15.31
97.41
18.80
96.76
3,51
99.18
iNatunliist
41.65
94
36.85
95.30
33.19
95.52
5.05
98 84
Average
45.77
93.56
24.81
96.04
25.57
95.89
12.63
97.40
Table 2: OOD detection performance comparison of HE and SHE. All values are averaged over the nine OOD datasets described in Section 4.1
The detailed Table is displayed in the Appendix [
CIFARIO
CIFAR100
Backbone
HE
SHE
HE
SHB
Network
FPR95 (4)
AUROC (f)
FPR95 (L)
AUROC (f)
FPR95 (4)
AUROC (?)
FPR95 (L)
AUROC (f)
ResNet18
11.77
97.13
97.15
48.73
87.16
48.86
87.14
ResNer34
6.03
98.69
6.02
98.68
63.23
81.42
63.28
8L48
WRN40-2
12.69
97.35
12.63
97.40
37.22
91.66
36.16
92.17
distribution (colored red), the less data in blue indicates the better capability in distinguish OOD data and ID data. It can be seen that SHE can obtain the minimum blue areas under the red peak among three methods.
4.3.2 SHE VERSUS HE
Notice that SHE is derived from HE that is inspired by Modern Hopfield Network, make com- parison between them. We can discover from Table that SHE and HE are competitive, indicating that the patterns of samples from the same class similar and contain redundancy for OOD detec- tion. As mentioned c HE detects OOD samples via the energy function as shown in Eq.
We evaluate the feasibility of HE and show the results in Table 2 which demonstrates that the energy function (Eq. 4 is able for OOD detection. For SHE, memory cost is reduced to the number of classes instead of the number of samples and the energy-based measure frequently used inner products without any hyperparameter, which is clegant and efficient. We also provide detailed comparison results on nine OOD detection Table [ proving that in most cases SHE can achieve similar performance to HE.
be simplified into
4.3.3 PERTURBATIONS ON SHE
Adding perturbations to input samples for OOD detection is proposed by (Liang et al. 2017). which can be formulated as follows:
/' - x sign (Var log S(x"))
(8)
where x' denotes the testing sample to be detected while E' the perturbed one. S(z") is the maximum Softmax probability of network outputs, and is the perturbation magnitude. Such
8

---

Published as a conference paper at ICLR 2023
perturbation- methods have been proved to be more beneficial for ID data than OOD data, and are adopted in extensive studies (Lee et al. 2018: Hsu et al. 2020 DeVries & Taylor 2018 To verify whether perturbation applies to SHE, also carry out the experiments and organize the results in Table 3 It demonstrates that introducing perturhations can be heneficial to SHE as the performance of SHE could be further improved with perturbations.
Note that, the perturbation requires an additional boowward-backwand procedure to retrieve the gra- dient, leading an increase in computational complexity and time. As comparison, we record the time overhead before and after adding perturbation to SHE: when we choose ResNet34 as the backbone with CIFARIO as ID data and Tinylmagenet as OOD data, the consuming time raises from 35.61s to 105.88 that is around 3 times. Therefore, a balance between efficiency and compu- tational overhead needs to be considered when adopting perturbation. Nevertheless, SHE always efficient and effective, which can be combined with perturbations to achieve even higher accuracy.
Table 3: OOD detection performance comparison of SHE and SHEP (i.c.. SHE + Perturbation). All values are averaged over the nine OOD datasets described in Section The detailed Table I0 is displayed in the Appendix.
CIFARIO
CIFARI00
Backhone Network
SHE
SHEP
SHE
SHEP
PPR95 (L)
AUROC (f)
FPR95 ())
AUROC (f)
FPR95 (1)
AUROC (t)
FPR95 (L)
AUROC (1)
ResNer18
11.74
97.15
6.52
98.46
48.86
87.14
43.62
88,40
ResNer34
6.02
98.68
4.26
99.09
63.28
81.48
57.54
84.50
WRN40-2
12.63
97.40
13.69
96.97
36,16
92.17
41422
90.26
Table 4: OOD detection performance comparison deriving pattern from penultimate and logits layer. All values are averaged over the nine OOD datasets described in Section 4.1 displayed in the Appendix.
The detailed Table
CIFARIO
CIFARI00
Backbone Network
Logits
Penultimate
Logits
Penultimate
(P)
AUROC (t)
FPR95 (L)
AUROC (1)
FPR95 (J)
AUROC (1)
FPR95 (L)
AUROC (1)
ResNer18
31.70
94.32
11.74
97.15
76.94
77.77
48.86
87.14
ResNer34
13,92
97.33
6.02
98.68
80.75
77.88
63.28
8L.48
WRN40-2
24.28
96.02
12.63
97.40
55.99
88.12
36.16
92.17
4.3.4
PENULTIMATE LAYER VS. LOGITS LAYER
Note that all the stored patterns or test patterns in approach are obtained from the penultimate layer output of neural networks. But in most methods, logits layer (i.e., the last layer) output is adopted for the confidence computation (Hendrycks & Gimpel 2016 Liang et al., 2017 Liu al.) 2020 [unn et al.] (021). To verify the significance of layer selection, we also provide an experi- mental evaluation on SHE that chooses patterns from the penultimate layer outputs final logits, respectively, for comparison. The average results on nine datasets are presented in Table 4 We can conclude that our approach gets much better performance when using the penultimate layer output, instead of final logits, as patterns to apply SHE.
5 CONCLUSION
In this paper, we propose a novel approach named HE for OOD detection based on a new "store- then-compare" paradigm. The key idea is store patterns of ID data and then leverage the energy function defined in the Modern Hopfield Network (Ramsauer et al. 2020) for measuring the simi- larity between the new test patterns and the stored ID patterns. To reduce storage and computational overhead, we simplify the energy function with the theoretical analysis by appropriate approxi- mations and obtain the simplified approach named SHE. In addition to the great efficiency and effectiveness, SHE does not have any hyperparameterss to tune, which is more convenient than most OOD detection methods with cumbersome hyperparameters tuning. Also, different from most OOD detection methods focusing final output logits, we find that the penultimate layer output, rather than the final output logits, is more suitable to be used as patterns in our approach for OOD detec- tion. The conducted evaluations demonstrate the superiority of proposed simple yet effective approach on nine widely-used OOD datasets.

---

Published as a conference paper at ICLR 2023
REFERENCES
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Con- crete problems in ai safety. arXiv preprint arXiv: 06065, 2016.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255, leee, 2009.
Taylor Denouden, Rick Salay, Krzysztof Czamecki, Vahdat Abdelzad, Buu Phan, and Sachin Vernekar. Improving reconstruction autoencoder out-of-distribution detection with mahalanobis distance. arXiv preprint arXiv:1812.02766 2018.
Terrance DeVries and Graham W Taylor. Learning confidence for outto--distribution detection in neural networks. arXiv preprint arXiv: 04865, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02133 2016.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint 1812: 04606, 2018.
John Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 2958):2554-2533 1982.
John J Hopfield. Neurons with graded response have collective computational properties like those of two-state neurons. Proceedings of the national academy of sciences, 81(10))0088-302) 1984.
Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting out- of-distribution image without learning from out-of-distribution data. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10951-10960, 2020.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4700-4708, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
Dmitry Krotov and John J Hopfield. Dense associative memory for pattern recognition. Advances in neural information processing systems, 29, 2016.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint arXiv:1706.0200 2017.
Weitang Liu, Xiaoyun Wang. John Owens, and Yixuan Li. Energy-based out-of-distribution detec- tion. Advances in Neural Information Processing Systems, 33:21464-2144 214755, 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011. 1.
10

---

Published as a conference paper at ICLR 2023
Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlović, Geir Kjetil Sandve, et al. Hopfield networks is all you need. arXiv preprint arXiv:2008.02220 022111000.....,,,..00 2020.
Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7263-7271, 2017.
Vikash Sehwag, Mung Chiang, and Prateek Mittal. Ssd: A unified framework for self-supervise outlier detection, arXiv preprint arXiv:2103./205/ 2021.
Joan Serrà, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José F Núñez, and Jordi Luque. In- put complexity and outoof-distribution detection with (kelihood-basse generative models. arXiv preprint arXiv:1909. 1480, 2019.
Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribttion detection with rectified activa- tions. Advances in Neural Information Processing Systems, 34:1 10:1444-57,, 2021.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribttion detection with deep nearest neighbors. arXiv preprint arXiv:2204.06500 2022.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine
learning research, 9(11), 2008.
Grant Van Hor, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1169-8778, 2018.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer sociery conference on computer vision and pattern recognition, pp. 3492. IEEE, 2010.
Pingmei Xu, Krista A Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev R Kulkarni, and Jianxiong Xiao. Turkergaze: Crowdsourcing saliency with webcam based eye tracking. arXiv preprint 10XY::5504.0655.0 2015.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 150.033506.03355 2015.
Qing Yu and Kiyoharu Aizawa. Unsupervised dusttf-dist detection by maximum classifier discrepancy. In Proceedings of the IEEE/CVF international conference on computer vision, pp- 9518-9526, 2019.
Sergey Zagoruyko and Nikos Komodakis. 1rXiv:1605.07146, 2016.
Wide residual networks.
arXiv preprint
Hongjie Zhang, Ang Li, Jie Guo, and Yanwen Guo. Hybrid models for open recognition. In European Conference on Computer Vision, pp. 102-117. Springer, 2020.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 152-11444 2017.
11

---

Published as conference paper at ICLR 2023
A
EXPERIMENTAL RESULTS.
Table 5: OOD detection performance of SHE using CIFAR100 as ID dataset.
Methods
Backhone Network
OOD
Energy
SHE (Ours)
AUROC
AUROC
AUROC (1)
SVHN
61.61
87.54
46.01
92.74
53.07
89.90
12.56
97.64
LSUN-C
63.09
84.66
43.98
91.68
19.95
96.67
27.93
95.15
LSUN-R
89.15
62.05
77.84
78.65
60.06
88.66
67.36
86.59
ISUN
88.25
66.06
76.66
80.16
58.08
89.78
53.33
90.10
ResNer18
Places
87.14
74,96
85.84
74.81
75.66
84,94
51.14
88.03
DTD
91.47
66.50
93.16
60.86
86.07
78.30
65.30
68. 16.16
Tiny Imagenet
72.28
89.42
72.85
88.63
73.08
79.69
83.56
44.48
90 00.62
INaturalist
92.95
67.60
94.31
68.45
85.15
81.73
59,42
78.85
Average
83.01
72.62
75.41
77.97
64.02
87.02
48.86
87.14
SVHN
58.43
88.65
42.75
93.08
26.46
94.443
14.38
97.32
LSUN-C
78.25
81,79
69.77
87.41
35,10
94.98
33.04
94.29
LSUN-R
92.01
74.81
72.59
83.30
80.26
76.60
90.54
84.92
75.92
71.05
84.45
ResNec34
DTD
87.83
73.33
88.56
69.91
60.17
88.35
68.59
72.49
Tiny Imagenet
88.94
67.08
84.08
75.83
71.73
84.1
79.07
76.59
SUN
91.84
72.72
92.62
69.75
65.09
88.45
77.94
77
NNaturalist
88.41
79.13
90.03
79.67
54.06
91.37
66.22
80.29
Average
84.93
74.11
80.76
77.78
57.33
88.72
63.28
81.48
52.60
19.15
96.54
59.67
72.26
71.56
75.04
iSUN
86.06
67.06
69.92
84.04
66.83
85.68
63.44
86.05
Places
68.05
82.24
47.22
90.26
45.28
90.33
24.26
95.13
DTD
65.29
80.26
47.35
87.92
36.56
89.82
25.17
93.27
Tiny Imagenet
80.42
70.35
63.44
85.91
$6.69
88.64
62.71
86.66
SUN
71.00
80.81
49.33
90.13
47.33
90.26
20.76
95.95
INaturalist
47,20
90.85
17.76
96.76
18.82
96.38
9.25
98.11
92.17
The ReAct in this table refers 'Energy ReAct' described in Sun et al. (2021). auxillary method combined with other SOTA methods, we can also combine the ReAct with SHE (better than Energy), which will also outperform SHE itself, the results are shown in Table. 6
Table 6: OOD detection performance comparison of "ReAct + SHE" and "ReAct + Energy".
FPR95 (L)
OOD Datasets
Method
SVHN
LSUN-C
Place365
DTD
SUN
iNaturalist
Avg
ReAct+Energy
14.38
33.04
77.51
68.59
77.94
66.22
56.28
ReAct+SHE
23.94
36.79
62,38
52.53
56.50
57.28
48.24
12

---

Published as a conference paper at ICLR 2023
SWHN
LSON 4
LSUN_E
LSUNLR
1200
BUN
1200
1000
DID
1000
Ti
500
SUR
200
000
200
200
30
40
100
60
150
1200
1200
Tryfmagenet
1000
500
300
400-
400-
200
200
100
Figure& 4: The distribution of output num for each class from nine OOD datasets. The backbone network is ResNet34 and the ID dataset is CIFAR100.
75
25
IQ_D
IDU
03
10,2
4-
D2.2
-25
-25
10.6
-40
10_7
10.0
-75
10_9
10,5
0CD Dath
-03
0C0
-20
-40
-20
20
40
60
10
-00
-40
20
60
20
Epoch 10
Epoch 40
10,0
00,0
0-
5-
D3
-20
10.5
12.4
-22
02.
12.4
10.6
10.5
10_7
-42
10
10.0
10.5
-60
-00
-03
Epoch '70
Epoch 100
Figure 5: Visualization of the training process by t-SNE (Van der Maaten & Hinton 2008). From the The backbone network is ResNet18, and ID data and OOD data are CIFARTO Krizhevsky et al.) 2009) and SUN (Xiao et al. 2010) respectively.
13

---

Published as a conference paper at ICLR 2023
Table 7: OOD detection performance of SHE using ImageNet-Ik as ID dataset and ResNet50 as
backbone network.
Methods
000
MSP
Energy
Mahtalaohii
SHE
Dusasets
TPR95 (.) AUROC(() (1)
FPRO5 ())
AUROC (?)
(1)
AUROC (1)
FPR95 (J)
AUROC (1)
TPR95 (L)
(UROC((1)
(J)
AUROC (*)
Places
$4.99
87.74
53.72
89
47 76
89 86
97.00
52.65
59.00
86.47
45.35
90.15
DTD
70.83
80.86
99.26
55 89
60.15
$4.59
98.50
42.41
68.82
80.72
45.09
87.93
76.28
54.19
50.23
Average
65.95
81.99
38.41
96.17
56.48
85.41
87.43
53.47
53.97
85.01
44.71
$8.24
Table 8: OOD detection performance of Reergy+ReAct and SHE+ReAct using ImageNet as ID From the results, ReAct+Energy better which is limitation our method and will
dataset.
explore it in the future.
FPR95(%%
Place365
DTD
SUN
iNaturalist
Avg
Reergy+ReAct
20.38
24.20
33.85
47.30
31.43
SHE+ReAct 49.91
32.06
41.70
35.99
39.92
Table 9: OOD detection performance comparison of HE and SHE. All values percentages. Bold numbers with gray cell are superior results. The hyperparameter used for HE 0.01 for ResNet18,
ResNet34, 0.2 for WRN40-2.
CIFARIO
CIFARIOB
Backhone
OOD
HE
HE
SHE
Network
Datasets
FPR95
AUROC (f)
FPR95
AUROC:
FPR95
AUROC (1)
FPR95 (J)
AUROC
SVHN
6.31
98.70
5.87
98.74
12.97
97.63
12.56
97.64
LSUN-C
7.95
98.45
7.94
27.51
95.29
LSUN-R
6.93
98.42
6.67
98.42
66.81
86.78
67.36
86.59
ISUN
4.13
98.86 98.86
4.16
98.85
53.24
90.15
53.33
90.10
Places
6,44
98 66
6.31
98.70
50.95
88.17
51.14
88.03
DTD
32,18
88.95
32.02
89.11
65.4
67.87
65.30
68.16
Tiny Imagenet
11.62
97. 86
11.81
97.86
57.60
89.31
58.25
89.13
SUN
3.51
99.23
3.58
99.24
44.46
90.65
44.48
90.62
INaturalist
26.88
95 06
27.32
95.02
59.58
78.63
59.42
78.85
Average
11.77
97.13
11.74
97.15
48.73
87.16
87.14
SVHN
3.34
99.30
3.16
99.34
14.19
97.28
14.38
97.32
LSUN
2.17
99.46
2.37
99.44
33.37
94.21
33.04
94.29
LSUN
5.59
98.76
5.73
98.71
79.84
76.95
80.26
76.6
ISUN
4.14
99.05
4.13
99.04
72.52
80.28
72.51
80.2
99.35
77.40
ResNet34
Tiny Imagenet
76.43
99.64
NNaturalist
5.17
99.07
5.05
99.08
66.08
80.31
66.22
80.29
Average
6.03
98.69
6.02
98.68
63.23
81.42
63.28
81.48
SVHN
12.84
97. 97.65
12.58
97.7
23.49
95.29
19.15
96.54
LSUN-C
32.96
93,66
32.33
93.94
25.01
95.31
25.94
95.47
LSUN R
8.83
98.18
9.19
98.1
73.38
82.67
75.00
82.34
ISUN
7,83
98.40
8.32
98.3
62.75
86.02
63.44
86.05
Places
5.12
98.87
5.27
98.84
25. .6.46
94.85
24.26
95.13
DTD
14.44
95.67
13.26
96.1
29.94
91.28
25.17
93.27
Tiny Imagenet
23.83
95.66
24.2
95.56
61.85
86.32
62.71
86.66
SUN
3.58
99.18
3.51
99.18
23.36
95.19
20.76
95.95
iNaturalist
4.79
98.80
5.05
98.84
9.70
98.03
9.02
98.
Average
12.69
97.35
12.63
97.40
37.22
91.66
36.16
92.17
14

---

Published as a conference paper at ICLR 2023
Table 10: OOD detection performance comparison of SHE and SHEP (SHE Perturbation). All values are percentages. Bold numbers with gray cell are superior results.
CIFARIO
CIFARIO0
Backhone Network
OOD
Dalasets
SHE
SHEP
SHI
SHEP
FPR95 ())
AUROC (f)
FPR95 ())
AUROC (1)
FPR95 (1)
AUROC (1)
FPR95 (1)
AUROC (1)
SVHN
5.87
98.74
11.11
97.92
12.56
97.64
34.61
93.62
LSUN-C
98.45
ISUN
4.16
98.85
99.30
53.33
90.10
53.54
87.61
Places
6.31
98.70
1.89
99.49
51.14
88.03
36.83
90.71
R8
DTD
32.02
89.11
17.04
95.14
65.
68. 16
52.17
76.23
Tiny Imagenel
11.81
97.86
5.70
98.76
58.25
89.13
55.36
88.07
INaturalist
27.32
95.02
7.76
59.42
78.85
35.96
Average
11.74
97.15
6.52
98.46
48.86
87.14
43.62
88.40
SVHN
3.16
99.34
8.04
98.49
14.18
97.32
42.71
91.2
LSUN-C
2.37
99.44
3.40
99.31
33.04
94.29
33.57
93.62
4.13
99.04
72.51
80.2
70.73
Places
2.86
99.32
1.52
99.62
77.51
77.63
62.41
83.34
R4sNet34
DTD
12.76
96.57
7.05
98.21
68.59
72.49
54.34
81.4
Tiny Imagenct
16.66
97.03
9.76
98.17
79.07
76.59
74.25
79.64
77.94
iNatunlist
Average
6.02
98.68
4.26
99.09
63.28
81.48
57.54
84.50
SVIIN
12.58
97.7
20.37
95.99
19.15
96.54
37.69
92.09
LSUN
32.33
93.94
43.62
89.08
25.94
95.47
34.04
93.57
LSUN-R
98.1
8.84
75.04
82.34
Places
5.27
98.
4.96
98.94
24.26
95.13
28.54
94.47
WRNN0-2
DTD
13.26
96.1
10.57
97.23
25.17
93.27
25.31
93.94
Tiny Imagenct
24.2
95.56
19.91
96.34
62.71
86.66
66.18
83.84
SUN
3.51
99.18
3.71
20.76
95.95
24.39
95.27
iNaturalist
Average
97.40
13.69
36.16
92.17
41.42
15

---

Published as a conference paper at ICLR 2023
Table 11: OOD detection performance comparison deriving pattern from penultimate and logits layer. All values are percentages. Bold numbers with gray cell are superior results.
CIFAR
Backhone Network
Backbone Network
Logits
Penultimate
Logiss
Penultimale
FPR95 (1)
AUROC (1)
FPR9S (.)
AUROC (1)
FPR95 (1)
AUROC (1)
FPR95 (L)
AUROC (1)
SVHN
54.81
90.61
5.87
98.74
49.30
92.31
12.56
97.64
LSUN-C
L.SUN-R
77.38
ISUN
12.66
97.70
4.16
98.85
80.23
79.01
53.33
90.10
Places
21.56
96.52
6.31
98.70
86.03
75.32
51.14
88.03
ResNet18
DTD
53.37
87.21
32,02
89.11
92.35
61,60
65.30
68.16
Imagenet
27.43
95.85
11.81
97.86
76.37
80,50
58.25
89.13
SUN
96.81
3,58
88
73.47
44.48
INaturalist
64.59
27.32
95.02
88.95
69.19
59,42
Average
31.70
94.32
11.74
97.15
76.94
77.77
48.86
87.14
SVHN
15.17
97.4
3.16
99.34
44.22
92.78
14.38
97.32
LSUN-C
6.45
98.56
2.37
69
87.14
33.04
LSUN-R
ISUN
9.79
98.09
4.13
99.04
75.35
72.51
Places
12.44
97.62
2.86
99.32
87.77
74.25
77.51
77.63
R4sNet34
DTD
25.46
94.82
12.76
96.57
87.69
70.78
68.59
72.49
Tiny Imagenet
24.69
95.67
16.66
97.03
84.4
75,36
79.07
76.59
SUN
10.65
97.97
1.47
99.63
91.56
70.84
77.94
77.88
INaturalist
Average
77.88
SVIN
30.15
95.17
12.58
97.7
57.21
91.55
19.15
96.54
LSUN-C
23.74
96.05
32.33
93.94
42.27
92.69
25.94
95.47
LSUN-R
9.19
81.12
75.04
82.34
iSUN
Places
17.43
96.99
5.27
98.84
52.59
89
24.26
95.13
WRN40-2
DYTD
26.25
95.11
13.26
96.1
$1.65
86
25.17
93.27
Tiny Imagenet
38.79
93.75
24.2
95.56
68.91
84.33
62.77
86.66
SUN
16.29
97.23
3.51
99.18
55.55
88.68
20.76
95.95
INatoralist
30.46
95.69
5.05
98.84
21.81
96.3
9.02
98.11
Average
16

---

Published as a conference paper at ICLR 2023
B THEORETICAL ANALYSIS.
Test Kyttem Clsso Iicm 5991
Figure 6: Heatmap of the Stored/ID/OOI pattern derived by ResNet18, the ID dataset, and OOD dataset is CIFAR1O 10 and SUN (Xiao et al. 2010) respectively. For visualization, the pattern is re- shaped to 32*16 from the dimension 512. From the figure, we can see the expectation of the ID/Stored pattern is larger than the expectation of the OOD pattern, which supports the theory men- tioned below.
Let = [V1. U2m) and == [=1, E.. be the output logits and the penultimate layer output, respectively. Here, m the dimension logits output (i.c., the number of classes) and n is the dimension of the penultimate layer output. We superscript the vector id and ood (e.g., ydd denotes logits output derived from an ID sample) to indicate the input type of the vector. For every good in goo we assume that they are independent random variables following the same Gaussian distribu- tion, i.e., 25 N (0, )²) Let (01, Umm be the weight matrix of the last layer (linear layer)
with each vy denotes the categorical vector for class i. Thus, have y [01, 122--, Um!] )
For a test sample from OOD data, we assume that it classified as the category k; and we have the following formula. Among them, M represents a distribution of the maximum value of Gaussian random variables.
yk aod max (v1 T 2odd U2 Tond Um 20d)
yh Mood max (0.00"/loz)l2)) NN (0.02)10z11e)) ....N (0, (0.0211mmll2")
(9)
y ood ~ N (0, (110211u))1?")
For id we assume that it follows another distribution I whose expected value is a positive number oood larger than the expected value of york When calculated by Eq. 6 the expectation calculated from output logits and penultimate layer outputs are shown as follows (* denotes inner product):
1yy * + and =
ral
T ood
Σ
m
a 05 T ooo
Uk
j=1jzkk
(10)
E [id]] E [Mood]]
Up [zp id E
[2g ooo
F.
[rid]
E [Mood]]
>
0
E id
oo0
E
E [Wood]
0 < E
[ydi y oodd
(11)
Therefore, compared with using the penultimate layer as the pattern to calculate the energy function, logits output will assign OOD samples with higher scores. Therefore, will be more challenging to distinguish ID samples from OOD samples.
17

---

Published as a conference paper at ICLR 2023
C ABLATION EXPERIMENT.
The ablation experiment consists of two parts. First, evaluate different metrics instead of the inner product which is used in our approach to measuring the similarity the stored pattern and test pattern. To be specific, shown in Table. T2] Second, we use the output from other layers (e.g.. layer3 in ResNet) instead of the penultimate layer (layer4 in ResNet) to act the representation of our approach. The results are shown in Table.
"Euclidean Distance" and "Cosine Similarity", the results
Table 12: OOD detection performance with different metrics, the ID dataset is CIFARIO. From the results, the inner product used in our approach performs better than another two metrics.
Metric
Backbone Network
OOD
Euclidean Distance
Cosine Similarity
Inner Product (Ours)
Datasets
FPR95 (1)
AUROC (t)
FPR95 (J)
AUROC (t)
FPR95 (L)
AUROC (1)
SVHN
38.59
93.76
39.59
93.70
5.87
98.74
LSUN-C
22.57
96.07
13.22
97.65
7.94
98.45
LSUN-R
20.50
96.35
17.95
96.93
6.67
98.42
iSUN
14.72
97.14
14.90
97.35
4.16
98.85
Places
12.18
97.53
17.73
96.84
6,31
98.70
ResNet18
DTD
13.59
97.22
13,42
97.57
32.02
89.11
Tiny Imagenet
27.65
95.01
25.15
95.53
11.81
97.86
SUN
8.98
98.12
12.89
97.68
3.58
99.24
NNturallit
21.69
95.84
22.04
96.19
27.32
95.02
Average
20.05
96.34
19.65
96.60
11.74
97.15
SVHN
15.78
97.03
28.70
95.52
3.16
99.34
LSUN-C
11.78
97.71
13.94
97.62
2.37
99.44
LSUN-R
16.31
96.86
22.79
96.00
5.73
98.71
iSUN
14.39
97.15
23.16
95.95
4.13
99.04
Places
19.09
96.26
20.69
96.27
2,86
99.32
ResNet34
DTD
10,28
98.01
12.07
98.00
12.76
96.57
Tiny Imagenet
30.16
94.04
37.07
92.76
16.66
97.03
SUN
14.92
96.92
16.80
96.81
1.47
99.63
iNatrrllitt
8.11
98.37
9.36
98.40
5.05
99.08
Average
15.65
96.93
20.51
96.37
6.02
98.68
SVHN
42.96
92.27
39.79
93.21
12.58
97.70
LSUN-C
42.92
91.67
38.33
93.11
32.33
93.94
LSUN-R
31.19
94.57
29.55
94.77
9.19
98.10
iSUN
29.35
94.96
28.51
95.01
8.32
98.30
Places
35.18
93.55
38.38
93.15
5.27
98.84
WRN40-2
DTD
14.41
97.25
15.74
97.27
13.26
96,10
Tiny Imagenet
48.18
90.67
43.30
91.78
24.20
95.56
SUN
30.93
94.43
35.22
93.87
3.51
99.18
iNaturalist
23.50
95.77
20.35
96.53
5,05
98.84
Average
33.18
93.90
32.13
94.30
12.63
97.40
Table 13: OOD detection performance comparison of shallow layer (the layer before penultimate layer) and penultimate layer the representation. All values are averaged over the nine OOD datasets.
CIFARIO
CIFAR100
Backbone
shallow layer
penultimate layer
shallow layer
penultimate layer
Network
FPR95 (L)
AUROC (1)
FPR95 (L)
AUROC (t)
FPR95 ({)
AUROC (?)
FPR95 (L)
AUROC (1)
ResNet18
86.83
34.25
11.74
97.15
86.75
43.43
48.86
87.14
ResNer34
83.87
51.30
73.77
63.28
WRN40-2
91.84
47.13
12.63
97.40
89.97
48.65
36.16
92.17
18

---

Published as a conference paper at ICLR 2023
D EXPERIMENTAL DETAILS.
Software and Hardware. The experiments are performed on one linux server (Operation system: Ubuntu Linux 18.04.1). For GPU resource, four NVIDIA GeForce RTX 3090 used for ResNet and WRN. The python environment is 3.7 and the libraries we use to implement our experiments is PyTorch 1.12.1.
Number of Evaluation RunsReported Following (Liu et al. 2020: Hendrycks et al., 2018), per- formance for each OOD dataset averaged over 10 random batches of samples.
Training Details. During the training procedure, some data augmentation (e.g.. rotation, flipping, resizing) and training techniques (e.g.. learning rate decay) are adopted improve the model's accuracy. In more detail, we set the batch size as 128, the image size as 112 for ResNet, and 64 for WRN, respectively. And we use SGD as the optimizer with 0.1 the initial learning rate and apply the learning rate decay at epochs 50 and 75, respectively.
19


Reviews:
[{"review_id": "nDkKy_IlEsV", "reviewer": "Reviewer_ANA9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a store-then-compare paradigm for out-of-distribution detection. The proposed method is motivated by Modern Hopfield Energy, and a simplified version of the proposed method is also derived.", "summary_of_the_review": "Overall, while I have concerns about whether the proposed method is just an alternative to metric learning, I think even if that is the case, the proposed method is still interesting in the field of OOD detection. Thus, I tend to accept this paper.", "strength_and_weaknesses": "Strength:\n1. The paper is well-written and easy to follow.\n2. The experiment result is quite significant and impressive, demonstrating the strong performance of the proposed method.\n3. Further analysis of the penultimate layer versus the logits layer is also interesting.\n\nWeaknesses:\n1. The authors mention that \"the energy function of Hopfield Network is well suited as a desirable measure of the discrepancy between the OOD sample and the stored patterns\". Could the authors elaborate more on the reason behind this statement? Specifically, there exists a lot of metrics that can measure the discrepancy between features, such as measuring the distance of mean, the distance of covariance, the earth mover's distance, and so on. Thus, I am curious on is there a specific reason that the authors choose to use the energy function of the Hopfield Network to measure the discrepancy.\n2. To further elaborate on my point 1, I think the idea proposed in this paper is quite similar to the metric learning methods in other areas (e.g., few-shot learning). Could the authors also discuss the relationship between the proposed method and metric learning?\n3. Another small confusion I have is about the title. Does the \"efficient\" here imply that SHE is more efficient than HE or SHE is more efficient than the other methods as well? If the formal case, I suggest the authors delete the word \"efficient\" in the title as it is slightly misleading, if the latter case, I hope that some ablation study w.r.t. efficiency can be given to demonstrate this.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dlAWVfYqnS", "reviewer": "Reviewer_sLwk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel out-of-distribution (OOD) detection algorithm with “store-and-compare” fashion: store the average pattern in the penultimate layer for each class, and during test, compute the inner product for the test blob with the stored pattern for the predicted class, and this inner product will be thresholded to determine OOD or not. The authors try to motivate the algorithm from Hopfield energy with a series of approximations, and also argue that the penultimate layer is more effective than the logit layer as used in most existing work. Experiments are conducted on vision datasets (two training sets CIFAR10 and CIFAR100, and 9 OOD sets), with comparison with three baselines, and demonstrated superior results most of the time.  ", "summary_of_the_review": "This paper proposed a novel OOD algorithm with an intuitive idea that simply compares the test examples with stored average patterns for the predicted class on the 2nd-to-last layer. It attempts to motivate it from Hopfield energy, which I don’t think is necessary; the intuitive argument that the penultimate layer is better than the logit layer is convincing and well supported by experiments, but theoretical analysis seems futile. Experiments are relatively comprehensive in terms of datasets, baselines and average performance, but not results with ImageNet dataset as state-of-the-art method ReAct did, and lack insights on when the algorithm should perform better than baseline. The writing is clear and easy to follow, w/o little typos. Overall I can see this becoming a good paper, but not in current form. \n\n\n", "strength_and_weaknesses": "strengths\n* The algorithm is intuitive and simple with strong empirical results, writing is easy to follow. \n\nweakness\n* Lack of analysis and understanding on when the algorithm should be better than baseline, there are huge tables on many datasets and backbone networks, but not much insights on more granular analysis.  \n* Lack empirical analysis and visualization on the stored patterns\n* no results with ImageNet dataset with 1k+ classes, which seems a critical aspect of the algorithm: number of classes, especially given that the performance seems not as good on CIFAR100 than on CIFAR10. \n\n\n\nMore details below: \n\nI don’t feel the need to motivate the algorithm with Hopfield network. At the end of the day, it seems all it does is to store the average outputs from the penultimate layer for each class, and during test time, we compute the inner product between the test blob and the stored “pattern” for the predicted class. I don’t feel motivating it with Hopfield energy is necessary, especially as there is a series of approximations not fully justified with empirical data demonstrated.  \n\nWhat I would prefer is to just start with the algorithm and briefly mention it could be motivated from Hopfield energy and put section 3.2-3.4 in the appendix, and leave more space for empirical analysis and visualization of the patterns remembered? For example, could we calculate the distances between classes and visualize the patterns in 2-D (or 3-D), and see where the ID and OOD examples land? even more interestingly, how are they evolving during training?  \n\nIt is misleading to say “Similar performance gain can be observed when CIFAR100 is used as the ID training data” – if we look at Table 5 in Appendix A, where with ResNet34, ReAct is around 6% better on average. This raises several questions we should study, e.g., why on certain training sets the OOD performance is better,  why on certain backbone networks the OOD performance is worse than baseline, etc.. I believe these are important analyses to be done. Also interesting to see results on ImageNet with even more classes – Will the algorithm still perform better? or is it only good in the small-num-classes regime?   \n\nThe proposed algorithm is not w/o any hyperparameters: we still need a threshold deciding what are OOD examples? OOD examples can be of any form. How do we determine such a threshold? is the threshold the same for each class or different for different classes? \n\nAbout why the penultimate layer is better than the logit layer, I find the intuitive argument more convincing than the theoretical analysis. First, there are two strong assumptions w/o any justification: (1) z follows zero-mean Gaussian distribution, (2) $y_k^{id}$ follows another distribution $\\cal{I}$ whose expected value is a positive number larger than the expected value of $y_{q\\not=k}^{ood}$. At least, show empirical histograms to justify such assumptions? Second, there seems not much point to compare the inner products of two layers, what we care about is the relative values of the inner product of ID vs OOD examples in the same layer.   \n\ntypos\n* “Intuitive Explanation.Given”, add space \n* section 4.3.4, “computation(Hendr”, add space\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "F5kTPPwx4-", "reviewer": "Reviewer_boCN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to use the energy function of modern Hopfield networks to detect out-of-distribution datapoints.", "summary_of_the_review": "Nice paper, I would like to see the results of the ablation studies that I requested, and revisions regarding the more accurate description of various models of the Hopfield family. ", "strength_and_weaknesses": "The premise of Hopfield networks is to place plausible data points at the local minima of the energy function. The authors propose to use this property for detecting datapoint that are located far away from those local minima, which correspond to in-distirbution training examples. Overall, this is a very natural idea, and the empirical evaluation supports its usefulness. I have a few suggestions that I describe below that may improve the proposed method and the quality of the presentation.  \n\nThroughout the paper the authors focus on one specific model from the modern HN family. This model uses the dot-product similarity function between the memories $S$ and the datapoints $\\xi$.  Although this is a plausible measure, it is unclear why the representations in the penultimate layer of the front-end networks (e.g. ResNet18) would respect this measure. I suspect that a much more natural measure of the similarity would be the Euclidean distance between the memories and the data points, see for example [(Millidge et al., 2022)](https://arxiv.org/abs/2202.04557) for a description of the HN with this similarity measure. A third possible choice would be a cosine similarity between the memories and the new test datapoints. I suspect that among these three networks the one with the Euclidean distance will demonstrate the best performance in terms of out-of-distribution sample detection, the dot-product based network (considered in this work) would be second, and the cos-similarity would perform the worst among these three. I would like to see the results of these ablation studies in the revised manuscript. \n\nThe description of the prior work on Hopfield networks is not entirely accurate. Binary HN were introduced in [(Hopfield PNAS 1982)](https://www.pnas.org/doi/10.1073/pnas.79.8.2554), continuous HN were introduced in [(Hopfield PNAS 1984)](https://www.pnas.org/doi/10.1073/pnas.81.10.3088). Modern HN (both continuous and binary) were introduced in (Krotov & Hopfield 2016). The contribution of (Ramsauer et al., 2020) was to generalize the theory of (Krotov & Hopfield 2016) to the case of softmax activation function, and point out the relationship with transformers. For example, the first sentence of the second paragraph of section 3.2 is misleading - HN have been used for continuous pattern retrieval since 1984, see [(Hopfield PNAS 1984)](https://www.pnas.org/doi/10.1073/pnas.81.10.3088).  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3OARy3JpfU", "reviewer": "Reviewer_bCQs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies out-of-distribution detection, aiming at making classification models excel at discerning ID and OOD data. It is an important problem for safety-critical applications, and has attracted increasing attention recently. In this paper, the authors state that previous OOD methods estimate the OOD confidence in the logit space with tedious hyperparameters, which are hard to be deployed in reality. To this end, the authors suggest a new store-then-compare paradigm. It is motivated by the modern Hopfield networks for the discrepancy score calculation, which is free from hyper-parameters and is easy in deployment. The authors conduct experiments on nine widely-used OOD datasets, and the authors claim their superiority over the state-of-the-arts.  ", "summary_of_the_review": "This paper adopts the Hopfield network in OOD detection, which may have some novelties to benefit the community. However, to improve the quality of the paper, it will be great if the authors could address my concerns in the Summary of the Paper. ", "strength_and_weaknesses": "> Strength:\n\nTo the best of my knowledge, this paper is the pioneer in using the modern Hopfield networks in OOD detection. The authors interpret the energy function as the measure of similarity, which can be used to discern ID and OOD data. Further, the authors suggest a simplified realization named SHE, which approximate the original energy function by the Taylor series, which eases the burden in memorizing all embedding features in energy calculation. \n\n> Weakness:\n\n- *The motivation of the proposed method is not well supported*. The authors claim the deficiencies of previous works in using logit outputs. However, the discussion is only valid for their proposed method in using the energy function. To fully demonstrate the deficiencies in using logit outputs, I think the authors could take other advanced works as examples, e.g., [1,2]. To me, these methods can also demonstrate promising results regarding various OOD setting. Further, I think there are also many other works that use the embedding features in OOD detection [3], which may challenge the novelty of the paper. \n\n[1] Yiyou Sun, et al. Out-of-distribution Detection with Deep Nearest Neighbors. ICML'22. \n\n[2] Vikash Sehway, et al. SSD: A Unified Framework for Self-supervised Outlier Detection. ICLR'21. \n\n[3] Kimin Lee, et al. A Simple Unified Framework for Detecting Out-of-distribution Samples and Adversarial Attacks. NeurIPS'18. \n\n- *The novelty of the proposed method is limited*. Both the energy functions [4] and the so-called store-then-compare paradigms [5] have been studied. Therefore, I think it can improve the quality of the paper in discussing the superiority of the proposed method over these representative baselines. \n\n[4] Weitang Liu, et al. Energy-based Out-of-distribution Detection. NeurIPS'20. \n\n[5] Yiyou Sun, et al. Out-of-distribution Detection with Deep Nearest Neighbors. ICML'22. \n\n- *The experiments are not enough*. I do not think the authors conduct extensive experiments that can fully support the effectiveness of the proposed method. Experiments with large semantic space [6] (e.g., ImageNet benchmark) and hard OOD scenario [5] (e.g., CIFAR-10 vs. CIFAR-100) are not considered. Further, comparison with advanced baseline methods (e.g., [1,2,5]) is also missing. \n\n[6] Rui Huang and Yixuan Li. MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Space. CVPR'21. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster