Image Background Serves as Good Proxy for Out-of-distribution Data
Authors: Sen Pei
Venue: ICLR 2024 poster (2024)
Keywords: Out-of-distribution detection
OOD supervision
robust image classification
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: An unified perspective to view the out-of-distribution detection, and an effective method to generate natural OOD supervision.

Abstract: Out-of-distribution (OOD) detection empowers the model trained on the closed image set to identify unknown data in the open world. Though many prior techniques have yielded considerable improvements in this research direction, two crucial obstacles still remain. Firstly, a unified perspective has yet to be presented to view the developed arts with individual designs, which is vital for providing insights into future work. Secondly, we expect sufficient natural OOD supervision to promote the generation of compact boundaries between the in-distribution (ID) and OOD data without collecting explicit OOD samples. To tackle these issues, we propose a general probabilistic framework to interpret many existing methods and an OOD-data-free model, namely $\textbf{S}$elf-supervised $\textbf{S}$ampling for $\textbf{O}$OD $\textbf{D}$etection (SSOD). SSOD efficiently exploits natural OOD signals from the ID data based on the local property of convolution. With these supervisions, it jointly optimizes the OOD detection and conventional ID classification in an end-to-end manner. Extensive experiments reveal that SSOD establishes competitive state-of-the-art performance on many large-scale benchmarks, outperforming the best previous method by a large margin, e.g., reporting $\textbf{-6.28}$% FPR95 and $\textbf{+0.77}$% AUROC on ImageNet, $\textbf{-19.01}$% FPR95 and $\textbf{+3.04}$% AUROC on CIFAR-10, and top-ranked performance on hard OOD datasets, i.e., ImageNet-O and OpenImage-O.

Published as a conference paper at ICLR 2024
IMAGE BACKGROUND SERVES AS GOOD PROXY FOR OUT-OF-DISTRIBUTION DATA
Sen Pei
ByteDance Inc.
pe senebytedance.cco
ABSTRACT
(OOD) detection empowers the model trained on the closed image set to identify unknown data in the open world. Though many prior tech- niques have yielded considerable improvements in this research direction, two crucial obstacles still remain. Firstly, a unified perspective has yet to be presented to view the developed arts with individual designs, which vital for providing insights into future work. Secondly, we expect sufficient natural OOD supervision to promote the generation of compact boundaries between the in-distribution (ID) and OOD data without collecting explicit OOD samples. To tackle these issues, we propose a general probabilistic framework to interpret many existing methods and OOD-data-free model, namely Self-supervised Sampling for OOD Detection (SSOD). SSOD efficiently exploits natural OOD signals from the ID data based the local property of convolution. With these supervisions, it jointly optimizes the OOD detection and conventional ID classification in an end-to-end manner. Extensive experiments reveal that SSOD establishes competitive state-of-the -f-ttr-art performance on many large-scale benchmarks, outperforming the best previous method by a large margin, e.g., reporting -6.28% FPR95 and +0.77% AUROC on ImageNet, 19.001 % FPR95 and +3.04% AUROC on CIFAR-10, and top-ranked performance on hard OOD datasets, ie., ImageNet-O and OpenImage-O.
INTRODUCTION
Identifying the out-of-distribttion (OOD) samples is vi- tal for practical applications since the deployed systems can not handle these unknown data in human-liis like fashion, e.g., performing rejection. Based on research in animal behavior, such as Tierney & Jane (1986) and Shettleworth (2001), ), these abilities demonstrated by humans are rooted in experience, which means we earn them from interactions with environments. For instance, the adults will not reach their fingers towards the flames in case of burning, while the children may take this risky action due to their limited cognitive abil- ity. This principle reveals that the failure encountered by the well-trained classification models in detecting OOD examples results from the lack of interactions with OOD data. Consequently, acquiring sufficient and diversified OOD signals from the in-distribution (ID) data has become promising direction, which helps mitigate the gap between classification and OOD detec- tion. Fig. provides intuitive illustration of natural supervision's importance in OOD detection. In particular, we employ the pre-trained ResNet-50 (He et al., 2016) as the backbone and ImageNet (Russakovsky et al., 2015) as the ID data. The OOD samples
S5OD
GlobalAvgPopl
iNaturalist
Figure Feature visualization the ID and OOD images. The green/orange dots are surrogate ID/OOD features generated by SSOD. The blue/gray dots natural ID/OOD features of ImageNet//Naturais:
from iNaturalist (Hom et al., 2018). The blue points are pooled features from ImageNet, i.e., the natural ID features. The green/orange points indicate the ID/OOD signals generated by our SSOD, i.e., the ID/OOD proxy. We can see that

---

Published as a conference paper at ICLR 2024
the OOD signals sampled by SSOD build defensive wall between the real ID and OOD data, which can serve as the proxy of authentic OOD images to supervise the training procedure.
Recall other existing OOD detection methods that primarily rely on the perspective of statistical difference, i.e., observing distinctions of the pre-trained features between ID and OOD samples. These methods use heuristic rules to filter out OOD data in two-stage manner, i.e., pre-training and post- post-processing, which suffer from the following drawbacks. Firstly, the frozen model weights are obtained on the ID classification task with limited OOD supervision. Therefore, the extracted features inherently carry bias, which is not distinguishable enough for identifying OOD data (cf. Fig. 4). Secondly, the two-stage design yields poor scalability and efficiency since it is unsuitable for scenarios without tre-trained models, e.g., given insufficient training data, a two-stage model may fail to obtain high-quality weights, thus encountering the performance drop in detecting OOD data.
To tackle the issues above, this paper interprets the OOD detection task with a unified probabilistic framework, which can widely include many previous individual designs. Concretely, our framework starts from Bayes' rule and divides the robust classification problem into two tasks: conventional ID classification and OOD detection. According to our theoretical analysis, the deficiency encountered by traditional neural networks in identifying OOD data arises from the absence of critical component, i.e., an OOD factor that estimates the likelihood of images belonging to the in-distribution. Further- more, sailing from this general foundation, present Self-supervised Sampling for OOD Detection (SSOD), an end-to-end trainable framework w/o resorting to explicit OOD annotations. In contrast to the observed paths that hug synthetic OOD features, SSOD directly samples natural OOD supervision from the background of ID images, i.e., self-supervised, getting rid of the constraints resulting from the lack of labeled OOD data and the deviation introduced within the OOD feature syntheses stage. Extensive experiments demonstrate that the joint end-to-end training significantly improves the OOD detection performance and guides the model focus more on the object- discrriiiiiiiittivve characters instead of the meaningless background information (cf. Fig. 2). The major contributions of this paper are summarized as follows:
We establish a general probabilistic framework to interpret the O0D detection, where various OOD methods can be analyzed, with main differences and key limitations clearly identified.
To promote ID/OOD features being more distinguishable, we design an end-to-end trainable model, namely supervied Sampling for OOD Detection (SSOD), to sample natural OOD signals from the ID images. SSOD avoids the labor-intensive work of labeling sufficient OOD images.
SSOD is evaluated across various benchmarks and model architectures for OOD detection, where it outperforms current state-of-the tf-tte--rttttttte approaches by a large margin, e.g., improving KNN (Sun et al., 2022) w/ and w/o contrastive learning with -20.23% and -38. % FPR95 on Places (Zhou et al., 2018), and Energy (Liu et al., 2020) with 30.74% FPR95 and +8.44 % AUROC SUN (Xiao et al., 2010), to name a few. The scalability and superiority of SSOD promise its potential to be starting point for solving the OOD detection problem.
METHODS
We introduce the unified probabilistic OOD framework, present the detailed interpretation of existing OOD detection methods from our unified view, and formulate the SSOD finally.
2.1 PROBABILISTIC OOD DETECTION
We formalize the OOD detection task as binary classification problem. Concretely, we consider two disjoint distributions on the data and label space, denoted as Sip x Vip and Scon x Doon. representing the ID/OOD distribution. We note that 3'D and VoOD have no overlap. i.e., ViD n DooD = ø. OOD detection aims to train a model that can effectively distinguish the source distribution of a given image Moreover, for y E SIp VID. the classifier f() should correctly predict its category.
Supposing that we have M known classes within the ID data, depicting as [Wi, W2...., war) We don't 'tn't distinguish the categories of OOD data. Thus, the images sampled from unknown classes are depicted as the Was+1 1 no matter their categories or domains. Given an image  we aim to learn a classifier f(-) that reports the posterior probability of x belonging each category, which is ((
2

---

Published as a conference paper at ICLR 2024
(a) Locality of ResNet-50. (b) Locality of SSOD. (c) Samplers of ResNet-50. (d) Samplers of SSOD,
Figure 2: (a): The image patches blue are recognized as penguins with over 40% confidence by the vanilla ResNet-50. (b): The SSOD counterpart of (a). We see that SSOD focuses more discriminative characters, reducing the false negatives. (c): The image patches in green/red sampled ID/OOD supervision with over 95% confidence by ResNet-50. (d): The SSOD counterpart of (c), we notice that SSOD identifies more background as OOD data. SSOD is motivated by the local property of conventional networks as illustrated in (a), and the results depicted in (b) reveal that the joint training enhances the local property. The confidence is manually selected.
for i E classification problems, and correspondingly, we have M discriminative functions verify whether the input image x belongs to category wy, Taking (x = -; + 'I' as the score function, where s; is learnable with respect the classifier and T is the bias term. Greater ge(x) yields higher confidence of x belonging to category w; To make the confidence meet intuition, we employ (-)) function to map the logits g.).) the form probability, saying (",, |x) = o (z)())) where the superscript indicates the binary classifier. With these auxiliary notations, we have the following formula:
M). Noting that P(wM++1/z) = 11 1- ΣM P(w;(2). We first consider M binary
(w;lx) (g)(()) e
Based on the DS evidence theory (Dempster, 1967), i.e., DST, we calculate the posterior probability of æ belonging to any ID category w; as follows:
M II j=1jsf
P(u,(=)= 2 p
(1 (wcl)z)
(2)
where normalization factor. Based on the equation that P(wM+11()) = 1- DM Pww(2x) we can obtain the expression of Z as follows:
M
M
M
P (w/l))
II
(1 - Pd (w,
II (1 pb (u (x)
(3)
ID Confidence
OOD Confidence
In the equation above, the first term of indicates the sum of probability that x belongs any known classes, while the last term represents that of is OOD data. We substitute the expression of Z into Eq for simplification. The obtained results are depicted as follows:
P(wx)or)
1 I I+ -
We assume that 7' is 8M41 since they are both trainable. Concerning that e -M++++ = 1, then, Eq 4 can be transformed as:
P(ws(z)= = M+1 e-
=
M+1 e
(5)
3

---

Published as a conference paper at ICLR 2024
The above expression of P(W;(22) is in the form of softmax classification, except the denominator contains an extra term, i.e., e 11,11 With simple post prot-proesssing transformation, Eq can be depicted follows:
レ= E=
P(W;(z)) = EM+1 e
(6)
Σ j==
ID Classification
0OD Detection
Clearly, the first factor of Eq 6 is the conditional probability that æ belongs to category w; assuming is sampled from ID data simultaneously. The second factor of the above equation indicates the probability that x is ID images. Recall that Sip and SOOD are the sets of in-distribution and out-of-distribution data, thus, Eq 6 tells us conclusion that:
e 8
25=1e A P(w:|æ E Sm, P(r E Sin
())
ΣM Σn E MMI1 E
factor
OOD factor
Since the output of conventional neural networks activated by Softmax is P(wsl |r E SID, x), we focus formulating the second term (OOD factor), ie.. P(x SD)(z).
2.2 REVISIT OOD DETECTION METHODS FROM THE PROBABILISTIC VIEW
We interpret several classic OOD detection techniques from the perspective our proposed proba- bilistic framework and find that most OOD detection methods hold P(w, (a: E Sm, x) fa(x)), ie., the i-th dimension of the classifier's output activated by Softmax function. Thus, the crucial point is how to compute the OOD factor P(x SD)(z).
Methods based on logits, e.g., Max-Softmax Probability (MSP) (Hendrycks, 2017), which directly employs the Softmax output of classifiers as the ID/OOD score, aiming distinguish them with classification confidence. Concretely, given image æ, MSP uses the following expressions to depict the procedure of OOD detection:
x E Soon, max fxx < f()) E SID, max f(x) 2
-
(8)
MSP expects the classifier f(-) to assign higher confidence, i.e., max f(x), to ID samples while lower of that to the OOD. Obviously, for MSP, the OOD factor is built as:
P( E Sin x>) = P(max f=) 27).
(9)
Methods based on features try to distinguish ID/OOD data based on their deep features extracted by the backbone (t()), such ReAct (Sun et al., 2021), BAL (Pei al., 2022), VOS (Du al., 2022), and KNN (Sun et al., 2022), etc. Taking ReAct (Sun et al., 2021) as an example, it builds the OOD factor P(r Sp((z) (z)) in hard threshold manner with linear projection, depicted as:
P(x Smp/2) P(W w-w.- ReAct(h(z),c) + b 7),
(10)
where WT and b the weight matrix and bias vector, ReAct ReAct(((x), element-wise truncation function dominated by the threshold c, and is a hard threshold. Instead of the OOD-syntheses-free schemes like ReAct (Sun al., 2021) and KNN (Sun et al., 2022), BAL (Pei et al., 2022) and VOS (Du et al., 2022) generate ID/OOD supervision in the feature space to optimize the ID/OOD classifier with P( SIn(z) (d(h(x)), where d(-) is discriminator that expect to assign higher confidence for ID features.
min(h(x)), c} is an
In summary, most OOD methods approximate the OOD factor P(z E Sm(z) by P(f(x) f(SD))z) as MSP (Hendrycks, 2017). or P(h(z) e h(S:n)))) [z) like ReAct (Sun et al., 2021). BAL (Pei et al., 2022), VOS (Du et al., 2022), and KNN (Sun et al., 2022). We note here that f(x) is the Softmax output, and h((z) is the feature extracted by the backbone. Nevertheless, there is a significant bias introduced by f(-) and (t.)) since they are trained for the ID classification. It adversely affects the discrimination of ID and OOD data. Furthermore, the generated OOD signals in feature space, e.g. BAL (Pei et al., 2022) and VOS (Du et al., 2022), do not necessarily lead to the existence a corresponding natural OOD image. Consequently, its effectiveness in open world scenarios may be limited. To remove these obstacles, we propose to sample OOD supervision from the images and optimize the OOD factor directly.
4

---

Published as a conference paper at ICLR 2024
CLS Head
CNN
ID/OOD Sampler
Self-supervised
DOD Picid
0.2
0.3
0.4
0.8
0.3
0.9
Input image
Feature map
0.2
0.7
0.6
0.1
Figure Self-supervised Sampling for Out-of-distribution Detection (SSOD). We adopt self- supervised sampling scheme to train the OOD discrimination branch, i.e., the OOD head with the supervised signals injected by the CLS head. The image patches green/red/gray are ID/OOD/nnvllid signals identified by the CLS head. SSOD jointly trains these two branches.
2.3 SELF-SUPERVISED SAMPLING FOR OOD DETECTION
Inspiration of SSOD. In Fig. we notice and verify that the image background is a good proxy for OOD data. Thus, we expect to extract the background information from the feature maps, which can be regarded as the natural OOD supervision. Prior studies, e.g., Redmon et al. (2016) and Carion et al. (2020), have demonstrated that traditional neural networks are capable of retaining spatial information, Le., a position of the feature map reflects the corresponding position the input image. With these foundations, aim design an OOD patch sampler to collect background information from the feature maps. In Fig. 2 (a). the ResNet-50 (He et al., 2016) trained on ImageNet (Russakovsky et al., 2015) downsamples the input image and yields corresponding feature map (R HR-H XC), where each image patch is projected to feature vector (RC) located at the corresponding position. The classification head reports the category for each feature vector. We highlight the correctly classified patches in Fig. 2 (a) and (b). The results suggest that the confidence scores are much higher for patches contained the main objects while lower for the backgrounds.
Formulation of SSOD. For a feature map within RC (i.e., the channel, height, and width) produced by the neural networks, we can apply the classifier along spatial axes and obtain confidence score map within R HxW where M is the number of categories in ID data. The patches with a low confidence score, egg. lower than 5% on the ground-truth category, are recognized as OOD ID patch sampler selects some
samples highlighted in red in Fig. (c) and (d). Symmetrically, image patches with high confidence scores, e.g., greater than 95%, as the ID samples (cf. Fig. 2, the green patches), helping balance the positive (ID) and negative (OOD) samples.
Formally, we use h(.). (.),()), and fooe(-) ) to denote the backbone removing the classification head, the multi-category classification head, and the binary ID/OOID discrimination head, respectively. Given an input image x with label y, CCHHX h(x) is the feature map. The prediction result of the classification model is:
= fal (GAP(((()) = fal fG.P
(11)
where GAP is the global average pooling the spatial dimensions. Similarly, when applying f.)e(.) on each patch XCxXX without pooling operation, we can get the confidence score map MMW fas (x (xXXXW within Mx Moreover, pick the confidence along the target axis, egg., if the target label of x is then we collect the confidence along the j-th axis of M, yielding target confidence map within R &HXW i.e., i HWW We use the ID/OOD sampler to select patches with high/low scores on the target label as the ID/OOD supervision. Concretely, for i E {1, 2, 3, HW} }. obtain the following self-supervised OOD labels from the classification head:
0, 1.
HW HW
god
7
(12)
N/A,
1 - Y < 0 HW
<7
where gW indicates the predicted confidence of each image patch belonging to the target category, and yy confidence threshold, e.g., 95%. Remind that the image patches assigned with the positive
5

---

Published as a conference paper at ICLR 2024
ResNet ID ResNet OOD
SSOD ID
SSOD OOD
(a) ID vs. OOD features in ResNet-50.
(b) ID vs. OOD features in SSOD.
Figure 4: The I-SNE visualizations of deep features for the ImageNet (ID) and iNaturalist (OOD), (a): features extracted by the conventional ResNet--0 (He et al., 2016), and the FPR95 is 54.99 %. (b): features extracted by our SSOD, and the FPR95 14.80% %. The green/red dots represent ID/OOD features, i.., the ImageNetiNaturraisst SSOD improves the ID/OOD discriminability notably. The overlap of ID/OOD features reflects the false positive rate (FPR95).
label are highlighted as green in Fig. and the negative patches are marked in red. We drop the left image patches (i.e., the non-highlighted patches in Fig. 2, the invalid patches in Fig. 3, and N/A in Eq 12), and therefore, they provide no ID/OOD supervisions during the training. With the OOD head, we obtain the ID/OOD prediction (jood):
good = food fx (xHXXW)
(13)
Since only part of the image blocks is selected as ID/OOD supervision in Eq 12, consequently, the loss is performed on the corresponding predicted results in Eq 12 and Eq 13. The overall objective of SSOD is formulated with the cross entropy loss (CE):
L = CE( y) + &CE(
.y pod)
(14)
where a is a balance parameter, good E 2 and yood E IR During the training/inference phase, the OOD factor of input images can be calculated as follows:
P(x Sep (x) ((AA
(15)
where Sigmoid function is used to predict the probability of input image belonging the ID data. With the proposed SSOD above, we can train the OOD detection branch end-to-end with natural OOD supervisions sampled from the patches of ID background as illustrated in Fig. 3.
EXPERIMENTS
This section addresses the following problems: 1) How does SSOD perform OOD detection benchmarks? 2) Whether SSOD is stable under different hyper-parameter settings? 3) Whether SSOD generalizable across different backbones?
3.1 EXPERIMENTAL. SETUP
Benchmarks. We employ large-scale benchmarks in OOD detection, including ImageNet (Rus- sakovsky et al., 2015) groups, CIFAR-10 (Krizhevsky et al., 2009) groups, and hard OOD groups In ImageNet groups, we set four OOD datasets, which are iNaturalist (Horn et al., 2018), SUN (Xiao et al., 2010), Places (Zhou et al., 2018), and Texture (Cimpoi et al., 2014). In CIFAR-10 10 groups, we set five OOD datasets, which are SVHN (Netzer et al., 2011), LSUN (Yu et al., 2015), iSUN (Xu et al., 2015). Texture (Cimpoi et al., 2014), and Places (Zhou et al., 2018). Under hard OOD setting, the ImageNet is employed as ID data while ImageNet-O (Huang & Li, 2021) and OpenImage-O (Wang et al., 2022) are selected as OOD data. The detailed information of these datasets and training/evaluation protocol is attached in Appendix A.3 and A.4.

---

Published as a conference paper at ICLR 2024
Table OOD detection results on ImageNet. + indicates lower is better, 1 means greater better, We highlight the best and second results using bold and underlining. We use (w/) and (w/o to indicate using and without using supervised contrastive learning. If not specified, all methods employ ResNet-50 pre-trained on ImageNet- Ik as the backbone. F and R indicate FPR95 and AUROC.
iNaturalist
SUN
Places
Texture
Average
Method
1
A
4 F
A
4
A
F
A
F
1 A
MSP
54.99
87.74
70.83
80.86
73.99
79.76
68.00
79.61
66.95
81.99
MSP (CLIP-B)
40.89
88.63
65.81
81.24
67.90
80.14
64.96
78.16
59.89
82.04
MSP (CLIP-L)
34.54
92.62
61.18
83.68
59.86
84.10
59.27
82.31
53.71
85.68
MaDist
97.00
52.65
98.50
42.41
98.40
41.79
55.80
85.01
87.43
55.47
ODIN
47.66
89.66
60.15
84.59
67.89
81.78
50.23
85.62
56.48
85.41
GODIN
61.91
85.40
60.83
85.60
63.70
83.81
77.85
73.27
70.43
82.02
KLM
27.36
93.00
67.52
78.72
72.61
76.49
49.70
87.07
54.30
83.82
Energy
55.72
89.95
59.26
85.89
64.92
82.86
53.72
85.99
58.41
86.17
KNN (w/o)
59.08
86.20
69.53
80.10
77.09
74.87
11.56
97.18
54.32
84.59
KNN (W/)
30.18
94.89
48.99
88.63
59.15
84.71
16.97
94.45
38.82
90.67
MOS
9.28
98.15
40.63
92.01
49.54
89.06
60.43
81.23
39.97
90.11
Fort (ViT-B)
15.07
96.64
54.12
86.37
57.99
85.24
53.32
84.77
45.12
88.25
Fort (ViT-L)
15.74
96.51
52.34
87.32
55.14
86.48
51.38
85.54
43.65
88.96
MCM (CLIP-B)
30.91
94.61
37.59
92.57
44.69
89.77
57.77
86.11
42.74
90.77
MCM (CLIP-L)
28.38
94.95
29.00
94.14
35.42
92.00
59.88
84.88
38.17
91.49
SSOD (Ours)
14.80
96.91
28.52
94.33
38.92
90.78
45.32
87.02
31.89
92.26
Comparable methods. We choose both the classic and latest OOD detection methods for comparison. With regard to the classic schemes, we select the MSP (Hendrycks, 2017), MaDist (Lee al., (b), ODIN (Liang et al., 2018), KL Matching (KLM) (Hendrycks et al., 2019), MaxLogit (MaxL) (Hendrycks et al., 2019), GODIN (Hsu et al., 2020), CSI (Tack et al., 2020), and MOS (Huang & Li, 2021). Besides, we also use the Energy (Liu et al., 2020), which is the representative of the score-based calibration method, and the KNN (Sun et al., 2022), which is one of the latest schemes, comparable methods. ResNet-18 and 50 (He al., 2016) chosen the backbone for CIFAR-10 and ImageNet. KNN (Sun et al., 2022) has two different versions, i.e., w/ and w/o contrastive learning. For fair comparison with other methods, for all comparable techniques. Except for the above traditional methods, we also provide the results of large languaeee models, such MCM (Ming al., 2022), FLYP (Goyal et al., 2022), and Fort (Fort et al., 2021). The detailed information about them is attached in Appendix A.2.
employ contrastive learning
3.2 COMPARISON WITH STATR-OF-THE-ARTS
OOD detection on ImageNet. We use iNaturalist, SUN, Places, and Texture as the OOD data. Following Sun et al. (2022), we randomly select 10,000 OOD images from each dataset for evaluation and keep the quantity of ID and OOD data the same for reliable FPR95. The KNN (Sun et al., 2022) methods have two versions, Le., using and w/o using contrastive learning. We provide both results for comparison and mark the contrastive version with (w/). MSP. Fort, and MCM employ strong backbones improve the detection performance, such as CLIP (Radford et al., 2021) and ViT (Dosovitskiy et al., 2021). We annotate these backbones in the wake of each method. If not specified, all left methods use no contrastive learning and employ ResNet-50 as the backbone. We evaluate the performance of each method based on its averaged FPR95 and AUROC on the above four datasets. The technique yielding the best detection performance is highlighted using boldface, while the closely following one is underlined. From the results depicted in Table 1, we notice that SSOD yields competitive state-of-the-art performance on iNaturalist, SUN, and Places. Specifically, on iNaturalist, SSOD and MOS report comparable detection FPR95 as 14.80% and 9.28%. In contrast, other complicated methods such as VIT (Dosovitskiy et al., 2021) based Fort and CLIP (Radford et al.. 2021) based MCM only achieve a FPR95 of 15.74% and 28.38%. Moreover, Places, SSOD consistently establishes the top detection results (FPR95) as 28.52% and 38.92%, outperforming other techniques with simple ResNet-50 backbone. On Texture, most methods
SUN and
7

---

Published as a conference paper at ICLR 2024
Table 2: OOD detection results on CIFAR-10. J indicates lower is better, ? means greater better, We highlight the best and second results using bold and underlining. All values are percentages.
Methods GODIN
0OD
Metrics
MSP
MaDist
ODIN
Energy
CSI
KNN
SSOD
FPR95
59.66
9.24
20.93
15.51
54.41
37.38
24.53
2.12
SVHN
1 AUROC
91.25
97.80
95.55
96.60
91.22
94.69
95.96
99.44
FPR95
45.21
67.73
7.26
4.90
10.19
5.88
25.29
4.42
LSUN
AUROC
93.80
73.61
98.53
99.07
98.05
98.86
95.69
99.11
FPR95
54.57
6.02
33.17
34.03
27.52
10.36
25.55
10.06
iSUN
AUROC
92.12
98.63
94.65
94.94
95.59
98.01
95.26
98.16
FPR95
66.45
23.21
56.40
46.91
55.23
28.85
27.57
1.91
Texture
AUROC
88.50
92.91
86.21
89.69
89.37
94.87
94.71
99.59
FPR95
62.46
83.50
63.04
62.63
42.77
38.31
50.90
7.44
Places
AUROC
88.64
83.50
86.57
87.31
91.02
93.04
89. 14
98.42
FPR95
57.67
37.94
36.16
32.80
38.02
24.20
30.80
5.19
Average
AUROC
90.90
89.29
92.30
93.52
93.05
95.90
94.15
98.94
ID ACC
94.21
94.21
94.21
93.96
94.21
94.38
94.21
94.17
fail to identify the OOD images and demonstrate high FPR95. Nevertheless, SSOD detects most outliers of Texture and reports FPR95 of 45.32%, which is better than all methods except for KNN (Sun al., 2022). This failure case is caused by the overlap between ImageNet-Il 1k and Texture (cf. Appendix A.5). Overall, SSOD obtains the FPR95 and AUROC as 31.89% and 92.265 on the four above large-scale datasets, surpassing the closely following method with -6.28 % FPR95 and +0.77 % AUROC. Besides, we also demonstrate the confidence distribution of ID/OOD images in Appendix Fig. 6, evidencing that SSOD significantly reduces the overlap between IDD and OOD confidence distributions, promising a much better performance in detecting OOD examples.
OOD detection on 100.R-10. Following previous methods, we adopt SVHN, LSUN, iSUN, Texture, and Places as the OOD data. All comparable techniques take ResNet-188 18 as the backbone. SSOD resorts no pre-trained weights and trains the classifier from scratch. CIFAR-10 and other OOD samples are resized to 224 x 224 to obtain sufficient background information from the images. The detailed results are depicted in Table 2. On iSUN, the detection performance of SSOD is marginally lower than MaDist, while on the left four datasets, SSOD consistently outperforms other methods by a significant margin. In particular, on Places and Texture, SSOD reduces the FPR95 of 30.87% % and 21.30 % compared to the closely following one, evidencing its effectiveness. From an overall view, on the above OOD datasets, SSOD improves the detection ability of ResNet-18 by -19001 % FPR95 and +3.04% % AUROC on average, which can be established as one of the state-of-the-arts.
Hard OOD detection. ImageNet-O (Huang Li, 2021) and OpenImage-O (Wang et al., 2022) are employed hard OOD examples for ImageNet, since they have similar natural scenes to those in ImageNet. All methods use ResNet-50d (He et al., 2016), a variant of ResNet-50,, as the backbone. As shown in Table 3. SSOD achieves top-ranked performance on OpenImage-O O (Wang et al., 2022). Since the ImageNeet-O mainly contains adversarial images, leading to the classifier's 's wrong prediction, SSOD reports higher FPR95 compared the best previous methods. Overall, SSOD still achieves comparable state-of-the-ar performance, yielding 4.38% higher FPR95 compared to ViM (Wang al., 2022) and 1.15% lower AUROC compared to ReAct (Sun et al., 2021).
3.3 ABLATION STUDY
Ablations on hyper-parameter a. Recall that controls the importance of loss generated by the OOD head, balancing classifiers' classification performance and OOD detection ability. We employ CIFAR-10 (Krizhevsky et al., 2009) and Places (Zhou et al., 2018) as the ID and OOD data validate the stability of a. SSOD uses ResNet-18 as the backbone. From the ablations depicted in Table 4. notice that the classifier detects OOD input better with increasing a, while the ID ACC is gradually descending. We expect to boost the robustness of classifiers while not affecting the model's performance. Therefore, we set to 1.0 throughout the experiments.
8

---

Published as a conference paper at ICLR 2024
Table 3: Hard OOD detection results.
Table 6: OOD detection with different back- bones. We highlight the best and second re- sults using bold and underlining. indicates the SSOD counterpart.
ImageNet-O Openlmage-4 O
Average
Method
F
t
+ A
LF
A
MSP
93.85
56.13
63,53
84,50
78.69
70.32
Energy
90.10
53.95
76.83
75.95
83,47
64.95
iNaturalist
Places
Average
Method
ODIN
93.25
52.87 64.49
81.53
78.87
67.20
F
A
1F
A
F
A
MaxI.
92.65
54.39 65.50
81.50
79.08
67.95
KLM
88.50
67.00 60.58
87.31
74.54
77.16
R-18
60.00
86.32
76.96
76.86
67.42
81.65
ReAct
72.85
81.15 60.79
85,30
66.82
83.23
R-18*
19.96
95.21
46.48
88.31
33.22 91.76
MaDist
78.45
68.02 55.91
89.52
67.18
78.77
R-34
57.88
86.44
75.96
78.16
66.92 82.30
VIM
76.00
74.80 50.45
90.76
63.23
82.78
R-34*
15.88
96.28 43.44 87.33
29.66 91.81
SSOD
79.80 74.43 55.41
89.73
67.61
82.08
R-50
54.99
87.74 73.99 79.76 64.49 83.75
Table 4: Ablations on the peree-paraeeter a.
R-50*
14.80
96.91 38.92 90.78 26.86 93.85
0.5
0.7
1.0
1.2
1.5
R-101
56.58
86.08 70.78 80.18
63.68 83.13
4. FPR95
11.64
9.44
7.44
8.56
10.80
R-101*
18.78
95.56 42.72 87.84 30.75 91.70
1 AUROC
97.33
97.74
98.42
98.19
97.49
1 ID ACC
94.25
94.17
94.17
92.53
91.36
D-121
59.44
86.79 69.88 80.12 64.66 83.46
D-121
21.64
94.84 44.88 85.51 33.26 90.18
Table 5: Ablations of balance schemes.
RegN
54.96
87.74 70.28 80.03
62.62 83.89
Scheme
LW
DR
LWB
RegN'
16.36
96.05 35.80 89.79 26.08 92.92
J. FPR95
10.72
8.48
7.44
1 AUROC
97.66
98.05
98.42
MobN
52.52 88.19 73.36
79.41 62.94 83.80
T ID ACC
93.97
94.11
94.17
MobN* 24.76 94.47 42.52 88.38 33.64 91.43
Imbalance issue between ID/OOD features. During the training of the OOD head, we obtain much more background features since the objects only occupy small part of the image. To promote training stability, we design three ways to tackle this issue, which are Loss Weighting (LW), Data Resampling (DR), and Loss-Wise Balance (LWB). LW multiplies a balance factor the loss generated by the background features, DR randomly samples equivalent ID/OOD features within each image, and LWB calculates the cross entropy generated by the ID/OOD features separately and picks their mean value as the loss objective. CIFAR-10 10 and Places are the ID and OOD data. Based on the ablations depicted in Table SSOD employs LWB for data balancing.
OOD detection across different model architectures. To validate the transfer ability of SSOD on different model architectures, select ImageNet as the ID data, iNaturalist, and Places the OOD data. Considering the deployment on portable devices, we test both the conventional and lite models, such as ResNet (He et al., 2016) series. DenseNet-122 121 (Huang et al., 2017), RegNet (Y-800MF) (Radosavovic et al., 2020), and MobileNet (V3-Large) (Howard et al., 2019). Table 6 uses L F and t A to indicate the FPR95 and AUROC. R- 1-1344//50/101 are ResNett18//4/500/ 18/34/50/101. D-121, RegN, and MobN are DenseNet-121 121, RegNet, and MobileNet. Compared to Table 1. methods (+) shown in Table 6 achieve state-of-the-art performance, evidencing the scalability of SSOD. The improvements between the vanilla classifier and its improved SSOD version are also significant, e,g. -37.63% FPR95 on ResNet and -36.54% % FPR95 on RegNet.
4 CONCLUSIONS
This paper proposes a probabilistic framework that divides the robust classification into ID and OOD factors. This provides comprehensive overview of existing OOD methods and highlights the critical constraint of relying on pre-trained features. To address this limitation, we introduce an end-to-end scheme called SSOD, which optimizes the OOD detection jointly with the ID classification. This approach leverages OOD supervision from the background information of ID images, eliminating the data-collecting need. Extensive experiments have validated that SSOD achieves competitive performance in detecting OOD data, which can be a starting point for future research.

---

Published as a conference paper at ICLR 2024
REFERENCES
Haoyue Bai, Fengwei Zhou, Lanqing Hong, Nanyang Ye, S.-H. Gary Chan, and Zhenguo Li. Nas-ood: Neural architecture search for out-of-distribution generalization. In ICCV, 2021.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey 7agoruyko. End-to-end object detection with transformers. In ECCV, 2020.
Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny. Visualgpt: Data-efficient adaptation of pretrained language models for image captioning. CVPR, 2022a.
Minghui Chen, Cheng Wen, Feng Zheng, Fengxiang He, and Ling Shao. VITA: A multi-source vicinal transfer augmentation method for out-of-distributio generalization. In AAAI, 2022b.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, 2014.
A. P. Dempster. Upper and lower probabilities induced by multivalued mapping. In The Annals of Mathematical Statisties, 1967.
Patrick Dendorfer, Sven Elflein, and Laura Leal-Taixé. Mg-gan: A multi-generator model preventing out-of-distribution samples in pedestrian trajectory prediction. In ICCV, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. VOS: learning what you don't know by virtual outlier synthesis. In ICLR, 2022.
Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of out-of-distribution detection. In NeurlPS, 2021.
Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. In CVPR, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Dan Hendrycks. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR, 2017.
Dan Hendrycks, Steven Basart, Mantas Mazeika, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. A benchmark for anomaly segmentation. In CoRR, 2019.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alexander Shepard, Hartwig Adam, Pietro Perona, and Serge J. Belongie. The inaturalist species classification and detection dataset. In CVPR, 2018.
Andrew Howard, Ruoming Pang, Hartwig Adam, Quoc Le, Mark Sandler, Bo Chen, Weijun Wang. Liang-Chieh Chen, Mingxing Tan, Grace Chu, Vijay Vasudevan, and Yukun Zhu. Searching for mobilenetv3. In ICCV, 2019.
Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized ODIN: detecting out-of- distribution image without learning from -ui-tf-distribution data. In CVPR, 2020.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In CVPR, 2017.
Rui Huang and Yixuan Li. Mos: Towards scaling out-of-distribution detection for large semantic space. In CVPR, 2021.
Rui Huang, Andrew Geng, and Yixuan Li. On the importance gradients for detecting distributional shifts in the wild. In NewrlPS, 2021.
10

---

Published as a conference paper at ICLR 2024
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. In CoRR, 2023.
Alex Krizhevsky, Geoffre Hinton, et al. Learning multiple layers of features from tiny images. In CoRR, 2009.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-callirated classifiers for detecting out-of distribution samples. In ICLR, 2018a.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NeurlPS, 2018b.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In CoRR, 2023.
Shiyu Liang Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection neural networks. In ICLR, 2018.
Weitang Liu, Xiaoyun Wang, John D. Owens, and Yixuan Li. Energy-based out-of-distributioo detection. In NeurlPS, 2020.
Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into out-of- distribution detection with vision-language representations. In NeurlPS, 2022.
Yifei Ming, Yiyou Sun, Ousmane Dia, and Yixuan Li. How to exploit hyperspherical embeddings for out-of-distribution detection? In ICLR, 2023.
Felix Möller, Diego Botache, Denis Huseljic, Florian Heidecker, Maarten Bieshaar, and Bernhard Sick. Out-of-distribution detection and generation using soft brownian offset sampling and autoencoders. In CVPR Workshops, 2021.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NeurIPS Workshops, 2011.
Sen Pei, Xin Zhang, Richard YiDa Xu, Bin Fan, and Gaofeng Meng. Out-of-distribution detection with boundary aware learning. In ECCV, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross B. Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In CVPR, 2020.
Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In CVPR, 2016.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, 2015.
Sara Shettleworth. Animal cognition and animal behaviour. In Animal behaviour, 2001.
Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. In NewrlPS, 2021.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In ICML, 2022.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. CSI: novelty detection via contrastive learning distributionally shifted instances. In NeurIPS, 2020.
11

---

Published as a conference paper at ICLR 2024
Keke Tang, Dingruibo Miao, Weilong Peng, Jianpeng Wu, Yawen Shi, Zhaoquan Gu, Zhihong Tian, and Wenping Wang. Codes: Chamfer distof-distribution examples against overconfidence issue. In ICCV, 2021.
Tierney and Ann Jane. The evolution of learned and innate behavior:
Contributions from genetics and neurobiology to a theory of behavioral evolution. In Animal Learning and Behavior, 1986.
Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distribution with virtual-
logit matching. CVPR, 2022.
Hongxin Wei, Renchunzi Xic, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. In ICML, 2022.
Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010.
Pingmei Xu, Krista A. Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev R. Kulkarni, and Jianxiong Xiao. Turkergaze: Crowdsourcing saliency with webcam based eye tracking. In CoRR, 2015.
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. LSUN: construction
large-scale image dataset using deep learning with humans in the loop. In CoRR, 2015.
Bolei Zhou, Ágata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10
million image database for scene recognition. In TPAMI, 2018.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 RELATED WORK
Score-based posterior calibration. This line of research aims to find differences between the ID and OOD data, thus designing model-specific discriminative functions to identify the OOD samples. The related work includes ODIN (Liang al., 2018), LogitNorm (Wei et al., 2022), GradNorm (Huang et al., 2021), 1) ReAct (Sun et al., 2021), Energy (Liu et al., 2020), and CIDER (Ming et al., 2023), to name a few. Generally, these methods are usually pre- or post-processing schemes that demonstrate no need for retraining the neural networks. Although these methods above report considerable performance improvements and sometimes are training efficient, they do not necessarily lead to significant generalization ability. For example, ReAct (Sun et al., 2021) investigates the distinct behaviors of ID and OOD data after ReLU function, and therefore, fails to perform on architectures adopting other activations, such as GELJU, Sigmoid, and Tanh, etc. Similarly, the ODIN (Liang et al., 2018) investigates posc-proessiing schemes specially designed for Softmax, i.e., the temperature scaling. These specific designs promote OOD detection but limit the model's scalability. In contrast, our SSOD doesn't suffer from this limitation as it addresses the OOD detection directed by Bayes' theorem, which widely holds in general scenarios.
Auxiliary supervision from synthetic OOD data. The lack of OOD supervision is a critical factor leading to unsatisfactory performance in OOD detection. Thus, significant interest has been raised in generating synthetic OOD data. Existing approaches tackling this issue can be roughly divided into two manners, which are feature and image generation. The former samples OOD features from the ID boundary, such as VOS (Du et al., 2022), or generates them using GAN, such as BAL (Pei et al., 2022). In contrast, the image generation yields more training tax since it directly generates the OOD images, such as Conf (Lee et al., 8a), SBO (Möller et al., 2021), MG-GAN (Dendorfer et al., 2021), NAS-OOD (Bai et al., 2021), CODEs (Tang et al., 2021), and VITA (Chen et al., 2022b). In summary, these methods either introduce bias as they only consider the approximated feature space or are costly due to the image generation. Unlike the above methods, SSOD avoids feature bias and training tax by utilizing the universal local property of neural networks, extracting realistic OOD supervision from the ID images without generation cost.
A.2 OOD DETECTION VIA VISION-LANGUAGE PRE- TRAININGG
Recently, vision-language pre-training and large language models (LLMs) have become a trend to include various downstream tasks, such as text-image retrieval (Radford et al., 2021), VQA (Chen et al., 2022a), objects segmentation (Kirillov et al., 2023), and image captioning (Li et al., 2023), to name a few. Empowered by the massive amount of training data and model parameters, we expect to see blossom in detecting OOD data with these cross-modal models. For instance, MCM (Ming et al., 2022) and Fort (Fort et al., 2021) resort to the two-tower architectures, i.e., CLIP (Radford et al., 2021), for matching the text descriptions within closed set and open images. The matching score is used for ruling out the OOD data. Except for designing matching rules along the CLIP paradigm, FLYP (Goyal et al., 2022) notices that vison- lasou-lan fine-tuning also plays a decisive role in the final OOD detection performance.
A.3 BENCHMARKS
We perform experiments on ImageNet (Russakovsky al., 2015) and CIFAR-IO (Krizhevsky et al., 2009). For ImageNet, we follow the settings from Sun et al. (2022) and employ ImageNet-O (Huang & Li, 2021), OpenImage-O (Wang et al., 2022), iNaturalist (Horn et al., 2018). SUN (Xiao et al.. 2010), Places (Zhou et al., 2018 8), and Texture (Cimpoi et al., 2014) as the OOD images. For CIFAR- 10, we select SVHN (Netzer et al., 201 1), LSUN (Yu et al., 2015), iSUN (Xu et al., 2015), Places (Zhou et al., 2018), and Texture (Cimpoi al., 2014) as the OOD images. Images in CIFAR- 10 and ImageNet are resized and cropped to 224 x 224. All OOD images enjoy the identical pre-processing method. The detailed information of all employed datasets is presented as follows.
ImageNet. ImageNet (Russakovsky et al., 2015) is well known in image classification problems, containing 1,000 classes from the natural scene such tiger. goldfish, and house, to name few. This dataset is used as the ID data, expecting to get higher confidence from the classifier, i.e., higher OOD factor (cf. Eq 7).
13

---

Published as a conference paper at ICLR 2024
CIFAR-10. CIFAR-10 (Krizhevsky et al., 2009) is smaller compared to the ImageNet. It consists of tiny images within 10 classes, such as airplane, bird, dog, etc. All images contained CIFAR-10 are in the shape of 32 x 32. Our experiments treat CIFAR- 10 the in-distribution data and resize images into 224 x 224 to get bigger feature maps. The above operation marginally influences the classification performance since up-sampling these tiny images yields information gain.
SVHN. This dataset (Netzer et al., 2011) indicates the street view house number, consisting of digit numbers from the natural street view. Following Sun et al. (2022), we randomly select 10,000 images from this dataset to serve as the OOD data. All pictures from OOD data are expected to get lower confidence from the classifier.
LSUN. This dataset is used for visual recogmition, which was presented in Yu et al. (2015). It consists of over one million labeled images, including 10 scene and 20 object categories. Following Sun et al. (2022), 10,000 images from LSUN are treated as the OOD data.
iNaturalist. The existing dataset the image classification problem usually has a uniform distribution across different objects and categories. However, in the real world, the images could be heavily imbalanced. To bridge this gap between experimental and practical settings, iNaturalist (Horn et al., 2018), consisting of over 859, 5,0000 images within about 5,000 species (i.e., planets and animals), is presented. We randomly selected 10,000 images from this dataset as OOD pictures.
iSUN. This dataset is constructed based on the SUN (Xiao et al., 2010). iSUN (Xu et al., 2015) is standard dataset for scene understanding, containing over 20,000 images from SUN database. We use 10,000 iSUN images as the OOD data.
Texture. This dataset consists images carrying vital characters, i.e., patterns and textures of natural objects. Presented in Cimpoi al. (2014), Texture aims at supporting the analytical dimension in image understanding. In our experiments, we treat Texture as the OOD data. Data cleaning has been performed in Sun et al. (2022).
Places. This dataset is used for scene recognition, which was presented by Zhou et al. (2018), including over 10 million images such as badlands, bamboo forest, and canal, etc. Following Sun et al. (2022), we use 10,000 of these images to play the role of OOD data.
ImageNet-O. This dataset is released in MOS (Huang & Li, 2021), which is built as hard adversarial OOD data for ImageNet, consisting of 2,000 images.
OpenImage-O. O. This dataset is presented in ViM (Wang al., 2022), which follows natural class statistics and is manually labeled at the image level. The whole dataset contains 17,632 images.
A.4 TRAINING AND EVALUATION.
All images used in our experiments are resized to 224 x 224. We use AdamW as the optimizer. The learning rate starts from le-4 and halves every 30 epochs. The experiment runs on 8 NVIDIA Telsa V100 GPUs. The batch size set to 256, i.e., 32x8, each GPU is allocated with 32 images. We store the checkpoints yiclding the best FPR95 performance. About the evaluation, positive rate of the OOD dataset when the true positive rate ID images 95%, i.e., FPR95. We also compare the area under the receiver operating characteristic curve (AUROC) and the classification accuracy of ID images (ID ACC). We keep the quantity of ID and OOD data consistent following Hendrycks (2017).
report the false
A.5 FALIURE CASE ANALYSES
Recall that SSOD extracts OOD information from the background of training images and employs them as the proxy of OOD characters, revealing the potential of suffering limited diversity of the OOD supervision if the training images are not diverse. This phenomenon will be perceived if the OOD data differs domains from the training images. For example, the training images are natural scenes, while the testing OOD data is synthetic color blocks textures. To check this issue, We train the SSOD on ImageNet (Russakovsky et al., 2015) while testing it on Texture (Cimpoi et al., 2014). From the results depicted in Table 1, though SSOD achieves tankrdnked performance, it is KNN (Sun et al., 2022), increasing the FPR95 by about 33.76% %. The overlap between ImageNet and Textures causes this issue. Concretely, many images in the Texture (Cimpoi al., 2014) carry
than
14

---

Published as a conference paper at ICLR 2024
(a) SUN: 6.56%
(b) iNaturalis:: 0.59%
(c) Places: 5.71%
(d) Textures: 8.87%
(e) T'extures: 39.77%
(f) Textures: 86.40%
Figure 5: Confidence of OOD images assigned by SSOD. Successful cases: (a). (b), (c), and (d). These images are randomly sampled from several OOD datasets and assigned nominal ID confidence by the SSOD, Faliure cases: (e) and (f). These images carry vital symbols of objects appearing in ImageNet, e.g., (e) is the braided and (f) is cobwebbed, which are similar with the knot and spider in ImageNet. This phenomenon indicates the importance of data cleaning during the evaluation phase.
vital symbols of objects included in ImageNet (ef. Fig. 5). These overlaps lead to the inefficiency of SSOD during the comparison with KNN (Sun et al., 2022) on Texture dataset in Table
A.6 VISUALIZATION RESULTS
Inselat
Iopeket
Deriiy
Dealty
Inuge Sac
Imagaiee
Imagelet
Dentbyy
Der
Figure 6: Confidence distribution of images from ImageNet (ID) and other OOD datasets. The curve in orange indicates the confidence distribution of ImageNet, while the curves blue are that OOD images, including iNaturalist, SUN, Places, and Texture from left to right. In the figures above, the r-label and y-label are the confidence of images belonging to ID data and their overall density. Top: MSP yields bigger overlap between the ID and OOD confidence distribution, which means higher FPR95. Bottom: SSOD assigns higher confidence to ID images, and the overlap between ID/OOD confidence is relatively small, corresponding to the low FPR95.
In the figure above, the curves in orange the confidence distribution of pictures in ImageNet, while the blue curves are that of OOD data, including iNaturalist, SUN, Places, and Texture. As clearly depicted in Fig. 6, MSP (top) yields greater overlap between the distributions, which means more ID/OOD images are confused. In contrast, SSOD (bottom) significantly reduces the overlap between ID and OOD confidence distributions, promising much better performance in detecting OOD examples.
15

---

Published as a conference paper at ICLR 2024
CNN
Feature Map
(a) The local receptive field of convolution.
(b) Patch-wise classification results using ResNet-50.
Figure 7: Illustration of the spatial correspondence between the image and its feature. (a): Convolu- tional neural networks project the image patch to feature point within the feature map. The pooled feature map is then fed into the linear classifier determine its category. The representation scope (i.e., receptive field) of each feature point is dominated by the model's 's down down-sampling rate. (b): The trained classification head of ResNet-50 (He al., 2016) can correctly identify the feature point carrying significant characters. Concretely, we feed the model with a Great Pyrenees image, and the outputted feature map directly used for classification without pooling. We mark the correctly predicted image patch as red and otherwise lime green. is clear that the neural network can correctly identify the determinative parts within the image. The number in the right panel indicates the index of the image patch, and we only demonstrate part of the image for saving space.
16


Reviews:
[{"review_id": "w7p7jKJVma", "reviewer": "Reviewer_f65W", "summary": "The paper addresses the task of out-of-distribution detection in image classification tasks. The authors identify two main challenges the existing methods are facing, namely the absence of a unified perspective to interpret existing techniques and the need for natural OOD supervision to enhance model robustness without the necessity of explicit OOD data collection. Towards this end, the authors introduce a novel probabilistic framework that provides a unified interpretation of many existing OOD detection methods and the SSOD model that efficiently leverages the CNNs properties (retain spatial information) to create a distinction between ID and OOD samples. The paper's contributions are substantiated through extensive experiments (both is terms of benchmarks, as well as proper evaluation w.r.t state-of-the-art methods). SSOD shows consistent improvements in OOD detection metrics over previous methods, highlighting the model's effectiveness in recognizing and handling OOD samples without the need for additional OOD data.", "questions": "No further questions, besides addressing the weaknesses mentioned above.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Originality >> SSOD uses image backgrounds as natural proxies for OOD samples, which is a novel perspective in the field.\n\nQuality >> The best paper I read in a while, both in terms of problem statement/formulation and execution (exemplar experimental analysis).\n\nClarity >> The paper is well-articulated - clearly presents the research problem, the proposed solution, and the insights from the conducted experiments. Also, the authors seem to have made an effort to ensure that the concepts are accessible to anyone, regardless of their expertise in OOD detection.\n\nSignificance >> By proposing a general probabilistic framework, the paper unifies various existing approaches under a single interpretative lens, on top of SSOD that offers an evident advancement for OOD detection (relevant problem).", "weaknesses": "No fundamental flaws with the current submission, but just a suggestion to the authors - to include a more thorough discussion on the limitations of SSOD - what are the potential biases, the impact of background complexity on the model's performance, and scenarios where the model may not perform as expected - to start a discussion towards further improvements. Maybe within a dedicated section, tackling these aspects would be valuable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nQyUeatA5D", "reviewer": "Reviewer_ajFn", "summary": "This paper focuses on Out-of-Distribution (OOD) detection and introduces a unified probabilistic framework that divides the OOD detection problem into In-Distribution (ID) and OOD components. This framework presents an insightful overview of existing OOD detection methodologies and pinpoints their limitations (classifiers and features are often biased towards ID data). To address this challenge, the authors introduce Self-Supervised Sampling for OOD Detection (SSOD), utilizing image backgrounds as effective proxies for OOD data. The model employs separate ID and OOD heads, with the OOD head being self-trained through the utilization of confidence scores derived from the classification head. The results demonstrate that the proposed method significantly outperforms existing approaches by a substantial margin.", "questions": "1. Could you specify the number of patches used in the SSOD model? More specifically, what are the dimensions (height H and width W) in the last feature map?\n\n2. Does the performance of SSOD show sensitivity to the classification confidence parameter gamma?\n\n3. While SSOD generally surpasses other baselines in performance, as seen in the tables, there are instances where other baselines demonstrate notably strong results (e.g., MOS in iNaturalist and KNN in Texture). What could be the main reason for these exceptional cases?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation derived from the proposed general OOD detection framework seems both intuitive and solid. \n2. The paper is well-written and figures / tables are easy to follow.\n3. The proposed SSOD approach, while seemingly simple, demonstrates strong effectiveness.\n4. The experimental results are quite strong across various datasets.", "weaknesses": "1. Given that SSOD necessitates pseudo-labels for each patch (like semantic segmentation), \nSince SSOD requires the pseudo-labels for each patch (like semantic segmentation), I presume that the training expenses could surpass those of conventional pre-training methods. Could the authors provide a computational comparison of SSOD with other baseline OOD detection models, as well as standard classification models (e.g., ResNet-18, ResNet-50, etc.)?\n\n2. It appears that the principal interpretation derived from the probabilistic framework (Appendix A.1) might be more aptly positioned between Sections 3.1 and 3.2. Currently, the main motivation behind SSOD doesn't seem to be adequately emphasized.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BuOtPa3FGJ", "reviewer": "Reviewer_TJ6P", "summary": "- Authors tackle the problem of Out-Of-Distribution (OOD) detection in this work and show that the image background in In-Distribution (ID) datasets can act as a good proxy for OOD data, preventing the necessity to collect real/synthetic OOD data for training strong OOD detectors. \n- First, authors propose a general probabilistic framework that can explain existing OOD detection methods.\n- Next, using this interpretation, authors propose **S**elf-**S**upervised **O**OD **D**etection (SSOD) to exploit the natural OOD signals present in ID data. This prevents the need for collecting explicit real or synthetic OOD data making the pipeline more efficient.\n- With extensive experiments, authors show impressive results on several OOD benchmarks proving that image backgrounds of ID data can serve as a good proxy for OOD data.", "questions": "**Questions**\n- In the abstract, authors mention they leverage \"local property of convolution\" for OOD. But it hasn't been mentioned anywhere else. Can you elaborate what they mean by this?\n- In eq. 9 the $y_i^{\\text{OOD}}$ is the label for detecting OOD patches right? In which case, patches with confidence lower than 5% ($1-\\gamma$) should have a label 1 and not 0 right?\n- In the 2nd sentence below Eq. 11, is it \"During inference\" or \"During training/inference\"? The second loss term in Eq. 11 is applied on a spatial map from what I understood, so why do we have to compute $P(x\\in \\mathcal{S}_{\\mathbb{ID}}|x)$ explicitly during training? We just need that during inference correct?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The probabilistic interpretation of OOD detection methods is very useful to advance future research. The factorized interpretation also helps tune each independent component accordingly to optimize ID or OOD performance depending on the downstream objective.\n- The paper is well written, and the math is easy to follow once derived on paper. \n- The experimental section supports all the claims made in the paper.", "weaknesses": "I will summarize my concerns with this work under three broad sections.\n\n**Nomenclature**\n- Authors chose to proceed with the name SSOD for their work but the whole field of Semi-Supervised Object detection (SSOD) [1] already exists creating a bit of a confusion.\n- I recommend using SSOOD to avoid any confusion with an already established sub-field. \n\n**Presentation of results**\n- Authors claim that their first contribution is to provide a probabilistic interpretation of OOD, using which existing methods can be analyzed, but pushed the analysis section to the supplementary. In my opinion, if authors claim the probabilistic interpretation as an analysis tool, then it shouldn't be delegated to the supplementary.  \n\n**Motivation and intuition**\n- Authors show impressive results on several OOD benchmarks but the motivation that image background can serve as a good proxy for OOD data has some flaws.\n- First, using the penguin image example from Fig. 2, teaches the network to consider the background (in this case \"water\") as a signal for OOD. Now this will be ineffective in datasets constructed from comics or cartoons which is not one of the domains that authors evaluate their method on. This also explains why the scores are lower on OOD datasets constructed from SUN, Places because Imagenet is predominantly biased towards \"organisms\" and \"food\" with lower signals from indoor scenes and places. This raises the question \"Does this method work because of the choice of the OOD datasets used?\" This is partly explained by the results on \"Textures\" (I agree with the authors that textures has some overlap with patterns in the ID classes and the results are low, but I believe that is just part of the story). \n- Second, I think some additional analysis is required on the iNaturalist dataset (or maybe another toy setup on the subset of imagenet) which explains why the method works. If the OOD head is learning to detect any background patch as OOD, then what is the role of this in iNaturalist, where the background is usually water/trees/nature etc? From Imagenet, the network learnt to flag any background containing these regions as OOD, so does it ignore the ID category entirely and just focus on background? But that can't be true because the ID performance is also higher. The interplay between the CLS head and OOD head is very important to completely understand why image background is a good proxy and is missing from the paper. \n\n**I recommend authors to consider answering these questions for me to improve on my rating**\n\n[1] Liu, Yen-Cheng and Ma, Chih-Yao and He, Zijian and Kuo, Chia-Wen and Chen, Kan and Zhang, Peizhao and Wu, Bichen and Kira, Zsolt and Vajda, Peter, Unbiased Teacher for Semi-Supervised Object Detection, ICLR 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x0mcJpZcV5", "reviewer": "Reviewer_f34F", "summary": "The article focuses on Out-of-distribution (OOD) detection and proposes a model called Self-supervised Sampling for OOD Detection (SSOD) that does not require explicit OOD data annotation, which is able to extract natural OOD signals from the background of ID images, end-to-end end-to-end training of OOD detection branches. Experiments are conducted on several large-scale OOD detection datasets to demonstrate the effectiveness and superiority of SSOD.", "questions": "1.\tThe derivation of formula (1) seems to lack a detailed explanation. Can a more complete mathematical derivation of this formula be provided?\n2.\tThere are some grammatical and spelling errors in the text, please fix them\n3.\tThere seems to be a subjective bias in the interpretation of the experimental results. Can more objective evidence be provided to support these interpretations?\n4.\tIn the first paragraph on page 1, the author mentions that \"OOD detection empowers the model trained on the closed image set to identify unknown data in the open world\". But there is no definition or difference between what is closed image set and open world. Please define and explain these two concepts clearly in the introduction.\n5.\tOn page 4, paragraph 5, the author mentioned \"Since the ImageNet-O mainly contains adversarial images, leading to the classifier's wrong prediction, SSOD reports higher FPR95 compared to the open world\". SSOD reports higher FPR95 compared to the best previous methods\", but no reason or mechanism is given as to why the adversarial images cause SSOD's performance to degrade.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe article proposes a general and reasonable probabilistic framework to understand the OOD detection problem, which can cover a wide range of existing OOD detection methods in an innovative way.\n2.\tThe article proposes an effective self-supervised sampling mechanism capable of extracting useful OOD signals from ID images, avoiding the difficulty and cost of collecting and labeling large amounts of OOD data.\n3.\tThe article provides a detailed description and derivation of the working principle and design of SSOD.", "weaknesses": "1.\tThe article uses some uncritical and unreasonable assumptions in the derivation of the probabilistic framework, such as setting T as sM+1, ignoring the possible differences between sM+1 and T; treating P(wi|x) as P(wi|x∈SID,x), ignoring the possibility that x may belong to the OOD data.\n2.\tThe article uses a fixed and subjective threshold γ in the self-supervised sampling mechanism to determine whether an image block belongs to ID or OOD, which does not consider the possible differences and variations between different datasets, models, and categories.\n3.\tThere are some spelling mistakes, grammatical errors, and punctuation errors in the article; some irregular or inappropriate terms are used in the article, such as OOD-data-free model, OOD patch sampler, and so on.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qLOIuXY0jw", "reviewer": "Reviewer_u8Y6", "summary": "The paper addresses challenges in Out-of-distribution (OOD) detection by:\n1. Proposing a unified probabilistic framework to understand existing methods.\n2. Introducing a new model, SSOD, that uses natural OOD proxy from in-distribution data without needing explicit OOD samples.\n3. Demonstrating that SSOD significantly outperforms previous methods on major benchmarks.", "questions": "Please see the content in Weakness.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "1. The paper is articulately composed and organized, facilitating a clear understanding of most sections.\n2. To my understanding, the method introduced is innovative.\n3. The study is underpinned by a comprehensive set of experiments.", "weaknesses": "Major Points:\n\n1. **Concerns about the First Contribution:**\n\na. The derivation in Eqs 1-6 appears not helpful to the proposed method and Eq. 7 could be introduced more directly with \n\n$$P(w_i|x)=P(w_i,x\\in S_{ID}|x) = P(w_i|x\\in S_{ID},x) \\cdot P(x\\in S_{ID}|x), i=1,\\dots,M$$\n\nb. The analysis of prior methods seems not necessarily dependent on a probabilistic perspective, since $P(x\\in S_{ID}|x)$ is essentially a rephrasing of existing methods (in A.1).\n\nThus, the emphasis on the probabilistic viewpoint might not be as novel as suggested.\n\n2. **Impact of Confidence Threshold:** How does the confidence threshold, $\\gamma$, in Eq. 9 influence the performance? Could the authors elaborate on how they determined its value during experiments?\n\n3. **Bias in the Proposed Method:** The paper indicates that existing post-hoc methods are influenced by biases from pretrained models. Yet, as Table 3 reveals, the proposed technique doesn't perform optimally on ImageNet-O, which contains adversarial images for ImageNet. Given that the method's training relies on both the original dataset and model's intermediate predictions, is it possible the method still suffers from the bias?\n\n4. **Effect of OOD Training Target on ID ACC:** In the experiments, what is the ID ACC performance when $\\alpha=0$? Essentially, does including the OOD training target lead to a noticeable decrease in ID ACC?\n\n5. **Fairness of Comparison in Experiments:** In the experimentation section, multiple methods such as MSP, ODIN, ReAct, and many others are post-hoc techniques that work off a fixed pre-trained model. These methods don't adjust the training process, distinguishing them from the proposed approach. As a result, juxtaposing these techniques might not offer a balanced comparison. The majority of experiments may not necessarily highlight the superiority of the proposed method. It would be beneficial for the authors to contrast their strategy with other training-based techniques.\n\nMinor Points:\n1. **Table Formatting Issues:** In Tables 4 & 5, the shaded regions seem to obscure the lines, affecting clarity.\n\n2. **Discussion Location:** In the introduction, the authors state that \"various OOD methods can be analyzed, with main differences and key limitations clearly identified\". However, this discussion is relegated to the appendix. It would be more helpful if the main content were self-contained and inclusive of this analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eQWSNjcYUE", "reviewer": "Reviewer_EtXA", "summary": "The text addresses challenges in Out-of-distribution (OOD) detection, emphasizing the need for a unified perspective and the generation of compact boundaries between in-distribution (ID) and OOD data without explicit OOD samples. A general probabilistic framework is proposed to interpret various existing OOD detection methods, providing insights for future research. Concurrently, a Self-Supervised Sampling for OOD Detection (SSOD) model is introduced, which leverages natural OOD signals from ID data through convolution’s local property, allowing for the joint optimization of OOD detection and ID classification in an end-to-end manner. Extensive experiments demonstrate that SSOD achieves state-of-the-art performance on large-scale benchmarks, significantly outperforming previous methods, and showing remarkable results on both standard and challenging OOD datasets.", "questions": "see Weakness", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well-written and easy to understand.\n2. The author conducts extensive experiments for evaluation.\n3. The visualization is helpful to better understand SSOD.", "weaknesses": "1. SSD [1] also uses a self-supervised algorithm for OOD detection; however, the article does not make any comparisons with SSD.\n2. As shown in Tables 2, 4, and 5, SSOD results in a decrease in the model's accuracy for ID classification, which is not permissible for OOD detection. OOD detection requires the model to identify OOD data without affecting the accuracy of ID classification.\n3. The results reported in Table 1 show that SSOD does not achieve state-of-the-art (SOTA) performance on all OOD datasets, and there are many works with better performance not compared in Table 1: React [2], Dice [3], Ash [4].\n4. The motivation is not convincing enough: I do not agree with the author's critique of the two-stage manner approach. Post-processing algorithms are actually more suitable for adapting to various pre-trained models, regardless of whether the model is trained with supervised learning or self-supervised learning.\n5. Table three indicates that SSOD performs poorly on hard OOD detection tasks, does this highlight a flaw in the algorithm: its inability to differentiate between ID and OOD with similar backgrounds? Take the following more realistic example of OOD input: if a network trained to distinguish between different types of apples (e.g., fuji, red, honey-crisp, etc.) is presented with a different fruit as OOD data (e.g., peach, plum, tomato, etc.), the background features could be similar for both ID and OOD data. Alternatively, consider a scenario like scene classification (indoor scene vs beach scene vs forest scene, just as a hypothetical example). In this case, there is no background as such, because the entire scene constitutes the 'foreground.' I am very curious about how SSOD would perform in such scenarios.\n6. I recommend the author conduct experiments about ViT architectures.\n\n[1]  Sehwag, Vikash, Mung Chiang, and Prateek Mittal. \"Ssd: A unified framework for self-supervised outlier detection.\" ICLR 2021.\n\n[2] Sun, Yiyou, Chuan Guo, and Yixuan Li. \"React: Out-of-distribution detection with rectified activations.\" Advances in Neural Information Processing Systems 34 (2021): 144-157.\n\n[3] Sun, Yiyou, and Yixuan Li. \"Dice: Leveraging sparsification for out-of-distribution detection.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n[4] Djurisic, Andrija, et al. \"Extremely Simple Activation Shaping for Out-of-Distribution Detection.\" The Eleventh International Conference on Learning Representations. 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)