The Devil is in the Wrongly-classified Samples: Towards Unified Open-set Recognition
Authors: Jun CEN
Di Luan
Shiwei Zhang
Yixuan Pei
Yingya Zhang
Deli Zhao
Shaojie Shen
Qifeng Chen
Venue: ICLR 2023 poster (2023)

Abstract: Open-set Recognition (OSR) aims to identify test samples whose classes are not seen during the training process. Recently, Unified Open-set Recognition (UOSR) has been proposed to reject not only unknown samples but also known but wrongly classified samples, which tends to be more practical in real-world applications. In this paper, we deeply analyze the UOSR task under different training and evaluation settings to shed light on this promising research direction. For this purpose, we first evaluate the UOSR performance of several OSR methods and show a significant finding that the uncertainty distribution of almost all these methods is actually closer to the expectation of UOSR than OSR. We show that the reason lies in the known but wrongly classified samples, as their uncertainty distribution is extremely close to unknown samples rather than known and correctly classified samples. Second, we analyze how the two training settings of OSR (i.e., pre-training and outlier exposure) influence the UOSR. We find although they are both beneficial for distinguishing known and correctly classified samples from unknown samples, pre-training is also helpful for identifying known but wrongly classified samples while outlier exposure is not. In addition to different training settings, we also formulate a new evaluation setting for UOSR which is called few-shot UOSR, where only one or five samples per unknown class are available during evaluation to help identify unknown samples. We propose FS-KNNS for the few-shot UOSR to achieve state-of-the-art performance under all settings.

Published as a conference paper at ICLR 2023
THE DEVIL IS IN THE WRONGLY-CLASSIIED SAM- PLES: TOWARDS UNIFIED OPEN-SET RECOGNITION
Jun Cen'r, Di Luan': Shiwei Zhang?, Yixuan Pei",
Yingya Zhang', Deli Zhao?, Shaojie Shen', Qifeng Chen'
'Cheng Kar-Shun Robotics Institute, The Hong Kong University of Science and Technology AAlibaba Group 'Xi'an an Jiaotong University
(Jcenaa, d wusoonnsstustttt.. hk, {zhangiin. 2aw, yingya. del1 11111.200 2dl))alllllll lbaba inc. com, geiyixuanestin stu. x ett..eda.co {eeshaojle, eqt .ustt.hk
ABSTRACT
Open-set Recognition (OSR) aims to identify test samples whose classes are not seen during the training process. Recently, Unified Open-set Recognition (UOSR) has been proposed to reject not only unknown samples but also known but wrongly classified samples, which tends to be more practical in real-world applications. In this paper, we deeply analyze the UOSR task under different training and evalua- tion settings to shed light this promising research direction. For this puppose, we first evaluate the UOSR performance of several OSR methods and show a significant finding that the UOSR performance consistently surpasses the OSR performance by large margin for the same method. We show that the reason lies in the known but wrongly classified samples, as their uncertainty distribution is extremely close to unknown samples rather than known and correctly classi- fied samples. Second, we analyze how the two training settings of OSR (i.e. pre-training and outlier exposure) influence the UOSR. We find although they are both beneficial for distinguishing known and correctly classified samples from unknown samples, pre-training is also helpful for identifying known but wrongly classified samples while outlier exposure not. In addition to different training settings, also formulate new evaluation setting for UOSR which is called few-shot UOSR, where only one or five samples per unknown class are available during evaluation to help identify unknown samples. We propose FS-KNNS for ttt-hh-rtt art performance under all settings.
the few-shot UOSR to achieve state-of ofite--f-the-ass
INTRODUCTION
Neural networks have achieved tremendous success in the closed-set classification (Deng et al., 2009), where the test samples share the same In-Distribution (InD) class set with training samples. Open-Set Recognition (OSR) (Scheirer et al., 2013) is proposed tackle the challenge that some samples whose classes are not seen during training, which are Out-of-Distribution (OoD) data, may occur in the real world applications and should be rejected. However, that the model should not only reject OoD) samples but also InD samples that are Wrongly classified (InW), as the model gives the wrong answers for both of them. So Unified Open-set Recognition (UOSR) is proposed to only accept InD samples that are correctly classified (InC) and reject OoD and InW samples (Kim et al., 2021) simultaneously. The difference between the UOSR and OSR lies in the InW samples. where OSR is supposed to accept them while UOSR has the opposite purpose. Actually, UOSR more useful in most real-world applications, but it receives little attention from the research community as it has been proposed very recently and lacks comprehensive systematic research. Therefore, we deeply analyze the UOSR problem in this work to fill this gap.
researchers have argued
We first apply existing OSR methods for UOSR in Sec. 3, and then analyze UOSR under different training settings and evaluation settings in Sec. 4 and Sec. 5 respectively. In Sec. 3, several existing OSR methods are applied for UOSR, and we find that the UOSR performance is consistently and significantly better than the OSR performance for the same method, as shown in Fig. (a). We show
"qqull contribution. Code; httpa //gs wthub.com. CRN/UI (ed___pppp Set, Set__eooonnteeeeeeeeeeeeeee

---

Published as a conference paper at ICLR 2023
90
OSR
97
(Ours)
(Sec. 5)
86.69
VS wi DI:
64.9
05.33
94
(Sec. 4)
TP wd DI
84.16
AUROC
91.63
AUROC
91
77.1
(Sec.
76.37
76.23
75.59
PROSER
MCD
SoftMax
Vim
38 KNN
OpenMax
DF
55D
EB
SIRC
SoftMax ODIN
LC
OpenMax
OLTR
225
245
265
285
305
325
345
365
385
Methods (a)
AURC
(b)
Figure I: (a) shows that the UOSR performance is significantly better than OSR performance for the same method, which illustrates the uncertainty distribution of these OSR methods is actually
closer to the expectation of UOSR than OSR. (b) shows the UOSR performance under different settings and the skeleton of this paper. Results are based on the ResNer50 backbone. CIFARIO and ImayeNteeNet are InD and OoD datasets, respectively. (TS: Train from Scratch. TP: Train from Fre-training. OE: Outlier Exposure. FS: Few-shot.)
that this phenomenon holds for different network architectures, datasets, and domains (image and video recognition). We find the devil is in the InW samples that huve similar uncertainty distribution with OoD samples rather thun InC samples. Therefore, the false positive predictions in OSR tend to be InW samples, which extremely important but dismissed by all existing OSR works.
In Sec. 4, we introduce two training settings into UOSR, including pre-training (Hendrycks et al., 2019a) and outlier exposure (Hendrycks et al., 2019b; Yu & Aizawa, 2019; Thulasidasan et al., 2021), they both helpers of the OSR that introduce extra information beyond the training set. Pre-training is to use the weights that are trained on a large-scale dataset for better down-task performance, and outlier exposure is to introduce some background data without labels into training to help the model classify InD and OoD samples. We find both of them have better performance for InC/OoD discrimination, which explains why they are beneficial for OSR. However, pre-training is also helpful for InC/InW discrimination. while outlier exposure has a comparable or even worse performance distinguish InC and InW samples. The performance of UOSR can be regarded as the comprehensive results of InC/OoD) and InC/InW discrimination so that both techniques can boost the performance of UOSR. We build up a comprehensive UOSR benchmark that involves both pre- training and outlier exposure settings, as shown in Fig. (b).
In addition to the two aforementioned training settings, we introduce a new evaluation setting into UOSR in Sec. 5. We formulate the few-shot UOSR, similar to SSD (Sehwag et al., 2022) that pro- poses few-shot OSR, where or 5 samples per OoD class are introduced for reference to better iden- tify OoD samples. We first develop KNN-based baseline (Sun et al., 2022) KNN for the few- shot UOSR. Although InC/OoD discrimination is improved due to the introduced OoD reference samples, the InC/InW discrimination is severely harmed compared to SoftMax baseline (Hendrycks & Gimpel, 2017). To alleviate this problem, we propose FS-KNNS that dynamically fuses the FS-KNN with SoftMax uncertainty scores to keep high InC/InW and InC/OoD performance si- multaneously. Our FS-KNNS achieves state-of-the-art performance under all settings in the UOSR benchmark, shown in Fig. (b), even without outlier exposure during training. Note that InC/OoD performances are comparable between FS- FS-KNNS and FS- IN-K,N but their distinct InC/InW perfor- mances makes FS-KNN better at OSR and FS-KNNS better at UOSR, which illustrates the differ- ence between few-shot OSR and UOSR and the importance of InW samples during evaluation.
TOWARDS UNIFIED OPEN-SET RECOGNITION
In this section, we first formalize the UOSR problem and then discuss the relation between UOSR and other uncertainty-relate tasks.
Unified Open-set Recognition. Suppose the training dataset D Dirain = {(x, y:) y C X x C, where X' refers to the input space, e.g., images videos, and C refers the Inl sets. In closed-set
2

---

Published as a conference paper at ICLR 2023
recognition, all test samples come from the InD sets, i.e., Didgred = {(x, 1:) C X x C. In OSR and UOSR, the test samples may come from OoD sets U which are not overlap with InD sets C, so
MM C A' xl. The InD test samples we have Dape = test test where Dusknown = [(x,, yi) y:)
Dested can be divided into two splits based on whether the sample is correctly classified or wrongly classified, i.e., Delased Dilgued-e U Delssedww where Delasede {(x, 3/) 0 ys ) Pest {(x;, Mb lis * yi i= and ÿi refers to the model classification results of sample x,. The goal of UOSR is to reject W and Ool samples and accept InC. samples, so the ground truth uncertainty Delueed- and fest Dusknown is I while for Dissred it is O, as shown in Table 1. The key of UOSR is how to estimate the uncertainty to be close to the ground truth uncertainty u.
of fest
The UOSR is proposed by (Kim et al., 2021) very recently, so it has not attracted many researchers to this problem yet. SIRC (Xia & Bouganis, 2022) augments SoftMax (Hendrycks & Gimpel, 2017) baseline for the better UOSR performance. Build upon these existing methods, we make a deep analysis of UOSR in this work. In Table 1, we compare different settings of the related un- certainty estimation tasks, and the detailed dis-
Table I: Comparison of uncertainty-related task settings. Cls: Classification. 0 and I refer to the corresponding ground truth uncertainty u, and is not fixed in MC.
InC InW OoD Ordinal Rank InD Cls
SP
0
X
AD/OD
x
OSR
0
UOSR
0
cussions are as follows.
MC
X X
Selective Prediction (SP). Apart from the classical classification, SP also tries to estimate which sample is wrongly classified (Corbière et al., 2019; Moon et al., 2020; Granese et al., 2021), so the ground truth uncertainty of Wn is SP constrained under the closed-set scenario and does not consider OoD samples during evaluation, as shown in Table 1, which is the key difference with UOSR. We involve some SP methods in the UOSR benchmark in Sec. 4.
Anomaly/Outlier Detection (AD/OD). AD is to detect anomaly patches within an image or anomaly events within a video (Deng & Li, 2022; Zaheer et al., 2022). OD regards a whole dataset as InD and samples from other datasets as OoD (Chauban et al., 2022; Goodge et al., 2022). Both AD and OD do not require InD classification, so there is no InC/InW discrimination problem.
Open-set Recognition (OSR) and OSR and OoDD are same, but their datasets might be different. Both of them aim to accept all InD samples matter they are InC or InW samples, and reject OoD samples. OSR divides one dataset into two splits and uses one of them as InD data to train the model, while another split is regarded as OoD samples (Scheirer et al., 2013). In contrast, OoDD uses a whole dataset as InD data and regards another dataset as OoD data (Hendrycks & Gimpel, 2017). However, in the recent works about OSR in the video domain (Bao al., 2021), utilizes the OoDD setting rather than the OSR setting. In this work, we use OSR to represent the task setting and use one dataset as InD data and another dataset as OoD data. As mentioned before, the distinction between UOSR and OSR is InW samples, where OSR aims to accept them, and UOSR aims to reject them, so the ground truth uncertainty of InW samples is O in OSR and I in UOSR, as shown in Table 1. Better InC/OoD performance is beneficial for both UOSR and OSR, but higher InC/InW discrimination is preferred by UOSR but not wanted by OSR.
Detection (OoDD). The task settings of
Model Calibration (MC). All tasks mentioned above solve the uncertainty ordinal ranking problem, i.e., the ground truth uncertainty of each type of data is fixed, and the performance will be better if the estimated uncertainty is closer to the ground truth uncertainty (Geifman al., 2019). In contrast, MC uses uncertainty to measure the probability of correctness (Guo et al., 2017). For a perfect calibrated model, if we consider all samples with confidence of 0.8, then 80% of samples are correctly classified. In this case, 20% InW samples with a confidence of 0.8 are perfect, but in other seeertainty-relatee tasks, Wn samples are supposed to have confidence of either I or O. MC also does not consider OoD samples. See Appendix A for more relation between UOSR and MC.
3 OSR APPROACHES FOR UOSR
In this section, we evaluate the existing OSR approaches for the UOSR problem, and show that the InW samples play a crucial role when evaluating the uncertainty quality. Specifically, simply changing the ground truth uncertainty u of InW samples from 0 to I can bring large performance
3

---

Published as a conference paper at ICLR 2023
Table Uncertainty distribution analysis in im- age domain with ResNet50. OoD dataset: Tiny- ImageNet. AUORC (%) ) reported.
Uncertainty distribution analysis in video domain with TSM backbone. OoD dataset is HMDB51. AUORC (%) is reported.
Table 3:
Methods InC/OoD W InW/OoD OSR UOSR
Methods InC/OoDInC//mi InW/OoD OSR UOSR
SoftMax
84.69
85.68
50.64
75.59
84.90
OpenMax
88.56
82.53
64.47
82.27 85.95
ODIN
88.35
80.76
64.36
81.65
86.69
Dropout
85,36
85.86
48.92
75.75
85.58
LC
84.60
82.58
55.14
76.37
84.16
BNN SVI
89.93
88,85
55.44
80.10
89.44
OpenMax
86.16
85,96
51.27
76.23
85.33
SoftMax
88.73
88.01
54.19
79.72
88.42
OLTR
85.99
85.74
52.22
77.10
85.94
RPL
89.22
88.06
55.74
79.67
88.70
PROSER
87.04
77.84
62.57
79.23
84.82
DEAR
89.33
85.47
56.78
80.00
87.56
Nes LSON
(84,62,88.0)
HMDRSI MITV2
(79.48, 59.64)
(82.58, 87 46)
(94.42, 58.74)
(77.10, 85.94)
88
(79.23, 84 82))
+1.88
(75 (25.66, 85.76)
(82.27, 85.951
+4.32
AUROC-UOSR
AUROC-UOSR
+3,68
+8.84
+5.59
82
SoftMox OCIN
DEAR
SoftMax
OpenMax
OpennMaxx OLTR
RPL
MC Dropout
75
81
84
87
90
76
79
82
91
AUROC OSR
AUROC OSR (b)
Figure 2: (a) and (b) show the relation between UOSR and OSR performance in the image and video domain under ResNet50 and TSM backbones. Different color indicates different OoD datasets. The red-dotted diagonal where UOSR has the same AUROC as OSR. Green arrows show the performance gap between UOSR and OSR for the same method.
boost, as shown in Fig. 2. Then we provide the comprehensive experiment results and discussion of this phenomenon.
Applied Methods. We reproduce several classical OSR methods and evaluate their UOSR and OSR performance in our experiments, including SoftMax (Hendrycks Gimpel, 2017), ODIN (Liang et al. 2018), LC (DeVries & Taylor, 2018), OpenMax (Bendale & Boult, 2016), OLTR (Liu et al., 2019) and PROSER (Zhou et al., 2021) in the image domain, as well as DEAR (Bao et al., 2021), RPL (Chen al., 2020), MC Dropout (Gal & Ghahramani, 2016) and BNN SVI (Krishnan al., 2018) in the video domain.
Datasets. In the image domain, the InD dataset is CIFAR100 (Krizhevsky al., 2009) and the OoD datasets are TinyImageNet (Le & Yang, 2015) and LSUN (Yu et al., 2015). In the video domain, we follow (Bao et al., 2021) and adopt UCF101 (Soomro et al., 2012) as InD dataset, while the Ool) datasets are HMDB51 (Kuehne et al., 2011) 1) and MiTv2 (Monfort et al., 2021).
Experiments settings. We train the network using the InD dataset and evaluate the UOSR and OSR performance based on the ground truth in Table 1. The evaluation metric is AUORC (Hendrycks & Gimpel, 2017) which is a threshold-free value. The AUROC reflects the distinction quality of two uncertainty distributions. We adopt VGG13 (Simonyan & Zisserman, 2015) and ResNet50 (He et al., 2016) as the network backbone in the image domain and TSM (Lin et al., 2019) and I3D (Carreira & Zisserman, 2017) in the video domain. More experiment settings are in Appendix E.
Results. We provide the UOSR and OSR performance of different methods in Fig. 2. shows that all data points are above the red dotted diagonal, illustrating that the UOSR performance is significantly better than the OSR performance of the same method, and this relationship holds across different datasets, domains (image and video), and network architectures (See Appendix C for results of more network backbones). The performance gap between UOSR and OSR of the same method can be very large, such 8.84% for OLTR when TinylmageNet is the OoD dataset, and 10.16% for BNN SVI when the OoD dataset is MiTv2. Therefore, existing OSR methods have uncertainty distributions that actually closer the expectation of UOSR than OSR.

---

Published as a conference paper at ICLR 2023
InC
InC
InC
Inc
Inia
InW
1WW
Inil
10-
0:0
G=O
Dv0
DeD
8-
uncetaiiity
6X)
(h)
69)
68)
Figure 3: (a) and (b) are the SoftMax and ODIN methods in the image domain, while (c) and (d) are the SoftMax and DEAR methods the video domain. OoD datasets are TinyImageNet for the
image domain and HMDB51 for the video domain.
Analysis. To better understand our findings, we provide detailed analysis of the uncertainty dis- tribution relationships between InC, InW, and OoD samples in Tables and 3. Note that higher AUROC means a better distinction between two uncertainty distributions, and AUROC=50% means two distributions overlap with each other. From Tables and we can clearly see that AUROC of InC/OoD and InC/InW are significantly higher than InW/OoD, and AUROC of InW/OoD is very close to 50%. Therefore, the uncertainty distribution of InC samples is distinguishable from OoD and InW samples, and there is a lot of overlap between the uncertainty distributions of InW and OoD samples. Several uncertainty distribution visualizations are in Fig. 3, where we can see that InW and OoD samples share a similar uncertainty distribution, while InC samples have smaller uncertainty.
Importance. Based on the above analysis, we conclude that the false positive predictions in OSR tend to be InW samples, since they share similar uncertainty distributions with Ool) samples. This conclusion is extremely important as InW samples significantly deteriorate the OSR performance. Without InW samples, the OSR performance increases by large margin (75.59 to 84.69 for Soft- Max method in Table 2). However, no existing OSR works mentioned and considered this phe- nomenon. We explicitly point out this conclusion and hope the following researchers take into account when they design new OSR or UOSR methods.
Why? To deeply understand why InW samples share similar uncertainty distribution with OoD samples instead of InC samples, we begin by analyzing the feature distributions of InC/InW/OoD samples. We first find that the features of InC/InW/OoD follow the hierarchy structure where InC/InW/OoD samples are gradually far away from the training samples, so their features separa- ble. Please see Fig. 10 and l for reference. Then we calculate the feature similarity and surprisingly find that InW features are more similar to InC features rather than OoD features in Table 9, which contradicts the uncertainty score phenomenon. Therefore, the reason that Wn samples have similar uncertainty scores with OoD samples is not they have similar features, but lies in the uncertainty estimation methods. We provide explanations about how the uncertainty estimation process causes the contradictory phenomenon in Appendix D.
A better closed-set model is better for UOSR. Recently, (Vaze et al., 2022) found that better closed-set performance means better OSR per- formance. We provide a deeper explanation of this finding and show that this conclusion also holds for UOSR. Table 4 shows that we improve the closed-set accuracy through data augmenta- tion and longer training (Vaze et al., 2022). The
Table 4: Relation between closed-set accuracy Acc. (%) and open-set performance, Aug: Aug- mentation; Ep: Epoch. AUROC (%) is reported.
Aug. Ep. Acc. InC/OoD InC/InW OSR UOSR
100 59.41
81.65
82.47
68.64 81.89
100 68.83
84.18
85.13
73.71
84.40
300 73.28
84.69
85.68
75.59
84,90
open-set method is the SoftMax baseline with ResNet50 backbone, and the OoD dataset is Tiny- ImageNet. For OSR, we can see that AUROC of InC/OoD is significantly better than OSR, which indicates the uncertainty distribution of InW samples are contradictory with the expectation of OSR. So less InW samples and better InC/OoD performance are two reasons for better OSR performance when closed-set accuracy is higher. For UOSR, both the InC/InW and InC/OoD performance are improving with the growth of closed-set accuracy, which brings better UOSR performance.
4 PRE-TRAINING AND OUTLIER EXPOSURE
After directly applying existing OSR methods for UOSR in Sec. 3, we explore two additional training settings in this section, including pre-training (Hendrycks et al., 2019a) and outlier
5

---

Published as a conference paper at ICLR 2023
Table 5: UOSR benchmark in the image domain under the ResNet50 model. InD dataset is CI- FAR100 while the OoD dataset is TinylmageNet. f. refer to OSR-based SP-based, UOSR- based methods. OD: Outlier Data. N/G/R means No/Generated/Real OD. AUORC (%), AURC
(x10") and Acc. (%) are reported.
w/o pre-training
w/ pre-training
UOSR
OSR
UOSR
OSR
Methods
OD
Acc.t
AURCL
AUROCT
AUROCT
Acc.t
AURCI
AUROC1
AUROCT
ODIN
N
72.08
371.92
86.69
81.65
86.48
261.42
91.81
91.47
LC
N
72.08
372.55
84.16
76.37
86.48
280.69
87.68
85.
OpenMax'
73.64
361.22
85.33
76.23
86.43
261.53
90.29
85.86
Moxioogits
N 73.89
351.39
87.09
80.11
86.42
249.86
92.91
91.11
Entropy!
N
73.89
355.54
86.25
78.14
86.43
257.36
91.21
87.22
OLTR
N 73.69
357.79
85.94
77.10
86.22
260.17
90.59
86.05
Ensemble
N
76.78
314.56
89.59
83.93
87.41
240.34
93.67
92.02
Vim
73.89
341.88
87.37
80.92
86.43
246.59
93.01
92.33
BCE!
N
73.29
369.91
84.29
74.59
86.39
257.74
90.79
86.50
TCP
N
71,80
369.17
85.17
75.46
86.83
261.97
89.95
85.67
DOCTOR*
N
72.08
378.45
84.48
75,06
86.48
262.0S
90.24
85.75
SIRCOMP, ||=||
N
73.13
358.93
85.77
76.67
86.44
260.08
90.53
85.98
SIRC(MSP, Res.)
73.13
348 40
87.26
80.29
86.44
247.75
92,99
90.39
SIRC(-HA ||
N
73.13
355.62
86.57
78.16
86.44
257.28
91.23
87.27
SIRC(-H. H. Res.)
N
73.13
346.21
87.71
81.43
86.44
244.55
93.68
91.62
SoftMax
N
73.28
363.55
84,90
75.59
86.44
260. 14
90.50
85.93
SofIMax (OE!)
73.54
339.59
88.78
84.35
85.43
255.77
92.54
90.65
ARPLI
N
73.03
345.84
88.13
80.49
84.67
301.27
87.01
83.49
ARPL+CS
R
72.78
349.50
87.65
82.81
83.60
268.00
92.00
91.17
MCD (Dropout)
N
76.49
375.01
82.25
79.21
87.21
301.71
83.57
81.69
MCD
70.88
365.82
86.97
79.88
81.96
287.21
90.43
86.59
PROSER
68.08
394.48
84.82
79.23
81.32
301.78
90.20
89.29
PROSER (EB')
71.82
366.65
86.06
81,95
85.06
269.79
90.84
90.38
VOS
G
73.44
356.68
86.65
79.72
86.62
249.24
92.94
91.37
VOS
73.18
331.12
89.96
85.78
85.93
251.44
92.92
91.34
OpenGAN
R
73.61
334.04
88.92
85.48
86.25
260.19
91.21
90.94
posure (Hendrycks al., 2019b), which are effective methods improve the OSR performance because of introduced extra information beyond the training set. Pre-training to use large-scale pre-trained weights for initialization for better down-task performance. Outlier exposure is to intro- duce some unlabeled outlier data (OD) during training and regard these outlier data as the proxy as OoD data to improve the open-set performance. We find both of them also have positive effect on UOSR, but for different reasons.
Pre-training settings. In the image domain, 2020) for ResNet50 and ImageNet (Deng et al., 2009) pre-training weights for VGG13, 13. In the video domain, Kinetics400 (Carreira & Zisserman, 2017) pre-trained weights are used for initialization.
BiT pre-training weights (Kolesnikov al.,
Outlier exposure settings. In the image domain, we use 300K Random Images dataset (Hendrycks al., 2019b) outlier data. the video domain, outlier data comes from Kinetics400, and filter out the overlapping classes with InD and OoD datasets. We implement several outlier exposure based methods, including OE (Hendrycks et al., 2019b), EB (Thulasidasan et al., 2021), VOS (Du et al., 2022) and MCD (Yu& & Aizawa, 2019).
UOSR benchmark settings. We include OSRebased, SP-based, and UOSR-based methods to build comprehensive UOSR benchmark. We implement several Selective Prediction (SP) based meth- ods, including BCE, TCP (Corbière et al., 2019), CRL (Moon et al., 2020), and DOCTOR (Granese et al., 202 1). Although these SP methods are originally designed to differentiate InC and InW sam- ples, we adopt them in the UOSR setting to build a more comprehensive benchmark. The only UOSR-based method is SIRC (Xia & Bouganis, 2022), which combines two uncertainty scores for better UOSR performance. The evaluation metrics are AURC and AUROC (Kim al., 2021). More details can be found in Appendix E.

---

Published as a conference paper at ICLR 2023
Table 6: UOSR benchmark in the video domain under the TSM model. InD dataset is UCFI01
while the OoD dataset is HMDB51. 4, 4,4 O refer to based SP-based, UOSR-basee methods.
OD: Outlier Data. AUORC (%), AURC (x10²) and Acc. (%) are reported.
w/o pre-training
w/ pre-training
UOSR
OSR
UOSR
OSR
Methods
OD
Ace.t
AURCI
AUROCT
AUROCT
Acc.
AURCI
AUROCT
AUROCT
OpenMax
73.92
185.81
85.95
82.27
95.32
75.75
91.22
90.89
BNN SVIt
N
71.51
181.45
89.44
80.10
94.71
69.89
93.58
91.81
RPL
71.46
186.18
88.70
79.67
95.59
72.88
92.44
90.53
DEAR
N
71.33
215.80
87.56
80.00
94.41
102.01
91.50
91.49
BCE
69.90
223.57
83.27
78.96
93.66
110.42
83.83
81.64
CRL
N
71.80
183.76
88.75
78.57
95.22
67.61
93.36
91.38
DOCTOR
72.01
182.04
88.73
79.76
95.06
65.61
93.89
91.80
SIRCMSP ||= )
N
73.59
173.32
88.71
80.33
95.00
65.97
93.74
91.42
SIRCMMSP, R
73.59
174.76
88.17
78.74
95.00
65.43
93.83
91.73
SIRC( H. ||=||]
73.59
172.30
89.27
81.61
95.00
66.11
94.06
91.95
SIRC(-H, H. R
N
73.59
175.79
88,50
79.62
95.00
69.06
94.07
92.18
SoftMax SoftMax (OE!)
73.92
173.14
88,42
79.72
95.03
68.08
93.94
91.75
74.42
162.36
90.19
86.29
94.71
67.93
94.33
93.40
MCD (Dropout
N
73.63
184.66
85.58
75.75
95.06
79.53
90.30
88,23
MCD
R
72.49
168.83
91,26
85.57
93.47
71.19
95.34
93,68
VOS
74.00
187.82
86.10
84.51
95.27
65.68
94.44
93.62
VOS
R
74.68
172.71
87.98
87.09
94.79
64.99
94.97
93.72
EB
70.90
212.01
85.32
86.47
94.66
67.83
94.40
93.06
100
Entropy
Y OE
* BCE
TS
SoftMax
1 ARPL_cs
TCP
TP
OpenMax
96-
RPL
vos
SIRC_1
97
vos
SIRC_1
TP OF
MCD
SIRC_2
OLTR
+ MCD
SIRC,2
AUROC-InC/OuD
94
AUROC-InCIos0
93
90-
88
85
87:
14
TP
TS OE
82
80
83
86 89
92
78
81
84
87
90
93
96
AUROC-InC/nW (a)
AUROC-InCAnW (b)
Figure 4: (a) and (b) plot the InC/InW and InC/OoD discrimination in the image and video domain. We set the SoftMax method training from scratch as the original point and divide the coordinate system into 4 quadrants (Q1 to Q4). (TS: Train from Scratch. TP: Train from Pre-training. OF: Outlier Exposure.)
Results. The UOSR benchmarks for image and video domains are Table 5 and 6 respectively. See Appendix for the results of different model architectures. First, we can see the AUROC of UOSR is higher than OSR for almost all the methods under both settings, i.e., pre-training and outlier exposure, which further strengthens the conclusion that the uncertainty distribution of OSR methods is closer to the ground truth of UOSR. For instance, AUROC is 89.59 for UOSR and 83.93 for OSR under the Ensemble method w/o pre- pre-training Table 5. Second, pre-training and outlier exposure can effectively boost the UOSR and OSR performance e.g., the pre-training boost the AUROC of UOSR/OSR from 84, 90/75.59 to 90.50/85.92 90.19/86.29 29 AUROC of UOSR/OSR compared to 84.90/75.59 of SoftMax in Table 5. Note that to ensure outlier data is useful, we keep methods w/ and w/o outlier data as similar as possible, like SoftMax/OE, ARPL/ARPL+CG generated outlier data in the UOSR and OSR tasks. The UOSR/OSR AUROC of VOS method is 89.96/85.78 78 for real outlier data and 86.65/79.72 for generated outlier data, provided in Table 5.
the SoftMax, and outlier exposure method OE has
and Dropout/MCD. Third, real outlier data is more beneficial than
Analysis. To better understand why pre- pre-training and outlier exposure are helpful for UOSR and OSR, provide the InC/InW and InC/OoD discrimination performance of each method in Fig. 4.
7

---

Published as a conference paper at ICLR 2023
Table 7: Results of few-shot UOSR in the image domain. Model is ResNet50 with pre-training. InD and OoD datasets are CIFAR 100 and TinyImageNet. AUORC (%) and AURC (x109) are reported.
5-shot
1-shot
AURCI
AUROCT
AURCI
AUROCT
Methods
UOSR
UOSR OSR
InC/OoD InC/InW
UOSR
UOSR OSR
InC/OoD InC/InW
SoftMax
260.14
90.51
85.93
90.64 94.86
89.58
260.14
90.51
85.93
90.64 94.86
89.58
KNN
245.48
93.45
92.34
83.08
245.48
93.45
92.34
83.08
FS-KNN
238.54
95.09
95.91
97.20
79.58
239.71
94.67
94.91
96.52
81.04
SSD
246.42
94.95
97.89
98.32
70.14
245.99
94.16
96.51
97.16
72.09
FS- DS-KNN+S
239.49
94.14
91.64
95.27
85.85
240.30
93.97
91.36
94.99
86.45
FS. KNN*S
255.82
91.26
87.10
91.47
89.67
255.69
91.30
87.16
91.51
89.69
SIRC
241.58
93.85
91.42
94.44
89.58
239.64
94.20
92.03
94.87
89.26
FS-KNNS
231.61 95.51
94.16
96.54
87.98
234.84
94.91
93.10
95.84
88.08
We can that almost all outlier exposure methods are in the Q2, which means that outlier exposure methods have lower InC/InW AUROC and higher InC/OoD AUROC than the SoftMax baseline. For OSR, both lower InC/InW and higher InC/OoD AUROC are beneficial, but only higher InC/OoD AUROC is wanted for UOSR, while lower InC/InW AUROC is not. This can explain why some of the UOSR performances with outlier exposure are comparable or even worse than the baseline, such as EB and VOS in the video domain Table 6, but all of the OSR performances are increased. In contrast, pre-training helpful for both InC/InW and InC/OoD AUROC, as methods with pre- training are in the Q1. Outlier exposure methods also benefit from pre-training, green marks are at the upper right location compared to orange marks. Therefore, outlier exposure and pre-training bring the UOSR performance gain for different reasons. InC/OoD performance is improved by both of the techniques, but pre-training is also helpful for InC/InW, while outlier exposure is not. This can be explained by the fact that the model may see additional InD samples during pre-training so that the closed-set set accuracy and InC/InW discrimination are improved. In contrast, outlier exposure only provides OoD data but more InD data, only InC/OoD performance is improved.
FEW-SHOT UNIFIED OPEN-SET RECOGNITION
In addition to the analysis of two useful training settings in Sec. 4, we introduce a new evaluation setting into UOSR in this section. Inspired by the recent work SSD (Sehwag et al., 2022) which proposes the few-shot OSR, we introduce the few-shot UOSR, where or 5 samples per OoD class can be introduced for reference to better distinguish OoD samples. The introduced reference OoD datasets marked as D {x),, } C X x U. We show that few-shot UOSR has distinct challenges from few-shot OSR because of InW samples, and we propose our FS-KNNS method to achieves the state-of-the-ard UOSR performance even without outlier exposure during training. Baselines and dataset settings. SSD is the only existing few-shot OSR method that utilizes the Ma- halanobis distance with InD training samples Drrain and OoD reference samples Dieff KNN (Sun et al., 2022) utilizes the feature distance with samples in Dirain as uncertainty scores and shows it is better than Mahalanobis distance. So we slightly modify KNN to adapt to the few-shot UOSR and find the modified FS-KNN can already beat SSD. Dref comes from the validation set of Tiny geNet and training set of UCF101 in the image and video domain. We repeat the evaluation process until all OoD reference samples are used and calculate the mean as the final few-shot result.
FS-KNN, Given test sample x" and feature extractor f. the feature of x" is z- = f(x)). The cosine similarity set between z and the feature set of Derain Strain = {d: (d di (z* 212((11="| [|z,]]), z, = f(x,),x, x, E Derain- Stee is similar with Strain except x; comes from Dref We believe the uncertainty should be higher if the test sample is more similar to reference OoD samples and not similar with InD samples. Therefore, the uncertainty score is
0f knn = topK (Strain) + topK (Ste().
where topk means the Th largest value. The performances of SoftMax, KNN, ES-KNN, and SSD are in Table 7 and 19. Pre-training is used for better performance. FS-KNN has better InC/OoD performance than KNN as reference OoD samples are introduced. Although the overall UOSR per- formance of FS-KNN is better than the MoftMax baseline, the InC/InW performance is significantly
8

---

Published as a conference paper at ICLR 2023
urceewary 15-0h
Ina
0e0
0.4
0.2
uncormiityy Softhaar 20
0.0
02
0.6
0.6
2.6
2.5
0.0
0.
0.6
2.6
0.8
1.5
0.5
0.
0A
0.6
0.6
1.0
(e) Jont uncertaary disribution
(b) SotbMas incermiity diaribution
(4) KNN unceriallly diszituriion
(d) FS KNNS unceriaity ditribution
Figure 5: Uncertainty scores of each test sample (a) and uncertainty distribution of SoftMax (b),
FS-KNN (c), and FS-KNNS (d).
sacrificed, which is also important aspect of UOSR. For example, InC/InW performance drops from 89.58 to 79.58 in the 5-shot results of Table 7. So we naturally ask question: Can we improve the InC/OoD performance based on the introduced OoD reference samples while keeping similar InC/InW performance with SoftMax baseline? This is the key difference between few-shot OSR and few-shot UOSR, as low InC/InW performance is wanted in OSR but not preferred in UOSR. FS-KNNS. Inspired by SIRC (Xia & Bouganis, 2022), we aim to find a way to fuse SoftMax and FS-KNN scores that the mixed score can keep the high InC/OoD performance of FS-KNN and meanwhile has the comparable InC/InW performance with SoftMax. The uncertainty distributions of SoftMax and FS- FN-K are depicted in Fig. 5. We find that OoD samples have larger uncertainty than InW and InC samples in the FS-KNN method, but the uncertainty of InC samples overlaps a lot with WnW samples, which explains the reason that InC/OoD performance is high but InC/InW performance is low. In contrast, InW and OoD samples share similar uncertainty in the SoftMax method, which brings higher InC/InW performance. Therefore, we want to keep the uncertainty of InW samples in the SoftMax method, well as the uncertainty of OoD samples in the FS-KNN method. this way, the mixed scores obtain the high InC/OoD performance from FS-KNN while keeping the comparable InC/InW performance of SoftMax. We call this method FS-KNN with SoftMax (FS-K (ES-K,NS), and the uncertainty is
úfs ifs-knns ui 1 e-te-aluee ae-aau--eenna 01
where ug and i1 refer to the uncertainty score of SoftMax and FS-KNN, respectively. Aknne is a threshold to determine when the weight of un becomes large, and is coefficient to control the change rate of the weight. ūfs-knns will be largely influenced by on when U1 > Agnnss A proper Akens should be located between the InW and OoD samples, as shown in Fig. 5 (a) and (c). In this way, the uncertainty of InW samples is mainly controlled by SoftMax, and OoD samples are strengthened by FS-KNN. Ablation study about hyperparameters K, ar, and Nknns is in Appendix L.
The uncertainty distribution of FN-K is shown in Fig. 5 (d), which shows the un- certainty of InC and InW samples are similar to SoftMax (b). but the uncertainty of OoD samples is larger. From Table 7 and 19, we can see our FS-KNNS has significantly better InC/InW perfor- mance than FS-KNN (87.98 and 79.58 under ResNet50), and meanwhile keeps the high InC/OoD performance, so the overall UOSR performance is better than both SoftMax and FS- KNN.. We also try three score fusion methods, including FS-KNN+S, FS-KNN and SIRC, but these methods are general score fusion methods, while our method is specifically designed for UOSR, so our FS- KNNS surpass them. Our FS- FS-KNNS is totally based on the existing model trained by the classical cross-entropy loss, so there is no extra effort and outlier data during training, and our FS-KNNS still has better performance than all outlier exposure methods and achieves state-of-the-art UOSR performance, as shown in Fig. (b) and 16. Note that the best choice for OSR (FS-KNN SSD) may not be the best choice for UOSR (FS-KNNS), as their expectation of InC/InW is contradictory, which shows the necessity of our proposed few-shot UOSR.
Discussion.
6 CONCLUSION
Rejecting both In W and OoD samples during evaluation has very practical value in most woal-world applications, but UOSR is a newly proposed problem and lacks comprehensive analysis and re- search, so we look deeper into the UOSR problem in this work. We first demonstrate the uncertainty distribution of almost all existing OSR methods is actually closer to the expectation of UOSR than OSR, and then analyze two training settings including pre-training and outlier exposure. Finally, we introduce a new evaluation setting into UOSR, which is few-shot UOSR, and we propose a FS-KNNS method that achieves state-of-the-art performance under all settings.

---

Published as a conference paper at ICLR 2023
Acknowledgement: This work is supported by Alibaba Group through Alibaba Research Intern Program. Thanks to Xiang Wang and Zhiwu Qing who gave constructive comments on this work.
REFERENCES
Wentao Bao, Qi Yu, and Yu Kong. Evidential deep learning for open set action recognition. In ICCV, 2021.
Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In CVPR, 2016.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, 2017.
Kushal Chauhan, Pradeep Shenoy, Manish Gupta, Devarajan Sridharan, et al. Robust outlier detec- tion by de-biasing vae likelihoods. CVPR, 2022.
Guangyao Chen, Limeng Qiao, Yemin Shi, Peixi Peng, Jia Li, Tiejun Huang, Shiliang Pu, and Yonghong Tian. Learning open set network with discriminative reciprocal points. In ECCV, 2020.
Charles Corbière, Nicolas Thome, Avner Bar-Hen, Matthieu Cord, and Patrick Pérez. Addressing failure prediction by learning model confidence. In NeurIPS, 2019.
Hanqiu Deng and Xingyu Li. Anomaly detection via reverse distillation from one-class embedding. In CVPR, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In CVPR, 2009.
Terrance DeVries and Graham W Taylor. Learning confidence for out-of-distribation detection in neural networks. arXiv preprint /802: 2018.
Xuefeng Du, Zhaoning Wang. Mu Cai, and Yixuan Li. Vos: Learning what you don't know by
virtual outlier synthesis. In ICLR, 2022.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In ICML, 2016.
Yonatan Geifman, Guy Uziel, and Ran El-Yan El-Yaniv. Bias-reduced uncertainty estimation for deep neural classifiers. In ICLR, 2019.
Adam Goodge, Bryan Hooi, See-Kiong Ng. and Wee Siong Ng. Lunar: Unifying local outlier detection methods via graph neural networks. In AAAI, 2022.
Federica Granese, Marco Romanelli, Daniele Gorla, Catuscia Palamidessi, and Pablo Piantanida. Doctor: A simple method for detecting misclassification error.. In NewrIPS, 2021.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. In ICML, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and dutto--dist examples in neural networks. In ICLR, 2017.
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In ICML, 2019a.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. ICLR, 2019b.
Jihyo Kim, Jiin Koo, and Sangheum Hwang. A unified benchmark for the unknown detection capability of deep neural networks. arXiv preprint 12.03721 2021.
10

---

Published as a conference paper at ICLR 2023
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung. Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In ECCV, 2020.
Ranganath Krishnan, Mahesh Subedar, and Omesh Tickoo. Bar: Bayesian activity recognition using variational inference. In NeurIPS, 2018.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: large video datahase for human motion recognition. In ICCV, 2011.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 23/N, 2015.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In ICLR, 2018.
Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. In ICCV, 2019.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X. Yu. Large- scale long-tailed recognition in an open world. In CVPR, 2019.
Mathew Monfort, Bowen Pan, Kandan Ramakrishnan, Alex Andonian, Barry A McNamara, Alex Lascelles, Quanfu Fan, Dan Gutfreund, Rogerio Feris, and Aude Oliva. Multi-moments in time: Learning and interpreting models for multi-action video understanding. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2021.
Jooyoung Moon, Jihyo Kim, Younghak Shin, and Sangheum Hwang. Confidence-aware learning for deep neural networks. In ICML, 2020.
Walter J. Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E. Boult, Toward open recognition. IEEE Transactions Pattern Analysis and Machine Intelligence, 2013.
Vikash Sehwag, Mung Chiang, and Prateek Mittal. Ssd: A unified framework for self-supervised outlier detection. In ICIR, 2022.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. ICLR, 2015.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah, Ucf101: dataset of 101 human actions classes from videos in the wild. arXiv preprint 12Xiv:122.0044 2012.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribttion detection with deep nearest neighbors. ICML, 2022.
Sunil Thulasidasan, Sushil Thapa, Sayera Dhaubhadel, Gopinath Chennupati, Tanmoy Bhat- tacharya, and Jeff Bilmes. An effective baseline for robustness to distributional shift. In ICMLA, 2021.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set classifier is all you need. In ICLR, 2022.
Zhi-Fan Wu, Tong Wei, Jianwen Jiang, Chaojie Mao, Mingqian Tang, and Yu-Feng Li. Nge: unified framework for learning with open-world noisy data. In ICCV, 2021.
Guoxuan Xia and Christos-San Savvas Bouganis. Augmenting softmax information for selective classi- fication with nut-of-distributin data. arXiv preprint arXiv:2207.075506 2022.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint urXiv:1506.03330 2015.
Qing Yu and Kiyoharu Aizawa. Unsupervised out-of-disttibuttin detection by maximum classifier discrepancy. In ICCV, 2019.
11

---

Published as a conference paper at ICLR 2023
Dahua Lin Yue Zhao, Yuanjun Xiong. Mmaction. htttss///giiaubb.cc//ppppmmmmaaa mmaction, 2019.
M Zaigham Zaheer, Arif Mahmood, M Haris Khan, Mattia Segu, Fisher Yu, and Seung-lk Lee. Generative cooperative learning for unsupervised video anomaly detection. In CVPR, 2022.
Da- Wei Zhou, Han-Jia Ye, and De-Chuan Zhan, Learning placeholders for open-set recognition. In CVPR, 2021.
12

---

Published as a conference paper at ICLR 2023
A RELATION BETWEEN MODEL CALIBRATION AND UOSR
Different from Selective Prediction (SP), Anomaly Detection (AD), OSR, and UOSR, Model Cali- bration (MC) is not an uncertainty ordinal ranking problem. In other words, all other settings expect the uncertainty of test sample to be either 0 or 1, while MC is not. A perfect calibrated model should meet (Guo al., 2017)
P P(ー=v\f()=pp> =
vp E [0,1]
(3)
where p is the confidence the negative version of uncertainty p 100 test samples whose confidence scores are all 0.8, then the model is perfectly calibrated if 80% samples are correctly classified. In this case, 20% samples with confidence 0.8 is consistent with the requirement of a perfect calibrated model, but the confidence is supposed to be either 0 or 1 in other settings. The performance of MC evaluated by ECE, and the readers may refer to (Guo et al., 2017) for the formal definition. Smaller ECE means better calibrated model. We provide 5 examples to illustrate the relation between the performance of UOSR and MC in Fig. 6. Note that MC does not consider OoD samples during evaluation, we also do not involve OoD samples in Fig. 6. Therefore, UOSR in Fig. is also equal to SP. From (f) we can see that the performance of UOSR and model calibration is not perfectly positively correlated, i.e., the best case for model calibration (c) is not the best case for UOSR and vice versa (e).
u. For example, given
(UROOS SUUN
64
Aec:
Acc Map
Lapp
Dup
0.5
Tue
oos
E
0.2
60
02
04
266
00
1.
0.6
0.5
26
1.0
00
02
05 04 24 26
00
1.3
0.5 0.5
05
10
(2)
(b)
(C)
(d)
(5)
(0)
Figure 6: We provide 5 samples in (a)-(e), where keep the confidence distribution of InC and change the confidence distribution of InW samples. The evaluation metrics of UOSR are AUROC and AUPR, and ECE is for the MC.
B TEMPERATURE SCALING FOR UOSR
Temperature scaling is convenient and effective method for model calibration (Guo et al., 2017). We study how this method influences the UOSR performance. The experiments are conducted under R50 backbone while InD and OoD datasets CIFAR100 and TinylmageNet, respectively. The quantitative results are Table 8. The uncertainty distribution under different temperatures T are in Fig. 7 and 8.
Table 8: UOSR and MC performance under different temperatures T.
w/o pre-training
w/ pre-training
ECE| AURCI
AUROCT
ECE|
AURCI
AUROCT
T
MC
UOSR
UOSR OSR InC/InW InC/OoD
MC
UOSR UOSR OSR InC/InW InC/OoD
0.1
0.247
355.69
66.58
62.06
66.96
66.48
0.128
257.20
69.07
66.67
68.49
69.15
0.5
0.207
351.39
84.10
74.87
85.01
83.86
0.106
257.26
88.66
83.71
88.61
88.67
1
0.146
358.31
85.57
76.90
85.18
85.67
0,081
260.14
90.51
85,93
89,58
90.64
2 0.119
352.60
86.79
79.19
84.62
87.35
0.018
250.99
92.40
89.09
89.08
92.85
5
0.344
351.45
87.05
79.94
84.05
87.84
0.523
249.00
92.96 90.86
86.00
93.90
0.256
351.39
87.08
80.04
83.94
87.90
0.509
249.55
92.93
91.02
85.22
93.97
20
0.197
351.38
87.09
80.08
83.89
87.92
0.414
249.73
92.92
91.07
84,99
93.99
13

---

Published as a conference paper at ICLR 2023
(A: T40.1
TW0S
60 Tml
Woz
6c1 Tw5
00 T+
Figure 7: Uncertainty distribution under different temperatures T without pre-training.
244
02
Figure 8: Uncertainty distribution under different temperature T with pre-training
From Table 8 we can see that the optimal T for MC 2) not the best case for UOSR. When T grows, the InC/OoD discrimination increases, but the InC/InW discrimination drops. Therefore, the OSR performance keeps improving with larger T, but the UOSR may not benefit from larger T because of lower InC/InW discrimination. For example, the best T for UOSR with pre- training is rather than 20. But in general, temperature scaling a simple and useful technique for both MC and UOSR, as T has better MC and UOSR performance than T = 1 (without temperature scaling). The only drawback of temperature scaling is needs the validation set to determine the optimal T.
C EXISTING OSR METHODS ARE MORE SUITABLE FOR UOSR
In addition to Fig. 2, we provide the results of VGG13 backbone in the image domain and I3D backbone in the video domain. Fig. further proves the finding that existing OSR methods are actually more suitable for UOSR holds for differnt model architectures and domains.
87
TinylmaasNees LSUN
(78.32, $7.13)
HADBSI
90.12)
(81.99
88
(72 (8,
82.81)
+7.29
UUROC-UOSR
+8.33
AUROC-UOSR
85
(75 85, 85.68)
81
+5.09
78
+10.13
+9.83 (78.9), 80
ODIN
79
+1.77
SoltMax
75
LC
OpenMax
OLTR
Dropout
IROSER
76
BNN Svi
72
75
78
81
84
87
76
79
82
85
88
91
AUROC (4)
AUROC- OSR (b)
Figure 9: (a) and (b) are conducted using the VGG13 and I3D backbone in the image and video domain respectively. InD datasets are CIFAR100 and UCF101 for (a) and (b), and OoD datasets are shown with different colors.
14

---

Published as a conference paper at ICLR 2023
D FEATURE AND UNCERTAINTY SCORE BEHAVIOR
To deeply understand why the InW samples share similar uncertainty distribution with OoD samples rather than InC samples, analyze the feature representation and uncertainty score behavior in this section. Surprisingly, we find the W samples have more similar features with InC samples than OoD samples, which is opposite from the uncertainty score behavior.
Features of InC/DDW/OoD samples are separable. We visualize the feature representations in Fig 10. We find the feature distribution of InC, InW and OoD samples follow a hierarchy structure. The InW samples surround the InC samples, and Ool samples are further far away and located at the outer edge of InW samples, such as three distributions class A and B in Fig 10 (b). Therefore, separable from the feature representation perspective.
the features of InC, InW, and OoD samples To further testify this idea, we calculate the similarity between InC/InW/OoD samples and training samples of each class, shown in Fig. 11. We samples with training samples, and InW samples have smaller similarity, and OoD samples have the smallest similarity. In conclusion, features of ICC/nW//ooD samples are in hierarchy structure and distinguishable.
that the InC samples the most similar
1.0
1.0
InC InW OoD
0.8
0.8
Class
0.6
0.6
0.4
0.4
Class
0.2
0.2
0.0
0.0
0.0
0.2
0.4
10
0.6
(a)
(b)
Figure 10: (a) and (b) are t-SNE visualization results of the whole test dataset and 10 classes.
1.0
InC
0.9
InW
OoD
Similarity
0.8
0.6
0.5
20
40
60
80
100
Class index
Figure 11: Similarity between with training samples of each class.
Table 9; We provide the feature similarity of W/ICCnC and InW/OoD, the mean of uncertainty score, and the AUROC of InW/InC and WnODOoD in this table.
Similarity
Uncertainty
Discrimination (AUROC)
InW/InC
InW/OoD
InC
InW
OoD
InW/InC
InW/OoD
Feature space
0.797
0.733
0.057
0.158
57.25 0.583
0.179
84.91 83.36
Logit space
0.718
0.571
0.314
0.537
$7.95
15

---

Published as a conference paper at ICLR 2023
Features of InW samples are more similar with InC samples than OoD samples. Our finding in Sec. 3 is that InW samples share similar uncertainty scores with OoD samples rather than InC samples, so we analyze whether the feature representation also follows the same behavior. We cal- culate the similarity between In W/InC samples and InW/OoD samples in the feature space and logit space. Then provide the mean of uncertainty scores based on the KNN method and MaxLogit method, as well as the AUROC of InW/InC and InW/OoD to illustrate the uncertainty discrimina- tion performance. In Table we can see that the similarity of InW/InC is larger than InW/OoD (0.797-0.733), which means InW samples have more similar features with InC samples than OoD samples. However, the uncertainty scores of InW samples are more similar with OoD scores than InC samples (0.158/0.1 (0.158/0.179-0005) 179-0.057), so that WnWDOoD can not be distinguished very well like InW/InC (57.25-84.91). Therefore, we draw a very interesting conclusion that the feature behavior and uncer- tainty score behavior of InW/ and InW/OoD are contradictory. InW samples are more similar to InC samples in the feature/logit space but more similar to OoD samples from the uncertainty score perspective.
Let us formulate this phenomenon mathematically. Suppose we have Tei Tway To which represents the feature of an InC, InW, and OoD sample, respectively. From Table we know that
sim(rw, Te) > sim(Ia;;R))
(4)
where sim refers to the similarity. Then, we have an uncertainty estimation function f to measure the uncertainty u of sample based on the features, so Uh [(xe), th = [( Uh f(x)).
Based on our finding in Sec. 3 that InW samples share similar uncertainty distribution with OoD samples rather than InC samples, we have
sim Mhe, Ue) < sim(uw, Uo), sim(f f(x)) <
(5)
Comparing Eq. 5 and Eq. 4 we find that the uncertainty estimation function f changes the similarity relationship between InW/InC and InW/OoD.
Let us give toy example of how f changes the similarity relation- ship. Suppose the logit space is un- der 2 dimensions (s, t). and Ee = (1.3). The
Uncertainty v
t
Uncertainty
f(x,)
f(xo)
f(x)
f(xw)
f (xx 1/s
f(xw)
(2,2), Fw 11. similarity is measured with Euclidean distance, and in this case The is closer to Te.. 50 Eq. 4 holds for this example. If f(x)
X
[(x))
%
f(x))
X
X
(a)
(b)
like Fig. 12 (a), then Ue = 2, uw -1, uo = -1. In this case, Uh Uo Up. So the un- certainty score of InW sample is sim- ilar with OoD sample instead of InC sample. This example illustrates why an InW sample has a similar feature to an InC sample, but has similar uncertainty score with OoD sample. This is how existing uncertainty estimation methods work, as the results in Sec. 3 and Sec. 4 show that InW samples have similar uncertainty distribution with OoD samples. This kind of method is suitable for UOSR problem where InW and OoD samples are supposed to be rejected at the same time.
Figure 12: (a): Tu is close to x, in (s, t) space, but ff (( is
close f((..) in uncertainty space; (b) The is close to 25 in
(s, 1) space, and (((..) is also close to f(( in uncertainty space.
We provide another uncertainty estimation function f in Fig. 12 (b), where f(x) = t/s. In this case, Uw = Ue = 1 < u. = 3, so the uncertainty score of the InW sample is similar to the InC sample instead of the OoD sample. This is the ideal case for the traditional OSR problem to reject OoD samples and accept InC and InW samples.
Conclusion. In this section, first find that the features of InC. InW and OoD samples are sep- arable, and show that InW samples have more similar features to InC samples than Ool samples, which seems contradictory with the finding in Sec. 3 that InW samples share more similar uncer- tainty scores with OoD samples than InC samples. Then we give a toy example to illustrate that the uncertainty estimation method is the reason for this phenomenon. Although existing uncertainty estimation methods are designed to behave like Fig. 12 (b) for the OSR problem, they actually work as Fig. 12 (a) which is more suitable for the UOSR problem.
16

---

Published as a conference paper at ICLR 2023
E IMPLEMENTATION DETAILS
E.1 DEFINITION OF AURC
We follow (Kim al., 2021) to give the definition of AURC here. Suppose we have dataset D = {(x,, 34 ) where x, and denote the training sample and corresponding closed-set label. Then, we use a confidence estimation method (methods in Table 5) assign confidence score k(p;) for each X,. Given a threshold 8, the sample coverage is computed as
SSg (D)
c
where is the number of samples in set. Then the risk r(0) is computed as
1 # vi)
r(0) wss,
SS
where 1(-) and ds denote the indicator function the predicted class respectively. So we have (c(0), r(0)) pairs which can consist of a risk-coverage curve (RC-curve). The AURC is the area under the RC-curve.
E.2 IMAGE DOMAIN
w/o pre-training. When we train the model from scratch, we find that setting the base learning rate as 0.1 and step-wisely decayed by 10 every 24000 steps with totally 120000 steps achieve good enough closed-set performance. We use a linear warmup strategy to warmup the training in the first 500 steps. We SGD with momentum, batch size 128 for all models.
w/ ImageNet-21kk pre-training. When we fine tune the model with ImageNet pretrained model from (Kolesnikov et al., 2020), we the base learning rate 0.003 and step-wisely decayed every 3000 steps with totally 10000 steps. We use linear warmup strategy to warmup the training in the first 500 steps. We use SGD with momentum, batch size 512 for all models. For outlier exposure methods, the batch size of outlier data is set to 128.
InD and OoD datasets. We follow the datasets setting in (Liang et al., 2018). The training InD dataset is CIFAR-100 100, which contains 100 classes with 50000 training images and 10000 test im- ages. The OoD datasets for open-set evaluation are TinylmageNet and LSUN. The TinylmageNet dataset contains 10000 test images from 200 different classes, we use the Tiny ImageNet (resize) (Liang et al., 2018). The Large-scale Scene UNderstanding (LSUN) dataset consists of 10000 images of 10 different scenes categories like classroom, conference room, dining room, etc. We use LSUN (resize) from (Liang et al., 2018). The size of images in both InD and OoD datasets is 32 32. We the same strategy in image preprocessing stage resolution the training image is lower than 96 x 96, we use 128 x 128 image cropping technique and random horizontal mirroring followed by 160 x 160 image resize. Test images and OoD images are directly resized to 128 x 128.
(Kolesnikov et al., 2020), if the
Outlier datasets. We 300K Random Images dataset from (Hendrycks al., 2019b) outlier dataset for those outlier exposure methods. The 300K Random Images dataset is a debiased dataset with real images scraped from online. According to (Hendrycks et al., 2019b), all images that belong to CIFAR classes and images with divisive metadata have been removed.
E.3 VIDEO DOMAIN
w/o pre-training. When we train the model from scratch, we find that setting the base learning rate as 0.05 and step-wisely decayed every 160 epochs with totally 400 epochs can achieve good enough closed-set performance. The batch size is 256 for all methods.
w/ Kinetics400 pre-training. We follow (Bao et al., 2021) 1) to set the base learning rate 0.001 and step-wisely decayed every 20 epochs with totally 50 epochs. For those methods without outlier exposure, fix the parameters of all Batch Normalization layers except the first one, and set the learning rate of the fully connected layer to be 10 times the base learning rate. For those methods with outlier exposure, all parameters updated with the same learning rate.
17

---

Published as a conference paper at ICLR 2023
InD and OoD datasets. We follow the datasets setting in (Bao et al., 2021). The training InD dataset is UCF101, which contains 101 classes with 9537 training samples and 3783 test samples. HMDB51 and MiT-v2. We use the test sets of them which contain 1530 samples and 30500 samples respectively. For UCF101 and HMDB51, we follow the MMAction (Yue Zhao, 2019) to use the split for training and evaluation, which is the same with (Bao et al., 2021). Note that in (Bao et al., 2021), they find some classes in HMDB51 overlap with those in UCF101 but they do not clean them. We remove the overlapping classes in UCF101 and HMDB51 so that OoD data does not contain any samples of InD classes. The classes we remove HMDB51 and the corresponding same classes in UCF101 are in Table 10.
The OoD datasets for open-set evaluation
Table 10: Overlapping classes in HMDB51 and UCF101.
HMDB51
35, Shoot bow
29, Push
15, Golf
26, Pull up
UCF101
2, Archery
71, PushUps
32, GolfSwing
69, PullUps
HMDB51
30, Ride bike
34, Shoot ball
43, Swing baseball
31, Ride horse
UCF101
10, Biking
7. Basketball
6, BascballPitch
41, HorseRiding
Outlier datasets. We use Kinetics400 as our outlier datasets for those outlier exposure methods. To ensure that the classes of outlier data is not overlapping with InD data and OoD data, we re- move corresponding classes in Kinetics400. The overlapping classes between Kinetics400 and UCF101/HMDBB1 are too many to be listed here ( (129 overlapping classes). The available training sample ID list of Kinetics400 and all codes will be public. We pick up 271 classes from Kinetics400 and 25 samples in each class as outlier data.
F DETAILED UNCERTAINTY ANALYSIS OF TABLE 5
Table 11 is the complementary analysis of Table 5, which further illustrates that outlier data can improve InC/OoD discrimination but may not be helpful for InC/InW discrimination.
Table 11:
Uncertainty distribution analysis in image domain with ResNet50. Pre-training is not used. OoD dataset: TinylmageNet. AUORC (%) is reported.
Methods
ODInC/Oa InW/oDD OSR UOSR
ODIN
N
88.35
80.76
64.36
81.65 86.69
LC
N
84.60
82.58
55.14
76.37
84.16
OpenMax
N
85.16
85.96
51.27
76.23 85.33
OLTR
N
85.99
85.74
52.22
77.10
85.94
SoftMax
N
84.69
85.68
50.64
75.59 84,90
(oE)
R
90.04
84.00
68.54
84.35
88.78
ARPL
N
88.76
85.77
$8.09
80.49
88.13
ARPL+CS
R
89.36
81.40
65.32
82.81 87.65
MCD(Dropout)
N
84.15
74.17
63.15
79.21 82.25
MCD
R
87.47
85.23
61.39
79.88
86.97
PROSER
G
87.04
77.84
62.57
79.23
84,82
PROSER (EB)
R
88.36
77.88
65.60
81.95
86.06
VOS
G
87.51
83.38
58.17
79.72
86.65
VOS
R
91.44
84.41
70.32
85.78
89,96
G UOSR BENCHMARK UNDER TRADITIONAL OSR SETTING
We evaluate the UOSR and OSR performance under the traditional OSR dataset setting (Vaze et al., 2022), where a part of data within one dataset is regarded as InD and the remaining data regarded as OoD. The experiments are conducted under the most challenging TinylmageNet dataset and the results are Table 12. We can see that the UOSR performance still higher than OSR performance for most methods, which means InW samples share more similar uncertainty distribution with InC samples than OoD samples. Surprisingly, we find the InWDOoD AUORC of ODIN method achieves 84.32, which means InW and OoD samples can be well distinguishable. This proves claim in Appendix that the features InC/InW/ooD samples are separable and it is possible to find a proper uncertainty estimation method to distinguish these three groups of data.
18

---

Published as a conference paper at ICLR 2023
Table 12: Unified open-set recognition benchmark in the image domain under the traditional OSR dataset setting. All methods are conducted under the R50 model. Dataset is Tiny ImageNet. 4, V refer OSR-based, SP-based, UOSR-base methods. Pre-training weights are used.
UOSR
OSR
InC/InW
InC/OoD In/DODD
Methods
Acc.t
AURCI
AUROCT
AUROCT
AUROCT
AUROCT
AUROCT
SoftMax
87.40
730.10
94.06
90.59
91.02
94.11
66.21
ODINI
87.40
724.90
96.62
95.21
85.50
96.78
84.32
LC
87.40
756.49
88.02
84.32
81.27
58.05
OpenMax
86.60
731.73
94.49
90.25
91.77
94.53
62.59
OLTR
87.60
731.93
93.69
90.09
89.96
93.74
64.34
PROSER
86.90
750.26
92.29
90.21
77.01
92.51
74.91
BCE
87.60
732.79
93.74
90.46
89.63
93.80
66.92
TCP:
87.70
731.66
94,20
90.54
90.20
94.25
64.06
DOCTOR*
87.40
731.69
93.97
90.54
90.25
94.02
66.40
SIRC(MSP, ||= )
87.40
731.48
94.02
90.53
91.04
94.06
SIRC(MSP, Res.)
87.40
729.96
94.77
91.71
91.19
94.82
70.13
SIRC(-H, |=11,) )
87.40
730.12
94.74
91.65
91.24
94.79
69.89
SIRC(-H, Res.)
87.40
731.34
94.06
90.59
91.02
94.10
66.22
H UOSR BENCHMARK UNDER DIFFERENT TRAINING SET
We provide the UOSR evaluation results in Table 13 when InD and OoD datasets are TinylmageNet and CIFAR 100 respectively. Pre-training weights are used. TinylmageNet has 200 classes in the training which is more diverse than CIFAR100. We ImageNet is more challenging than CIFARI00, as the Acc. of CIFAR 100 is 86.44 and Acc. of TinylmageNet only reaches 77.02. In this way, the impact of InW samples becomes further huge. For example, the performance gap between UOSR and OSR for the SoftMax method is 8.95 when InD dataset is TineemageNet, and this value is only 0.34 when InD dataset CIFARI00. So the InW samples are more important for the performance when InD dataset is difficult, as lower closed-set Acc. means more InW samples.
that training the model Tiny-
Table 13: Unified open-set recognition benchmark in the image domain. All methods conducted under the R50 model. InD and OoD Dataset are TinylmageNet and CIFAR 100 respectively. i, O refer
SP-based, UOSR-based methods. Pre-training weights are used.
UOSR
OSR
InC/InW
InC/OoD InW/OoD
Methods
Acc.t
AURC|
AUROCT
AUROCT
AUROCT
AUROCI AUROCT
SoftMax
77.02
340.70
86.22
77.27
88.83
85.62
49.27
ODINI
77.23
359.57
84.01
75.32
86.53
83.43
47.81
LC
77.23
385.15
79.76
71.85
83.21
78,98
47,69
OpenMax
76.90
340.21
86.53
77.09
89.63
85.81
48.04
OLTR
77.00
341.60
86.04
76.75
89.02
85.36
47.92
PROSER
75.93
392.91
80.50
74.65
79.20
80.50
55.20
BCE!
76.91
339.43
86.40
76.83
89.64
85.66
47.44
TCP*
77.82
336.61
86.48
77.47
89.72
85.76
48.38
DOCTOR
77.23
339.02
86.69
77.62
89.86
85,96
49.30
SIRC(MSP, ||=||,> )
77.03
337.28
86.82
78.86
88.78
86.37
53.65
SIRC(MSP, Res.)
77.03
316.14
90.66
87.00
88.82
91.08
73.31
SIRC(-H, |=11,
77.03
333.73
87.60
80.13
89.00
87.28
56.19
SIRC(-H, H. R
77.03
311.98
91.39
88.20
89.02
91.94
75.67
19

---

Published as a conference paper at ICLR 2023
1 UOSR BENCHMARK UNDER SSB DATASETS
(Vaze et al., 2022) proposed the Semantic Shift Benchmark (SSB) which compose several fine- grained datasets, including CUB, Standford Cars, FGCV-Aircraft, and a part of ImageNet. OSR in SSB is more challenging as OoD samples share the same coarse labels with InD samples, and only have some minor differences the fine-grained properties. We evaluate the UOSR performance on CUB and FGCV-Aireras and provide the results of EASY and HARD modes. Pre-training weights are used for better performance. From Table 14 and 15 we can see the UOSR and OSR performance are higher under the EASY mode compared to HARD mode as expected, since the OoD samples are more similar with InD samples in the HARD mode. Our conclusion that W samples share similar uncertainty with OoD samples still holds, the AUROC of WnW/OoD is close to 50 and much lower than InC/OoD and InC/InW.
Table 14: Unified open-set recognition benchmark in the image domain. All methods are conducted under the R50 model. Dataset is CUB-200-2011. 1. t, O refer OSR-based, SP-hased, UOSR- based methods. Pre-training weights are used. EASY/HARD
UOSR
OSR
InC/InW
InC/OoD
InW/OoD
Methods
Ace.t
AURCI
AUROCT
AUROCT
AUROCT
AUROCT
AUROCT
SoftMax
91.78
77.69/120.46
92.79/84.31
90.33/78.78
90.56
93.37/82.81
56.34/33.73
ODIN!
91.61
86.89/157.09
91.20/77.37
91.45/73.11
82.42
93.52/76.14
68.90/40.09
LC
91.61
78.34/121.66
92.66/84.35
89.86/78.63
91.15
93.06/82.69
54.95/34.38
OpenMax
91.30
78.07/119.61
92.87/85.43
90.01/79.78
91.14
93.35/83.98
54.99/35.67
OLTR
91.33
80.38/118.50
92.43/85.30
89.42/79.72
90.66
92.91/83.95
52.65/35.19
PROSER
91.33
79.39/128.14
92.50/83.81
90.32/78.53
89.31
93.37/82.42
58.21/37.60
BCE:
91.50
79.75/122.19
92.24/84.76
89.34/79.27
90.64
92.67/83.31
53,43/35.80
TCP#
92.06
77.31/116.93 92.55/84.85
90.28/79.92
90.07
93.17/83.64
56.71/36.77 78
DOCTOR
91.61
78.09/121.61 92.76/84.37
90.09/78.68
91.13
13.18/82.71
56.26/34.70
SIRC(MSP, |= 1
91.78
18.04/119.42
92.72/84.46
90.19/78.99 95
90.59
93.27/83.00
55.78/33.70
STRC(MSP, R
91.78
77.69/120.66
92.79/84.31
90.33/78.7 78
90.56
93.37/82.81
56.33/33.74
SIRC(-H. [=]]))
91.78
76.97/119.40 93.11/84.52
91.06/83.08
90.52
93.78/83.08
78..65/34.8
SIRC(-74 H. R
91.78
76.65/120.65 93.18/84.33 91.19/78.91
90.47
93.88/82.86
61.20/34.81
Table 15: Unified open-set recognition benchmark in the image domain. All methods are con- ducted under the R50 model. Dataset is Fine-Grained Visual Classification of Aireraft (FGVC- Aireraft). 1. O refer to based SP-based, UOSR-based methods. Pre-training weights are used. EASY/HARD
UOSR
OSR
InC/InW
InC/OoD
InW/OoD
Methods
Acc.t
AURCI
AUROCT
AUROCT
AUROCT
AUROCI
AUROCT
S
85.61
129.71/127.188
89.46/82446
84.24/72.85
88.54
89.80/79.08
51.18/35.75
ODIN"
85.25
152.17/173.15
87.11/75.75
84.92/68.36
80.04
89.72/73.32
57.17/39,70
LCt
85.25
134.44/144.29
88.98/80.15
83.55/69.74
87.58
89.50/75.93
49.13/33.95
OpenMax
86.69
123.31/123.86
90.13/82.10
85.63/73.20
88.12
90.80/79.01
51.99/35.36
OLTR
85.97
128.01/124.92
89.86/82.79
85.11/73.45
88.57
90.31/79.66
53.26/35.35
PROSER
85.97
124.15/137.70
90.73/80.40
86.92/70.44
87.31
91.93/76.67
56.24/32.22
BCE!
85,37
134.56/127.05
88.75/82.37
84.05/73.93
86.49
89.58/80.06
51.75/38.17
TCP
85.25
132.09/131.88 80
89,30/82.19
83.60/72.12
88.41
89.63/78.66
48.73/34.30
DOCTORR
85.25
133.93/144.07
19.16/80.24
83.79/69.89 80
87.68
89.71/76.00
49.58/33.96
SIRCMMSP, ||*||]
85.61
130.50/126.36
89.2 89.24/82.57
83.86/73.04
88.52
50.31/35.99 89,49/79.27 .2
SIRC(MSP, Res.)
85.61
129.71/127.18
89.46/82.46
84.24/72.85
88.54
89,80/79.00
51.17/35.76
SIRC(-H, ||=|], o
85.61
128.97/125.59
89.71/82.85
84.70/73.60
88.55
90.13/79.70
52.40/37.35
SIRC( H. 8
85.61
128.04/126.58
89.97/82.69
85.16/73.35
88.52
90.49/79.46
53.46/37.00
20

---

Published as a conference paper at ICLR 2023
J UOSR BENCHMARK AND FEW-SHOT RESULTS OF VGG13
We provide the UOSR benchmark and few-shot results of VGG13 Table 16 and 17 respectively. Our FS-KNNS still achieves the best performance under all settings.
Table 16: Unified open-set recognition benchmark in the image domain. All methods conducted under the VGGI3 model. InD dataset is CIFAR 100 while OoD dataset is SinyImageNet. 1. O refer to OSR- SSR-based, SP- COSR-based methods. OD: use Outlier Data in training.
w/o pre-training
w/ pre-training
UOSR
OSR
UOSR
OSR
Methods
OD
Acc.t
AURC|
AUROC1
AUROCT
Acc.t
AURC|
AUROCT
AUROC
SoftMax
X
75.07
341.06
85.87
76.44
74.69
311.70
91.01
85.13
ODIN!
75.07
346.70
85.27
78.13
74.69
314.80
91.24
88.90
LC
X
75.07
365.96
81.77
76.68
74.69
339.96
88.27
88.95
OpenMax
74.52
368.58
82,81
72.68
75.05
312.70
90.47
84.25
OLTR
73.80
365.61
84.68
74.37
74.52
306.73
92.12
87.08
PROSER
70.95 X
376.63
84.00
79.29
71.11
367.16
86.45
85.97
BCE
X 74.74
339.25
86.54
76.99
74.45
325.29
88.85
81.14
TCP:
75.09
340.41
85.94
76.18
74.83
313.80
90.57
84.35
DOCTOR
X
75.07
340.69
85.95
76.72
74.69
310.29
91.37
85.97
SIRC(MSP, || 10
X
75.07
343.44
85.33
75.63
74.69
313.22
90.67
84.59
SIRC(MSP, Res.)
75.07
336.87
86.77
78.23
74.69
309.67
91.46
86.14
SIRC(-H, H. |=]|,
75.07
343.53
85,25
76.37
74.69
309.19
91.56
86.82
SIRC H. R
75.07
335.71
86.93
79.13
74.69
305.34
92.39
88.34
OE
71.71
312.44
93.71
89.76
73.38
306.72
93.34
92.77
EB
74.19
340.66
87.67
85.73
72.76
340.81
89.40
89.79
VOS
71.73
312.75
93.68
90.76
73.08
306.37
93.65
92.97
MCD
70.45
316.57
94.21
91.54
72.08
300.87
95.40
94.26
Table 17: Results of few-shot UOSR in the image domain. Model is VGGI3 with pre-training. InD and OoD datasets are CIFAR100 and TinylmageNet respectively.
5-shot
1-shot
AURCI
AUROCT
AURCI
AUROCT
Methods
UOSR
UOSR OSR InC/OoD InC/
UOSR
UOSR OSR InC/OoD InC/InW
SoftMax
311.68
91.00
85.13
92.10
86.68
311.68
91.00
85.13
92.10
86,68
KNN
306.04
93.04 91.61
95.75
82.32
306.04
93.04 91.61
95.75
82.32
FS IN-KNN
295.80
95.44
98.87
99.39
79,83
296.75
95.27
98.48
99.16
79.87
SSD
321.31
92.57
99.26
99.36
65.72
315.46
93.11
99.10
99.27
68.77
FS B-NNN
296.42
94.22
92.15
96.50
85,20
296.67
94.20
92.15
96.48
85.16
FS-KNN'S
303.01
92.85
89.42
94.38
86.83
302.84
92.88
89.45
94.41
86.83
SIRC
287.91
95.71
96.33
98.00
86.66
288.12
95.68
96.30
98.00
86.53
FS-KNNS
285.85
95.95
96.52
98.30
86.66
287.02
95.75
95.96
98.04
86.67
K UOSR UNDER NOISY OUTLIER EXPOSURE
When we introduce outlier data into the training process, the labels of some outlier data may be corrupted and become the labels of InD class. We call this kind outlier data as noisy outlier data. We study how noisy outlier data influence the UOSR and OSR performance in this section. In addition, we use NGC (Wu et al., 2021) to find the noisy outlier data and correct them. The results shown in Table 18. We noise level when NGC is not used. This is natural as some noisy outlier data corrupt the InD data distribution. In contrast, the closed-set Acc. does not drop with the growth of noise level when NGC
that the closed-set Acc. gradually decreases with the growth of
21

---

Published as a conference paper at ICLR 2023
used. So NGC is very effective in finding corrupted samples. When the noise level is 0%, the UOSR performance with NGC is better than the performance without NGC (lower AURC value), meaning that other parts in NGC that not related label coruuption, such contrastive learning, are helpful for UOSR. Surprisingly, the performance of UOSR and OSR are relatively stable under different noise levels matter closed-set Acc. when NGC is not used. So the model is robust in open-set related performance when noisy outlier data is introduced. Our finding that InD samples share similar uncertainty scores with OoD samples still holds as AUROC of InW/OoD is close to 50.
NGC not, compared
the clear performance drop of
Table 18: UOSR and OSR performance under noisy outlier data. InD dataset is CIFAR 100 and outlier dataset 300K Random Images. OoD dataset is TinylmageNet. Experiments are conducted with ResNet18 backbone.
UOSR
OSR
InC/InW
InC/OoD
InW/OoD
Noise level
NGC
Acc.
AURC|
AUROCT
AUROC
AUROC1
AUROCT
AUROC1
0%
X
76.19
334.13
85.50
77.29
85.05
85,60
50.70
20%
X
72.15
343.46
87.84
77.53
87.34
87.98
50.45
40%
X
70.06
343.60
89.91
79.86
88.36
90.37
55.25
60% X
69.46
346.62
90.07
79.79
88.76
90.47
55.52
80%
X 69.20
348.18
89.95
79.05
88,84
90.29
53.77
100%
68.05 X
356.35
89.97
78.31
89.63
90.08
53.23
0%
77.52
315.32
87.75
81.12
85.61
88.23
56.61
20%
V
77.50
316.86
87.81
80.46
86.88
88.02
54.40
40%
76.87
321.66
87.58
79.48
87.15
87.68
52.23
60%
77.07
319.58
87.91
79.88
87.43
88.02
52.53
80%
77.12
317.90
88.17
80.14
87.41
88.34
52.49
100%
77.38
328.85
86.24
78.00
86.46
86.20
49.94
L ABLATION STUDY OF FEW-SHOT UOSR
Analyze of K. Similar with (Sun et al., 2022), we find the value of K influences the performance lot. The ablation study of is shown in Fig. 13. It shows that the best K for InC/OoD discrimination is between and 5, and drops quickly after 7. In contrast, the InC/InW performance keeps increasing until 20 and then drops. The overall UOSR performance achieves the best when K
96
AUROC
97
AUROC 28
95
AUROC
84
5
50
3
30
20
50
10
20
5
30
InC/OoD
InC/InW
UOSR
OSR
(0) 5-shot
95
AUROC
96
AUROC
80
94
AUROC
78
05-
93
34
InCoOol)
InC//nW
UOOK
OSK
(b) 1-shot
Figure 13: Ablation study of used in FS-KNN. The backbone is ResNet50.
Analyze of and Aknns- Two hyper parameters FS-KNNS including a and Aknns are significant for the performance. Atnns is a threshold to determine when the weight of becomes large, i.e., ufs-knns is important when U1 > Aknna- a to control the change rate of the weight. The ideal Akenn should be located between the uncertainty of InW and OoD samples as shown in Fig. 5 (a) and (c), so that the uncertainty of InW samples is still determined by SoftMax and the uncertainty of OoD samples is enlarged because of FS FS-KNN. However, which sample is InW OoD is unknown during test, so we cannot determine the Agnns based on the uncertainty distribution of test samples. (Xia & Bouganis, 2022) proposed to determine the hyper parameters of SIRC based the training
22

---

Published as a conference paper at ICLR 2023
set, but find that the uncertainty distribution ú fnnnknn of training samples significantly different from the test InC samples, as shown in Fig. 14. Therefore, we seek help from the OoD reference
10
InC-test
abbicid C₁
InW-test
OoD-test
InC-train
density
OoD-reference
0.0
0.2
0.4
0.6
0.8
1.0
uncertainty
Figure 14: Uncertainty distribution of u fennnnns knns- InC-train samples have distinet uncertainty distri- bution with InC-test samples, but OoD reference samples share similar uncertainty distribution with OoD test samples. e correspond to the Aknna when 1.5, 0.5, 0, -0.5 Eq. 6.
0
0g
2.5
0g
94.73
94 75
94.77
79
96.5
93.76
77
93.79
93.81
95.5
89
95.40
65.49
95.57
95 62
95
86
96.0
96 49
86.33
96.17
86.03
85.90
5- 94
94.4
94.44
34.45
94.50
95.0
15.5
88
94.5
05.48
56.05
96.40
86.52
96.45
88.62
88.02
$7.26
86.41
85.60
94.66
$5.09
05.30
95.31
95.16
94.0
50
93.89
55.33
96.24
96.39
$4.0
50
09.62
89.53
89.00
50 93.30
94.64
95.53
95.23
93.5
160000
InCTTsW
1008
Figure 15: Ablation study of 3 and in Eq. and 6.
samples, as find their uncertainty distribution is extremely similar with OoD test samples. We calculate the mean u and standard deviation of the fssa-ans of OoD reference samples, and we aim find proper Agnns through:
Aknns - 3
We draw several Agnnne with different 3 in Fig. 14 to A smaller Agene or a larger 3 means more samples will be influenced by the FS-KNN uncertainty, so that the ufs-knns of more OoD test samples will be strengthened by the FS-KNN uncertainty which brings better InC/OoD) performance, but meanwhile more InW samples will also be influenced by the FS-KNN uncertainty which brings shown in Fig. 15. In other words, Aknns controls the trade-off
worse InC/InW performance, between InC/OoD and InC/InW performance. Overall, 3 = achieves the best UOSR performance which is balanced result of InC/OoD and InC/InW trade-off. Larger means the weight of FS- KNN uncertainty grows quickly when FS-KNN uncertainty is closed to Aenns- We find that smaller makes the performance insensitive of Aisnns and vice versa. So finally pick and as 50 and as the optimal hyper parameters.
23

---

Published as a conference paper at ICLR 2023
M FEW-SHOT UOSR IN THE VIDEO DOMAIN
Table 19: Results of few-shot UOSR in the video domain. Model is TSM with pre- pre- vre-trinning InD and OoD datasets are UCF101 and HMDB51. AUORC (%) and AURC (x109) are reported.
5-shot
1-shot
AURCJ
AUROCT
AURCJ
AUROCT
Methods
UOSR
UOSR OSR
InC/OoD InC/InW UOSR
UOSR OSR
InC/OoD InC/InW
SoftMax
66.55
93.66
91.44
93.46
94,99
66.55
93.66
91.44
93.46
94.99
KNN
73.73
93.38
92.99
94.11
88.44
73.73
93.38
92.99
94.11
88.44
FS-KNN
68.44
95.04
96.14 93.91
96.71
83.74
73.19
93.60
94.32
95.09
83.53
SSD
70.36
95.41
95.97
79.97
75.61
92.48
93.34
94.07
81.75
ES-KNN+S
65.64
95.44
94.69
96.13
90.78
68.98
94.54
93.66
95.14
90.48
FS- FS-KNN'S
65.29
94.09
92.13
93.96
94.99
65,42
94.04
92.08
93.91
94.97
SIRC
62.41
95.11
93.79
95.13
94.94
63.11
94.88
93.49
94.88
94.84
FS-KNNS
60.00
96.19
95.37
96.40
94.82
62.65
95.35
94.26
95.48
94.49
98
(Our)
TS
7S wt
95
TP wo OE
AUROC
92
89
DLAR
UNN SVI
MCD
86
SoftMax
OE
# FS, KNN
OpenMax
EB
MC Dropout
Vos
FS, KNNS
83 35
75
95
115
135
155
175
195
215
Figure 16: UOSR performance under all settings of TSM backhone in the video domain. Ool dataset is HMDB51.
24


Reviews:
[{"review_id": "o6dMS46nKhO", "reviewer": "Reviewer_q3SW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work authors evaluate various open set recognition methods on the task of Unified Open set recognition(UOSR) where we need to detect novel classes but also misclassified/incorrect prediction. Authors also investigate training methods which reportedly improve open set recognition on UOSR. In addition, provide few-shot UOSR evaluation.\n", "summary_of_the_review": "Authors evaluate methods on important task of UOSR but given lack of contributions in terms of providing new method, insight or unknown empirical behavior. Given UOSR is already established & evaluated in earlier work, and the emphasized phenomenon of wrongly classified examples well. Based on GCD, we know clustering works even in unsupervised fashion for OSR and hence few-shot learning would work well too albeit with lesser data. Although it is interesting to validate empirically that expectation of Uncertainty distributions is close to UOSR, in my opinion not sure if the work is worthwhile of ICLR acceptance. \n\n", "strength_and_weaknesses": "Strengths:\n\nPaper is easy to follow and task of UOSR & evaluations is important to community.\n\n\nWeakness:\n\nLacks novel insights, methods for UOSR based on observed phenomenon, please see next section for more details. \nIs there any particular reason to not consider baselines like Deep ensembles [2] which is best performing method in initial UOSR evaluation benchmarking study? Also, as the paper makes a point w.r.t entropy distributions, it might be worthwhile to evaluate calibrated models (e.g. temperature scaling) and see the relative change in behavior compared to baseline models.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cYMBoSHkY9", "reviewer": "Reviewer_ZHBz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This article demonstrates the uncertainty distribution of almost all existing OSR methods is actually closer to the expectation of UOSR than OSR. This article also introduces a new evaluation setting into UOSR, which is few-shot UOSR. Then, the FS-KNNS method is proposed, which achieves state-of-the-art performance under all settings.", "summary_of_the_review": "This paper mentions many important issues that have not been paid attention to by the current research on OoD detection. However, there are still some problems with this article. If the authors can provide convincing replies to the above issues, I am willing to improve my initial score.", "strength_and_weaknesses": "Strength:\n- This article deeply analyzes the UOSR task under different training and evaluation settings through extensive experiments.\n- Although Vaze et al. [7] have demonstrated the importance of a good closed set, this paper further illustrates the impact of InW samples on OSR performance, providing a meaningful reference for future OOD detection research.\n- The article analyzes how pre-training and outlier exposure influence the UOSR.\n- This paper proposed a new evaluation setting called few-shot UOSR, and proposed a new method named FS-KNNS.\n- Detailed ablation experiments are included in the appendix of this paper, and the results of the paper are reproducible.\n\nWeaknesses:\n- In Section 2, the authors mentioned the definition of UOSR, which rejects InW and OoD samples. The authors may also consider discussing outlier detection [1,2] as related work.\n- Some OSR methods need to use the generated OoD samples to assist in training [3,8], while some OSR methods only need to train InD samples to identify OoD samples [4]. This article should discuss whether the InW samples will play an equally important role in the above cases in Tables 2 and 3. Perhaps it is possible to report the performance of OSR methods with/without OoD sample-assisted training separately in the table.\n- The authors discuss the impact of InW on UOSR. Did the authors consider the effect of some noisy outliers in the training set on the results? Or just consider the impact of misclassified samples in the test set on the results? How to discover them and use them to improve the model? I am glad to see related discussions.\n- In Table 5, the authors compare the methods of outlier exposure and claim that the method of outlier exposure can effectively improve UOSR. However, I am concerned that this is the superiority of the OOD detection method itself, since the methods not based on outlier exposure are older. Should the author try to introduce outlier exposure settings for the same method and observe the AUROC changes to confirm the final effect, or try to compare some SOTA methods that do not require outlier exposure, e.g., [5,6]?\n\nSome Questions:\n- Appendix C mentioned the outlier datasets used for the study. However, the outliers of other methods appear to be generated from known data [3,8], while the authors use additional data directly. Should the authors discuss the difference between using real and virtual outliers, and how they relate to the conclusions drawn in this paper?\n- The InD dataset used for OSR in this paper is CIFAR-100, a relatively simple dataset. Whether the author considers using InD dataset with large-scale semantic space such as TinyImageNet, because this dataset seems to be more challenging and more realistic to reflect the negative effects of InW, I am glad to see related discussions.\n\nOverall:\nThis paper mentions many important issues that have not been paid attention to by the current research on OoD detection. However, there are still some problems with this article. If the authors can provide convincing replies to the above queries, I am willing to improve my initial score.\n\n\n[1] Goodge, Adam, et al. \"Lunar: Unifying local outlier detection methods via graph neural networks.\" AAAI. 2022.\n[2] Chauhan, Kushal, et al. \"Robust outlier detection by de-biasing VAE likelihoods.\" CVPR. 2022.\n[3] Wang, Yezhen, et al. \"Energy-based open-world uncertainty modeling for confidence calibration.\" CVPR. 2021.\n[4] Bao, Wentao, Qi Yu, and Yu Kong. \"Evidential deep learning for open set action recognition.\" ICCV. 2021.\n[5] Cao, Senqi, and Zhongfei Zhang. \"Deep Hybrid Models for Out-of-Distribution Detection.\" CVPR. 2022.\n[6] Wang, Haoqi, et al. \"ViM: Out-Of-Distribution with Virtual-logit Matching.\" CVPR. 2022.\n[7] Vaze, Sagar, et al. \"Open-Set Recognition: A Good Closed-Set Classifier is All You Need.\" ICLR. 2021.\n[8] Du, Xuefeng, et al. \"VOS: Learning What You Don't Know by Virtual Outlier Synthesis.\" ICLR. 2021.\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "afTT-d6I1_L", "reviewer": "Reviewer_zcng", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses an extended problem of open-set recognition (OSR), called Unified Open-set Recognition (UOSR). Unlike OSR, which only rejects testing samples from classes that the model has not seen during training, UOSR aims at rejecting both samples from unseen classes and also samples from seen classes but wrongly classified. The paper aims at providing a comprehensive study of the UOSR problem by adopting existing OSR methods. Meanwhile, analysis is also given on the effects of supervised pretraining on ImageNet or Kinetics 400 and the effects of training the model with outlier exposure with extra out-of-distribution samples. Finally, the paper introduces a simple technique for few-shot UOSR by introducing a KNN-based margin in SoftMax for uncertainty estimation, showing improved performance compared with the baselines.", "summary_of_the_review": "Overall, I find this paper interesting with promising analysis. However, given the concerns above on the significance of the findings, which appear to be quite straightforward after having the findings by Vaze et al 2022, as well as the concerns on compared methods and datasets, I would hold a relatively conservative view about the paper and not recommend for acceptance.", "strength_and_weaknesses": "Strength\n- The paper introduced a relatively thorough analysis of OSR methods for UOSR, revealing that the wrongly classified samples from known classes have a huge impact on the performance.\n- The paper is generally well-written and easy to follow.\n\nWeaknesses\n- Though a good set of analysis is given, some of them appear to be straightforward and intuitive. For example, The wrongly classified samples from known classes have a strong impact on the final performance, which appears to be very obvious especially when the correlation between InD performance and OoD performance is founded (Vaze et al, 2022) because better  InD performance means less InW and more InC.  Hence, the additional value of the study here seems to be limited.\n- It is unclear why state-of-art methods such as ARPL, ARPL+CS, and OpenGAN are missed in the comparison and analysis.\n- The SSB benchmark introduced by Vaze et al, 2022 appears to be better suited for OSR performance evaluation (thus UOSR), especially when there are fine-grained datasets and ImageNet-scale OSR data. It would be more convincing to evaluate the method on SSB.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-6vz25GyHs_", "reviewer": "Reviewer_sZtz", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the problem of 'Unified Open-Set Recognition' (UOSR), a variant of open-set recognition (OSR) which also considers how to deal with misclassified closed-set test samples. Concretely, the closed-set test samples are split into InW and InC (wrongly and correctly classified samples respectively), and the model must predict 'unknown' (i.e have a high open-set score) for InW. \n\nI would summarise the central finding of this paper as \"False Positives in standard OSR tend to be misclassified closed-set samples\" (by False Positives, I mean examples from the training classes which have been predicted as 'open-set', and by misclassified, I mean wrongly predicted within the closed-set classes). \n\nBesides the above, the authors provide an empirical investigation into OSR and OoD techniques for the UOSR problem, presenting some interesting findings such as that Outlier Exposure helps the InC/open-set decision but hurts the InW/InC decision. They further tackle the UOSR problem in a few-shot manner, and present a combination of KNN and a softmax score to solve it.", "summary_of_the_review": "Overall, I believe this paper has an interesting central finding for OSR which is intuitive but has, to my knowledge, not been explored in the OSR literature before. The paper further conducts relatively rigorous experiments and makes effort to provide all details and code. That said, I believe there are issues with the exact execution of some experiments and presentation of findings, which prevent the findings from being easily accessible to the reader.\n\nUPDATE AFTER REBUTTAL:\n\nI have now looked through the other reviews and the corresponding authors' responses. The authors have responded to my main concerns and clarified some of my confusions.\n\nMy fellow reviewers' main concern seems to be that Vaze et al. [1] already show a correlation between ID accuracy and OSR performance. Though it is, perhaps, a corollary that the 'False Positive' OSR predictions tend to be InW samples, I believe it is an important point which has not received sufficient emphasis in the literature. It further seems to me that the authors have provided substantial additional empirical evidence to respond to all reviewers' concerns. Thus I believe this paper provides a valuable empirical contribution to the field.\n\nAs such, I maintain my previous rating (8, Good Paper) and recommend the paper for acceptance.\n\n[1] OSR: A Good Closed-Set Classifier is All You Need? Vaze et al, ICLR 2022", "strength_and_weaknesses": "The main strength of this paper is the finding that, for most OSR methods, 'false positive' predictions tend to be misclassified closed-set samples. Though this may seem obvious, I do not believe this has been reported in the OSR literature. The main weaknesses for me revolve around the way this finding is phrased, and some of the experimental setups.\n\nStrengths:\n* As mentioned above, it is interesting to know that false positive open-set predictions tend to be misclassified closed-set examples. This is useful to researchers as the authors highlight a specific sub-problem, separating InC from open-set samples, as a key challenge. It is also important to practitioners, as both open-set examples and incorrectly predicted samples should be rejected by a closed-set classifier.\n* The paper provides a thorough set of experiments (though I have some questions about their execution and details, see weaknesses) to demonstrate interesting empirical findings, such as the one regarding outlier exposure mentioned above. \n* As needed for an empirical paper, the authors provide extensive additional implementation details in the appendix. They further include code to reproduce the paper's empirical findings, which is greatly appreciated.\n\nWeaknesses:\n* I am not sure that framing UOSR as an entirely new task is warranted or helpful in this case (e.g with new evaluation datasets). Rather, it seems that this paper highlights that a specific subset of the OSR problem (delineating InC vs open-set) is the most interesting part of the problem, and a new evaluation metric which focusses just on this decision could be interesting. \n* I am a little confused as to why authors have used non-standard datasets for evaluation. The authors rely heavily on a CIFAR100 (ID) vs TinyImageNet/LSUN (OoD) experiment to demonstrate their findings. This is more similar to an OoD evaluation rather than OSR. Standard practise in the OSR literature is to use held-out categories from a single dataset as 'unseen' (this is especially an issue for the LSUN experiment, whose taxonomy is unrelated to CIFAR100). \n* I am not sure why the authors have used different methods to compare video and image datasets, rather than a unified evaluation. A number of the image-based methods could be equally run on video and vice-versa. This lessens the clarity of the findings.\n* The authors compare and contrast training with Outlier Exposure to using pretrained models. However, I think it is important to distinguish the *data* from the *method* used. As such, I think a better experiment would be to pre-train models with the OE data and see if the findings hold. I would also suggest the authors discuss how much overlap there is in the semantics of the pre-training/outlier training set with the OoD test set.\n* (Minor): The authors do not introduce AURC, which I understand is from the Selective Prediction literature. Given this paper is targeted at the OSR community, I would suggest explaining it.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster