Open-Set Recognition: A Good Closed-Set Classifier is All You Need
Authors: Sagar Vaze
Kai Han
Andrea Vedaldi
Andrew Zisserman
Venue: ICLR 2022 Oral (2022)
Keywords: open set recognition
image recognition
computer vision

Abstract: The ability to identify whether or not a test sample belongs to one of the semantic classes in a classifier's training set is critical to practical deployment of the model. This task is termed open-set recognition (OSR) and has received significant attention in recent years. In this paper, we first demonstrate that the ability of a classifier to make the 'none-of-above' decision is highly correlated with its accuracy on the closed-set classes. We find that this relationship holds across loss objectives and architectures, and further demonstrate the trend both on the standard OSR benchmarks as well as on a large-scale ImageNet evaluation. Second, we use this correlation to boost the performance of the maximum softmax probability OSR 'baseline' by improving its closed-set accuracy, and with this strong baseline achieve state-of-the-art on a number of OSR benchmarks. Similarly, we boost the performance of the existing state-of-the-art method by improving its closed-set accuracy, but the resulting discrepancy with the strong baseline is marginal. Our third contribution is to present the 'Semantic Shift Benchmark' (SSB), which better respects the task of detecting semantic novelty, as opposed to low-level distributional shifts as tackled by neighbouring machine learning fields. On this new evaluation, we again demonstrate that there is negligible difference between the strong baseline and the existing state-of-the-art. Code available at: https://github.com/sgvaze/osr_closed_set_all_you_need.

Published as a conference paper at ICLR 2022
OPEN-SET RECOGNITION: A GOOD CLOSED-SET CLASSIFIER IS ALL YOU NEED?
Sagar Vaze*
Kai Han*t
Andrea Vedaldì
Andrew Zisserman"
*Visual Geometry Group, University of Oxford The University of Hong Kong
sagar, vedaldi, az) robots.. cx..xx....o. ox.ac.uk
kaihanx@hku.h
ABSTRACT
The ability to identify whether or not a test sample belongs to one of the seman- tic classes in a classifier's training is critical to practical deployment of the model. This task is termed open-set recognition (OSR) and has received signifi- cant attention in recent years. In this paper, we first demonstrate that the ability of classifier to make the 'none-of-above decision is highly correlated with its accuracy on the closed-set classes. We find that this relationship holds across loss objectives and architectures, and further demonstrate the trend both on the standard OSR benchmarks as well as on a large-scale ImageNet evaluation. Sec- ond, use this correlation to boost the performance of the maximum softmax probability OSR 'baseline' by improving its closed-set accuracy, and with this strong baseline achieve state-of-the-are on number of OSR benchmarks. Similarly, we boost the performance of the existing state-of of-the-art method by improving its closed-set accuracy, but the resulting discrepancy with the strong baseline is marginal. Our third contribution is to present the 'Semantic Shift Benchmark' (SSB), which better respects the task of detecting semantic novelty, opposed low-level distributional shifts as tackled by neighbouring machine learning fields. On this new evaluation, we again demonstrate that there is negligible difference between the strong baseline and the existing state-of-the-art. Code available at: https://githu..com/ http.//o//wgweeosssoeoooooo closeed__eeee__eeeeeee _ettdsset___eee aallllllll Saall_yu___eeee _eed.
INTRODUCTION
Given the success of modern deep learning systems on closed-set visual recognition tasks, a natural next challenge is open-set recognition (OSR) (Scheirer et al., 2013). In the closed-set setting, a model is tasked with recognizing set of categories that remain the same during both training and testing phases. In the more realistic open-set setting, a model must not only be able to distinguish between the training classes, but also indicate if an image comes from class it has not yet encountered.
The OSR problem was initially formalized (Scheirer et al., 2013) and has since inspired a rich line of research (Bendale & Boult, 2016; Chen al., 2020a; Ge al., 2017: Neal et al., 2018; Sun et al., 2020; Zhang et al., 2020; Shu et al., 2020). The standard baseline for OSR is model trained with the cross-entropy loss on the known classes. At test time, the maximum value of the softmax probability vector is used to decide if an input belongs to the known classes or not. We henceforth refer to this method as the 'baseline' or 'maximum softmax probability (MSP) baseline'. Most existing literature reports significantly outperforming this OSR baseline on standard benchmarks of re-purposed image recognition datasets, including MNIST (LeCun et al., 2010) and TinyImageNet (Le & Yang, 2015).
In this paper we reappraise these approaches, by asking whether a well-trained closed-set classifier can perform as well as recent algorithms, and by analyzing the benchmark datasets. To do this, we first investigate the relationship between the closed-set and open-set performance of classifier (sec. 3). Though one may expect stronger closed-set classifiers to overfit to the training classes (Recht al., 2019; Zhang al., 2017). and so perform poorly for OSR, and open-set performance are highly correlated. We show this trend holds across datasets, objectives and model architectures, and further demonstrate the trend on an IeageNet-scale evaluation.
show instead that the closed-set

---

Published as a conference paper at ICLR 2022
85
Known Classes
Unknown Classes
Open Set Purfomance (AUROC)
80
SOTA
Carrent
70-
Go
65
60-
Proposed
(8)
Previously
Oues
(b)
Figure 1: (a) We show that we can push OSR baseline performance to be competitive with or surpass state-of-the-art methods (shown, ARPL + CS (Chen et al., 2021)). (b) We propose the 'Semantic Shift Benchmark' datasets for OSR, which are larger scale and give precise definitions of what constitutes a 'new class'.
Secondly, following this observation, we show that the open-set performance of a classifier can be improved by enhancing its closed-set accuracy, tapping the numerous recent advances in image classification (Loshchilov & Hutter, 2017; Szegedy et al., 2016; Cubuk et al., 2020; Bello et al., 2021). Specifically,
introduce strategies such more augmentation, better learning rate schedules and label smoothing, that significantly improve the closed-set performance of the MSP baseline (sec. 4). We also propose the of the maximum logit score (MLS), rather than normalized softmax probabilities, as an open-set indicator. With these adjustments, we push the baseline to become competitive with or outperform state-of-the-ar OSR methods, substantially outperforming the currently reported baseline figures. Notably, we surpass state-of- of-the-art figures on four of the six OSR benchmark datasets.
Furthermore, we transfer these improvements to two previous OSR methods, including the current state-of-the-are from (Chen et al., 2021). While this does boost its performance, we observe that there is negligible difference with that of the improved 'MLS' baseline (see fig. 1a). This finding is important because allows us to better assess recent reported progress the area.
Finally, we turn to the experimental setting for OSR (sec. 5). Current OSR benchmarks are both small scale and lack a specific definition of what constitutes a 'visual class'. As an alternative, we propose the 'Semantic Shift Benchmark' suite (SSB). We propose the use of fine-grained datasets including CUB (Wah et al., 1), Stanford Cars (Krause et al., 2013) and FGVC-Aircraft (Maji et al., 2013) which all have clear definitions of semantic class (see fig. 1b), well evaluation based on the full ImageNet database (Ridnik et al., 2021). Furthermore, we construct open-set splits with explicit focus on semantic novelly, which hope better separates this avenue of research from related machine learning sub-fields such as out-of-diitribution (Hendrycks & Gimpel, 2017) and anomaly detection (Kwon et al., 2020). Our proposed splits also offer a better way of quantifying open-set difficulty; we find that different splits lead to much larger discrepancy in open-set performance than the current measure of open-set difficulty 'openness' (Scheirer al., 2013), which focuses only on the number of open-set classes. We evaluate our strong baseline as well as the state-of-the-an art method on this new configuration to encourage future research this direction.
ImageNet-scale
2 RELATED WORK
Open-set recognition. Seminal work in (Scheirer et al., 2013) formalized the task of open-set recognition, and has inspired a number of subsequent works in the field. (Bendale & Boult, 2016) introduced the first deep learning approach for OSR, OpenMax, based on the Extreme Value Theory (EVT). GANs have also been used to tackle the task (Ge al., 2017; Neal et al., 2018). OSRCI (Neal et al., 2018) generates images similar to those in the training set but that do not belong to any of the known classes, and uses the generated images to train an open-set classifier. This work also established the existing OSR benchmark suite. (Kong & Ramanan, 2021) achieve strong OSR performance by using an adversarially trained discriminator to delineate closed from open-set images, leveraging real open-set images for model selection. Other approaches include reconstruction based methods (Yoshihashi et al., 2019; Oza & Patel, 2019; Sun et al., 2020) which use poor rest-time reconstruction as an open-set indicator, and prototype-based methods (Shu et al., 2020; Chen et al.,
2

---

Published as a conference paper at ICLR 2022
2020a; 2021) which represent known classes with learned prototypes, and identify open-set images based on distances to the prototypes.
State-of-the-art. In this work, we compare against methods which achieve state-of-the-as in the controlled OSR setting (with no extra data for training or model selection, for instance as demonstrated in (Kong& & Ramanan, 2021)). To knowledge, these methods are ARPL (Adversarial Reciprocal Point Learning) (Chen et al., 2020a; 2021) and OpenHybrid (Zhang et al., 2020), which we detail in sec. 3.1 and sec. respectively. In this paper, we show that the MSP baseline can be competitive with or outperform the more complex methods listed above. Finally, we note recent works (Zhou et al., 2021; Miller et al., 2021: Guo et al., 2021) with which we do not compare as they report lower performance than ARPL and OpenHybrid.
Related subfields. OSR also closely related out-of-distribution (OoD) detection (Hendrycks & Gimpel, 2017; Liang et al., 2018: Hsu et al., 2020), novelty detection (Abati et al., 2019; Perera et al., 2019; Tack et al., 2020), anomaly detection (Hendrycks et al., 2019; Kwon et al., 2020; Bergman & Hoshen, 2020) and novel category discovery (Han et al., 2019; 2020; 2021). Amongst these, OoD is perhaps the most widely studied and is similar nature to OSR. As noted by (Dhamija et al., 2018; 8; Boult et al., 2019), OSR is similar to the OoD problem with an additional multi-way classification component between known categories. In fact, there is currently significant overlap in the evaluation datasets between these settings, though cross-setting comparisons are difficult due to different evaluation protocols. Specifically, the OoD setting permits the use of additional data as examples of "Ool"' data during training. (Chen et al., 2021) and (Zhang et al., 2020) evaluate their OSR methods on OoD benchmarks, with both showing competitive results despite not having access to additional data during training. In this paper, we distinguish the OSR problem from OoD and other related fields by proposing a new suite of benchmarks. While OoD encompasses all forms of distributional shift, including those based on low-level features, OSR specifically refers to semantic novelty. We propose new benchmarks that respect this distinction.
3 CORRELATION BETWEEN CLOSED-SET AND OPEN-SEN PERFORMANCE
One may expect that stronger closed-set classifiers have overfit their learned representations the closed-set categories, and thus perform poorly for OSR (Recht et al., 2019; Zhang et al., 2017). Furthermore, existing literature largely considers the closed and open-set tasks separately, with works generally emphasising good open-set performance despite no degradation in closed-set accuracy (Neal et al., 2018; Zhou al., 2021; Miller et al., 2021). On the contrary, in this section we show that the closed-set and open-set performance of classifiers are strongly correlated. We first demonstrate this for the baseline and a state-of-the-ar the-art method on the standard OSR benchmarks (sec. 3.1) and then large scale evaluation across a number of model architectures (sec. 3.2).
Open-set recognition. We formalize the problem of OSR, and highlight its differences from closed- set recognition. First, consider labelled training set for a classifier Durai (x,, y1)) C X x C. Here, X' is the input space (e.g., images) and C is the set of 'known' classes. In the closed-set scenario, the model is evaluated on test set in which the labels
also drawn from the same set of classes, i.e., Diest = {(x,, y1) ) M C X' x C. In the closed-set setting, the model returns a distribution over the known classes as p(y Conversely, in OSR, test images may also come from unseen classes U4 giving Dusstopenn (x,, yt) ) C X' x (C (Z). In the open-set set setting, in addition to returning the distribution p(y Xx, C) over known classes, the model also returns a score S(y E C)x) to indicate whether or not the test sample belongs to any of the known classes.
3.1 BASELINE AND STATE-OF-THE-ARRY ON STANDARD BENCHMARKS
We first experiment with three representative open-set recognition methods across the standard benchmark datasets in the literature (Neal et al., 2018; Oza Patel, 2019; Sun et al., 2020; Chen et al., 2020a; Zhang et al., 2020). The methods include the standard MSP baseline as well as two variants of ARPL (Chen et al., 2021). We use the standard network from the open-set literature (Neal et al., 2018), lightweight model similar to the VGG architecture (Simonyan & Zisserman, 2015) which we henceforth refer to as 'VGG32' (refer to appendix D for details). The three methods are summarised below, followed by a description of the most commonly used benchmarks.
Methods. Maximum Softmax Probability (MSP, baseline): The model is trained for closed-set classification using the cross-entropy loss between one-hot target vector and the softmax output
3

---

Published as a conference paper at ICLR 2022
Figure 2: Correlation between closed set performance (accuracy) and open-set per- formance (AUROC). We train three methods on the standard open-set benchmark datasets, including the MSP baseline, ARPL and ARPL + CS (Chen et al., 2021). Foreground points in bold show results averaged across five 'known/unknown' class splits for cach method-dataset pair (following standard prac- tise in the OSR literature) while background points, shown feint, indicate results from the underlying individual splits.
100
Melhod MSP ARPI ARPL:CS
Open Set Performance (AUROC)
90-
85-
Datser MNIST
80
SVIIN CIFRR CIFAR 50 TisykuugeNs
75-
70
75
80
85
90
95
100
Closed Performance (Ascuracy)
p(y]x) of the classifier. This training strategy, along with the of the maximum softmax probability as Sly E C|x) = maX x())x), is widely used in both the OSR and OoD literature as a baseline (Hendrycks & Gimpel, 2017). ARPL (Chen al., 2021): This method an extension of the recent RPL (Reciprocal Point Learning) optimization strategy (Chen et al., 2020a). Here, the probability that a sample belongs to a class is proportional to its distance from a learned 'reciprocal point' in the feature space. A reciprocal point aims to represent 'otherness' with respect to class, with the intuition being that open-set examples are different to all known classes. ARPI, extends RPI. by computing feature distances as the sum of both the Euclidean and cosine distances. In this case, S(y E Clx) is equal to the maximum distance in feature space between the image and any reciprocal point. ARPL + CS (Chen et al., 2021) augments ARPL with 'confusing samples': adversarially generated latent points to stand in for 'unseen class' samples. The confusing samples are encouraged to be equidistant from all reciprocal points, with the same open-set scoring rule used as in ARPL. We train both ARPL and ARPL + CS based on the official public implementation (Chen et al., 2021).
Datasets. We train the above methods the standard benchmark datasets for open-set recognition. In all cases, the model is trained on a subset of classes, while other classes are reserved as 'unseen' for evaluation. MNIST (LeCun et al., 2010). SVHN (Netzer et al., 1), CIFAR1O (Krizhevsky, 2009): These are ten-class datasets, with MNIST and SVHN containing images of hand-written digits and street-view house numbers respectively. Meanwhile, CIFARIO is a generic object recognition dataset containing natural images from ten diverse classes including animals and vehicles. In these cases, the open-set methods are evaluated by training on six classes, while using the other four classes for testing (IC) = 6; [44| = 4). CIFAR + N (Krizhevsky, 2009): In an extension to the CIFAR 10 evaluation protocol, open-set algorithms are benchmarked by training on four classes from CIFAR 10, while using N classes from CIFAR100 for evaluation. where N denotes either 10 50 classes (CC) = 4: [4| E {10,50}). FinyImageNee (Le & Yang, 2015); In the final and most challenging case, exisiting open-set algorithms are evaluated on the TinylmageNet dataset. This dataset contains 200 classes sub-sampled from ImageNet (Russakovsky et al., 2015), with 20 classes used for training and 180 as unknown (IC) = 20; [4] = 180).
Experimental setup. At test time, the model is fed test images from both known and novel classes, and is tasked with making a binary 'known/unknown' decision on a per-image basis. Following standard practise in the OSR literature, the threshold-free area under the Receiver- -pperato curve (AUROC) is used as an evaluation metric. We train with the same hyper-parameters as (Chen et al., 2021) and, following standard practise, train on five different splits of closed and open-set classes for each dataset and method combination. When evaluating on existing benchmarks throughout this paper, we use the same data splits as (Chen et al., 2021).
Fig. 2 gives the AUROC (open-ser set performance) against the Top- multi-way classification accuracy (closed-set performance). We show the averaged results as well as the individual split results, omitting the CIFAR+10 setting for clarity (as the scatter points are almost coincident with the CIFAR+50 setting). It is clear that there is positive correlation between the closed-set accuracy and open-set performance: we find a Pearson Product-Moment correlation = 0.95 between the accuracy and AUROC, indicating a roughly linear relationship between the two metrics.
Results.
Discussion. To justify our findings theoretically, we look to the model calibration literature (Guo al., 2017). Intuitively, model calibration aims to quantify whether the model 'knows when it doesn't 'tt know", in that low confidence predictions are correlated with high error rates. Specifically, assume classifier, fx), returns probabilities for each class, making predictions as ý arg max fxx).
4

---

Published as a conference paper at ICLR 2022
AURROC (Dyensee Petommnce))
0.825
VIT-B-16
AUROC (Ogem set Pedornumc))
0.825
Hard
0.800
Bany
ResNot18
0.775
0.775
ResNe134
0,150
0,750
ResNe150
RetNe1111
0.725
ResNa152
0.700
0.700
0.675
0.075
0.650
0.675
0.700
0,725
0,750
0.775
0,800
0.825
0250
0.70
0.72
0.74
0.76
0.78
Accuncy
Accuracy Perforsence)
(a)
(b)
Figure 3: (a) Open-set results range of architectures the ImageNet dataset. 'Easy' and 'Hard' OSR splits are constructed from the ImageNet-21K-P dataset. (b) ImageNet open-set results within a single model family (ResNet).
Further assume labelled input-output pairs, (x,y) C X' X C, where C is the label space. Then, the classifier is said to be perfectly calibrated if:
P - v)f((( - pp) p Vp [0
=
It is further true that if a classifier is trained with proper scoring rule (Gneiting et al., 2007) on infinite data, then the classifier will perfectly calibrated the loss function's minimum (Minderer et al., 2021). Many losses used to train deep networks are proper scoring rules (e.g., the cross-entropy loss). Thus, assuming that generalization error on the test set correlated with the infinite-data loss value, we would suspect models with lower generalization (test) error to be better calibrated. If we use low-confidence predictions as an indicator that a test sample belongs to a new semantic class, we would expect stronger models to be better open-set detectors.
3.2 LARGE-SCALE EXPERIMENTS AND ARCHITECTURE ABLATION
So far, we have demonstrated the correlation between closed and open-set performance on single, lightweight architecture and on small scale datasets though we highlight that they are the standard existing benchmarks in the OSR literature. Here, we experiment with range of architectures large-scale dataset (lmageNet).
We experiment with architectures from number of popular model families, including VGG (Simonyan Zisserman, 2015), ResNet (He et al., 2016) and EfficientNet (Tan Le, 2019). We further include results for the recently proposed non-convolutional VIT (Dosovitskiy al., 2021) and MLP-Mixer (Tolstikhin et al., 2021; Melas-Kyriaze 2021) models. All models were trained with the cross-entropy objective for classification.
Methods.
Dataset. For large-scale evaluation, we leverage the recently released ImageNet-22 22K- (Ridnik al., 2021). This dataset contains subset of the full ImageNet database, processed and standardized to remove small classes and leaving around 1 11K object categories. Note that ImageNet-21 2-P is strict superset of ImageNet-IK (ILSVRCI2). As such, models are trained the standard 1000 classes from ImageNet-Ik K, and we select two 1000-category subsets from the disjoint categories in ImageNet-2 1 K-P as the open sets. Differently to existing practise the standard datasets, our two open-set splits for ImageNet are not randomly sampled, but rather designed to be 'Easy" and 'Hard' based on the semantic similarity of the open-set categories the training classes. In this way we better capture a model's ability to identify semuntic novelty opposed to low-level distributional shift. This idea and split construction details are expanded upon sec. 5. For both 'Easy" and 'Hard' splits, we have [C] = 1000 and [ 1000.
Results. Fig. 3a shows our open-set results on ImageNet. Once again, we find a positive correlation between closed and open-set performance. In this case we find the linear relationship to be weaker, with p = 0.88 for the 'Hard' evaluation and p 0.63 for the 'Easy'. This unsurprising given the large discrepancy in architecture styles. In general, remarkably better for OSR than others. The exception is the ViT model (highlighted), which bucks the OSR trend for both 'Fasy" and 'Hard' splits. When looking within single model family,
do not find any particular model family be
5

---

Published as a conference paper at ICLR 2022
find the linear relationship to be substantially strengthened. Fig. 3b demonstrates the trend within the ResNet family, with p = 1.00 and p = 0.99 for the 'Easy' and 'Hard' OSR splits respectively.
Discussion. We again note that the ViT model, despite its size (86M parameters) and few inductive biases (no convolutions), does not overfit its representation to the training classes. The fact that outperforms the OSR trend supports recent findings the benefits of purely attention-based vision models (including similar findings in (Fort et al., 2021)), as well as the benefits of good closed-set performance for OSR. Finally, we note the practical utility of our findings in sec. 3. Namely, the fact that the open and closed-set performance are correlated allows OSR to readily improve with the extensive research in standard image recognition.
4 A GOOD CLOSED-SET CLASSIFIER IS ALL YOU NEED?
In this section, we demonstrate that we can leverage the correlation established in sec. 3 improve the performance of the baseline OSR method. Specifically, we improve the closed-set accuracy of the maximum softmax probability (MSP) baseline and, in doing so, make it competitive with or stronger than state-of-the the open-set models. Specifically, we achieve new state-of-the-ee figures on four of the six OSR benchmarks.
We find that we can significantly improve the MSP baseline performance by leveraging techniques from the image recognition literature, such as longer train- ing, better augmentations (Cubuk et al., 2020) and label smoothing (Szegedy et al., 2016). Fig. 4 shows how open-set performance of the baseline model in- creases
AUROC (Operesat Parfarmmnc)
76- 72-
introduce these changes the Tiny-
ImageNet benchmark. For example: longer training (scatter point 7 scatter point 8); better augmentations (3 5); and ensembling (8- 9). Full details and a tabular breakdown of the methods used to increase closed-set performance can be found in appendix C.
73
Accuncy (Closod Set Performace))
Figure 4: Gains in open-set performance closed-set performance increases on TinyImageNet.
We take these improved training strategies and train the VGG32 backbone on the standard benchmark datasets. We train all models for 600 epochs with a batch size of 128, training models on a single NVIDIA Titan GPU. We do not include ensemble results for fair comparison with previous methods. Full training strategies and implementation details can be found in appendices C and D. We report our results as 'Baseline (MSP+)' in table
Logit scoring rule. Next, we also change the open-set scoring rule. Previous work has noted that open-set examples tend to have lower feature norms than closed-set ones (Dhamija et al., 2018; Chen et al., 2021). As such, we propose the use of the maximum logit score (MLS) for the open-set scoring rule. Logits are the raw outputs of the final linear layer in a deep classifier, before the softmax operation normalizes these such that the outputs can be interpreted as a probability vector summing to one. As the softmax operation normalizes out much of the feature magnitude information present in the logits, we find logits lead better open-set detection results. We provide detailed analysis and discussion of this effect in appendix B. We further provide a more general study of the representations learned with cross-entropy models, including visualizations of the learned feature space. We present results of our maximum logit score baseline as 'Baseline (MLS)' in table 1.
We compare against OpenHybrid (Zhang al., 2020) and ARPL CS (Chen et al., 2021), which hold state-of-the-ar performances on the standard datasets in the controlled setting (with no extra data for training or model selection). We also compare against OSRCI (Neal et al., 2018), which established the current OSR benchmark suite. While OSRCI and +RPL+ CS have been described in sec. 2 and 3.1 respectively, OpenHybrid tackles the open-set task by training a flow-based density estimator top of the classifier's 's feature representation, jointly training both the encoder and density model. In this way, a distribution over the training data log (xx is learned, which is used to directly provide S(y E C)x). Comparisons with more methods be found in appendix E.
We find that our MLS baseline substantially improves the previously reported baseline figures, with an average absolute increase in AUROC of 15.6% across the datasets. In fact, MLS surpasses the existing ofate-of-the-ar the-art on the SVHN, CIFAR+10, CIFAR+50 and TinyImageNet benchmarks and is, on average, 0.7% better across the entire suite.

---

Published as a conference paper at ICLR 2022
Table 1: Comparisons of our improved baselines (MSP+, MLS) against state-of-the-art meth- ods on the standard OSR benchmark datasets. All results indicate the area under the Receiver- Operator curve (AUROC) averaged over five 'known/unknown' class splits. '+' indicates prior
methods augmented with improved closed-set optimization strategies, including: MSP+ (Neal et al.,
2018), OSRCI+ (Neal et al., 2018) and (ARPL + CS)+ (Chen et al., 2021).
Method
MNIST
SVHN
CIFARIO
CIFAR 10
CIFAR 50
TinyImageNet
Baseline (MSP) (Neal et al., 2018)
97.8
88.6
67.7
81.6
80.5
57.7
OSRCI (Neal al. 2018)
98.8
91.0
69.9
83.8
82.7
58.6
OpenHybrid (Zhang et al., 2020)
99.5
94.7
95.0
96.2
95.5
79.3
ARPL + CS (Chen et al., 2021)
99.7
96.7
91.0
97.1
95.1
78.2
OSRCI+
98.5 (-0.3)
89.9 (-1.1)
87.2 (+17.3)
91.1 (+7.3)
90.3 (+7.6)
62.6 (+4.0)
(ARPI. CS)+
99.2 (-0.5)
96.8 (+0.1)
93.9 (+2.9)
98.1 (+1.0)
96.7 (+1.6)
82.5 (+4.3)
Baseline (MSP+) Baseline (MLS)
98.6 (+0.8)
96.0 (+7.4)
90.1 (+22.4)
95.6 (+14.0)
94.0 (+13.5)
82.7 (+25.0)
99.3 (+1.5)
97.1 (+8.5)
93.6 (+25.9)
97.9 (+16.3)
96.5 (+16.0)
83.0 (+25.3)
We also take the OSRCI and ARPL CS algorithms (Neal et al., 2018; Chen et al., 2021), and augment them with our proposed training strategies for a fair comparison, reporting the results under OSRCI+ and (ARPL + CS)+. Specifically, we train them for longer, include label smoothing and use better data augmentations (see appendix D for full details). We also trained OpenHybrid in this controlled setting, but significantly underperformed the reported performance. This likely because the method was trained for 10k epochs and with a batch size of 1024, which are both 10x larger than those used in these experiments. Note that, despite this, the stronger baseline still outperforms OpenHybrid in number of cases.
In almost all cases we are able to boost the open-set performance of OSRCI and ARPL+CS, especially for the former. In the case of (ARPL+CS)+, we achieve new state-of-the-ar results on the CIFAR+10 and 50 benchmarks, and also report a 4.3% boost on TinylmageNet. However, we note that on average, (ARPL+CS)+ is almost indistinguishable from the improved MLS baseline (with 0.03% difference in average open-set performance).
Discussion. A number of increasingly sophisticated methods have been proposed for OSR in recent years. Typically, proposed methods have carefully tuned training strategies and hyper-parametert such as custom learning rate schedules (Zhang al., 2020), non-standard backbones (Guo al., 2021) and novel data augmentations (Zhou et al., 2021). Meanwhile, the closed-set accuracy of the methods is often unreported. As such, is difficult to delineate what proportion of the open-set performance gains come from increases in closed-set accuracy. Our findings in this section suggest that many of the gains could equally be realised through the standard baseline. Indeed, in sec. we propose new evaluation protocols and find that once the closed-set accuracy of ARPL. and the baseline are made comparable, there negligible difference in open-set performance. We further experiment on OoD benchmarks in appendix F and report similarly improved baseline performance.
5 SEMANTIC SHIFT BENCHMARK
Current OSR benchmarks have two drawbacks: (1) they all involve small scale datasets; (2) they lack clear definition of what constitutes a 'semantic class'. The latter is important to delineate the open-set field from other research questions such as out-of-distribttion detection (Hendrycks & Gimpel, 2017) and anomaly detection (Kwon et al., 2020). Specifically, OSR aims to identify whether a test image is semantically different to the training classes, not whether, for example, the model is uncertain about its prediction or whether there has been low-level distributional shift,
To address these issues, we propose a new suite of evaluation benchmarks. In this section, we first detail a large-scale ImageNet evaluation (introduced in sec. 3.2) before proposing three evaluations on fine- grained datasets which have clear definitions of a semantic class. Differently to previous work, our evaluation settings all aim to explicitly capture the notion of semantic novelty. Finally, we benchmark MI.S and ARPL. on the new benchmark suite to motivate future research.
5.1 PROPOSED BENCHMARK DATATSETS
ImageNet. We introduce a large-scale evaluation for category shift, with open-set splits based on semantic distances to the training set. Specifically, we designate the original ImageNet-Ie 1K classes for the closed-set, and choose open-set classes from the disjoint set of ImageNet- et al., 2021). We exploit the hierarchical, tree-like semantic structure of the ImageNet database. For
(Ridnik
7

---

Published as a conference paper at ICLR 2022
Figure 5: Open-set class pairs for CUB. For three difficulties [ ('Easy' (green/left), 'Medium (orange/middle), 'Hard' (red/right)), we show an image from an open-set class (right) and its most similar closed-set class (left). Note that the harder the difficulty, the more visual features (e.g., foot colour or bill shape) the open-set class has in common with the closed-set. Further examples can be found in appendix H.
instance, the class 'elephant' can be labelled multiple levels of semantic abstraction ('elephant', "placental', 'mammal', 'vertebrate', 'animal"). Thus, for each pair of classes between ImageNet-IKK IK and ImageNet-21I--P we define the semantic distance between two classes as the total path distance between their nodes in the semantic tree. We then approximate the total semantic distance from the ImageNet-22 2oggeNett-21-PP classes the closed-set by summing distances all ImageNet- IK classes. Finally, we select "Fasy" and 'Hard' open-set splits by sorting the total distances to the closed-set and selecting two sets of 1000 categories. We note that the larger ImageNet database has been used for OSR research previously (Bendale & Boult, 2016; Kumar al., 2021: 1: Hendrycks et al., 2021). However, we structure explicitly for semantic similarity with ImageNet- IK similarly to concurrent work in (Sariyildiz et al., 2021).
Fine-grained classification datasets. Consider the properties of fine- Ene-grained visual categorization (FGVC) datasets. These datasets are defined by 'entry level' category, such as flowers (Nilsback & Zisserman, 2008) or birds (Wah et al., 2011). Within the dataset, all classes are variants of that single category, defining single axis of semantic variation, e.g., 'bird species' in the case of birds. Because the axis of variation is well defined, it is reasonable to expect a classifier to learn it given a number of example classes- namely, to learn what bird species are and how they can be distinguished.
Contrast FGVC datasets with the current OSR benchmarks, such as the CIFAR+10 evaluation. In this case, model is trained on four CIFARIO classes such (airplane, automobile, ship, truck), J. all of which could be considered 'entry level', before having to identify images from CIFAR100 classes such as (bicyele, bee, porcupine, baby) as belonging to new classes. In this case, the axis of variation is much less specific, and it is uncertain whether the OSR model is responding to a true semantic signal or simply low-level distributional shifts in the 'unseen' data. Furthermore, because of the small number of training classes in the current benchmark settings, it is unrealistic for a classifier to learn such high-level class definitions. We give an illustrative example of this in appendix G.
As result, we propose three FGVC datasets for OSR evaluation: Caltech- UCSD Birds (CUB) (Wah et al., 2011), Stanford Cars (Krause et al., 2013) FGVC-Aireraft (Maji et al., 2013). These datasets come with labelled attributes (e.g.. has bi f1ll shape: hhokedd in CUB), which can be used to characterize the differences between classes and thus the degree of semantic shift. We use attributes construct open-set FGVC class splits which with the difficulty depending on the similarity of labelled visual attributes with any of the training classes. We sketch the split-construction process for CUB here and refer to appendix H for more details on Stanford Cars and FGrC-Aircraft.
binned into 'Easy', 'Medium' and 'Hard' classes,
Every image in CUB is labelled for the presence of 312 visual attributes such as has bill wtlbb_lhappee:_____a__aa_aaaaaaa :hokeed fas_breast Dareast_color:: color:
Table 2: Statistics the Semantic Shift Benchmark. We show "CClasses(FTed Images) for the known classes, and for the 'aayy', 'Medium' and 'Hard' open-set classes.
and
Datsor CLB RIVC-AAcee
Known
Fasy
Medium
Hard
This in- formation aggregated for each class, resulting in matrix M E 00, 1)xA, describing the frequency with which each attribute appears in each class.
:yello..
(915)
(1004)
InagsNet
1000 (50000)
1000
1000 (50000)
Treating each row in M as semantic class descriptor, this allows us to compute the semantic similarity of every pair of classes and, given a set of closed-set classes, identify which remaining classes are 'Easy", 'Medium' and 'Hard' (least to most similar) with respect to the closed-set. Examples of 'Easy', 'Medium' and 'Hard' open-set classes, along with their closest class in the closed-set, are shown in fig. 5 for CUB.
8

---

Published as a conference paper at ICLR 2022
We note that fine-graine fine-grained OSR has been demonstrated in (Chen et al., 2021; 2020a) on dataset of 300 aircraft classes. However, this dataset does not come with labelled attributes, making it harder to construct open-set splits with varying levels of semantic similarity to the training set, which is our focus here. Finally, while prior works have recognised the difficulty of OoD detection for more fine-grained data (Bodesheim et al., 2015; Perera & Patel, 2019; Lee al., 201 8a), we propose them for OSR because of their clear defimition of a semantic class rather than their increased difficulty. A further discussion of these ideas is presented in appendix G. We provide statisties of the splits from all proposed datasets in table 2, and the splits themselves in the supplementary material.
5.2 BENCHMARKING FOR OPEN-SET RECOGNITION
Evaluation Protocol. For the 'known/unknown' class decision, we report AUROC as is standard practise, as well as accuracy to allow potential gains in open-set performance to be contextualized in the closed-set accuracy of model. We also report Open-Set Classification Rate (OSCR) (Dhamija et al., 2018) which measures the trade-off between accuracy and open-set detection rate as a threshold on the confidence of the predicted class is varied. We report results on 'Easy' and 'Hard' splits for all datasets, combining 'Medium' and 'Hard' examples into a single bin when applicable.
In fine-graid fine-grained classification, it is standard to pre-train models on ImageNet. This unsuitable for the proposed fine-grained OSR setting, as ImageNet contains overlapping classes with the proposed datasets. Instead, we pre-train the network on Places (Zhou et al., 2017) using MoCoV2 self- supervised weights (Chen al., 2020b; Zhao et al., 2021). For the ImageNet benchmark, we can train with labels on the ImageNet-Iee IK dataset and evaluate on the unseen classes. We finetune the ARPL model from a pre-trained ImageNet checkpoint.
Results. In table 3 we test MLS and ARPL+ (Chen et al., 2021) using a ResNet50 backbone on the proposed benchmarks (we found ARPL CS to be prohibitively expensive to train in this setting, see appendix D for details). The results corroborate the trends found in sec, 4: strong closed-set classifiers produce open-set results with good AUROC performance, and the MLS baseline performs comparably to the state-of-the-ar method. In fact, while we find ARPL+ achieves slightly better AUROC on the ImageNet benchmark, MLS outperforms in terms of OSCR across the board.
Finally, more careful consideration of the semantics of the open-set classes leads to harder splits significantly reducing OSR performance. This is in contrast to 'openness' (Scheirer et al., 2013), the current measure used to assess the difficulty of an OSR problem. dependent on the ratio of the number of closed open-set classes. For instance, in the ImageNet case, we find the harder split to be lead to around 6% worse AUROC for both methods. We also experimented with randomly subsampling first 1K and then 10K open-set classes, finding that introducing more classes during evaluation only reduced open-set performance by around 0.6% (10x less than our proposed splits).
Table 3: OSR results on the Semantic Shift Benchmark. We measure the closed-set classification accuracy and AUROC on the binary open-set decision. We also report OSCR, which measures the trf between open and closed-set performance. OSR results shown on 'Easy / Hard' splits.
Method
CUB
SCars
RGYC-Airenmh
Image
Acc.
AUROC
Acc.
AUROC
OSCR
Acc.
AUROC
OSCR
Acc.
AUROC
OSCR
ARPL+ MLS
85.9
83.5/ 75.5
76.0/ 69.6
96.9
94.8/83.6
92.8/82.3
91.5
87.0/77.7
83.3/ 74,9
78.1
79.0 73.6
65.9/62.6
86.2
89,3/79.3
79.8/73.1
97.1
94.0/82.2
92.2/8111
91.7
90.7/82,3
868879 76,8
78.8
78.2/72.6
66.1/62.7
6 CONCLUSION
In this work we have demonstrated a strong correlation between the closed-set and open-set perfor- mance of models for the task of open-set recognition. Leveraging this finding, we have demonstrated that a well-trained closed-set classifier, using the maximum logit score (MLS) at test-time, can be competitive with or outperform existing state-of-the-art methods. Though we believe OSR is a critical problem which requires further investigation, our findings give us insufficient evidence to reject our titular question of 'is a good closed-set classifier all you need?'. We have also proposed the 'Semantic Shift Benchmark' suite, which isolates semantic shift from other low-level distributional shifts. Our proposed benchmark suite allows controlled study of semantic novelty, including stratification of the degree of semantic shift.

---

Published as a conference paper at ICLR 2022
ACKNOW
We would like to thank Andrew Brown for many interesting discussions on this work. This research funded by Facebook AI Research Scholarship, Royal Society Research Professorship, and the EPSRC Programme Grant VisualAI EP/128572/1
ETHICS STATEMENT
Open-set recognition of immediate relevance the safe and ethical deployment of machine learning models. In real-world settings, it is unrealistic to expect that all categories of interest to the user will be represented in the training set. For instance, in autonomous driving scenario, forcing the model to identify every object as an instance of a training category could lead it to make unsafe decisions.
When considering potential negative societal impacts this work, we identify the possibility that OSR research may lead to complacent consideration of the training data. As we have demonstrated, OSR models are far from perfect and cannot be exclusively relied upon in practical deployment. As such, it remains of critical importance to carefully curate training data and ensure its distribution is representative of the target task.
Finally, we comment on the dataset privacy considerations for the existing and proposed benchmarks. datasets are licensed for academic/non--commrriii research. However, CIFAR, TinylmageNet and ImageNet contain some personal data for which consent was likely not obtained. The proposed FGVC datasets have the added benefit of containing no personal information.
REFERENCES
Davide Abati, Angelo Porrello, Simone Calderara, and Rita Cucchiara. Latent space autoregression for novelty detection. In CVPR, 2019.
Irwan Bello, William Fedus, Xianzhi Du, Ekin Cubuk, Aravind Srinivas, Tsung- Lin, Jonathon Shlens, and Barret Zoph. Revisiting resnets: Improved training and scaling strategies. arXiv preprint arXiv:2 103.092003.0757, 2021.
Abhijit Bendale and Terrance E. Boult. Towards open deep networks. In CVPR, 2016.
Liron Bergman and Yedid Hoshen. Classification based anomaly detection for general data. In ICLR, 2020.
Paul Bodesheim, Alexander Freytag, Erik Rodner, and Joachim Denzler: Local novelty detection in multi-class recognition problems. In WACV, 2015.
Terrance Boult, Steve Cruz, Akshay Raj Dhamija, Manuel Günther, James Henrydoss, and Walter J. Scheirer. Learning and the unknown: Surveying steps toward open world recognition. In AAAI, 2019.
Guangyao Chen, Limeng Qiao, Yemin Shi, Peixi Peng. Jia Li, Tiejun Huang, Shiliang Pu, and Yonghong Tian. Learning open set network with discriminative reciprocal points. In ECCV, 2020a.
Guangyao Chen, Peixi Peng, Xiangqian Wang, and Yonghong Tian. Adversarial reciprocal points learning for open recognition. IEEE TPAMI, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.00 042900 2020b.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In CVPR, 2014.
Ekin Dogus Cubuk, Barret Zoph. Jon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with reduced search space. In NeurlPS, 2020.
Akshay Raj Dhamija, Manuel Günther, and Terrance E. Boult. Reducing network agnostophobia. In NeurIPS, 2018
10

---

Published as a conference paper at ICLR 2022
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Xuefeng Du, Zhaoning Wang. Mu Cai, and Yixuan Li. Vos: Learning what you don't know by virtual outlier synthesis. ICLR, 2022.
Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of out-of-distribution detection. ICML Workshop on Uncertainty & Robustness in Deep Learning. 2021.
Zongyuan Ge, Sergey Demyanov, and Rahil Garnavi. Generative openmax for multi-class open set classification. In BMVC, 2017.
Tilmann Gneiting, Fadoua Balabdaoui, and Adrian Raftery. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society, 2007.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. ICML, 2017.
Yunrui Guo, Guglielmo Camporese, Wenjing Yang. Alessandro Sperduti, and Lamberto Ballan. Conditional variational capsule network for open set recognition. ICCV, 2021.
Kai Han, Andrea Vedaldi, and Andrew Zisserman. Learning to discover novel visual categories via deep transfer clustering. In ICCV, 2019.
Kai Han, Sylvestre-Alviss Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Automatically discovering and learning new visual categories with ranking statistics. In ICLR, 2020.
Kai Han, Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Autonovel: Automatically discovering and learning novel visual categories. IEEE TPAMI, 2021.
Kaiming He, Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR, 2017.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. ICLR, 2019.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. CVPR, 2021.
Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting out-of- distribution image without learning from out-of-distribution data. In CVPR, 2020.
Shu Kong and Deva Ramanan. Opengan: Open-set recognition via open data generation. ICCV, 2021.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In International IEEE Workshop on 3D Representation and Recognition (3dRR), 2013.
Alex Krizhevsky. Learning multiple layers of features from tiny images. University of Toronto, 2009.
Pulkit Kumar, Anubhav, Abhinav Shrivastava, and Shu Kong. Open world vision challenge. CVPR Workshop Open World Vision, 2021.
Gukyeong Kwon, Mohit Prabhushankar, Dogancan Temel, and Ghassan AlRegib. Backpropagated gradient representations for anomaly detection. In ECCV, 2020.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. In CS23/N, 2015.
11

---

Published as a conference paper at ICLR 2022
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yamn.. lew.unn.leee....o www.:lewwwn.....ceieeeee www.exxd//miistt 2010.
Kibok Lee, Kimin Lee, Kyle Min, Yuting Zhang, Jinwoo Shin, and Honglak Lee. Hierarchical novelty detection for visual object recognition. In CVPR, 2018a.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. NeurIPS, 2018b.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distribttion image detection in neural networks. In ICLR, 2018.
Sungbin Lim, Iddoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. In NeurIPS, 2019.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li, Energy-based out-of-distribution detection. NeurlPS, 2020.
Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In ICLR, 2017.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint 1rXiv:130..515,, 2013.
Luke Melas- Kyriazi Do you even need attention? stack of feed-forward layers does surprisingly well on imagenet. ArXiv e-prints, 2021.
Dimity Miller, Niko Sünderhauf, Michael Milford, and Feras Dayoub. Class anchor clustering: distance-based loss for training open set classifiers. In WACV, 2021.
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. ArXiv e-prints, 2021.
Lawrence Neal, Matthew Olson, Xiaoli Fern, Weng- Wong, and Fuxin Li. Open set learning with counterfactual images. In ECCV, 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In NeurlPS Workshop Deep Learning and Unsupervised Feature Learning, 2011.
Maria-Elee Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing, 2008
Poojan Oza and Vishal M. Patel. C2ac: Class conditioned auto-encoder for open-set recognition. In CVPR, 2019.
Pramuditha Perera and Vishal M. Patel. Deep transfer learning for multiple class novelty detection. In CVPR, 2019.
Pramuditha Perera, Ramesh Nallapati, and Bing Xiang. Ocgan: One-class novelty detection using gans with constrained latent representations. In CVPR, 2019.
Pramuditha Perera, Vlad I. Morariu, Rajiv Jain, Varun Manjunatha, Curtis Wigington, Vicente Ordonez, and Vishal M. Patel. Generative-dissriimnnat feature representations for open-set recognition. CVPR, 2020.
Rajeey Ranjan, Carlos Domingo Castillo, and Rama Chellappa. 1.2-constrained softmax loss for discriminative face verification. arXiv preprint UrXiv:1703.09500 2017.
Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML, 2019.
12

---

Published as a conference paper at ICLR 2022
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses. ArXiv e-prints, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJC,, 2015.
Mert Bulent Sariyildiz, Yannis Kalantidis, Diane Larlus, and Karteek Alahari. Concept generalization in visual representation learning. In ICCV, 2021.
Walter J. Scheirer, Anderson Rocha, Archana Sapkota, and Terrance Boult. Towards open set recognition. IEEE TPAMI, 2013.
Yu Shu, Yemin Shi, Yaowei Wang, Tiejun Huang, and Yonghong Tian. P-odn: Prototype-based open deep network for open set recognition. Scientific Reports, 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Xin Sun, Zhenning Yang. Chi Zhang, Guohao Peng, and Keck-Voon Ling. Conditional gaussian distribution learning for open set recognition. In CVPR, 2020.
Xin Sun, Chi Zhang, Guosheng Lin, and Keck- Ling. Open set recognition with conditional probabilisticc generative models. arXiv preprint arXiv:2008.055 05109, 2021.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In CVPR, 2016.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive
learning on distributionally shifted instances. In NewrIPS, 2020.
Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. ICML, 2019.
Iyya O. Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. Mlp-mixer: An all-mlp architecture for vision. In arXiv, 2021.
Antonio Torralba, Rob Fergus, and William T. Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE TPAMI, 2008.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech- UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-2011-68 1-001, California Institute of Technology, 2011.
Ross Wightman. Pytorch image models.
GitHlu repository [Online]. Available: moddee/eoooooiiiiaammmmeelsss 2019.
Pingmei Xu, Krista A Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev R. Kulkarni, and Jianxiong Xiao. Turkergaze: Crowdsourcing saliency with webcam based eye tracking. ArXiv e-prints, 2015.
Ryota Yoshihashi, Wen Shao, Rei Kawakami, Shaodi You, Makoto lida, and Takeshi Naemura. Classffcatioo-rcconstuuttto learning for open-set recognition. In CVPR, 2019.
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. ArXiv e-prints, 2015.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017.
Hongjie Zhang, Ang Li, Jie Guo, and Yanwen Guo. Hybrid models for open set recognition. In ECCV, 2020.
Nanxuan Zhao, Zhirong Wu, Rynson W.H. Lau, and Stephen Lin. What makes instance discrimination good for transfer learning? In ICIR, 2021.
13

---

Published as a conference paper at ICLR 2022
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. In IEEE TPAMI, 2017.
Da- Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Learning placeholders for open-set recognition. In CVPR, 2021.
14

---

Published as a conference paper at ICLR 2022
A EXPANSION OF FIG. 2 OF THE MAIN PAPER WITH STANDARD DEVIATIONS
For completeness, we include another version fig. which includes OSRCI models (Neal et al., 2018) in fig. 6. We find the correlation between the closed and open-set performance continues to hold with the inclusion of this additional method. We further report the standard deviations this plot in table 4. It can be seen that, for the same dataset, the standard deviations of all four methods appear to be similar. The standard deviations on the most challenging Tiny ImageNet benchmark is greater than on the other datasets.
Finally, we note in fig. that the trend seems less clear at very high accuracies. This may be because AUROC also becomes very high, making it difficult identify clear patterns. However, it may also indicate that the relationship between the metrics becomes weaker as closed-set performance saturates.
OSRCI
90
ARPL-+CS
Thaaeet MNIST
SVHN
CIFARIO
60
CIFAR
They EmageNee
100
(Accurscy)
Figure 6: Correlation between open-set and closed-set performances the standard OSR benchmarks. This plot is similar to fig. but includes scatter points for OSRCI (Neal et al., 2018).
Table 4: Standard deviations our experiments in fig. of the main paper. We report the standard deviations for both the closed-set and open-set performance (accuracy/AUROC) across the five 'nown/unknown' class splits.
Method
MNIST
SVHN
CIFAR10
CIFAR + 50
TinylmageNet
MSP
0.20/1.29
0.36/0.55
1.64/1.34
0.79/1.23
4.83/1.36
OSRCI
0.22/0.52
0.47/2.97
1.99/1.80
0.63/1.47
3,27/3.02
ARPL
0.21/0.77
0.43/0.79
2.10/1.56
0.66/0.44
5,40/1.63
ARPL + CS
0.29/1.04
0.51/0.31
1.70/1.68
0.63/0.23
4.40/1.55
B ANALYSING THE CLOSED-SET AND OPEN-SET CORRELATION
Here, we aim to understand why improving the closed-set accuracy may lead increased open-set performance through the MSL baseline. To this end, we train the VGG32 model the CIFAR10 10 benchmark setting with the cross-entropy loss. We train the model both with a feature dimension of D 128 (as is standard for this model) as well with D 2 for feature space visualization. We also train without bias in the linear classifier for more interpretable features and classification boundaries (so class boundaries radiate from the origin of the feature space). Specifically, train a model to make predictions as ÿ, = softmax W (x;).;)). where ( is a CNN embedding
function (x) E RD) and W I CxD D is the linear classification matrix. Here C |C| 6 and D E {2, 128), and we optimise the loss with one-hot target vector y; and batch size B, as B -i=1 DD y; log(}.).
Next, we interrogate the learned embeddings by plotting the mean vector norm of the features from all test images, for both the known and unknown classes, as training proceeds. These are shown in fig. 7a and fig. 7b for the models with D = 128 and = 2 respectively. We also show the average vector norm for the per-class weights in the linear classifiers dashed lines. Furthermore, snapshots of how these images are embedded for the model with = 2 are shown in fig. 7 7f representative epochs. The plots of the mean feature norms show that, the start of training, all
15

---

Published as a conference paper at ICLR 2022
Clovite Weiglt
Weght
03-
Noa Vers
077
66-
D-1 Kax Logtt
Kis Sattas
90
75
100
125
150
175
200
75
100
125
150
175
200
23
375 200
Spudbs
(a)
(b)
(c)
1.5
150
130
12.
Knowa
Known
Known
Uaknows
Unakownn
7.5
Epecls
77.35
Eprach:
75
Epoch
95
AUROC: 7M
53
AUROP:
AUROC 91.31
0--
-25
10
20
10
23
(d)
(I)
Jek--k
Jek-d -- Secha
Coed GpasSerluans
Drons
(g)
(h)
Figure 7: Plots showing how the feature representations and linear classification weights of deep classifier evolve as training proceeds (CIFARIO OSR setting). (a), (b) show the average feature norm for and unseen classes, well the per-class vector norms for the weights in the linear classification head, for models with D = 128 and D = 2 respectively. (c) shows how the open-set performance of the classifier with D) 128 develops as training proceeds, using three different OSR scoring rules. (d), (e). (f) show the feature projections for images from seen and unseen classes different epochs (indicated by vertical dashed lines in (b)) for the model with D 2. We show test images from known classes in colour and unknown classes in black. (g) (h) (i) show how classifier weight and feature norms change as function of weight decay strength (A)
images are embedded with a similar magnitude. However, as training proceeds, the magnitude of features for the known classes increases substantially more than for the unknown classes.
To understand this, consider the cross-entropy loss for a single sample in the batch, shown in eq. (2):
(O, W) - e + log( exp (xp()).)) w. (x,)) + log Σ exp(w, (x.))))
j=1
where e refers to the correct class index, and w, refers to the classification vector corresponding to the jth class. Empirically, we find that the linear classifier's weights and the feature norms for known classes increase during training, which is justified as increasing both |wa| and (x(x))) reduces the loss value. Note that we observe this despite training with weight decay, which we omit from eq. (2) for clarity. However, for 'hard' or 'uncertain' training examples (for which the classifier's prediction may be incorrect) the model is encouraged to reduce W,- (x;) Vj # e through the second term of eq. (2), While the only way to do this for the D 2 case is to reduce the feature norm (fig. 7b and fig. 7d to 7f), we show in fig. 7a that this also holds true for the D = 128 case which > C. The tendency of deep networks to map 'hard' samples closer to the origin has been noted in (Ranjan et al., 2017).
This suggests that stronger cross-entropy models project features further from the origin, while still ensuring that any 'uncertain' samples have lower feature norms. This, in turn, suggests stronger cross-entropy classifiers would perform better for OSR, with images from novel categories likely to
Ablltiooss for different weight decay values shown in fig. 7g to 7j (we use À 1e in this paper).
16

---

Published as a conference paper at ICLR 2022
be interpreted as 'uncertain' during evaluation. Our analysis also suggests that cross-entropy training already provides a strong signal and thus a strong baseline for open-set recognition.
Finally, this motivates to propose the maximum logit score (MLS) provide open-set score, i... Sly E C]x) = maxjec w, (x(x), rather than the softmax output as in the standard MSP baseline. Normalizing the logits via the softmax operator cancels the feature representation, which we have demonstrated useful for the OSR decision. Fig. 7c shows how the AUROC evolves as training proceeds when both the maximum logit and maximum softmax value are used for OSR scoring. The plot demonstrates that softmax normalization noticeably reduces the model's ability to make the open-set decision. We also show the OSR performance if we use the feature norm as our open-set score (S(y E C(x) = { showing that this simple indicator can perform remarkably well.
the magnitude information of
C IMPROVING OPEN-SET PERFORMANCE WITH STRONGER CLOSED-SET CLASSIFIERS
Here, we describe how we improve the open-set performance of the baseline method in sec. 4 of the main paper. and provide full breakdown of fig. 4. The methods include better learning rate schedules and data augmentations, as well as the use of logits rather than the softmax output for OSR scoring. We document the closed-set and open-set performance on the TinyImageNet dataset (the most challenging of the OSR benchmarks) in table 5. We further include the 'Open Set Classification Rate' (OSCR (Dhamija et al., 2018)) which summarises the trade-off between closed-set accuracy and open-set performance (here, in terms of the False Positive Rate) as the threshold the open-set score is varied. As demonstrated see. 4 of the main paper, the findings of this study generalize well to other datasets.
Table 5: Breakdown of methods used to improve the closed-set classification accuracy of the baseline method. All experiments were conducted with a VGG32 backbone over five 'known/unknown' splits of the TinylmageNet dataset. The bracketed number with the Cosine scheduler indicates the number of learning rate restarts used during training. We find a Pearson Product-Moment correlation of 0.93 between the closed-set accuracy and the open-set AUROC.
Setting
Closed Set
Open Set (AUROC)
Combined (OSCR)
(Accuracy)
Epochs
Scheduler
Aug.
Logit Eval
Warmup
Label Smoothing
Ensemble
100
Step Step
RandCrop
x
X
64.3
68.9
$1.4
RandCrop
64.3
69.6
200
Cosine
RandCrop
/
X
77.7
74.8
64.3
200
Cosine (0)
CutOut
/
X
X
77.6
75.4
64.7
200
Cosine
RandAug
/
X
79.8
76.6
67.3
600
Cosine (2)
RandAug
X
82.5
78.2
70.3
600
Cosine (2)
RandAug
X
82.5
78.4
70.3
600
Cusine
RandAug
X
84.2
83.0
74.3
600
Cosine (2)
RandAug
85.3
84.0
76.1
We first train the baseline with the same hyper-parameters as in (Chen al., 2021), training for 100 epochs and using a step learning rate schedule, with basic random crop augmentation strategy. We evaluate using both softmax and logit scoring strategies. can be seen that using maximum logit scoring gives better open-set performance (AUROC), while softmax scoring appears to be better in terms of OSCR. This is likely due to the fact that softmax normalization cancels the effect of the feature norm, which results in more separable scores that are beneficial to the OSCR calculation.
Here, we are interested in boosting the open-set set performance (AUROC) by improving the closed-set accuracy. Hence, we use the maximum logit for open-set scoring as discussed in appendix B. This already gives an open-set performance of 69.6% AUROC, which is significantly higher than the softmax thresholding baseline reported for these datasets in almost all of the comparisons in the literature, which report a baseline 57. 57.7% AUROC. The discrepancy between the reported baseline and our simplest setting is the result of reported figures originating in (Neal al., 2018), wherein all models were trained only for 30 epochs (according to the publicly shared code) while our simplest model is trained for 100 epochs.
Following this trend, we find that training for longer (200 epochs) and using a better learning rate schedule (cosine annealed schedule (Loshchilov & Hutter, 2017)) significantly enhances both
17

---

Published as a conference paper at ICLR 2022
closed-set and open-set performance. We further find that stronger augmentations boost accuracy, where we leverage RandAugment (Cubuk et al., 2020) to find an optimal strategy. Finally, we find that learning rate warmup and label smoothing (Szegedy et al., 2016) can together significantly increase accuracy. We select the RandAugment and label smoothing hyper-parameters by maximizing closed-set accuracy on a validation set (randomly sampling 20% of the training set).
In summary, we find that simply leveraging standard training strategies for image recognition models leads significant boost in open-set performance. Specifically, we find that the combination of the above methodologies, including longer training and better augmentations boosts the AUROC to 82.6%. Finally, we find that open-set performance can be boosted to 84.0% AUROC by bootstrapping the training data and training K = 5 ensembles. The improvements in open-set performance strongly correlate with the boosts to the closed-set accuracy, with p = 0.93 between accuracy and AUROC.
D IMPLEMENTATION DETAILS D.1 VGG32 ARCHITECTURE
This backbone architecture is commonly used in the open-set literature (Neal et al., 2018). The model consists of a simple series of nine 3x3 convolution layers, with downsampling occurring through strided convolutions every third layer. Batch normalization and LeakyRelu (slope of 0.2) used after every convolution layer, with dropout used on the input image, and then after the third and sixth layer. Finally, after the ninth layer, the spatial feature is reduced with average pooling to a feature vector with dimensionality D = 128. This is fed to the linear classifier (fully connected layer) to give the output logits.
D.2 STANDARD DATASETS
Here, we describe the experimental setup for our results in sec. 4 of the main paper.
All models were trained on a single 12GB GPU (mostly a NVIDIA Titan X). When optimizing with the cross-entropy loss, training took between 2 and 6 hours for a single class split, depending on the dataset (for instance, training on TinylmageNet took 2.5 hours). All hyper-parameters were tuned on validation set which was constructed by holding out a randomly sampled 20% of the closed-set training data from a single split of seen/unseen classes.
Baselines, MSP+/MSS We trained the VGG32 model with a batch size of 128 for 600 epochs. For each dataset, we train on five splits of 'known/unknown' classes as is standard practise, training each run with the random seed '0'. We use initial learning rate of 0.1 for all datasets except minylmageNet, for which we use 0.01. We train with a cosine annealed learning rate, restarting the learning rate the initial value at epochs 200 and 400. Furthermore, we 'warm up" the learning rate by linearly increasing it from 0 to the 'initial value' at epoch 20.
We use RandAugment for all experiments, tuning its hyper-parameters on validation set from single class split for each dataset. We follow a similar procedure for the label smoothing value s, though we find the optimal value to be for all datasets except TinylmageNet, where it helps significantly at s = 0.9.
We use the same experimental procedure for ARPL CS (Chen et al., 2021) as for the baselines, again tuning the RandAugment and label smoothing hyperparameters for this method. Here, following the original implementation, we find batch size of 64 and learning rate of 0.001 lead to better performance on TinylmageNet. This method also took significantly longer to train, taking 7.5 hours per class split on TinylmageNet.
(ARPL + CS)+
OSRCI+ OSRCI involves multiple stages of training, including first training GAN to synthesize images similar to the training data, before using generated images as 'open-set' examples to train a (K + 1)-way classifier (Neal et al., 2018). As our focus is on the effect of improving classification accuracy on open-set performance, we augment the training of the latter stage of OSRCI. We again train the (K + 1)-way classifier for 600 epochs with cosine annealed learning rate schedule and RandAugment. For this method, we find that reducing all learning rates by a factor 10 compared to the baselines significantly improved performance.
18

---

Published as a conference paper at ICLR 2022
D.3 PROPOSED BENCHMARKS
Here, we describe the experimental setup for our results in sec. 5 of the main paper.
ImageNet. For this evaluation, leverage ResNet50 model pre-trained with the cross-entropy loss on ImageNet-IK from (Wightman, 2019). We evaluate the model directly for our MLS baseline. For ARPL+ we finetune the pre-trained model for 10 epochs with the ARPL optimization strategy.
FGVC datasets. We use a similar experimental setting for the FGVC datasets as we do for the standard benchmarks. Specifically, for both MSP+/MLS and ARPL+, again train for 600 epochs, using a cosine annealed learning rate and learning rate warmup. We also re-tune the Rand Augment and label smoothing hyper-pann peeer-paraeeerss on validation set. Differently, however, we backbone with 448 x 448 image size as is standard in the FGVC literature. We further initialize the network with weights from MoCoV2 training on Places, using an initial learning rate of 0.001 and batch size of 32. Training for both methods took between one and two days depending on the dataset.
ResNet50
We attempted to train ARPL+CS on our proposed datasets but found it computationally infeasible. Specifically, the memory intensive nature of the method meant we could only fit a batch size of on a 12GB GPU. We attempted to scale it up for the FGVC datasets, fitting batch size of 16 across xx 24GB GPUs, with training taking a week. However, we found its performance after week to be slightly lower than +RPL+ in this setting.
Note:
E COMPARISONS WITH OTHER DEEP LEARNING BASED OSR METHODS
Table 6: Comparing our improved baseline with other deep learning based OSR methods on the standard benchmark datasets. All results indicate the under the Receiver- curve (AUROC) as percentage. We also show the backbone architecture used for each method, showing results with multiple backbones when reported.
Method
Backbone
MNIST
SVHN
CIFARIO
CIFAR- 10
CIFAR + -00
IitgInageNet
MSP (Neal al., 2018)
VGG32
97.8
67.7
81.6
NO.5
17.7
Ope (Bendale
2016)
VGG32
98.1
81.7
79.6
57.6
G-OpenMax (Cic ot al., 2017)
VGGG3
98.4
89.6
67.5
82.7
81.9
$8.0
OSRCI (Neal et al. 2018)
VGG32
98.8
91.0
69.9
$3.8
82.7
58.6
CROSR (Yoshiludhi
2019)
DHRNeL
C2AE (Oza & Patel. 2019)
VGG32
98.9
92.2
89.5
95.5
93.7
74.8
GFROSR (Perera et al., 2020)
VGG32/ WRR-28-10 10
93.5 / 95.5
80.7/83.1
92.8/9155
92.6/91.3
60.8 61.7
CGDL (Sun al., 2021)
CPGM-AAE
99.5
95.3
96.5
96.1
77.0
OpenHyhhid (Zhang et 20200)
VGG32
99.5
94.7
95.0
96.2
95.5
79.3
RPI. (Chen et al., 20120s)
VGG32/ WRN-40-4
99.3 99.6
95.1 96.8
86.1 90.1
85.6/97.6
85.0/ 96.8
70.2/ 80.9
PROSER (Zhon al., 2021)
WRN-28-10
94.3
89.1
96.0
85.3
69. 3
ARPI. (Chen
2021)
VGGG32
90.1
94.3
76.2
ARPL CS (Chen et al., 2021)
VGG32
99.7
96.7
91.0
97.1
95.1
78.2
OSRCI
VGG32
(-0.3)
(-0.3)
(-1.))
(+17.3)
91.1 (+7.3)
90.3 (+7.6)
62.6 (+4.0)
(ARPI
VGG32
(-0.5)
(40.1)
(+2.9)
98.1 (+1.0)
(+4.3)
Baseline (MSP4)
VGG32
98 (+0.8)
96 0 (47.4)
90.1 (+22.4)
95.6 (+14.0)
(=1.5)))
82. (+25.0)
Baseline (MLS)
VGG32
99.3 (+1.5)
97.1 (+8.5)
93.6 (+25.9)
97.9 (+16.3)
96.5 (+16.0)
83.0 (+25.3)
In table 6, we provide comparisons with more methods, including those using different backbone architecture, to supplement table from the main paper. The overrall conclusion is the same as the main paper. Specifically, our improved baseline significantly outperforms reported baseline figures and outperforms sfate-of-the-art OSR models on a number of standard benchmarks. Training other OSR methods (OSRCI, ARPL + CS (Neal et al., 2018; Chen et al., 2021)) on top of our improved baseline can boost also their OSR performance. However, the discrepancy between the state-of-the-art and the baseline is now negligible.
F OUT-OF-DISTRIBUTTI
DETECTION RESULTS
In this section, we run experiments on OoD benchmarks, separate but related machine learning sub-field to OSR. OoD deals with all forms of distributional shifts, whereas OSR focusses on semantic novelty. Specifically, in the 'multiclass" OoD setting, a model is trained for classification dataset, before being tasked with detecting test samples from other datasets as 'unknown' (Hendrycks & Gimpel, 2017). Once again, this task is evaluated problem. A notable difference with the OSR setting is that OoD models often have access to auxiliary data as examples of 'OoD' during training (Hendrycks et al., 2019).
a given
binary classification ( ('known'// /'unknown')
19

---

Published as a conference paper at ICLR 2022
F.1 CORRELATION BETWEEN CLOSED-SET AND OoD PERFORMANCE
First, we conduct similar experiments to sec. 3. We evaluate four ResNet models trained on CI- FARIO0 the OoD task, using IO for examples of 'OoD'. We show the closed-set and OoD performances of these models are correlated in fig. 8, with a Pearson Product-Momest correlation of 0.97. This trend is similar to the one observed in the ImageNet OSR evaluation in fig. 3b.
0.78
ResNer20
AUROC (OaD AORD (OaD Pefformanc))
ResNer32
0.76
ResNer44
ResNet56
0.74
0.69
0.70
0.71
0.72
Accuracy (Closed Sot Performunce)
Figure 8: OoD against closed-set performance for four ResNet models trained CIFARI00, using CIFARIO as OoD. The plot indicates similar performance correlation as observed in fig. 3b.
F.2 OoD PERFORMANCE WITH DIFFERING TYPES OF DISTRIBUTION SHIFT
Next, in table 7, we evaluate OoD performance when different datasets are taken as examples of 'OoD' with respect to CIFARI00 100. Specifically, we compare OSR methods (and an OoD baseline), taking Gaussian Noise, SVHN and CIFARIO 'OoD'.
Table 7: Results on out-of-distributio detection benchmarks. We evaluate two MLS models: one represents a model which we train ourselves; the second represents a strong pre-trained model from (Lim et al., 2019).
Outlier Exposure
OpenHyhrid
ML.S
(Hendrycks et al., 2019)
(Zhang el al., 2020)
ARPL++CS
MLS
(Lim et al., 2019)
CIFARI00 Gaussian Noise CIFARI00 - SVHN CIFARI00 CIFARIO
95.7
67.6
73.5
78.9
86.9
77.9
83.3
88,9
75.7
85.6
73.0
77.7
83.2
As a strong baseline from the OoD literature, report results from Outlier Exposure (O.E.) (Hendrycks et al., 2019), which encourages the classifier to predict a uniform distribution when fed auxiliary 'OoD' images from 80 Million Tiny Images (Torralba al., 2008). We also report results from OpenHybrid (Zhang et al., 2020) which reports CIFAR 100 + CIFAR10 result. Furthermore, we train ARPL+CS and MLS in this setting, training ResNet50 for 200 epochs. As final experi- ment, we take strong model pre-trained on CIFAR100 from (Lim et al., 2019) and evaluate it on the OoD benchmarks. Our results show that, while OpenHybrid performs strongly on the CIFARTON 100 -> CIFAR 10 experiment, the two MLS models outperform the O.E baseline on this evaluation despite not having seen extra data during training.
F.3 EVALUATION ON OoD BENCHMARKS
Finally, we run our MLS method on the standard OoD benchmark suite. Specifically, we take models trained on CIFAR1O and CIFAR100, and evaluate them when Places365 (Zhou al., 2017), Textures (Cimpoi et al., 2014 LSUN-Crop (Yu et al., 2015), LSUN-Resize (Yu et al., 2015). iSUN (Xu et al., 2015) and SVHN (Netzer et al., 2011) are used in turn as 'OoD' datasets. We take well-trained NedeResNNt-400 models (trained with Fast Auto-Augment et al., 2019)) and run our MLS baseline on top. We compare against sfate-of-the-ar OoD methods which do not use extra data for fine-tuning, and report results in table 8. We report average AUROC across the six OoD datasets.
CIFARI0 and CIFAR100 from (Lim
We find that strong closed-set classifiers with our MLS baseline achieve highly competitive performance on the OoD benchmarks, once again substantially closing the gap between the MSP baseline (Hendrycks & Gimpel, 2017) and state-of-the-art.
20

---

Published as a conference paper at ICLR 2022
Table 8: Results of our strong baseline on the full OoD benchmark suite. We take strong WideResNet-400 models from (Lim et al., 2019) and run our MLS baseline on top. Models are trained on CIFARIO and CIFAR100 as 'in-distribution" and we report AUROC averaged across six OoD datasets. All compared figures are taken from (Du et al., 2022) and Liu et al. (2020).
Method
CIFARIO
CIFAR100
MSP (Hendrycks & Gimpel, 2017)
90.9
75.5
ODIN (Liang et al., 2018)
91.1
77.4
Energy Score (Liu et al., 2020)
91.9
79.6
Mahanabolis (Lee et al., 80)
93.3
84.1
VOS (Du et al., 2022)
94.1
MLS (Ours)
95.1
80.8
Discussion. Our results show that strong closed-set classifiers can also perform well in the OoD setting, even compared to very recent methods such as Virtual Outlier Synthesis (VOS, (Du et al., 2022)). In fact, in some cases, we find the MISS baseline exceeds state-of-the-art for this task.
(e.g. SVHN and CIFARIO as 'OoD' in table 7, i.e. in the more similar settings to OSR). In fact, the MLS models trained on CIFAR are worse at detecting Gaussian Noise than CIFARIO images 'OoD'. We present this peculiar finding as evidence that the OoD and OSR research questions may have different, and possibly orthogonal, solutions. We hope that benchmarks which can isolate semantic novelty from low-level distributional shifts, such as the Semantic Shift Benchmark from sec. 5, can facilitate more controlled OSR and OoD research.
Interestingly, the MLS baseline performs best with in the 'near-OoD'
G DISCUSSION: UNDERSTANDING SYSTEMS OF CATEGORIZATION
Before one can establish if an image belongs to a new class, one must first understand what constitutes a single class, or how the system of categorization is constructed. To illustrate this, consider classifier trained on instances of two household pets: [Labrador (dog). British Shorthair (car)). Now consider an open-world setting in which the model must be able to distinguish previously unseen objects, perhaps: (Poodle (dog), Sphynx (cat)). In this case, understanding the categorization system essential to making the open-set decision. Does the classification system delineate individual animal species? In this case, both 'Poodle' and 'Sphynx' should be identified as 'open-set" examples. Or does instead simply separate 'cats' from 'dogs'? ? which case neither object belongs to the open-set.
To solve this problem, and to perform OSR reliably, the model must understand the set of invariances within single category, well set of 'axes of variation' to distinguish between categories. Specifically, different instances within a single category will have a set of features which can be freely varied without the category label changing. In computer vision, this often refers characteristics such as pose and lighting, but could also refer to more abstract features such as animal gender or background setting, Meanwhile, the classification system will also have a (possibly abstract) set of axes of variation to which the category label is sensitive.
In the current OSR benchmarks, with either abstract class definitions or a small number of classes, the set of axes of variation which can distinguish between categories is diverse. In this sense, the problem is ill-posed, with many axes likely being equally valid distinguish between the training classes, including those based on semantically meaningless low-level features. In contrast, within our proposed fine-grained setting, the set of axes of variation which can distinguish between categories is far more constrained. For instance, in the CUB case, given training task of classifying 100 bird species, there is little uncertainty as to what the axis of semantic variation could be.
H CREATING SPLITS FOR THE SEMANTIC SHIFT BENCHMARK
H.1 SPLIT CONSTRUCTION
In sec. 5 of the main paper, sketched the process for constructing open-set splits from the CUB dataset. Here, we describe the process in detail for both CUB, Stanford Cars and FGVC-Aireraft, which each have different attribute structures.
21

---

Published as a conference paper at ICLR 2022
For each FGVC benchmark, we split its classes into two disjoint sets, C and U, containing closed-set and open-set classes respectively. 24 is further subdivided into disjoint ( ('Eay', 'Medium', 'Hard') ) sets with varying degrees of attribute similarity with any class in C. Specifically, we measure the difficulty of an open-set class by its semantic similarity with its most similar training class (where similarity is defined in terms of attribute overlap).
In practice, we found the semantic similarity of the 'Medium' and 'Hard' splits of Stanford Cars to the closed-set to be very similar, hence we combine them into single 'Hard' split.
CUB. In CUB, each image is labelled for the presence of 312 visual attributes such as jas_bill __shape: hooked and caa__breat__oo) :oo::::::l:..ooooo low. Note that images from the same class do not all share the same attributes, both because of standard factors such as pose and occlusion, but also because of factors such as the age and gender of the bird.
This information is summarised on a per-class basis, describing how often each attribute occurs in each class; i.e., a matrix M E [0, 1] is available, where CUB and A = 312 is the number of attributes. This allows us to construct a class similarity matrix S E [0, c where Snj m, my and m; is the 1.2-normalized jth row of M. Thus, given a set of closed-set classes in C, we can rank all remaining classes (14) according to their maximum similarity with any of the training classes. Finally, we bin the ranked open-set classes into { ('Easy', "Medium', 'Hard") sets. In practice, we randomly sample I million combinations of C, and select the combination which results in the most difficult open-set splits.
200 is the total number of classes in
Stanford Cars. Each class name in Stanford Cars follows the format of 'Make' "Mode''' "Type' 'Year'; ; for instance 'Aston Martin V8 Vantage Convertible 2012' is class. In this case, we create open-set splits of different difficulties based on the similarity between class names.
We first create the 'Hard' open-set split by identifying pains of classes which have the same 'Make', 'Model' and 'Type' but come from different 'Years'. Next, we create the 'Medium' split from class pairs which have the same 'Make' and 'Model' but have different "Types'. Finally, the 'Easy' split is constructed from pairs which have the same 'Make' but different 'Models'.
We note that open-set bins of different difficulties in Stanford Cars are the most troublesome to define. This is because the rough hierarchy in the class names may not always correspond the degree of visual similarity between the classes. For instance, two cars from the same 'Year' but of different 'Makes' (e.g. a Ford and Nissan both made in 2010) may look more similar than cars of the same "Make'-'Modee "Model'-' but from different years (e.g. Audi S4 Sedan 2007 and Audi S4 Sedan 2012).
FGVC-Aireraft. We leverage the hierarchy of class labels in FGVC-Aireraff each image is labelled with 'manufacturer' (e.g.. 'Airbus' or 'Boeing'), ) a 'family' (e.g.. 'A320' or 'A330') and 'variant' (e.g. '3330-2000 or 'A330-300') The hierarchy constructed as a tree, with 'manufacturer' classes at the top level, 'family' classes at the second, and 'variant" classes at the bottom. The standard image classification challenge operates at the variant level, meaning all variant classes are visually distinct with identifiable features. Furthermore, the hierarchy corresponds to visual similarity. i.c there is more inter-class variation between manufacturers than between variants from the same manufacturer. Thus, given the closed-set classes C, we can create 'Easy' open-set split from variants which do not share a manufacturer with any closed-set class. Meanwhile, 'Medium' open-set classes share a manufacturer with closed-set classes but come from different families, and 'Hard' open-set classes share families with closed-set classes but are different variants.
H.2 SPLIT EXAMPLES
We include examples of images from the closed-set and open-set splits of the proposed FGVC datasets in fig. 9 and 11. For each dataset, we show examples of "Easy" (green/top), 'Medium' (orange/middle) and 'Hard' (red/bottom) classes. For each difficulty, the open-set (right) and their most similar class in the closed-set (left). We note that 'Hard' open-set classes are far more visually similar their corresponding closed-set class than 'Easy' open-set classes.
show three images from three classes from
H.3 SPLIT DETAILS
All split details can be found here: http:///sithbbbooomgggrr/ooooodd need.
22

---

Published as a conference paper at ICLR 2022
Rose breasted Grosbeak
Eastern Towhee
Cerulean Warbler
Black throated Blue Warbler
Yellow bellied Flycatcher
Palm Warbler
White crowned Sparrow
White throated Sparrow
Herring Gull
Red legged Kittiwake
Red faced Cormorant
Brandt Cormorant
Blue Grosbeak
Indigo Bunting
Herring Gull
California Gull
Elegant Tern
Caspian Tern
Figure 9: Sample classes from closed and open-set splits for the CUB dataset. We show 'Easy' (green/top), 'Medium' (orange/middlee) and 'Hard' (red/bottom) classes. Classes the left (solid outline) are in the closed-set, while classes on the right (dashed outline) are in the open-set.
23

---

Published as a conference paper at ICLR 2022
Acura RL Sedan 2012
Acura TSX Sedan 2012
Audi RS Convertible 2008
Audi AS Coupe 2012
BMW ActiveHybrid 5 Sedan 2012
BMW Series Convertible 2007
Aston Martin Virage Convertib e 2012
Aston Martin Virage Coupe 2012
Audi 100 Sedan 1994
Audi 100 Wagon 1994
Audi S5 Convertible 2012
Audi S5 Coupe 2012
Audi Sedan 2012
Audi 54 Sedan 2007
Bentley Continental GT Coupe 2012
Bentley Continental GT Coupe 2007
Dodge Caliber Wagon 2012
Dodge Caliber Wagon 2007
Figure 10: Sample classes from closed and open-set splits for the Stanford Cars dataset. We show 'Easy" (green/top). 'Medium' (orange/middle) and 'Hard' (red/bottom) classes. Classes on the left (solid outline) are in the closed-set, while classes the right (dashed outline) are in the open-set. In practice we combine the MMediumm and 'Hard' splits during evaluation.
24

---

Published as a conference paper at ICLR 2022
Baeing 767-300
Luckheed Corporation C-130
Embraer EMB-120
Lockheed Corporation L-1011
Bueing 737-200
Becchcraft Beechcraft 1900
Douglas Aircraft Company 47
Douglas Aircraft Company DC
de Havilland DH-82
de Havilland DHC-6
Embracr EMB-120
Embraer ERJ 135
Boeing 737-200
Boeing 737-600
Bocing 747-100 100
Bocing 747 747-300
Airbus A340 200
Airbus A340 500
Figure 11: Sample classes from closed and open-set splits for the FGVC-Aireraft dataset. We show "Easy' (green/top), 'Medium' (orange/middle) and 'Hard' (red/bottom) classes. Classes on the left (solid outline) are in the closed-set, while classes on the right (dashed outline) are in the open-set.
25

---

Published as a conference paper at ICLR 2022
I AVERAGE PRECISION EVALUATION ON PROPOSED BENCHMARKS
We report average precision (AP) for the binary 'known/unknown' decision for the proposed bench- mark evaluations in table 9. AP is standard metric in the OoD literature and is better suited for dealing with class imbalance at test time. We note that the 'Hard' FGVC open-set splits (with a small number of classes) report substantially poorer AP than AUROC in absolute terms. We treat open-set examples as 'positive' during evaluation.
Table 9: Average Precision (AP) results on the proposed benchmark datasets for 'Easy' 'Medium'/ 'Hard' splits.
CUB
FGVC-Aireraft
ImageNet
ARPL+
59.9/53.3/455 /55.3
66.9 06.9/58.9/30 34.4
78.2/-/71.2
MLSS
67.1 58.2 47.2
69.2 / 58.2 39.6
16.6//688.6
26


Reviews:
[{"review_id": "sohU-Ri_CS_", "reviewer": "Reviewer_PPFy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is dealing with the open set recognition problem which is a challenging research problem. Based on some findings about the correlation of closed set recognizer and open set recognizer, this paper found a way to achieve a strong OSR through an enhanced close set recognizer. This is a valuable finding for future open set recognition research. \nThe weakness of this paper is that there are not more in depth analysis the behind reason for these findings, thus the value of this paper is not that significant. So this is possibly a direction to make this paper even stronger.", "summary_of_the_paper": "This paper first analyzed the correlation between closed set classifier and open set classifier. Then based on this finding, cross entropy closed set classifier is enhanced with a few recent accuracy improvement methods, then it is transferred to a open set classifier and achieve good performance. And also experimentation is conducted to compare with other SOTA close set classifiers, and that the proposed enhanced OSR classifier based on cross entropy baseline can achieve similar accuracy. Finally a new benchmark dataset is generated. ", "summary_of_the_review": "This paper proposed interesting finding about the correlation between closed set classifier and open set classifier, and leveraging this finding, a new SOTA open set classifier is generated from a baseline classifier, which has similar performance as  other SOTA close set classifiers. Also a new benchmark dataset is generated.\nOverall it is a valuable paper, which is clearly written, and with good experimentation results to support. The weakness of this paper is lacking of more in depth analysis, and providing more insights to the behind rationale, which lower the value of this paper.  \nSo overall I will recommend marginal acceptance of this paper. \n ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YVyQ-6ARiZF", "reviewer": "Reviewer_HAFU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The core technical contribution or claim of this paper is that closed set accuracy is important for openest detection. This is known for researchers working on openest detection, though not highlighted enough. For any representation one key challenge of open set detection is in distinguishing confusing instances (incorrect predictions) within closed set vs novel category instances along the decision boundary. So as the quality of closed set classification improves this overlap could decrease and hence would enable easier detection of open-set instances.\n\n[1] makes similar claims that vision transformers result in better open-set detection without complex detection mechanisms. [3],[4] highlight the fact that good representation is all you need for meta learning or out of distribution generalization which though orthogonal to open-set detection, informs the research community that ‘quality of representation & performance on closed set’ is useful for auxiliary and relevant tasks of OOD, few-shot learning.  \n\nSimilar to the observation of this paper of closed set accuracy, many works in ‘openworld’ competition [5] at CVPR has taken various strategies to improve closed set accuracy in pre-training phase to improve closed set accuracy and leverage it for open-set performance boost.  \nAnother key contribution of this paper is proposing large scale ImageNet benchmark, [5] is a CVPR’21 competition and existing benchmark along the similar lines which unfortunately makes benchmarking contribution not significant enough. \n\nStrengths:\n\nThis paper focuses on an important problem and highlights an observation and it’s simplicity is especially relevant for practical settings and safety-critical applications.\nThis paper is well written and has great empirical evaluation methodology and demonstrate SOTA compared to other methods with additional components.\nThis paper proposes additional benchmarks for openset detection leveraging fine-grained classification datasets, and this is adds a valuable dimension to openset benchmarks.\n\nWeakness:\n\nThe core technical contribution(s) does not seem convincing to warranty a conference acceptance, especially given previous works making similar conclusions and contributions. \nThis lacks sufficient commentary on why a good closed set accuracy would result in better open-set detection.\n\n\nSuggestion to Authors:\n        Visualization of representations (albeit noisy) with and without augmentations/modified training could be insightful. May be including few examples of how hard novel instances/samples map onto different representations.\n\tFor completeness please consider including CIFAR100 (close set) vs CIFAR10(open set), as the number of closed set classes increase the task of openset detection becomes more challenging, this version of experiment is included in [2] which this paper includes as baseline and [1].\n\nReferences:\n\n[1]  Exploring the Limits of Out-of-Distribution Detection. http://www.gatsby.ucl.ac.uk/~balaji/udl2021/accepted-papers/UDL2021-paper-001.pdf\n \n[2] H. Zhang et al. Hybrid Models for Open Set Recognition https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123480103.pdf \n\n[3] Y. Tian et al. Rethinking Few-shot Image Classification: A Good Embedding is All You Need?  \nhttps://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123590256.pdf\n\n[4] J. Miller et al. Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization. https://proceedings.mlr.press/v139/miller21b/miller21b.pdf\n\n[5] Open World Vision CVPR’21 Competition and workshop https://www.cs.cmu.edu/~shuk/open-world-vision.html, https://eval.ai/web/challenges/challenge-page/1041/overview \n", "summary_of_the_paper": "This paper makes an observation that good representations for open set detection would be correlated with high closed set accuracy.\nTo validate and illustrate this observation they improve closed set accuracy on existing open set benchmarks with different architectures and demonstrate  improvement in open set detection performance. \nThis paper also proposes new benchmarks for openest detection with fine grain details, which haven’t been studied in the setting of openest detection.", "summary_of_the_review": "This paper focuses on an important problem of open-set detection and demonstrate that a good closed-set representation is a very strong baseline and competitive with more complex methods for open-set detection, which is valuable for many practical settings.\nThough empirical evaluation is comprehensive and illustrate performance on large scale benchmarks, given existing prior works novelty of contributions seems rather limited", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-ehYDB8ZFT5", "reviewer": "Reviewer_dw7J", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper is well written, clear and straight to the point.\n- The appendix add more experiments and details to further understand the main paper statements.\n- Up to my knowledge the most important references in the field have been considered\n- The paper belongs to the category of very important papers that show that sometimes in research we consider very complex solutions, whereas a properly tuned standard approach can do the job as well.\n- The authors propose a clear distinction between open-set recognition (in which the none-of-the above class comes form well defined classes) and out-of-distribution (in which the none-of-the-above can be any kind of image). It is important to clarify the different tasks.\n- It is interesting to see that in new and not overfitted datasets, cross-entropy baseline seems to perform better than the best method for open-set scenario. \n\nWeaknesses:\n- Authors compare with only ARPL and OpenHybrid approaches because they say that the other approaches perform lower on open-set benchmarks. I think it is still important to consider also the other methods especially because the authors propose new benchmarks and on these benchmarks the ranking between the methods changes. So it could be for the other methods. I see that a more complete evaluation is performed in the appendix E (Table 6), however this is not done for the new datasets (Table 3).\n- In Fig.2 I would expect to see also results for OpenHybrid. Also, in the figure, for the simplest datasets (with performance close to 100) there is not much correlation between open-set can close-set performance as ARPL does not improve over cross-entropy on the close-set scenario. It is something that should be mentioned.\n- Results in Table 1 for the cross-entropy are presented with a ranking based on the classifier logits, which in my opinion makes more sense. However, as previous works use the softmax scores, it is important to compare the two ways. I see a comparison in Appendix B (Fig. 6c), but this is only for a model. I would like to see this comparison at least for the models in tab. 1.\n\nAdditional questions/comments:\n- Fig.1a presents ARPL in the legend, without really having yet introduced the method.\n- The authors propose a clear distinction between open-set recognition and out-of-distribution. However, is this distinction really necessary. Could not the two fields be unified? It would be more interesting to know if the same conclusions of this paper are valid also for out-of-distribution problems.\n- The \"more theoretical\" justification of the results at bottom of page 4 does not add much, but I do not have suggestions on how to improve it.\n- In Fig.3 it is interesting to see that ViT seems to have a better generalization to the open-set scenario on ImageNet. Is it due to the fact that it has been trained on more data or it is the reduced inductive bias (no convolutions)? It would be interesting to see if with different sizes of ViT, the correlation between open and close-set scenario still holds.", "summary_of_the_paper": "This paper considers the problem of open-set recognition, in which a visual classifier should be able to distinguish images of the trained categories from images of other different categories.\nThe authors show that there is a strong correlation between results on open-set scenario and the close set-scenario (the classic problem in which the model is trained and tested on the same semantic categories). Then, they show that a baseline based on ranking the logits of a model trained on standard cross-entropy training can be very competitive with more complex methods when trained with strong data augmentation and other improvements. \nFinally, they propose new benchmarks for open-set recognition emphasising the fact that in contrast to other similar tasks, in open set scenario, the images that should be classified as other classes not seen during training should belong to a different semantic category, and not to other kind of distributional shifts.\nThe take home message of this paper is that at this stage of research for improving open-set recognition, the best that we can do (or almost) is to use stronger models for close-set recognition (as the title suggests).", "summary_of_the_review": "The paper is well written and introduces interesting results that can change the understanding of the open-set recognition. Experimental results prove the main conclusion of the paper, but some additional experiments are needed (see above) to further understanding the problem. Also, in my understanding the distinction between open-set recognition and out-of-distribution is ficticial and the proposed cross-entropy baseline should evaluated also on out-of-distribution settings.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides well-written and thorough analysis demonstrating that closed-set recognition performance correlates with open-set recognition performance, and that simply making the close-set model strong via augmentation, label smoothing, etc. along with small scoring changes (using logits rather than softmax probabilities) can get close to (or better than in some cases) performance than much more complicated methods. The authors also propose a large-scale benchmark that varies the semantic similarity across classes, allowing for a more fine-grained analysis of this problem. 

Overall, all of the reviewers thoughts that the paper provides very thorough validation of an insight that would be very interesting to the community. Reviewer HAFU had some concerns about novelty, since a number of papers have shown closed-set classifier improvements (and therefore better embeddings) benefit related problems such as few-shot learning and generalization to novel domains, as well as proposed large-scale experiments. The rebuttal convinced this reviewer, however, that some of the contributions and findings are unique and provide additional evidence to the community, and the new setting provides more fine-grained analysis. Reviewer dw7J had a number of suggestions in terms of additional evaluations, and the rebuttal either clarified why it is not possible or added them. As a result, after the discussion the reviewers all supported acceptance of this paper. 

Given the above discussion, and rebuttal/changes to the paper, I recommend acceptance. It is a very well-done empirical paper, provides interesting findings, stronger baselines, and thorough experimentation. Further, some of the smaller findings (ViT correlation experiment) as well as larger relationship between open-set recognition and out-of-distribution detection are valuable contributions to the community. Finally, I would recommend this paper as oral, given that it may garner a good discussion of these contributions.
Decision: Accept (Oral)