Open-Set Recognition: A Good Closed-Set Classifier is All You Need

1 INTRODUCTION . Given the success of modern deep learning systems on closed-set visual recognition tasks , a natural next challenge is open-set recognition ( OSR ) ( Scheirer et al. , 2013 ) . In the closed-set setting , a model is tasked with recognizing a set of categories that remain the same during both training and testing phases . In the more realistic open-set setting , a model must not only be able to distinguish between the training classes , but also indicate if an image comes from a class it has not yet encountered . The OSR problem was initially formalized in ( Scheirer et al. , 2013 ) and has since inspired a rich line of research ( Bendale & Boult , 2016 ; Chen et al. , 2020a ; Ge et al. , 2017 ; Neal et al. , 2018 ; Sun et al. , 2020 ; Zhang et al. , 2020 ; Shu et al. , 2020 ) . The standard baseline for OSR is a model trained with the cross-entropy loss on the known classes . At test time , the maximum value of the softmax probability vector is used to decide if an input belongs to the known classes or not . We henceforth refer to this method as the ‘ baseline ’ or ‘ cross-entropy baseline ’ . Most existing literature reports significantly outperforming this OSR baseline on standard benchmarks of re-purposed image recognition datasets , including MNIST ( LeCun et al. , 2010 ) and TinyImageNet ( Le & Yang , 2015 ) . In this paper we reappraise these approaches . Firstly , we investigate the relationship between the closed-set and open-set performance of a classifier ( sec . 3 ) . While it might be expected that stronger closed-set classifiers might overfit to the training classes ( Recht et al. , 2019 ; Zhang et al. , 2017 ) , and thus might perform poorly for OSR , we show instead that the closed-set and open-set performance are highly correlated . We show this trend holds across datasets , loss objectives and model architectures , and further demonstrate the trend on an ImageNet-scale evaluation . Secondly , following this observation , we show that the open-set performance of a classifier can be improved by enhancing its closed-set accuracy , tapping the numerous recent advances in image classification ( Loshchilov & Hutter , 2017 ; Szegedy et al. , 2016 ; Cubuk et al. , 2020 ; Bello et al. , 2021 ) . Specifically , we introduce strategies such as more augmentation , better learning rate schedules and label smoothing , that significantly improve the closed-set performance of the cross-entropy baseline ( sec . 4 ) . With these adjustments , we push the baseline to become competitive with or outperform state-of-the-art OSR methods , substantially outperforming the currently reported baseline figures . Notably , we achieve a new state-of-the-art on the most challenging TinyImageNet benchmark . Furthermore , we transfer these improvements to two previous OSR methods , including the current state-of-the-art from ( Chen et al. , 2021 ) . While this does boost its performance , we observe that the improvement with respect to our improved baseline is marginal , further demonstrating the power of the simple cross-entropy classifier ( see fig . 1a ) . This finding is important because it allows us to better assess recent reported progress in the area . Finally , we turn to the experimental setting for OSR ( sec . 5 ) . Current OSR benchmarks are both small scale and lack a specific definition of what constitutes a ‘ visual class ’ . As an alternative , we propose an ImageNet-scale evaluation based on the full ImageNet database ( Ridnik et al. , 2021 ) , as well as the use of fine-grained datasets , including CUB ( Wah et al. , 2011 ) and FGVC-Aircraft ( Maji et al. , 2013 ) , which have clear definitions of a semantic class ( see fig . 1b ) . Furthermore , we construct open-set splits with an explicit focus on semantic novelty , which we hope better separates this avenue of research from related machine learning sub-fields such as out-of-distribution ( Hendrycks & Gimpel , 2017 ) and anomaly detection ( Kwon et al. , 2020 ) . Our proposed splits also offer a better way of quantifying open-set difficulty ; we find that different splits lead to a much larger discrepancy in open-set performance than the current measure of open-set difficulty ‘ openness ’ ( Scheirer et al. , 2013 ) , which focuses only on the number of open-set classes . We evaluate our strong baseline as well as the state-of-the-art method on this new configuration to encourage future research in this direction . 2 RELATED WORK . Open-set recognition Seminal work in ( Scheirer et al. , 2013 ) formalized the task of open-set recognition , and has inspired a number of subsequent works in the field . ( Bendale & Boult , 2016 ) introduced the first deep learning approach for OSR , OpenMax , based on the Extreme Value Theory ( EVT ) . GANs have also been used to tackle the task ( Ge et al. , 2017 ; Neal et al. , 2018 ) . OSRCI ( Neal et al. , 2018 ) generates images similar to those in the training set but that do not belong to any of the known classes , and uses the generated images to train an open-set classifier . This work also established the existing OSR benchmark suite . ( Kong & Ramanan , 2021 ) achieve strong OSR performance by using an adversarially trained discriminator to delineate closed from open-set images , leveraging real open-set images for model selection . Other approaches include reconstruction based methods ( Yoshihashi et al. , 2019 ; Oza & Patel , 2019 ; Sun et al. , 2020 ) which use poor test-time reconstruction as an open-set indicator , and prototype-based methods ( Shu et al. , 2020 ; Chen et al. , 2020a ; 2021 ) which represent known classes with learned prototypes , and identify open-set images based on distances to the prototypes . State-of-the-art In this work , we compare against methods which achieve state-of-the-art in the controlled OSR setting ( with no extra data for training or model selection , for instance as demonstrated in ( Kong & Ramanan , 2021 ) ) . To our knowledge , these methods are ARPL ( Adversarial Reciprocal Point Learning ) ( Chen et al. , 2020a ; 2021 ) and OpenHybrid ( Zhang et al. , 2020 ) , which we detail in sec . 3.1 and sec . 4 respectively . In this paper , we show that the cross-entropy baseline can be competitive with or outperform the more complex methods listed above . Finally , we note recent works ( Zhou et al. , 2021 ; Miller et al. , 2021 ; Guo et al. , 2021 ) with which we do not compare as they report lower performance than ARPL and OpenHybrid . Related subfields OSR is also closely related to out-of-distribution ( OoD ) detection ( Hendrycks & Gimpel , 2017 ; Liang et al. , 2018 ; Hsu et al. , 2020 ) , novelty detection ( Abati et al. , 2019 ; Perera et al. , 2019 ; Tack et al. , 2020 ) , anomaly detection ( Hendrycks et al. , 2019 ; Kwon et al. , 2020 ; Bergman & Hoshen , 2020 ) and novel category discovery ( Han et al. , 2019 ; 2020 ; 2021 ) . Amongst these , OoD is perhaps the most widely studied and is similar in nature to OSR . As noted by ( Dhamija et al. , 2018 ; Boult et al. , 2019 ) , OSR is similar to the OoD problem with an additional multi-way classification component between known categories . In fact , there is currently significant overlap in the evaluation datasets between these settings , though cross-setting comparisons are difficult due to different evaluation protocols . Specifically , the OoD setting permits the use of additional data as examples of ‘ OoD ’ data during training . ( Chen et al. , 2021 ) and ( Zhang et al. , 2020 ) evaluate their OSR methods on OoD benchmarks , with both showing competitive results despite not having access to additional data during training . In this paper , we distinguish the OSR problem from OoD and other related fields by proposing a new suite of benchmarks . While OoD encompasses all forms of distributional shift , including those based on low-level features , OSR specifically refers to semantic novelty . We propose new benchmarks that respect this distinction . 3 CORRELATION BETWEEN CLOSED-SET AND OPEN-SET PERFORMANCE . One may expect that stronger closed-set classifiers have overfit their learned representations to the closed-set categories , and thus perform poorly for OSR ( Recht et al. , 2019 ; Zhang et al. , 2017 ) . Furthermore , existing literature largely considers the closed and open-set tasks separately , with works generally emphasising good open-set performance despite no degradation in closed-set accuracy ( Neal et al. , 2018 ; Zhou et al. , 2021 ; Miller et al. , 2021 ) . On the contrary , in this section we show that the closed-set and open-set performance of classifiers are strongly correlated . We first demonstrate this for the baseline and a state-of-the-art method on the standard OSR benchmarks ( sec . 3.1 ) and then on a large scale evaluation across a number of model architectures ( sec . 3.2 ) . Open-set recognition . We formalise the problem of OSR , and highlight its differences from closedset recognition . First , consider a labelled training set for a classifier Dtrain = { ( xi , yi ) } Ni=1 ⇢ X ⇥ C. Here , X is the input space ( e.g. , images ) and C is the set of ‘ known ’ classes . In the closed-set scenario , the model is evaluated on a test set in which the labels are also drawn from the same set of classes , i.e. , Dtest-closed = { ( xi , yi ) } Mi=1 ⇢ X ⇥ C. In the closed-set setting , the model returns a distribution over the known classes as p ( y|x ) . Conversely , in OSR , test images may also come from unseen classes U , giving Dtest-open = { ( xi , yi ) } M 0 i=1 ⇢ X ⇥ ( C [ U ) . In the open-set setting , in addition to returning the distribution p ( y|x , y 2 C ) over known classes , the model also returns a score S ( y 2 C|x ) to indicate whether or not the test sample belongs to any of the known classes . 3.1 BASELINE AND STATE-OF-THE-ART ON STANDARD BENCHMARKS . We first experiment with three representative open-set recognition methods across the standard benchmark datasets in the literature ( Neal et al. , 2018 ; Oza & Patel , 2019 ; Sun et al. , 2020 ; Chen et al. , 2020a ; Zhang et al. , 2020 ) . The methods include the standard cross-entropy baseline as well as two variants of ARPL ( Chen et al. , 2021 ) . We use the standard network from the open-set literature ( Neal et al. , 2018 ) , a lightweight model similar to the VGG architecture ( Simonyan & Zisserman , 2015 ) which we henceforth refer to as ‘ VGG32 ’ ( refer to appendix D for details ) . The three methods are summarised below , followed by a description of the most commonly used benchmarks . Methods . Cross-entropy ( baseline ) : The model is trained for closed-set classification using the cross-entropy loss between a one-hot target vector and the softmax output p ( y|x ) of the classifier . This training strategy , along with the use of the maximum softmax probability as S ( y 2 C|x ) = maxy2C p ( y|x ) , is widely used in the OSR literature as a baseline . ARPL ( Chen et al. , 2021 ) : This method is an extension of the recent RPL ( Reciprocal Point Learning ) optimization strategy ( Chen et al. , 2020a ) . Here , the probability that a sample belongs to a class is proportional to its distance Figure 2 : Correlation between closed set performance ( accuracy ) and open-set performance ( AUROC ) . We train three methods on the standard open-set benchmark datasets , including the cross-entropy baseline , ARPL and ARPL + CS ( Chen et al. , 2021 ) . Foreground points in bold show results averaged across five ‘ known/unknown ’ class splits for each method-dataset pair ( following standard practise in the OSR literature ) while background points , shown feint , indicate results from the underlying individual splits . from a learned ‘ reciprocal point ’ in the feature space . A reciprocal point aims to represent ‘ otherness ’ with respect to a class , with the intuition being that open-set examples are different to all known classes . ARPL extends RPL by computing feature distances as the sum of both the Euclidean and cosine distances . In this case , S ( y 2 C|x ) is equal to the maximum distance in feature space between the image and any reciprocal point . ARPL + CS ( Chen et al. , 2021 ) augments ARPL with ‘ confusing samples ’ : adversarially generated latent points to stand in for ‘ unseen class ’ samples . The confusing samples are encouraged to be equidistant from all reciprocal points , with the same open-set scoring rule used as in ARPL . We train both ARPL and ARPL + CS based on the official public implementation ( Chen et al. , 2021 ) . Datasets . We train the above methods on the standard benchmark datasets for open-set recognition . In all cases , the model is trained on a subset of classes , while other classes are reserved as ‘ unseen ’ for evaluation . MNIST ( LeCun et al. , 2010 ) , SVHN ( Netzer et al. , 2011 ) , CIFAR10 ( Krizhevsky , 2009 ) : These are ten-class datasets , with MNIST and SVHN containing images of hand-written digits and street-view house numbers respectively . Meanwhile , CIFAR10 is a generic object recognition dataset containing natural images from ten diverse classes including animals and vehicles . In these cases , the open-set methods are evaluated by training on six classes , while using the other four classes for testing ( |C| = 6 ; |U| = 4 ) . CIFAR + N ( Krizhevsky , 2009 ) : In an extension to the CIFAR10 evaluation protocol , open-set algorithms are benchmarked by training on four classes from CIFAR10 , while using N classes from CIFAR100 for evaluation , where N denotes either 10 or 50 classes ( |C| = 4 ; |U| 2 { 10 , 50 } ) . TinyImageNet ( Le & Yang , 2015 ) : In the final and most challenging case , exisiting open-set algorithms are evaluated on the TinyImageNet dataset . This dataset contains 200 classes sub-sampled from ImageNet ( Russakovsky et al. , 2015 ) , with 20 classes used for training and 180 as unknown ( |C| = 20 ; |U| = 180 ) . Experimental setup . At test time , the model is fed test images from both known and novel classes , and is tasked with making a binary ‘ known/unknown ’ decision on a per-image basis . Following standard practise in the OSR literature , the threshold-free area under the Receiver-Operator curve ( AUROC ) is used as an evaluation metric . We train with the same hyper-parameters as in ( Chen et al. , 2021 ) and , following standard practise , train on five different splits of closed and open-set classes for each dataset and method combination . When evaluating on existing benchmarks throughout this paper , we use the same data splits as ( Chen et al. , 2021 ) . Results . Fig . 2 gives the AUROC ( open-set performance ) against the Top-1 multi-way classification accuracy ( closed-set performance ) . We show the averaged results as well as the individual split results , omitting the CIFAR+10 setting for clarity ( as the scatter points are almost coincident with the CIFAR+50 setting ) . It is clear that there is a positive correlation between the closed-set accuracy and open-set performance : we find a Pearson Product-Moment correlation ⇢ = 0.95 between the accuracy and AUROC , indicating a roughly linear relationship between the two metrics . Discussion . To justify our findings theoretically , we look to the model calibration literature ( Guo et al. , 2017 ) . Intuitively , model calibration aims to quantify whether the model ‘ knows when it doesn ’ t know ’ , in that low confidence predictions are correlated with high error rates . Specifically , assume a classifier , f ( x ) , returns probabilities for each class , making predictions as ŷ = argmax f ( x ) . Further assume labelled input-output pairs , ( x , y ) ⇢ X ⇥ C , where C is the label space . Then , the classifier is said to be perfectly calibrated if : P ( ŷ = y|f ( x ) = p ) = p 8p 2 [ 0 , 1 ] ( 1 ) It is further true that if a classifier is trained with a proper scoring rule ( Gneiting et al. , 2007 ) on infinite data , then the classifier will be perfectly calibrated at the loss function ’ s minimum ( Minderer et al. , 2021 ) . Many losses used to train deep networks are proper scoring rules ( e.g.the cross-entropy loss ) . Thus , assuming that generalization error on the test set is correlated with the infinite-data loss value , we would suspect models with lower generalization ( test ) error to be better calibrated . If we use low-confidence predictions as an indicator that a test sample belongs to a new semantic class , we would expect stronger models to be better open-set detectors .

Summary: This paper considers the problem of open-set recognition, in which a visual classifier should be able to distinguish images of the trained categories from images of other different categories. The authors show that there is a strong correlation between results on open-set scenario and the close set-scenario (the classic problem in which the model is trained and tested on the same semantic categories). Then, they show that a baseline based on ranking the logits of a model trained on standard cross-entropy training can be very competitive with more complex methods when trained with strong data augmentation and other improvements.  Finally, they propose new benchmarks for open-set recognition emphasising the fact that in contrast to other similar tasks, in open set scenario, the images that should be classified as other classes not seen during training should belong to a different semantic category, and not to other kind of distributional shifts. The take home message of this paper is that at this stage of research for improving open-set recognition, the best that we can do (or almost) is to use stronger models for close-set recognition (as the title suggests).