An Empirical Exploration of Open-Set Recognition via Lightweight Statistical Pipelines
Authors: Shu Kong
Deva Ramanan
Venue:  (2021)
Keywords: open-set recognition
anomaly detection
statistical models
Gaussian Mixture Models
open-world image classification
open-world semantic segmentation

Abstract: Machine-learned safety-critical systems need to be self-aware and reliably know their unknowns in the open-world. This is often explored through the lens of anomaly/outlier detection or out-of-distribution modeling. One popular formulation is that of open-set classification, where an image classifier trained for 1-of-$K$ classes should also recognize images belonging to a $(K+1)^{th}$ "other" class, not present in the training set. Recent work has shown that, somewhat surprisingly, most if not all existing open-world methods do not work well on high-dimensional open-world images (Shafaei et al. 2019). In this paper, we carry out an empirical exploration of open-set classification, and find that combining classic statistical methods with carefully computed features can dramatically outperform prior work. We extract features from off-the-shelf (OTS) state-of-the-art  networks for the underlying $K$-way closed-world task. We leverage insights from the retrieval community for computing feature descriptors that are low-dimensional (via pooling and PCA) and normalized (via L2-normalization), enabling the modeling of training data densities via classic statistical tools such as kmeans and Gaussian Mixture Models (GMMs).

Under review as a conference paper at ICLR 2021
AN EMPIRICAL EXPLORATION OF OPEN-SET RECOG- NITION VIA LIGHTWEIGHT STATISTICAL PIPELINES
Anonymous authors Paper under double-blind review
ABSTRACT
Machine-learned safety-critical systems need to be self-aware and reliably know their unknowns in the opendworld. This often explored through the lens of modeling. One popular formula-
anomaly/outlier detection or out-of-distributte tion is that of open-set classification, where an image classifier trained for 1-of-K K classes should also recognize images belonging to (K + 1)th "other" class, not present in the training set. Recent work has shown that, somewhat surprisingly, most if not all existing open-world methods do not work well on high-dimensional open-world images (Shafaei et al., 2019). In this paper, we carry out an empirical exploration of open-set classification, and find that combining classic statistical methods with carefully computed features can dramatically outperform prior work. We extract features from off-the-shelf (OTS) state-of-the-art networks for the underlying K K-way closed-world task. We leverage insights from the retrieval community for computing feature descriptors that are dow-dimensional (via pool- ing and PCA) and normalized (via L2-normalizatiom) enabling the modeling of training data densities via classic statistical tools such as kmeans and Gaussian Mixture Models (GMMs). Finally, we (re)introduce the task of open-set semantic segmentation, which requires classifying individual pixels into one of known classes or "other" class. In this setting, our feature-based statistical models noticeably outperform prior open-world methods.
INTRODUCTION
Embodied perception and autonomy require systems to be self-aware and reliably know their wn- knowns. This requirement is often formulated as the open set recognition problem (Scheirer et al.) 2012), meaning that the system, e.g., a K K-way classification model, should recognize anomalous examples that do not belong to one of K closed-world classes. This is a significant challenge for machine-learned systems that notoriously over- geerreener to anomalies and unknowns on which they should instead raise warning flag (Amodei et al.) 2016).
world benchmarks: Curating open- world benchmarks is hard (Liu et al. 2019). One common
strategy re- purpossss existing classification datasets into closed open examples e.g., declaring MNIST digits 0-5 as closed and 6-9 as open (Neal et al. 2018 Oza & Patel 2019 Geng et al.) (020). In contrast, sotmaall/ouuto--ddsribbiill
(OOD) benchmarks usually generate anomalous samples by adding examples from different datasets e.g., declaring CIFAR as anomalous for MNIST (Ge et al.)
2017 Oza & Patel 2019 Liu al. 2019). Most open-world protocols assume open-world data not available during traminn Liang et al.) 2018 Oza & Patel 2019). Interestingly, Dhamija et al. (2018); Hendrycks et al. (2019b) find that, if some open examples are available during training, one can learn simple open-vs- vs-closed binary classifiers that are remarkably effective. However, [Shafaei et al. (2019) comprehensively compare various well-known open-world methods through rigorous experiments, and empirically show that none of the compared methods generalize to high- -ihhdmmmeniinnll open-world images. Intuitively, classifiers can easily overfit to the available set of open-world images, which won't 't likely exhaustively span the open world outside the K classes of interest.
In this paper, we carry out a rigorous empirical exploration of open-set recognition of high- dimensionial images. We explore simple statistical models such Nearest Class Means (NCMs), kmeans and Gaussian Mixture Models (GMMs). Our hypothesis that such classic statistical methods reliably model the closed-world distribution (through the closed-worle training data),

---

Under review as a conference paper at ICLR 2021
SOTA model recognizes its unknown "stroller"
as "molorryyle" inducing serious safety concerne.
1010
11
road
build
veget.
Sidewalk
"Otber" pixels ignored Cityscapes benchmark.
number of pixels
cur
region
pioels
predictioo
10a
sky
gole
person
terrain
fence
wall
urffic sign
bicyele
traffic light
static groundd
parkking
dynanic
br bridge
rall track
10
mm 1
train
rider
10
Lonnel
curavvan
trailes
wuard rall
pole group
10
road person building car
motorcycle
Figure 1: We motivate open-set recognition with safety concerns in autonomous systems. Left: State-of-
the-art semantic segmentation networks (Wang ct al. 2019) do not model "strollers", which are outside the K' closed-set categories in Cityscapes benchmark (Cordis et al. 2016). Here, the network misclassifies the "stroller" as "motorcycle" which can critical mistake when fed into autonomy stack because the two objects exhibit different behaviours (and so require different plans for ohstacle avoidance). Right: While classic semantic segmentation benchmarks explicitly evaluate background pixels outside the set of K classes (Everingham et al. 2015). contemporary benchmarks such as Cityscapes ignore such pixels during evaluation. As result, most segmentation networks also ignore such pixels during training. Perhaps surprisingly, such ignored pixels include vulnerable objects like wheelchairs and strollers (sec left). We repurpose these ignored pixels as open-set examples that are from the (K+1) oh "other" class, allowing for a large-scale exploration of open-set recognition via semantic segmentation.
and help avoid overfitting (an issue in open-vs-closed classifiers). Traditionally, such simple models have been used to address the open-world (Chandola et al.) 2009, Geng et al. 2020), but are largely neglected in the recent literature. We revisit these simple methods, and find them quite effective once crucial techniques are considered, as summarized by contributions below.
We build classic statistical models on top of off-the-shelf (OTS) features computed by the underlying K K-way classification network. We find it crucial to use OTS features that have been pre-trained and post-pree post-processed appropriately (discussed further below). Armed with such features, we find classic statistical models such as kmeans and GMMs (Murphy 2012) can outperform prior work. We describe two core technical insights below.
Contribution 1:
Insight-1 1 Pre-training networks (e.g., on ImageNet (Deng et al. 2009)) is common practice for traditional closed world tasks. However, to the best our knowledge, open-world world methods do not sufficiently exploit pre-training (Oza & Patel 2019). Hendrycks et al. (20T9a) report that pre-training improves anomaly detection using softmax confidence thresholding (Hendrycks & Gimpel 2017). We find pretraining to be crucial factor in learning better representations that support more sophisticated open-world reasoning. Intuitively, pre-trained networks expose themselves to diverse data that may look similar open- world examples encountered at test-time. We operationalize this intuition by building statistical models on top of existing discriminative networks, which tend to make use of pre-training by design. We demonstrate this significantly outperforms features trained from scratch, as most prior open-set work does.
Insight-2 Low-dimensional normalized features. While some existing open-world methods also exploit OTS features (Lee et al. 2018)) we find it crucial to make use of insufficiently well-known best practices for feature extraction. Specifically, reduce dimensionality, we pool spatially (Gong et al. 2014) and use principle component analysis (PCA) (Turk& & Pentland 1991). Then, to ensure features are invariant to scalings, we adopt L2 normalization (Gong et al., 2014; Gordo et al., 2017). While these are somewhat standard practices for deep feature extraction in areas such as retrieval, their combination is not well explored the open-set literature (Bendale & Bould 2016 Grathwohl al. 2019). Given particular OTS K K-way classification network, we determine the "right" feature processing through validation. In particular, we find that L2-normalization greatly boosts open-world recognition performance; spatial pooling and PCA altogether reduce feature dimension by three orders of magnitude without degrading performance, resulting in a lightweight pipeline.
Contribution 2: We re(introduce) the problem of open-set semantic segmentation. Interestingly. classic benchmarks explicitly evaluate background pixels outside the set of K classes of interest (Ev eringham al. 2015). However, contemporary benchmarks such as Cityscapes (Cordts al. 2016)
2

---

Under review as a conference paper at ICLR 2021
extracting off-the-shelf features from underlying networks originally trained for closed-world tasks Pyramid module networks for se sementic segmentation
feed-forward CNN architecture
penutining layar
sp. spatial pooling
features opan-set seranntcc segmentation object
festures used for open-set image recognttion
L2: L2-nomaliationn
Figure 2: Flowchart for extracting off- off-the-shelff (OTS) features used for open-set recognition. We determine the appropriate feature processing steps on validation set, including spatial pooling (sp) and L2 normalization (L.2). Left: for open-set image recognition, we extract OTS fcatures the last convolution layer of K K-way classification network. Right: for open-set semantic segmentation, we extract OTS features from the "pyramid head" module, which has sufficiently captured multi-scale information. We do not adopt spatial pooling and instead use the per-pixel features to represent pixels. Note that, different from our practice, many other methods like OpenMax (Bendale & Boult 2016) and generative methods (Grathwohl al. 2019) work on logit features, which are too invariant to be effective for open-se set recognition (cf. Figure and Table 2).
ignore such pixels during evaluation. As result, most contemporary segmentation networks also ignore such pixels during training. Perhaps surprisingly. such ignored pixels include vulnerable objects like strollers and wheelchairs. Misclassifying such objects may have serious implications for real-world autonomous systems (see Figure D Instead of ignoring these pixels, we use them to explore open-worke recognition by repurposing them as open-wone world examples. Interestingly, this setup naturally allows for open-pixels in the train-set, a protocol advocated by (Dhamija et al. 2018 Hendrycks et al., 2019b). We benchmark various open-world methods on this setup, and show that suggested simple statistical models still outperform typical open-world methods. Similar to past work, we also find that simple open-vs-closee vs-closed binary classifiers serve as strong baselines, provided one has enough training examples of open pixels that span the open-world.
2 RELATED WORK
Open-set recognition. There are multiple lines of work addressing the open-world problems in the context of K K-way classification, such as amomallylottoo--istrrbuuuoo detection (Chandola et al. 2009)
Zong et al. 2018 Hendrycks al. 2019b), novelty/outierr detection (Pidhorskyi et al. 2018). Defined
on K -wy classification, these problems can be crisply formulated as open-set recognition (Scheirer
et al. 2012 Bendale & Boult 2016 Lee et al.) 2018 Geng et al. 2020 Given testing example, these methods compute the likelihood that it be belongs to the open-world via post-hoc functions like
density estimation (Zong al. 2018), uncertainty modeling (Gal & Ghahramani] 2016 Liang et al.)
2018: Kendall & Gal. | 2017) and reconstruction error of the testing example (Pidhorskyi et al. 20188)
Dehacne et 2020 Different from the above sophisticated methods, we train simple statistical
models (c.g.. GMM) which can work much better by following our proposed pipeline.
Feature extraction. Off-the-shelf (OTS) features can be extracted from the discriminative network and act as powerful embeddings (Donahue et al. 2014). Using OTS features for open-set recognition
has been explored in prior work (Oza & Patel 2019 Grathwohl al., [019 Lee et al.] 2018). OTS features can be logits, softmax and other intermediate feature activations computed by the discriminative network. Early open-set methods modify the softmax (Hendryeks & Gimpel| 2017 Bendale & Boult 2016). Grathwohl et al.) (2019) learn an energy-based model over the logit features for anomaly detection. Ozza & Patel (019)) reconstruct input images from penultimate-be features and use the reconstruction eтror the open-set likelihood. Most related to work is Lee) et al. (2018), who build Gaussian models over OTS features for anomaly detection, but relies on input image perturbation for better open-set classification performance. In contrast, we study simpler statistical models such as kmeans and GMM, and show that proper feature processing (via L2- 12-norralization and PCA) greatly boosts the efficacy and efficiency of open-set recognition.
OPEN-SET RECOGNITION VIA LIGHTWEIGHT STATISTICAL PIPELINES
In this section, we discuss various design choices in our pipeline, including (1) training schemes for the underlying cloled-world task, (2) methods for extracting and repurposing closed-workle feature descriptors for open-world recognition, and (3) the statistical density estimation models built such
3

---

Under review as a conference paper at ICLR 2021
extracted features. We conclude with (4) analysis of the additional compute required for self-aware processing (via the addition of an open-world "head" on top of the flosed-world network), pointing out that minimal additional processing is needed.
1. Network training strategies. Virtually all bate-of-the-an deep classifiers make use of large-scald pre-training e.g., on ImageNet (Deng al. 2009). which seems to consistently improve towards the sfate-of-the-art performance on the closed-world data (Sun et al.] 2017)] Mahajan et al.) 2018). However, many, if not all, open-world methods trains the discriminant network purely on the closed- world data withoul pre-training (Oza & Patel 2019: Hendrycks & Gimpel 2017). We argue that a pre-trained network also serves as an abstraction of the (pseudo) open world. Intuitively, such a pre-trained model has already seen diverse data that may look similar to the open- -ooldd examples that will be encountered at test-time, particularly if ImageNet does not look similar to the (closed) training set for the task of interest. Recently. [Hendrycks et al.] (2019a) show that pre-training improves open-world robustness with a simplistic method that thresholds softmax confidence (Hendrycks & Cimpel 2017). Our diagnostic study shows that our explored statistical models, well as prior methods, do perform much better when built on a pre-trained network than a network trained from scratch!
2. Feature extraction. OTS features generated at different layers of the trained discriminative model can be repurposed for open-set recognition (Lee et al. 2018). Most methods leverage softmax (Hendrycks & Gimpel| 2017) and logits (Bendale & Boult 2016]] Grathwohl et al.) 2019) which can be thought of as features extracted top layers. Similar to (Lee et al.]2208)) we find it crucial to analyze features from intermediate layers for open-set recognition, for which logits and softmax may be too invariant to be effective for open-set recognition (see Figure One immediate challenge to extract features from an intermediate layer is their high dimensionality, e.g. of size 512x7x7 from ResNet18 (He et al. 2016). To reduce feature dimension, we simply (max or average) pool the feature activations spatially into a 5 512-dim feature vectors (Yang & Ramanan 2015). We further use PCA, which can reduce dimension by 10x (from 512-dim to 50-dim) without sacrificing performance. We find this dimensionality particularly important for learning second-order covariance statistics as in GMMs, described below. Finally, following (Gong et al. 2014 Gordo et al. 2017). we find it crucial to L2-normalize extracted features (see Figure
3. Statistical models. Given the above extracted features, we can learn various generative statis- tical models to capture the confidence/probabbiity that a test example belongs to the closed-world distribution. We explore simple parametric models such as Nearest Class Means (NCMs) (Mensink)
et al. (013)) and class-conditional Gaussian models (Lee al.) 2018 Grathwohl et al. 2019), as well as non-parametric models such has nearest neighbors (NN) (Boiman et al. 2008 Júnior et al. 2017). We finally explore an intermediate regime of mixture models, including class-conditional)
GMMs and kmeans (Chandola et al. 2009 7, Cao et al.) 2016 Geng et al. 2020). Our models label a test example as open-world when the inverse probability (e.g., of the most-likely class-conditinnal GMMs) or distance (e.g., to the closest class centroid) is above a threshold. One benefit of such simple statistical models is that they are interpretable and relatively easier to diagnose failures. For example, one failure mode is an open-world sample being misclassified as a closed-world class. This happens when open-worke data lie close to a class-centroid or Gaussian component mean (see Figure leeft). Note that single statistical model may have several hyperparameters GMMs can have multiple Gaussian components and different structures of second-order covariance, c.g., either a single scalar, a vector full-rank general covariance per component, as denoted by "spherical", "diag" and "full", respectively. We make use a validation set to determine the hyperparameters (as well as feature processing steps listed above).
4. Lightweight Pipeline. We re-iterate that the above feature extraction and statistical models result in lightweight pipeline for open-set recognition. We now analyze the number of additional parameters in our pipeline. Naively learning a GMM over features from the last convolutional layer result in massive second-order statistics, on the order of (512 x 7)2 for a 512x7x7 Res18 feature map. We find that spatial pooling and PCA can reduce dimensionality to 50, which requires only 502 covariance parameters (a reduction of 10*). We find linear dimensionality reduction more effective than sparse covariance matrices (e.g., assuming diagonal structure). The appendix includes additional experiments. Given a class- conditioon five-component GMM (the largest found to be effective through cross validation), this requires 128KB storage per class, or 594KB for all 19 classes in Cityscapes. This is less than 1% of the compute of the underlying closed-word network
4

---

Under review as a conference paper at ICLR 2021
(e.g., HRNet at 250 MB), making it quite practical addition that enables self-aware processing on real-time autonomy stacks.
4 EXPERIMENT
We extensively validate our proposed lightweight statistical pipeline under standard open-set recog- nition benchmarks, typically focused image classification. We also consider open-set semantic segmentation, revisiting classic formulations of semantic segmentation that make use of a background label (Everingham al. 2015). We start by introducing implementation details, evaluation metrics and baselines. We then present comprehensive evaluations on each setup.
Implementation. As discussed earlier, open-world recognition is often explored through the lens of open-set classification. To ensure our approaches retain high-accuracy on the original closed- world tasks, we build statistical models on top of off-the-shelf (OTS) state-of-the-an networks. For open-set image classification, we fine-tune an SaageNet--reetraing
ResNet network (Res 18/50 in our experiments) (He et al.) 2016) exclusively on the bosed-train-set using cross-entropy loss. For open-set semantic segmentation we use HRNet (Wang et al. 2019), highly-ranked model on the Cityscapes leaderboard (Cordts et al.] 2016). We extract features at the penultimate layer of each discriminative network (other layers also apply but we do not explore them in this work). We conduct experiments with PyTorch (Paszke et al. 2017) on a single Titan GPU.
Evaluation Metric. Following past work (Hendrycks & Gimpel 2017 Lee et al. 2018), we evaluate binary detection of open-world examples using the area under the receiver operating characteristic curve (AUROC) (Davis & Goadrich| 2006). AUROC is calibration-free -free and threshold-less metric, simplifying comparisons between methods. For open-set semantic segmentation, we also use AUROC to evaluate the performance of recognizing "background" pixels as open-world examples. This is different from traditional practice in segmentation benchmarks (Everingham et al.] 2013) which treat such "background" pixels as just another class.
Baselines. Our statistical pipeline supports various statistical models. We study the simple models proposed in Section 3 including NN, kmeans, NCM, and GMMs. All models, including baselines to which we compare, are based on the same underlying classification network. Hyperparameters for all models (e.g., number of mixtures) are tuned on validation se
Classifiers. Hendrycks et al. (2019b) learn a binary open-vs-closed classifier (CLS²) for anomaly detection. Following classic work in semantic segmentation (Everingham et al. 2015), we also evaluate (K+1)-way classifier (CLS((+1)). We use the softmax score corresponding to the (K+1)th "other" class as the open-set likelihood. Both methods require open-set examples during training.
Likelihoods. Many probabilistic models measure open-set likelihood on OTS features, including Max Softmax Probability (MSP) (Hendrycks Gimpel| 2017) and Entropy (Steinhardt & Liang 2016 (derived from softmax probabilities). OpenMax (Bendale & Boult 2016) fits logits to Weibull distributions (Scheirer et al. 2011)) that recalibrate softmax outputs for open-set recognition.
C2AE Ozza & Patel] 2019) learns additional K-way classifier OTS features based on reconstruction errors, which are then used as open-set likelihood function. GDM (Lee et al. 2018) learns Gaussian Discriminant Model on OTS features and designs open-set likelihood based on Mahalanobis distance. CROSR (Yoshibashi et al.) 2019) trains reconstruction-buit that jointly performs closed-set K K-way classification and open-set recognition. G-Open (Ge et al. 2017) and OSRCI (Neal et al.] (2018) turn to Generative Adversarial Networks (GANs) to generate fake images that augment the closed-set training set, and train a discriminative model for open-set recognition. CGDL (Sun et al.) (0200) learns class-conditional Gaussian model and relies on reconstruction error for open-set recognition. The last three methods (CROSR, G-Open and CGDL) train ground-up models in contrast to our statistical models that operate on OTS features of an aiready-trained K K-way classification network. As we focus on an empirical exploration rather than achieving the state-of-the-an we refer readers to more recent approaches for the state-of-the-ee art by training ground-up models with sophisticated techniques (Zhang et al.) 2020 Chen et al.) 2020).
model
We use open-source code when available. We implemented C2AE and its authors validated our code through personal communication.
5

---

Under review as a conference paper at ICLR 2021
Table 1: Single-dataset open-set recognition (Setup-1) AUROCI. Error bars are shown gray rows. Bolded numbers mark the top-S ranked methods on each dataset. Because GDM does not L2 n2-normalize features, we add a variant that does (denoted GDMz2). L2-normalization clearly improves performance, particularly on CIFAR. Interestingly, GDMz2 underperforms NCM, implying that full-covariance Gaussian (used by GDM) overfits compared an identity covariance. GMM makes use of low-dimensional covariances (learned via PCA) and strikes balance between flexibility and generalization, achieving comparable performance to many
sophisticated approaches.
MSP
Entropy
OpenMax
G-Open
OSRCI
CROSR
C2AE
CGDL
MSP
MCdrop
GDM
GGM
NN
NCM
kmeans
GMM
977
.988
981
9964
-988
.998
.989
.994
.985
.981
.984
.984
.989
.991
.991
.993
MNIST
002
.022
.022
.005
.004
004
.022
.002
.002
.002
.002
.002
.002
.002
.002
002
886
895
894
896
910
955
922
995
888
862
868
866
901
.94
SVHN
006
006
013
017
006
004
009
003
006
.005
005
0003
003
003
.004
003
757
788
.99
903
808
686
752
8877
CIPAR
032
030
.02
035
029
908
009
0155
.011
.009
.008
007
007
008
.088
Bayesian Networks. Bayesian neural networks compute uncertainty estimates via Monte Carlo estimates (MCdrop) (Gal & Ghahramani] 2016, Loquercio et al.] 2020) and calibrated Max Softmax
Probability (MSP.) (Liang et al. (2018), which can also be used as open-set likelihoods. We
implement MCdrop via 500 samples.
4.1 SETUP-I: SINGLE- OPEN-SET RECOGNITION
Setup. We begin by following the standard benchmark protocol used in most prior work; split single dataset into open and closed sets w.r.t class labels (e.g., define MNIST digits 0-5 as the closed-set, and digits 6-9 as the open-set). This is a common practice open-set recognition (Neal et al. 2018 Oza & Patel 2019). Notably, methods do not have access open-set examples during training.
Datasets. MNIST CIFAR SVHN are popular datasets used in the open-set recognition litera- ture (Neal et al. 2018 Hendrycks & Gimpel 2017). All three datasets contain ten classes with balanced numbers of images per class. Standard protocol randomly splits six (four) classes of train/validation-set into closed (open) train/validation-sen sets. respectively. We repeat five times and report average AUROC for each method. Through cross-validation, we find reliable OTS features can computed by average-pooling features from the last convolutional layer down to 512-dim, projecting down to 50-dim via PCA, and 1.2-nommalziin.
Results. Table D shows that, perhaps surprisingly, simple statistical models (like kmeans and GMMs) defined on such normalized features already performs on par to many prior methods/ Because GDM (Lee et al., 2018)) does not L2-normalize features, we evaluate a variant that does (GDM L2). The improved performance demonstrates the importance of feature normalization, which although is well known in the image retrieval community, is not widely used in open-set recognition. We hereby focus statistical models trained normalized features, providing raw vs. normalized comparisons in the appendix.
4.2 SETUP-II: CROSS-DATASED OPEN-SET RECOGNITION
Setup. In these experiments, we use the cross datased protocol advocated by (Shafaci et al. 2019), where some outlier examples are sampled from a different dataset for training/validation. (c.g.. train on TinyImageNet-Close CIFAR-open as open-set). Conclusions drawn under this setup may generalize better due to less dataset bias in the experimental protocol (Torralba & Efros 2011).
as closed-set, validate on MNIST-open as outlier set, and test on
Datasets. We use TinylmageNet as the closed-world dataset (for K-way classification), which has 200 classes of 64x64 images, split into 500/50/50 images as the train/val/test sets. Following (Shafaei) al. 2019), we construct open val/test sets using cross-dataset images (Torralba Efros (011]. 1). including MNIST, SVHN, CIFAR and Cityscapes. For example, we use outlier dataset (c.g.. MNIST train-set) to tune/train an open- method, and test on another dataset as the open-set (c.g.. CIFAR test-set). We use bilinear interpolation resize all images into 64x64 to match Tiny Imagent image resolution. Through cross validation, we find reliable OTS features can be computed by wverage-pooling features from the last convolutional layer down to 2048-dim dim, projecting down to 200-dim via PCA, and L2-normalizing.
Results for Table are summarized below:

---

Under review as a conference paper at ICLR 2021
logit features
L2-normelized pooling features
Figure tSNE plots (Maaten & Hinton, 2008
of open-vs-closed data, as encoded by ditfer- ent features from a Res50 model (trained with pre-training in the closed world, cf. Tahle 2 Points colored w.r.t closed-world class la- bels. Left: Logit features mix open and closed data, suggesting that methods based on them (Entropy, SoftMax and OpenMax) may strug- gle in open-set classification. Right: Convolu- tional features better separate open vs closed data (cf. Figure
ciose set: tiny mny-lmmgeNle
open-set: Cityscapes
open-sat
CIFAR10
MNIST
Table 2: Cross-dataset open-set image recognition (Setup-II) AUROC1. In this setup, we train on Tiny- ImageNet, validate using outlier images from second dataset, and test using open-set images from third dataset. For each open-set dataset, we compute the average AUROC over all results when using different outlier datasets. We study two Res50 models either trained from scratch (pink row), or fine-tuned from an ImageNet-prees pretraiied model (blue row). Clearly, simple statistical models can handily outperform much prior work. Pre-training boosts open-set recognition performance for all methods (see last row pair). Binary classifiers CLS do not generalize well, presumably duc to overfitting. Somewhat surprisingly, OpenMax works quite poorly. We conjecture that the regularized logit fcatures on which is based may too invariant be effective for supplement this table with more details.
cross-datased open-set recognition. Table [
open-test
MSP
Entropy
Opx
MSP.
MCdrop
GDM
kmeans
GMM
CL.S
CI.S
MNIST
709
.712
.144
.773
.657
.811
.54
.799
.966
.961
.939
.940
.963
.939
.755
.789
.453
.832
.001
.766
.723
.957
901
.979
9663
.964
.986
.444
.522
.768
.33
.991
.944
.907
SVIIN
.310
.787
.123
863
783
380
820
.999
994
995
.993
.900
.701
,488
694
.703
.338
.750
741
.719
.712
.866
.822
.963
.977
.688
.739
.677
CIFAR
.755
.732
.471
791
809
7663
.38
.61
927
.975
948
961
.754
880
Cityse.
.739
.753
.604
862
.877
.753
.725
650
.559
.839
.903
885
.601
.919
.751
.762
851
868
784
651
.13
.715
.833
.866
867
.91
723
.734
377
752
.32
940
908
average
.75
.768
397
834
.15
781
758
857
884
946
.488
9455
.712
.96
Simple statistical models (c.g., NCM and kmeans) can outperform prior open-set methods (c.g.. C2AE and GDM). We find that 1.2-normalization greatly contributes to the success of these simple statistical methods (cf. details in the appendix). Both the metric learning and image retrieval (Mensink al. 2012 Musgrave et al. 2020) literature have shown the importance of L2-normalization. Informally, open-set recognition queries the testing example and measures how close it is to any of the closed-world training examples (Musgrave et al.) 2020).
Interestingly, kmeans performs slightly better than GMMs. Considering that the former can be seen as a special case of GMMs that have an identity covariance, we conjecture that learning other types of covariance (e.g., full-rank covariance matrix) does not help when the underlying K-way network has already provided compact feature representations.
From last row pair, we the original GDM which operates on raw features (without 1.2-normalization) This further confirms the importance of L.2 n.2noommaiizatin in feature extraction for open-set recognition.
pre-training notably improves all the methods. GDM1,2 outperforms
Perhaps surprisingly, OpenMax does not work well in this setup (though we have spent considerable effort wning it). This is consistent with the results in (Dhamija et al. 2018 Shafaei et al. 2019). and we conjecture the reason is that OpenMax cannot effectively recognize cross-datasel anomalous inputs using logit features because they are too invariant to be useful for open-set recognition (Figure 3). Similar lackluster results hold for other methods that operate on logit features (Entropy and MSP).
4.3 SETUP-III: OPEN-SET SEMANTIC SEGMENTATION
Setup. In these experiments, we (re)introduce the task of open-set segmentation by repurposing "background" pixels in contemporary segmentation benchmarks (Cityscapes) as open-world pixels. As elaborated before, such pixels are either traditionally treated just another class for segmentation evaluation (Everingham et al. 2013) or ignored completely. Instead, we evaluate them using open- world metrics such as AUROC. We will show our statistical methods also outperform other typical open-world methods. As this setup has natural access to open-world pixels during training, we explore the training of simple open-vs-elose classifiers.
7

---

Under review as a conference paper at ICLR 2021
mage
ground-truth
prediction
void mark
MSP
Figure 4: Two random images from Cityscapes-val. val, visualized with ground-truth and the predicted semantic segmentation maps by the HRNet. We visualize open-world pixels in the ground-truth (white regions), as well predicted open-world pixels for standard baseline (MSP) and our method (GMM). MSP tends to predict segment boundaries open-pixels, while GMM tends find open-set objects (street-shop and rollator as pointed by the red arrows).
Figure Performance CLS versus amount of (open) training data. We train CLS on the OTS features of the state-of-the-art HRNet, which has already exploited all closed-world pixels in the train-set. The binary open-vs-closed classifier CLS outperforms (L)(K+1) presumably due to that the former being trained on balanced batches, With fewer than 50 images, GMMs outperform such discriminative models. But with enough open training examples, simple binary classification performs remarkably well. However, because GMMs do not require any open training examples, it cannot overfit to them and so may generalize better to the open-world.
0.850
0.825
AUROG
0.800
0.775
0.750
CLS E
0.725
CLS?
0.700
GMM
0
100
200
300
400
500
eimages
Datasets. Cityscapes (Cordts et al., 2016) provides per-pixel annotations for urban scene images 2014x2048-resolutton for autonomous driving research. We construct our train- and val-sets from its 2,975 training images, in which we use the last 10 images as val-set and the rest as train-set. We use its official 500 validation images as our test-set. The "background" pixels (shown white of ground-truth visual in Figure 4 are the open-world examples in this setup. Through validation, we find reliable OTS features can be computed by projecting features from the last convolutional layer from 720 down to 100-dim via PCA, and L2-normalizing.
Results. For our statistical models (as well as GDM), we randomly sample 5000 closed-world pixel features from each class, it is prohibitively space-consuming use all the pixel features from the Cityscapes train-set. We show quantitative comparison in Table 3 and list salient conclusions below.
Clearly, simple statistical models (c.g., NN and GMM) perform significantly better than the classic open-world methods (e.g.. MSP and OpenMax). However, when training on large amounts of open-pixels, CLS methods achieve significantly better performance. This clearly shows the benefit of training on open-world pixels (Hendrycks et al. 2019b)). We do note that GMMs do not need any open pixels during learning, and so may generalize better to novel open-world scenarios not encountered in the training set (Figure 5).
GDM performs poorly, probably due to arbitrary scales of the raw features that are too uninformative to be used for open-set pixel recognition. We note that other statistical methods all struggle with raw pooled features (cf. appendix). However, once we L2-normalize the pixel features to be scale-invariant, these statistical methods perform significantly better (as reported in this table).
Figure 4 shows qualitative results. MSP predicts segment boundaries as open-pixels. This makes sense the MSP mostly returns aleatoric uncertainties corresponding to ambiguous pixel sensor measurements around object boundaries (Kendall & Gal 2017). In contrast, GMM reports open- pixels on truly novel objects, such the street-shar and ro collarrr, both of which ignored by the semantic segmentation network during training HRNet (Wang et al. 2019). These regions appear be caused by epistemic uncertainty arising from the lack of training data (Kendall & Gal 2017).
Figure plots AUROC performance vs model size for various statistical models. Notably, NN consumes the most memory, even more than the underlying networks. GMMs perform the best and are quite lightweight, only consuming 0.6MB when built on the HRNet model (250MB).
8

---

Under review as a conference paper at ICLR 2021
Table 3: Open-set semantic segmentation (Setup-ILI) AUROC1. Simple statistical methods (GMMs) out- perform prior methods, with the notable exception of discriminative classifiers (CLS² and CLS(K+1) that have access open-set training examples. F Janalyzes this further, demonstrating that GMMs outperform such discriminative models when they have access to less open training examples, suggesting GMMs may better generalize to never-before-see open world scenarios.
MSP
Entropy
OpenMax
C2AE
MSP.
MCdrsp
GDM
NN
NCM
kmeans
GMM
CLS
CLS((4+>>
600
603
612
.63
.795
open set performance (AUROC)
1.0
Figure 6: AUROC vs. memory cost (MB) for various statistical models for open-set semantic segmentation. NN stores ~-10kk OTS features, which is larger than the underlying network (HRNet). We explore GMMs with various covariance structures (spherical, diago- PCA, and mixture components.
0.8
0.6
kmeans-pca
GMM spherccll
0.4
GMM--daag-pca
nal, full), feature dimensionality We find the best AUROC-memory tradeoff on the validation set (shown here to be single-mixture GMM with full-covariance and PCA), and find generalizes well held-out test (cf. Appendix).
GMK full pcs
NNH-pc
SOTA network (HRNet)
0.0
10
memory
CONCLUSION
We explore an empirical exploration of open-set recognition via lightweight statistical pipelines. We find simple statistical models quite effective if built on properly processed off-the-shelf features computed by the discriminative networks (originally trained for the closed-world tasks). Our pipelines endow K-wat K-way networks with the ability to be "self-aware", with negligible additional compute costs (0.1%). Finally, we (ittoouree the task of open-sed semantic segmentation by repurposing background pixels open-world examples, requiring classification of individual pixels into of K' known/closed-wer classes and an "other" open-world class.
REFERENCES
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv:1606.06500 2016.
Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In CVPR, 2016.
Oren Boiman, Eli Shechtman, and Michal Irani. In defense of nearest-neighhor based image classification. In CVPR, 2008.
Yue Cao. Mingsheng Long, Jianmin Wang. Han Zhu, and Qingfu Wen. Deep quantization network for efficient image retrieval. In AAAI, 2016.
Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly detection: survey, ACM computing surveys (CSUR), -1(3):1-58, 2009.
Guangyao Chen, Limeng Qiao, Yemin Shi, Peixi Peng, Jia Li, Tiejun Huang, Shiliang Pu, and Yonghong Tian. Learning open set network with discriminative reciprocal points. In ECCV, 2020.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
Jesse Davis and Mark Goadrich. The relationship hetween precision-recall and roc curves. In ICMI., 2006.
David Dehaene, Oriel Frigo, Sébastien Combrexelle, and Pierre Eline. Iterative energy-based projection on normal data manifold for anomaly localization. In ICLR, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Akshay Raj Dhamija, Manuel Günther, and Terrance Boult. Reducing network agnostophobia. In NeurIPS, 2018

---

Under review as a conference paper at ICLR 2021
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: deep convolutional activation feature for generic visual recognition. In ICML, 2014.
Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision, 111(1)98-133, 2015.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In ICML, 2016.
Zong Yuan Ge, Sergey Demyanov, Zetao Chen, and Rahil Garnavi. Generative openmax for multi- class open set classification. In BMVC, 2017.
Chuanxing Geng, Sheng-jun Huang, and Songcan Chen. Recent advances in open set recognition: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Yunchao Gong, Liwei Wang, Ruiqi Guo, and Svetlana Lazebnik. Multi-scale orderless pooling of deep convolutional activation features. In ECCV, 2014.
Albert Gordo, Jon Almazan, Jerome Revaud, and Diane Larlus. End-to-end learning of deep visual representations for image retrieval. International Journal of Computer Vision, 2017.
Will Grathwohl, Kuan-Chieh Wang, Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. In ICLR, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Dan Hendrycks and Kevin Gimpel. baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR, 2017.
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In ICML, 2019a.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. ICLR, 2019b.
P.R.M. Júnior, R.M. De Souza, Rafael de O Werneck, Bernardo Stein, Daniel V Pazinato, Waldir de Almeida, Otávio AB Penatti, R.S. Torres, and Anderson Rocha. Nearest neighbors distance ratio open-set classifier. Machine Learning, 2017.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In NeurlPS, 2017.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NeurlPS, 2018.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of -uitt--dist image detection in neural networks. In ICLR, 2018.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long-tailed recognition in an open world. In CVPR, 2019.
Antonio Loquercio, Mattia Segu, and Davide Scaramuzza. A general framework for uncertainty estimation in deep learning. IEEE Robotics and Automation Letters, 2020.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 2008.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, 2018.
10

---

Under review as a conference paper at ICLR 2021
Thomas Mensink, Jakob Verbeek, Florent Perronnin, and Gabriela Csurka. Metric learning for large scale image classification: Generalizing to new classes at near- zero cost. In ECCV, 2012.
Thomas Mensink, Jakob Verbeek, Florent Perronnin, and Gabriela Csurka. Distance-based image classification: Generalizing to new classes near-zero cost. IEEE Transactions Pattern Analysis and Machine Intelligence, 65(11):2624-233) 2013.
Kevin P Murphy, Machine learning: probabilistic perspective. MIT press, 2012.
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. A metric learning reality check. In ECCV, 2020.
Lawrence Neal, Matthew Olson, Xiaoli Fern, Weng-Keen Wong, and Fuxin Li. Open set learning with counterfactual images. In ECCV,,20
Poojan Oza and Vishal M. Patel. C2AE: class conditioned auto-encoder for open-set recognition. In CVPR, 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Stanislav Pidhorskyi, Ranya Almohsen, and Gianfranco Doretto. Generative probabilistic novelty detection with adversarial autoencoders. In NeurlPS, 2018.
Walter Scheirer, Anderson Rocha, Ross J Micheals, and Terrance E Boult. Meta-recognition: The theory and practice of recognition score analysis. IEEE Trunsactions on Putlern Anulysis and Machine Intelligence, 13(8):1689-16) 2011.
Walter Scheirer, Anderson Rezende Rocha, Archana Sapkota, and Terrance Boult. Toward open set recognition. IEEE Transactions Pattern Analysis and Machine Intelligence, 35(7):
1757-1772, 2012.
Alireza Shafaei, Mark Schmidt, and James Little. A less biased evaluation of sample detectors. BMVC, 2019.
Jacob Steinhardt and Perey Liang. Unsupervised risk estimation using only conditional indepen- dence structure. In NeurIPS, 2016.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. ICCV, 2017.
Xin Sun, Zhenning Yang, Chi Zhang, Keck-Voon Ling, and Guohao Peng. Conditional gaussian distribution learning for open set recognition. In CVPR, 2020.
Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR, pp. 1521-1528, 2011.
Matthew A Turk and Alex P Pentland. Face recognition using eigenfaces. In CVPR, 1991.
Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, and Bin Xiao. Deep high-resolution representation learning for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019.
Songfan Yang and Deva Ramanan. Multi-scale recognition with dag-cnns. In CVPR, 2015.
Ryota Yoshihashi, Wen Shao, Rei Kawakami, Shaodi You, Makoto lida, and Takeshi Naemura. Clasiffiatio--eccostttuttioo learning for open-set recognition. In CVPR, 2019.
Hongjie Zhang, Ang Li, Jie Guo, and Yanwen Guo. Hybrid models for open set recognition. In ECCV, 2020.
Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Dacki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection, In ICLR, 2018.
11

---

Under review as a conference paper at ICLR 2021
APPENDIX OUTLINE
As elaborated the main paper, we introduce a lightweight statistical pipeline for open-set recognition by repurposing off-the-shelf (OTS) features computed by state-of-the-ar recognition network. As our pipeline does not require (re)training the underlying network, it is guaranteed to replicate the state-of-the-art performance of the network on the (closed-world) task for which it was trained, but still allows the final recognition system to properly identify never-before-seen data from the open-world. In the appendix, we expand on our pipeline, including more experiments, analyses and visualizations. We outline the appendix below.
Section A Data statistics for open-set semantic segmentation. We provide data details used for open-set semantic segmentation (Setup-III), motivated by safety concerns in autonomous stacks as shown in Figure | left.
Section B: Detailed results by statistical models. We provide detailed results on the open-set recognition including open-set image recognition (Setup-II) and open-set semantic segmentation (Setup-IIf). We detail the performance of the various statistical models studied in the main paper, including Nearest Neighbor (NN), Nearest Class Mean (NCMs), kmeans and Gaussian Mixture Models (GMMs).
Section C Reduced dimension via PCA. We show that PCA can reduce dimensionality significantly (making our pipeline quite lightweight), while maintaining or even improving performance.
Section D Performance vs. memory/compute. We rigorously evaluate the memory/compute costs of our varous statistical pipelines, emphasizing solutions that are both accurate and lightweight.
Section E Visualization of Gaussian component means. One benefit of our simple statistical mod- els is their interpretability; visualize Gaussian means through centroid images, and demonstrate that they correspond to canonical objects (e.g., those with standard poses and clean background).
Section F Open-Source demonstration. We include code (via Jupyter Notebook) for open-set semantic segmentation, assuming one has access to precomputed features from HRNet (Wang et al.) 2019)
A SETUP FOR OPEN-SET SEMANTIC SEGMENTATION
As we (re)introduce the task of open-set semantic segmentation for exploring open-set recognition, in which we re-purpose "backgroud" pixels of Cityscapes as open-world examples (that are from the (K+1)th "other" class). We hereby list the statistics of open and wlosed-world examples (pixels). Cityscapes training set has 2.975 images. We use the first 2,965 images for training, and hold out the last 10 as validation set for model selection. We use the 500 Cityscapes validation images as our test set. Here are the statistics for the full train/val/test sets.
train-set set for closed-pixels: 2,965 images providing 334M closed-sed set pixels.
train-set for open-pixels: 2.965 images providing 44M open-set pixels.
val-set for closed-pixels:
images providing 1M closed-set pixels.
val-set for open-pixels: 10 images providing 0.2M open-set pixels.
test-set for closed-pixels 500 images providing 56M pixels.
test-set for open- open-pixels: 500 images providing 8.3M pixels.
B DETAILED RESULTS BY STATISTICAL PIPELINE
In Figure 10 on the last page of this document, we provide detailed results of various statistical models for open-worte tasks, including open-set image recognition and open-set semantic segmentation. In the main paper, we state that we tune and select statistical models (if it has hyper-parameters to tune) on the small validation set, and report on the test set with the selected (best-perforning) model. Such hyper-parameters can be the number of means/components in kmeans and GMM models, and covariance types in GMM - "spherical", "diagonal" and "full" denote that the covariance matrix of each Gaussian component is controlled by single scalar, vector and full-rank matrix, respectively. It demonstrates that validation can reliably tune the statistical models whose performance can be translated to the test sets. Moreover, also record the detailed results of whether using
12

---

Under review as a conference paper at ICLR 2021
0.95
open-set open-set performance (AUROC)
val-set
0.94
test-set
0.93
0.92
0.91
0.90
200
400
600
800
reduced dimension by PCA
Figure 7: Study of open-set performance versus feature dimension reduced by PCA. We study this with the experiment of open-set image classification, where Tiny SunyllmagNNe/Cityscaaaa We extract OTS features from the Res50pt network (as detailed in the paper). The spatially pooled (and 1.2- normalized) feature has 2048 dimension. To avoid randomness (as in some statistical models like GMM and k-means), NCM model which simply computes per-class mean feature and computes the distance of nearest center as the open-set likelihood. Surprisingly, using PCA to reduce feature dimension can improve open-set performance!
are the closed/open sets.
L2-normalization on the features. We can see that L2-normalization greatly boosts open-world performance.
Table 4 and Bist details of various methods on cross-dataset evaluation (Setup-II), supplementing
Table Please refer to the caption for details.
C PERFORMANCE VS. PCA REDUCED DIMENSION
As analyzed in the main paper, PCA is an important technique to make our pipeline lightweight by considerably reducing feature dimensions. We study how statistical model performs under different reduced feature dimensions by PCA. We choose the simplistic NCM method which does not induce randomness (unlike kmeans and GMM which require random initialization for learning). We study this through open-set image recognition under SeIup-II. To simplify the study, choose (resized) Cityscapes images as open-set data, i.c., we use TinyImagene//Ciyscaapp the network Res50 in the diagnostic study, the original dimension of the pooled
images as the closed/open
set. As features is 2048. In Fig. [ we plot the performance (AUROC) of NCM as a function of reduced dimension by PCA. Perhaps surprisingly, PCA even improves the open-world performance while significantly reducing feature dimension (from 2048 to 100)!
D PERFORMANCE VS. MEMORY/COMPUTE
As seen previously, PCA reduces the feature dimension greatly and hence makes the statistical models quite lightweight. We now study how lightweight different statistical models can be by considering the open-world performance. We analyze the models learned for two tasks (open-worid image classification and open- world semantic segmentation). where the OTS features have dimension 2048 (extracted from Res50) and 720 (extracted from HRNet), respectively. We use PCA to reduce the features dimensions to 200 and 100, respectively.
We focus on NN, kmeans and GMMs, all of which operate the PCA reduced features (with 1.2- normalization). NN is the straightforward baseline that memorizes all training examples to recognize open-set examples. For GMM, study it by specifying three types of covariance "spherical", "diag" and "full" meaning the covariance matrix of each Gaussian component controlled by single scalar, vector full-rank (symmetric) matrix, respectively. For open-set semantic segmentation, is prohibitively prace-consuming to memorize all the pixel features the whole train set. So we randomly sample 5000 pixels from each of the 19 classes defined by Cityscapes (~200k in total).
In Figure g we draw the open-world performance (AUROC) for the two tasks w.r.t the total memory cost (i.e., the required space model's parameters). We that NN takes the most memory
13

---

Under review as a conference paper at ICLR 2021
Open Classification SaneylmmaeNeetts-Ciyyappppp val
Open Segmentation (Cityscapes): val
20t perfoetoo (NRRC))
(MUROC)
0.8
8.8
0.6
0.6
k-meens
I-meams
GMHH-spherica
0.4
0.4
GMM-dlag
GMM4 full
GMK
0.2
NN
0.2
SOTA network (Res50)
network (HRNet)
10
10a
10a
momery cost (MB)
momory cost (MB)
Figure 8: Open-set performance w.r.t memory cost (MB) by different models. Memory cost means the space required to store parameters these models. Left: open-set (200-way) image recognition for TinyemageNer- vs-Cityscapes. Right: open-set semantic segmentation (19 classes) on the Cityscapes. NN memorizes training examples for open-world recognition, and hence it consumes huge memory to store OTS features of training examples (more memory consumption than the underlying SOTA models). Compared with the underlying networks, GMM-spherical and k-means induce negligible computation cost. They also perform considerably GMM-full on
better than NN and -faMu on open-set image classification, but not as well speldworld semantic segmentation. These plots clearly serve as guidelines to choose the appropriate statistical models on specific tasks. Note that the validation performance shown here can he nicedly translated to test sets, as detailed in Figure T0
usage, which is even more than the underlying networks. In contrast, GMM-spherical and k-means models are significantly more compact, i.e., ~0.3MB for both tasks. Moreover, on open-wote image classification (Figure&-left), k-means and GMM-spherical achieve much better performance than the other models. Interestingly GMM-full achieves the best and stable performance for open-set semantic segmentation (Figure rightt but not for open-set image classification (Figure 8 left). Despite this, we note that the validation performance (as plotted here) can be nicely translated to real test sets, as shown in Figure IOp.
is worth noting that the specified PCA-reduced dimension is not optimal that an even lower dimension can lead to better open- world performance (cf. Figure (7). We do not exhaustively explore this in this work, but instead emphasize that our pipeline is quite lightweight that can be tuned on specific tasks, e.g., 0.6MB GMM-full compared with 250MB HRNet for semantic segmentation.
E VISUALIZATION OF GAUSSIAN MEANS
As statistical models interpretable, we visualize what the statistical model can capture. To do so, we visualize per-class Gaussian means through medoid images, which are training images that have features closest to their corresponding per-class mean feature. We show the medoid images in Figure ( as well as some random images sorted by the cosine similarity (i.c., Euclidean distance on 12-normalized features) the Gaussian means within each class. We can see the medoid images most likely capture the canonical objects of each class, e.g., those of with "standard" pose and clean background.
F OPEN-SOURCE DEMONSTRATION
We attach our code (via three Jupyter Notebook files) to demonstrate our exploration of open-set set recognition. One can run the code with access to networks (Res50 and HRNet (Wang et al. 2019)) trained for closed-wond space limit, but we are committed to relcasing them to the public after paper notification. We refer readers to the Jupyter Notebook files for self-explanatory descriptions.
tasks. We are not able to upload models or pre-computed features due to
Lacppp_pp______aa 1.2norm.pynb' We show how we train, select and evaluate GMMs on cross- dataset open-set image recognition (Setup-II).
"demo_ Oilemo_Open-See Open- Beeen-Se--Image-Rese
"demo _sne tsne_visual res50pt.ipynb". We show t-SNE visualizations of OTS features of cross-dataset open-set examples (IItup-II). This intuitively demonstrates the benefit of exploiting OTS features for open-set recognition.
"emoo voemanti--semmentttioo under Setup-III, open-set semantic segmentation.
We demonstrate how train and evaluate GMM
14

---

Under review as a conference paper at ICLR 2021
Table 4: Cross-dataset evaluation (Sctup-IT) with K-way classification network that is trained-from: scratch. We report performance with the AUROC metric. This table supplements Table Recall that train on TinyImageNet the closed set, use another dataset outlier set tune and select model, and report on third dataset as the open-set. All the methods operate on off-the-sheil features extracted from the underlying classification network. We report their averaged performance and standard deviation in the last two columns.
outlier set
MNIST
SVHN
CIFAR
Cityscapes
avg
std
open
CF
C5
MN
SV
CF
CS
MN
SV
CF
CS
SV
CF
CS
MSP
Entropy
OpenMax
773
.803
.750
.862
.773
.803
.750
.862
.773
.803
.750
.862
.773
.803
.750
.862
.797
.042
MC-dropout
.07
.821
.725
.854
.835
.941
.749
.890
.099
.830
.739
.872
.635
.841
.749
.890
.777
.066
C2AE
864
7744
.683
.708
.864
744
.683
.706
.757
.701
.755
.797
.757
.701
.756
.797
.752
.055
GDM
.712
.724
.45
822
.712
.725
.457
.822
.712
.142
GDM-L2
980
.993
.923
.928
.947
.998
.974
.906
.979
.995
.975
.833
.849
.943
.877
.945
.940
.050
GMM
.907
:999
.979
.905
.987
.999
.979
.905
.973
.989
.985
.821
.811
.930
.817
.928
.930
.05
CLS-2
.999
:362
A28
.514
.978
.909
.955
.380
.933
.68
.999
.544
.959
.621
.530
.999
.764
.260
Table 5: Cross-dataset evaluation (Setup-II) with K-way classification network that is finetuned from an ImageNet pre-trained model. We report performance with the AUROC metric. This table supplements Table [2] Recall that we train on Tinylmagele ImageNet as the closed-set, use another dataset as outlier set to tune and select model, and report on the third dataset as the open-set. All the methods operate on off-the-shell features extracted from the underlying classification network. We report their averaged performance and standard deviation in last two columns.
outlier set
MNIST
SVHN
CIFAR
Cityscapes
avg
open test set
MN
SV
CF
CS
MN
SV
CF
CS
MN
SV
CF
CS
MN
SV
CF
CS
MSP
.755
.770
.725
.751
.775
.770
.725
.751
.775
.770
.725
.751
.775
.770
.725
.751
.756
.000
Entropy
.789
.787
.732
.702
.789
.787
.732
.762
.789
.787
.732
.762
.789
.787
.732
.702
.788
.03
OpenMax
458
.25
473
515
.588
125
.473
.15
.458
.125
.473
.515
.430
.116
.464
.25
.39
164
MSP
.032
.953
.722
.633
.791
851
.832
.863
.791
.051
.832
.063
.791
.851
.334
.27
MC MC-dropout
.783
C2AE
GDM
.945
.37
.717
.842
.845
.637
.780
.831
.896
.758
.079
GDM-L2
857
.99
961
513
957
.99
.961
.13
.957
.999
.961
.513
.957
.999
.91
.531
.357
196
NN
.001
.994
.227
.725
.902
.994
.527
.715
.901
.994
.927
.715
.901
.994
.927
.715
.384
103
NCM
.799
.956
975
.83
.799
.95
.975
.833
.979
.995
.975
.633
.979
.995
.775
.838
946
065
kmeans
.90
.961
.994
.963
.899
.57
.981
.97
.904
.968
042
GMM
CLS-2
15

---

Under review as a conference paper at ICLR 2021
per lass centroid image
nore nove
Enyptten
cov-simllaity
rocting cheir
convaimantty 819
con--l ttv-iiiaantyy 700
con-timilar Ly:B 765
com-imlarkpp..M CovemmaarBB....7
Stn-smlantlyy.oo 5.600
brown beer
cow-sillrtty 0.0
cos-tmiaaityy. 794
con-dimiler byd 774
con-tmiarrhy 759
cos-siile 0.76
cos-simlartly S. 6560
con-simlatlly
pizza
mosimilatty 0.854
m-si-iiaily...mm cos-imilla by: 782
com mw-imllaffydd..m mm-smlldtly 0.618
con-simllatty oll-tttlaatty.omm
ty: cos-simterky.0.777
con-simlarlly
hourdlass
con-siiaatty 0.834
cos wit.tiitttyyy...o 7012
o09- cos-simllarty by:D 785
cos-timlarky-..700
con-smlarEy-..rr
COS- www-simlarttlyyo 62727
con-simflanty
dbeinoss
stneiiilanil 0.822
masimileilyy..om 783
cossimilar Ly:B 784
cowumlarkppt..0r
comomlerdyd...gg
mn-simladly....
Figure 9; Visualization of per-class Gaussian with medoid images (whose features closest means within their corresponding classes). comparison, we show some random images sorted by their cosine similarity to the per-class Gaussian mean. We can see the medoid images capture "canonical" objects representing the corresponding classes, e.g., those with "standard" shape and cleaner background. This visualization suggests statistical models quite interpretable.
16

---

Under review as a conference paper at ICLR 2021
Rertte
Beaffrt
ReeSiee
Reedito
0.60226
softmax
083089
379942
679553
0.59017
043982
375010
679473
000221
0.57166
033722
12:00
092295
entropy
0,59956
18000
1948
013003
0,61358
openmax
0.65467
1.00
1,0000
076:08
raw feat.
w/ L2
1+13
043155
13543
33500
075455
047122
1040
5012
000833
1900
13000
03003
033250
077544
30730
C91607
0 0.50528
0 170
NN
pkrwar
061915
13000
2340
077272
000079
036700
15770
08147
0,49209
0.76910
020107
13130
3008
E3000
082069
091111
class centroid
0.50273
0.61522
015143
0,48175
0.71535
1.800
kmeans k=1
0.55085
0.62010
17862
13605
653311
6000
600MT
340304
189G1
DAMOS
078145
0.877
0.0003
3.53733
kmeans 0.60075
0.64429
1DMM
2,008
19000
23003
076419
037000
073905
1.9091
06781
000031
kmeans k=5
0.60218
0.64907
068349
1300%
19000
036045
spreiil (m5)
1985
22700
01815
1796
kmeans k=7
0.61918
0.65465
LUMB
238
LIERRY
28481
E51394
0.75522
054025
165817
22007
078872
DANE:
060155
0329-1
56641
1DMM
18790
030133
CARME
GMM spherical k=1
0.53039
0.63315
GMM
0.62693
0.63907
Ad% 1t
1.800
1.00
03212
01603
1.800
spherical k=3
A80 v0
374955
04020
080360
18005
LUMR
2752
24180
10000
1M0M
1859
spherical k=5
GMM
0.62961
0.65128
074405
130715
5300
029134
07727
271256
C2915
0TE22
155150
C5:60
201710
GMM diag k=1
0.53820
0.62669
079907
GMM diag k-3
0.61556
0.64717
19009
110007
D10m
341977
01701
1.00
17403
10:00
159100
028411
12001
20686
0.62053
0.63280
062233
13140
-20007
ERWAS
048506
1567
diag k=5
C2MB
06081
15201
0,0002
000
25000
GMM full k=1
0.70429
0.74774
0.79511
01407
371866
k8k-1
11711
10074
18/377
038719
1850
GMM full k=3
0.71349
0.72420
178.01
18843
1375.0
000/35
1488
Lo0n
130/77
63718
253217
E2027
13007
GMM full k=5
0.71389
0.73348
060911
375617
D4R424
078013
2-0030
1.000
0.77999
Figure 10: Detailed results of (left) open-set image recognition using Cityscapes images as cross-dataset open-world examples under Setup-ll) and (right) open-set semantic segmentation (Setup-IlI). In each cell, first and second (if existing) row numbers denote AUROC performance on the val and test sets, respectively. We highlight the best performance on the test set. As for notation, gGMM means learn GMM "globally" on the whole closed train-sct, agnostic class labels; while cGMM means that leam class-conditional GMMs. For open-set semantic segmentation, we only train GMM in a cansscooniiiioal fashion (i.c., cGMM), because using pixel features from all classes to train global GMM is prohibitively time-consuming. "Raw feat." means the feature we extract from the last convolution layer without 1.2-normalization, and "w/ L2" means we L2-normalize the extracted features. Clearly, n2-normalization greatly boosts open-world performance.
the val set, on which we tune the hyper-pann pprrmmeer and report the performance
17


Reviews:
[{"review_id": "jVXQb4TxgaN", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an open-set recognition approach that uses simple statistical measures (such as GMM’s and KMeans) on top of post-processed intermediate features extracted from closed-set deep models. It finds that i) this “lightweight” pipeline outperforms prior methods on open-set image recognition across multiple evaluation protocols at much lesser memory and compute cost, and ii) Open-world recognition generally benefits from using models pre-trained on large datasets such as ImageNet rather than training from scratch, iii) the technique also generalizes to open-world semantic segmentation.\n\nStrengths\n\n– The paper studies an important problem, is motivated clearly, and is well-written\n\n– The proposed approach is intuitive, memory-efficient, and appears to clearly and consistently perform prior work\n\n– The modeling choices are motivated and analyzed well, for eg. Fig 3 clearly illustrates why l2-normalized and pooled intermediate features more clearly capture in-domain-ness than logits used predominantly in prior work\n\n– The set of experiments and baseline comparisons appears comprehensive\n\n– The performance v/s memory tradeoff of the current method vs prior work is analyzed well\n\n– The limitations of the proposed method are also explored, for eg. Fig 5 that shows how for open set semantic segmentation, the proposed method is a better alternative than training a binary classifier only when very little out-of-domain data is available to train on\n\nWeaknesses\n\n– Studying how/whether the choice of layer from which features are extracted affects performance would have been an interesting addition\n\nAdditional comments / suggestions\n\n– Fig 6 is difficult to read since some of the lines overlap, varying opacity might help with readability\n\nOverall comments\n\nThis is an interesting and well-written paper that proposes simple and memory-efficient alternatives for open-world recognition that consistently outperform more complex methods from prior work.\n\n-------- post-rebuttal comments --------\n\nI have read the concerns raised by other reviewers as well as the author response. I still feel that this is an interesting work and its findings are of potential value to the community: There has been a considerable amount of recent work in open set recognition for deep models, and this paper calls into question the need for sophisticated techniques by showing (fairly rigorously, in my opinion) that simple strategies and the right choice of feature engineering works better. I agree with the authors that not using ImageNet pretraining is an unrealistic and unnecessary constraint – moreover, the method generalizes even when evaluated on datasets such as MNIST and SVHN, which are distributionally very different from ImageNet. Further, I think the paper acknowledges prior work appropriately and did not find any of the claims made to be unreasonable. I agree with R1's concerns about overlap with two recent papers, but found the author response to be satisfactory. Overall, I will retain my accept rating.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "hGo92z2m7QD", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of Paper: The main claim of the paper is that out of distribution (OOD) detection can be done by use of pre-training and appropriately deriving a feature space from SOTA activations  via pooling, PCA based dimensionality reduction, L2 normalization.  Classical methods such as GMMs, k-means etc. can then be used to estimate the probability density function of features for use in OOD detection.  Several alternative schemes are compared against many OOD detection schemes.  \n\nKey contributions claimed are that pre-trained nets have information about open-world statistics and off-the-shelf net features along with appropriate choice of a low-dimensional representation helps in outperforming conventional OOD schemes.\nThe paper builds upon recent work on OOD method benchmarking method by Shafaei et al (2019) that argues that most OOD schemes are not able to pass a less unbiased test designed ( wherein a Source data set is used for training using standard methodology, Validation data set for estimating a decision function between source and validation,  and finally the probability of outlier detection on other datasets and their variability is estimated to get robust view of OOD.).   \n\n1. The general approach is, as I think, quite misguided: the idea is to use pretrained datasets, extract features from it and apply a Gaussian mixture model. This means though, that such an approach will be able to recognize only classes for which features were made available (hence, the closed-world problem is just extended by additional knowledge about more classes from selected datasets, no open-world problem is solved).\n\n2. The fact that pretraining helps, is a generic statement which depends on the statistics of the dataset used for pretraining and then the dataset on which it is tested.\n\n3. the setup in 4.1 (split a single dataset into open and closed sets w.r.t class label) is questionable - as the image statistics in the partition are same.  \n\n4. in 4.2 the authors use an 'open' dataset for validation/tuning - this makes this dataset not open per definition. It may be true that it helps in the model generalizing better, but the terminology is still misguided.  In my view, the paper applies the testing methodology of Shafaei et al (2019) incorrectly to design an OOD algorithm and claim its superiority.   \n\nNone of the tables in the paper provide error bars.  In order to convince that the insights are correct I would expect that the experiments need to be run with a larger sampling of outlier datasets (as done in Shafaei et al).   \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "VaGuRP-R0kR", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Methodology:\nThe paper tackles the so-called open-set classification where query examples outside any of the classes in the training set should be detected at inference. By combining the feature extractor based on fashionable deep models and the classical clustering methods (k-means, GMM, etc), the paper empirically shows that this pipeline can address many open-set problems in realistic scenarios.          \n            \nPros:\nUnfortunately, none.\n\nCons:\nThe paper basically reinvents the wheel. The so-called open-set classification problem is precisely the anomaly detection problem that has been studied for decades (see V. Chandola, A. Banerjee, and V. Kumar, “Anomaly detection: A survey,” ACM Computing Surveys, vol. 41, no. 3, p. 15, 2009), and the use of classical clustering methods (k-means, GMM, etc) was among one of the first efforts for anomaly detection. Applying the identical strategy on top of the fashionable deep features is the routine treatment for anomaly detection 101 in 2020, and brings in no novelty to the community. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "FtJVUjKewGi", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary and Contributions\nThe paper presents an empirical study of the open set image recognition (OSR) problem. It considers two classification setups -- single-domain and cross-domain, and, one semantic segmentation setup. In particular, the paper evaluates simple statistical models like the Nearest Class Means (NCMs), KMeans, and, Gaussian Mixture Models (GMMs) built atop OTS deep features against deep SOTA baselines like CLS^(K+1), MSP, OpenMax, C2AE, GDM, etc.  \n\nThe paper demonstrates that simple statistical models outperform SOTA models if deep OTS features pretrained on large datasets are used, normalized and their dimensionality is reduced through spatial pooling and PCA. This also results in really light-weight image recognition pipelines for OSR settings. \n\n## Detailed Review \nThe following is the detailed review of the paper, organized into strengths and weaknesses subsections. \n\n### Strengths \n#### Relevance and Significance \nMost ML models, when deployed in real-world settings, often need to operate in open-set (or, open world) conditions and need to provide robust estimates including an 'out-of-class'/ 'unknown' label when encountering data from unknown classes. Models that have good OSR properties should be of broad interest to the ML community.  \n\nEmpirical results in the paper suggest that simple statistical techniques can be quite effective in performing open-set recognition. The paper also validates the significance of performing feature preprocessing steps towards obtaining performance on par with (or exceeding) that of more complex SOTA methods. \n\n#### Clarity \nThe paper is written well and is easy to understand. \n\n### Weaknesses \n#### Relation to Prior Art \nThe paper does a reasonable job of presenting the prior art, identifying the challenges and need for the presented work. However, it doesn't cite the following relevant works:  \n\n[1] Sun et al, Conditional Gaussian Distribution Learning for Open Set Recognition, CVPR 2020. \n\n[2] Geng et al, Recent Advances in Open Set Recognition: A Survey, IEEE PAMI 2020. \n\n#### Novelty \nThe study involving the exploration of preprocessing steps on a variety of simple statistical models seems novel. However, the survey paper by Geng et al [2] also conducts an empirical study, involving many of the models considered in this paper. The paper needs to reference and compare against [2].  \n\n#### Empirical Evaluation \nThe empirical evaluation is inadequate. Experiment results are shown only for limited closed sets for training. Being an empirical paper it is expected for the paper to have more detailed experiments. In addition, \n\n(1) The paper needs to redesign the study in light of [2]  \n\n(2) It should compare results against more recent SOTA models, for example, [1] given that both uses class-conditional GMMs. \n\n(3) Authors mention in Section 4 that hyperparameters are selected based on a small-scale validation set. However, the size of this validation set is not specified. Details about the validation set are needed to determine performance of the experiments. \n\n(4) In Section 4.1, the C2AE scores in Table 1 seems to be lower than what is reported in their paper, however experimental setup the same: https://arxiv.org/pdf/1904.01198.pdf. Is there any specific reason for this? \n\n(5) Section 4.2: Results on more than one close-world training dataset including MNIST, CIFAR for cross-world testing should be provided as also done by Shafaei et al. \n\n(6) The experimental results need to be explained better. In sections 4.1 and 4.3, GMMs gives good results while on the setting of section 4.2, GMMs don't do well. What is the explanation for this? \n\n## Assessment \nThis paper presents a study showing that simple, lightweight, statistical models can outperform deep SOTA models on the image OSR problem. This should be of interest to the ML community. However, the paper seems agnostic of a couple of recent works which render the study and the empirical evaluation inadequate. In addition, experimental evaluation and analysis needs to be improved as per the observations made above. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The addresses open-set recognition, namely, detecting anomalous samples that belong to classes not observed during training. 
It has been shown that existing methods fail on open-world images. The current paper shows empirically that performance can be greatly improved if based on low-dimensional features. 

Reviewers had grave concerns about the novelty of the approach and the logic behind the workflow. They found merit in the paper but chose to retain their scores after reviewing the rebuttal. As a future recommendation, it would be useful to provide more evidence about what component of the method or workflow are novel and what makes them work well. 
Decision: Reject