Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: ## Summary and Contributions
The paper presents an empirical study of the open set image recognition (OSR) problem. It considers two classification setups -- single-domain and cross-domain, and, one semantic segmentation setup. In particular, the paper evaluates simple statistical models like the Nearest Class Means (NCMs), KMeans, and, Gaussian Mixture Models (GMMs) built atop OTS deep features against deep SOTA baselines like CLS^(K+1), MSP, OpenMax, C2AE, GDM, etc.  

The paper demonstrates that simple statistical models outperform SOTA models if deep OTS features pretrained on large datasets are used, normalized and their dimensionality is reduced through spatial pooling and PCA. This also results in really light-weight image recognition pipelines for OSR settings. 

## Detailed Review 
The following is the detailed review of the paper, organized into strengths and weaknesses subsections. 

### Strengths 
#### Relevance and Significance 
Most ML models, when deployed in real-world settings, often need to operate in open-set (or, open world) conditions and need to provide robust estimates including an 'out-of-class'/ 'unknown' label when encountering data from unknown classes. Models that have good OSR properties should be of broad interest to the ML community.  

Empirical results in the paper suggest that simple statistical techniques can be quite effective in performing open-set recognition. The paper also validates the significance of performing feature preprocessing steps towards obtaining performance on par with (or exceeding) that of more complex SOTA methods. 

#### Clarity 
The paper is written well and is easy to understand. 

### Weaknesses 
#### Relation to Prior Art 
The paper does a reasonable job of presenting the prior art, identifying the challenges and need for the presented work. However, it doesn't cite the following relevant works:  

[1] Sun et al, Conditional Gaussian Distribution Learning for Open Set Recognition, CVPR 2020. 

[2] Geng et al, Recent Advances in Open Set Recognition: A Survey, IEEE PAMI 2020. 

#### Novelty 
The study involving the exploration of preprocessing steps on a variety of simple statistical models seems novel. However, the survey paper by Geng et al [2] also conducts an empirical study, involving many of the models considered in this paper. The paper needs to reference and compare against [2].  

#### Empirical Evaluation 
The empirical evaluation is inadequate. Experiment results are shown only for limited closed sets for training. Being an empirical paper it is expected for the paper to have more detailed experiments. In addition, 

(1) The paper needs to redesign the study in light of [2]  

(2) It should compare results against more recent SOTA models, for example, [1] given that both uses class-conditional GMMs. 

(3) Authors mention in Section 4 that hyperparameters are selected based on a small-scale validation set. However, the size of this validation set is not specified. Details about the validation set are needed to determine performance of the experiments. 

(4) In Section 4.1, the C2AE scores in Table 1 seems to be lower than what is reported in their paper, however experimental setup the same: https://arxiv.org/pdf/1904.01198.pdf. Is there any specific reason for this? 

(5) Section 4.2: Results on more than one close-world training dataset including MNIST, CIFAR for cross-world testing should be provided as also done by Shafaei et al. 

(6) The experimental results need to be explained better. In sections 4.1 and 4.3, GMMs gives good results while on the setting of section 4.2, GMMs don't do well. What is the explanation for this? 

## Assessment 
This paper presents a study showing that simple, lightweight, statistical models can outperform deep SOTA models on the image OSR problem. This should be of interest to the ML community. However, the paper seems agnostic of a couple of recent works which render the study and the empirical evaluation inadequate. In addition, experimental evaluation and analysis needs to be improved as per the observations made above. 

review_title: Good Empirical Study Falls Short 

final_score: rating: 3: Clear rejection
review_content: Summary of Paper: The main claim of the paper is that out of distribution (OOD) detection can be done by use of pre-training and appropriately deriving a feature space from SOTA activations  via pooling, PCA based dimensionality reduction, L2 normalization.  Classical methods such as GMMs, k-means etc. can then be used to estimate the probability density function of features for use in OOD detection.  Several alternative schemes are compared against many OOD detection schemes.  

Key contributions claimed are that pre-trained nets have information about open-world statistics and off-the-shelf net features along with appropriate choice of a low-dimensional representation helps in outperforming conventional OOD schemes.
The paper builds upon recent work on OOD method benchmarking method by Shafaei et al (2019) that argues that most OOD schemes are not able to pass a less unbiased test designed ( wherein a Source data set is used for training using standard methodology, Validation data set for estimating a decision function between source and validation,  and finally the probability of outlier detection on other datasets and their variability is estimated to get robust view of OOD.).   

1. The general approach is, as I think, quite misguided: the idea is to use pretrained datasets, extract features from it and apply a Gaussian mixture model. This means though, that such an approach will be able to recognize only classes for which features were made available (hence, the closed-world problem is just extended by additional knowledge about more classes from selected datasets, no open-world problem is solved).

2. The fact that pretraining helps, is a generic statement which depends on the statistics of the dataset used for pretraining and then the dataset on which it is tested.

3. the setup in 4.1 (split a single dataset into open and closed sets w.r.t class label) is questionable - as the image statistics in the partition are same.  

4. in 4.2 the authors use an 'open' dataset for validation/tuning - this makes this dataset not open per definition. It may be true that it helps in the model generalizing better, but the terminology is still misguided.  In my view, the paper applies the testing methodology of Shafaei et al (2019) incorrectly to design an OOD algorithm and claim its superiority.   

None of the tables in the paper provide error bars.  In order to convince that the insights are correct I would expect that the experiments need to be run with a larger sampling of outlier datasets (as done in Shafaei et al).   


review_title: Paper empirically explores open set classification. It claims that leveraging pre-training information and classical learning schemes with appropriately chosen feature perform better than OOD schemes.  The experimental approach and results are not convincing.

final_score: rating: 3: Clear rejection
review_content: Methodology:
The paper tackles the so-called open-set classification where query examples outside any of the classes in the training set should be detected at inference. By combining the feature extractor based on fashionable deep models and the classical clustering methods (k-means, GMM, etc), the paper empirically shows that this pipeline can address many open-set problems in realistic scenarios.          
            
Pros:
Unfortunately, none.

Cons:
The paper basically reinvents the wheel. The so-called open-set classification problem is precisely the anomaly detection problem that has been studied for decades (see V. Chandola, A. Banerjee, and V. Kumar, “Anomaly detection: A survey,” ACM Computing Surveys, vol. 41, no. 3, p. 15, 2009), and the use of classical clustering methods (k-means, GMM, etc) was among one of the first efforts for anomaly detection. Applying the identical strategy on top of the fashionable deep features is the routine treatment for anomaly detection 101 in 2020, and brings in no novelty to the community. 

review_title: A reinventing-the-wheel paper 

final_score: rating: 7: Good paper, accept
review_content: This paper proposes an open-set recognition approach that uses simple statistical measures (such as GMM’s and KMeans) on top of post-processed intermediate features extracted from closed-set deep models. It finds that i) this “lightweight” pipeline outperforms prior methods on open-set image recognition across multiple evaluation protocols at much lesser memory and compute cost, and ii) Open-world recognition generally benefits from using models pre-trained on large datasets such as ImageNet rather than training from scratch, iii) the technique also generalizes to open-world semantic segmentation.

Strengths

– The paper studies an important problem, is motivated clearly, and is well-written

– The proposed approach is intuitive, memory-efficient, and appears to clearly and consistently perform prior work

– The modeling choices are motivated and analyzed well, for eg. Fig 3 clearly illustrates why l2-normalized and pooled intermediate features more clearly capture in-domain-ness than logits used predominantly in prior work

– The set of experiments and baseline comparisons appears comprehensive

– The performance v/s memory tradeoff of the current method vs prior work is analyzed well

– The limitations of the proposed method are also explored, for eg. Fig 5 that shows how for open set semantic segmentation, the proposed method is a better alternative than training a binary classifier only when very little out-of-domain data is available to train on

Weaknesses

– Studying how/whether the choice of layer from which features are extracted affects performance would have been an interesting addition

Additional comments / suggestions

– Fig 6 is difficult to read since some of the lines overlap, varying opacity might help with readability

Overall comments

This is an interesting and well-written paper that proposes simple and memory-efficient alternatives for open-world recognition that consistently outperform more complex methods from prior work.

-------- post-rebuttal comments --------

I have read the concerns raised by other reviewers as well as the author response. I still feel that this is an interesting work and its findings are of potential value to the community: There has been a considerable amount of recent work in open set recognition for deep models, and this paper calls into question the need for sophisticated techniques by showing (fairly rigorously, in my opinion) that simple strategies and the right choice of feature engineering works better. I agree with the authors that not using ImageNet pretraining is an unrealistic and unnecessary constraint – moreover, the method generalizes even when evaluated on datasets such as MNIST and SVHN, which are distributionally very different from ImageNet. Further, I think the paper acknowledges prior work appropriately and did not find any of the claims made to be unreasonable. I agree with R1's concerns about overlap with two recent papers, but found the author response to be satisfactory. Overall, I will retain my accept rating.

review_title: Interesting and intuitive approach for open-set recognition

Meta-review: The addresses open-set recognition, namely, detecting anomalous samples that belong to classes not observed during training. 
It has been shown that existing methods fail on open-world images. The current paper shows empirically that performance can be greatly improved if based on low-dimensional features. 

Reviewers had grave concerns about the novelty of the approach and the logic behind the workflow. They found merit in the paper but chose to retain their scores after reviewing the rebuttal. As a future recommendation, it would be useful to provide more evidence about what component of the method or workflow are novel and what makes them work well. 
Decision: Reject