Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new approach for generalized open-set recognition (OSR) in classification setting where the idea is to build a model which could simultaneously detect unknown classes and maintain the accuracy of traditional closed-set classification of known classes. The proposed OSR is based on background class regularization (BCR) where data of known unknown classes are utilized in the training phase of the classifiers based on typical deep neural network (DNN) architectures.

Previous works applying BCR have treated known-classes as one group and unknowns as other with SoftMax classification layer, leading to limited OSR performance, especially near the known classes, whereas proposed work regularizes and limits feature space in class-wise manner, making the data of unknown classes to be located far away from the known classes. Distance-based classifier applying linear discriminative analysis is used to model the latent features from DNN and to design novel loss function which can regulate the solution between known classes and unknown background classes in OSR setting, simultaneously keeping the robust performance of closed-set classification.

Proposed approach is experimentally compared to three previous works applying BCR-based OSR with SoftMax classifiers. Proposed method outperforms those three related methods in well-known public benchmarks datasets of image and text classification tasks. Different performance metrics of classification accuracy, area under the receiver operating characteristic curve (AUROC), and the open-set classification rate (OSCR) are utilized in comparison.

To summarize the contribution, paper presents LDA principled distance-based classifier (applied to output of neural network latent feature layer) and new loss function to train OSR-sensitive classifiers, developed and formulated based on the limitations of previous work and experimentally validated against previous BCR-based approaches.
main_review: The main goal of the work is to define a novel open-set classifier on the top of DNN which can improve the detection of unknown classes while keeping the accuracy of original closed-set recognition. This is a well-studied problem and typical applications of image and text classification are considered. A group of different techniques from prior work are combined in a novel way and each part of the model is justified and the whole model is experimentally verified against state-of-the-art.

Strong points
 
The paper is presenting an interesting approach to open-set recognition in the case of background class information (i.e., data of known-unknown classes) is available. Although, proposed method combines several different techniques (i.e., LDA classifier, inclusion probability modelling, and loss function extended with class-inclusion loss), the model choices are clearly and well justified. Furthermore, the limitations, issues, and differences of the most similar previous works are described in detail to support and motivate the proposed methodologies and model. The claims of model properties are experimentally validated against three previous approaches, and it is shown that open-set recognition rate is improved, and the similar or better close-set accuracy is achieved simultaneously. It is also stated that code and experiments will be available to reproduce the results.

Summary of pros
- Clearly written
- Clear and theoretically sound techniques
- Comprehensive experiments on benchmark datasets
- Reproducibility based on detailed equations and code/experiments to be released

Weak points

Although the whole classifier is well-validated, there is room for improvement in studying the behavior of different parts and parameters of the model. Especially, in ablation study section, to get better view how different part of the model and parameters are behaving, it would be useful to give more detailed description and illustrations: 1) When experimenting with different lambda parameter, it would be useful to show graphically how the change of lambda affects the classification accuracy and OSCR in different datasets. 2) When experimenting different class mean vectors initialization, it would be useful to show the detailed results and collect them to a single table, to get better view how robust they are and how significant the differences are.

As claimed, a simple inference process, loss function with fewer data-dependent parameters, and ability to bound open-space risk without additional post-analysis, are proposed. To strengthen the presentation and claims, it would be good to show how computational complexity and efficiency of the proposed classifier in relation to training and inference times compares to previous methods utilizing SoftMax classifiers.

Summary of cons
- Ablation study is somewhat vague; more detailed and illustrative experiments would strengthen the analysis of model parameters and choices (e.g., selecting lambda, detailed performance of different initialization of class mean vectors)
- Missing computation complexity and efficiency analysis of proposed method in relation to training and inference times (if there are any significant differences compared to previous methods?)

Other (minor) comments and questions
- Eq. 9: It would be useful to write out the final loss equation with classification and class-inclusion parts included. Now they are given in the middle of the text only
- In 4.1: use similar term S1/S2 or setting 1 / setting 2 in each following subsection when referring to different experiments
- Background dataset (ImageNet) could be introduced already in 4.1 (because it is used in both S1 and S2)
- Result presentation (Table 1 and Table 2): Equally good results of previous methods should be also bolded (Table 1: SVHN accuracy of Energy model, Table 2: CIFAR100 average AUROC of Energy model), or are these rounded to third decimal and class-inclusion model is marginally better?
- In ablation studies related to classification accuracy (with conventional loss function), it would be useful to collect these results to a new table for comparison
- How is the best/optimal lambda selected in experiments? How much does it affect the classification accuracy and OSCR numerically?
- How much does different class mean vector initialization strategies (empirical mean, feature mean, random sampling, re-computed empirical mean) affect the model performance? and are there any explanations of why they behave/perform differently?
- Are there some good practices of how “known unknown classes” (KUC) data should be chosen or generated? Could this be automated? How robust will the proposed model be given the different choices of KUC data?
summary_of_the_review: The paper presents an interesting and novel approach combining several techniques to improve DNN-based open-set recognition with distance-based classifier. Paper is well-written and the methodological choices are properly justified. More detailed ablation study and the analysis of the model efficiency would still improve the presentation and verify the claims better. This is an important and practical problem and can bring new insight for the field. I recommend the paper to be accepted.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: For open set recognition (OSR), the authors propose using background data to represent instances from unknown classes.  The background data are from data outside the known classes.  Based on LDA, they proposed using the distance from an instance to the class mean to estimate the posterior class probability via softmax.  The class means are randomly chosen beforehand.  Their loss functions includes cross entropy, L_cf, and background class regularization (BCR), which is denoted as L_bg.   For L_bg, they introduce probability of inclusion, P_I, which is estimated by the CDF for Weibull distribution.  With the background data, they calculate L_bg,u  based on 1-P_I.  With the known data, they calculate L_bg,k based on P_I.

Empirical evaluations have two settings.  The unknown classes from the first setting are from the same dataset as the known classes.  In the second setting the unknown classes are from another dataset.  Background data are from another dataset.  In the both settings, the proposed method compares favorably against a few existing algorithms on a few datasets.


main_review: Strengths:

The main contribution is a loss function that includes background class regularization (BCR) based on probability of inclusion estimated by the CDF of Weibull distribution, though BCR and CDF of Weibull distribution are not new. Empirical results indicate that the proposed method compares favorably against a few existing algorithms on a few datasets.  The paper is generally well written.

Weakness:

Since both L_cf and L_bg,k try to get known instances close to their class means,  an ablation study on excluding either one of them would be interesting.  Also, a more detailed analysis and discussion of why L_bg,k is needed would be helpful. 

Minor:

Paragraph next to Figure 2: Eq 7 -> Eq 8 for P_I
summary_of_the_review: While the empirical results are favorable, justification and analysis of including L_bg,k could be more detailed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a distance-based background class regularization (BCR) method for open-set recognition (OSR), in which the distance-based classifier that uses the principle of linear discriminant analysis is utilized to limit the feature space of known-class data in a class-wise manner and make background-class samples far away from the limited feature space. Experiments show the robust OSR performance of the proposed method.
main_review: The strengths of the paper:
+ The paper is well-written and easy to read.
+ A distance-based background class regularization (BCR) method using the principle of linear discriminant analysis is proposed to address the problem of open-set recognition.
+ Ablation study is well designed to show the performance of the proposed distance-based method.

The weaknesses of the paper:
- The novelty of this work is limited. The distance-based classifier that uses the principle of linear discriminant analysis has been studied in previous method, e.g., [1] Thomas Mensink, Jakob Verbeek, Florent Perronnin, Gabriela Csurka, Distance-Based Image Classification: Generalizing to New Classes at Near-Zero Cost, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013.
- The experimental evaluation of this paper is not sufficient. The current version lacks comparisons with existing Softmax based methods, e.g.,  [2] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song, Sphereface: Deep hypersphere embedding for face recognition, IEEE Conference on Computer Vision and Pattern Recognition, 2017; [3] Feng Wang, Jian Cheng, Weiyang Liu, and Haijun Liu, Additive margin softmax for face verification, IEEE Signal Processing Letters, 2018.
summary_of_the_review: The paper is well-written and easy to read. However, the novelty is limited and the experiments are not sufficient.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose a new method for open set recognition. To this end, the authors use distance based classifiers and they minimize the distance between the class samples and their corresponding means for the labeled data samples and try to enforce the background data samples to be outside the class acceptance regions. Then the samples coming from the unknown classes can be rejected based on the distances to the centers of the known classes. The authors use ImageNet data samples are used as background class samples. The proposed method is compared to related open set recognition methods and better accuracies are reported.
main_review: In this paper, the authors propose a new method for open set recognition. To this end, the authors use distance based classifiers and they minimize the distance between the class samples and their corresponding means for the labeled data samples and try to enforce the background data samples to be outside the class acceptance regions. 
The strengths of the paper can be summarized as follows:
1) The paper is written well and the motivation and the proposed methodology are described well.
2) The authors aim to propose compact acceptance regions for the known classes and they enforce the background samples to lie outside of these regions. The background samples help to return more compact and correct class regions. This idea makes sense and there are recent studies applying the same idea.
3) Experimental results are good in the sense that the proposed method outperforms the competing methods.
The weaknesses of the paper can be summarized as follows:
1) The idea of using distance-based classifiers for returning compact class regions where the known class samples lie inside these regions and negative (background) samples lie far from these regions have been used before. The authors cited some of these studies, but some important references are missing. Among these, the method using center loss function [R1], polyhedral conic classifiers [R2] and Convolutional prototype networks [R3] should be mentioned. Especially, the deep polyhedral conic classifier [R2] implements exactly the same idea. It returns bounded convex acceptance regions for the class samples and enforces the negative samples to lie outside of these regions. 
The authors use a similar idea, and they enforce to minimize the distances between the known class samples and their corresponding class centers. They also use background class samples to lie outside the class acceptance regions. To this end, they adopt the idea from Explainable Deep One-Class classification method and they introduce a slightly different loss term that utilized background samples. Therefore, the overall novelty is limited in my opinion.
2) I strongly believe that setting class specific means in the proposed methodology is problematic. The authors randomly select the class means from the standard Gaussian distribution and fix them. This is unacceptable. It should be noted that the features change in each epoch in deep neural network classifiers, and the class means also change in the process. Therefore, the class means must be also updated based on the changed features. This way, local similarities between the semantic classes can be preserved (For example, in the feature space we expect that the class samples belonging to the cat and dog classes lie in close regions and they are probably far from other classes such as aeroplanes, cars, etc.). But when the class samples are chosen randomly and the class samples are enforced to lie in the vicinity of these randomly chosen centers the semantic similarities are completely destroyed. Note that Izmailiov et al. use this setting in the generative models and this is not appropriate for the classification setting. Furthermore, they also recommend the updating class centers. 
The authors report that updating class mean vectors yield worse results than fixed class mean vectors. In contrast, both [R1] and [R2] also update the class mean vectors and report good accuracies. The authors must look into this issue in more details (change parameter settings or update centers using the formulations given in [R1] with appropriate weights).
3) The authors use ImageNet dataset for background classes. I was wondering if the overlapping classes between tested datasets and this one are removed. Also, the authors use a completely different setting (using a background class) in their experiments and therefore it is hard to compare the results to the some published related methods. Of course, it is hard to implement every method, but the authors must definitely give results for a distance metric learning method using triplet loss function. Anchors can be set class means and constraints can be easily formed by using known class labels and background class labels. Lastly, an ablation study showing the importance of the utilized background class is necessary in my opinion to judge the effect of background class samples for returning more compact class acceptance regions.
References:
[R1] Wen et al., A comprehensive study on center loss for deep face recognition, Int. Journal of Computer Vision, 127, 2019.
[R2] Cevikalp et al., Deep compact polyhedral conic classifier for open and closed set recognition, Pattern Recognition, 2021.
[R3] Yang et al., Convolutional prototype network for open set recognition IEEE Trans. on Pattern Analyis Machine Intelligence, 2020.

summary_of_the_review: In general, the novelty is kind of limited but the experimental results are quite good. There are some issues that must be addressed regarding the setting class centers. Also, more comparison to the related methods and some ablations studies are needed in experiments. A borderline paper closer to the acceptance region in my opinion.
review_title: null

Meta-review: This paper tackles the open-set recognition problem, specifically the subset that looks at rejecting test data that with unknown classes that are related to the training data. The proposed approach uses an existing distance-based classifier (based on LDA) combined with a new background class regularizer. Results, comparing to a few prior OSR methods, are shown across image/text datasets. 

  The reviewers gave a mixed set of scores, with concerns about visualization/ablation studies and the lambda parameter with affect on classification accuracy (1wRX, ujMG, rop6), computation complexity and efficiency (1wRX), limited novelty and discussion of relationship to prior works (Mkdh, rop6), and limited comparison to state of art as only a few algorithms are compared to the proposed approach (Mkdh), and initialization method. Notably, the authors make a strong claim for the latter point that the method should only be compared to previous BCR methods (as opposed to softmax-based classifiers, for example); this seems to ignore whole classes of different methods that can approach the OSR problem. While it is true that comparing to previous BCR methods can directly show your approach is superior to them under similar class of algorithms (thereby showing that it is an improvement), putting the method within the context of the entire literature is absolutely necessary to discuss relative impact to the field. For example, the improvements in AUROC are not that great (and in some cases worse) than even the methods you compare to, while OSCR is improved significantly, so it is not clear how it stacks up with respect to the current state of art. Even if it doesn't beat it, you could argue your contribution, but not presenting it all prevents a holistic perspective that is necessary. 

  The authors provided thorough rebuttals, including additional ablations and experiments. However, after the review period the scores remain mixed (5,5,6,8) and the reviewers expressed remaining concerns about novelty and comparison to the current state of art (not just BCR-based methods). As a result of these remaining concerns, I recommend rejection at this time.
Decision: Reject