Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduced a new evaluation metric for OSR, OpenAUC, which jointly measures the binary open-set performance and multi-class closed-set performance, and also introduced a simple OSR method by minimizing the OpenAUC risk with synthetic open-set samples by mixing up features of closed-set samples. Thorough theoretical analysis is presented and promising performance on public datasets is shown. 
strengths_and_weaknesses: Strengths:
+ The joint evaluation of open-set and closed-set performance is an important problem for the OSR problem.
+ Thorough theoretical analysis is given for existing methods and the proposed method; the proposed method is simple and effective. 
+ Good results are obtained on public datasets. 

Weaknesses:
- The experimental datasets seem already quite saturated, except TinyImageNet. The more challenging SSB datasets, introduced in [14] with a particular focus on better evaluating OSR, should be evaluated, to strengthen the paper.
- Only the VGG32 backbone is used. It would be good to see how the proposed method works on other backbones as well, such as ResNet, ViT.
- Figure 1 is partly unclear to me. What is the difference between m-f-score and M-f-score? Why OSCR curves are not presented? OSCR is the most relevant metric, and the OSCR curves should be presented.

One suggestion (not weakness): when describing the weaknesses in the main text on other methods, it would be better to add some demonstrative figures to show some examples and how the proposed method can handle them. 

Overall, I hold a positive view on the proposed method and think it can be helpful for future OSR research, especially evaluation, but the experiment parts need improvement, especially on the more challenging datasets.
questions: Please refer to the weaknesses above. In addition, 
- The OpenAUC requires measuring pairs of closed and open set samples, which is different from existing methods that only measure individual data points.  How the pairs are generated during the evaluation? Are the same pairs used for all methods? The possible pairs will be way more than the individual data points? This will significantly increase the evaluation cost from O(n) to O(n^2), thus reducing efficiency.
limitations: Briefly mentioned the future work on generalization bound which is not considered at the moment for OpenAUC. No societal impact is discussed, and I didn't see any major concerns here.
review_title: null

final_score: rating: 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper focuses on the evaluation issue for the Open-Set Recognition (OSR) problem. Specifically, the authors point out that existing metrics for OSR are inconsistent with the goal of OSR: some poor open-set predictions can escape from the punishment of classification-based metrics, while novelty detection AUC ignores the close-set performance. In view of this, a novel metric, named OpenAUC, is proposed. Theoretical analysis reveals that OpenAUC overcomes the limitations of the existing metrics. Moreover, an end-to-end learning algorithm is proposed to optimize OpenAUC. Finally, empirical results on six benchmark datasets validate the effectiveness of the proposed method.
strengths_and_weaknesses: Strengths:
 The authors make a systematic analysis of existing metrics for Open-set Recognition. Concretely, existing metrics are grouped into classification-based ones and novelty-detection-based ones. And their limitations are collectively categorized into three types of inconsistency properties.
 A novel metric named OpenAUC is proposed for OSR. Compared with existing metrics, OpenAUC (1) evaluates the performance on close-set and open-set in a unified manner; (2) aggregates the performance under different thresholds; (3) enjoys a concise formulation and thus is easier to optimize.
 The empirical results are convincing. On one hand, it validates the inconsistency properties of existing metrics. On the other hand, the effectiveness of the proposed learning method is validated.

Weakness:
It is recommended that more related work can be provided, such as the details of the competitors and AUC optimization.
Besides, there exist some minor issues: It is recommended to use “Eq.(1)” instead of “Equation (1)”. Some punctuations are missing such as the full stops of Eq.(9) and Eq.(12).

questions: In Appendix, Figure.3 specifies the indices of classes, that is, “FP1”, “FP2”, “FN1” and so on. How do these classes correspond to the notations in the proof such as $y_1$ and $y_2$? It might be a bit confusing, and more clarification is necessary.
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper introduces a new metric called OpenAUC as a summary number to jointly measure the closed-set classification accuracy and open-set detection accuracy. It is a threshold-free metric. The paper compares existing metrics used in the literature of open-set recognition and points out their limitations. Further, the paper develops a loss to train neural networks by directly optimizing the OpenAUC metric. Experiments on standard open-set recognition datasets validate the effectiveness of the loss.


strengths_and_weaknesses: Strengths:

- The motivation of the paper is good. It is desired to design a metric for open-set recognition. 

- The paper nicely analyzes existing metrics and explain why they fail to jointly measure closed-set and open-set accuracies. 

- The developed metric and the derived loss function make sense.


Weaknesses:

- One important weakness is that the paper does not justify why OpenAUC is better than OSCR. As pointed out in the paper (Line189), calculating the area under the OSCR curve can be a summary number (as done by [13,14]). 

- Following the above, Line188 argues that "a numeric metric is generally necessary for model comparison." This argument conflicts that in [4] -- quote here: "The application of any algorithm to a real world problem involves the selection of an operating point, with the natural choices on a PR curve being either high precision (low number of false positives) or high recall (high number
of true positives)." That said, a curve offers richer information and allows comparing methods at different operating points. The paper should discuss this.

- While the derived loss helps train a model that directly optimizes OpenAUC, it should evaluate the trained model using other metrics as well, including OSCR, F-measure, OSCR, closed-set accuracy, and AUROC. Admittedly, these metrics (except OSCR) have issues, given the standard datasets, using all these metrics give a balanced understanding -- it is important to justify the proposed loss function helps, or doesn't decrease much, the accuracies of closed-set classification and open-set detection. 

- It is not clear what the final open-set function r is in experiments.

- Line 90 and 91: Is it a typo "close-set score function r"? Isn't r an open-set score function as stated in Line86?
questions: The paper is above average. As for questions, authors are encouraged to address the weaknesses listed above. Answers in the rebuttal can sway the rating.
limitations: The paper does not effectively discuss limitations and potential negative societal impacts. The design of the new metric is a summary number that is based on the heuristics (a model should be good at both close-set classification and open-set detection). There are indeed potential limitations and negative impacts. For example, such a summary number makes it non-trivial to select operating points in real-world systems and hide critical failures in the real world. Some applications require high recall of the open-set (e.g., autonomous vehicles) and some other favor high accuracy on the close-set (e.g., image tagging).
review_title: null

Meta-review: metareview: The paper proposes OpenAUC, which is a novel metric designed specifically for evaluating Open-Set Recognition (OSR) performance. OpenAUC is motivated by a formal analysis on existing OSR evaluation metrics, which suffer from three types of inconsistency properties. Theoretical results show that OpenAUC is consistent with the goal of OSR while free of all identified inconsistency properties. An empirical loss function is developed accordingly that enables model training to optimize the proposed OpenAUC.  

Overall, the paper is well-written. The proposed OpenAUC metric can potentially benefit future research in OSR as recognized by the reviewers. Authors and reviewers engaged in a productive discussion, which helped to further improve the quality of the paper. The authors are encouraged to address some remaining suggestions from the reviewers, including adding results on other backbones in the experiments and extending the related work by discussing more recent literature in OSR and AUC optimization.

confidence: Certain
Decision: Accept