OpenAUC: Towards AUC-Oriented Open-Set Recognition
Authors: Zitai Wang
Qianqian Xu
Zhiyong Yang
Yuan He
Xiaochun Cao
Qingming Huang
Venue: NeurIPS 2022 Accept (2022)

Abstract: Traditional machine learning follows a close-set assumption that the training and test set share the same label space. While in many practical scenarios, it is inevitable that some test samples belong to unknown classes (open-set). To fix this issue, Open-Set Recognition (OSR), whose goal is to make correct predictions on both close-set samples and open-set samples, has attracted rising attention. In this direction, the vast majority of literature focuses on the pattern of open-set samples. However, how to evaluate model performance in this challenging task is still unsolved. In this paper, a systematic analysis reveals that most existing metrics are essentially inconsistent with the aforementioned goal of OSR: (1) For metrics extended from close-set classification, such as Open-set F-score, Youden's index, and Normalized Accuracy, a poor open-set prediction can escape from a low performance score with a superior close-set prediction. (2) Novelty detection AUC, which measures the ranking performance between close-set and open-set samples, ignores the close-set performance. To fix these issues, we propose a novel metric named OpenAUC. Compared with existing metrics, OpenAUC enjoys a concise pairwise formulation that evaluates open-set performance and close-set performance in a coupling manner. Further analysis shows that OpenAUC is free from the aforementioned inconsistency properties. Finally, an end-to-end learning method is proposed to minimize the OpenAUC risk, and the experimental results on popular benchmark datasets speak to its effectiveness.

OpenAUC: Towards AUC-Oriented
Open-Set Recognition
Zitai Wang Qianqian Xu Yuan He
Zhiyong Yang" Qingming Huang 4,3,7,8
Xiaochun Cao
SKLOIS, Institute of Information Engineering, CAS School of Cyber Security, University of Chinese Academy of Sciences Key Lab. of Intelligent Information Processing, Institute of Computing Tech., CAS School of Computer Science and Tech., University of Chinese Academy of Sciences Alibaba Group
6 School of Cyber Science and Tech., Shenzhen Campus, Sun Yat-sen University
7 BDKM, University of Chinese Academy of Sciences
Peng Cheng Laboratory
wangzitai@iie ac yangzhiyong21 (ecaiuucaacee.. ac cn caoxiaochun@mail, sysu, edu.
xuqianqianOict. ac. heyuan Jeyuan.hydall1 baba-inc. com
qmhuangducas ac.
Abstract
Traditional machine learning follows close-set assumption that the training and test set share the same label space. While in many practical scenarios, is inevitable that some test samples belong to unknown classes (open-set). To fix this issue, Open-Set Recognition (OSR), whose goal is to make correct predictions on both close-set samples and open-set samples, has attracted rising attention. direction, the vast majority of literature focuses on the pattern of open-set samples. However, how to evaluate model performance in this challenging task still unsolved. In this paper, a systematic analysis reveals that most existing metrics are essentially inconsistent with the aforementioned goal OSR: (1) For metries extended from close-set classification, such as Open-set F-score, Youden's 's index, and Normalized Accuracy, a poor open-set prediction can escape from a low performance score with superior close-set prediction. (2) Novelty detection AUC, which measures the ranking performance between close-set and open-set samples, ignores the close-set performance. To fix these issues, propose novel metric named OpenAUC. Compared with existing metrics, OpenAUC enjoys a concise pairwise formulation that evaluates open-set performance and close-set performance a coupling manner. Further analysis shows that OpenAUC is free from the aforementioned inconsistency properties. Finally, end-to-end learning method is proposed to minimize the OpenAUC risk, and the experimental results popular benchmark datasets speak to its effectiveness.
this
Introduction
Traditional classification algorithms have achieved tremendous success under the close-set set assumption that all the test classes are known during the training period. However, in many practical scenarios, it is inevitable that some test samples belong none of the known classes. In this case, close-set model will classify all the novel samples into known classes, inducing significant performance degeneration. To fix this issue, Open-So Set Recognition (OSR) has attracted rising attention in recent years [1. 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14], where the model is required to not only (I) correctly classify the close-set samples but also (2) discriminate the open-set samples from the close-set ones.
"Corresponding authors.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

In this complicated setting, how to evaluate model performance becomes a challenging problem. Existing work has proposed several metrics, which fall into two categories:
The first direction extends traditional classification metrics to the open-set scenario. To this end, one should first extend the close-set confusion matrix with unknown classes, where threshold decides whether the imput sample belongs to the unknown classes, On top of this, open-set F-score [2, 9, 11, 12, 15, 16] summarizes the True Positive (TP), False Positive (FP), and False Negative (FN) performance of known classes. Youden's index [17] takes the sum of the True Positive Rate (TPR) and True Negative Rate (TNR) performance of known classes as the performance measure. Besides, Normalized Accuracy [15] summarizes the close-set accuracy and the open-set set accuracy via a convex combination. Although is intuitive extend close-set metries, we point out that these metrics are essentially inconsistent with the goal of OSR. Specifically, for open-set F-score and Youden's index, only the FP/EN performances known classes evaluate the open-set performance implicitly. As a result, these metrics will encourage classifying close-set samples into the open-set to decrease the FN of known classes. Moreover, Normalized Accuracy encourages selecting the threshold classifying more open-set samples into known classes. In extreme cases, even close-set model (i.e., all the open-set samples are classified into known classes) can obtain a high performance on these metrics.
The second category regards OSR as novelty detection problem [18, 19] with multiple known classes. Based on such observation, the Area Under ROC Curve (AUC) [20, 21], which measures the ranking performance between known classes and unknown classes, has become a popular metric [3, 4, 5, 6, 8, 10]. Compared with classification-basel metrics, AUC is insensitive to the selection of threshold since summarizes the True Positive Rate (TPR) performance for all possible thresholds. However, the limitation of AUC is also obvious: the close-set performance is ignored. A natural remedy is to adopt the close-set accuracy as a complementary metric [3]. However, what we expect is model that can make correct predictions on close-set and open-set simultaneously. This decoupling strategy will induce a challenging multi-objective optimization problem and also unfavorable to comparing the overall performances of different models. What's more, simply aggregating these two metries will induce another inconsistency property.
In view of this, natural question arises:
Whether there exists a numeric metric that is consistent with the goal of OSR?
To answer this question, we propose novel metric named OpenAUC. Specifically, the proposed metric enjoys a concise pairwise formulation, where each pair consists of a close-set sample and an open-set sample. For each pair, only if the close-set sample has been classified into the correct known class, OpenAUC will check whether the open-set sample is ranked higher than the close-set one. In this sense, OpenAUC evaluates the close-set performance and the open-set performance in a coupling manner, which is consistent with the goal of OSR. What's more, benefiting from the ranking operator, OpenAUC overcomes the sensitivity of the threshold, and further analysis shows that maximizing OpenAUC will guarantee a better open-set performance under a mild assumption on the threshold. Considering these advantages, we further establish an end and-to-end learning method to maximize OpenAUC. Finally, extensive experiments conducted on multiple benchmark datasets validate the proposed metric and learning method. To sum up, the contribution of this paper three-fold:
We make a detailed analysis of existing metrics for OSR. The theoretical results show that existing metrics, including the classification-bseed ones and AUC, are essentially inconsistent with the goal of OSR due to their own limitations.
A novel metric, named OpenAUC, is proposed. Benefiting from its concise formulation, further analysis shows that OpenAUC overcomes the limitations of existing metrics and thus is free from the inconsistency properties.
An end-to-end learning method is proposed to optimize OpenAUC, and the empirical results on multiple benchmark datasets validate its effectiveness.
2 Preliminary
Problem definition. In open-set recognition, the training samples {z, = (æi, vi) are drawn from product space Z₁ X x Vk. where X' is the input space, and V/. {1, ,C} is the

---

Table 1: The consistency analysis of existing metrics for OSR.
Metric
P1 (close)
P2 (open)
P3 (threshold)
P4 (numeric)
Open-set F-score [15]
Youden's 's index [17]
X
Normalized Accuracy [15]
X
AUC [3]
V
V
The OSCR curve [4]
V
V
OpenAUC (Ours)
label space known classes, During the test period, some samples might belong to none of the known classes. For the sake of simplicity, all these samples can be allocated to one super unknown class. In other words, the open-set samples are drawn from product space Z, = x Va. where ) = {C + 1} is the label space of unknown classes. To make predictions, OSR first requires rejector R = 91 o r to judge whether an input sample comes from open-set, where r X' I is the open-set score function, and II- + {0, 1) is the open-set decision function. To be specific, an input æ will classified as an open-set sample if r(æ) is greater than a given threshold E IR. For the samples with a(x)) = 0, classifier / 92 ʃ is further required to make predictions on known classes, where f X + RC the close-set set score function and 92 RC + 3%k is the close-set decision function. In view of this, proper metric for OSR should enjoy the following properties:
(P1) For close-set samples, the metric not only evaluates whether the open-set score function outputs low open-set scores but also requires that the classifier h make correct predictions.
(P2) For open-set samples, the metric should check whether the open-set score function outputs high open-set scores.
(P3) The metric should be insensitive to the threshold t because different ratios of open-set samples will induce different optimal thresholds, but such ratio unavailable during the training period.
(P4) The metric should be single numeric number to favor comparing the overall perfor- mances of different models.
Roadmap. Next, first present detailed analysis of existing metrics in Sec.3. The results show that these metrics are essentially inconsistent with the aforementioned properties, which are summarized in 1. Furthermore, novel metric named OpenAUC and its end-to-end learning method is proposed in Sec.4 to overcome the inconsistency of existing metrics.
3 Existing metrics for Open-set Recognition
Existing metrics for OSR fall into two categories: the classification- ones and the novelty- detection ones. The first category extends existing classification metrics to the open-set scenario, while the second one regards OSR as a generalized novelty detection problem. We will present detailed analysis of these metrics in the rest of this section.
3.1 Open-set F-score and Youden's Index
To extend classification metrics, one should first extend the confusion matrix with the unknown class. Let TP;, TN,, FP,, FN, denote the True Positive (TP), True Negative (TN), False Positive (FP), False Negative (FN) of the class i E V% U Pd. under the given threshold respectively, Note that we omit the classifier and the rejector since there exists no ambiguity.
Open-set F-score [15] is representative classification- based metric for OSR. Compared with its close-set counterpart, this metric evaluates the open-set performance via FP; and FN;, where i 3k- To be specific, this metric summarizes TP;, FP,, FN, of known classes by the harmonic mean of Precision and TPR (i.e., Recall):
Pμ X TPR P TPR
F-score = 2
(1)
3

---

where
P
1
E i=1
TP;
TPR.
1
Σ i=1
TP: TP: + FN,
(2)
C
TP, + FP;
C
if one aggregates model performances in macro manner, and
TP; TPRR
TP:
Pk
(3)
ITP+
(TP; + FN,)
when model performances are summarized in micro manner. Compared with open-set F-score, Youden's index additionally considers TN,, where E 3% [17, 22]:
J TPR; TNR;;
where TNR; denotes the TNR of known classes. However, as illustrated in Prop. these two metrics suffer from an inconsistency property. Please refer Appendix.Bi B.1 for the proof.
Proposition 1 (Inconsistency Property 1). Given a dataset S and metric M that is invariant to TPC+1, FNC+1 and FPC+1, then for any (h, R) such that Σ (P, 2 IPc+1 R), there exists (h, R) such that Mh,, R) = Mh, R) but TPC+1(h, 0
Remark 1. If a metric suffers from the inconsistency property I, then for any (h, R), we can construct (h, Rt) that performs as well as (h, R) on but actually misclassifies all the open-set samples as known classes, which is inconsistent with (P2).
Remark 2. E (h, (n R) 2 TPC-1 (h, R) is a mild condition. To be specific, when TPC-1 is O(C), it only requires that FP; is O(1) for any i E Yg. What's more, even if this condition does not hold, still have TPc+1 (h, It) < TPc+ (h, R) as long Σ FP:(h, R) + 0.
Corollary 1. Open-set F-score and Youden's 's index both suffer from the inconsistency property I.
3.2 Normalized Accuracy
Normalized Accuracy (NAcc) [15] summaries the accuracy performances on close-set and open-set:
NAcc := AnS.KS + (1 Ana) AUS,
where Ama E 1) is the balance constant, and
Di=1
TP, TN:]
TPc+1
AKS
AUS
Li=1
TP: TN, FP; + FN.]
TP + FPC+1
are the Accuracy on Known and Unknown Samples (AKS, AUS), respectively. Since the close-set performance is explicitly involved, NAcc avoids the inconsistency property Ideally, if Ana P Py- C 1], NAce becomes exactly the close-set accuracy. However, it is generally hard to decide the balance constant Ana since we have no idea about the ratio of open-set samples in the test set. Besides, as shown in Prop.2, this metric suffers from another type of inconsistency property. Please refer to Appendix.B. B.2 for the proof.
Proposition 2 (Inconsistency Property II). Given dataset S.for any classifier-rejector pair (h, R) such that Di= FN, FN, (h, R) 2 (Ph+1(h, R) and TPc+1(h, R) > FPC+1(h, R). there exists (h, R) such that [Ac Rt) > NAcc(h, R)) but TPc Rt) -
Remark 3. For any (h, R), we construct (h, It) such that NAcc(h, Ri NAcc(h, R) but actually misclassifies all the open-set samples to known classes. In other words, NAce encourages selecting threshold that classifies more open-set samples to known classes, which inconsistent with (P3).
Remark 4. Similar to the condition in Prop. I, Ee (N, R) 2 Th R) is a mild condition. And TPc+1 R) FPc+1 (h, R) is also mild since is a basic requirement for open-set models.
3.3 The Area Under the ROC Curve (AUC)
When regarding OSR as a novelty detection problem, we no longer need to extend the confusion matrix, and AUC [3] comes into play. Traditional AUC is a popular metric for the imbalanced binary

---

classification problem since it is insensitive to the label distribution [23, 24]. To extend this metric to OSR, one has allocate all the known classes one super known class, which the same as the label space of unknown classes. Let T FPR, denote the TPR and FPR of the super known class, respectively. Then, AUC is defined as the area under the -PR,, curve:
AUC ==
(FPR, (()) dl.
If we assume that there exist ties in r, AUC will enjoy much simpler formulation [23]:
AUC =
E
[1 (r(x.) > r(xx)]]
(8)
where DR, Du are the distribution of known and unknown classes, respectively. In other words, AUC equals the probability that the close-set score function ranks an open-set sample higher than close-set one. Compared with classification- under different thresholds, and the pairwise formulation (8) requires no thresholds. However, its limitation is also obvious: the close-set performance is ignored, which is inconsistent with (P1). To address this issue, one can adopt the close-set accuracy, denoted by Accs, as complementary metric. In other words, this strategy divides OSR into two traditional tasks: multiclass classification and novelty detection. Although intuitive, it also inconsistent with the goal of OSR since the open-set performance and the close-set performance are evaluated in decoupling manner. To be specific, when R(x)= = 1 for a close-set sample x, even if h makes a correct prediction on æ, which will improve the Accı performance, (R, h) will misclassify æ to the unknown class. In this case, Accę is inconsistent with the actual model performance. What's more, simply aggregating the two metries will induce another inconsistency property, whose proof is presented in Appendix.B.3.
metrics, AUC summarizes the model performance
Proposition 3 (Inconsistency Property III). Given a dataset S, for any (h, r) satisfying Acck, AUC + 1, there exists (h, F) that performs worse on the OSR task but sutisfies:
agg( Acc, (h), AUC()) = ogg ogg(AAc)(()) AUC(r)),
where agg :R R
+ R the aggregation function.
Corollary 2. The following metrics from the inconsistency property III: (1) The product of the close-set accuracy and AUC, i.e., Acc; AUC, (2) the summation of the close-set accuracy and AUC, i.e., Acck AUC, and (3) the pointwise summation of the close-set set accuracy and AUC:
Acck ① AUC = E
h((x))))) [r( > r(*x)]]]
4 OpenAUC: novel metric for Open-set Recognition
Considering the limitations of existing metrics, we present a novel metric in this section. To this end, we first review non-numeric metric named the OSCR curve.
4.1 Open Set Classification Rate (OSCR)
The Open Set Classification Rate (OSCR) curve [4] an adaptation of the Detection and Identification Rate (DIR) curve that is commonly used in open-set face recognition [25]. In this curve, f(x), and r(z) are assumed be proportional is defined as the FPR performance on open-set samples, while y-axis is the Correct Classification Rate (CCR) defined on the close-set:
ly = c|ær] and 1/ maxeeyy. f(x) respectively. Then, x-axis
No
1
CCR(() = N E
Di arg max fxx)) 1 [[f()) f]. cEVe
where N₂ is the number of close-set samples, and [A] is the indicator function of an event A. Compared with AUC, the OSCR curve considers the close-set performance via CCR. Meanwhile, as FPR varies from 0 to this curve will present the CCR performance under different thresholds.
Compared with the aforementioned metries, the OSCR curve contains richer information and allows comparing model performances at different operating points. Even though, our goal is to optimize
5

---

the overall performance of the curve. Hence, it is necessary to find numeric metric that aggregates the information of the entire curve, such that the models can be trained by optimizing the loss of the metric. To this end, [13] and [14] estimate the area under the OSCR curve by directly calculating the numeric integral with histograms. However, this number is hard to optimize due to multiple non-differential operators such as ranking and counting. Moreover, the assumptions on (f,r) also limit the application of this metric. For example, considering that Zu = -2 -Z2 n -.. -2-c. [13, 26] propose to model potential unknown space via the prototypes of { {-Zole= Inspired by this, we adopt the following definition, where the aforementioned assumptions on (f.r) do not hold:
Definition 1. Let f(x) denote the probability that the input does not belong to the class c, that is, Ve E fax), x P [u + c/z).. In this case, r(x) is defined as minkey, f(z), since given an open-set sample æ, well-trained model tends output a large (()) for any k E Yn.
4.2 The Definition of OpenAUC
To overcome the limitations of the OSCR curve, we first remove the assumptions on (h, and present generalized formulation of open-set FPR and CCR, denoted by OFPR and Conditional OTPR (COTPR), respectively:
OFPR(t) := E
[ (r(x) S f]], COTPR(() == E z~Dz
[1 S y y h(x)]],
(10)
In other words, OFPR represents the probability that r misclassifies open-set sample known classes, and COTPR equals the probability that (h,r) makes correct prediction on a close-set sample. Then, OpenAUC is defined the area under the OFPR-COTPR curve:
OpenAUC
COTPR(OFPR- (t)) dt.
(11)
However, this integral formulation is still hard to calculate. To fix this issue, we present the following reformulation, whose proof is presented in Appendix.C.I.
Proposition 4. Given (h, r) and sample pair (Zk, zu), 2x E 2k, Zu E Zw, OpenAUC equals the probability that h makes correct prediction on z and ranks zu higher than zy:
OpenAUC- = JE D.
1 1yk = h( 1 [r(x.) ) > r(æw)]
(12)
(b)
Remark First, for close-set samples, term (a) and term (b) require that makes correct predictions and r outputs low open-set scores, which is consistent with (PI). Second, term (b) requires to rank open-set samples higher than the close ones, which is consistent with (P2) and (P3). Last but not least, the product operator guarantees that OpenAUC evaluates the close-set performance and the open-set performance in a coupling manner, which essentially different from the metrics mentioned in Prop.3.
Compared with existing numeric metrics, OpenAUC free from the aforementioned inconsistency properties, and we will make detailed discussion in Sec.4.3. Moreover, Compared with the OSCR curve, OpenAUC enjoys a concise formulation, based on which we can design a differentiable objective function for Empirical Risk Minimization (ERM) in Sec.4.4.
4.3 OpenAUC vs. Existing Metries for OSR
Essentially, inconsistency property is induced by the fact that exchanging the predictions of open-set samples and close-set samples will not lead to larger error on these metrics. However, such an exchange at least breaks a sample pair that has been correctly ranked by r, while the close-set accuracy will not be improved. Thus, we have the following proposition, whose details are presented in Appendix.C.. C.2.
Proposition 5. Given sample pair ((x1, C C 11, (2)), where y2 + C 1, for any (h,r) such that R(x,) 1, R(x2) = 0, h(x;2) + y2, if (h, f) makes the same predictions as (h,r) expect that R(x:) 0, ((xx)) h(x2) and R(x22) 1, we have OpenAUC(h, F) OpenAUC(h,
Remark 6. If we construct (h. F) the way that leads to the inconsistency property I, (h, F) will perform inferior to (h,r) )) OpenAUC, that is. OpenAUC is from the inconsistency property I.

---

Besides, the following proposition reveals that optimizing OpenAUC will guarantee a lower bound of 1I which helps avoid the inconsistency property II. Please refer to Appendix. C.3 for the proof.
Proposition 6. Given a datasel S. for any (J,r) such that OpenAUC k and any threshold LC+1 such that FP = a # O, we have TPR 2 (1 k)/a.
Remark 7. It is clear that TPRc+1 > 0 when a > 1 - k. As we optimize OpenAUC, that is, k - 1, it will become easier to select a threshold tc++1 such that > 1--hh k. In other words, OpenAUC is free from the inconsistency property Il under a mild condition.
Moreover, inconsistency property III is induced by the fact that agg AUC) evaluates the close- set performance and the open-set performance essentially in a decoupling manner. As a comparison, only if (h, r) makes correct predictions on both close-set samples and open-set samples, it will be free from the punishment of OpenAUC. In light of this, we have the following proposition, whose details will be presented in Appendix. C.4.
Proposition 7. Given two close-set samples (æ1,31) and (æ22, 12) and an open-set sample (323, C+1), if (h, F) makes the same predictions as (h, expect that h((1)) h(x,)) V/. h((22)) ((2))) + ya. ((22) > r(æ3) > r(æ11,r(23) = f((xs) and F(z2) r(<1), = r((2)) we have OpenA UCC, () F) < OpenAUC(h,r)) r).
Remark 8. If we construct (h, f) in the way that leads to the inconsistency property III, (h. f) will perform inferior to (h.r) on OpenAUC, that is, OpenAUC is free from the inconsistency property III.
To sum up, OpenAUC is free from the aforementioned inconsistency properties. In view of this, has become an appealing problem to design learning methods that can optimize OpenAUC efficiently.
4.4 Learning Method for OpenAUC
Following the standard machine learning paradigm [27], we should first reformulate the metric optimization problem to risk minimization problem. For traditional metrics such as the close-set accuracy, one can simply replace [A] with 1 [-A]] However, due to the product term, is clear that
R(f. r) = OpenAUc
E
[1m * h(*x)) 1 (r(xx) S r(æx)]]
(13)
To address this issue, we reformulate the product term to a simpler summation term, whose proof is presented in Appendix.C.5.
Proposition 8. Optimizing OpenAUC is equivalent to the following risk minimization problem:
min R(J,r)= f8
E #g
1 [une +
h(ax))
+ 1 [yve h(#x)] 1 [r(w,) S r(#w))
(14)
(a)
(b)
(c)
Remark 9. If we view term (b) as a switch of term (c), then R(frr) essentially is conditional combination of the close-set error and the AUC risk, that is, term (a) and term (c) respectively. In other words, we first minimize the close-set error, and only if h makes correct prediction, we will optimize the open-set AUC risk. This intuition is consistent with an OSR model's decision process described in Sec.2.
On top of this observation, we have the following empirical minimization objective:
1 N2 NR
1 N₂ N Σ> 1 h(x;)) - -((()))))() r(xx)),, (15)
RLe(frr)
Σ L(h(z:).
where L is a common close-set classification loss function such as the cross-entropy loss; é represents continuous surrogate loss for AUC optimization such as the square loss (t) (1 -1-18 t)² whose details can be found in the recent survey [21, 28, 29]; [y* = h(xx)) acts as the switch of l and will not be updated in each epoch. Note that there exist open-set samples in the training set. Inspired by [1i], we adopt manifold mixup [30] to generate open-set samples. Specifically, the convex combinations of samples from different known classes are regarded as open-set samples:
Tu >xfpre(xx)) + (1 Aa) (2x)) V # 4b
(16)
7

---

Table 2: The OpenAUC results on six benchmark datasets for OSR, where C+10, C+50 and CE+
represent CIFAR+10, C5FAR+50 and Cross-Entropy+, respectively. The best and the runner-up method on each dataset are marked with red and blue, respectively.
Method
MNIST
SVHN
CIFARIO
C+10
C+50
TinyImageNet
Softmax
99.2.1.11
92.810.4
83.8=1.5
90.9111.
88.5
60.8
GCPL [31]
99.1=0.2
93.4+0.6
84.3=1.7
91.0=1.77
88.3#1.1
59.315.3
RPL [26]
99.4=0.1
93.610.5
85.2=1.4
91.8#1.2
89.6#0.99
53,2+4.6
ARPL [13]
99.4=0.11
94.0=10.6
86.6=1.4
93.5++0.8
91.61#.4
62.3#3.3
ARPL+CS [13]
99.5=0.1
94.3110.3
87.9=1.5
94.7110.7
92.9
65.9
CE+ [14]
99.1=#022
93.9++0.4
88.1=1.7
93.2++0.6
90.2++04
74.3++399
Acc+AUC
09.
94.0++0.9
87.6=1.9
93.6+1.0
92.0+0.5
74.0++4.0
Ours
99.4=0.1
95.0+10.4
89.2 1.9
95.2+10.7
93.6
75.9
1.00
ooooo
Value
0.75
DC
0000000
Value
C-O O-D-O-
0.75
OOO
0.50
Mi ecare
0.50
Mi-spore
0.50
M-T.score
Incorsstant
NAce
Incorsistntt
NADe
0.25
NAco TPR
0.00
0.00
Threshold
Threshold
Threshold
(a) CIFAR+10
CIFAR+50
TinyImageNet
Figure The inconsistency property of classification-base metries. We can find that all these metrics decrease rapidly as the TPR performance of unknown classes increases.
where æt and æj are samples from different known classes; f is decomposed as fpost (prre (ppre(x)) (æ)):
Ag (0, 1) is sampled from Beta distribution B(a, a.). Then, the score of x. is obtained by
( Compared with other generative models [3, 12, 16], manifold mixup enjoys significant efficiency advantage. While a potential problem is that might locate close to the manifolds of other known classes, i.e., A (ax)) + (1- fpre (x,) (xxe( k i, To this end, we set an extra hyperparameter for the AUC risk, and the final objective becomes:
N₂ L(h(z:),
RE)e((,r)) N
1
Σ
A
[1 h(x:)) -xx))){((((x)) r((x)))] (17)
No
j=1
5 Experiments
In this section, extensive experiments are conducted to answer the following research questions: (Q1) Does the inconsistency property exist in empirical results? (Q2) Does optimize OpenAUC avoid the inconsistency properties and thus help the model outperform the state-off oftte-of-thearn methods? (Q3) How does the proposed method perform under different hyperparameter settings?
5.1 Protocol
Following the protocol in [13] and [14], the experiments are conducted on the following datasets:
(1) MNIST' [32], SVHN2 [33] and CIFAR1O [34], where 4 classes are randomly sampled as the unknown classes; (2) CIFAR+10 and CIFAR+50. where classes are sampled from CIFAR 10 as known classes, and N non-overlapping classes sampled from the CIFAR100 dataset [34] act as the unknown classes; (3) TinyImageNet' [35], where and 180 classes are randomly sampled to evaluate the close-set and open-set performance; (4) Fine-grained datasets such CUB [36].

---

lambda
lambda
lambda
0.4
0.4
0.4
0.5
0.5-
0.5
0.6
0.8
0.8
93.0
92.5
94.0
34.5
96.0
95.5
85.0
87.5
50.0
91
92
53
54
OpenAUC
OpenAUC
OpenAUC
(a) SVHN
(b) CIFARIO
(c) CIFAR+50
Figure 2: The sensitivity analysis of the proposed learning method for OpenAUC.
where known classes are subdivided into three disjoint sets {Easy, Medium, Hard) according to their semantic novelty.
To answer the question (Q1), we record the model perform of Cross-Entroon Cooss-Entropy++ [14] on open-set F-score, Youden' index, Normalized Accuracy, AUC, close-set accuracy and TPR of the open-set class. To answer the question (Q2), we compare the proposed method with the state-of-the-an methods, including Softmax, GCPL [31], RPL. [26]. ARPI. [13], ARPL+CS [13], Cross-Entropy++ [14]. To validate the inconsistency property III, we compare the objective that removes the term 1 1y h((x))]] in (17),,), which is denoted by Acc+AUC, Besides, the implementation details are presented in Appendix.D
5.2 Results and Analysis
Metric comparison (Q1). We present the model performance on classification- based metrics under different thresholds in Fig.1. From these results, we can find that all the classification-bass metries decrease rapidly as the TPR performance of the unknown class increases. Moreover, on CIFAR+10 10 and TinyImageNer the best NAce performance corresponds to a very small TPR performance of the unknown class, which is consistent with our analysis in Prop.2. In other words, all these metrics encourage us selecting a threshold that misclassifies more the open-set samples. All these results show that the inconsistency properties do exist in practical scenarios.
Performance Comparison (Q2). We compare the proposed method with competitors in Tab.2 and have the following observations: (1) Our method outperforms the competitors consistently on all the datasets, which validate the effectiveness the proposed learning method. (2) Acc+AUC performs inferior to the proposed method, which is consistent with the inconsistency property III. Besides, the results in fine-grained datasets are presented Appendix.F.
Sensitivity Analysis (Q3). We present the sensitivity in terms of the hyperparameter À in Fig.2. From the results, we have the following observation: (1) Optimizing the OpenAUC risk consistently helps improve the model performance, which is consistent with our goal to design the OpenAUC objective. (2) When À equals 0.1 0.2, the proposed method achieves the best performance. (3) As increases, there exists a decreasing trend in model performance, which might be induced by the noise in generated samples.
6
Broad Impact
This work provides novel metric named OpenAUC for the OSR task, well as its optimization method. We expect our research could promote the research open-set learning, especially from the perspective of model evaluation. Moreover, no metric is perfect, and, of course, it is no exception
'nttp://yann.lecmm www//xxd/mmist//////coommm/ http:////saann/oooocccc////////a Licensed GPL3.
Licensed GPL3.
ptttpp::/www.cs.torre
http://.ttorrttto.eee//////////a html. Licensed MIT.
http:////sc2t...tanfooee attaaffor..eb/ttttg-iaaaeeeeeee 200 zip. Licensed MIT.
'nttps://ww.viie vision. caltech. wete:/eatttttttsuuaaaaaaaaa 2011101100000010___11111111111 Licensed MIT.

---

for OpenAUC. To be specific, OpenAUC summarizes the COTPR performance under all the OFPR performance. However, some applications, such self-driving, require high recall of open-set. According to Prop.6, only the performance under low OFPR is of interest. In this case, OpenAUC might be biased due to considering irrelevant performances. This might cause potential negative impact concerning safety and security. To fix this issue, optimizing the partial OpenAUC, which summarizes the COTPR performance under some given OFPR performance, might be better choice. Of course, there is no free lunch. Partial OpenAUC will be more difficult to optimize due to the selection operation. Besides, the generalization bound of open-set learning is still an opening problem, and we leave the corresponding analysis of OpenAUC optimization in future work.
7 Conclusion
This paper presents an extensive analysis of existing metrics for OSR. The theoretical results show that existing metrics are essentially inconsistent with the goal of OSR. To fix this issue, novel metric named OpenAUC is proposed. Compared with existing metries, OpenAUC enjoys a concise formulation that evaluates the close-set performance and open-set performance in coupling manner. A series of propositions show that OpenAUC is free from the inconsistency properties of existing metrics. Finally, an end-to-end learning method proposed for OpenAUC, and the experimental results validate the theoretical results and the effectiveness of the proposed method.
Acknowledgments
This work was supported in part by the National Key R&D Program of China under Grant 0018AAA0102000, in part by National Natural Science Foundation of China: U21B2038, 61931008, U2001202, 61733007, 6212200758 and 61976202, in part by the Fundamental Research Funds for the Central Universities, in part by Youth Innovation Promotion Association CAS, in part by the Strategic Priority Research Program of Chinese Academy of Sciences, Grant No. XDB28000000, in part by the China National Postdoctoral Program for Innovative Talents under Grant BX2021298, and in part by China Postdoctoral Science Foundation under Grant 2022M713101.
References
[1] Abhijit Bendale and Terrance E. Boult. Towards open set deep networks. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1563-1572, 2016.
[2] Zongyuan Ge, Sergey Demyanov, and Rahil Garnavi. Generative openmax for multi-class open set classification. In British Machine Vision Conference, 2017.
[3] Lawrence Neal, Matthew L. Olson, Xiaoli Z. Fern, Weng- Wong, and Fuxin Li. Open set learning with counterfactual images. In European Conference on Computer Vision, pages 620-635, 2018.
[4] Akshay Raj Dhamija, Manuel Günther, and Terrance E. Boult. Reducing network agnostophobia. In Advances in Neural Information Processing Systems, pages 9175-9186, 2018
[5] Poojan Oza and Vishal M. Patel. C2AE: class conditioned auto-encoder for open-set recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2307-2316, 2019.
[6] Pramuditha Perera and Vishal M. Patel. Deep transfer learning for multiple class novelty detection. In IEEE Conference on Computer Vision and Pattern Recognition, pages 11544- 11552, 2019.
[7] Ryota Yoshihashi, Wen Shao, Rei Kawakami, Shaodi You, Makoto lida, and Takeshi Nae- mura. Clasifiiattion learning for open-set recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 4016-4025, 2019.
[8] Pramuditha Perera, Vlad I. Morariu, Rajiv Jain, Varun Manjunatha, Curtis Wigington, Vicente Ordonez, and Vishal M. Patel. ceeneratveediscriiinaat feature representations for open- set recognition. JEEE Conference on Computer Vision and Puttern Recognition, pages 11811-11820, 2020.
10

---

[9] Xin Sun, Zhenning Yang, Chi Zhang, Keck Voon Ling, and Guohao Peng. Conditional gaussian distribution learning for open set recognition. In IEEE Conference on Computer Vision and Puttern Recognition, pages 13477-13486, 2020.
[10] Hongjie Zhang. Ang Li, Jie Guo, and Yanwen Guo. Hybrid models for open set recognition. In European Conference on Computer Vision, pages 102-117, 2020.
[11] Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Learning placeholders for open-set recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1401-4410, 2021.
[12] Zhongqi Yue, Tan Wang, Qianru Sun, Xian-Sheng Hua, and Hanwang Zhang. Counterfactual zero-shot and open-set visual recognition. In IEEE Conference Computer Vision and Pattern Recognition, pages 15404,,, 2021.
[13] Guangyao Chen, Peixi Peng, Xiangqian Wang, and Yonghong Tian. Adversarial reciprocal points learning for open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-1, 2021.
[14] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set classifier is all you need. In International Conference on Learning Representations, 2022.
[15] Pedro Ribeiro Mendes Júnior, Roberto Medeiros de Souza, Rafael de Oliveira Werneck, Bernardo V. Stein, Daniel V. Pazinato, Waldir R. de Almeida, Otávio A. B. Penatti, Ricardo da Silva Torres, and Anderson Rocha. Nearest neighbors distance ratio open-set set classifier. Mach. Learn., 106:359-3888 386, 2017.
[16] Shu Kong and Deva Ramanan. Opengan: Open-set recognition via open data generation. In IEEE/CVF International Conference on Computer Vision, pages 793-802, 2021.
[17] Matthew D. Scherreik and Brian D. Rigling. Open set recognition for automatic target classifi- cation with rejection. IEEE Trans. Aerosp. Electron. Syst., 52:632-642, 2016.
[18] Marco A. F. Pimentel, David A. Clifton, Lei A. Clifton, and Lionel Tarassenko. A review of novelty detection. Signal Process., 99:215-249, 2014.
[19] Mohammadreza Salehi, Hossein Mirzaei, Dan Hendrycks, Yixuan Li, Mohammad Hossein Rohban, and Mohammad Sabokrou. A unified survey on anomaly, novelty, open-set, and out-of-distributioo detection: Solutions and future challenges. CoRR, abs/21 10. 14051, 2021.
[20] Tom Fawcett. An introduction to ROC analysis. Puttern Recognit. Lett., 27:861-874, 2006.
[21] Tianbao Yang and Yiming Ying. Auc maximization in the era of big data and ai: survey. ACM Computing Surveys (CSUR), 2022.
[22] W. Youden. Index for rating diagnostic tests. Cancer, 3:32-35, 1950.
[23] Charles X. Ling, Jin Huang, and Harry Zhang. AUC: a statistically consistent and more dis-
criminating measure than accuracy. In International Joint Conference Artificial Intelligence, pages 319-526,70 2003.
[24] David J. Hand. Measuring classifier performance: a coherent alternative to the area under the ROC curve. Mach. Learn., 77:103-123, 2009.
[25] Jonathon Phillips, Patrick Grother, and Ross J. Micheals. Evaluation methods in face recognition. In Handbook of Face Recognition, 2nd Edition, pages 551-574. Springer, 2011.
[26] Guangyao Chen, Limeng Qiao, Yemin Shi. Peixi Peng. Jia Li, Tiejun Huang, Shiliang Pu, and Yonghong Tian. Learning open set network with discriminative reciprocal points. In European Conference on Computer Vision, pages 507-522, 2020.
[27] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar, Foundations of Machine Learning. MIT Press, 2012.
[28] Wei Gao and Zhi-Hua Zhou. On the consistency of AUC pairwise optimization. In International Joint Conference on Artificial Intelligence, pages 939-945, 2015.
[29] Mingrui Liu, Zhuoning Yuan, Yiming Ying, and Tianbao Yang. Stochastic ane maximization with deep neural networks. In International Conference on Learning Representations, pages 1-24, 2019.
11

---

[30] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najañi Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning, pages 6438-64444 2019.
[31] Hong-Ming Yang, Xu- Yao Zhang, Fei Yin, and Cheng-I.in Liu. Robust classification with con- volutional prototype learning. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3474-3482, 2018.
[32] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. IEEE, 86:2278-233 1998.
[33] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In Neural Information Processing Systems Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
[34] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[35] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei- Fei. Imagenet large scale visual recognition challenge. Int. Comput. Vis., 115:21 1-252, 2015.
[36] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-uesd birds-200-2011 dataset. Technical Report T2-201R-20001111 California Institute of Technology, 2011.
[37] Kang Zhou, Shenghua Gao, Jun Cheng, Zaiwang Gu, Huazhu Fu, Zhi Tu, Jianlong Yang, Yitian Zhao, and Jiang Liu. Sparse-gan: Sparsity-constraneed generative adversarial network for anomaly detection in retinal OCT image. In IEEE International Symposium on Biomedical Imaging, pages 1227-1231, 2020.
[38] Alan Herschtal and Bhavani Raskutti. Optimising area under the ROC curve using gradient descent. In International Conference on Machine Learning, 2004.
[39] Toon Calders and Szymon Jaroszewicz. Efficient AUC optimization for classification. In European Conference on Principles and Practice of Knowledge Discovery in Databuses, pages 42-53, 2007.
[40] Xinhua Zhang, Ankan Saha, and S. V.N. Vishwanathan. Smoothing multivariate performance measures. J. Mach. Learn, Res., 13:3623-3684 2012.
[41] Peilin Zhao, Steven C. H. Hoi, Rong Jin, and Tianbao Yang. Online AUC maximization. In International Conference on Machine Learning, pages 233-240. 2011.
[42] Yiming Ying, Longyin Wen, and Siwei Lyu. Stochastic online AUC maximization. In Advances
in Neural Information Processing Systems, pages 451-459, 2016.
[43] Michael Natole, Yiming Ying, and Siwei Lyu. Stochastic proximal algorithms for AUC maximization. International Conference on Machine Learning, pages 3707-3716, 2018.
[44] Michael Natole, Yiming Ying, and Siwei Lyu. Stochastic AUC optimization algorithms with linear convergence. Frontiers Appl. Math. Stat., 5:30, 2019.
[45] Zhenhuan Yang. Wei Shen, Yiming Ying. and Xiaoming Yuan. Stochastic auc optimization with general loss. Communications on Pure & Applied Analysis, 19(8), 2020.
[46] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang. Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, pages 8024-8035, 2019.
[47] Charles R. Harris, K. Jarrod Millman, Stéfan van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus. Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fer- nández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with numpy. Nat., 585:357-362, 2020.
12

---

[48] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vander- Plas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Edouard Duchesnay. Scikit-learn: Machine learning python. J. Mach. Learn. Res., 12:2825-255 2830, 2011.
[49] Sébastien Marcel and Yann Rodriguez. Torchvision the machine-vision package of torch. In Proceedings of the 18th International Conference on Multimedia 2010, Firenze, Italy, October 25-29, 2010, pages 1485-1488, 2010.
[50] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference Computer Vision and Pattern Recognition, pages 770-778, 2016.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you descrihe the limitations of your work? [Yes] See Sec.7.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] There are no potential negative societal impacts associated with our research.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yss]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See the supplementary materials.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] The experiments are conducted on five different splits of each dataset, and we report the standard divation in Tab.2.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "wxegDpgJBbC", "reviewer": "Reviewer_kn16", "summary": "This paper focuses on the evaluation issue for the Open-Set Recognition (OSR) problem. Specifically, the authors point out that existing metrics for OSR are inconsistent with the goal of OSR: some poor open-set predictions can escape from the punishment of classification-based metrics, while novelty detection AUC ignores the close-set performance. In view of this, a novel metric, named OpenAUC, is proposed. Theoretical analysis reveals that OpenAUC overcomes the limitations of the existing metrics. Moreover, an end-to-end learning algorithm is proposed to optimize OpenAUC. Finally, empirical results on six benchmark datasets validate the effectiveness of the proposed method.", "questions": "In Appendix, Figure.3 specifies the indices of classes, that is, “FP1”, “FP2”, “FN1” and so on. How do these classes correspond to the notations in the proof such as $y_1$ and $y_2$? It might be a bit confusing, and more clarification is necessary.", "limitations": "Yes", "rating": 9, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n The authors make a systematic analysis of existing metrics for Open-set Recognition. Concretely, existing metrics are grouped into classification-based ones and novelty-detection-based ones. And their limitations are collectively categorized into three types of inconsistency properties.\n A novel metric named OpenAUC is proposed for OSR. Compared with existing metrics, OpenAUC (1) evaluates the performance on close-set and open-set in a unified manner; (2) aggregates the performance under different thresholds; (3) enjoys a concise formulation and thus is easier to optimize.\n The empirical results are convincing. On one hand, it validates the inconsistency properties of existing metrics. On the other hand, the effectiveness of the proposed learning method is validated.\n\nWeakness:\nIt is recommended that more related work can be provided, such as the details of the competitors and AUC optimization.\nBesides, there exist some minor issues: It is recommended to use “Eq.(1)” instead of “Equation (1)”. Some punctuations are missing such as the full stops of Eq.(9) and Eq.(12).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ew10SzwKNh", "reviewer": "Reviewer_tgk6", "summary": "The paper introduces a new metric called OpenAUC as a summary number to jointly measure the closed-set classification accuracy and open-set detection accuracy. It is a threshold-free metric. The paper compares existing metrics used in the literature of open-set recognition and points out their limitations. Further, the paper develops a loss to train neural networks by directly optimizing the OpenAUC metric. Experiments on standard open-set recognition datasets validate the effectiveness of the loss.\n\n", "questions": "The paper is above average. As for questions, authors are encouraged to address the weaknesses listed above. Answers in the rebuttal can sway the rating.", "limitations": "The paper does not effectively discuss limitations and potential negative societal impacts. The design of the new metric is a summary number that is based on the heuristics (a model should be good at both close-set classification and open-set detection). There are indeed potential limitations and negative impacts. For example, such a summary number makes it non-trivial to select operating points in real-world systems and hide critical failures in the real world. Some applications require high recall of the open-set (e.g., autonomous vehicles) and some other favor high accuracy on the close-set (e.g., image tagging). ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The motivation of the paper is good. It is desired to design a metric for open-set recognition. \n\n- The paper nicely analyzes existing metrics and explain why they fail to jointly measure closed-set and open-set accuracies. \n\n- The developed metric and the derived loss function make sense.\n\n\nWeaknesses:\n\n- One important weakness is that the paper does not justify why OpenAUC is better than OSCR. As pointed out in the paper (Line189), calculating the area under the OSCR curve can be a summary number (as done by [13,14]). \n\n- Following the above, Line188 argues that \"a numeric metric is generally necessary for model comparison.\" This argument conflicts that in [4] -- quote here: \"The application of any algorithm to a real world problem involves the selection of an operating point, with the natural choices on a PR curve being either high precision (low number of false positives) or high recall (high number\nof true positives).\" That said, a curve offers richer information and allows comparing methods at different operating points. The paper should discuss this.\n\n- While the derived loss helps train a model that directly optimizes OpenAUC, it should evaluate the trained model using other metrics as well, including OSCR, F-measure, OSCR, closed-set accuracy, and AUROC. Admittedly, these metrics (except OSCR) have issues, given the standard datasets, using all these metrics give a balanced understanding -- it is important to justify the proposed loss function helps, or doesn't decrease much, the accuracies of closed-set classification and open-set detection. \n\n- It is not clear what the final open-set function r is in experiments.\n\n- Line 90 and 91: Is it a typo \"close-set score function r\"? Isn't r an open-set score function as stated in Line86?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Y3TyYjJ8x5o", "reviewer": "Reviewer_DygR", "summary": "This paper introduced a new evaluation metric for OSR, OpenAUC, which jointly measures the binary open-set performance and multi-class closed-set performance, and also introduced a simple OSR method by minimizing the OpenAUC risk with synthetic open-set samples by mixing up features of closed-set samples. Thorough theoretical analysis is presented and promising performance on public datasets is shown. ", "questions": "Please refer to the weaknesses above. In addition, \n- The OpenAUC requires measuring pairs of closed and open set samples, which is different from existing methods that only measure individual data points.  How the pairs are generated during the evaluation? Are the same pairs used for all methods? The possible pairs will be way more than the individual data points? This will significantly increase the evaluation cost from O(n) to O(n^2), thus reducing efficiency.", "limitations": "Briefly mentioned the future work on generalization bound which is not considered at the moment for OpenAUC. No societal impact is discussed, and I didn't see any major concerns here. ", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The joint evaluation of open-set and closed-set performance is an important problem for the OSR problem.\n+ Thorough theoretical analysis is given for existing methods and the proposed method; the proposed method is simple and effective. \n+ Good results are obtained on public datasets. \n\nWeaknesses:\n- The experimental datasets seem already quite saturated, except TinyImageNet. The more challenging SSB datasets, introduced in [14] with a particular focus on better evaluating OSR, should be evaluated, to strengthen the paper.\n- Only the VGG32 backbone is used. It would be good to see how the proposed method works on other backbones as well, such as ResNet, ViT.\n- Figure 1 is partly unclear to me. What is the difference between m-f-score and M-f-score? Why OSCR curves are not presented? OSCR is the most relevant metric, and the OSCR curves should be presented.\n\nOne suggestion (not weakness): when describing the weaknesses in the main text on other methods, it would be better to add some demonstrative figures to show some examples and how the proposed method can handle them. \n\nOverall, I hold a positive view on the proposed method and think it can be helpful for future OSR research, especially evaluation, but the experiment parts need improvement, especially on the more challenging datasets.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept