Adaptive Robust Evidential Optimization For Open Set Detection from Imbalanced Data
Authors: Hitesh Sapkota
Qi Yu
Venue: ICLR 2023 poster (2023)
Keywords: Open Set Detection
Imbalanced Data

Abstract: Open set detection (OSD) aims at identifying data samples of an unknown class ($i.e.$, open set) from those of known classes ($i.e.$, closed set) based on a model trained from closed set samples. However, a closed set may involve a highly imbalanced class distribution. Accurately differentiating open set samples and those from a minority class in the closed set poses a fundamental challenge as the model may be equally uncertain when recognizing samples from the minority class. In this paper, we propose Adaptive Robust Evidential Optimization (AREO) that offers a principled way to quantify sample uncertainty through evidential learning while optimally balancing the model training over all classes in the closed set through adaptive distributively robust optimization (DRO). To avoid the model to primarily focus on the most difficult samples by following the standard DRO, adaptive DRO training is performed, which is governed by a novel multi-scheduler learning mechanism to ensure an optimal model training behavior that gives sufficient attention to the difficult samples and the minority class while capable of learning common patterns from the majority classes. Our experimental results on multiple real-world datasets demonstrate that the proposed model outputs uncertainty scores that can clearly separate samples from closed and open sets, respectively, and the detection results outperform the competitive baselines. 

Published as a conference paper at ICLR 2023
ADAPTIVE ROBUST EVIDENTIAL OPTIMIZATION FOR OPEN SET DETECTION FROM IMBALANCED DATA
Hitesh Sapkota & Qi Yu
Rochester Institute of Technology (hxs1 1943, q1 yu) @rit.edu
ABSTRACT
Open set detection (OSD) aims at identifying data samples of an unknown class (i.e., open set) from those of known classes (i.e., closed set) based on a model trained from closed set samples. However, a closed set may involve highly imbalanced class distribution. Accurately differentiating open set samples and those from a minority class in the closed set poses a fundamental challenge the model may be equally uncertain when recognizing samples from the minority class. In this paper, propose Adaptive Robust Evidential Optimization (AREO) that offers principled way to quantify sample uncertainty through evidential learning while optimally balancing the model training over all classes in the closed set through adaptive distributively robust optimization (DRO). To avoid the model to primarily focus on the most difficult samples by following the standard DRO, adaptive DRO training is performed, which is governed by a novel multi-scheduler learning mechanism to ensure an optimal model training behavior that gives sufficient attention to the difficult samples and the minority class while capable of learning common patterns from the majority classes. Our experimental results on multiple real-world datasets demonstrate that the proposed model outputs uncertainty scores that can clearly separate samples from closed and open sets, respectively, and the detection results outperform the competitive baselines.
INTRODUCTION
In many practical scenarios (e.g., drug discovery, anomaly detection efc.), it is likely to encounter unknown samples and it is desirable that the model can properly detect these samples as unknown. Various approaches have been proposed to tackle the unknown sample detection problem (Bendale & Boult, 2016; Sun et al., 2020), using techniques such as Weibull-Calbbration SVM (W-SVM) (Scheirer et al., 2013), reconstruction error (Zhang & Patel, 2017), nearest neighbor (Júnior et al., 2016), and quasi-linear function (Cevikalp & Yavuz, 2017). As a representative example, the Openmax framework removes softmax from the last layer of neural network and includes an additional layer to produce the probability of a sample being unknown. This essentially redistributes the probability mass to (K + 1) classes (with unknown being a new class). Multiple efforts follow this direction (Sun et al., 2020; Neal et al., 2018). While this technique is viable to detect open-set samples, the additional layer is included during the testing phase. As result, the training still follows the closed set assumption.
Recent advances in uncertainty quantification provide more systematic way to break the closed set limitation by explicitly modeling the uncertainty mass that corresponds to the unknown class. One representative work is the evidential deep learning (EDL) model (Sensoy et al., 2018), which treats the predicted multi-class probability as a multinomial opinion according to subjective logic (Jøsang, 2016). Similar to EDI., Prior Networks (PNs) (Malinin & Gales, 2018) explicitly considers the distributional uncertainty that quantifies the distributional mismatch (Malinin & Gales, 2018). The Posterior Networks further improves PNs by leveraging normalizing flows for density estimation in the latent space predict posterior distribution, which can be used identify -istr--disttibution (OOD) samples from in-distribution ones (Charpentier et al., 2020).
Despite the promising progress in OSD that focuses differentiating samples from the closed and open sets, respectively, limited attention has been devoted to the situation where the closed set involves highly imbalanced classes, which may be quite common in many practical settings. For

---

Published as a conference paper at ICLR 2023
example, for anomaly detection, the known types of anomalies available for model training usually unevenly distributed into multiple categories (e.g., car accident vs. shooting). Similarly, for computer-aided medical diagnosis, the known diseases (to the model) may be highly imbalanced based on the available cases. Thus, following the standard Empirical Risk Minimization (ERM) framework for training, the model may not learn properly from the minority class due to the lack of positive samples. As a result, it is more likely to misidentify a minority-class sample as an unknown-class sample during OSD, leading to a high false-positive rate.
Distributionally Robust Optimization (DRO) offers an effective means to handle the imbalance class distribution in the closed set setting (Qi et al., 2020; Zhu et al., 2019). In DRO, the worst case weighted loss is optimized, where the weights are searched in given neighborhood (referred to as the uncertainty set) of the empirical sample distribution such that the overall loss maximized. By expanding the uncertainty set, the model is encouraged to assign higher weights to difficult samples. As a result, samples from the minority class will be given more emphasis during model training if not properly learned (which incurs a larger loss). Another common solution to handle imbalanced class distribution in the closed set is through oversampling to achieve a more balanced class distribution (Chawla al., 2002). While both oversampling and DRO may help to improve the closed set performance, neither of them is adequate to address OSD from imbalanced data.
A fundamental challenge lies in the interplay between samples from the minority class and the difficult samples from the majority classes. As a result, simply oversampling the minority class may neglect these difficult samples. Similarly, applying DRO with flexible uncertainty much emphasis on these difficult samples and ignore the minority class as well as some representative samples from the majority classes, which affects proper model training. In fact, directly applying these models for OSD may lead to even worse detection performance, which is evidenced by our experimental results. Few recent approaches try to address the OSD under iaas-immbllneeed setting. Liu et al. (2019) leverage the visual similarity across the centroids of closed set classes to allow more effective training from the minority class samples. However, it is possible that the samples from the minority class may look quite different from most other samples, making such strategy less effective. Further, Wang et al. (2022) try to push minority class samples away from open set ones in the feature space using contrastive learning. However, the final OSD depends heavily on the selection of open set samples as evidenced by our experiment results.
may put too
To systematically tackle the fundamental challenge outlined above, we propose Adaptive Robust Evidential Optimization (AREO) that offers a principled way to quantify sample uncertainty through evidential learning while optimally balancing the model training over all classes in the closed set through novel adaptive DRO learning. To avoid the model from primarily focusing on the most diffieult samples by following the standard DRO, the adaptive learning strategy gradually increases the size of the uncertainty set using Multi Scheduler Function (MSF), which allows the model to learn from easy to hard samples. class-ratio biased loss is further assigned to the minority class to ensure proper learning from its limited samples. Our main contribution is fourfold:
a novel extension of DRO to evidential learning, which enables principled uncertainty quantifica- tion under the class imbalanced setting, critical for many applications, including OSD. adaptive DRO training governed by a uniquely designed -uuti-scheduulrr learning mechanism to ensure an optimal model training behavior that gives sufficient attention to the difficult samples and the minority class while capable of learning common patterns from the majority classes, theoretical connection to boosting model (i.e., AdaBoost), which ensures the nice convergence and generalization properties of AREO, of art OSD performance on various datasets.
2 RELATED WORK
Open set detection. Various SVM based techniques (Scheirer et al., 2013; Jain et al., 2014; Scheirer et al., 2014) have been proposed for OSD. For instance, Scheirer et al. (Scheirer et al., 2013) proposed an SVM based model, which performs detection using a Weibull-calibrated SVM (W-SVM) by leveraging Extreme Value Theory (EVT). Reconstruction based approaches have been proposed (Zhang & Patel, 2017), where a threshold defined over the reconstruction error is used to decide whether the sample is from known or an unknown class. Other traditional models, such as nearest neighbor (Júnior et al., 2016), quasi-lines function (Cevikalp & Yavuz, 2017), have also been explored as well. Deep learning models have been increasingly applied for open
2

---

Published as a conference paper at ICLR 2023
set detection (Yoshihashi et al., 2019; Sun al., 2020: Bendale & Boult, 2016). As example, OpenMAX replaces the softmax function and probability of the softmax is redistributed to produce the probability of a sample being unknown (Bendale & Boult, 2016). Sun et al. (2020) proposed VAE based open set recognition, where the probability of a sample belonging to each of the known classes is used as a proxy detect whether the sample is known or unknown. Each known class distribution is modeled as a Gaussian using the training data. Some recent approaches aim to learn a more compact representation of closed set samples (Cevikalp et al., 2021; Yang et al., 2020) or push the open set class samples to specific region in an embedding space for better recognition (Chen et al., 2021). Special loss functions (Dhamija et al., 2018) and generative processes (Perera et al., 2020) have also been leveraged to separate open set samples from closed set ones.
Recently, systematic approaches have been presented to break the closed set limitation by explicitly modeling the uncertainty mass belonging to the unknown distribution. One of the representative work inline with this is the evidential deep learning (EDL) model (Sensoy et al., 2018). Similar to this work. Malinin & Gales (2018) propose Prior Networks (PNs) that explicitly consider the distributional uncertainty to quantify the distribution mismatch. Despite having a natural way to quantify the uncertainty, both of these methods require OOD data samples for model training, which is less practical. Charpentier et al. (2020) propose the posterior networks that leverage the normalizing flows for density estimation in the latent space in order to predict the posterior distribution by only using in-distribution samples. Despite the significant progress in OSD, limited attention has been drawn to the scenario, where the closed set involves highly imbalanced classes. Few recent works try to tackle this fundamental challenge of OSD under Hlass-imbalanced setting. Liu et al. (2019) propose a technique based on the assumption that visual similarity exists between head and tail classes in the closed set. A model is designed to leverage this similarity to make it more robust for recognizing minority class samples. However, such an assumption may not universally hold, which limits the applicability of the model in general settings. Further, Wang et al. (2022) leverage contrastive learning to push the minority class samples away from the open set ones in the feature space during the training process. However, the final OSD performance highly depends the training open set samples.
Distributionally robust optimization. Distributionally robust optimization is based on principled statistical learning theory, where the worst case weighted loss is optimized by scarching the weights in given uncertainty set (Duchi & Namkoong, 2019; Zhu et al., 2019; Namkoong & Duchi, 20.6). DRO offers a systematic way to handle the imbalanced class distribution and has been commonly used in supervised learning setting (Qi al., 2020; Zhu et al., 2019) as well as in multiple instance learning (Sapkota et al., 2021). I). In a similar way. Li et al. (2020) propose a technique called Tilted Empirical Risk Minimization (TERM) by redefining the ERM with the introduction of hyperparameter I. Depending on the tunable parameter / value, different variants of loss (maximum, minimum, and average) are recovered and thereby provide a unified way to perform effective training in the presence of outlier and class imbalance scenarios. While DRO may help to improve the closed set performance, it is not sufficient to address the OSD problem with imbalanced data. This is because DRO with a flexible uncertainty set may put too much emphasis on the difficult samples and ignores the ones from the minority class as well as representative samples from majority classes.
Our proposed AREO model offers an adaptive learning strategy to learn from easy samples in the early training phase and gradually shift the focus to the difficult samples. Furthermore, the class-ratio biased loss ensures proper learning from the limited samples in the minority class.
METHODOLOGY
3.1 PRELIMINARIES
Evidential Learning for OSD. Let DN -X ((),, (x., be set of training samples in the closed set. Each x, E RD is a D-dimensional feature vector and yn E {0, lj indicates the one hot encoding associated with its class label: Yhy and ynk = 0 for all + with j being the true label. Following the principle of Subjective Logic (SL) (Jøsang, 2016), we consider a total of C' + mass values with C' being the number of classes. We assign a belief mass be, Ve E [C], to each singleton, which corresponds to one class in the closed set and the remaining mass is referred to as the uncertainty mass, denoted by u. The belief masses and the uncertainty mass
are all non non-negative and sum to one: u + C b = 1, auu 2 0 and b. 2 0. They can be evaluated as
=1=
Ce u
C S
b
S
=
(1)
3

---

Published as a conference paper at ICLR 2023
where S = == 11) with e. 2 0 being the evidence derived for the ct singleton, which can be generated by a neural network enabled with a non-negen non-negative output. The belief mass assignment in the above expression corresponds to Dirichlet distribution with the concentration parameters Oe = + 1:
={ 1 P8=118 for p E Sc (ia) B(a) -1, (0,
Dir(pla) -
otherwise
where Se is probability for the t singleton is given by Elpe] 3.. Consider sample x, and let f(x,, +) denote the evidence vector generated by an evidential neural network parameterized by θ. This allows us to fully characterize the Dirichlet distribution, whose mean vector gives rise to the probability of assigning x, to each class. There are multiple ways to design a loss function to train the evidential neural network (Sensoy et al., 201 8). A simple but effective option is the sum of square loss:
and B(a) is beta function. Given the evidences, the expected
(0) lly, - + XxK Dir( (1,...,,,),11) 1)
(3)
where min(1, ( the annealing coefficient at epoch t and o, = Yn + (1- yyy) 3 an.
Remark. Besides being used powerful model for closed set classification, a unique benefit of evidential learning is that offers principled way to quantify the uncertainty mass, which is explicitly allocated to account for something that is "unknown" to the model. Intuitively, a properly trained evidential model will output a high total evidence for data samples whose features are sufficiently exposed to the model during training. In contrast, it should predict low total evidence for less representative samples in the training data. For these samples, their corresponding uncertainty mass will be large (as the total mass sums to one). As result, the uncertainty mass fits squarely for detecting open set samples, which have not been exposed to the model that is trained using the closed set samples.
Distributionally Robust Optimization. Distributionally Robust Optimization (DRO) is inherently used to handle the minority and/or difficult class samples by optimizing the worst-case loss where weights assigned to each sample are given by uncertainty set. Let l, (9) be the loss for the x,, sample network parameterized by e. Then the corresponding DRO loss is given as
N
CDRO (9)
max Σ (0) PEPORO m=
(4)
The uncertainty set defined to assign weights (p) is given as
1 PDRO {DER" p 1 (pl N S =n}
:
(5)
where (plla) is f-divergence between two distributions p and q and n) controls the size of the uncertainty set. When v is large, the weight distribution can deviate a lot from the uniform distribution, making it possible to assign a very high weight to certain data samples. In contrast, a small 77 will constrain to be close to the uniform distribution and all samples share similar weight.
3.2 DISTRIBUTIONALLY ROBUST EVIDENTIAL OPTIMIZATION
The standard evidential learning does not explicitly consider an imbalanced class distribution. Further, it also does not focus on the difficult samples resulting from multi-modality where a single class can have multiple types of samples. As a result, minority classes and/or difficult samples are usually assigned higher uncertainty mass due to lack of sufficient training data. While this may not sigmificantly impact the closed set performance (i.e., accuracy). it poses a more severe issue for OSD as difficult/minority class samples become equally uncertain as those open set samples. To address this challenging solution, one straightforward way would be to integrate evidential learning with DRO for robust uncertainty mass quantification on minority class/difficult samples in the close-set. Intuitively, since the model explicitly focuses on learning from minority class/difficult samples, it provides a low uncertainty mass for minority/difficult samples while remain high (in terms of uncertainty mass) for those open set samples. This novel integration of DRO and evidential learning allows us to define a distributionally robust evidential loss (DREL) given as
N
DREL (0) = max
P EL
(O)
(6)
PEP P
4

---

Published as a conference paper at ICLR 2023
1.5
04
14
5.
Ntarber 2R
Numbh Eaech
(a) Cosine
(b) Offset Cosine
(c) Exponential
(d)
Figure I: Examples of Scheduler Functions
Details of solving (6) is provided in Appendix Depending on 1j the uncertainty set, we can decide whether we want to assign equal weight to all data samples or focus on the most difficult ones. The lemma below reveals the relationship between DREL and the standard evidential loss.
Lemma 1. With - 0, the EDL loss under DRO reduces to the standard EDL loss.
When is set to be very small, the model gives similar weights to all samples, which allows them to participate equally in the training process. At another extreme, we can direct the model to fully focus on the most difficult sample with the maximum loss, summarized in the lemma below.
With rj + oc, the loss under DRO becomes equivalent a maximum loss based approach focusing only on the hardest sample.
Lemma 2.
The above lemma implies that highly flexible uncertainty set may cause the model put much emphasis on difficult samples. Since these difficult samples may come from the majority classes, simply setting a large 7) will not necessary direct the model's attention the samples from the minority class. Furthermore, using a flexible uncertainty set in the initial phase of the model training may misguide the model to neglect a large number of representative data samples. As a result, the model will not be able to capture the common patterns that exhibit in most of the training samples. As such, the direct integration of DRO and EDL does work well which is also justified experimentally through the comparison of the proposed technique with DRO technique.
3.3 ADAPTIVE ROBUST EVIDENTIAL. OPTIMIZATION (AREO)
The key idea to address the limitations in the distributionally robust evidential optimization is to gradually increase the size of the uncertainty set, which allows the model learn from easy to hard samples from closed set classes. Scheduler functions (SF) provide a natural way to achieve the desired training behavior. Figure (a-c) shows three typical SFs, including cosine in (a): cos 27
(F) 1.2 and exponential in (c): exp (-$). where i denotes the index of the training epoch, T is the terminating epoch, and / is specific parameter of the exponential function. It can be seen that while the general trends of different SFs are similar, they exhibit some key differences that may lead to quite distinct model training behaviors. For example, a cosine function can help to ensure the uncertainty set to stay small for a relatively longer time in the beginning of model training. This ensures the model to learn from the representative samples in the majority classes (according to Lemma 1). In contrast, an exponential function can change the size of the uncertainty set very rapidly, which can give the model more time to learn from the difficult samples at the later phase (according to Lemma 2). The offset cosine function can offer both a relatively long initial learning and later learning phases. However, choosing SF that best matches the nature of given dataset poses key challenge. Furthermore, single SF may not be rich enough to express the desired training behavior of complex dataset.
offset cosine in (b): 1 cos
To address this key challenge, we propose to conduct multi-scheduler learning to automatically construct a composite scheduler function that can be automatically learned for each given dataset to deliver the optimal training behavior. More specifically, the multi-scheduler function (MSF) is formulated as a convex combination of a of atomic SFs:
M
M
Wm SF (8m, 1, T),
Σ m=1
Wm = 1, Wm 2 0 Vm E [M]
(7)
m=11
where w are the mixing weights and B a set of specific parameters for the atomic SFs. Figure (d) visualizes example MSF that combines cosine and exponential functions with different mixing
weights and fixed 20, T 600. As can be seen, the MSF is much more expressive then either its
component SF, which makes it capable represent much broader range of training behaviors.
5

---

Published as a conference paper at ICLR 2023
By leveraging the proposed MSF to control the size of the uncertainty set, achieve adaptive robust training. Let 17o be the initial size of the uncertainty set and the size of the set at epoch is
Te 7
(8)
MSF(W, B, T)
Based on this adaptive uncertainty set, we define the Adaptive Robust Evidential Loss (AREL) as
N
CAREL (O) =
max PE DEPARO
Pr
(())
where IEL is the uncertainty loss for the xn given by Eq. (3) under adaptive robust optimization framework and pARO is the adaptive robust uncertainty set defined as
PARO SERE 1 2 0, D,
1 N
S
(10)
As Th increases, the model gradually shifts its focus from easier samples to the more difficult ones. In this way, the model can be trained first capture the common patterns the data and then conduct fine-tuning by attending to those difficult samples. However, for imbalanced classes, there may be good number of difficult samples from the majority classes. Therefore, solely controlling the size of the uncertainty set does not guarantee sufficient training over the minority class. To address this, we further leverage the label of the minority-class c to formulate a ratio biased weight augmentation on samples from this class. Let p(c) -ygn.=1 Pn be the total weight for minority class e obtained by solving (9). Then, the weights for the minority class samples are adjusted as:
plc), if p(c) 2 d min (d. MSS (oolc) ( S.A...
p(e) p Pui if Yno = 1 1 p(e)
p(c)=
otherwise
Pn
(11)
I-P(c) Pn, otherwise
As the MSF monotonically decreases over the training epochs, the total weight for the minority class samples will eventually reach t. making it equally weighted as the other (C - 1) classes.
Remark. Our approach considers a minority class if there an obvious gap between the percentage of samples from the minority class over the total samples from all C classes and c. Any other class that not minority one is regarded as majority class. Our approach can handle the multiple minority classes which can be achieved by applying the ratio biased weighted augmentation (given by Eq. (1 (11)) to each minority class.
The adaptive robust training is achieved through bi-level optimization, where the inner loop optimizes the the model parameters (0) and the outer loop optimizes the MSF parameters W {w, w', 3'3';
min CAREL ( W), s.t. O- arg min CAREL (O, W)
(12)
where CAREL Faal are training and validation losses, respectively. The outer loop optimization can be solved by computing the Hypergradients (Maclaurin et al., 2015; Pedregosa, 2016) or through a population-base baseed methods (Jaderberg al., 2017), where the former may easily get stuck in local optimum (Tao et al., 2020). To this end, we extend the existing population based method to learn an optimal MSF and the details are given in Appendix B.
3.4 THEORETICAL ANALYSIS
We establish the key theoretical properties of AREO, including the convergence speed in model training and the generalization capability by formally demonstrating the equivalence between AREO and AdaBoost under a non-convex robust uncertainty loss. The key idea is to leverage the equivalence between AdaBoost and the gradient descent search of an optimal function from linear combination of a set of (weak) learners (Mohri et al., 2012: Blanchet et al., 2019). Let F = [fi,... JK be a set of different classifiers, and the linear span generated by the set F is
{
K
LS(F) f:f Σ o
(13)

---

Published as a conference paper at ICLR 2023
AREO training consists of two alternative updates between optimizing the worst probability and predicting function f. The update in function prediction can be regarded as finding sub- sub-gradient G. aLAREL(f and updating with ILss (9) arg (in)E Il Gellow where DN is the training data. Letting (fe be the loss associated with the data sample Xn, the update of p involves the optimization of the following objective with ft. being fixed:
CREL (J:)
max (fft)
(14)
PEPARO
where the uncertainty set is given by (10). The corresponding Lagrangian of the above optimization problem is given by
[E k==1
N Pr. log Pr.
p20,p 1=1 Σ (fi)
7
(15)
It should be noted that finding the optimal f value is non-trin non-trivial because the optimization involves the nonconvex loss (i.e., CARÉU). This creates difficulty showing equivalence between AREO and AdaBoost. To ensure the convergence of f to stationary point, we adapt the ProbAbilistic Gradient Estimator technique (PAGE) (Li al., 2021) to our unique adaptive robust evidential optimization setting. This convergence guarantee helps to move forward showing the equivalence between AREO and Adaboost given by the theorem below.
Theorem 3. Under the assumption of finite exponential moment for Cn(f), with a sufficiently large and
0 being
Th 3 (3*) rb33))
(16)
the worst case probability p is given by
(nllfe) a
exp
Pr
(17)
(.(Se)
S==1 exp
En exp(BCa(f.))
where === = a" 2 O be the optimal a, and (b(3) log o. optimization between f and with above worst case probability solution exactly recovers the AdaBoost algorithu proposed in (Freund & Schapire, 1997).
The alternative
N
Remark. There are several key benefits of connecting AREO with AdaBoost. First, AdaBoost is less prone to overfitting even running for a large number of iterations (Mease & Wyner, 2008). Inheriting such a property crucial for OSD as an overfitted evidential model can produce highly confident wrong predictions. This implies that low uncertainty may be predicted for samples that the model is less familiar with, resulting in a false negative detection of an open set sample. Furthermore, since the target function is expressed as linear combination of set of weak learners, the optimal function can be regarded as maximizing the 4 geometric margin among the training samples to ensure good generalization capability like other maximum-mane marginn classifiers (Mohri et al., 2012). This ensures decent closed set performance from AREO (as shown by our experiments). The proof of Theorem is provided in Appendix C.
EXPERIMENTS
We perform extensive experimentation to evaluate the effectiveness of the proposed AREO model. We first describe five real-world image datasets where minority class is introduced to create an imbalanced setting. We then assess the OSD performance the proposed technique by comparing with competitive baselines. Finally, we conduct some qualitative analysis, which uncovers deeper insights on the performance advantage of the proposed model.
4.1 DATASETS
Our experiments involve five real-world image datasets: Cifar10. Cifar100 (Krizhevsky, 2009), ImageNet (Deng al., 2009), MNIST (Deng, 2012), and Architecture Heritage Elements Dataset (AHED) (Llamas, 2017). In our experimentation, model training is performed solely based on the closed set samples. During the detection phase, the testing samples corresponding to the closed
7

---

Published as a conference paper at ICLR 2023
set classes will be assessed against the samples from open set classes. For all datasets, for the hyperparameter optimization, randomly selected 20% of the training set is used. The brief description for each dataset is given below. For the detailed description and data sample distribution in majority and minority classes, please refer to the Appendix.
MNIST:
Five classes are treated as open and the rest the closed set. To make the dataset imbalanced, we consider class '3' as minority class and randomly select 30% data samples as compared with other majority classes. The same imbalanced ratio is applied both training and testing sets. In addition to the MNIST open set classes as described above, we follow other existing works (Sun et al., 2020) and further test the OSD performance on additional open set samples from three more sources: (1) MNIST-Noise. (2) Noise, and (3) Omnigolot (Lake et al., 2015).
Cifar10: Five classes are assigned as open set and closed set, respectively. Bird is made as the minority class using the same strategy introduced above. In addition to the open set classes from Cifar10 itself, we further assess the OSD performance with Cifar+10 and Cifar+50.
Cifar100: "Living being' related super classes are assigned as the closed set and the remaining super classes are assigned as the open set. We make 'insect' related classes as the minority one. ImageNet: Five classes are assigned as open set and closed set, respectively. We make "king crab' as the minority class.
Architectural Heritage Elements Dataset (AHED): Five classes are assigned as open set and closed set, respectively. This is inherently highly imbalanced dataset where number of data points are unevenly distributed across different classes. The class 'portal' the minority one.
4.2 EXPERIMENTAL. SETTINGS
Evaluation metric. To assess the model performance, we report mean average precision (MAP) score which summarizes the precision-recall curve as a weighted mean of precision achieved at each threshold, with the increase in recall from previous threshold as the weight. Specifically, in the OSD, we treat the open set samples as positive and closed set samples as negative and compute the MAP score based on the uncertainty score produced by the trained model. Different from AUROC, MAP places more emphasis on initial part of the ROC curve, which gives preference if model can rank the open set samples on the top based their predicted uncertainty scores. This MAP metric works well in practice as the main focus may be devoted to the first few predicted candidate samples, especially when there is a long candidate list. The theoretical result shows that MAP is approximately the AUROC times the initial precision of the model (Su et al., 2015). Therefore, we focus on reporting the MAP performance and leave the AUROC results in Appendix D. It is worth to note that our AUROC results also show consistent trend as the MAP results.
Network architecture. In terms of the architecture of the evidential neural network, for all datasets, we use an LeNet5 network with tanh activation in the feature extractor and ReLU in the fully connected layers. For training, we use the Adam optimizer with a learning rate of 0.001 and l2 regularization with coefficient of 0.001. The detailed hyperparameter setting is provided in Appendix.
4.3 PERFORMANCE COMPARISON
In our comparison study, we include baselines that are most relevant to our model, including EDL, EDL augmented with oversampling using SMOTE (Chawla et al., 2002) (referred to AEDI.,, and EDL with standard DRO training (referred to as DRO). Further, we also compare the performance with the Posterior networks (Charpentier et al., 2020) and its robust form, PostNet (RS), proposed by Kopetzki et al. (Kopetzki et al., 2021). In addition, we also compare with representative baselines with outstanding OSD performance: OpenMAX (Bendale & Boult, 2016), CGDL (Sun et al., 2020), and OLTR (Liu et al., 2019). Please refer to the Appendix D for the more detailed description of the baselines used in our comparison study along with additional results and an ablation study.
Tables presents the OSD performance comparison between different models for all five datasets. AREO consistently outperforms all the baselines across all the datasets. For certain datasets, the performance advantage over the second best model is more than or close to 10%. This clearly demonstrates the benefits of conducting evidential learning through adaptive DRO training to achieve an optimal balanced learning from all classes and different types of data samples. We also observe that EDL consistently performs better than other non-evidential learning based models, such as OpenMAX. in most cases. The better OSD performance from EDL is attributed to its explicit modeling of the uncertainty mass that works naturally for detecting the open set samples. In contrast, directly applying DRO with flexible uncertainty set, which aims address the imbalanced class
8

---

Published as a conference paper at ICLR 2023
Table 1: OSD (MAP) performance all datasets
Cifarlò
Approach
Cifar100
ImageNet
Cifar10
Cifar+10
Cifar+50
EDL
62.42 + 0.31
29.23 =
65.75 1.11
62.00 = 2.40
55.93 + 4.30
AEDI
54.19 + 0.77
26.21 0.68
63.04
0.70
52.79 + 0.91
57.94
+ 0.07
1:
1.63
50.78
4.44
55.67
3.86
OpenMAX
59.65 + 1.03
21.18 1.34
62.80 + 2.08
50.88
= 0.60
53.21 = 0.39
CGDL
54.27 + 2.06
16.83 +: 0.20
50.13
1.08
50.50
+ 4.56
55.47 + 1,53
5.39
62.51
4.68
53.85
2.76
56.83
1.52
PostNet (RS)
#1.31-11.22
18.25
1.50
53.13
1.33
51.75
0000000000 0.98
56.21
+ 0.75
OLTR
56.37 + 0.25
19.59
0.49
53.98 + 0.68
48.48 = 0.29
50.71 + 0.06
4.08
57.52
Approach
MINSS
AHED
MNIST
Noise
MNIST Noise
Omnigoloi
87.32
8.74 71.30
77.62 = 6.79
50.23
AEDL
75.37 # 11.14
11.45
76,23 = 12.67
67.29 + 10.77
52.22 + 0.26
DRO
63.25 1: 4.32
46.78
1.22
49.59 3.98
48.15 4: 1.70
42.28 4: 0.18
84.11
1.35
1.71
81.14
CGDL
61.33 1.33
74.88
8.42
13.92
10.7222.11
48.57 = 1.39
ProtNet
55.58 4.1.2
13.88 47.53
43.94 + 8.00
72.79 + 4.24
46.69 + 1.90
PostNet (RS)
49.31
36.14
0.59
46.10
4.37
OLTR
86 38
+ 0.31
90.01
1:43
83.73 1 1.28
55.27 = 3.01
26
+ 1.66
AREO
90.80 + 0.058
94.24
0.32
94.18
0.21
93.80 = 0.2
53.21
0.65
Approach
Sample
EDL
AEDL
DRO
AREO
Birdl
137
1533
3109
Bird2
4548
4225
3985
100
Bird1
Bird2
Bird3
1928
2258
4452
183
Boat
1274
1520
1249
223
Truck
3308
1318
4208
51
Boat
Truck
(a) Representative difficult samples
(b) Ranking of samples
Figure 2: (a) Top row: minority class; bottom-row: majority classes; (b) sample ranking.
distribution, leads rather poor OSD performance due to the reasons analyzed in prior sections. Similarly, AEDL does not perform better than the standard EDL due to the lack of fine-tuning of the difficult examples from the majority classes that become inseparable from the open set samples with high predicted uncertainty score. Table 10 in the Appendix also shows the closed set performance as reference. Further, for the deeper insight on the superior OSD of AREO please refer to Appendix.
4.4 QUALITATIVE EXAMPLES
We perform a qualitative analysis to further assess the effectiveness of AREO. Figure 2 (a) top row shows representative testing samples from the minority class ('bird') in Cifar10. These images appear to be difficult even for the humans to identify the bird as only small part is visible. Thus, EDL, AEDL, and DRO assign a relative higher uncertainty score for them. As result, many open set samples may be assigned relatively lower uncertainty score, leading to false negative detection on these samples. Figure (b) shows the ranking of these samples according lower ranking indicates lower uncertainty). In contrast, AREO assigns much lower rankings for these birds objects. This analysis justifics the effectiveness AREO for detecting minority class data samples in the closed set. Similarly, Figure 2 (a) bottom show representative images from some majority classes. Again, AREO is able to recognize these difficult samples and assign a relatively low uncertainty score avoid them being mis-identified open samples shown by Figure 2 (b).
uncertainty scores (a
5 CONCLUSION
In this paper, we focus on open set detection from imbalanced closed set data. To address the fundamental challenge due to the interplay between the minority-class samples and difficult samples from the majority classes, we propose important extension of DRO to the evidential learning setting, leading to novel Adaptive Robust Evidential Optimization (AREO) model. As an evidential learning model, AREO effectively breaks the closed set assumption by explicitly modeling the uncertainty mass that uniquely suitable for detecting open set samples. An adaptive DRO training process is achieved through multi-scheduler learning achieve an optimal training behavior. The experimentation conducted on five real-world datasets with diverse types of open set data samples justifies the effectiveness of the proposed model.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This research was supported in part by NSF IIS award IIS-1814450 and an ONR award N00014- 18-1-2875. The views and conclusions contained in this paper are those of the authors and should not be interpreted representing any funding agency.
REFERENCES
Abhijit Bendale and Terrance E. Boult. Towards open set deep networks. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1563-1572, 2016.
José H. Blanchet, Yang Kang, Fan Zhang, and Zhangyi Hu. distributionally robust boosting algorithm. 2019 Winter Simulation Conference (WSC), pp. 3728-3739, 2019.
Hakan Cevikalp and Hasan Serhan Yavuz. Fast and accurate face recognition with image sets. 2017 IEEE International Conference Computer Vision Workshops (ICCVW), pp. 1564-1572, 2014 2017.
Hakan Cevikalp, Bedirhan Uzun, Okan Köpüklüü and Gurkan Ozturk. Deep compact polyhedral conic classifier for open and closed set recognition. Pultern Recognition, 119:108080 2021.
Bertrand Charpentier, Daniel Zügner, Stephan Günnemann. Posterior network: Uncertainty estimation without ood samples via density-based pseudo-counts. ArXiv, abs/2006.09239, 2020.
Nitesh V. Chawla, Kevin W. Bowyer, Lawrence Hall, and W. Philip Kegelmeyer. Smote: Synthetic minority over-sampling technique. J. Artif. Int. Res., 16(1):321-357, June 2002. ISSN 1076-9757.
Guangyao Chen, Peixi Peng, Xiangqian Wang, and Yonghong Tian. Adversarial reciprocal points learning for open set recognition. IEEE Trunsactions Puttern Analysis and Machine Intelligence, 2021. doi: 10.1109/PAMM.22213300077
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.
Icee, 2009.
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):14-114, 2012.
Akshay Raj Dhamija, Manuel Günther, and Terrance E. Boult, Reducing network agnostophobia. CoRR, abs/181 1.04110, 2018. URL httt:://aaadv.orr/abb////L/ 1811.11111111 04110.
John Duchi and Hongseok Namkoong. Variance-based regularization with convex objectives. Mach. Learn. Res., (101)))44022555 January 2019. ISSN 1532-4435.
Yoav Freund and Robert E. Schapire. theoreici generalization of on-line learning and an application to boosting. In COLT 1997, 1997.
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojeiech M. Czamecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu. Population based training of neural networks. ArXiv, abs/17 1711099111,,, 2017.
Lalit P. Jain, Walter J. Scheirer, and Terrance E. Boult. Multi-class open set recognition using
probability of inclusion. In ECCV, 2014.
Minki Jeong, Scokeon Choi, and Changick Kim. Few-shot open-set recognition by transformation consistency. In CVPR, 2021.
Audun Jøsang. Subjective logic. Springer, 2016.
Pedro Ribeiro Mendes Júnior, Roberto Medeiros de Souza, Rafael de Oliveira Werneck, Bernardo Stein, Daniel V. Pazinato, Waldir R. de Almeida, Otávio Augusto Bizetto Penatti. Ricardo da Silva Torres, and Anderson Rocha. Nearest neighbors distance ratio open-set classifier, Machine Learning, 106:359-386, 2016.
10

---

Published as a conference paper at ICLR 2023
Shu Kong and Deva Ramanan. Opengan: Open-set recognition via open data generation. CoRR, abs/2104.02939, 2021. URL https: http:///xxvvv.roo http//ab///a/// 02939.
Anna-Kathrin Kopetzki, Bertrand Charpentier, Daniel Zügner, Sandhya Giri, and Stephan Günnemann. Evaluating robustness of predictive uncertainty estimation: Are dirichlet-based models reliable? In ICML, 2021.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
Brenden M. Lake, Ruslan Salakhutdinov, and Joshua Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 1332- 1338, 2015.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6((1))277822224 1998. doi: 10.1109/5.726791
Tian Li, Ahmad Beirami, Maziar Sanjabi, and Virginia Smith. Tilted empirical risk minimization. CoRR, abs/2007.01 2020. URL https:
owg. /207 01162.
Zhize Li, Hongyan Bao, Xiangliang Zhang, and Peter Richtarik. Page: A simple and optimal
probabilistic gradient estimator for nonconvex optimization. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Reseurch, pp. 6286-6295. PMLR, 18-24 Jul 2021. URL https:/ http///ooeeiiccooooooooooooo ngs. ml www..s.prssss.........e.... / vi 39/ 1019//121e 12lla. html.
Bo Liu, Hao Kang, Haoxiang Li, Gang Hua, and Nuno Vasconcelos. Few-shot open-set recognition using meta-learning. 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 8795-8804, 2020.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boging Gong, and Stella X. Yu. Large-scale long-tailed recognition an open world. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2532-25411 2019.
Jose Llamas. Architectural heritage elements image dataset, 2017.
David G. Luenberger. Optimization by Vector Space Methods. John Wiley and Sons, Inc., USA, 1st edition, 1997. ISBN 047155359X.
Dougal Maclaurin, David Duvenaud, and Ryan P. Adams. Gradient-based hyperparameter opti- mization through reversible learning. In Proceedings of the 32nd International Conference on International Conference on Machine Learning Volume 37, ICML' 15, pp. 2113-2222. JMLR.org, 2015.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, pp. 7047-7058, Red Hook, NY, USA, 2018. Curran Associates Inc.
David Mease and Abraham J. Wyner. Evidence contrary to the statistical view of boosting, J. Mach. Learn. Res., 9:131-156, 2008.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning. The MIT Press, 2012. ISBN 026201825X.
Hongseok Namkoong and John C Duchi. Stochastic gradient methods for distributionally ro- bust optimization with f-divergences. In Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29, Curran As- sociates, Inc., 2016. URL https: h/tpoc/eero//g//p/////ee neurips cctpapper/21///EL...... pdf.
4588e67 4 d3 2at.aa/00000000000000000011111e1
Lawrence Neal. Matthew Lyle Olson, Xiaoli Z. Fern, Weng-Keen Wong, and Fuxin Li. Open set learning with counterfactual images. In ECCV, 2018.
Maria-Elena Nilsback and Andrew Zisserman. visual vocabulary for flower classification. IEEE Conference Computer Vision and Pattern Recognition, volume 2, pp. 1447-1454, 2006.

---

Published as a conference paper at ICLR 2023
Fabian Pedregosa. Hyperparameter optimization with approximate gradient. ArXiv, 102/160.
2016.
Pramuditha Perera, Vlad Morariu, R. Jain, Varun Manjunatha, Curtis Wigington, Vicente Ordonez, and Vishal M. Patel. Generative-diserimnaati feature representations for open-set recognition. 2020 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11811- 11820, 2020.
Qi Qi, Yan Yan, Zixuan Wu, Xiaoyu Wang. and Tianbao Yang. A Simple and Effective Framework for Pairwise Deep Metric Learning, pp. 375-391. arXiv, 11 2020. ISBN 978-3-030-58582-2. doi: 0000/9988--00--58888-
Ariadna Quattoni and Antonio Torralba. Recognizing indoor scenes. In 2009 JEEE Conference on Computer Vision and Patlern Recognition, pp. 413-420, 2009. doi: 10.
Hitesh Sapkota, Yiming Ying, Feng Chen, and Qi Yu. Distributionally robust optimization for deep kernel multiple instance learning. Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings of The 24th International Conference Artificial Intelligence and Statisties, volume 130 of Proceedings of Machine Learning Research, pp. 2188-2196. PMLR, 13-15 Apr 2021. URL https: http:///poceediggee ml prw.spppessse.... vv130 http://///0//aaaaaaaae sapkota2kaaaaammm html.
Walter J. Scheirer, Anderson Rocha, Archana Sapkota, and Terrance E. Boult. Toward open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35:1757-1772. 2013.
Walter J. Scheirer, Lalit P. Jain, and Terrance E. Boult. Probability models for open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36:2317-2324, 2014.
M. Sensoy, Melih Kandemir, and Lance M. Kaplan. Evidential deep learning to quantify classification uncertainty. In NeurlPS, 2018.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Pro- ceedings of the 31st International Conference Neural Information Processing Systems, NIPS 17, pp. 4080 4090, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 97815 10860964.
Wanhua Su, Yan Yuan, and Mu Zhu, A relationship between the average precision and the area under the curve. In Proceedings of the 2015 International Conference on The Theory of Information Retrieval, pp. 349-352, 2015.
Xin Sun, Zhenning Yang. Chi Zhang, Guohao Peng, and Keck-Voon Ling. Conditional gaussian distribution learning for open set recognition. 2020 JEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 13477-13486, 2020.
Zhiqiang Tao, Yaliang Li, Bolin Ding, Ce Zhang, Jingren Zhou, and Yun Fu. Learning mutate with hypergradient guided population. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20 20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Haotao Wang, Aston Zhang, Yi Zhu, Shuai Zheng, Mu Mu Alex Smola, and Zhangyang Wang. Partial and asymmetric contrastive learning for cuttr--distribution detection in long- recognition. In International Conference on Machine Learning, pp. 23446-23455 2022.
Mong-Ming Yang, Xu-Yas Yao Zhang, Fei Yin, Qing Yang, and Cheng-Lin Liu. Convolutional prototype network for open set recognition. IEEE Transactions Pattern Analysis and Machine Intelligence, 2020.
Ryota Yoshihashi, Wen Shao, Rei Kawakami, Shaodi You, Makoto lida, and Takeshi Naemura. Clasification-recnntttcciioo learning for open-set recognition. 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 401 1-4000, 2019.
He Zhang and Vishal M. Patel. Sparse representaation- open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19:1690-1696, 2017.
Dixian Zhu, Zhe Li, Xiaoyu Wang, Boqing Gong, and Tianbao Yang. A robust zero-sum game framework for pool-based active learning. In AISTATS, 2019.
12

---

Published as a conference paper at ICLR 2023
Appendix
In this appendix, we first summarize the major notations used in the main paper in Appendix A. We then present the detailed description of the training process obtained through a bi-level optimization in Appendix B. Proofs of main theoretical results provided in Appendix C. Finally, we present more detailed experimental dataset, result, and settings in Appendix D. The link to the source code is provided in Appendix E.
A SUMMARY OF NOTATIONS
Table 2 summarizes all the major symbols along with their descriptions.
Table 2: Symbols with Descriptions
Notation
Description
by
Belief mass associated with class c
Total number of classes
Ce
Evidence for the c" singleton
Uncertainty mass
a
Dirichlet Parameters
Probability for the ed singleton
Pc
Yn
One hot encoded C dimensional multinomial variable
Vne
Class lahel for the nt data sample for class Probability of the v." th data sample belonging to class c
Pnc
Uncertainty set size for AREO Hyperparameter controlling the schedule Hyperparameter controlling the emphasis in a minority class
B
P
Probability distribution in the DRO framework
PDRO
Uncertainty set of DRO
paRo
Uncertainty set of ARO
Evidential network parameters
(F"(O)
Evidential loss with the n1 data sample
(A)
Adaptive robust evidential Loss
F
Set Different classifiers
Weight associated with 6tth weak learner
dk
p(c)
Weight associated with the t class from Eq. (6)
p(c)
Readjusted weight associated with the cd class from Eq. (11)
Mixing weights associated with the MSF to control uncertainty set ?
w
Mixing weights associated with MSF to readjust the class-specific weights
Set of Specific parameters for the SFs to control uncertainty set f):
Set of Specific parameters for SFs to readjust the class- specific weights MSF Parameter sets associated in our model training Total number of Epcochs
W
T
B TRAINING THROUGH BI-LEVEL OPTIMIZATION
Our training involves a bi-level optimization, where we jointly optimize the network parameter θ along with the MSF parameters W. Algorithm shows the overall training process based on the population based optimization. We randomly initialize the MSF parameters W, and network parameters G, from the corresponding spaces H. and Q Oprrmm respectively shown in Line 3. We perform this initialization for P different models. Next, in each epoch we independently optimize P models using the proposed objective function defined in Eq. (6). After s epochs, we evaluate the accuracy of each model by using 'eval' as the evaluation metric in the validation set. It should be noted that in our case, we used closed set classification performance (MAP) as 'eval' metric. We identify P (with P < P) worst performing models and replace their model parameters by the randomly selected model parameters from set of b highest accurate models. This process is known as
13

---

Published as a conference paper at ICLR 2023
exploitation and is demonstrated in Line 12. MSF parameters for those worst performing model can be obtained either through random selection from the original space H. or through small perturbation of the W of the model whose parameter is copied. This process is called exploration for the new MSF parameters and is shown in Line 13. The best performing model parameters and accuracy are stored in the @ and acc" respectively. Finally, the best model @* is returned as the optimal model for the testing.
searching
Algorithm 1: Multi-Scheduler Learning Process
Input: H. P. s, eval, P, T Initialize: epoch = 0, 0* = None, acc* = None
for p E [P] do
L Pp. Wp + initialize (0 param, H)
while epoch<T do
Op pp--oiiizee@0,1WW p.. p E [P]
eppoch% = then
accp + ev. Wp Wp).p p [P]
sorted_ide idr t arg sort (accp)
bottom_idaa idx + sorted corted_idx[: - P] top top_idx= t sorted_ idx|: (: P1 for idr &bottommide do
10
O_idr,j uniform([O)))) (O top
13
W Wiide +- explore Wi) best model idr +- top idæ if 0* not None then
14
15
16
if acc aCCbest model_ idz then
acc* = aCCbest model_dde idr O" Obest meest_mmdeliidee
18
19
else
+ Obsttmmodee model_idæ
21
epoch + epoch
return O* with the highest acc
The optimization specified in (9) involves an inequality constraint, which incurs a higher computa- tional overhead. Therefore, in our actual optimization process, we consider a regularized version of the AREO loss as follows:
AREL
max D2O.
Σ Pnlh AD, pl N
(18)
1=1 n=1
Solving the above maximization problem leads to closed form solution for p' shown by the following lemma. It should be noted that the role of the À is exactly opposite as that of the Specifically, we start from high À that the model gives equal emphasis to all data samples. Next, in each step we decrease X using the following Equation
(MMFFWWWFF
(19)
Decreasing X helps the model focus on the difficult samples as training progresses.
Lemma 4. Assuming that Dy is the KL. divergence, then solving (18) leads to the following solution
N
C
Σ
P
(20)
n=1
where p", is given by
P exp
(21)
N j=1
exp
14

---

Published as a conference paper at ICLR 2023
The detailed proof is below in C.
C PROOFS OF THEORETICAL RESULTS
In this section, we present the detailed proofs for Lemmas 1, 2, 4, and Theorem 3.
Proof Lemma 1: By setting + O, we have (p } This implies that p is uniform with each element as N. As a result, the optimization problem becomes
(DRO (0)
n (O)
(22)
Proof of Lemma 2: With - oc, the uncertainty set defined in (5) reduces to the following
PDRO_ = E R :p=1= 1, 20 0}
(23)
Now, the corresponding Lagrangian form of (6) becomes
N u, Σ (pnl) + uuPm)) +
DRO (O,
Σ n=1
Pn -
(24)
where and are Lagrangian multipliers. Taking gradient with respect to pa and setting it zero, we get
(0) + un ====0
(25)
Let h: arg max, (0) be the index of data sample with the maximum loss (and assuming it is unique). Then, the following holds true
Vn E [1, n K < Uni
(26)
This consequently leads un Vn E [1, NI, # k. Due to the KKT conditions,
unPn = O: Vn E NJ
(27)
we have Pa = 0, Vn E [1, n k. By using the following constraint
Σ
Pa =1
(28)
7=1
we have the following conclusion
i, 0. otherwise
Pn
(29)
This means our optimization reduces to the following
[ CAREL max EL (e)
(30)
which proves the lemma.
Proof of Lemma 4:
The the Lagrangian of the regularized loss in (18)
-1
CAREL (A,,)(
Σ P
Σ Pm log Pm log N
Σ
Pn
(31)
where v the Lagrangian multiplier. Taking the derivative with respect to pn. and setting to
1 A lo -
(32)
Simplifying above equation, get Pr,
1 +
Pn = exp
(33)
A
15

---

Published as a conference paper at ICLR 2023
Using the summation constraint over Pn. i.e.,
Σn= Pr 1, it leads to following
1 + -1)=
Σ exp
=1
(34)
Solving the above equation we get the expression for v as follows
1
e = A log
(35)
4 -1-1
n=1 exp
Substituting the value into (33) gives
1/2 x
exp
Pn N 1=
(36)
1t exp
The concludes the proof of Lemma
Proof of Theorem 3. AdaBoost can be achieved through alternative optimization between classification function of and the worst case probability solution (Freund & Schapire, 1997). To show equivalence with the proposed AREO, our proof includes three steps: (i) a specially designed deep neural network (DNN) architecture and a loss function adapted to match the learning process of AdaBoost, (ii) projected functional aub-gradient descent optimize the classification function f. and (iii) optimizing the worst case probability solution.
Step 1: A specially designed DNN. Let Φ(x) R. denote a M Mddimensional feature vector learned using a DNN. By applying a fully connected linear layer with a weight matrix E R x M on top of the feature vector, we obtain Then, the final output of the DNN is obtained by aggregating these K functions, leading to f f= a f.
of K (discriminative) functions: f = (f1,-.., fk., T (ø(x).
where (..,, σ T As result of this design, the final function output by the DNN can be regarded as lying in the linear span of set of functions F (i.,...JJ fK given by
{ f:f
K
f:f= Σ onfE. 1 S < K, on E 20)
LS(F)
(37)
k=1
Training of AREO involves alternating between re-weighting using the worst case probability dis- tribution and updating the prediction function f. Next, we prove that given the specially designed DNN, we can exactly optimize the classification function by keeping the worst case probability fixed and vice versa.
Optimizing the classification funciioo f under the worst case probability. We first formulate the distributional robust evidential loss, which given by
Step 2:
N
CAREL
Σ Cn)
(38)
PEPARO #=1
where C, (f) is the loss associated with the datasample X.. Then, the optimal f" can be obtained by minimizing the distributional robust loss:
min CAREL SELS(F)
(39)
This optimization involves a nonconvex loss CAREL, To ensure the convergence of f to stationary point, adapt the ProbAbilistic Gradient Estimator (PAGE) technique (Li et al., 2021) to the DRO setting (shown Algorithm 2) which ensures the convergence in O(b + 4 steps with 5 being the batch size. Please refer to Theorem 6 further details.
To show that an optimal fr can be achieved, we first verify that the specially designed DNN and the loss function described above meet number key conditions specified by (Blanchet et al., 2019): (i) the loss functional CRREL is L-smooth, (ii) for two different functions f f e I.S(F), ( (((()) # (²)(((x))) )), and (iii) LS(F) has finite dimensional basis. First, (i) is true
16

---

Published as a conference paper at ICLR 2023
because LAREL. is the convex combination of the losses (f). As each individual loss involves the ReLU term with ReLU added in the output of DNN (to ensure non-negan negattiitii of the evidence), the resulting convex combination may not be smooth. Therefore, function to approximate the ReLUU. The the convex combination of SoftPlus results in the function LAREL be I--smooth. Second, the rich and high dimensional input data (i.e., diverse images) and the feature encoding through the deep architecture of the DNN ensures (ii) is true. Last, since the dimensionality of the weight matrix W is K M, implies that the dimensionality of the basis of LS(F) is bounded by K, so (iii) holds true.
the SoftPlus which is smooth
The smoothness of L LARL ensures that stationary solution is achieved within the O(b + - gradient
steps. This allows us to have a guaranteed stationary solution with E []] S in a non- convex optimization setting. Furthermore, since CARET. is functional on f. the next two conditions ensure that the functional gradient exists and can be evaluated (Blanchet et al., 2019). During the optimization process, we need to make sure that the trajectory of the functional gradient lies the space LS(F), which can be achieved through functional gradient projection.
Step 3: Optimizing the worst case probability solution. Let ft denote the optimal classification func- tion for the current iteration t. Next, we continue to optimize the worst case probability solution. The following lemma shows that such an optimal solution exists.
Lemma 5. Assuming that E (ft) has a finite exponential moment with a: 2 0 being sufficiently large and
Th = 3 (5*) - (33)
(40)
the worst case probability is given as
(n(ffe)
exp
Pr
(41)
N
(.(se) a
j==1
exp
where 3* = 1 ar, a² 2 0 be the optimal o, and (B) = loo
En exp(
N
Proof. Taking the derivative of the Lagrangian for the optimization problem given in (15) with respect Pa leads
(x(f)) - a log
+ un =
(42)
where un is the Lagrangian multiplier for the constraint p 2 0 and a is the Lagrange multiplier for the DRO constraint with the size defined by th. Simplification of the above expression yields
Cn(
log Pr.
(43)
For some X' with p.. = X cxp ( (Cn(ft) a candidate solution
(n((s))
exp
P
(44)
(f,)))
exp
The above equation involves the expression in terms of the Lagrangian multiplier. By leveraging the sufficiency result presented in Chapter 8 Theorem of (Luenberger, 1997), can find the relationship between the multiplier and our constraint parameter 7/t- As such, our optimal solution be expressed in terms of original constraint. Suppose that we can find a* > 0 and p* E pARo such that p" maximizes (15) for a a" and Σ P" log = 7J with the optimal solution defined in (44). Considering this, we have the following
N Ph log = Σ P
N
C.(fe)
N Σ cxp
((.CA)) a
1)
- -log
8ar (8") 3133)) (45)
n=1 a
1
where we define 3" = o- and vB3 = log >N= exp (BC(f)). This allows us to express the Lagrangian multiplier using ne. Next, verify that there exists unique solution defined in (44) by
17

---

Published as a conference paper at ICLR 2023
leveraging the convexity of the exponential function. Specifically, substituting (44) in (15), get the following
N Σ
N
N
(ft(t)
a
Σ
Pn
log pm
= log Σ
exp
(46)
a
==1
1==1
If could show the following inequality holds true
(fee
N Σ
a
log
exp
E
(() -
Pn log Pr
(47)
then we can claim that the above candidate solution is the optimal solution. Rearranging the terms, we get the following
(fi)
N
(f
exp
2 exp
Pn logPn
(48)
a
This can be shown
N
(fu(fe)
[f((fe)
(n((f)
Σ 1=1
exp
Σ
Poln'
exp
Σ n=1
Pn exp
log Pn
n=1
Now applying Jensen inequality to the exponential function vb following
Ene
Σ Do(ee
we have the
(Cn(ft) lo
Σ Pu (fi
Pn exp
2 exp
Pn log Pr.
(49)
This completes the proof of the lemma.
Theorem 6. Suppose that CAREL holds the L-smoothness criteria with following inequality
A ALRLL (fa) L f21
(50)
Then choosing a learning rate size b the number of iterations required be performed by our algorithm for finding can be bounded by the following:
1(tt 355 with minibatch size b secondary minimbatch
approximate solution i.e., [[[[[[ L CAREL s c]
2 2
(1+11
1-P
T
(51)
pb'
Further the gradient complexity in terms of number of gradient steps is given as
N Ngrad b+ 21pL
2 (1+V
(1 (1+
1-P
(pb (1 p)b')
(52)
pb'
Before giving the formal proof, first show two lemmas that used during the proof.
Lemma 7. The L-smoothness condition given by Eg. (50), leads the following inequality
CAREL( < CAREL (OEEL(())++ CVC (AAREL(F)), (f),, f4) 11 - 1TP vft f R. (53)
where (a, af and || is the Euclidean norm
18

---

Published as a conference paper at ICLR 2023
Proof of Lemma 7. For the completeness the proof of the above Lemma is follow.
CAREL(P2
(AREL(S)) !
-CAREL(F') (F),
+ ff" ('IAAAAELL + + CCAREL (
Cauchy-Schward innuuuuat (u, v) S ||||||/||| leads to the following
CAREL(F))
S (AREL(F)) + (VCAREL(()), (2), -f)))
+ L/ I + - VCAREL (CAREL(F))III - fl1dr
Now lets use the L-smoothness assumption from Eq. (50), we have
(AREL(F))
/' LT 11
< CAREL (f') +
L
= CAREL(( + (VC (f', f() + - 11 - 81112
2
Now, we provide another important Lemma required to prove the above Theorem based on Lemma
Lemma 8. Considering I-sooothssss assumption in Eq. (50), and let s == Se 79t. Then for any Dr RM and y > we have the following
CAREL (fet1)
< <CARLL(F.) (ft) - 2 C 2
-(1
L
llfe11 - =foll + - 11.. - CAREL
2
2 2
2
(54)
19

---

Published as a conference paper at ICLR 2023
Proof of Lemma 8.
Let fit1 == fo -VCAREI((F) Then using L-smoothness of C CAREI,
have the following
(AREL(f(t1)
2 CAREL (ft) (FARET(ff)..ann f + L llfe+1 fell?
L f1lfe11 fell?
CAREL(F)) (VCAREL(()) 9t. fi+1 -ft) (gt, fet1 - -fi)
CAREL C (f() 9t. -791) - ( I.
7 2 11se-1- - fell
CARREL (ft) + + + (f() - 9 -V1V CAEL (fr) gt, AAELL (fi)
-G- llfet1 fell"
1 -AARET +3/1 (AE grll - seti,Je Se-1)
- |lfet1 fell
CAREL +000++/VV2AAB (II - 9ell - ||fet1 - fill?
L 2
27 (11f-1 - Fetll + llfe - F142112 - [1fe+1 - fa fz)
LAREL (ft) + CAREL (fi) - -9e112 ( - I1se+1 - Sell2
1 (1 -Jol- + 2
CAREL CARE)(J))13 - -115+1 - -1stt1-fell?
27
1 1 C (fa)12 (F))) - 27 - 4 [l,e+1 fell + E19. - CARLL
= CAREL(Fe)
This completes the Proof of Lemma 8. The last term is the variance and it can be bounded using the following lemma.
Lemma 9. Suppose that the smoothness assumption in Eq. (50) holds. If the gradient estimator ge+1 defined in Algorithm Line 13, then we have the following
(1-p:)L? (fo)
E[]|9(+1 - ARELL ( (1 - -P)) ALAR
Ilf - fell (55)
Proof of Lemma 9. According to Algorithm 2, we have the following
with probability Pt
ge41 =
91 + 1/ (.. (f))
with probability 1 Pe
(56)
20

---

Published as a conference paper at ICLR 2023
Using this the left hand side of the above lemma can be written
E Ell|g++1
CAREL (f++1)11
= =pill
Σ
an an((fet) CLARE (f++)]1?
nEB
+ (1 Pt)EE
119e + V
Σ
- - VCAREL (fe+1)1?
NEB'
1 Σ -Col)
= (E
1191
+
- VCAREL (fot2)l1
NEB
119. VLAREL ( 1/ (ua.
(1 P.)
NEB
+ (1 PE)
CAREL (S(t)) + V (1))11
= (1 -PPLE
Σ
-VCAEEL VCRELL (fet1) + (F)))
+ (1 Pe)ll9 (CAIE
1 1-Pt Pt
L
V(CL) a (L)
8/2
Pt E [[VLANEL (S+11) VCAREL
+ (1-P:))les VCAREL
1/2
L1)
(AAEEE(((H)))) CAREL(F.)1 + +(1 (1-Polle: (LAREL((F)))
Using the L-smoothness assumption in Eq. (50), we have
(1 (1-p
Ell99+11 AARREI (f+t)113
1f+11
-fll fell? +(1++111111111
(1- - -:))ll AREE (fe)ll
Proof Theorem 6. We leverage the above lemmas to prove the Theorem. Adding Eq. (54) with 7. x Eq. (55) and taking expectation results in the following:
2p
E
CAREL (fe+1) TAREL + 1199-11 VL
(fe+a113
2p
<
E
C CREEL(f))
CAREL
7 (ICLLLL (f))l
(1 I
1l. - fell?
2
22 2
+
2
E
[llge
AARELL (f)))
+
2
[[1
pp)ll9
ALRELL (fo)l22
2
2p
[1
L2
I1 -fal?]
2p
1/
E
CAREL (ft))
LAREL
llge
(f))
2p
E
1
L
(1
122
llfe+1
fell?
2y 2
2pb'
where LAREL is the loss the optimal S Using the inequality of by
1/4
(1-phmL² 2pb
2 0. iee
7 5
(57)
L
( +
p pb
21

---

Published as a conference paper at ICLR 2023
write the following
Ell VLAREL (f++1)l1²)
S CAREL (ft) CAREL
211900 VOCAAAE (f))l
7 C (fa)l
+ 2p
2
Now let us define Φi = CAREL(J.) CAREL following
+ 2p llst
V
CLAREL (f)) then we can write the
E[ $ Elod]
E (f+)
(58)
2
Now summing from t to results the following
T-1
[[or] < [[oo]
Σ I=0
AREL (f:)1l
(59)
According to the Algorithm 2, fr is chosen from (feher and Φu = CAREL CAREL
2p I1:00 VCAREL (Jo)lI² = CAREL((J)) LAREL Do. we have
20o
ARREL (fr
(60)
Setting T= 2Aa and using Jensen's inequality results in the following 2"
20.
E (fr)1l S E ARELL (fr)PP)
(61)
1
7T
With the following total number of iterations
2A 2
2D0L
11 (
T-P P pb'
T
(62)
can obtain e-approximate stationary point solution. The number of gradient steps required in the Algorithm 2 is given as
Ngrad = b+
+ ((ppp(pppp)))))))1 (1-p)6)
(63)
Replacing by Equation (62), we have the following
2.
(1 V
(pb + (1-+(1)p)) p)b') (+ pb'
1-P
N Ngrad= b+
(64)
2
This proves Theorem 6.
D ADDITIONAL EXPERIMENTAL DETAILS
D. DATASET DESCRIPTIOM
MNIST. In this dataset, classes corresponding to digits '1', '3', '5', '7', and '9' are treated as closed set classes and the rest as the open set. As the number of data samples per class not exactly the same, we first sample 5,000 samples per class in the training set. For testing, we sample 1,000 samples per class. To make the dataset imbalanced, we consider class '3' as minority class and randomly select 30% data samples compared with other majority classes. The same imbalanced ratio is applied to both training and testing sets. Table 3 shows the number of data samples from both the minority class and majority classes. In addition the MNIST open set classes described above, we follow other existing works (Sun et al., 2020) and further test the OSD performance on additional open samples from three more sources: (1) MNIST-Noise, (2) Noise, and (3) Omnigolot (Lake et al., 2015). More specifically, MNIST-Noise set testing samples, Noise consists of random noises, and Omnigolot consists of data samples from the Omnigolot dataset. For those classes, we select the same data samples as those of the closed set samples.
constructed by adding random noises to the closed
22

---

Published as a conference paper at ICLR 2023
Algorithm 2: Alternative Optimization between f and p using Probabilistic SGD
Initialize: fo. stepsize y, minibatch sizes b, < b. pt [0, t= pa (fo)) = 1An E [1, compute 90 = + Enen On (fo) Cn (fo) (fo) = b * p. with B, B' being random
minibatch samples with B and B
while <<T do
Se+1 6- ft 79th prev_use Ber(pe)
preu use = then
Find loss (f) associated with datasample Xn, Vn E B
Find an (f = b
ΣE exp
Find g++1 = EnEB On ((+11) V (
10
else
Find loss (f) associated with data sample X Vn B'
exp
Find an (St++)) = b1
E exp
Find g++1 = + by EnEo (a VCn((e)) (o.
13
14
1+t+1
15 return ir chosen from (fr)
Table Number closed samples in different datasets
Split
Cifar100
MNIST
ImageNet
AHED
Majority
Minority
Majority
Minority
Majority
Minority
Majority
Minority
Majority
Minority
Train
20000
1500
22500
800
20000
1500
2960
222
5071
216
Test
4000
4500
150
300
744
1271
Cifar10. In the Cifarl0 dataset, classes 'Airplane', 'Automobile', 'Bird', 'Ship', and 'Truck' are considered as the closed set classes and 'Cat', 'Deer', 'Dog', 'Frog', and 'Horse' as open set classes Furthermore, the class 'Bird' is made as the minority class using the same strategy introduced above. In addition the open set classes from Cifar10 itself, we further assess the OSD performance with Cifar+ 10 and Cifar+50. In particular, Cifar+ 10 includes data samples from the randomly selected 10 classes of the 10 dataset and C0 includes data samples from 50 randomly selected classes of the Cifar100 dataset.
Cifarl00. In the Cifar100 dataset, 'living being' related super classes are regarded as the closed set classes and the remaining 'non-living being' related super classes are regarded as the open set classes. We make 'insect' related classes as the minority one.
ImageNet. In the ImageNet dataset, we performed experimentation by randomly picking 5 classes known classes and five classes unknown classes. Specifically classes 'ant', "king crab', 'lion', 'French bulldog', 'great white shark' are treated as known classes whereas, classes 'iPod', 'lipstick', 'street sign', 'bookshop', and 'miniskirt' as unknown classes. We make 'king crab' as the minority class in this dataset.
Architectural Heritage Elements Dataset (AHED). In this dataset, we pick classes 'bell tower', 'portal', 'gargoyle', 'dome'. and 'column' as open set classes whereas, classes 'apse", 'vault', 'altar', "stained glass', and 'flying buttress' as unknown classes. This is inherently highly imbalanced dataset where number of data points are unevenly distributed across different classes. The class 'portal' is the minority class in this dataset.
D.2 DETAILED EXPERIMENTAL SETTING
For all datasets, we use LeNet5 (Lecun et al., 1998) as the network architecture with Tanh activation in the CNN layers and SoftPlus activation in the fully connected layers. Training involves the Adam optimizer with a learning rate of 0.001 and /2 regularization coefficient of 0.001. We initialize the uncertainty set λ 100 for all datasets that model gives the equal emphasis the all
23

---

Published as a conference paper at ICLR 2023
Table 4: Ablation Study on OSD Results for the Different Datasets.
Cosine
Exponential
Cifar10
Cifar100
ImageNet
Cifarl0
Cifar+10
Cifar+50
70.43 + 0.48
29.41 1.86
69.71 2.32
55.61 = 1.85
60.68 + 0.79
61.62 + 0.26
25.17 + 1.22
60.94 = 1.19
53.22 = 0.70
59.59 + 1.48
72.48 + 4.08
37.14 + 2.06
73.87 1.42
57.52 1.60
62.02 + 1.11
Cosine
Exponential
MNIST
AHED
MNIST
Noise
MNIST-Noise
Omnigolot 85.72 = 6.55
80.82 = 10.49
88.81 + 6.42
84.01 = 4.30
52.18 + 1.18
87.38 + 2.96
89.85 + 7.47
85.29 6.26
82.44 5.88
53.08 + 1.43
90.80 + 0.06
94.25
1 0.32
94.18 0.21
93.80 0.20
53.21
data samples. In terms of MSF parameters (w and w') for each model in P. we initialize them by uniformly sampling from (0, 1]. Next, we follow the training procedure shown in Algorithm I with random parameter selection in the exploration phase.
D.3 ABLATION STUDY ON SCHEDULER FUNCTIONS
Table compares the OSD performance between the MSF obtained through the proposed multi- scheduler learning strategy with prefixed atomic scheduler function, including cosine and exponen- tial. Due to the lack of expressiveness, single atomic SF function usually cannot work well for all datasets. For example, in Cifarl0, cosine yields a relatively better OSD compared exponential, whereas in the case of MNIST, exponential produces a relatively higher OSD performance. In contrast, by combining both of them and properly balancing their contribution based upon the nature of the dataset, MSF achieves the best performance in all cases.
Figure below shows the resulting MSF function output for the Cifarl0 and Cifar100 datasets that exhibit quite different learning behaviors. For Cifar10, the model can quickly learn from the classes due relatively easy data samples. As the MSF function decreases quickly, the resulting ?) value in AREO increases quickly making the model focusing mostly on the difficult samples. In case of Cifar (far100) 100, because its difficult nature, the model takes more time to learn well from all samples. Only in the latter phase of training, the model starts to put more emphasis the difficult samples by increasing the 7) value in the AREO.
1
04
6.2
69
30
Number Epoch
Number of Epoch
(a) Cifarl0
(b) Cifar100
Figure 3: MSF Function output Visualization for Multiple Datasets
D.4 PERFORMANCE ON MULTIPLE IMBALANCED CLASSES
To demonstrate the capability of handling the multiple minority classes with severe imabalanced condition, we conduct an experiments with multiple minority classes under two different settings discussed below
Same Cardinality: this first setting we demonstrate the ability our technique handling multiple imbalanced classes. In this case, consider two minority classes with e₁ 6% and c2 6%. In each step, we randomly select minority classes and repeat the experimentation two times and take an average over those runs to get the final score. Table 5 shows the performance for different baselines along with our proposed AREO. As shown, our technique has a far better performance terms of OSD compared the existing baselines.
24

---

Published as a conference paper at ICLR 2023
Table 5: OSD (MAP) performance on Multiple Imbalanced Classes with Same Cardinality
Approach
Cifar10
MNIST
Cifarl0
Cilar+ 10
Cifar+50
MNIST
Noisc
MNIST-Nois
Omnigolot
EDL
65.04
30.61
67.71
85.50
90.42
86.15
90.26
AEDL
58.12
23.20
58.70
87.57
94.96
89.26
90.67
DRO
53.40
22.19
58.62
60.89
29.04
37.88
33.57
OLTR 59.06
22.06
56.85
84.24
91.65
80.26
53.68
AREO
71.38
31.21
68.78
90.16
98.15
93.45
95.18
Different Cardinality with Severe Imbalance: In this setting, we demonstrate the ability of our approach on the multiple imbalanced classes with different cardinality. this case, we consider two minority classes with e₁ = 6% and 3%. Similar to first case, we randomly select minority classes and repeat the experimentation two times and take an average over those runs to get the final score. Further, this also demonstrates the ability of our technique handling severe type of the imbalanced scenario where 02 = 3%. Table shows the performance comparison across different techniques. As shown, AREO has better performance compared to the existing baselines.
Table 6: OSD (MAP) performance on Multiple Imbalanced Classes with Different Cardinality
Approach
Cifar10
MNIST
Cifar10
Cifar+ 10
Cifar+50
MNIST
Noise
MNIST-
Omnigolot
EDL.
68.46
33.05
69. 17
87.16
93.87
90.00
91.37
AEDL
59.60
23.85
62.44
87.98
92.45
89.26
89.91
DRO
57.38
22.72
61.82
67.72
80.32
66.59
71.74
OLTR 59.33
21.71
56.00
84.46
84.94
79.94
60.28
AREO
70.01
33.79
71.31
89.97
96.31
94.24
92.22
D.5 DETAILED DESCRIPTION OF BASELINES AND ADDITONAL COMPARION RESULT
D.5.1 DESCRIPTION OF BASELINES
CGDL uses variational autoencoder that can learn the class conditional posterior distribution for each class in the latent space (Sun al., 2020). Any sample with a low probability of belonging to any of the classes are regarded the outliers. CGDL is consistently outperformed by the proposed AREO model. One possible reason that CGDL may not learn the minority class posterior distribution properly in the latent space. Further, the approach may ignore the difficult samples when forming the posterior distribution. As such, the model is not able to diffcrentiate the minority class and OOD samples as both may have low probability of belonging to any of the classes resulting in the lower performance.
Posterior networks (Charpentier et al., 2020) leverage the normalizing flows to obtain the posterior distribution over the predicted probabilities. In this case, the latent representation is learned using the encoder and per class probability density value for that latent representation is determined to get the posterior distribution. In case of the minority class, the normalizing flow may not learn well to have high probability density value the sample. As such, while computing the uncertainty, may still assign to the low uncertainty to minority class, Further, the model may not learn well to produce the high density value for difficult samples from other classes. As such, it is likely the model may have confusion between hard samples and OOD samples resulting in difficulty in OSD. As shown in Table the posterior networks have consistently lower performance compared to AREO.
Kopetzki et al. (2021) propose a more robust form of the posterior networks by training the network using randomized smoothing (RS). The key idea is to draw multiple samples x ~ ~x(x', a) around the input sample x². Although this technique has shown improvement over the adversarial attack, it is not designed for the imbalanced situation as demonstrated by its lower OSD performance Table 1.
We have also included an approach called OLTR proposed by Liu al. (2019) baseline. This work has proposed a way to deal with the OSD in the imbalanced data distribution however it has several limitations. First, the approach is based across the minority and majority classes and thereby having robust learning even for the minority class. So, if the minority class is different from other majority classes, the model may not properly
the assumption that visual similarity is shared
25

---

Published as a conference paper at ICLR 2023
learn the minority classes and thereby the model may be confused between minority and open set samples. Also, the proposed OLTR does not have a mechanism to focus on the difficult samples from the majority classes while training. As such, the model may detect difficult samples as open set samples. Those limitations are reflected in the performance shown in Table 1. Recently, Wang et al. (2022) propose a contrastive loss based approach where minority classes pushed from the OOD samples in the feature space. Although, this paper also considers the open set detection under class imbalanced-setting it heavily relies on the selection of open set samples involved during the training process. Table 7 shows the OSD performance with respect to open set training datasets Flower (Nilsback & Zisserman, 2006) and MIT Indoor Scene (Quattoni & Torralba, 2009). As shown, the performance using this method is highly dependent on the selection of the OOD dataset during training process.
Table 7: OSD (MAP) performance on (Wang et al., 2022)
Openset
Cifar10
Cifar100
Cifar
Cifar+10
Cifar+50
Flower
70.67 + 1.69
25.78 + 2.08
64.09 + 3.00
53.96 + 0.57
Indoor
72.94 0.52
31.70 + 3.35
69.65 3.71
55.74 0.35
D.5.2 ADDITIONAL COMPARISON RESULTS
In this section, we first provide detailed discussion on recently developed general open set recogni- tion methods, which do not specifically focus on imbalanced data. We then choose some representative methods and present a comparison with the proposed AREO model. Finally, we discuss some recent OSD models designed for few-shot learning under the meta-learning setting.
Chen et al. (2021) conduct Adversarial Reciprocal Point Learning (ARPL), where the adversarial point is generated for each known class in the embedding space by leveraging representations of other known class samples along with the unknown samples generated using an adversarial mechanism. During training, it maximizes the gap between the representations of known class samples and that of the adversarial point. Meanwhile, the model tries to push the unknown samples' representations into a specific region using the adversarial margin constraint. In order to achieve this, diverse and confusing training samples are generated through adversarial learning. Cevikalp et al. (2021) leverage the polyhedral conic function and define two losses. The first loss ensures a good separation among the known classes whereas the second loss achieves the compactness within each class so that the open set samples could be easily rejected. Dhamija et al. (2018) also leverage two losses, where the Entropic open set loss is to ensure that the softmax output for the open set samples are uniformly distributed to the all known classes and the Objectosphere loss aims to assign a higher feature magnitude to the known class samples in the embedding space than those from the open set classes. The proposed approach requires open set datasets to be available during the training set, which may limit its applicability in more general settings. In (Perera et al., 2020), self supervision is performed to construct the decision boundary between classes based on the semantics in the feature space. A generative model is then trained based on the known class samples. Thus, the generated images will be close to that of the closed set class samples. As the open set samples are not seen during the generative modeling process, the produced images will exhibit a high disparity with those of the closed set samples. Finally, Yang et al. (2020) propose the Convolutional Prototype Network (CPN) where a prototype for each known class is constructed in the feature space and two different loss functions are defined. To define the generative loss, generative assumption is followed where the class spasisse features are drawn from certain distributions (e.g., Gaussian) with the mean given by the prototype representation. This generative loss helps to reduce the intra-class variance and thereby making the known class sample representation very compact. Thus, the model can reserve more spaces for unknowns, making OSD relatively casier. The second loss (i.e., discriminative loss) encourages the separation of the class samples from different classes based on the distance between sample and prototype representations.
We perform comparison with the first three methods discussed above, including two most recent baselines with competitive OSD performance. We report the comparison results on Cifar 10, Cifar 100, and MNIST datasets in Table 8. As discussed earlier, all these methods are general open set recognition models and hence suffer from lower OSD performance for the more challenging setting
26

---

Published as a conference paper at ICLR 2023
that involves imbalanced data. The results further justify the effectiveness of the proposed AREL model.
Table 8: OSD (MAP) performance on Multiple Imbalanced Classes with Different Cardinality
Approach
Cifarl0
MNIST
Cifar
Char+50
MNNST
MNIST-Nobs
Ommigolxx
ARPL (Chen et al. (2021))
58.30
29.33
69.00
90.27
74.51
50.38
39.08
48.74
Agnostophobia (Dhamuja (2018))
03.30
20.11
68.30
85.03
78.43
72.72
75.70
52.93
DC ECPP (Covikalp (2021))
61.22
23.54
58.30
77.56
56.80
71.11
89.44
53.22
AREO
72.48
37.14
73.87
90.80
94.24
94.18
93.80
57.52
In addition to the above baselines, there are also recent OSD models specifically developed for few-shot learning (FSR) under the meta-leariing setting. For instance, Jeong et al. (2021) propose few-shot open-set recognition (FSOSR) model. This approach designed to work with testing tasks with limited labeled data through meta-learning. The design of the training paradigm is based episodic learning (Snell et al., 2017) widely used in meta-learning, where query and support sets are constructed by selecting subsets of the meta-training data. In contrast, for few-shot setting through meta-lcarning. Furthermore, the FSOSR approach in (Jeong et al., 2021) does not consider the imbalanced class distribution, either. Therefore, the problem setting, model training, and evaluation process are all different. Similarly, Liu et al. (2020) propose an oPen mEta LEaRning (PEELER) algorithm that adapts ProtoNet to FSOSR under the meta-learning setting. There are two key differences between our technique and PEELER. First, PEELER assumes that the unknown samples are also available during the training process. Second, the algorithm also designed under the meta-learning setting that makes the direct comparison with our approach infeasible.
model is not designed
Kong & Ramanan (2021) propose the OpenGAN model to discriminate the open set samples from close-set ones. There are some key differences from our work. First, OpenGAN introduces open set samples in the training well validation sets whereas our approach does not involve any open set samples in training or validation sets. Second, OpenGAN only works in the binary classification setting where the loss function is proposed to discriminate whether sample is in the open set closed set. In this case, the loss function does not perform closed set classification. In contrast, our approach achieves a state-of-the-arr OSD performance while ensuring decent closed set performance. Finally, OpenGAN does not have specific mechanism to handle the imbalanced setting, which is one primary design focus of our approach.
D.6 PERFORMANCE COMPARISON USING AUROC
In addition the MAP scores, the AUROC scores, which are reported in Table 9, also show consistent trend in terms of OSD performance.
Table 9: OSD (AUC) performance on Multiple Datasets
Approach
Cifar10
MNIST
Cifar100
Cifar10
Cifar+10
Cilar+50
MNIST
Noise
MNIST-Noie T-Nise
Omnigolot
EDL
66.36
77.66
69.07
88. 72
95.36
93.08
91.23
52.38
DRO
61.67
51.62
56.49
64.64
51.58
53.41
53.29
51.12
AEDI
51.38
63.31
59.89
76.36
81.35
83.41
75.51
50.62
OpenMAX
64.98
65.37
65.54
91.21
92.26
93.38
92.53
52.62
AREO
72.52
74.33
72.69
92.75
96.86
96.69
96.02
55.49
D.7
CLOSED SET PERFORMANCE
Table 10 below shows the closed set performance for competitive baselines. It is interesting to see that DRO with flexible uncertainty set performs the worst in the closed set setting does not learn properly from the most representative samples in the training data while only focusing on the difficult ones. AEDL performs very competitively and achieves the best performances on two datasets. This is partly because are evaluating MAP by treating the minority class as positive and oversampling helps to improve the prediction on the minority class quite significantly. AREO also performs competitively and achieves the best performance the other two datasets. The good closed set performance further confirms our theoretical result that proves the equivalence between AREO and AdaBoost, which justifies its strong generalization capability.
27

---

Published as a conference paper at ICLR 2023
Table 10: closed performance (MAP) all datasets
Approach
Cifar10
Cifar100
ImageNet
MNIST
AHED
EDL
55.39 + 3.78
31.80 + 2.37
55.84 = 1.60
99.58 + 0.26
40.48 + 2.65
AEDL
54.98 + 0.63
36.11 + 00008
55.62 = 0.58
99.62 = 0.23
41.36 +: 3.98
DRO
27.16
5.94
10.50
0.25
20.71 + 1.00
90.87
+ 3.89
30.02 0.93
AREO
54.65 + 1.02
36.44
E 0.23
55.31 = 1.11
99.88 + 0.01
49.68 1.58
L.000
3.000
3.8
0.39
8.000
0.94
8,007
0.4
0.80
1.006
orde
And Jun wit
Aurlin Taruahh
L
0-40
(a) EDL
(b) DRO
(c) AEDL
(d) AREO
Figure 4: OSD performance comparison from imbalanced Cifarl0 dataset.
D.8 SUPERIOR OSD PERFORMANCE OF AREO
Figure 4 provides a deeper insight on the superior OSD performance of AREO than other competitive baselines, including EDL., DRO, and AEDL. Cifar10 used as an illustrative example and similar patterns are obtained on other datasetts. First, while EDL is able to separate outliers from most samples in the majority classes based on their predicted uncertainty scores, it assigns much higher uncertainty scores to samples from the minority class, making them hard to be separated from the outliers. Second, the uncertainty scores for the majority classes span a wide range, which implies that several (difficult) samples from these classes have also been assigned very high uncertainty scores. If the goal to ensure that most top-ranked samples are true outliers for effective detection in practice, these highly uncertain close-set samples may significantly affect the detection effectiveness. Third, while oversampling can help to better detect the samples from the minority class, which is indicated by lower uncertainty scores achieved by AEDL, most majority classes become much more uncertain and some of them have even a higher average uncertainty score than the outliers. Furthermore, the uncertainty scores from most classes also span a wide range. Finally, DRO effectively narrows down the range of the uncertainty scores as it allows the model to focus more on the difficult samples. However, does not effectively bring down the high uncertainty scores of the minority class, either, which is still higher than outliers. Similar to DRO, the proposed AREO also manages to keep the uncertainty scores data samples from the majority classes low so that even the difficult samples are unlikely to be mis-identified as outliers. Mcanwhile, it effectively lowers the uncertainty scores of the minority-clan examples so that they can better separated from the outliers.
E
LINK TO SOURCE CODE
For the source code, please click here.
28


Reviews:
[{"review_id": "ZTIE_8Ov6Tb", "reviewer": "Reviewer_HU7D", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors address the problem of imbalanced data in open set detection (OSD) and proposed AERO (Adaptive Robust Evidential Optimization).  Based on evidential deep learning (EDL), their approach includes an uncertainty mass for the unknown class, which is used for detecting the unknown class.  Also, they use DRO (Distributionally Robust Optimization) to handle imbalanced data via learning weights for each instances to focus on harder instances.    They propose distributionally robust evidential loss (DREL), which is a combination of DRO and EDL.  They replace the loss function for each instance in DRO with the loss function from EDL.  To learn from easy to hard instances (hence not focusing only on hard instances), they propose MSF (mutli-scheduler function) to adaptively increase the size of uncertainty set.  MSF is a weighted sum of multiple scheduler functions, and the weights are learned.  Adaptive Robust Evidential Loss (AREL) adds an adaptive size of the uncertainty set $\\eta$ according to MSF to DREL.   To increase the chance of minority instances to be included in the uncertainty set (in additional to hard majority instances), they increase the weights for the minority instances.  During training, they alternate between AREL loss and MSF loss.  In their theoretical analysis, they link their approach to AdaBoost.\n\nThey compare their proposed AREO with 8 existing algorithms and 5 datasets.  Their results indicate AREO compares favorably.\n", "summary_of_the_review": "The paper addresses an interesting problem in OSD with imbalanced data.  The authors proposed combining existing EDL (for OSD) and DRO (for imbalanced data) with MSF (for adaptively increasing the uncertainty set).", "strength_and_weaknesses": "Strengths:\n\n1.  The problem of OSD with imbalanced data is interesting\n\n2.  The empirical results indicated the proposed AREO compares favorably with 8 algorithms and 5 datasets.\n\nWeaknesses:\n\n1.  The proposed AREO is a combination of  the existing approaches DRO and EDL, with MSF (multi-scheduler function), which also is not new.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SUG1KcYq1MS", "reviewer": "Reviewer_c1ud", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper enhances evidential deep learning (EDL) for open set detection by incorporating distributionally robust optimization with a scheduler function to boost hard samples to predict while at the same time enhancing the boosting for minority classes so they can be learned.  The new method is termed  adaptive robust evidential optimization (AERO), and experimental results clearly demonstrate the utility of the method. ", "summary_of_the_review": "The paper is very interesting with impressive results for open set detection.  The description of the method neglects a critical regularization term used in EDL, which is one of key component the proposed AERO method builds upon.  Without proper discussion of this term and how it is incorporated into AERO, the results in this paper are suspect. \n\nThe authors have addressed my major concerns, I can increase my rating to over the acceptance threshold to 6.  This paper has a lot of merits, and I would have no problem if it appears in the program.  The use of EDL, DRO, and MSF has novelty, but the integration of all three methods does not seem to require any new processing ideas or theory that would bring this paper to a definite accept.", "strength_and_weaknesses": "The strength of the paper is the that the method appears novel, and the experimental results are impressive.  Furthermore, the paper includes some theoretical linkage to boosting (which this reviewer does not have time to verify).  The weakness of the paper is that the Kullback-Leibler regularization term in EDL seems to be ignored, which makes the results questionable. Also, the composition of the scheduler function as a weighted sum of convex and concave functions is a bit ad-hoc.  The main concern is that the convex functions are parameterized by beta, but this does not seem to be the case for the concave functions.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RLDehtKwvd6", "reviewer": "Reviewer_gFy7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an Adaptive Robust Evidential Optimization (AREO) technique to tackle sample uncertainty by evidential learning. The main emphasis in the paper is evidential learning in the presence of minority classes. The paper starts with motivating the need for a tradeoff between distributive robust optimization (which deals with difficult samples in the training set) vs. oversampling (which deals with minority classes). An adaptive evidential learning strategy is proposed that gradually increases the size of the uncertainty set by a composite scheduler function. This function optimizes the weight to be assigned to each training sample at every epoch depending on whether they are from minority classes or majority ones. The optimization involves jointly solving for parameters of the model and the composite scheduler function. A bilinear optimization based on population-based training is proposed. Additional valuable insight about the connection between AREA and AdaBoost is provided under certain assumptions.   \n", "summary_of_the_review": "The paper studies an important aspect of open-set classification often overlooked in the literature. The proposed idea of using composite scheduler function for adaptive evidential learning is quite intriguing. However, there were some concerns about whether the improvement comes from training P different models at the same time or by optimizing the scheduler function that adjusts the size of the uncertainty set. An ablation study would have been useful to disentangle these two components. It would also be very useful if the weights assigned to different samples during training and evidence computed for open-set samples during testing could have been demonstrated in a simple illustrative set-up.\n\nQuestions: \n\n1. It was not also immediately clear what the significance of optimizing \\beta would be. SF_m functions can be prefixed,  perhaps a large number of them with different \\beta as it is commonly done in composite kernel learning techniques[*]\n\n*Fung, G., Dundar, M., Bi, J., & Rao, B. (2004, July). A fast iterative algorithm for fisher discriminant using heterogeneous kernels. In Proceedings of the twenty-first international conference on Machine learning (p. 40).\n \n2. Training uses the Adam optimizer with a fixed learning rate of 0.001. Neural nets tend to  learn from easy samples (dominant patterns) in earlier epochs and from difficult samples (rare patterns) in later stages. However the fact that no learning rate scheduler is used may not allow enough time for the network to learn from difficult samples, because training may stop prematurely because of the high learning rate without fully exploring the weight space. When comparing the proposed approach against state-of-the-art it may be more convincing if the AREO is also compared against an evidential network where the learning rate is gradually decreased. \n\n\n\n", "strength_and_weaknesses": "\nStrengths:\n\nEvidential learning in the presence of minority classes while acknowledging the potential presence of difficult samples in majority classes is a real-world problem that is often overlooked in most open-set classification and evidential learning papers. On that regard this paper studies a highly significant problem and proposes a scheduling algorithm that adjust uncertainty set size during network training.\n\nThe connection between AdaBoost and AREO is quite motivating.   \n\nIt is good to see that the improvement in performance holds even when different number of minority classes with different sizes are included in experiments. Qualitative analysis ranking sample difficulty by different techniques was also very useful.\n\nWeaknesses:\n\nIt is not immediately clear whether the improvement is due to population-based training (PBT) or weighting samples by the proposed composite scheduler. An ablation study would be very useful.  This is a concern because none of the compared techniques seem to be using PBT. \n\nIt is also concerning that no run-time comparison is provided between proposed technique and others. PBT would significantly increase run-time and the limited improvement in some datasets may not quite justify this significant increase in run-time.\n\n\n-----------------\nAuthors' responses satisfactorily address my main concerns about PBT and run time. If there was a score of 7 I would have upgraded my ranking, regrettably there is none, and I don't think this is an \"8\" paper.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IZDnZMJFAY", "reviewer": "Reviewer_FxMN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors focus on open set recognition (detection) and propose a method when the known classes are imbalanced. The proposed method is built based on evidential learning and distributionally robust optimization. To this end, the authors first integrate the evidential learning loss function to distributionally robust optimization and introduce distributionally robust evidential optimization. Then Adaptive Robust Evidential Optimization method is proposed by utilizing multi-schedular function. The main idea is to introduce an optimal training behavior that gives sufficient attention to the difficult samples and minority class and at the same to detect the unknown class samples with high accuracy. The authors show the connection between their proposed method and AdaBoost and compare their method to the related methods. Better accuracies are reported against the rival tested methods.", "summary_of_the_review": "In general this is a good paper. The proposed method is built based on existing methods, yet it has some novelty. Experimental results support that the proposed method outperforms related methods. However, some important references on open set recognition are missing, the paper also lacks comparison to these methods as well. Also, in addition to mAP scores, AUC scores should be reported as well. ", "strength_and_weaknesses": "The main strengths of the paper can be summarized as follows:\ni) Despite some Grammar mistakes, the paper is mainly well written.\nii) Although the authors build their proposed method based on the known methods, the resulting method seems effective. Especially, integrating evidential optimization loss in distributionally robust optimization makes sense. \niii) Showing the theoretical connections between the proposed method and AdaBoost is a plus.\niv) The proposed method outperforms other tested methods.\nThe main weaknesses of the paper can be summarized as follows:\ni) There are some missing references related to general open set recognition methods. Especially there are recent successful open set recognition methods that estimate compact class acceptance regions for open set recognition. These methods must be also discussed in the paper and used for experimental comparison.\nii) The authors use mAP scores for assessing accuracy. However, most of the papers reported Area Under the ROC curve (AUC) scores since this is the common setting used for measuring the detection of performance of the unknown samples.  Therefore, please report AUC scores as well.\n\nMinor Issues:\nThe authors use the term “close set”, but the correct term is the closed set. Please correct it. Also, there are some minor Grammar mistakes that must be corrected (e.g., have been proposed – page , at the beginning of Section 2).\n\nReferences:\n[1] G. Chen, P. Peng, X. Wang, Y. Tian, Adversarial reciprocal points learning for open set recognition, in: arXiv:2103.00953, 2021.\n[2] P. Perera, V. I. Morariu, R. Jain, V. Manjunatha, C.Wigington, V. Ordonez, V. M.  Patel, Generative-discriminative feature representations for open-set recognition, in: CVPR, 2020.\n[3] H.-M. Yang, X.-Y. Zhang, F. Yin, Q. Yang, C.-L. Liu, Convolutional prototype network for open set recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence (2020) 1–1 doi:10.1109/TPAMI.2020.3045079.\n[4] H. Cevikalp, B. Uzun, O. Kopuklu, G. Ozturk, Deep compact polyhedral conic classifier for open and closed set recognition, Pattern Recognition 119 (2021).\n[5] A. R. Dhamija, M. Gunther, T. E. Boul, Reducing network agnostophobia, in: Neural Information Processing Systems (NeurIPS), 2018.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster