Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors address the problem of imbalanced data in open set detection (OSD) and proposed AERO (Adaptive Robust Evidential Optimization).  Based on evidential deep learning (EDL), their approach includes an uncertainty mass for the unknown class, which is used for detecting the unknown class.  Also, they use DRO (Distributionally Robust Optimization) to handle imbalanced data via learning weights for each instances to focus on harder instances.    They propose distributionally robust evidential loss (DREL), which is a combination of DRO and EDL.  They replace the loss function for each instance in DRO with the loss function from EDL.  To learn from easy to hard instances (hence not focusing only on hard instances), they propose MSF (mutli-scheduler function) to adaptively increase the size of uncertainty set.  MSF is a weighted sum of multiple scheduler functions, and the weights are learned.  Adaptive Robust Evidential Loss (AREL) adds an adaptive size of the uncertainty set $\eta$ according to MSF to DREL.   To increase the chance of minority instances to be included in the uncertainty set (in additional to hard majority instances), they increase the weights for the minority instances.  During training, they alternate between AREL loss and MSF loss.  In their theoretical analysis, they link their approach to AdaBoost.

They compare their proposed AREO with 8 existing algorithms and 5 datasets.  Their results indicate AREO compares favorably.

strength_and_weaknesses: Strengths:

1.  The problem of OSD with imbalanced data is interesting

2.  The empirical results indicated the proposed AREO compares favorably with 8 algorithms and 5 datasets.

Weaknesses:

1.  The proposed AREO is a combination of  the existing approaches DRO and EDL, with MSF (multi-scheduler function), which also is not new.

clarity,_quality,_novelty_and_reproducibility: The paper is generally well written.

Below Eq.6: $\eta$ could be defined.  The uncertainty set seems to be central to the approach, a clear description of it would be helpful.

The novelty is not high because the proposed AREO is a combination of existing approaches.
summary_of_the_review: The paper addresses an interesting problem in OSD with imbalanced data.  The authors proposed combining existing EDL (for OSD) and DRO (for imbalanced data) with MSF (for adaptively increasing the uncertainty set).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors focus on open set recognition (detection) and propose a method when the known classes are imbalanced. The proposed method is built based on evidential learning and distributionally robust optimization. To this end, the authors first integrate the evidential learning loss function to distributionally robust optimization and introduce distributionally robust evidential optimization. Then Adaptive Robust Evidential Optimization method is proposed by utilizing multi-schedular function. The main idea is to introduce an optimal training behavior that gives sufficient attention to the difficult samples and minority class and at the same to detect the unknown class samples with high accuracy. The authors show the connection between their proposed method and AdaBoost and compare their method to the related methods. Better accuracies are reported against the rival tested methods.
strength_and_weaknesses: The main strengths of the paper can be summarized as follows:
i) Despite some Grammar mistakes, the paper is mainly well written.
ii) Although the authors build their proposed method based on the known methods, the resulting method seems effective. Especially, integrating evidential optimization loss in distributionally robust optimization makes sense. 
iii) Showing the theoretical connections between the proposed method and AdaBoost is a plus.
iv) The proposed method outperforms other tested methods.
The main weaknesses of the paper can be summarized as follows:
i) There are some missing references related to general open set recognition methods. Especially there are recent successful open set recognition methods that estimate compact class acceptance regions for open set recognition. These methods must be also discussed in the paper and used for experimental comparison.
ii) The authors use mAP scores for assessing accuracy. However, most of the papers reported Area Under the ROC curve (AUC) scores since this is the common setting used for measuring the detection of performance of the unknown samples.  Therefore, please report AUC scores as well.

Minor Issues:
The authors use the term “close set”, but the correct term is the closed set. Please correct it. Also, there are some minor Grammar mistakes that must be corrected (e.g., have been proposed – page , at the beginning of Section 2).

References:
[1] G. Chen, P. Peng, X. Wang, Y. Tian, Adversarial reciprocal points learning for open set recognition, in: arXiv:2103.00953, 2021.
[2] P. Perera, V. I. Morariu, R. Jain, V. Manjunatha, C.Wigington, V. Ordonez, V. M.  Patel, Generative-discriminative feature representations for open-set recognition, in: CVPR, 2020.
[3] H.-M. Yang, X.-Y. Zhang, F. Yin, Q. Yang, C.-L. Liu, Convolutional prototype network for open set recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence (2020) 1–1 doi:10.1109/TPAMI.2020.3045079.
[4] H. Cevikalp, B. Uzun, O. Kopuklu, G. Ozturk, Deep compact polyhedral conic classifier for open and closed set recognition, Pattern Recognition 119 (2021).
[5] A. R. Dhamija, M. Gunther, T. E. Boul, Reducing network agnostophobia, in: Neural Information Processing Systems (NeurIPS), 2018.


clarity,_quality,_novelty_and_reproducibility: In general, the paper is written well and the proposed method makes sense and it has some novelty. Experimental studies support the main claims regarding the proposed method. The authors provide algorithm for the proposed methods, but the source codes are needed to reproduce the experimental results.
summary_of_the_review: In general this is a good paper. The proposed method is built based on existing methods, yet it has some novelty. Experimental results support that the proposed method outperforms related methods. However, some important references on open set recognition are missing, the paper also lacks comparison to these methods as well. Also, in addition to mAP scores, AUC scores should be reported as well.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an Adaptive Robust Evidential Optimization (AREO) technique to tackle sample uncertainty by evidential learning. The main emphasis in the paper is evidential learning in the presence of minority classes. The paper starts with motivating the need for a tradeoff between distributive robust optimization (which deals with difficult samples in the training set) vs. oversampling (which deals with minority classes). An adaptive evidential learning strategy is proposed that gradually increases the size of the uncertainty set by a composite scheduler function. This function optimizes the weight to be assigned to each training sample at every epoch depending on whether they are from minority classes or majority ones. The optimization involves jointly solving for parameters of the model and the composite scheduler function. A bilinear optimization based on population-based training is proposed. Additional valuable insight about the connection between AREA and AdaBoost is provided under certain assumptions.   

strength_and_weaknesses: 
Strengths:

Evidential learning in the presence of minority classes while acknowledging the potential presence of difficult samples in majority classes is a real-world problem that is often overlooked in most open-set classification and evidential learning papers. On that regard this paper studies a highly significant problem and proposes a scheduling algorithm that adjust uncertainty set size during network training.

The connection between AdaBoost and AREO is quite motivating.   

It is good to see that the improvement in performance holds even when different number of minority classes with different sizes are included in experiments. Qualitative analysis ranking sample difficulty by different techniques was also very useful.

Weaknesses:

It is not immediately clear whether the improvement is due to population-based training (PBT) or weighting samples by the proposed composite scheduler. An ablation study would be very useful.  This is a concern because none of the compared techniques seem to be using PBT. 

It is also concerning that no run-time comparison is provided between proposed technique and others. PBT would significantly increase run-time and the limited improvement in some datasets may not quite justify this significant increase in run-time.


-----------------
Authors' responses satisfactorily address my main concerns about PBT and run time. If there was a score of 7 I would have upgraded my ranking, regrettably there is none, and I don't think this is an "8" paper.  

clarity,_quality,_novelty_and_reproducibility: Clarity: 
The paper is organized well and reads smoothly. However, there were many typos some of which are listed here. 
Please proofread the entire paper including the appendix.

Various SVM based techniques  have proposed for OSD --> please correct as "have been proposed"

The grammar in the following sentence is not right. Not sure what the subject of the second part of the setence is starting with "while remain" or (remaining).  

… thei predicted uncertainty scores. Should be "their".

This process is known is exploitation … Should be "as exploitation"

It was not clear how (w and \beta) were different than (w' and \beta'). Not mentioned in the text. 

Quality/Novelty:

Composite scheduler idea and the way to jointly optimize model parameters and  hyperparameters of the composite scheduler by bilinear optimization and population based training seems to be novel.  The connection between Adaboost and AREO is also interesting.

Reproducibility:

Link to code is provided. However, it is hard to judge whether the results are reproducible without running the code given some key parameters are not discussed (such as P).



summary_of_the_review: The paper studies an important aspect of open-set classification often overlooked in the literature. The proposed idea of using composite scheduler function for adaptive evidential learning is quite intriguing. However, there were some concerns about whether the improvement comes from training P different models at the same time or by optimizing the scheduler function that adjusts the size of the uncertainty set. An ablation study would have been useful to disentangle these two components. It would also be very useful if the weights assigned to different samples during training and evidence computed for open-set samples during testing could have been demonstrated in a simple illustrative set-up.

Questions: 

1. It was not also immediately clear what the significance of optimizing \beta would be. SF_m functions can be prefixed,  perhaps a large number of them with different \beta as it is commonly done in composite kernel learning techniques[*]

*Fung, G., Dundar, M., Bi, J., & Rao, B. (2004, July). A fast iterative algorithm for fisher discriminant using heterogeneous kernels. In Proceedings of the twenty-first international conference on Machine learning (p. 40).
 
2. Training uses the Adam optimizer with a fixed learning rate of 0.001. Neural nets tend to  learn from easy samples (dominant patterns) in earlier epochs and from difficult samples (rare patterns) in later stages. However the fact that no learning rate scheduler is used may not allow enough time for the network to learn from difficult samples, because training may stop prematurely because of the high learning rate without fully exploring the weight space. When comparing the proposed approach against state-of-the-art it may be more convincing if the AREO is also compared against an evidential network where the learning rate is gradually decreased.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper enhances evidential deep learning (EDL) for open set detection by incorporating distributionally robust optimization with a scheduler function to boost hard samples to predict while at the same time enhancing the boosting for minority classes so they can be learned.  The new method is termed  adaptive robust evidential optimization (AERO), and experimental results clearly demonstrate the utility of the method. 
strength_and_weaknesses: The strength of the paper is the that the method appears novel, and the experimental results are impressive.  Furthermore, the paper includes some theoretical linkage to boosting (which this reviewer does not have time to verify).  The weakness of the paper is that the Kullback-Leibler regularization term in EDL seems to be ignored, which makes the results questionable. Also, the composition of the scheduler function as a weighted sum of convex and concave functions is a bit ad-hoc.  The main concern is that the convex functions are parameterized by beta, but this does not seem to be the case for the concave functions.
clarity,_quality,_novelty_and_reproducibility: The paper is well written.  The approach seems novel and for the most part makes sense despite what could be a fatal flaw. The experimental results are impressive, but this reviewer really does not have a good grasp of the MAP metric in this context.  The AUROC is much more intuitive for this reviewer. 

In any event, the concern is that the EDL loss function is only composed of the sum of square loss in (3) and is missing the KL term that was used in the EDL paper.  Without the KL term as shown in the original EDL paper, the evidence would go to infinity to minimize the sum of the square loss, meaning that EDL has no way to estimate uncertainty.  The KL regularization term anchors the uncertainty by decreasing evidence when the test samples cannot be predicted well. The problem is that the KL term must be properly annealed so that in can initially learn to discriminate before learning how to calibrate the evidence. If the lack of inclusion of the KL term in (3) was a simple reporting error, the paper still would need to explain how the annealing of its weigh parameter lambda is balanced in relation to opening up the DRO set via eta from epoch to epoch.

On page 4, the statement - "...it provides a low uncertainty mass for minority/difficult samples while remain high (in terms of uncertainty mass) for those open set samples." seem to conflict with an earlier statement that minority/difficult samples are assigned high uncertainty.
summary_of_the_review: The paper is very interesting with impressive results for open set detection.  The description of the method neglects a critical regularization term used in EDL, which is one of key component the proposed AERO method builds upon.  Without proper discussion of this term and how it is incorporated into AERO, the results in this paper are suspect. 

The authors have addressed my major concerns, I can increase my rating to over the acceptance threshold to 6.  This paper has a lot of merits, and I would have no problem if it appears in the program.  The use of EDL, DRO, and MSF has novelty, but the integration of all three methods does not seem to require any new processing ideas or theory that would bring this paper to a definite accept.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: **Summary:** the paper focuses on open set recognition (detection) and proposes a method when the known classes are imbalanced. The proposed method is built based on evidential learning and distributionally robust optimization. To this end, the authors first integrate the evidential learning loss function to distributionally robust optimization and introduce distributionally robust evidential optimization. Then Adaptive Robust Evidential Optimization method is proposed by utilizing multi-schedular function. The main idea is to introduce an optimal training behavior that gives sufficient attention to the difficult samples and minority class and at the same to detect the unknown class samples with high accuracy. The authors show the connection between their proposed method and AdaBoost and compare their method to the related methods. Better accuracies are reported against the rival tested methods.

**Strengths**: well written; although built on previous methods, the combination is effective. some theoretical connections with AdaBoost. Good performance compared with other tested methods.

**Weaknesses**: The following weaknesses were identified by reviewers:

1. incremental novelty, combination of existing approaches (DRO, EDL & MSF). [HU7D]
2. missing refs in related works and experiments. [FxMN]
3. use AUC for evaluation instead of mAP. [FxMN, c1ud]
4. missing ablation study on PBT or weighted sampling of the scheduler [gFy7]
5. missing run-time comparison [gFy7]
6. significance of optimizing beta in the scheduler? [gFy7]
7. also compare against learning rate decay [gFy7]
8. KL divergence term in EDL is missing [c1ud]
9. how to set the parameter lambda in EDL [c1ud]
10. adhoc nature of the composite scheduler, beta does not apply to concave functions? [c1ud]
11. various minor typos

**Discussion:**
The authors wrote a response to address these concerns. During the discussion, all reviewers were satisfied with the response. Reviewer FxMN noted that: 
  - However, I have some doubts regarding the latest results comparing their proposed method to recent SOTA open set recognition methods. Although SOTA open set recognition methods do not have any mechanism for handling imbalanced datasets, I would not expect such bad accuracies since only one class is imbalanced and it still has 30% of original sample size. Therefore, the authors should double check these results. 
  - There are still minor typos that must be corrected (e.g., although the authors replaced the close set with closed set in some places, there are still many close set terms in the paper). 

In the end, the final rating was 6666. Two reviewers would have liked to give a "7" if possible, thinking that the paper is not strong enough for "8".   The paper can be published after further minor revision according to the discussion and feedback.
justification_for_why_not_higher_score: The scope is quite focused (imbalanced data in open set detection).
justification_for_why_not_lower_score: All reviewers found the topic interesting.
Although the paper has incremental novelty (combining some methods), the authors justify their approach and make clear their contributions of their proposed AREO and theoretical analysis.
Decision: Accept: poster