Kati8/ICLR_2023_2025 huggingface dataset:
authors: Lydia Fischer
authors: Patricia Wollstadt
reviews: 
Comments: This submission focuses on the classifcation scenario with rejection option where accuracy-reject curve is usually used to evaluate and compare the performance of different certainty measures over a range of thresholds for accepting or rejecting classifcations.
Strengths: 1. The motivation of this paper is reasonable, i.e., using precision-reject curve and recall-reject curve in the class-imbalance problems.

2. Some experiments are performed to show the differences of the precision-reject curve and recall-reject curve from accuracy-reject curve.

3. Some discussions are given.
Weaknesses: 1. The contribution is very limited. Precision-reject curve and recall-reject curve are two intuitive extensions of accuracy-reject curve. It is not suggested to presented them independently in one paper. It will be better to present them as part of one paper that proposes a classifcation algorithm with rejection option.

2. It is unclear why precision and recall are used. For class-imbalanced problems, F1 is the more commonly used measure.

3. In experiments, authors should pay more attentions to the difference of precision and recall from accuracy (e.g., figure 2). For figure 1, it is hard to observe their differences.
Recommendation: 1: strong reject
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 2 fair
reviews: 
Comments: The article considers the problem of performance measurement under model uncertanty and extends a previously proposed method -- accuracy-rejet curves -- to precision and recall metrics rather than accuracy.
Strengths: Rejection options is an important area of research, particularly impactful in some domains like medicine.
Weaknesses: It is unclear how this is different from standard precision recall curves. In the binary classification case, given that precision and recall focus on the positive class only this work appears to simplify down to precision-recall curves. It therefore does not appear to propose anything novel that would help with choosing particularly thresholds for a task. It may be different in the case of multiple classes but none of  the experimental datasets have more than two classes.
Recommendation: 3: reject, not good enough
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 1 poor
Clarity: 3 good
reviews: 
Comments: Accucracy-Reject-Curves (ARCs) are frequently used for evaluating classifiers with a reject option, i.e., where the classifier can choose to not classify an example. The authors argue that for imbalanced datasets, precision and recall reject curves, and evaluate them on four small datasets.
Strengths: The assumptions upon which the work rests are valid, and it is quite likely that precision and/or recall reject curves are a good alternative to ARCs on imbalanced datasets.
Weaknesses: Having said this, this extensions is a very trivial extension, as it is well-known (as the authors also observe) that precision and recall (and F1) are better evaluation measures for imbalanced data. So this paper could, in my opinion, only be of interest if the performed evaluation is exceptionally thorough and valid, so that this is a paper that clearly establishes the value of this proposal. Unfortunately, I think the evaluation is lacking for several reasons:
- the comparison is only on 4 small datasets, one of them as simple as two Gaussian clusters. The datasets are also rather small (less than 1000 instances), and - contrary to the claims of the paper - *not* heavily imbalanced (no class has less than 25% examples).
- no systematic evaluation of the data space, such as comparisons for several different rates of unbalancedness
- it is not clear why precision and recall (which are separate curves) have been selected, and not F1-reject curves, where one would only have to deal with one curve. These are, as the authors mention, also not new (although they have been proposed in a somewhat different context)
- there are some obvious benchmark methods that could/should have been included. For example, a natural extension would be to use local reject options, which have different thresholds for the majority and minority class. What is the advantage of precision-reject curves over ARCs in such settings? In the limiting case, when the minority class has a regular threshold, and the majority class has a fixed threshold of 1 (i.e., is never predicted), wouldn't you get something like a precision-reject curve for the minority class as well?

Minor:

The paper seems to be adapted from a version that used numeric citations. With author-year citations, constructs such as "The authors of Artelt et al. (2022) show.." are not good, this can be simply said as "Artelt et al. (2022) show...".
Recommendation: 1: strong reject
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 1 poor
Clarity: 3 good
reviews: 
Comments: In many real-world classification problems, one may want to provide a classifier the option to reject (or abstain from making a prediction) on samples that it is not certain about. In such cases, it is common to evaluate the trade-off between accuracy and rejection costs by plotting the accuracy as a function of rejection rates. However, in applications where there is severe class imbalance, plotting accuracy vs rejection rate may not be the best approach to evaluate the classifier. The paper proposes to instead use plots of precision and recall as a function of rejection rate, and presents some experiments to argue why these are better alternatives.
Strengths: - The paper is well-written and does a good job of covering prior work on classification with rejection.
- Exploring alternate ways for evaluating a classifier with a reject option is an important problem to tackle.
Weaknesses: - The paper is limited in novelty. The fact that metrics based on precision and recall are better alternatives to accuracy under class imbalance is well known in the literature. Merely proposing their use in evaluating rejection-based classifiers does not make for a significant or novel contribution.
- The experimental conclusions aren't very strong either. It is interesting that precision and recall curves show different trends compared to accuracy curves, but this observation alone does not make for a strong contribution. Besides, the datasets used are all small scale, containing a few hundred samples.
Recommendation: 1: strong reject
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 2 fair
Clarity: 3 good
reviews: 
Comments: This work addresses classification with rejection options, where performance evaluation typically relies on accuracy-reject curves to assess various certainty measures across different decision thresholds. While the paper introduces precision-reject and recall-reject curves as alternatives for handling class imbalance scenarios, the contribution appears insufficient for independent publication. The proposed curves represent straightforward variants of the existing accuracy-reject framework and would be better incorporated into a broader paper that develops a novel classification algorithm incorporating rejection mechanisms. Additionally, the choice of precision and recall metrics lacks adequate justification—F1-score would be a more standard selection for imbalanced classification tasks. The experimental analysis also requires deeper investigation into how precision and recall diverge from accuracy metrics; current visualizations (particularly Figure 1) do not clearly demonstrate these distinctions, and Figure 2 could benefit from more careful attention to highlighting such differences.
Strengths: 1. The paper presents a sensible motivation by proposing precision-reject and recall-reject curves specifically designed for class-imbalanced classification problems. 2. Experimental results are provided to illustrate how precision-reject and recall-reject curves differ from traditional accuracy-reject curves. 3. The work includes analytical commentary on the proposed approaches.
Weaknesses: 1. The overall contribution is modest, as precision-reject and recall-reject curves are straightforward extensions of the accuracy-reject curve. Publishing these two curves as standalone contributions in a single paper is not advisable; they would be more appropriately integrated into a comprehensive paper that proposes a novel classification algorithm with rejection capability. 2. The rationale for selecting precision and recall remains unclear. For class-imbalanced problems, F1-score is the more conventional evaluation metric. 3. The experimental section needs stronger emphasis on demonstrating how precision and recall measurements diverge from accuracy measurements (see Figure 2). The distinctions are insufficiently visible in Figure 1, suggesting that the comparative analysis requires more rigorous presentation.
Recommendation: 1: strong reject
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 2 fair