Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper mainly focuses on reporting a cautionary finding on using selective classification, but didn’t provide any solution. That is, selective classification can magnify existing accuracy disparities between various groups within a population, especially when there are spurious correlations. The authors studied the margin distribution to get some understandings about the problem. 

If the disparity across different groups is important, performing a uniform standard classification over the unified data is not a good idea in the first place. It seems the group disparity problem is inherent to the full coverage classification and should be addressed at the full coverage in the first place, e.g., by using group-DRO models.  This is consistent with their finding (which is intuitive) that selective classification can uniformly improve group accuracies on group-DRO models, because the problem is inherent to the classification model used. 

Although the authors tried to analyze their findings using margin distribution under the concept of the left-log-concave distribution, which does not lead to any solution for solving the disparity problem over the worst-group, but remains an observation. Moreover, it is unclear whether all the phenomena of selective classification magnifying disparity can be captured in their concepts and analysis, since their findings are based on empirical observations over a few datasets. 

Overall I feel the current observation report requires further effort. It would be natural to expect the analysis on observations can lead to a solution for the observed problem. 

review_title: require some further effort

final_score: rating: 7: Good paper, accept
review_content: The paper draws attention to a problem in the selective classification. In particular, it implies that selective classifiers can perform unnegligible accuracy disparities across various groups. To capture such information, the authors theoretically analyze the margin distributions and compare to the group-agnostic baseline, which can help to explain the accuracy disparities shown in the training result of 5 datasets. 

Overall, I like the idea of theoretically analyzing the accuracy-coverage curve by margin distribution and I vote for accepting.

Pros:
1. The paper takes an important issue of selective classification: accuracy disparities between groups. For me, it's possibly related to ''fairness'' in classification, which we shall treat carefully in real-world problems. 

2. The proposed margin distribution for analysis is reasonable. The theoretical analysis is helpful to explain the observation in experiments. 

3. This paper provides experiments to illustrate their findings in typical datasets and defend their analysis.   

Below are my major concerns. Hopefully the authors can address them in the rebuttal period. 

Cons:
1. Although exploiting the margin distribution can provide good reasoning on the monotonicity of accuracy-coverage curves, analysis of comparison to group-agnostic baseline seems not very clear to me: 

(1) In that section, the authors consider the case of a mixture of two groups. Is this case typical enough to demonstrate the issues the authors want to imply?

(2) In proposition 5, the authors show that log-concave distributions always underperform the baseline. However, from my view, the importance of this issue lies on that the disparity is unnegligible, so a more refined estimate of the difference between accuracies, namely, $|A_F(\tau)-\tilde A_F(\tau)|$ (may need some extra conditions) will be more helpful to explain results on CheXpert-device.


Some typos:
1. Page 4: there are extra "]" in both C(tau) and I(tau).
review_title: Review of Paper2955

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary: This paper demonstrates and analyzes the accuracy disparities induced by selective classification on subgroups. It evaluates these induced disparities across five datasets from different domains, and provides a theoretical analysis of the conditions under which selective classification may decrease accuracy or magnify disparities via evaluation of the margin distribution.

Overall, this paper is a useful addition to a growing literature on the disparate impact of various machine learning techniques on subgroups. The analysis in terms of margin distribution is both appropriate to the selective classification task, and novel, as far as I am aware. The theoretical analysis clearly describes conditions under which the various observed experimental results may generally hold. The (short) section on DRO also suggests a promising way out of this problem, informed by the results and analysis in the paper. The paper could use some minor edits and clarifications, but I believe it would be a valuable addition to the conference.

Note: I did not evaluate or check all of the proofs.

#### Major Comments:

* I found the description of the baselines confusing; it is not made clear that C and I are being defined (I had to scan to see if they were previously discussed) and a great deal of text is used to describe a baseline construction procedure which could be encapsulated in a simple algorithm (my recommendation) or set of numbered steps.

* Throughout, the paper uses a definition of "average accuracy" (defined in Sec. 3) which is not actually average accuracy -- it is average accuracy on covered points. I find this to be a bit misleading, or at the very least a confusing overloading of the term, and would recommend introducing a new name for this metric and re-labeling the y-axis of the plots that use it.

* The DRO section (7) feels like an afterthought, but it is actually quite important to the paper. There are several recent works which identify disparate model performance impacts on subgroups (e.g. due to model compression, differential privacy, etc.), with no clear solution. The connection to DRO suggests a potential solution, at least in practice, for the problem identified in this paper. It would be nice to, at the very least, see some of the discussion of DRO upgraded from the appendix to the main text, if not a more thorough analysis.

#### Minor comments:

* I did not feel that the actual process of training a selective classifier (vs. a standard classifier) was clearly explained; since this is fundamental to the analysis, it would be useful to have this clearly signposted for readers unfamiliar with selective classification (such as this reviewer).

#### Typos etc.

Sec. 7: "to their average accuracies Figure 5"

####

UPDATE: I have reviewed the author response and the revisions. All of my main concerns were addressed. I believe that these improved the paper further, and while they do not change my rating, the paper is still a clear "accept" in my view.
review_title: Selective Classification Can Magnify Disparities Across Groups

final_score: rating: 7: Good paper, accept
review_content: This paper discusses the impact of classifier abstention on the performance obtained for different groups of data. In particular, the authors show that, while abstaining in case of uncertainty in general leads to an improvement in predictive performance, it can also lead to worse results for specific groups. In a somewhat atypical paper structure, the authors first illustrate this empirically on five banchmark datasets. Later in the paper, they also present theoretical results that explain the empirical results. 

This is an interesting paper about an original research topic. The take-home message is not so surprising, but the analysis is thorough. I particularly appreciated that the empirical results are also supported by theoretical results. I didn't do a detailed check of the proofs, but I don't see a reason why the propositions should not hold. 

Overall the paper is also well written and I enjoyed reading it. A few things are not very clear to me:

1. In Section 3 it is briefly mentioned that two scores are considered as a representation of uncertainty: softmax and a score based on MC dropout. For the latter it is not clear to me how the score is computed. Is this the variance over different predictions for an instance? Moreover, in Figures 2 and 4 it is not mentioned which of the two scores is used to obtain the graphs. I would have expected to see the results for softmax in the main paper, since this is most often used as uncertainty score, but the horizontal axis in Figure 4 is not limited to the interval [-1,1], so this cannot be softmax. This needs to be explained better. 

2. Related to the previous remark, I would also argue that the margin distributions depend very strongly on the uncertainty measure that is used. As discussed e.g. in the work of Kendall and Gal, softmax looks at aleatoric uncertainty, whereas MC dropout  looks at epistemic uncertainty. Furthermore, also a lot of other uncertainty measures exist in the literature. For example, in multi-class classification, one often constructs models that abstain on instances that are very dissimilar to the training data (so-called out-of-distribution samples). Thus, different measures represent different types of uncertainty, so I also expect to see very different margin distributions on specific datasets. 

3.  In the description of the experimental setup, I found the notion "spurious attribute" unclear. For "spurious correlation" it is clear what is meant, but I have never heard of a spurious attribute, and I also don't know what that would be. Aren't you just referring to a confounding attribute, as commonly used in the statistical literature? Does it make a difference for the analysis whether the group variable is observed or unobserved? It might be good to elaborate on this. 

4. It is also not very clear what the practical usefulness of the obtained results is. It is nice to know that the performance can drop for specific groups when abstention is allowed, but this is not necessarily a bad thing. In practice one encounters a lot of situations where machine learning models are used as a prescreening tool, followed by a manual check for cases that are hard to classify by models. In such situations one could decide to use a particular algorithm only for specific groups, whereas badly-scoring groups would deserve a special treatment, such as a manual screening. 
review_title: Interesting paper about potential biases induced by abstention

Meta-review: This is an interesting paper discussing the impact of classifier abstention on the performance obtained for different groups of data. The reviewers are either very (scores of 8, 7 and 7) or moderately (score of 5) positive about the paper. The main concern is that the paper does not directly propose a solution for the discovered problems. Nevertheless, it can initiate interesting discussions and research around them.
Decision: Accept (Poster)