Classification with Conceptual Safeguards
Authors: Hailey Joren
Charles Thomas Marx
Berk Ustun
Venue: ICLR 2024 poster (2024)
Keywords: classification
selective classification
concept bottleneck models
uncertainty quantification
selective classification
Primary area: societal considerations including fairness, safety, privacy
TL;DR: We introduce conceptual safeguards, a method for selective classification in concept bottleneck models that can reliably promote safety and interpretability in automation tasks

Abstract: We propose a new approach to promote safety in classification tasks with concept annotations. Our approach – called a *conceptual safeguard* – acts as a verification layer for models that predict a target outcome by first predicting the presence of intermediate concepts. Given this architecture, a safeguard ensures that a model meets a minimal level of accuracy by abstaining from uncertain predictions. In contrast to a standard selective classifier, a safeguard provides an avenue to improve coverage by allowing a human to confirm the presence of uncertain concepts on instances on which it abstains. We develop methods to build safeguards that maximize coverage without compromising safety, namely techniques to propagate the uncertainty in concept predictions and to flag salient concepts for human review. We benchmark our approach on a collection of real-world and synthetic datasets, showing that it can improve performance and coverage in deep learning tasks.

Published conference paper at ICLR 2024
CLASSIFICATION WITH CONCEPTUAL SAFEGUARDS
Hailey Joren
Charles Marx
Berk Ustun UC San Diego
UC San Diego
Stanford University
bjoren@ucsd.edi
ctmarx@stanford.edd
berk@ucsd.edu
ABSTRACT
We propose new approach to promote safety in classification tasks with es- tablished concepts. Our approach called a conceptual safeguard acts as a verification layer for models that predict target outcome by first predicting the presence of intermediate concepts. Given this architecture, a safeguard ensures that model meets minimal level of accuracy by abstaining from uncertain predictions. In contrast to a standard selective classifier, a safeguard provides an avenue to im- prove coverage by allowing human confirm the presence of uncertain concepts on instances on which it abstains. We develop methods to build safeguards that maximize coverage without compromising safety, namely techniques to propagate the uncertainty in concept predictions and to flag salient concepts for human review. We benchmark our approach on a collection of real-world and synthetic datasets, showing that it can improve performance and coverage in deep learning tasks.
INTRODUCTION
One of the most promising applications of machine learning is to automate routine tasks that human can perform. We can now train deep learning models to perform such tasks across applications be it identify bird in image [1]. detect toxicity in text [2], or pneumonia in a chest x-ray [3]. Even as these models may outperform human experts [4, 5, 6]. their performance falls short of the levels we need reap the benefits of full automation. A bird identification model that 80% accurate may not work reliably enough to be used in the field by conservationists. A pneumonia detection model with 95% accuracy is still not sufficient to eliminate the need for human oversight in a hospital setting.
One strategy to reap some of the benefits of automation in such tasks through abstention. Given any model that is insufficiently accurate, we can measure the uncertainty in its predictions and improve its accuracy by abstaining from predictions that are too uncertain. In this way, we can improve accuracy by sacrificing coverage i.e., the proportion of instances where model assigns prediction. One of the common barriers to abstention in general applications is how to handle instances where a model abstains. In applications where we wish to automate routine task, would only pass these to the human expert who would have made the decision in the first place. Thus, abstention represents a way to reap benefits from partial automation.
In this paper, we present a modeling paradigm to ensure safety in such applications that we call a conceptual safeguard (see Fig. 1). A conceptual safeguard an abstention mechanism for models that predict an outcome by first predicting a set of concepts. Given such a model, a conceptual safeguard operates verification layer- estimating the uncertainty in each prediction and abstaining when it exceeds a threshold needed to ensure minimal level of accuracy. Unlike a traditional selective classifier. a conceptual safeguard provides a way improve coverage by allowing experts to confirm the presence of certain concepts on instances on which we abstain.
Although conceptual safeguards are designed to be a simple component that can be implemented off the shelf, designing methods to learn them is challenging. On the one hand, concepts introduce degree of uncertainty that we must account for at test time. In the best case, we may have to abstain too much to achieve desired level of accuracy. In the worst case, we may fail to hit the mark. On the other hand, we must build systems that are designed for confirmation i.e., where we can reasonably expect that confirming concepts will improve coverage and where that improves coverage. Our work seeks to address these challenges so that we can reap the benefits of these models.
rank the concepts in way

---

Published conference paper at ICLR 2024
Conceptual saleguards abstain using concept uncertainty
Abstentions can be resolved by confirming concepts
Conceptual Safeguard
Conceptual Safeguard
Image Skin
Predicted Concepts
Uncertainty Propagator
Image of Skin Lesion X)
Predicted Concepts
Uncertaiity Propagator
Seeectiv Classifer
@(R2) m PN
(OX) P)
fin
9x10)
Prejected)
Pr[Dotbed]
PrEDLawetes) 10%
>1 89%
9 Abstain
PrFFigeacedd
90%
=1 Melanome
PlEEoewgRaac 1 0%
Figure 1: Conceptual safeguard to detect melanoma from image of skin lesion. We consider model that estimates the probabilities a conceptual safeguard will decide whether to output prediction 9 E {Mel {#elannne, NoMelanoma } or to abstain ý =1. The safeguard improves accuracy by abstaining on images that would receive low confidence prediction, and measures confidence in way that accounts for the uncertainty in concept predictions through uncertainty propagation. On the left, show image where safeguard abstains because confidence fails meet the threshold to ensure high accuracy Pr(xelanoma] = 62% I 90%. On the right, we show a human expert can resolve the abstention by confirming the presence of concepts Dotted and LrzegVasc in the image.
concepts: Dotted, P1 Pigmeated ... IrregularVasc. Given these probabilities,
Contributions Our main contributions include:
We introduce a general-purpont approach for safe automation in classification tasks with concept annotation. Our approach can be applied using off-the supervised learning techniques.
We develop a technique to account for concept uncertainty in concept bottleneck models. Our technique can use native estimates from the components of a concept bottleneck to return reliable estimates of label uncertainty, improving the accuracy-coverage trade-off in selective classification.
We propose confirmation policy that can improve coverage. Our policy flags concepts in instances on which a model abstains, prioritizing high-value concepts that could resolve abstention. This strategy can be readily customized to conform to a budget and applied offline.
4. We benchmark conceptual safeguards on classification datasets with concept labels. Our results show that safeguards can improve accuracy and coverage through uncertainty propagation and concept confirmation.
RELATED WORK
Our work is related to a large body of work on machine learning with a reject option esg. 7, for recent survey], and more specifically methods for selective classification [8, 9, 10, 11]. Our goal is to learn a selective classifier that assigns as many predictions as possible while adhering conceptual safeguards for this task through post-hoc approach [see e.g., 12] in which we are given model, and build a verification layer that estimates the confidence of each prediction and abstains on predictions where a model is insufficiently confident. The key challenge in our approach that we require reliable estimates of uncertainty to abstain effectively [13, 14], which we address by propagating the uncertainty in concept predictions to the uncertainty in the final outcome.
Selective Classification
minimal level of accuracy [i.e., the bounded abstention model of 11]. We build
Deep Learning with Concepts Our work is related to a stream of work on deep learning with concept annotations. A large body of work uses these annotations to promote interpretability either by explaining the predictions of DNN terms of concepts that a human can understand [15, 16], or by building concept bottleneck models i.c., model that predicts predicting concepts - i.c., [17]. One the key motivations for concept bottleneck models is the potential for humans to intervene test time [17] e.g., to improve performance by correcting concept prediction that was incorrectly detected. Recent work shows that many architectures that
outcome of interest by
2

---

Published conference paper at ICLR 2024
perform well cannot readily support interventions [see e.g., 18, 19]. Likewise, interventions may be practical in applications where we wish automate routine task. In such cases, we need mechanism to flag predictions for human review to prevent human experts from having to check each prediction. Our work highlights several avenues to avoid these limitations. In particular, we work with independent architecture that is amenable interventions, consider restricted class of interventions, and present an approach that does not require constant human supervision.
One overarching challenge in building concept bottleneck models is the need for concept annotations. In effect, very few datasets include concept annotations and those that do are often incomplete (c.g., an example may be missing some all concept labels). In practice, the lack of concept labels can limit the applicability and the performance of concept bottleneck models and has motivated work a recent stream of work on machine-driven concept annotation [see e.g.. 20] and drawing on alternate sources of information [21, 22]. Our work outlines an alternative approach to overcome this barrier adoption: rather than building new model that is sufficiently accurate, use much as possible by allowing it to abstain from prediction.
FRAMEWORK
We consider a classification task to predict a label from a complex feature space. We start with a dataset of i.i.d. training examples ((x; Ci, m))] where example i consists of:
a vector of features x; E 22' C R c.g., x, pixels in image i; vector of k concepts C₁ E C {0, 11mh e.g., Ci = 1 if x-ray contains a bone spur;
a label yi E E = {0, e.g. yi if patient has arthritis.
Objective We use the dataset to build a seleetive classification model + ( Given a feature vector X,, denote the output of the model as h(x;) where = 1 denotes that the model abstains from prediction for K;
In the context of automation task, would like our model assign many predictions as possible while adhering to a target accuracy to ensure safety at test time. In this setup, abstention reflects viable path meet this constraint by allowing the model abstain instances where would otherwise assign an incorrect prediction. Given an target accuracy a E (0, 1), we express these requirements as optimization problem:
max Coverage(1)
s.t. Accuracy(h)) 2 a,
where:
Coverage(h) := Pr (Ú AL) is the coverage of the model h - i.e., proportion of instances where h outputs a prediction.
Accuracy(h) (h) == Pr (y = ÿ # is the selective accuracy of the model h - i.e., the accuracy of h over instances on which it outputs a prediction
System Components We consider a selective classifier with the components shown in Fig. 1. Given the dataset, will first train the basic components of an independent concept bottleneck model:
A concept detector 9 X [0. Given features I,, the concept detector returns output a vector of m probabilities q I.. vi.m] E [0, 11 where each gi : Pr(ci,e = w:)) captures the probability that concept k is present in instance i.
A front-end model f : C V. which takes as input a vector of hard concepts e, and returns as output the outcome probability 7, == f(c.).
For example, we build sequential architecture- wherein we train front-end model to predict the label from predicted concepts then interventions may reduce accuracy as the front-end may rely on an incorrect concept prediction to output accurate label predictions.
3

---

Published conference paper at ICLR 2024
100%
Operating Poins $ 0.1
Selective Accuracy
Safeguard Confimmaton
Salequard
50%
0%
Coverage
100%
Figure We evaluate the performance of classification models that can abstain using an aceuracy-coverage curve [11]. Given a model that outputs probability prediction for each point, a conceptual safeguard flags points on which model abstains based on confidence threshold 7 E [0, 0.5j where setting T 0 lcads to 100% coverage and setting r = 0.5 leads to 0% coverage.
In what follows, we treat these models as fixed components that are given and assume that they satisfy two requirements: (i) independent training, meaning that we train the concept detector and the front-end model using the datasets {(x, c,) and {(C, b:))== respectively; (ii) calibration, i.e. that the concept detector and front-end model and return calibrated probability estimates of their respective outcomes. As we will discuss shortly, independent training will be essential for confirmation while calibration will essential for abstention.
In practice, these are mild requirements that we can satisfy using off-the-shelf methods for supervised learning. Given a dataset, for example, we can fit the concept detectors using a standard deep learning algorithm, and fit the front-end model using logistic regression. In settings where the dataset is missing concept labels for certain examples, we can train a separate detector for each concept to maximize the amount of data for each concept detector. In settings where have an embedding model [21] or an existing end-to-end deep learning model, we can dramatically reduce the computation by training the concept detectors via fine-tuning In all cases, can calibrate each model by applying a post-hoc calibration technique, e.g., Platt scaling [23].
Conceptual Safeguards Conceptual safeguards operate as confidence-based selective classifier abstaining points where they cannot assign sufficiently confident prediction. We control this behavior through an internal component called the selection gate ş [o, y' {1}, which is parameterized with a confidence threshold T E [0, IJ. Given threshold the gate takes as input soft label V, E [0, 1] and returns:
00 if
D. E,,)
i = (+())
I. if
D. E (r,
1 if
D. E
As shown in Fig. 2, set choose operating point for the model on the accuracy-coverage curve e.g., to meet a desired target accuracy rate at by sacrificing coverage. In settings where our front-end model returns calibrated probability predictions, we can use them
the threshold 7:
Definition 1. A probabilistic prediction
[0, for binary label y E {0, 1} is calibrated if Pr (y y = t) for all values E [0, 1].
Proposition 1. Suppose that y is a calibrated probability prediction for y. Then any selective classifier (v) that abstains when y has confidence below 1 achieves accuracy least 1 T.
Proposition is standard result ensuring high accuracy for selective classifiers that output calibrated probabilistic prediction (see Appendix for a proof). In cases where the front-end model is not calibrated, the result will hold only given the degree of calibration. Thus, the reliability of this approach hinges on the reliability of our confidence estimates. An alternative approach for such cases is to treat the output of the front-end model f as confidence score and tune using calibration dataset [see, e.g., the SGR algorithm of 12].
Confirmation We let users confirm the presence of concepts on instances where model abstains. In a pneumonia detection task, for example, we can ask a radiologist to confirm the presence of

---

Published conference paper at ICLR 2024
concept & in x-ray i. We refer to this procedure as confirmation rather than intervention to distinguish from other ways where human expert would alter the output from a concept detector."
We consider tasks where each concept is human-verifiabl human experts that we would query to confirm [c.f., concepts where human may be uncertain as in 24]. In such tasks, confirming concept will replace its probability fi.k to its ground-truth value C E {0, 1}. We cast confirmation as policy function Vs [0, 1 - [0, 1 where S C [m]] denotes subset of concepts to confirm. The function takes input vector of raw concept predictions
i.e., that it can be detected correctly by
and returns a vector of partially confirmed concept predictions: p; = PP....... Pi,ml E [0, 1 where:
Gi.k
if E
PLR =
if $
METHODOLOGY
In this section, describe how build the internal components of conceptual safeguard. We first discuss how to output reliable predicted probabilities given uncertain inputs at test time. We then introduce a technique flag promising examples that can be confirmed improve coverage.
3.1 UNCERTAINTY PROPAGATION
It is important that the concept detectors are probabilistic, so that we can prioritize confirming concepts that have high uncertainty. However, this creates an issue where the concept detectors output probabilities, but the front-end model requires hard concepts inputs.
Here, we describe simple strategy wherein we use uncertainty propagation allow the front-end model to accept probabilities rather than hard concepts as input. We make two assumptions about the underlying data distribution to motivate our approach.
Assumption 2. The label y and features
conditionally independent given the concepts Assumption 3. The concepts {Ci,..., Cm.) are conditionally independent given the features x.
We these assumptions to write the conditional label distribution p(y x:) in terms of quantities that we can readily obtain from a concept detector and front-end model:
ply x)
Σ
ply c, p(c x)
(1)
cE(0.1) m
Σ
ply c) p(c x)
(Assumption 2)
cE(0,1)
Σ
plu c))
II
p(ck x))
(Assumption 3)
cE(0,1)
KE [m)]
output from f(e)
output from gu (x)
We use this decomposition propagate uncertainty from the inputs of the front-end model its output. Specifically, we compute the expected prediction of the front-end model on all possible realizations of hard concepts and weigh each realization in terms of the probabilities from concept detectors. In practice, replace each quantity in Eq. (1) with an estimate computed by the concept detectors and front-end model. Given predicted probabilities from concept detectors p, = (Pi,1, ,P.kk). we compute the estimate of p(u æz) as:
f(p.) Σ f(c) (0.1 kg(m)
Pi (1(1)..))--- 2-c
Here, we abuse notation slightly and use (p;) to denote the front-end model applied to uncertain concepts, and f(c) to denote the front-end model applied hard concepts. This requires 2 calls to the front-end model, which negligible in practice as most front-end models are trained with limited number of concepts. In settings where is large, computation is prohibitive, we can use sample of concept vectors to construct Monte Carlo estimate.
2 For example, intervention may refer to "correction" in which a human replaces a hard concept prediction
with its correct value.

---

Published conference paper at ICLR 2024
Algorithm Greedy Concept Selection
Input: {i E [n] Pr (f({)) = }} instances on which model abstained
Input: 71.-.. 7'm O, cost to confirm each concept
Input: B>0, 0, confirmation budget
I: SI...,,,, -0
concepts to confirm for each instance
2: repeat
i" i" < arg max,, (qi k) s.t. le &
select best remaining concept
S.- +-+- S.- [k*}
4:
B ++ B - *
5: until < 0 Or = Im| for all i [n]
Output: Si...., Sn, concepts to confirm for each abstained instance
3.2 CONFIRMATION
Selective classification guarantees higher accuracy at the cost of potentially lower coverage. In what follows, we describe a strategy that can mitigate the loss in confirmation by human confirmation i.e., manually spotting concepts among instances which we abstain. In principle, confirmation will always lead an improvement in coverage. practice, human confirmation is labor intensive and may require expertise examples that are responsive to confirmation costs.
we want to develop techniques that can account by flagging promising
In Algorithm I, we present routine to flag concepts for human expert review among instances on which a model abstains. The routine iterates over a batch of n points on which the model has abstained and identifies salient concepts that can be confirmed by human expert to avoid coverage.
Algorithm computes the gain associated with confirming each concept on each instance and then returns the concepts with the maximum gain while adhering user-specified confirmation budget B > O. We associate the cost confirming each concept k with a cost % > 0, which can be set to control the time expertise that is associated with each confirmation. The routine selects concepts for review based on the expectation of the gain in certainty. We measure the gain in certainty in terms of the variance of the prediction after confirmation:
Gain(q, k)) Var -ernx=B [+(((
qı q1 (u) (f(glgx + 1l) - f(g/g= + 0]))
The gain measure in (3) captures the sensitivity of predictions from the front-end model by confirming concept k. In particular, we seek identify concepts that if confirmed would induce large change in the output of the front-end and thus resolve abstentions. Given that we do not know the underlying value of concept & prior confirmation, we treat qи as random variable that will be set to q + 1] with probability and q[qx <- 0] with probability 1 Qu. Here, qlys- +- refer to the vector q, except with d replaced with
4 EXPERIMENTS
We present experiments where we benchmark conceptual safeguards on a collection of real-world classification datasets. Our goal evaluate their accuracy and coverage trade-offs, and to study the effect of uncertainty propagation and confirmation through ablation studies. We include details on our setup and results in Appendix B, and provide code reproduce our results on GitHub.
4.1 SETUP
We consider six classification datasets with concept annotations:
The melannn melanoma and skincancer datasets are image classification tasks to diagnose melanoma and skin cancer derived from the Derm7pt dataset [25].
The warbler and flycat.cher datasets image classification tasks derived from the CalTech- UCSD Birds dataset [26] to identify different species of birds.

---

Published conference paper at ICLR 2024
The noi syconcept.s2, and noi syconcepts75 datasets are synthetic classification tasks de- signed to control the noise in concepts (see Appendix B.1).
We process each dataset to binarize categorical concepts (e.g,, WingColor to WingColorRed) We split each dataset into a training sample (80%, used to build a selective classification model) and a test sample (20%, used to evaluate coverage and selective accuracy in deployment).
Models
We train a selective classification model using one of the following methods:
X-+ MLP: multilayer perceptron trained top of the penultimate layer of the embedding model. This baseline represents an end-to-end deep learning model that directly predicts the output without concepts.
Baseline: An independent concept bottleneck model built from concept detectors g1. Im and front-end model f trained to predict the true concepts.
CS: Conceptual safeguard built from the same concept detectors and front-end as the baseline. This model propagates the uncertainty from the concept detectors 1m to the front-end model as described in Section 3.
We build Baseline and CS models using the same front-end model f and concept detectors 91.... Im- We train the front-end model f using logistic regression, and the concept detectors using embeddings from a pre-trained model [i.e., Inception V3, 27] (for all datasets other than noisyconcepts).
Evaluation We report the performance of cach model through accuracy-coverage Fig. 2, which plots its coverage and selective accuracy on the test sample across thresholds. We evaluate the impact of confirmation in concept based models i.e., Baseline and CS in terms of the following policies:
in
ImpactConf, which flags concepts review using Algorithm RandomConf, which flags random subset of concepts to review.
We control the number of examples to confirm by setting a confirmation budget, and plot accuracy- coverage curves for confirmation budgets of 0/10/20/50% to show how performance changes under no/low/medium/high human levels of human intervention, respectively.
4.2 RESULTS
We present the accuracy coverage curves for all methods and all datasets in Table Given these curves, evaluate the gains uncertainty propagation by comparing Baseline to CS, and the gains from confirmation by comparing Baseline + RandomConf to CS + ImpactConf).
Overall, results that methods outperform their respective baselines in terms of coverage and selective accuracy. In general, we find that the gains vary across datasets and confirmation budgets. Given desired target accuracy, for example, achieve higher coverage warble and fl Flycatcher rather than mel anoma and sk skincaner.. Such differences arise due to differences in the quality of concept annotations and their relevance for the prediction task hand.
On Uncertainty Propagation Our results highlight how uncertainty propagation can lead to im- provements in a safeguard. On the one hand, we find that uncertainty propagation improves the accuracy coverage trade-off. On the skí ncancer dataset, for example, we see a major improvement in the accuracy coverage curves between model that propagates uncertainty (CS + RandomConf, blue) and a model and comparable model that does not (Baseline RandomConf, green).
the other hand, accounting for uncertainty can improve these trade-off's by producing a more effective confirmation policy. Our results highlight these effects by showing gains of uncertainty may change under confirmation budget. On the #lycatchel cher dataset, for example, accounting for uncertainty leads to little difference when do not confirm examples (i.c.. a 0% confirmation budget). In a regime where we allow for confirmation setting a 20% confirmation budget we find that accounting for uncertainty increase coverage, from 46.2% to 63.5%, when the threshold is set at 7 = 0.05 (see Section 4.1).
7

---

Published conference paper at ICLR 2024
Confirmation Budget
23%
90%
92%
80%
[26]
Dambin Card
97%
600
100%
50%
7536
1005
100%
1005
3006
slycatchez
90%
60%
80%
65%
8-
99%
Jumking Randemtert
907
155
C5
53%
0%
23%
50%
75%
300%
06
255
35%
15%
100%
30%
0%
2555
50%
75%
201%
0%
255
507
15%
100%
139%
200%
100%
2004
23%
90%
15%
ralancra n. 616
45%
80%
70%
70%
104
Kiwahara ot (25)
55%
9%
23%
50%
75%
50%
100%
50%
25%
55%
75%
200%
181
Covergge
Craeage
Cocerag
Coverace
30%
50% Isselin
90%
10%
akincancez 616
17
7P%
165
200
1256
Kiwahara
59%
50%
Cosergge
Cooeraee
Coverage
Coverage
45%
90%
15%
90%
60%
RDY NELP
57%
Tamdine Caed
60%
976
125
100%%
50%
130%
100%
100%
2004
un nciayounsaps.778 We -
40%
60%
805
Jaseãie
95%
Sasedin RandommCen PrpuctCue
0%
25%
50%
75%
300%
25%
53%
75%
1.00%
07%
2575
50%
75%
200%
0.4
25.50
5000
300%
Table 1: Accuracy coverage curves methods on all datasets. include additional results Appendix for multiclass tasks. Note that Baseline (black) and Baseline + RandomConf (green) produce identical results without confirmation. Likewise, CS RandomCont (blue) and CS ImpactConf (purple) are also equivalent under the same condition.
On Confirmation Our results show that confirming concepts improves performance across all datasets. On flycatcher, we find that confirming random subset of instances 20% improves coverage from 26.9% to 46.2% for threshold of 0.05 (Baseline - Baseline RandomConf), In a conceptual safeguard, coverage starts from 63.5% due to uncertainty propagation (CS RandConf), and improves 84.6% as result of our targetted confirmation policy (CS ImpactConf). The gains of confirmation depend on the underlying task and dataset. For example, the gains may smaller when the concept detectors perform well enough without confirmation achieve high accuracy (e.g.. for the warbler and noi syconcecnsp datasets).

---

Published conference paper at ICLR 2024
Datoset
Prediction Thresholds
Baseline RandomCref
ImpoceCCon Conf
warbler
5,994
0.1
100,0/5
74.0%
87.34
89.3 89.3%
100.000
7
7 0.15
100.005
100.00/
100.00%
100.009
100.005
Wab et [26]
0.2
100.00%
100.00%
100.00%
100.00%
100.00%
Elycaticher
0.05
0.0%%
26.9%
0.0%
46.2%
84.62%
x 5,9994
7= 0.1
82.7%
44.2%
36.5%
46.2%
100.00%
0.15
92.3%
44.2%
36.5%
65.4%
100.00%
Wah et al. [26]
0.2
100.0088
57.7%
51.9%
100
100.0004
me. anooa
7 0.05
0.0%
0.0%
4.3%
8.6%
52.86%
0.0%%
0.0%%
72.86/5
mn 17
== 0.15
0.0%%
0.0%6
14.3%
41.4%
100.0005
Kowahum [25]
100.0098
skinconcer
0.0%
0.0%
0.0%
0.0%%
616
= 0.1
0.0%
0.0%
36.6%%
15. 15.9%
39.0216
7
85.375
Kawahara et [25]
0.2
86.599
11,0%
36.69
36.6%
85.49)
aoi soisyconcepta25
= 0.05
0.0%
0.0%
0.0%
0.0%
0.0%%
x 100,
32.39
52.3%
32,35
33 33.99
44.66%
0.15
44.1%
43.6%
43.6%
45.8%
69.16%
7= 0.2
80.4%
80.4%
80.4%
86.8%
93.31%
nai maisycnncept.15
7 - 0.05
0.0%
0.0%
0.0%
0.0%
0.0%
100, 000
8.19%
11.17%
m 3
0.15
0.0%8
0.0%3
0.0%
0.0%
11.17%
0.19%
11.17%
Table 2: Coverage at specific thresholds for confirmation budget of 20%. We present results for other datascts and confirmation budgets Appendix B.2.
Our results highlight the value of targetted confirmation policy i.e., a technique that lead to meaningful gains in coverage without compromising safety or requiring real-time human supervision. In practice, these gains only part of the benefits of our approach specify costs in a way that limits the experts required for confirmation. For example, non-expert users may be able to confirm concepts such as WingCol orRed, while confirming the final species prediction to RedF acedCormoraaae may require greater expertise.
practitioners may be able to
5 CONCLUDING REMARKS
Conceptual safeguards reflect general- purpose approach to promote safety through selective clas- sification. applications where we wish to automate routine tasks that humans could perform, safegaurds allow us to reap some of the benefits of automation through abstention in way that can promote interpretability and improve coverage. Although our work has primarily focused on binary classification tasks, our machinery can be applied build conceptual safeguards for multiclass tasks (see Appendix B.3), and extended to other supervised prediction problems.
Our approach has number of overarching limitations that affect concept bottlenecks and selective classifiers. As with all models trained with concept annotations, we expect to incur some loss in performance relative to end-to-end model when concepts information about the label from its input. In practice, this gap may be large and we may reap the benefits of automation using traditional selective classifier. As with other selective classification methods, we expect that abstention may exacerbate disparities in coverage or performance across subpopulations [see e.g., 28]. our setting, it may be difficult pin down the source of these disparities as they may arise from concept annotations, model concepts, or interaction effects. Nevertheless, we may be able mitigate these effects through suitable confirmation policy.
unlikely to capture all relevant

---

Published conference paper at ICLR 2024
ACKNOWLEDGMENTS
We thank Taylor Joren, Mert Yuksekgonul, and Lily Weng for helpful discussions. This work was supported by funding from the National Science Foundation Grants IIS-2040880 and IIS-2313105, the NIH Bridge2AI Center Grant U54HG012510 and Amazon Research Award.
REFERENCES
[1] Michael A Tabak, Mohammad Norouzzadeh, David Wolfson, Steven Sweeney. Kurt C VerCauteren. Nathan P Snow, Joseph M Halseth, Paul A Di Salvo, Jesse S Lewis, Michael D White, et al. Machine learning to classify animal species in camera trap images: Applications in ecology. Methods in Ecology and Evolution, (0(4):585-599 2019.
[2] Deepak Kumar, Patrick Gage Kelley, Sunny Consolvo, Joshua Mason, Elie Bursztein, Zakir Durumeric, Kurt Thomas. Michael Bailey. Designing toxic content classification diversity perspectives. In Seventeenth Symposium on Usable Privacy and Security (SOUPS 2021), pages 299-318, 2021.
[3] An Yan, Yu Wang, Yiwu Zhong, Zexue He, Petros Karypis, Zihan Wang, Chengyu Dong, Amilcare Gentili, Chun-Nan Hsu, Jingho Shang. al. Robust and interpretable medical image classifiers via concept bottleneck models. arXiv preprint arXiv:2310.03 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
[5] Wenying Zhou, Yang Yang, Cheng Yu, Juxian Liu, Xingxing Duan, Zongjic Weng, Dan Chen, Qianhong Liang, Qin Fang, Jiaojiao Zhou, et al. Ensembled deep learning model outperforms human experts in diagnosing biliary atresia from sonographic gallbladder images. Nature communications, 12(1))1259, 2021.
[6] Eunji Chong, Flysha Clark-Whitney Audrey Southerland, Elizabeth Stubbs, Chanel Miller, Eliana Ajodan, Melanic R Silverman, Catherine Lord, Agata Rozga, Rebecca M Jones, et al. Detection of eye contact with deep neural networks is accurate as human experts. Nature communications, 11(1):6386. 2020.
Kilian Hendriekx, Lorenzo Perini, Dries Van der Plas, Wannes Meert, and Jesse Davis. Machine learning with reject option: A survey. arXiv preprint arXiv:2 1177 2021.
[8] Chi- Keeng Chow. An optimum character recognition system using decision functions. IRE Transactions on Electronic Computers, pages 247-254, 1957.
[9] C Chow. On optimum recognition error and reject tradeoff. IEEE Transactions on information theory, 16 (1):41-46 46, 1970.
[10] Giorgio Fumera and Fabio Roli. Reject option with multiple thresholds. Pattern recognition, 33(12): 2099-2101, 2000.
[11] Vaclav Voracek Vojtech Franc, Daniel Prusa, and Vaclav Voracck. Optimal strategics for reject option classifiers. Journal of Machine Learning Research, 24(11):1-45) 2023.
[12] Yonatan Geifiman and Ran El- Yaniv, Selective classification for deep neural networks. Advances in neural information processing systems, 30, 2017.
[13] Nontawat Charoenphakdee, Zhenghang Cui, Yivan Zhang, and Masashi Sugiyama. Classification with rejection based on cost-sensitive classification, International Conference on Machine Learning, pages 1507-1517. PMLR, 2021.
[14] Adam Fisch, Tommi Jaakkola, and Regina Barzilay. Calibrated selective classification. arXiv preprint 2208 12084, 2022.
[15] Been Kim, Martin Wattenberg, Justin Gilmer, Carric Cai, James Wexler, Fernanda Viegas, al. In- terpretability beyond feature attribution: Quantitative testing with concept activation vectors (teav). In International conference on machine learning, pages 2668-2677. PMLR, 2018
[16] Zhi Chen, Yijie Bei, and Cynthia Rudin, Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2(12):772-782 2020.
10

---

Published conference paper at ICLR 2024
[17] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning, pages 5338-5345 PMLR, 2020.
[18] Marton Havasi, Sonali Parbhoo, and Finale Doshi Velez, Addressing leakage in concept bottleneck models. Advances in Neural Information Processing Systems, 35:23386-23397, 2022.
[19] Andrei Margeloiu, Matthew Ashman, Umang Bhatt, Yanzhi Chen, Mateja Jamnik, and Adrian Weller. Do concept bottleneck models learn intended? arXiv preprint arXiv:2105.04289, 2021.
[20] Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and Tsui-Wei Weng. Label-free concept bottleneck models. arXiv preprint arXiv:2304.06122 06129, 2023.
[21] Mert Yuksekgonul, Maggie Wang, and James Zou, Post-hoc concept bottleneck models. arXiv preprint arXiv:2205. 15480, 2022.
[22] Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister. and Pradeep Ravikumar. On completeness-awarc concept-based explanations in deep neural networks. Advances in neural information processing systems, 33:20554-20565, 2020.
[23] John Platt et al. Probabilistic outputs support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61 1-0,,,1,,,, 1999.
[24] Katherine Maeve Collins, Matthew Barker, Mateo Espinosa Zarlenga, Naveen Raman, Umang Bhatt, Mateja Jamnik, Ili Sucholutsky, Adrian Weller, and Krishnamurthy Dvijotham. Human uncertainty in concept-based ai systems. In Proceedings the 2023 AAAI/ACM Conference Al, Ethics, and Society. pages (699-89,.00 2023.
[25] Jeremy Kawahara, Sara Daneshvar, Giuseppe Argenziano, Ghassan Hamarneh. Seven-point checklist and skin lesion classification using multitask multimodal neural nets. IEEE journal of biomedical and health informatics, 23(2):538-546, 2018.
[26] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. Technical report, California Institute of Technology, 2011.
[27] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, Zbigniew Wojna. Rethinking inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818 --226,, 2016.
[28] Erik Jones, Shiori Sagawa, Pang Wei Koh, Ananya Kumar, and Percy Liang. Selective classification magnify disparities across groups. In Proceedings of the International Conference Learning Representations (ICLR), 2021.
11


Reviews:
[{"review_id": "VWmurjxnPM", "reviewer": "Reviewer_wCU7", "summary": "The authors present an approach to do selective classification in deep learning with concepts, by constructing a concept bottleneck model where the front end model can make predictions given soft concepts and leverage concept confirmations to improve coverage and performance under abstention.", "questions": "In the introduction its mentioned the front model can make predictions given soft concepts, however later in the text its mentioned in Section 2 under: `Propagating Concept Uncertaininty` its mentioned the front-end model requires hard concepts as inputs, which is not very clear?\n\nIn Introduction, its not very clear why the two objectives would conflict with other, if there are papers to cite that would help to make the claim stronger?\n\nHow does the choice of models to more complex architectures change the performance of the system?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The authors provide a good motivation and introduction. Authors also provide emperical validations on multiple datasets. The problem statement is very relevant to practical problems and provide an insight into how to automate classification tasks by making it safe and interpretable.", "weaknesses": "The writing and flow could be improved better, some of them are raised in questions below. Table 1 is referenced in Section 1, however what the columns means is defined only in Section 2, which makes it harder to read the table meaning. \n\nIt would also be better to provide more details in the evaluation dataset around what each datasets means, and some statistics around it.\n\nIn my opinion the paper lacks novelty in terms of the innovation, and answers to the questions raised would help to understand better. Its not very clear about dataset statistics and how it changes and aligns with the interpretations that are presented.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uVUSBFkXNO", "reviewer": "Reviewer_3F5i", "summary": "The authors propose the use of a concept bottleneck model as input for selective classification. Moreover, they propose a greedy algorithm to select concepts to be confirmed by human experts, with the objective to increase the coverage of the selective classifier while guaranteeing a minimum accuracy level of the selective classifier. They evaluate their method with competitive baselines using both synthetic and real datasets.", "questions": "1. Assuming that there is (small) calibration error of the predictions of the classifier, how would the results of proposition 4 change? \n2. It seems that abstention happens when $\\bar{y}_i = \\tau$. Could one also assume that abstention happens when $\\bar{y}_i \\in( \\tau_1, \\tau_2)$, that is when the prediction of the classifier is within some range? How could this affect the results of proposition 4, as well as the accuracy guarantees assuming a not perfectly calibrated classifier?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The work appears to be the first to use concept bottleneck models to capture the uncertainty of the entire model for selective classification. Moreover, the idea of getting human feedback to confirm concepts to improve selective classification is quite interesting and adds to the increasing literature of human-in-the-loop algorithms. \n\nThe paper is very well organized, has a clear structure, and is nicely written. The authors clearly state their contributions as well as the assumptions of their method. They also provide a detailed description of the experimental setup and provide the code for reproducibility in an anonymized repo. The experimental evaluation seems comprehensive including experiments with on both synthetic and real datasets, as well as a  robustness analysis under violations of the Assumptions 1 and 2.", "weaknesses": "Even though the meaning of coverage might be clear to experts in selective classification, it might be helpful to include a high level definition of coverage in the introduction, so that it is clear for a broader ML audience.  \n\nIn Proposition 4, the authors assume a perfectly calibrated predictor. However, in practice, perfect calibration is impossible. As a results, it would be useful to include theoretical results that complement proposition 4 that account for the calibration error a classifier.     \n\nStyle/Typos:\n1. Figure 3 has no caption.\n2. The style of citations and captions of tables and figures does not follow the ICRL author instructions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PClmqGeQy8", "reviewer": "Reviewer_msQs", "summary": "The authors propose a classification system that uses the combined approaches of a conceptual bottleneck and abstaining outputs to increase the reliability of models. The conceptual bottleneck approach trains a classification model for each concept identified in the training data. The end-model is a classifier that uses the presence or absence of concepts to make the target classification. The abstain mechanism allows the end-model to abstain from prediction.  When the model is uncertain about the presence of a concept, it may query the user for confirmation, thereby increasing trust and performance. Concept uncertainies are propagated through the end-model by using concept identification model scores as probabilities and sampling over potential concept vectors. This also improves performance.", "questions": "Can a deeper analysis of the consequences of abstaining be provided? Abstaining almost always improves average performance on the remaining predictions. Reporting the average is almost illusory, since those non-abstain predictions would have been correct or incorrect regardless. Rather, there is a real cost associated with refusing to provide an answer. The benefit is that the model reduces risk of error, but the costs are application dependent. How can we think about these costs in a constructive manner?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The three strengths of the proposed approach are a functional abstaining method, requests for confirmation, and uncertainty propagation. Together these methods raise a classification model to something that is more intelligent, capable of some corrective action when faced with unusual inputs.", "weaknesses": "1. The uncertainty propagation methodology doesn't seem computationally efficient.\n2. The performance of the default classifier (always predict majority class, uniformly randomly abstain) ought to be included in Table 2. The default performance ought to always be presented when using accuracy as a performance metric.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WWPCitXMwH", "reviewer": "Reviewer_1z66", "summary": "The paper introduces a perspective on selective classification within deep learning using concepts. It suggests a strategy for balancing accuracy and coverage by abstaining from predictions in situations where errors can be costly. The proposed approach involves creating a concept bottleneck model, enabling the front-end model to use soft concepts and improving coverage and performance through concept confirmation. The paper presents techniques for handling uncertainty and pinpointing concepts for confirmation.", "questions": "stated above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper exhibits good clarity in its articulation, with ideas clearly presented and structured in an organized manner. Exploring the integration of user feedback into ML models to enhance accuracy and ensure broad coverage is intriguing and holds significance for ML models in real life usage. Furthermore, the paper touches interpretability in machine learning, which is an important aspect for ML models in real life.", "weaknesses": "1. The abstract should be expanded to encompass key concepts that effectively summarize the paper's contributions. In the introduction, the authors emphasize the significance of interpretability and the challenges it poses in achieving high accuracy. By including these vital points in the abstract, the paper can provide a more comprehensive overview of its content and contributions.\n\n2. Regarding the abstention process, it appears to be based on a prediction probability threshold, where if the probability is lower than the threshold, the prediction is abstained? How does it different from a decision threshold used by the models? Can authors clarify that?\n\n3. In the results and discussion section, there's limited exploration and commentary on the impact of the solution on system accuracy, as seen in Table 2. Notably, the confirmation budget appears to have a limited effect on datasets like \"noisyconcepts25\" and \"warbler\" compared to others. The paper can delve into the reasons behind this discrepancy.\n\n4. In real-world applications of this solution, questions about the ease of concept approval and handling conflicting user feedback arise. While these aspects may be considered out of scope, addressing them would be beneficial for evaluating the practicality of implementing this approach in real-world scenarios. This is particularly important when considering the potential challenges of user feedback and conflicting inputs in such applications.\n\nMinor things:\nPage 4, confirm. we —> replace . with comma\nSection 4.2, Table Table 2 —> Table 2\nShouldn’t Table 2 rather be labelled as Figure 2?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)