Deferring Concept Bottleneck Models: Learning to Defer Interventions to Inaccurate Experts
Authors: Andrea Pugnana
Riccardo Massidda
Francesco Giannini
Pietro Barbiero
Mateo Espinosa Zarlenga
Roberto Pellungrini
Gabriele Dominici
Fosca Giannotti
Davide Bacciu
Venue: NeurIPS 2025 poster ()
Keywords: Concept Bottleneck Models
Learning to Defer
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: Concept Bottleneck Models (CBMs) are interpretable machine learning models that ground their predictions on human-understandable concepts, allowing for targeted interventions in their decision-making process. However, when intervened on, CBMs assume the availability of humans that can identify the need to intervene and always provide correct interventions. Both assumptions are unrealistic and impractical, considering labor costs and human error-proneness. In contrast, Learning to Defer (L2D) extends supervised learning by allowing machine learning models to identify cases where a human is more likely to be correct than the model, thus leading to deferring systems with improved performance. In this work, we gain inspiration from L2D and propose Deferring CBMs (DCBMs), a novel framework that allows CBMs to learn when an intervention is needed. To this end, we model DCBMs as a composition of deferring systems and derive a consistent L2D loss to train them. Moreover, by relying on a CBM architecture, DCBMs can explain the reasons for deferring on the final task. Our results show that DCBMs can achieve high predictive performance and interpretability by deferring only when needed.

Deferring Concept Bottleneck Models: Learning to Defer Interventions to Inaccurate Experts
Andrea Pugnana" University of Trento
Riccardo Massidda"
Francesco Giannini Scuola Normale Superiore
Pietro Barbiero IBM Research
University of Pisa
Mateo Espinosa Zarlenga
Roberto Pellungrini
Gabriele Dominici
University of Cambridge University of Oxford
Scuola Normale Superiore
USI
Fosca Giannotti
Davide Bacciu University of Pisa
Scuola Normale Superiore
Abstract
Concept Bottleneck Models (CBMs) are interpretable machine learning models that ground their predictions targeted interventions their decision- maaiin process. However, when intervened on, CBMs assume the availability of humans that can identify the need to intervene and always provide correct interventions. Both assumptions are unrealistic and im- practical, considering labor costs and human error-proneness. In contrast, Learning to Defer (L.2D) extends supervised learning by allowing machine learning models identify cases where human is more likely be correct than the model, thus leading to deferring systems with improved performance. In this work, we gain inspiration from L2D and propose Deferring CBMs (DCBMs), novel framework that allows CBMs to learn when an intervention is needed. To this end, we model DCBMs as composition of deferring systems and derive a consistent L.2 loss train them. Moreover, by relying reasons for deferring on the final task. Our results show that DCBMs can achieve high predictive performance and interpretability by deferring only when needed.
underrstaddtt concepts, allowing for
CBM architecture, DCBMs can explain the
Introduction
Concept Bottleneck Models (CBMs) Koh et al. 2020 (ML) models that incorporate human- caimann-nterpretablle concepts as part of their training and predictive process. At test time, CBMs enable experts correct any of their intermediate concepts' values, potentially triggering a change to the CBM's task prediction. This fosters a collaborative interaction between humans and Al systems, where CBM may improve its accuracy when deployed with the support an expert. However, CBMs suffer from few shortcomings: first, increasing inter- pretability often comes at the expense of predictive accuracy, leading to an interpretabillty trade-off (Zarlenga et al., [022];; second, CBMs often assume that their set of concepts can fully predict the final task (i.e., they are complete (Yeh et al. 2020 third, CBMs assume that human interventions are infallible, an siplliiliiiiitinnn that does not reflect the real world where human experts may introduce errors. be unaware of their own potential weaknesses, and have specific cub-expertise (Rastogi et al. [022]. These practical limitations muddle the effects and dynamics of the human-Al collaboration expected when using CBMs.
family of interpretable machine learning
Equal Contribution (andrea. wuw.rraaaannnaannnnniiie riccardo. www..mmaadddddddddaaaaaaaadd .niiiiiiiiipi.....
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Machine
Concepts
Wet
Lego
Mc
My
Wis Noe
Lego
A
A
Good Day
Fi
Food
Deg
Human
Cat
Concepta
Homan
Label
Figure 1: DCBM: Given an input, the concept predictors Mc output either a concept's value or defer its prediction to human (i.e., they predict L). Next, the deferring system Åc outputs the human labels only on the deferred concepts, returning the system's 's predictions otherwise, The same applies to the final task, where the task classifier My is an input of a dedicated deferring system Ay. DCBMs can be trained by considering the cost of deferring, thus regulating the expected number of human deferrals.
To address the complex dynamics of human-in-the-loop interactions, Learning to Defer (L.2D) has been introduced as an extension of supervised learning [Madras al.) 2018 Okati et al. 2021 Mozannar and Sontag 2020). In L2D, ML models can delegate challenging instances to human experts, enhancing human-Al team collaboration and outperforming both the ML models and the human experts [Mozannar et al. 2023). Notably, conventional L2D approaches have been applied to single-clas classification tasks are typically opaque, providing little insight into the reasons for deferring decisions [Ruggieri and Pugnana 2025].
In this work, introduce Deferring Concept Bottleneck Models (DCBMs), novel class of models enabling learning to defer on CBMs (Figure D A key advantage of DCBMs is their ability to effectively learn when concept task prediction could benefit from human intervention. To the best of our knowledge, DCBM represents the first more transparent human-AI collaborations. Moreover, resorting L2D, DCBMs introduce another variable to the accuracy-iterpretaallly trade-off, i.c. the so-called coverage, which measures the percentage of times the ML model provides the prediction. Indeed, by allowing DCBMs defer difficult cases to the human, one can achieve high accuracy and interpretability at the cost of deferring more to the human. Summarizing, our contributions are the following:
seaeettaabb---dessiie deferring system, enabling
We introduce the Deferring Concept Bottleneck Model, a novel interpretable model capable of autonomously deferring both its intermediate concepts and final task predictions (Section [.1))
2. We propose new deferral-aware loss for CBMs (Section 3 3.2 and prove that it is consistent surrogate loss w.r.t. the intractable zero-one loss on the deferral procedure (Section (33).
3. We experimentally show how DCBMs react varying costs and different human-accurage accuracy degrees for defer (Section 4). Moreover, DCBMs can significantly improve concept-incomplete tasks. Finally, we demonstrate how DCBMs can produce concept-based explanations for their final task deferrals by exploiting their imterpretable-b--despp architecture.
We organize the rest of the paper follows. In Section and L.D. Then, in Section 3 we propose DCBMs and prove that their loss function consistent with the L.2 problem. Next, in Section[4] we report empirical analysis highlighting the advantages of DCBMs. Finally, we discuss related works in Section 5 and summarize our work in Section 6
2 introduce the background CBMs
2 Background
Given a variable V, we denote domain as D(V), and realization as v D(V). Similarly, we use bold for sets of variables V and their multi variate realizations v E D(V).

---

Concept Bottleneck Models. Concept-basse based models are interpretable architectures that explain
their predictions using high-level units of information known as "concepts" [Kmm et al. 2018 Chen
et al. 2020, Marconato et al., 2022 Kim et al. 2023 Barbiero et al., 2023 Oikarinen et al. 2023
Bortolotti et al. |025]. Most these approaches can be formulated as Concept Bottleneck Model
(CBM) |Koh et al. 2020], an architecture where predictions are made by composing (i) concept
encoder g D(X) - DCC) that maps samples æ D(X) C Rd (e.g., pixels) to a set of ne concepts E D(C) = {0, 1 (e.g.. "red", "round"), and (ii) task predictor f D(C) + D(Y) that maps predicted concepts to a set of ny tasks y E D(Y) = {0, 1 (e.g., "apple", ", "pear").
CBMs can be trained (a) independently. where 9 and f are trained separately and later combined; (b) sequentially, where g is trained first, and its output is used to train f: or jointly where g and are trained together. All of these training paradigms operate under the assumption that the training concept labels are complete, meaning they are sufficient to predict the tasks y Yeh et al. 2020).
Learning to Defer.
Learning to Defer (L2D) [Madras al. 2018 combines human expert's
knowledge, modeled as a given and non-traie trainable predictor h: D(X) + D(Y), together with a learnable classifier m: ( - D(Y) u { over |Y| + classes, where the additional class, denoted as I,, stands for the deferral decision. We define a deferring system as a pair A = (m, h) s.t.
m(x) h(x))
if m(x)) otherwise.
Aax)=
deferring system is a human-AI team specifying who should predict between the human and the ML model. We stress here that the human predictions might differ from the ground-truth label, and thus trivially deferring each instance might not be optimal. According to Mozannar and Sontag 2020), L2D can be formalized as risk minimization problem of the following zero-one loss:
min IE Pex.y.. [m(a)= " (m(a))v) ME.
where Pxx, y. f)) is the distribution over (input, output, human (amman-prediction) triplets, and M is the hypothesis spaces for the model m. Since directly optimizing Equation (I) is intractable, many consistent surrogate losse:"l been proposed |Mzannnrr and Sontag 2020 Verma and Nalisnick 2022 Cao al. 2023 Charusaie al. 2022 to train single-task classifiers over the |Y| + classes. In a deferring system, the coverage counts the number of instances predicted by the model without
deferring to a human, i.e., given a dataset (x) 1= it corresponds 1/N Σ [(m(Z(0))
3 Deferring Concept Bottleneck Models
In this section, we first introduce Deferring Concept Bottleneck Models (DCBMs), a novel graphical probabilistic model for which we also define the exact but intractable lcarning-to-defer optimization problem (Section 3. Then, we introduce the surrogate loss and we show how can be derived as the maximum likelihood of our graphical model (Section 3.2p. Finally, we prove that our formulation results in a valid consistent loss for the L.2D problem (Section 3.3) and we discuss how the loss' consistency can be ensured, while efficiently training DCBMs (Section 3.4).
3.1 Model Formulation
As in CBMs, we consider the problem of predicting both concept variables that directly depend on the input and task variables that are conditionally independent of the input given the concepts. We define DCBMs as an extension of CBMs, where each concept variable C' E C' and each task variable Y E Y is dealt with as separate deferring system. Similar to CBMs, a DCBM can be framed as a probabilistic graphical model, with the difference that both concepts and tasks depend on human predictions when the model defers (Figure
Let the set of concepts and task variables be V CCU CUY. Y. We assign an expert Hy and a model My: D(Zv) + [nv] U {1} to each variable V E V. Here, the output space consists of ny + 1 classes, including the deferral choice 1, and either Zy = X, V E C, or Zy = C.
Let el and E be loss functions. E' is consistent surrogate of E whenever arg min E' arg min f.
3

---

cc
0Y
X
Mc
My
Y
Hc
Hy
CEC
YEY
A DCBM is Bayesian Network where inputs X, concepts C, tasks Y. and human labels are observed variables (in gray). As represented by the plate notation [Koller and Friedman 2009), assign human expert and latent model to each variable. We incorporate the deferral decision the model through a dedicated output, denoted as M Here, we learn each model Mv's parameters Oy via maximum likelihood.
Figure 2:
V E Y. Similarly, we denote the ground-truth output as kv, which is to be intended as the label of either concept or task.
In contrast to traditional L2D setups, in a DCBM we need to train a model composed of several for each concept and task variable. Hence, our objective would idcally minimize the number of mistakes made by all the deferring systems. This can be expressed through the following multi-variate zero-one loss, where model the cost of each deferral via hyperparameter E [0, 1]:
deferring systems,
Definition (Multivariate Zero-One Loss). Given set of variables V and set of deferring systems
A = (Ay (mv, hv)) Vvvv parameterized by set of parameters O {Bv (Bvirev. we define the multivariate zero-one loss as
Σ (my (mmy/a/// (mv(kv) + [n
(A [bv
(2)
VEV
where zy and ky are the realizations inputs and outputs, respectively, of each deferring system Av.
3.2 Maximum Likelihood and Surrogate Loss
By deriving the negative ligeliidl from our probabilistic formulation of DCBMs (Figure 2}. we can treat the maximum likelihood estimation of the parameters minimization problem. In this way, we obtain a loss function composed of two terms. Intuitively, the first term directly rewards the classifier for predicting the ground-truth class, while the second term rewards the model for deferring whenever the human prediction is correct.
Proposition 3.1 (Maximum Likelihood of DCBM), Let 0 be the parameters of a DCBM. Then, we can obtain the most likely parameters ê given observations on the inputs æ, the concepts c, the human h. and the task y, by minimizing the following loss function:
E(O c)). = Σ (a(zzvv kv) [[vv= V ((((zv;
(3)
VEV
where q( : D(Zv) Kv+1 returns the logits of the model My given zv D(Zv) and I (g((v;; 8y k k)))) is the negative log log-probability of the class k given the logits q(zv:Ov).
Proof. We report the proof in Appendix A.1
The negative log- ligellkkioood we derived in Equation 3 does not take into account the cost of deferring. In this way, scenarios where the human has a significant advantage, we can expect the model to underfit and almost always defer to the human [Mozannar et al. 2023). To overcome these limitations, we define a penalized loss function by constraining the parameters of the model to enforce two additional conditions: (1) the model should not always defer when the human is correct, and (2) when

---

the human is not correct, the model should not defer. We report the formalization of the constrained optimization problem Appendix A.22 and hereby report the resulting penalized loss,
e(O x, x.c.y,
Σ v()(zv:Br)..)
Vev
(1 A) hy) ((g(zv:
+ X [vv
Σ Ol),
kE[K]
where À E [0, 1] is an hyperparameter trading-off between deferrals and machine learning decisions. In practice, the negative log-proe pyy-prbbabllity ((((zv; (v)(: kv) = - log P(My = kv; 0v) of a class ky
according to the machine learning model My corresponds the usual cross-entropy formulation
exp(1(z)))
v (g(2),)) = log EMe[K+y exp(o(z)x)
(5)
The derivation of the negative log-likelihook (Equation B and its penalized counterpart (Equation costitute an original contribution of this work. We further notice that in the univariate scenario our formulation collapses to known formulations from the L2D literature [Mozannar and Sontag 2020, Eq. 6]. In this way. we first establish clear connection between the maximum likelihood problem and the learning to defer task that, to the best of our knowledge, has not been previously identified in the literature. Further, we empirically consider also different formulations of the negative log- pog-probability v from the L2D literature - see Table D in Appendix A..2 for viable alternatives.
3.3 Loss Consistency
The multivariate scenario exacerbates the fact that always deferring to a human might not be the right choice, as the human's 's feedback may be incorrect or costly, and propagate such error. Therefore, to ensure that our model effectively defers to the human only when needed, we have to prove that the cost-free loss (Equation 3 and the penalized loss (Fquation 4 of a DCBM are consistent surrogates of the ideal multivariate zero-one loss (Fquation 2 We prove this by first showing that the sum of consistent losses on deferring systems with distinct parameters is consistent for the whole system.
Lemma 3.2. Let Cg l,, ft fm be possibly distinct loss functions. Assume that, for every i E (1...., m). 1. Eg l RD: + RR being E2 a consistent surrogate of l.. Then E' Rn + R, with n1 + +... Nmm and E' (()0,.... ,m,)00))))))))))) 0m))= Eie (,(0,) is a consistent surrogate of { E with E(0 ..... mm) Σm ((O:).
Proof. We report the proof in Appendix
A.3
Theorem 3.3. The cost-free loss in Equation and the DCBM penalized loss in Equation are surrogate consistent losses of the multivariate zero-one loss of Equation when V = -CYY. and == 0 and 0, respectively.
Proof. We report the proof in Appendix A.4
L
Hence, under appropriate assumptions (whose practicalities we discuss the next subsection), minimizing our novel surrogate losses corresponds to minimizing exact multivariate zero-one loss.
3.4 Consistent Training of DCBMs
Theorem 3.3 ensures the consistency of our overall formulation, under specific assumption on the loss functions being summed together: they should depend on disjoint sets of parameters. In essence, there are two main requirements to ensure consistency while training DCBM. First, the model has to be trained independently, so that no information flows from the tasks' losses to the concepts' losses. Notably, independent training of CBMs is known to slightly decrease the performance compared to jointly training CBMs Koh et al. [2020]. However, independent training avoids the problem of concept leakage Mahinpei et al. [202]] Havasi et al. [2022). inherent to jointly trained models, thus maintaining the interpretability of the outcomes. For this reason, we focus on independently trained
5

---

models here and discuss additional experiments on jointly trained models, showing similar results to those seen for their independent counterparts, in Appendix E
The second requirement concerns concept and task predictors, which should not share their parameters in the DCBM's architecture. Parameter sharing is common in CBMs, especially for computer vision tasks (Zarlenga et al., [022], where an encoder produces a latent representation from the input space that is then fed to the concept predictors. To enable this in applications where parameter sharing is beneficial, we take the following two-step approach: first, we train an encoder to predict either all the concepts or the final task from the input features. Then, we freeze this encoder, discard the learned predictors, and independently train the concept predictors on the encoder's 's latent representation and the task predictor on the concepts using our consistent L2D loss. Still, for completeness' sake, we evaluate DCBMs when they share parameters across classifiers in Appendix E
4 Experimental Evaluation
Our experimental analysi aims answer the following research questions:
Q1 Does deferring to possibly imperfect human improve the performance of independently trained CBM-based approaches?
Q2 Does deferring mitigate the lack of completeness of a set of concepts for predicting task?
Q3 Can DCBMs help to interpret why task classification was deferred?
4.1 Experimental Settings
Datasets. We perform our analysis on two real-world datasets: cifar10-h [Peterson et al. 2019
and CUB Wah et al.. 2011]. The cifari0-D dataset is modified version of the cifar10 dataset Krizhevsky et al. 2009] containing 10,000 images with both ground-truth and human-annotated labels. We adapted for our scenario by adding as annotated concepts the 16 "superclass" concepts defined by Oikarinen et al. [2023] for each class. As human annotations are missing for the concepts, we treat humans as oracles on the concepts. Finally, CUB is dataset commonly used for image classification with CBMs. We consider the complete set of 112 concepts used by Koh et al. [2020]. Since the dataset reports annotator uncertainty on the concepts, we use them to produce random human concept labels as done by Collins et al. [2023] (Appendix B In the CUB task label, however, we treat humans as oracles. Finally, we employ the synthetic complet completeness (Laguna et al. 2024]
dataset to study possible variants of our method, whose results we report in Appendix E
Methods. We compare a complete DCBM architecture (DCBM) with some ablated variants and baselines. In particular, we consider the following baselines: (i) black-box model trained with standard supervised learning on the final task only (BB) (ii) a standard CBM without the deferring option (CBM). To evaluate the effect of deferring on concepts and tasks, we also compare with the following ablations: a DCBM that can not defer on the final task (DCBM-NT) and a DCBM that can not defer on the concepts (DCBM- -C). In all the datasets, we train the models using the sfate-of-the-ant Asymmetric Softmax [Cao al. 2023 ASM] parameterization of the negative-log -ggative-lo--likelhoood likelihood in our loss functions. We provide further details on the adopted architectures and the experimental setup in Appendix B We report in Appendix F results for other losses on the completenes dataset.
We report four main metrics: the accuracy on the final task (AccTask), the average accuracy among all concepts (AccConc), the coverage of the model on the final task (CovTask), which counts how many times the model directly classified the task label, and the average coverage on all concepts (CovConc), which is the percentage of concepts that are not deferred.
Metries.
4.2 Experimental Results
Q1: Improving CBM's Performance with Deferring. We study the performance improvement on CBMs the CUB dataset. First, consider the ideal scenario where the human predictions match the ground-truth (Figure Ba). Then, we exploit the human uncertainty annotations on the CUB dataset to study scenario where the human might wrongly classify concept (Figure 3b).
We provide the code for reproducing our experiments at wtthttss://gggie http:/hwww..cmmm//deruugg//////a

---

AncTrole
1206
CheTesk
80
ConCConc
07
800
000
3.3
8.2
(a) CUB: Oracle Human Concepts
-B-
ECBI
012
AncTaak
CanTark
CooConi
0.4
0.7
00
616
Dater Cost A
Defer Cost
Defor Cost
Defor Cost A
(b) CUB: Uncertain Human Annotations on Concepts
000
007
CooCone
804
0.0
00
0.5
0.5
113
0.4
112
3.2
0.4
02:
83
65
Deter Cost
Defer Cost
Defor Cost
Defer Cost
(c) cifar10-h: Oracle Human on Concepts
Figure 3: Results CUB dataset when human experts have perfect accuracy the concepts
(Figure Ba); on CUB dataset when human experts do not have perfect accuracy on the concepts (Figure 3by; on cifar10-h dataset when human experts have perfect accuracy on the concepts but not on the final task (Figure(3c) 3c) We report each metric's average and standard deviation as we increase the defer costs X. The CBM and BB baselines constant they are independent of the defer cost. DCBM outperform competing baselines for lower deferral costs X. Increasing the cost À reduces the number of deferrals and decreases the DCBM performance up the standard CBM performance.
In the first scenario, when the defer costs are significantly low (A < 0.05), the deferring systems tend to over-rely on the human and thus the coverage of the machine learning model is zero for both concepts and tasks. As expected, increasing the defer cost increases both the coverage on the task (CovTask) and on the concepts (CovConc). At the same time, it also reduces the accuracy of the prediction, which is, however, still over the standard CBM baseline without deferring capabilities. In summary, the performance of the ablated (DCBM-NC, DCBM-NT) and the full model (DCBM) tend to those of the standard non-deferring CBM for higher defer costs, while improving performance for lower defer costs. Therefore, standard practice in the L2D literature [Wei et al.) 2024], by leveraging the defer cost, we can ensure that the deferring systems do not over-rely on the human.
Notably, in the scenario where humans might wrongly classify some of the concept labels (Figure 3b the results emphasize how the human expert's s ability to correctly predict concepts affects the DCBM. In the presence of potentially incorrect humans on the concepts, the model correctly learns when there is no advantage in deferring concept predictions to humans.
detail, when the defer cost zero (A = 0), DCBM still has non-zero coverage (CovCone RS 0.2), meaning that it is not deferring 20% of instances to humans, even if it would "free" to do so. We provide additional results in Appendix E. showing how competitive intervention strategies fail to capture this aspect.

---

10 Rebson Shorp Iwlnd Sports 17
Jetered Corcepts: Numan
Delened Delened Coreepts NL
13
Delened Intersettonn:
30-
Perlect Intevertians
42
C
undeppan opppee cree Stack Whir Cesse phace tesermaan
cotet thel you grapie shak Case thet
Leete and coloer slack you coon Black
unseeppert get dhvan Cone
Coler Bhot
sacd colen But
Nesan sher teeed Sparns La Coess g Spansse porpln Fecch
Neteo thai taledd dppero cont goee ponninn Permen
Nasson ovep Dilee Spanss La tp Speeno popple Fechh iitesiee Sp gparoo
Detes
Figure 4: Interpretation of a DCBM with defer cost = 0.1 on an input sample. From left to right: (a) an example of an image from the CUB dataset; (b) the concepts that the model has deferred with the estimated probability, green bars stand for when the human correctly predicts the concept, red otherwise; (c) the estimated probability of each deferred concept being true according to the machine learning model, green bars stand for when the ML would have correctly predicted the concept, red otherwise; (d) the estimated probability of top-5 final task labels after deferring the concepts the human (standard DCBM behavior); (e) the estimated probability of top-5 final task labels without deferring the concepts to the human; (f) the estimated probability of top-5 final task labels from the ground-truth concepts.
Therefore, the DCBMs automatically adjust the coverage depending not only on the defer cost but also on the human competence, extending traditional CBMs to account also for incorrect humans.
Q2: Addressing Incompleteness through Human-AI collaboration. We investigate the impact of deferring in an incomplete scenario, where the set of concepts is not sufficient to distinguish between two or more classes. In practice, this means that we cannot learn good task classifier for those instances with the same concept-level representation dataset. better choice would then be defer human, who can distinguish between the two classes by also employing input data or additional information.
such as cat and deer in the cifar100 10-h
We validate our hypothesis cifar10-h (see Figure 3cb: results show that for low defer costs, the DCBM outperforms other baselines, with an AccTask Ro 0.984 at = 0 and a CovTask s 0.816. Furthermore, deferring on both the concepts and the task proves be better than deferring only on one of the two, as shown by the results of our ablated models DCBM-NC and DCBM-NT.
Since cifar10-h the human expert can make mistakes on the final task, our DCBM model correctly identifies that always deferring to the human would not be optimal: even when deferring to human would be "free" (A 0.0), the DCBM has high coverage the classification task (CovTask s 0.8). The previously discussed raise in performance then comes from the DCBM correctly deferring to human only when beneficial. Indeed, shown in Appendix B deferral occurs only for cat and deer instances. Moreover, increasing the defer cost À decreases the classification performance while increasing the coverage of the machine learning model. However, it is worth remarking that our DCBM still performs better than the CBM baseline even for higher costs (A Rs 0.3), where no concepts are deferred the human (CovConc 1.0) and most task classifications are performed by the machine (CovTask> R 0.8).
In nutshell, DCBMs provide useful mechanism to deploy model in an incomplete setting, addressing the risks that might arise from deploying a classifier that arbitrarily chooses one of the two (or possibly more) entangled classes.
Q3: Interpretable Learning to Defer in DCBMs. We show how DCBMs can help interpret the reasons for the final task deferring, by following similar approach to Zarlenga et al. 2023 For this purpose, we consider the CUB dataset and DCBM trained with cost À == 0.1. We discuss here an instance of bird image to be classified (Figure 5) and report further examples in Appendix C
main advantage of DCBMs their capability to identify which concepts should corrected by human intervention, without additional interventions. Since the human supervision is not perfectly accurate in the CUB dataset, the interventions might not correspond the ground-truth values (Figure/4b) Interestingly, the deferred concepts are not easy to grasp in the original image:
8

---

the underparts are partially covered by grass blades, the bill is not clearly cone-shaped, and the back not really visible the image. In general. highlighting the particular concepts on which the human should be a safer option than the MI. model favours the interpretation of the classifier. Moreover, we also stress that without deferring to the human, the machine learning model would have wrongly classified some deferred concepts (Figure 4
This example also stresses that DCBMs try to defer only when worthy and necessary, without involving humans when they are likely to make mistakes, Moreover, interventions allows us to reason on how concepts effectively lead the DCBM to predict the correct class (Figure ft). In fact, without intervening on the concepts, the final task model would have predicted a wrong label (Figure Finally, we also report how the final task model would have behaved under perfect interventions, i.e where human has access to ground -ruth concept labels, as assumed by the standard CBM literature (Figure I. As expected, the DCBM would increase own confidence in the correct label, at the cost of human supervision on all of the 1 112 concepts instead of the only five identified by the DCBM.
5 Related Works
Deferring systems. L2D, as introduced in Madras et al. 2018 is instance of hybrid decision-
making where humans oversee machines. Since directly optimizing Equation is NP-hard even in
[
simple settings [Mozannar al. 2023). Mozannar and Sontag [2020] proposed consistent surrogate
losses, which have since become the standard approach for jointly learning the deferral policy and the
ML predictor [Charusaie al., 2022 Verma and Nalisnick] 2022 Mozannar et al.) 2023 Cao et al.
2023 Liu et |0024| Wei et al. 2024]] A formal characterization of humans in the loop is provided
by Okati al. [2021]. Recent works extend the L2D problem to account for multiple human experts,
e.g., see Verma et al. (2023). Mao et al. (2023) and Cao et al. (2023), cases where the ML model is already given and not, jointly trained, c.g., [Charusaie et al.] 2022 Mao al., 2023 Montreuil et al.
2025a b), and how they relate to causal frameworks Palomba et al. 2025 Gao and Yin 2025
Concept Interventions. CBMs have seen a growth of interest in the context of concept interventions, operations that improve CBM's overall task performance in the presence of test-time human feedback. Works in this area have explored (1) how to best select which concepts to intervene on next when interventions are costly (Shin et al.) 2023 Chauhan et al., 2023 see Appendix [ for a comparison between DCBMs and Uncertainty on Concept prediction [UCP; et al. 2023]. popular and competitive strategy to perform interventions -2) (2) how to improve model's receptiveness to interventions and learn an intervention policy Zarlenga et al. 2002). and (3) how to intervene on otherwise black-box box models (Laguna al.) 2024 In particular, while policy-based methods [Chauhan eet all, 2023) rank which interventions should be prioritized to enhance the classification periormance of the model, they still require the human expert to initiate the procedure to request an intervention. By modeling the human predictive performance as done by the L,2 literature, DCBMs ask instead for interventions without further supervision at inference time. Other approaches have exploited inter-concept relationships to propagate single-concept interventions [Vandenhirtz et al. 2024 Raman
ct al. 2024| Dominici et al., 20255 and have used interventions as sources of continual learning labels Steiimannn et al. 2024]. Finally, [Sheth and Kahou [2023] and Collins et al. [2023) both
discuss notions of supervisor uncertainty, where we may be interested in modeling errors from an expert performing interventions. Nevertheless, works concept interventions fundamentally differ from our L2D-based approach in that they assume that experts themselves trigger a correction in a model's 's concept predictions. This makes it difficult for these approaches to adapt to expert-specific competencies and to be easily deployed in practice where it is desirable to know when a human should be called to intervene.
Conclusions and Future Work
This paper introduces DCBM, a novel approach that allows CBMs to defer a human without additional supervision. By training the CBM with an especially designed learning to defer loss function, a DCBM can implicitly model the predictive distribution of the human, and thus defer only on instances where the expert is more likely to be correct of the machine learning model. Moreover, we formally proved the consistency of our deferring loss function for independent training of CBMs. Our experimental results highlight that DCBMs effectively learn when to involve human, boosting

---

overall predictive performance only when the human is better than the ML model. Moreover, directly involving human helps mitigate cases where concepts are incomplete. Finally, the interpretable by-design nature of DCBMs offers ways to audit the deferring systems, showing promising results in explaining their limits.
Limitations and Future Works. We acknowledge a few limitations of our current work. First, the actual implementations of CBMs and deferring systems in real-life settings is still overlooked [Rug- gieri and Pugnana 2025 Hence, user studies are highly needed to evaluate how humans can benefit from ours and other concept-basee approaches.
Second, our theoretical approach considers concepts as independent variables. While DCBMs can be directly extended to group together sets of mutually exclusive binary concepts (thus not independent) into single multi-class concept, we are not considering more complex relationships among concepts. Extending our approach to account for hierarchical structure of concepts - as done e.g., in causal abstraction [Geiger et al. 2021 Massidda et al. 2024 is also open research direction.
Third, in this paper, we consider a single expert per concept/task and implicitly assume that the expert's ' costs are the same when deferring at the concept and task level. While our framework can be easily extended to account for different costs and multi-expert settings [Mao al. 2023), we did not explicitly investigate how these modelling choices can lead to different DCBMs. Furthermore, as all L2D approaches, our proposal models the predictive distribution of the human expert and assumes that the human will follow the same distribution at test time. Changes in how the human tackles the same task might affect the performance of the overall human-Al system. Detecting such distribution shifts and integrating with continual learning strategies (Parisi et al. 2019 could then help real-worll applications and constitutes a promising research line for L.D methods in general.
Finally, regarding the explanation of deferring systems, CBMs are interpretable models that can be used to enlighten the decision process toward either class or a defer prediction. This paper used defer on concepts and interventions to provide explanations on task's 's deferral. Since several methodologies for explaining CBMs have been considered, such as those based on logical rules, DCBMs could be extended to account for different kinds of explanations.
Acknowledgments and Disclosure of Funding
We thank Martina Cinquini for initial discussion and feedback on the article.
This work has been funded by the European Union under Grant Agreement no. 101 120763 TANGO. This work has been supported by the Partnership Extended PE00000013 "FAIR Future Artificial Intelligence Research" Spoke 1 "Huuan--cetters Science and technology for the eXplanation of Al decision making". This work has been supported by TAILOR, a project funded by EU Horizon 2020 research and innovation programme under GA No 952215. This work has been partially supported by IMAGINE, a project funded by the Swiss National Science Foundation (No. 224226). GD acknowledges support from the European Union's 's Horizon Europe project Smar(CHANGE (No. 101080965). MEZ acknowledges that the majority of this work was done with the support of the Gates Cambridge Trust via Gates Cambridge Scholarship.
AI" and ERC-2018-ADG G.A. 834756 "XAI:
References
Pietro Barbiero, Gabriele Ciravegna, Francesco Giannini, Mateo Espinosa Zarlenga, Lucie Charlotte Magister, Alberto Tonda, Pietro Lio, Frédéric Precioso, Mateja Jamnik, and Giuseppe Marra. Interpretable neural-symbolic concept reasoning. In ICMI., volume 202 of Proceedings of Machine Learning Research, pages 1801-1825, PMLR, 2023.
Samuele Bortolotti, Emanuele Marconato, Paolo Morettin, Andrea Passerini, and Stefano Teso. Shortcuts and identifiability in concept-based models from neuro-symbolic lens. In NeurIPS, 2025. 3
Yuzhou Cao, Hussein Mozannar, Lei Feng, Hongxin Wei, and Bo An. In defense of softmax parametrization for calibrated and consistent learning defer. In NeurIPS, 2023. 3 6 9 19
10

---

Mohammad-Amin Amin Charusaie, Hussein Mozannar, David A. Sontag, and Samira Samadi. Sample efficient learning predictors that complement humans. In ICML, volume 162 of Proceedings of Machine Learning Research, pages 2972-3005, PMLR, 2022.
Kushal Chauhan, Rishabh Tiwari, Jan Freyberg, Pradeep Shenoy, and Krishnamurthy Dvijotham.
Interactive concept bottleneck models. In AAAI, pages 5948-5955. AAAI Press, 2023. y
Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition.
Nature Machine Intelligence, 2(12):772-782, 2020.
g
Katherine Maeve Collins, Matthew Barker, Mateo Espinosa Zarlenga, Naveen Raman, Umang Bhatt, Mateja Jamnik, Ilia Sucholutsky, Adrian Weller, and Krishnamurthy Dvijotham. Human uncertainty in concept-based AI systems. In AIES, pages 869-889. ACM, 2023. 6
Gabriele Dominici, Pietro Barbiero, Mateo Espinosa Zarlenga, Alberto Termine, Martin Gjoreski, Giuseppe Marra, and Marc Langheinrich. Causal concept graph models: Beyond causal opacity in deep learning. In ICLR. OpenReview.net, 2025. 9
Ruijiang Gao and Mingzhang Yin. Confounding-robust deferral policy learning. In AAAI, pages
AAAI Press, 2025. 9
Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. In NeurlPS, pages 9574-9586, 2021. I0
Marton Havasi, Sonali Parbhoo, and Finale Deshi-Velez. Addressing leakage in concept bottleneck models. In NeurlPS, 2022. 5
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image
recognition. CVPR, pages 770-778, IEEE Computer Society, 2016. 20
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie J. Cai, James Wexler, Fernanda B. Viégas, and Rory Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In ICML, volume 80 of Proceedings of Machine Learning Research, pages 2673-2682. PMLR, 2018.
Eunji Kim, Dahuin Jung, Sangha Park, Siwon Kim, and Sungroh Yoon. Probabilistic concept bottleneck models. In ICML, volume 202 of Proceedings of Machine Learning Research, pages 16521-16540. PMLR, 2023. B
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 21
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In ICML, volume 119 of Proceedings of Machine Learning Research, pages 3338-5348. PMLR, 2020. 5
Daphne Koller and Nir Friedman. Probabilistic Graphical Models Principles and Techniques. MIT Press, 2009.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009. URL URLpttpp: http:://t/wt cs. toronto edu/
ttriz/cifar. html
20
Sonia Laguna, Ricards Marcinkevics, Moritz. Vandenhirtz, and Julia E. Vogt. Beyond concept bottleneck models: How to make black boxes intervenable? In NeurIPS, 2024. 6
Shuqi Liu, Yuzhou Cao, Qiaozhen Zhang, Lei Feng, and Bo An. Mitigating underfitting in learning to defer with consistent losses. In AISTATS, volume 238 of Proceedings of Machine Learning Research, pages 4816 4824. PMLR, 2024.
18 19 20 25
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR. OpenReview.net, 2019. 2T
David Madras, Toniann Pitassi, and Richard Zemel. Predict responsibly: Improving fairnsss and accuracy by learning defer. In NeurIPS, pages 6150-6160, 2018.

---

Anita Mahinpei, Justin Clark, Isaac Lage, Finale Doshi-Velez, and Weiwei Pan. Promises and pitfalls of black-box concept learning models. arXiv preprint arXiv:2106. 13314, 2021.
5
Anqi Mao, Christopher Mohri, Mehryar Mohri, and Yutao Zhong. Two-stage learning to defer with multiple experts. In NeurlPS, 2023.
9 10
Emanuele Marconato, Andrea Passerini, and Stefano Teso. Glancenets: Interpretable, leak-proof concept-based models. NeurIPS, 2022.
Riccardo Massidda, Sara Magliacane, and Davide Bacciu. Learning causal abstractions of linear structural causal models. In UA/, volume 244 of Proceedings of Machine Learning Research, pages 2486-25111 PMLR, 2024. I0
Yannis Montreuil, Axel Carlier, Lai Xing Ng, and Wei Tsang Ooi. Adversarial Robustness in Two-Stage Learning-tor toreeng-tt--eeeee to-Defer: Algorithms and Guarantees. In ICML. OpenReview.net, 2025a. 9
Yannis Montreuil, Yeo Shu Heng, Axel Carlier, Lai Xing Ng. and Wei Tsang Ooi. Two-stage learning-to-defer for multi-task learning. In ICML. OpenReview.net, 2025b. 9
Hussein Mozannar and David A. Sontag. Consistent estimators for learning to defer to an expert. In ICML, volume 119 of Proceedings of Machine Learning Research, pages 7076-7087. PMLR. 2020.
2
20
Hussein Mozannar, Hunter Lang, Dennis Wei, Prasanna Sattigeri, Subhro Das, and David A. Sontag. Who should predict? exact algorithms for learning defer to humans. In AISTATS, volume 206 of Proceedings of Machine Learning Research, pages 10520-10545. PMLR, 2023.
19
Tuomas P. Oikarinen, Subhro Das, Lam M. Nguyen, and Tsui-Wei Weng. Label-free concept
bottleneck models. In ICLR. OpenReview.net, 2023.
Nastaran Okati, Abir De, and Manuel Godez-Rodriggez. Differentiable learning under triage. In NeurIPS, pages 91 9140-9151, 2021. 2 9
Filippo Palomba, Andrea Pugnana, José M. Álvarez, and Salvatore Ruggieri. A causal framework for evaluating deferring systems. In AISTATS, volume 258 of Proceedings of Machine Learning Research, pages
PMLR, 2025. 9 20
German Parisi, Ronald Kemker, Jose I. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural networks, 11::54771, 2019. [O
Joshua C. Peterson, Ruairidh M. Battleday, Thomas L. Griffiths, and Olga Russakovsky. Human uncertainty makes classification more robust. In ICCV, pages 9616-9625. IEBE, 2019. 6
Naveen Raman, Mateo Espinosa Zarlenga, and Mateja Jamnik. Understanding inter-concept relation- ships in concept-based models. In ICML. OpenReview.net, 2024.
Charvi Rastogi, Yunfeng Zhang, Dennis Wei, Kush R. Varshney, Amit Dhurandhar, and Richard Tomsett. Deciding fast and slow: The role of cognitive biases in ai-assisted vocision-making. Proc.
ACM Hum. Comput. Interact., ((C3CW)):83:1-83222 2022. 0
Salvatore Ruggieri and Andrea Pugnana. Things machine learning models know that they don't 't know. In AAAI, pages
AAAI Press, 2025.
10
Ivaxi Sheth and Samira Ebrahimi Kahou, Auxiliary losses for learning generalizable concept- models. In NeurlPS, 2023. 9
Sungbin Shin, Yohan Jo, Sungsoo Ahn, and Namhoon Lee. A closer look at the intervention procedure of concept bottleneck models. In /CML, volume 202 of Proceedings of Machine Learning Research, pages 31504-31520. PMLR, 2023.
23
David Steinmann, Wolfgang Stammer, Felix Friedrich, and Kristian Kersting. Learning to intervene on concept bottlenecks. ICML. OpenReview.net, 2024. g
Moritz Vandenhirtz, Sonia Laguna, Ricards Marcinkevics, and Julia F. Vogt. Stochastic concept bottleneck models. In NewrlPS, 2024.
9
12

---

Rajeev Verma and Eric T. Nalismick. Calibrated learning to defer with one-vs-all all classifiers. In ICML. volume 162 of Proceedings of Machine Learning Research, pages 22184-22200 PMLR, 2022. 3
Rajeev Verma, Daniel Barrejón, and Eric T. Nalisnick. Learning to defer to multiple experts: Consistent surrogate losses, confidence calibration, and conformal ensembles. In AISTATS, volume 206 of Proceedings of Machine Learning Research, pages 11415-11444 PMLR, 2023.
g 19
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-ucsd birds-200-2011 I dataset. Technical Report CNS-TR-2011-00 California Institute of Technology, 2011. 6
Zixi Wei, Yuzhou Cao, and Lei Feng. Exploiting human-ai dependence for learning to defer. In
ICML. OpenReview.net, 2024.
9
Chih- Yeh, Been Kim, Sercan Ömer Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On completeness-aware concept-based explanations in deep neural networks. In NeurlPS, 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC. BMVA Press, 2016. 20
Mateo Espinosa Zarlenga, Pietro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Giannini, Michelangelo Diligenti, Zohreh Shams, Frédéric Precioso, Stefano Melacci, Adrian Weller, Pietro Lió, and Mateja Jamnik. Concept embedding models: Beyond the accuracy-explainablty trade-off. In NewrIPS, 2022.
Mateo Espinosa Zarlenga, Katie Collins, Krishnamurthy Dvijotham, Adrian Weller, Zohreh Shams, and Mateja Jamnik. Learning to receive help: Intervention-awant aware concept embedding models. In NeurIPS, 2023.
13

---

Supplementary Material
Table of Contents
Introduction
Background
3
Deferring Concept Bottleneck Models
Model Formulation
3.2
Maximum Likelihood and Surrogate Loss
3.3
Loss Consistency
3.4
Consistent Training of DCBMS
Experimental Evaluation
4.1
Experimental Settings
4.2
Experimental Results
5
Related Works
6
Conclusions and Future Work
Proofs
15
A.1 Proposition 3.1 Maximum Likelihood of DCBM
15
Regularized Optimization of DCBM
18
A.3
Lemma 3.2 Sum of Consistent Losses
19
A 4 Theorem Sum of Consistent Losses
20
Experimental Details
20
Additional Explanations
21
Additional Comparisons
D.I Defer on Task
21
D.2 Uncertainty Concept Predictions (UCP) vs DCBMs
23
Additional Results
24
14

---

A Proofs
A.1 Proposition 3.1 - Maximum Likelihood of DCBM
In this proof, we report the derivation of the maximum likelihood of the Bayesian Network corre- sponding Deferring Concept Bottleneck Model (DCBM), which we reported in Figure 2 We assume that our dataset is composed of i.i.d. samples from the joint distribution of the observable variables. Therefore, we consider the input data E D(X), ). the concept values E D(C), the task values y E DYY), ), and the human annotations on both concepts and tasks h E (H)) ). We first define the likelihood of the data by marginalizing over the latent variables, i.e., of each concept model Mc and task model My for all variables C E C and Y E E. We recall that for each concept model Mc we have one out of nc + 1 possible outcomes, where nc is the number of possible realizations of C' and the additional value accounts for the deferred decision, as in Mc T. Similarly, each task model My has ny possible outcomes. We then marginalize one variable a time from the joint likelihood, starting from an arbitrary task variable Y E Y.
C( x. x,c,y, = plr,c,y, y, 0)
(6)
=
Σ
plx,c,y,h, My = k)
(7)
+]
= -((x) Σ
plc,y, h My k x)
(8)
kE
= =p(x)
Σ
p(y = y My = k. Hy = hy)p(MM = k k = by x..c)
kE[ny+1]
(9)
plc,yrr hyy x., 0,r),
p( by x
Σ
p(Y = y My = Hy hh))p((MMM> - 0r
k][
(10)
plc.y x 0,y0)
where we employ the operator') \ to denote the removal from a of variable.
Before marginalizing the remaining variables, we focus on the sum over possible values of the model My. We can further decompose by considering whether the model value is possible value for Y or a deferral J.. According to our definition a deferring system, Y = y if and only if the model has value My Therefore, it holds that p(Y = My = k. Hy = fyy = 1 if and only if My y whenever My # 1 and p(Y = y My Hy hyy if and only if by y whenever My 1. Formally.
deferred the decision through My 1 but for the human expert holds Hy y.
Σ kE|ny+1] +1]
p(Y = y My = k, b
(11)
Σ kEIny]
p(Y y My = k, k, = k1c p(Y= = My = (, 1 0y)
(12)
Σ Fly = (My c,dy) + Is ylp(My 1 1
(13)
kE(ny]
=p(My y c, + Ilhyy vip(My 1 c,0r)
(14)
where [[[:] is the indicator function taking value one if the proposition is true, zero otherwise.
15

---

Therefore, we can apply the same decomposition to all tasks Y and rearrange terms as follows.
[(0 x. h.
(15)
=p(x)p(hr x (p((y - y c, 11hy - -/p(yy = 11 -/
x; 0000
(16)
=p(x)p(hr x.C)) II
(p(My y c, + I[hy vlp(My - |c, (c,,r)))((ee) he x. 0c)
YEY
(17)
=(x))((hr a c).ppc a, 0c)
Il
(p(My = c,,0r) +lhhy y)p(MMy = 1 c, 0y)).
YEY
(18)
Then, we can apply a similar decomposition to concepts, starting for an arbitrary concept C C.
p(c, he x, e.)c)
(19)
=E (ic,h,,ce- =c|
(20)
kE(mct1]
=p(he | a))
Σ
p(C = Mc = k,hc) = æ. 0c) plachoor æ (crc) (21)
kE|nC+1]
=p(hc (p(Mcc c (1,00,0)))))) + Il = clp(Mc = 1 (,,c).) pla,c, harc I x, (c)c))
(22)
=p(hic | x)) II (p(Mc= = 1, 0c) + Ile = clM(Mc= x 0c))
(23)
CEC
Finally, leading to the following form, which we further simplify by denoting as zv the input of each variable V E V and as v E D(V) its realization in the dataset.
C( 2, ,,,, h) = p(x)p(hh x))(hh x;, c)
(24)
II
(p(Mc = c x; 0c) + Ile = clp(Mc 1 x 0c))
CEC
(25)
II YeY
(p(My = v c, + 1hy = ylp(My = =1 1 (y).)
= =p(z)p(h x, c)
II (o(My= = v zv, 0) + 1ivvv = ulp(AVv = 1 (vv) VEV
(26)
Finally, we show that maximizing the likelihood equates to minimizing the loss function we defined in Section 3 We recall that to this end, we assume to have for each variable V V machine
learning model g(: 0v) that produces ny + 1 activations, one for each class and one additional for the defer action.
Ô = arg max C(0) x C, h.)
(27)
= arg max (( x, c)
II
pMy = u 0) + Ihhv = ulp(My = zv,
(28)
VEV
= arg max
II
pMM = - xv,0) 0) + +[hhv = u)p(Mv = (vv
(29)
VEV
arg max
Σ
Log(p(My = u 0) + Mhhv = ulp(Mv = 1 zv, Bv))
(30)
VEV
= arg max
Σ
log(p(My = u 0) + I[y = u/ log(p(MV = 1 Bv.
(31)
VEV
arg min
Σ
(og(p(My v 0)) - Tlyv = v) lo]go(p(( My = =1 0v)
(32)
VEV
= arg min Σ
((((zv; 0,), 6) + Ilay = 2) (V)((vv:V))
(33)
VEV
16

---

where (zv; 8y)) then corresponds to the standard formulation with the softmax operator, reported in Table [ In the same table, we report alternative formulations for the same object from the learning
to defer literature. Further, we can justify the transition from Equation 33 to Equation 31 since the following holds
log(p(M (My = v. 2v. + Ijy elp(My -1 zv, 0v)) 2 <l = e =v,, + Tlby = v) e]log(p(MV = 1 2v, 0v)).
(34)
17

---

A.2 Regularized Optimization of DCBM
As discussed in Section 3.1 we regularize the model to avoid trivially deferring whenever the human is correct. In this way, we can account for the cost of deferring and relegating it to the most significative cases. We formalize this intuition by requiring the log-probability of deferring when the human is correct to be smaller then zero. Formally, we define the following constraint over all variables of the deferring system
AA E V. Eech, [[yy v] log P(Mv 1 x. WEV.
- Ov]]] <0
(35)
Bech,ay |-1- Ihvv = e) log P(Mv = 1 x. 0v)]
(36)
VV...
Behh,ay [1yy u) (
O.
(37)
In practice, we treat the constraint as a regularization term controlled by an hyperparameter À E R. In particular, let
gv (c, h, x. y) Tl = ()(r
(38)
be the value of the constraint on the variable V E V. We treat the constrained optimization problem as the following regularized unconstrained problem.
min Eech,, [e(e) c, h x. y) - Σ
Eech,ay (ev h 2. y)
(39)
0
VEV
min E Ech,,,y [e(0] c, h x, 8)1 + Ec,n,
-r Σ sr(c,
(40)
0
VEV
min 2,,
((0 c, h x, y) - A Σ ov (c, h 2,
(41)
0
VEV
min e,,
Σ W(g(zv:8v),c) + I[hv =/llgv//(((2v:)))))))) AIihv u)V(g(zV; 0v),
VEV
(42)
-mnn E,
Σ VEV
V(o(zvi0v), + A)[(/r o (
(43)
Further, we show that the formulation from Liu et al. 2024 arises when explicitly constraining the model to avoid deferring whenever the human is incorrect in the training distribution in
E [Thy F P(M # Lxx 0v)] > O. By expressing the constraint in terms of log-probabilities, we get the following result.
WEV. Ech,, [hbv + log P(Mv + x. [v]0v]] > -E,
(44)
- V...
E.hhyyy
Thy # v] log Σ
P(Mv = k| (1,00v)) 0v)
> --6
(45)
kE(nv]
WV.. E.c,,
1lbv # v) Σ
log P(Mv = -k| 22,,0
-,,
(46)
AEjNv]
- VEV. Ecch,yyy
1 Ihhv vl Σ
log P(My k 1112000) <c
(47)
kE[Avl
- VE.. Ecch,yy
1lbv # v]
Σ ((((svi(v),)) <E.
(48)
kEjov]
18

---

Name
Loss Function
CB
Mozannar et 2023
@ (g(=). kj log
copteiell
log (1 +1++0 exp (- g(())))) log(1 (tg(())))))))
OVA Verma
2023
vi ()
log (1 exp ( (-u(s)e)) EVEINAAA log (1 exp (+o(s)e))
otherwise
log
explatsial Eate muptytabur maakrrr (xp()l)) sapfelelel
iF -1
ASM Cao et al. 2023
v (o(a),8)
log
Eate (S1p)
log
En Earnies (mplofa),)) ) mane caplalalg.)
Seewees meplelste J-mana a (mppot))e))
otherwise
Table I: Multiclass losses from Liu al. 2024
for positive threshold 0. Consequently, when introducing this constraints with the same penalty in the optimization problem, we obtain the following formulation
min Ee, ky 0
W()( + 2)I(RV = =o]W(g(zv;Bu))
VVEV
(49)
+ AT[hyy # v)
Σ ((g(zr;(r): ke|vvl
A.3 Lemma 3.2 - -Sum of Consistent Losses
Before proving Lemma 3.2 we prove the following result that is a fundamental property arising from the definition of the argmin function and the independence of variables.
Lemma A.1. Given f.g: A C R + R, we have
arg min (f(x) + g(v)) = ((7. v) E A²: EE arg min ((x), E arg min ((V) (*,))EA²
Proof. For the sake of simplicity we use the following shorteut:
L = arg min (f(x) + s(v)) and R = {(E D) E A2: I arg min f(x) A E arg min g(v) (22
PEA
yEA
First, we notice that in case any between f or g has no minimum in A, then the claim is trivially proved as L = R = 0. Indeed, let us assume, e.g., that f has no minimum in A. then clearly R = 0. Moreover, also L,
Indeed, if we assume by contradiction that I, + then exists (I, v) I, i.e.
(z. 77 E 2 with f() + g()) < fx) + g(y) for every (I., y) A2, By taking y ÿ and canceling g(ğ) on both sides we get that f(x) S f(x) for every x E A. Therefore f has at least a minimum (r)
in A, which is contradiction,
it must be L = 0. well.
So lets consider the case of both L + 0 and + 0. We show the double inclusion.
1. If (i, v) E then (ā) + g < f(x) glv) for every (ar, y) E A From this inequality, by taking x and canceling f())), we get y arg min gly). Identically, by taking y = 5. we get E arg min fxx) Therefore (ī, R.
2. If (a. 7) R then z e arg min fx) and ÿ arg wigminge.. gly). Namely, f(æ) S f())
for every I E A and g(g) S g(y) for every y E A. By summing on both sides, we get f(T) + g < f() + g()) for every (a, v) E A², and so (Z,7) E
Lemma 3.2
Let Cy L1, ... Ey Lm be (possibly distinct) loss functions. Assume that, for every + R. being & a consistent surrogate of 4, Then R" + R, with n1 + nm and El (01...... 0m) Σm (((),) is a consistent surrogate of L B + R. with
i E {1...., m. C l R
E(01..... ..... mm) 2m m ((0.).
19

---

Proof. The proof is a direct consequence of Lemma A.I For simplicity, we show the complete proof for m 2. To be precise, from the statement we report explicitly that, L & R + R. (2 f2 Ra -> IR and f", : R - Ih with = + n2, f' = [ + ly and f. = l + l2. We have to prove that L is a consistent surrogate of f, namely that arg mingeR (0) C arg mingeR {(0).
Let 0* (0T, 05) E 1 be a minimum of E' (the claim would be trivial in case e" has no minima). Then according Lemma A.I have:
0" arg min (0)))
arg min
(c1 + E2(02) =
DER"
(0, (2),02)ER*1t +22
(50)
{(0,, 02) E 1 01 E arg min (01) A O2 E arg min (02))
ERT Therefore of E arg mill ERR eg (01) and 0g E arg min (02). Since by hypothesis (1 l2 are consistent surrogates of &1, lg, respectively, it follows that: 01 E arg ming, (0₁) and 02 E arg min (2(02).
Finally, the proof concludes by using again Lemma A.1
0 02) E t 01 E arg min & (01)) 02 E arg min (2(02)) =
0,ER"
02ER
(51)
arg min (0) + (2(02)) = arg min ((0) (0) (2)
BER"
A.4 Theorem 1 1--uum Sum of Consistent Losses
Proof. By the previous Lemma 3.2 the sum consistent losses is consistent to the sum of the target loss functions. It is thus immediate how this applies optimization problem both for the unconstrained (Equation 3 and the penalized (Equation 4 losses. Formally,
Σ V(g(zv;0v). v) + (1 A)4(ky = () VEV
(52)
is the sum of losses consistent of the zero-one loss which reported in Equation I whenever À In fact, it corresponds to an equivalent formulation in Theorem from Mozannar and Sontag 2020 Similarly, for any other À E [0, 1]. the penalized version coincides in the single-varis variable case to the provably consistent formulation from Equation 4 in Liu et al. [2024). Therefore, the sum over different variables is consistent the sum of the zero-one loss.
B Experimental Details
Data Split. For the completeness synthetic dataset, we sample 1, 000 instances with an 80%-20% train-test split ratio. For cifar10h. we randomly split the dataset into training, validation and test. according to a 70%, 10%, 20% ratio. For CUB, we keep the original split.
Architecture of Concept and Task Predictors For each concept predictor qc. we employ three-layer MLP with a leaky-relu activation function. The black-box baselines and the CBM models, including our DCBM, adopt the same architecture and the same common frozen representation. Then, the task and concept classifiers are trained independently for the black box, the standard CBM, our dataset, DCBM, and its ablations (DCBM-NoTask and DCBM-NoConeepts For the compl eteness each concept encoder model takes as input the raw data. For the image datasets cifar and cub, concept predictors take instead an input the pre-trained embedding discussed Section 3.4 For CUB, we obtain such an embedding by training ResNet34 [He et al.) (0166) for 100 epochs to solve the final task using a cross entsspy loss function. The representations obtained by the pre-trained model are then frozen and used as the input for each concept encoder. For cifar10-A h we consider the pre-trained WideResNet [Zagoruyko and Komodakis) 2016) provided by Palomba al. 20025), who trained WideResNet architecture on the original cifar10 (Krizhessky et al.) 2009 training set for 200 epochs. We use the obtained representations to train all the concept encoders. All the final task classifiers consist of another three-layer MLP taking as input the concept values.
20

---

Table Mapping for uncertainty of concepts by Koh et
2020
0
0
0
0
4
2
3
4
2
3
4
Pa (c u)
0.00
0.50
0.75
1.00
0.00
0.50
0.25
0.00
Uncertain Concepts. To produce human expert labels in the CUB dataset, we employ the following strategy. Let e be the ground-truth label of concept for a given sample and be the corresponding
label of uncertainty, as provided by Koh et al. 2020J. Uncertainty labels have the following semantics: not visible (u 1), guessing (u 2), probably (u 3), and definitely (u 4). Koh al. [2020) translate the uncertainty labels in the following probabilities, which we use to sample the value of the concept provided by a human practitioner.
Training Procedure. We train every combination of models and defer costs À for 100 epochs. For completeness, we use Adam [Kingma and Ba 2015] with a learning rate equal to .001 and no scheduler. For both cifar10-b and CUB, optimizer, setting the initial learning rate to .001. We decrease the learning rate every 25 epochs by .5. Additionally, for CUB, following Zarlenga et al. [2022] guidelines,
AdamW (Loshchilov and Hutter 2019] as
consider weighted version of the loss on concepts to take into account their imbalance. To limit the computational burden, for both cifar10-b and CUB, we perform early stopping after 10 epochs if there is improvement for the loss on the validation set.
Evaluation. All the results are averaged over five and three runs on the synthetic and the other datasets, respectively, with fixed datasets' splits.
We train our baselines on a 224 cores machine with Intel(R) Xeon(R) Platinum 8480+ CPU and eight NVIDIA A100-SXM4-80GI,, OS Ubuntu 22.04.4 LTS. Notably, the cost of training a DCBM against a CBM is negligible, as they share essentially the same architecture, apart for an additional feature each concept task classifier. In detail, hardware, for a training epoch it takes se 9 seconds on CUB for a DCBM and s 7.5 for standard CBM. For sifarr0-be given the smaller number concepts, the difference even more negligible, with both taking approximately one second per epoch. We report epoch time as, due to early stopping strategies, the overall training time might vary across runs.
Hardware and Computational Time
C Additional Explanations
In Figure 5 we report additional samples from CUB that we analyze as the experimental results for research question Q4 in Section 4.2 While in the first two rows the effect of deferring on concepts is clearly beneficial, the other two examples require further discussion.
In both cases, we can see that the human would still mispredict a few concepts. Interestingly, such concepts are not visible in the image (e.g.. the upper tail the second-to-last example and the belly colour in the last example). Therefore, the perfect interventions are just an ideal scenario, as in practice, these concepts are not directly predictable from the image. Still, DCBM can "correctly defer" on the final task as concept interventions would not suffice to disambiguate the correct label.
D Additional Comparisons
D.1 Defer on Task
Deferring on the final task a useful strategy to mitigate risk incomplete scenarios where concepts alone are insufficient to determine the task label. For instance, in the cifar-10h dataset, concepts do not allow distinguishing cats from deer, since they have the same concept-level representation. In these cases, incomplete concept combinations should trigger the defer option the task, highlighting
21

---

wing Colert drewn
pomily cane ioot
way dermee
opecryann caso apone
eimii coler sou
weth Cate Cles aot pospoa Cuppri
Loni buoee Jaapyn
conciii Ber
Nicoeel Rutto
Geeet Bect
Lovee lasee peopos
clee feee Aecoeed Putto Seert
Loop tateo Saapen
Derio
Eoee fere
decoant Potto Bec
foom Coao Up
Now Colo
Duturred thmen
C.llladdd
Celared Henvertiro
No Heneeentroo
Af+
elAflordk
Tat netremt Wile coue Eoe carr coty ao Coter Onyy and ch puattsts Creer Cny
suneer mhod wole cater chock thst
fesall city
why Putten 2000 oe ciceppie grgtt dlyy
Oity
checa
stwnda poy Bae
tedip Stonioy
mosio sat bay
tedyy montiigg
aesett Jay
pay
cod dhusiing
10 Deternd Dencets- Hamen
Detoree Cancepts
Diteread Herversans
No Herversees
1.0
Pottec: Mervetions
So
crser restion shrol Peome
your se Pareroee
Wostens seeel Prenee
mowe Auxetee Weno
least Cower Cooato
pretes
dlw will keen Lewel
Nosee pndee Cootro
copeer
Dosle days stes Mottter dween took
ontvr Rested Phyeecter Seasidr Gyerto
ses Menobor dwee doo odloe dettid Povuttiie Feusion dourte
day teeebor conen doottre ustiee debted Fooltier spenian cource
shose ahd Cale
Figure 5: Interpretation of DCBM with defer cost 0.1 input sample. From left right: (a.) examples of an image from the CUB dataset; (b.) the concepts that the model has deferred with the estimated probability, green bars stand for when the human correctly predicts the concept, red otherwise; (c.) the estimated probability of each deferred concept being true according to the machine learning model, green bars stand for when the ML would have correctly predicted the concept, red otherwise; (d.) the estimated probability of top-5 final task labels after deferring the concepts to the human (standard DCBM behavior); (e.) the estimated probability of top-5 final task labels without deferring the concepts to the human; (f.) the estimated probability of top-5 final task labels from the ground-truth concepts;
22

---

Table Coverage of all concepts for increasing cost on cifar10-h. The DCBM correctly defers cats and deer when defer is not too costly, improving final performance. We highlight in bold the classes that are not possible to distinguish based on concept representations, i.e., cat and deer
pl
dog
freg
(000
(100
1.00
.000
000
000
-000
1.00 1000
1.00 000
1.00 .000
1.00
0.01
1.00 2: 0000
1.00 1: 1100
1.00 2: 0000
0000 1: 0000
000
000
1.00 =: 0000
1.00 1: 1000
1:00: 2: 0000
1 2: 0000
1.00 1: /000
0.06
.68 = 000
.938 d: J923
.962 .003
.092 0044
009
008
.00 = 0003
1984 4: .007
.979 2: 007
.987 .003
.960 .003
0.30
010
-033
.00G
004
008
-011
007
.977
(954
0.20
.900
023
999 .928
560 .222
1120 014
087
01T
.67 = 025
984 4: .003
.022 4: -010
.989 .005
.970
0.30
.88 - 008
.506 4: 1933
563 # 0012
115 :: 013
064 = 008
.61 = 007
.989 # .010
.961 # 000
.900 = .00
.50 #: .003
0.40
967 6 010
1,00 6: 0000
967 0088
-739 594
876 478
.78 020
.988 6: .016
.922 4: 010
.986 0003
.905 .006
003
005
Table 4: ConCone for the first 5 concepts, where the human is always correct. DCBM-NT always defer when possible (CovConcia is close to zero for small A), while UCP fails and defer less than optimal.
ComCome
CouCone
ConCome
ConCome
CouCome
UCP
DCEM--TT
DCP
DCBN-IT
UCP
DCBN- XT
UCP
DCBM NT
DCP
DCBN- WT
8.00
665 117
000
676 0091
.006
1.11
0.01
619 + 311
017 .019
680 + .888
.030 + 844
589 + 1888
.017 + +009
.713 + 137
.033 + 007
325 + +-04
.000 + .000
0.05
000 = 007
.132 = .084
.744 = .091
.177 4: .100
.01 4: 0077
.143 4 0079
.741 4: .13
.110 055
334 4: .00
.000 4: .00
0.10
.777 003
.382 .162
810 .087
.52 .073
0082
.262 133
.792 +11
.180
513++140
0.25
8022
2: 06
.800
-: .099
94 .053
.93
4 .018
.854
4. 004
.74 4. 063
.988 à: 0044
888 =: 0053
907 + .000
.459 2: .165
0.50
1,00 1 000
1,00 1. .000
1,00 1. .000
1.00 1. .000
1,00 1. .000
1.00 0.0000000000000000000000000
1.00 1. .000
1.00 1 000
1.00 1: .00
1.00 .000
instances that cannot be classified by looking at concepts only. We show this experimentally on cih where we can see that deferring on the final task improves the final accuracy (Figure(e)) Instead, the accuracy of the ablated DCBM-NT (i.e., DCBM with no possibility defer the final task) platcaus at around 90%. This is due to DCBM-NT randomly guessing between deer and cats, while correctly classifying other classes. Furthermore, Table 3 reports the coverage of all concepts for increasing cost X. Results show that, whenever defer is not too costly, DCBM correctly defers only instances of cats and deer to a human. Coherently with our formulation, when the cost increases, the model instead prefers to take a guess instead of deferring.
D.2 Uncertainty on Concept Predictions (UCP) vs DCBMs
Intervention strategies, such as those proposed by (Shin et al. 2023). allocate number of admissible interventions and then choose for each instance (or for batch of instances) on which concepts to intervene. Typically, such intervention strategies only consider the uncertainty of the model, which might disregard the fact that a human would not be better than the ML predictor in classifying a particular instance. Using learning to defer methodologies, we instead equip CBMs with the capability to (i.) autonomously ask for human intervention and (ii.) acknowledge the capabilities of the expert, i.e., we consider fallible human beings with variable performance.
We validate this intuition through two extra experiments, i.e., one in a fully controlled setting and one over CUB with uncertain humans. For both experiments, we compare DCEM-NT i.e., DCBM without the option to defer on the final task and the Uncertainty of Concept Predictions strategy (UCP) (Shin et al. 2023 which determines when to abstain based on the uncertainty on the concepts.
Synthetic Example We consider a slight modification of the completeness data we use for our ablation study (see Appendix B we define scenario where the human is always correct, on the first 5 concepts (out of 10) and always wrong on the remaining 5 concepts, i.e., the concept predictions for these last concepts always differ from the ground truth concepts.
We report in Tables and [ results for the coverage over the 10 concepts for both DCBM-NY -T and UCP applied on top of independent CBM. Recall that on the first 5 concepts the human is always correct, hence, if the cost allows it, DCBM-NT correctly learns to defer (CoeConc is 0 at À = 0), as shown in Table 4
Conversely, on the last concepts, where the human always makes mistakes (Table 5 we can see that the coverage for DCBM-NT is always one, i.e., the model has learned that interventions there would be harmful. On the other hand, UCP coverage is below one, meaning the intervention strategy would require intervening concepts where the human is wrong.
23

---

Table 5: CovCone for the last five concepts, where the human expert is always wrong. DCBM-NT correctly never defers (CorConc is one always), while UCP fails and defer more than ideal.
CovCone- -5
CovCone -
CocCome -?
ConCone- -
ConCone -
UCP
DCBM-NT
DCP
DCBX-E7
UCP
DCDM-NT
UCP
DCBM-XT
DCP
DCBX-W7
8.00
346 + 122
1.00 400
270 + 2203
1.00 + .000
.763 + 0000
1.00 + .000
.322 005
1.00 000
330--1172 117
1.00 +-00++ .000
1.00
4.73
0.05
.166 + 154
1.00= 4000
325 + 2005
1.00 + .000
.806 + .050
1,00 000
388 + 045
1:0+ 000
+ .146
1.00 + .000
0.10
506 = 174
1.00 .000
405 - .233
1.00 4. .000
.881 4. 003
1.00 2 000
.422 2: 003
1.00 000
381 2: 2227
1.00 4: .000
774
0.50
1.0 000
1.00+
000
1.00 + 900
1.00 + 000
1,00 + 0000
1,00 + 000
1.00 + 000
1.00 + 000
1.00 + .000
1.00 .000
Table 6: Comparison between DCBM-NT and UCP over CUB with uncertain humans. Results show that DCBM-NT is able to require intervention only when needed, while UCP over relies on humans, even these can make mistakes.
CovCone
UCP
DCBM-NT
0.00
.185 + .002
.655 + .015
800 .004
0.01
.188 .002
.666 + .015
803 0000
0.02
.271 + .029
.685
.96 = .001
0.0225
.424 + .049
7154
.822 = .003
0.025
.601 + .033
.012
.756 = .002
0.0275
.700 = .057
0.15
.743 .007
0.03
.808 .016
.771 .002
.738 = .004
0.04
.916 + .005
.751 .001
.724 = .002
0.05
.931 + .001
.745 .003
.723 + .004
0.10
.963 + .002
.719 .003
.704 = .004
0.20
.990 .002
.690 .00
.685 .001
0.30
.998 .001
.682 + .002
.677 = .002
0.50
1.000 + .000
.679 + .002
.676 = .003
CUB with uncertain humans
Table the comparison CUB with uncertain humans.
The results show that when we allow for a large number of interventions (A S .025), UCP underper- forms because it asks for interventions based solely model uncertainty, without accounting for whether the human likely to be more accurate. As result, it defers to the human on instances where the ML model predictions would have been better option.
On the other hand, when the budget of interventions limited, DCBMs have a more conservative approach and tend to prefer the use of the MI. model, leading to slightly worse outcomes.
E
Additional Results
The completeness dataset is a synthetic dataset that allows full control of the data- genteagenrrtionn process [Laguna et al. 2024). We add labels from human experts with different competencies by selecting the concepts' or task's correct labels with different probabilities. In particular, we denote as oracle, human-800 800% and human-60%, human that correctly predicts their labels with accuracy of 100%, 80% and 60%, respectively. For the oracle scenario, we plot the results Figure
We provide additional results on this dataset to investigate the following comparisons:
Shared parameters among the concept encoders,
Different human expert accuracy on both the concepts and task,
Joint vs Independent training,
Different learning-to-defe losses.
24

---

1100
ConCConn
00
1006
0
07
000
822
Figure 6: Results on completeness when human experts have perfect concept and task accuracy (i..,, they oracles). We report each metric's average and standard deviations increase the defer cost X. The black box and the CBM baselines are constant as they are independent of the defer cost.
Liu et al. 2024) studies the problem of label smoothing on learning to defer losses proposes a slightly different formulation of Equation which, once adapted our notation, we report as follows:
Label Smoothing.
Σ VEV
M(g(zv:0y). (1 A) - [yv hy] (tlp(((((v:@@)))))
(53)
A T # hv) arg min Ov)
kE(K]
In all the coming experimental results, the suffix -LS refer the results using Equation 33 while we use the suffix -NLS for the formulation Equation 4 As it is shown in the upcoming additional results, we do observe noteworthy differences the performance of the two loss functions.
Joint Learning. While independent training required guarantee the consistency of the learning- to-defer loss function, we implement joint learning to compare empirically. We implement the joint learning strategy by considering the following soft-labelled concept predictor:
((x) 91 (1) 91 (x)) + he (æ)),
(54)
where we produce the output as the weighted sum of the numan-providee concept eb and the machine learning model concept. The weight corresponds the probability of deferring or not the instance.
We study the different negative log-likelihood terms (Table T that can be employed within the saaning-to-defer (Equation 4 loss function for both independent (Tables 23 27 and (2) and joint learning (Tables 24 [28 and [0)] when dealing with oracle human experts on both concepts and tasks. Further, for the ASM Toss function, we also study how the model behaves when do not freeze the parameters of the encoder, also for independent (Table 255 and joint training (Table[6).. Finally, we study multiple combinations human expertise on the concepts and the tasks, whose reference summarize in the following table:
Human Task Expert
60
80
oracle
60
Tables and
Tables and
10
Tables and
Human Concept Expert
80
Tables and 14
Tables and 16
Tables 17
and
18
oracle
Tables
19 and 20
Tables
21
and 22
Tables 23 and
24
25

---

Table 7: Results for the completeness dataset when not allowing for shared parameters with independent training using ASM, and considering human60 task expert and human60 concept expert. LS refers to the label-smoothing- free implementation, while NLS to the one with label smoothing. We report avg + std and highlight the best baseline in bold.
Metric
DCBM-IS
DCBN-XT
024
.813 + 000
.226
= .011
.828 .010
+ .014
0.01
.199 + .017
.25 + 015
825 .009
881 = .018
831 .017
819 + .021
0.05
813 + .015
.836 + .013
.833 .008
.822 = .018
.833 .014
.819 + .014
0.10
.24 022
.844 .014
.322 .014
+ .010
834 .014
.829 +: .23
016
824
.188
.013
+:
AnTTus
0.20
819 + .016
.25 .114
0014
839 .2000
834 = 011
834 .06
.39 + .014
0,25
.23 + 0006
836 011
.199 = .016
823 = 016
817
815 115+.002 .023
0.30
.828 + .012
.811 0066
.828 .166
.822 = .014
#010
.819 + 0010
0.35
814
014
.823
839 = 010
.31
: .013
834
824 019
.225
= 015
834
+ .012
0.45
.26 + .010
835 = 014
833 = .021
823 = 008
827 .016
.24 + .018
0,50
.888 + .015
.818 + .112
.21 .015
810 + .009
828 .008
814 + .024
.800
.29
.007
.110
0.01
.827 + .012
843 010
831 .0U7
831 + 009
843 + 005
827 + .00
0.05
.244 2: .006
845 007
.244 + .012
828 = 0011
844 .003
.299 + .008
0.10
.833 + .005
.43 010
.822 .008
.222 = .008
846 .006
.844 + .008
AvcCCnn
.828 .008
+
.826
.004
.23 0011
844 006
.30
= 010
848 .077
+ .006
0,25
822 + .008
$43 003
826 1 .008
826 = 007
843 .007
827 + 0015
0.30
.87 + 000
849 008
.226 = 006
821 = 000
881 = 007
824 + .006
0.35
.821 + .011
.845 + .004
.877 .110
.80 + .006
.844 + .012
.824 + .006
.823 + .010
.845 012
.818 .017
= .006
844 .012
+ .013
0,50
.822 .008
.511 = 007
.16 + .006
814 .003
811 = .008
.817 + .008
0.00
1.000
1.000 0.000
1,000 0.000
0.000
1.000 0.000
0.000
1.000
0.000
1.000 0.000
1.000 0.000
1.000 0.000
1.000
0.05
1.000 0.000
1.000 :: 0.000
1.000 0.000
1.000 = 0.000
1.000 =000000.00
1.000 - 0.000
0.10
1,000 0.000
1.000 = 0.000
1,000 0,000
1.000 = 0.000
1.000=0.000
1. 1.000 + 0.000
ConTask
0.15
1.000 0.000
1.000 + 0.000
1,000 0.000
1.000 0.000
1.000 =
1.000 + 0.000
0.25
1.000
0.000
1.000 +: 0.000
1.000 0.000
1.000 = 0.000
0.000 1.000
1.000 + 0.000
0,30
1.000 0.000
1.000 : 0.000
1.000 0.000
1.000 = 0.000
1.000 =.00
1.000 0 0.000
0.35
1.000 0.000
1.000 = 0.000
1.000 0.000
1.000 0.000000000 0.000
1.000 0.000
1.000 + 0.000
0.40
0.0000
1.000 0.000
1,000 0.000
0.000
1.000 0.000
0.50
1.000
0.000
1. 1.000 +: 0.000
1.000 0.000
1.000
0.000
1.000 0.000
1 1.000 + 0.000
0.00
.933 + .018
1.000 1: 0.000
.971 .017
+ .013
1.000 0.000
.979 +:: .00
.983
.971
+: .003
.991 .017
1.000 + 0.000
.997 .004
.999 + .002
1.000
.999 + .002
0.10
1.000 0.000
1.000 : 0.000
.999 + .002
1.000 1: 0001
1.000 0.000
1.000 + 0.000
ConCone
0.15
1.000 0.000
1.000 - 0.000
1.000 0.000
1.000 = 0.000
1,000 + 0.000
0.20
0.000
1.000
0.000
1,000 0.000
- 1.000
1.000 0.000
1.000 0.000
0.30
1.000
0.000
1.000 + 0.000
1.000
0.000
1.000
0.000
1.000
0.000
1.000 + 0.000
0.35
1.000 + 0.000
1.000 + 0.000
1,000
0.000
1.000 + 0..00
1.000
0.000
1. 1.000 + 0.000
0.40
1.000 0.000
1.000 + 0.000
1,000
0.000
1.000 = 0.000
1.000 0.000
1.000 +: 0.000
0.45
1.000 0.000
1.000
0.000
1.000
0.000
0.000
1.000 0.000
1.000 - 0.000
1.000
1.000 0.000
1.000
0.000
1.000 - 0.000
0.000
1.000 0.000
26

---

Table 8: Results for the completeness dataset when not allowing for shared parameters with joint training using ASM, and considering human60 task expert and human60 concept expert. We report avg std and highlight the best baseline in bold.
Metric
DCBM-LS
OCBM-XC-LS DCBN-XT-CO XT-L8
DCBM-NLS
FERM-NC-XLA
DCBN-XT-BN XT-NX
0.00
.22 + 0001
.811 + .009
840 .000
.27 = 016
.82 .031
.27 + .019
0.01
816 + 0011
.888 = 0033
.815 .088
816 = .014
821 .012
820 + .006
0.05
.29 .08
.200 .118
.823 .122
.044
.828 .014
.877 +: .017
830
007
.814 2
.14 .133
022
0.15
813 + .012
837 + 010
.29 OU7
829 - 012
835 .033
828 + .000
AniFasi
0.20
.34 014
.822 + .017
827 = 010
824 = 0211
816 + .015
0.25
.17 + .006
.899 .012
.14 .008
823 - .018
833 .008
.831 + .022
0.30
.84
.833
.827
.166
.229
.13 .019
.36 + 018
.26
= 004
0011
+ .008
0.40
887 = .014
.31 + 005
820 + .013
828 = 010
827 = 008
.23 + .007
0.45
.033 + 034
.813 = .013
.177 = .25
800 = 025
816 016
.25 : .211
0.50
.800 + .057
.821 + .011
.009 .014
.855 + .007
.825 .005
.832 + .008
0.00
833 007
841 + 010
.177 .044
828 + .11
832 005
821 + 0011
0.01
825 + 008
837 + 008
822 007
821 # .005
843 = 0122
830 + 0011
0.05
.826 + 0044
.36 .009
.824 # .012
.81 = .015
839 .008
.827 + .011
.827
.899
.826
.811
+: .003
AncCome
.28 .007
007
834 .014
824 = .133
.31 .011
+: .009
0.20
.822 + .008
.39 007
824 .018
821 = .100
843 011
.255 + 0013
0.25
831 + 000
811 000
823 = 007
824 = .012
844 .009
819 : .012
0.30
.823 + .011
.845 .00
.822 .009
821 .006
.844 .01
.223 + .012
.00
0.40
.812 .006
841 + 002
.088 J011
822 + .009
839 005
.808 + .017
0.45
.816 + .006
.37 : 010
.817 + .011
825 = .10
838 1 007
.818 + .009
0.50
.14 .021
.86 + 011
.804 = .011
812 = .006
.32 008
8811 + .007
1.000
0.000
1.000 + 0.000
1.000 0.000
1.000 = 0.000
1.000 = 0.000
1.000 0.000
0.01
1.000 + 0.000
1.000 +: 0.000
1.000 0.000
1.000 = 0.000
1.000 0.000
1.000 + 0.000
0.05
1.000 0.0000
1.000 + 0.000
1,000 0.000
1.000 = 0.000
1.000:0.000
1. 1.000 + 0.000
0.10
1.000 0.0000
1.000 0.000
1,000 0.000
1.000 0.000
1.000
1.000 + O.E00.00 0.000
ConTask
1.000 0.000
1.000 +: 0.000
1.000 0.000
1.000 0.000
1.000 0.000
1.000 + 0.000
0,25
1.000 0.000
1.000 = 0.000
1.000 0.000
1.000 = 0.000
1.000 0.000
1.000 + 000
0.30
1.000 0.000
1.000 = 0.000
1.000
0.000
1.000 - 0.000
1.000
1.000 + 0.000
0.35
1.000 0.000
1.000 1 0.000
1,000 0.000
0.000
-.000
1.000 0.000
0.45
1.000
0.000
1.000 + 0.000
1.000
0.000
1.000
0.000
1.000
0.000
1.000 + 0.000
0,50
1.000 + 0.000
1.000 + 0.000
0.000 1,000
1.000
0.000
1.000
0.000
1. 1,000 + 0.000
0.00
.013
1.000
.979
1.000 0.000
+: .022
.989 .010
1.000 + 0.000
.984 .025
.974 = .034
1.000
0.000
.983 + .014
0.05
.999 + .002
1.000 + 0.000
.997 .005
.998 + .002
1.000 = 0.000
.977 + .01
0.10
.999 + .002
1.000 : 0.000
1.000 0.000
1.000 = 0.000
1.000 = 0.000
1.000 + 0.000
CocCone
0.15
1.000 0.000
0.000
1.000 0.000
1.000 0.000
1.000 0.000
1.000 0.000
1.000 - 0.000
1.000 1 0.000
1.000
0.000
1.000 - 0.000
1.000 0.000
1.000 - 0.000
0.25
1.000
0.000
1.000 + 0.000
1.000
0.000
1.000
0.000
1.000
0.000
1.000 + 0.000
0,30
1.000 + 0.000
1.000 + 0.000
1.000
0.000
1.000 + 0.000
1.000 + 0.000
1. 1.000 + 0.000
0.35
1.000 0.000
1.000 0.000
1,000
0.000
1.000 0.000
1.000
0.000
1.000 + 0.000
0.40
1.000 0.000
1,000
0.000
1.000 0.000
0.000
1.000 0.000
1.000
0.000
1.000
0.000
1.000 0.000
1.000 0.000
1.000
0.50
1.000 - 0.000
1.000
0.000
1.000 0.000
1 1.000 - 0.000
1.000 0.000
1.000 0.000
27

---

Table 9: Results for the completeness dataset when not allowing for shared parameters with independent training using ASM, and considering human80 task expert and human60 concept expert. LS refers to the label-smoothing- free implementation, while NLS to the one with label smoothing. We report avg + std and highlight the best baseline in bold.
Metric
DCBM-IS
DCBN-XT
014
.838 + 019
.266
= .008
.815
+ .014
0.01
.834 + .008
847 = .133
825 .009
822 = .088
8811 = .009
.19 + .001
0.05
.811 + .016
.42 = .019
.833 .008
.837 = .021
.833 .014
.819 : .014
0.10
.28 020
.812 .015
.832 .014
+ .015
832 .024
.829 +: .23
822
022
.244
+ .012
0022
008
AnTTus
0.20
821 + .016
824 .10
.399 .2000
826 = 010
.31 0014
.39 + .014
0,25
826 4002
.33 008
.899 = .016
823 = 0055
817
815 115+.022 .223
0.30
.828 + .012
.835 004
.828 .166
812 = .014
8014
.819 + .010
0.35
.866
: 0022
.823
011
832 .055
+: .013
.288 .022
.34 016
.225 2
= 014
832 .088
+ .012
0.45
.827 + .014
.31 = 014
833 + .021
829 = 009
837 0144
.21 + .018
0,50
.808 + .023
.23 + .011
.21 .015
806 + .011
834 + .010
814 + .024
.800
845
.29
.007
+ 0100
0.01
.827 .012
843 010
831 O0U7
831 + 009
843 + 005
827 + .00
0.05
.244 + .006
245+0007
.244 + .012
828 = 0111
844 .003
.299 + .008
0.10
.833 + .005
.43 010
.822 .008
.22 = .008
846 .006
.844 + .008
AvcCCnn
.828 .008
.826
.004
.23 0011
844 006
.30
= 010
848 .07
+ .006
0.25
822 + .008
$43 003
826 1 .008
826 = 007
843 .007
827 + 0015
0.30
887 + 007
849 008
.226 006
821 = 000
881 007
824 + .006
0.35
.821 + .011
.845 + .004
.877 .110
.30 + .006
.844 + .012
.824 + .006
.823 .000
.845 012
.818 .017
= 000
844 .012
+ .013
0,50
.812 .008
.511 = 007
.16 + .006
814 003
811 .008
.817 + .008
0.00
.933 .021
.961
1,000 0.000
.023
.943
0.000
.969 .015
.43 023
1.000 0.000
972 + .211
963 022
1.
0.05
.86 .012
.95 019
1.000 0.000
977 = 021
.96 007
1 1.000 + 0.000
0.10
.992 + .015
.999 002
1.000 0.000
.977 = .007
1.000 1:00:00 0.000
1. 1.000 + 0.000
ConTask
0.15
1.000
0.000
.997 + .007
1,000 0.000
.974 = .045
1.000 0.000
1.000 + 0.000
0.25
1.000 0.000
1.000 + 0.000
1.000 0.000
1.000 = 0.000
1.000 0.000
1.000 + 0.000
0,30
1.000 0.000
1.000 :: 0.000
1.000 0.000
1.000 = 0.000
1.000:00000
1.000 0 0.000
0.35
1.000 0.000
1.000 = 0.000
1.000 0.000
1.000 = 1.000000000 0.000
1.000 1000:00000 0.000
1.000 1 0.000
0.40
0.0000
1.000 0.000
1,000 0.000
1.000
1.000 0.000
0.50
1.000
0.000
1. 1.000 +: 0.000
1.000 0.000
1. 1.000 0.000
1.000
0.000
1 1.000 + 0.000
0.00
.933 + .018
1.000 1: 0.000
.971 .017
+ .013
1.000 0.000
.979 +:: .00
.983 .016
.971
1.000
+: .003
.991 .017
1.000 + 0.000
.997 .004
.999 + .002
1.000 0.000
.999 + .002
0.10
1.000
0.000
1.000 : 0.000
.999 + .002
1.000 1: 0001
1.000 0.000
1.000 + 0.000
ConCone
0.15
1.000 0.000
1.000 - 0.000
1.000 0.000
1.000 = 0.000
1,000 + 0.000
0.20
0.000
1.000
0.000
1,000 0.000
- 1.000
1.000 0.000
1.000 0.000
0.30
1.000
0.000
1.000 + 0.000
1.000
0.000
1.000
0.000
1.000
0.000
1.000 + 0.000
0.35
1.000 + 0.000
1.000 + 0.000
1,000
0.000
1.000 + 0..00
1.000
0.000
1. 1.000 + 0.000
0.40
1.000 0.000
1.000 + 0.000
1,000
0.000
1.000 = 0.000
1.000 0.000
1.000 0.000
0.45
1.000 0.000
1.000
0.000
1.000
0.000
0.000
1.000 0.000
1.000 0.000
1.000
1.000 0.000
1.000
0.000
1.000 - 0.000
1.000 0.000
1.000 0.000
28

---

Table 10: Results for the completenen dataset when not allowing for shared parameters with joint training using ASM, and considering human80 task expert and human60 concept expert. We report avg std and highlight the best baseline in bold.
Metric
DCBM-LS
-C---
DCBN-XT MONN-XT-IS
DCBM-NLS
DCBM-NC-NIL
DCBN-XT-HEL
0.00
.811 + 020
.855 + .008
.800 .010
883 = .199
856 = .22
.27 + .019
0.01
.839 + .016
.822 = .010
.855 .008
.815 = .009
851 .013
820 + .006
0.05
.822 008
.47 .010
.823 .012
021
.841 .015
.877 +: .017
832 0111
000
.814
4 .100
.844 011
002
0.15
.15 + 0008
.26 .012
.29 OU7
830 019
831 0011
828 + .010
AniFasi
0.20
823 + 2015
.28 014
.822 + .017
827 = .055
828 .018
816 + .015
0.25
.888 +
.33 023
.14 .008
.37 = .012
837 .012
.811 + .022
0.30
.01
.827
.004
.28
.26 - .099
018
.26
022
837 004
+: .088
0.40
.89 .019
842 008
820 = .013
836 = 011
827 = .100
823 : .007
0.45
810 .118
.818 008
.17 = .05
.35 = 0111
.22 .018
.25 : .01
0.50
.815 .038
.827 + .018
.009 .014
.825 + .014
.27 + .012
.82 + .008
0.00
.84 .110
839 008
.177 .044
829 + 007
834 006
821 + 0011
0.01
823 + 009
840 008
822 007
824 + 0008
844 000
830 + 0011
0.05
.827 + .004
839 .00
.824 .012
.23 = .011 2011
840
.827 + .011
.800
.86 .009
.226
008
+:
AncCone
829
006
.144
827 = .12
834 .022
+: .009
0.20
817 + .008
.38 + 007
821 .018
820 = 012
844 010
.255 + 0013
0.25
832 + 008
840 005
.23 = 007
826 = 006
843 = .006
819 : .012
0.30
.822 + .008
.846 + .009
.822 .009
.23 .007
-31
.223 + .012
0.40
.810 + .004
.39 0014
014
.088 J011
823 + 008
837 0044
.808 + .017
0.45
.811 + .00
.35 0088
.817 + .011
825 = 015
839 1 007
.818 + .009
0.50
.833 = .016
.866 + 015
.004 .011
814 .006
832 008
8811 + .007
.794 .277
.887 + .031
1.000 0.000
+ .406
.908 = .047
1.000 0.000
0.01
.722 + .408
.733 .68
1.000 0.000
921 = .44
805 249
1 1.000 + 0.000
0.05
.544 .012
.488 .122
1,000 0.000
826 = .229
970 020
1. 1.000 + 0.000
0.10
.83 .017
.92
1,000 0.000
= .009
1.000 0.000
ConTask
.966
+
1.000 0.000
1.000 +: 0.000
1.000 0.000
1.000 0.000
1.000 0.000
1.000 + 0.000
0,25
1.000 0.000
1.000
1.000 0.000
1.000 = - 0.000
1.000 0.000
1.000 + 000
0.30
1.000 0.0000
1.000 0.000
1.000 0.000
1.000 = 0 0.000
1.000
1.000 + 0.000
0.35
1.000 0.000
0.000
1,000 0.000
1.000=00000
1.000 0.000
0.45
1.000
0.000
1.000 + 0.000
1.000
0.000
1.000
0.000
1.000
0.000
1.000 + 0.000
0,50
1.000 + 0.000
1.000 + 0.000
0.000 1,000
1.000
0.000
1.000
0.000
1. 1.000 + 0.000
0.00
.011
1.000
.979
1.000 0.000
+: .022
.990
.010
1.000 + 0.000
.984 .025
.974 = .033
1.000 0.000
.983 + .014
0.05
.999 + .002
1.000 + 0.000
.997 .005
.999 + .001
1.000 = 0.000
.977 + .01
0.10
1.000 0.000
1.000 : 0.000
1.000 0.000
1.000 + .001
1.000 = 0.000
1.000 + 0.000
CocCone
0.15
1.000 0.000
0.000
1.000 0.000
1.000 1 0.000
1.000 =.000000000
1.000 0.000
1.000 0.000
1.000 1 0.000
1.000
0.000
1.000 0.000
1.000 1:000:000 0.000
1.000 - 0.000
0.25
1.000
0.000
1.000 + 0.000
1.000
0.000
1.000
0.000
1.000
0.000
1.000 + 0.000
0,30
1.000 + 0.000
1.000 + 0.000
1.000
0.000
1.000 + 0.000
1.000 + 0.000
1. 1.000 + 0.000
0.35
1.000 0.000
1.000
0.000
1,000
0.000
1.000 0.000
1.000
0.000
1.000 + 0.000
0.40
1.000 0.000
0.000
1,000
0.000
1.000 0.000
0.000
1.000 0.000
1.000
0.000
1.000
0.000
1.000 0.000
1.000 0.000
1.000
0.50
1.000 - 0.000
1.000
0.000
1.000
0.000
1 1.000 - 0.000
1.000 0.000
1.000 0.000
29

---

Table 11: Results for the completeness dataset when not allowing for shared parameters with independent training using ASM, and considering oracle task expert and human60 concept expert. LS refers to the saabeel-moothhig--iie free implementation, while NLS to the one with label smoothing. We report avg = std and highlight the best baseline in bold.
Metric
DCBM-I-
-C-I-
DCBN-XT
DCBM-NC-XLSS
DCBM-XT NINN-TT-LLSS
1.000 0.000
1.000 + 0.000
.226 .099
1.000 = 0.000
1.000 0.000
8816 + .014
0.01
1.000 - 0.000
1.000 + 0.000
825 .009
.909 = .002
.999 .002
819 + .021
0.05
.922 = 028
.976 + 027
.833 .008
.981 = .011
.986 .012
.819 + .014
0.10
898 0211
.888 + .118
.32 .014
+ .018
921 .038
.829 + .03
872 000
888 +
824
011
.89 - .114
008
AnTTus
0.20
844 + .119
842 .01
839 .20
853 - 006
862 015
839 + 014
0,25
826 + .004
.399 0013
.199 = .016
826 = 0133
822 = .018
815 10.+.022 .023
0.30
.828 + .008
.888 .012
.828 .016
.27 = .014
.19 .08
.819 + .000
0.35
.821 .015
832
.823
838 012
.37
:: .013
.888 .020
832 013
.825
= 013
828 .19
+ .012
0.45
.22 + .012
833 012
.333 .224
832 = 011
830 .015
.24 + .018
0,50
.800 + .017
831 + 010
.811 .015
803 + 020
830 .00
814 + .024
.800 +
.29
0007
.110
0.01
.827 .012
843 + 010
831 .0U7
831 + 009
843 005
827 + .00
0.05
.244 - .006
845 007
.244 + .012
828 = 0011
844 .003
.299 + .008
0.10
.833 + .005
.433 010
.822 .008
.222 + .008
846 .006
.844 + .008
AvcCCnn
.828 .088
+
.826
.004
.009
.23 0011
844 006
.30
= 010
848 = .077
+ .006
0.25
822 + .008
$43 003
826 1 .008
826 = 007
843 .007
827 + 0015
0.30
.87 + 007
849 008
.226 006
821 = 000
8811 000
824 + .006
0.35
.821 + .011
.845 + .004
.877 .110
.30 + .006
.844 + .012
.824 + .006
.09
.823 + .010
.845 012
.818 .017
= 000
844 .012
+ .013
0,50
.822 .008
.811 = 007
.16 .006
814 .03
811 = .008
.817 + .008
0.00
0.000 = 0.000
0.000 0.000
1,000 0.000
0.000 0.000
0.000
.002 .004
0.000 0.000
1.000 0.000
.20
.005 .011
1
0.05
.31 .173
.37 + 205
1.000 0.000
199 100
.59 : 103
1 1.000 + 0.000
0.10
.733 + 123
814 003
1,000 0,000
682 = .088
684 279
1. 1.000 + 0.000
ConTask
0.15
.833 + .031
.833 + 006
1.000 0.000
.875 = .041
.96 .20
1. 1.000 + 0.000
0.25
.944 + .005
.90 0011
1.000 0.000
.991 + .012
.98 004
1. 1.000 + 0.000
0,30
.997 + .007
.95 006
1.000 0.000
.981 = .200
1.000 0.000
1.000 0 0.000
0.35
1.000 = 0.000
.999 000
1.000 0.000
1.000 = 0.000
1.000 1:00::00 0.000
1.000 + 0.000
0.40
1.000 0.000
1.000
1,000 0.000
1.000
1.000 0.000
0.50
1.000
0.000
1.000 +: 0.000
1.000 0.000
1. 1.000 0.000
1.000 0.000
1 1.000 + 0.000
0.00
.973 .018
1.000 1: 0.000
.971 .017
+ .013
1.000 0.000
.979 1:: .00
.983 .016
.971
.00
.991 .017
1.000 + 0.000
.997 .004
.999 + .002
1.000 0.000
.999 + .002
0.10
1.000
0.000
1.000 : 0.000
.999 + .002
1.000 1: 0001
1.000 0.000
1.000 + 0.000
ConCone
0.15
1.000 0.000
1.000 - 0.000
1.000 0.000
1.000 = 0.000
1.000
1,000 + 0.000
0.20
0.000
1.000
0.000
1,000 0.000
- 1.000
1.000 0.000
1.000 0.000
0.30
1.000
0.000
1.000 + 0.000
1.000
0.000
1.000
0.000
1.000
0.000
1.000 + 0.000
0.35
1.000 + 0.000
1.000 + 0.000
1,000
0.000
1.000 + 0..00
1.000
0.000
1. 1.000 + 0.000
0.40
1.000 0.000
1.000 + 0.000
1,000
0.000
1.000 = 0.000
1.000 0.000
1.000 +:00 0.000
0.45
1.000 0.000
1.000
0.000
1.000
0.000
0.000
1.000 0.000
1.000 0.000
1.000
1.000 0.000
1.000
0.000
1.000 - 0.000
1.000 0.000
1.000 0.000
30

---

Table 12: Results for the complet dataset when not allowing for shared parameters with joint training using ASM, and considering oracle task expert and human60 concept expert. We report avg std and highlight the best baseline in bold.
Metric
DCBM-LS
DCBM-NC-LS
DCBN-XT-BE XT-
DCBM-NLS
DCBN-NC-NLL
DCBN-XT-HIL
0.00
1.000 = 0.000
1.000 + 0.000
.810 .010
1.000 = 0.000
1.000 0.000
.27 + .019
0.01
1.000 + 0.000
1.000 + 0.000
.15 .088
1.000 = 0.000
1.000 0.000
820 + .006
0.05
.91 .008
.98 + 004
.823 .012
.099
1.000 0.000
.877 +: .017
.64
023
.114
+ 022
.957
+: 002
0.15
916 + .034
903 024
.29 O07
944 = 039
899 .014
828 + .010
AndFast
0.20
.79 + .02
.833 020
.822 + .017
.69 = .117
878
816 + .015
0.25
.499 + .013
854 .013
.14 .008
845 = .017
848 -46
.831 + .022
0.30
.835 .008
.827
.005
.899
.855 .015
+ 016
.26
= 011
+ .008
0.40
.833 + .019
834 016
820 = .013
828 = 015
827 = 006
.23 - .007
0.45
.89 + 007
.822 - 0009
.17 = .05
826 = .009
.825 = 023
.25 : .021
0.50
.818 .023
.821 + .020
.009 .014
.823 + .013
.824 .007
.82 + .008
0.00
830 + 007
839 009
.177 .044
829 + 007
834 007
821 + 0011
0.01
.25 + 009
841 + 008
822 007
820 + .007
844 .008
830 + 0011
0.05
.828 + .004
.39 + 0007
.824 .012
.844 = .009
842 .011
.827 + .011
.829
.899 0044
.226
.31
+: .003
AncCome
826 .008
+ 008
.34 .144
823 013
834 .11
+: .009
0.20
817 + .009
.39 + 006
821 .018
820 = 010
844 008
.255 + 0013
0.25
.26 + .009
813 000
823 = 007
826 = 003
843 = 000
819 : .012
0.30
.819 .011
.844 .012
.822 .009
.22 + .008
834
.823 + .012
.00
0.40
.810 .009
840 + 012
0012
.808 J011
819 .009
838 0007
.808 + .017
0.45
.807 + .0088
834 0088
.817 .011
826 = 0111
839 = 006
.818 + .009
0.50
.806 + .019
.834 + 009
.804 .011
810 = .003
.82 .007
.811 + .007
0.000 0.000
0.000 + 0.000
1.000 0.000
0.000 = 0.000
0.000 0.000
1.000 + 0.000
0.01
0.000 0.000
0.000 + 0.000
1.000 0.000
0.000 = 0.000
0.000 0.000
1 1.000 + 0.000
0.05
.141 + .132
.88 090
1,000 0.000
.040 .089
.00 = .067
1. 1.000 + 0.000
0.10
.200 117
: 1139
1,000 0.000
=
.455
1.000 0.000
ConTask
.007
+
840
.791 .178
.893 + .033
1.000 0.000
.944 + .077
893 .028
1. 1.000 + 0.000
0,25
949 - .012
.955 022
1.000 0.000
.957 = .16
961 .017
1. 1.000 + 0.0
0.30
.944 + .007
.984 0055
1.000 0.000
.981 = .016
.991 =: .008
1. 1.000 + 0.000
0.35
1.000 1 0.000
1,000 0.000
.002
1.000 0.000
1.000 0.000
0.45
1.000
0.000
1.000 +: 0.000
1.000
0.000
1.000 0.000
1.000 0.000
1.000 + 0.000
0,50
1.000 + 0.000
1.000 + 0.000
0.000 1,000
1.000
0.000
1.000 0.000
1. 1.000 + 0.000
0.00
.017
1.000
.979
1.000 0.000
+: .022
.986 .016
1.000 + 0.000
.984 .025
.974 = .036
1.000
0.000
.983 + .014
0.05
.999 + .001
1.000 + 0.000
.997 .005
.998 + .001
1.000 = 0.000
.977 + .01
0.10
1.000 0.000
1.000 : 0.000
1.000 0.000
1.000 = 0.000
1.000 = 0.000
1.000 + 0.000
CocCone
0.15
1.000 0.000
0.000
1.000 0.000
1.000 1 0.000
1.000 0.000
1.000 0.000
1.000 0.000
1.000 1 0.000
1.000
0.000
1.000 - 0.000
1.000 0.000
1.000 0.000
0.25
1.000
0.000
1.000 + 0.000
1.000
0.000
1.000
0.000
1.000
0.000
1.000 + 0.000
0,30
1.000 + 0.000
1.000 + 0.000
1.000
0.000
1.000 + 0.000
1.000 + 0.000
1. 1.000 + 0.000
0.35
1.000
0.000
1.000 0.000
1,000
0.000
1.000 0.000
1.000
0.000
1.000 + 0.000
0.40
1.000 0.000
1,000
0.000
1.000 0.000
0.000
1.000 - 0.000
1.000
0.000
1.000
0.000
1.000 0.000
1.000 0.000
1.000
0.50
1.000 - 0.000
1.000
0.000
1.000 0.000
1 1.000 - 0.000
1.000 0.000
1.000 0.000
31

---

Table 13: Results for the completenee dataset when not allowing for shared parameters with independent training using ASM, and considering human60) task expert and human80 concept expert. LS refers to the label-smoothing free implementation, while NLS to the one with label smoothing. We report avg + std and highlight the best baseline in bold.
Metric
DCBM-IS
DCBM-NO -C-I-
CO-M-NC-XLS
DCBN-XT MINN-TT-MLA
.822 .016
.813 + 000
.200
= .025
.28 .100
+ .013
0.01
.823 + .019
.825 + 015
.877 .023
817 = .013
881 .017
824 + .007
0.05
.811 + 0211
.836 .013
.838 .024
.816 = .012
.833 .014
.27 + .010
0.10
.233 008
414 .044
.014
.835 .006
.23 + .012
834 .014
.80 +: .223
824 .066
.28 016
.244
+ .055
.033
AnTTus
0.20
813 .019
.25 0044
832 .018
825 - 009
834 .166
836 + .015
0,25
.19 $19
836 011
.244 + .02
823 021
817
821 + .010
0.30
.844 .011
.831 .066
.835 .010
.23 = .012
.811 #010
.816 .011
0.35
.821 .013
014
.826
1 0177
831 .019
.877 : .018
.840 023
824 019
.821
011
831 .01
835 .016
0.45
.822 + 006
.335 014
842 = .008
825 = 013
827 = .016
.25 + 009
0,50
.81 + .015
.888 + .012
.226 .122
801 + .012
828 .008
827 + .018
008
+ .005
0.01
884 006
885 .022
885 + .05
843 + 005
883 + 003
0.05
881 + .005
$45
881 + .004
.81 = ,011
844 .003
.86 + .005
0.10
.699 + .003
843 .010
.800 = .007
.864 = .003
846 .006
.833 + .007
AvcCCnn
.846
847
.850
= .007
887
.003
.35
844 006
.388 .008
837 = 006
848 .077
+ .00
0,25
830 + .006
$43 003
832 + .005
833 = 008
843 = .007
.39 + 0015
0.30
836 + 007
849 008
.31 # .001
832 = .008
811 = .00
881 + .005
0.35
.830 + .008
.845 + .004
.824 .013
.37 + .002
8844 + .012
.831 + .008
.833
+ = .100
.007
.832 + .009
.845 012
.829 # .011
844 .012
.822 + .007
0,50
.826 + .003
.511 = 007
.21 + .008
827 = .004
811 11 .008
.831 + .008
0.00
1.000
1.000 0.000
1,000 0.000
1.000 0.000
0.000
1.000 + 0.000
1.000 0.000
1.000 0.000
1.000 0.000
1.000
0.05
1.000 0.000
1.000 :: 0.000
1.000 0.000
1.000 = 0.000
1.000 0.000
1.000 - 0.000
0.10
1.000 0.000
1.000 = 0.000
1,000 0,000
1.000 = 0.000
1.000
1. 1.000 + 0.000
ConTask
0.15
1.000 0.000
1.000 + 0.000
1,000 0.000
1.000 0.000
=.000000000
1.000 + 0.000
0.25
1.000 0.000
1.000 +: 0.000
1.000 0.000
1.000 = 0.000
0.000 1.000
1.000 + 0.000
0,30
1.000 0.000
1.000 : 0.000
1.000 0.000
1.000 = 0.000
1.000 -.00000000
1.000 0 0.000
0.35
1.000 0.000
1.000 = 0.000
1.000 0.000
1.000 = 0.000
1.000 0.000
1.000 1 0.000
0.40
1.000 0.0000
1.000 0.000
1,000 0.000
0.000
1.000 0.000
0.50
1.000
0.000
1. 1.000 +: 0.000
1.000 0.000
1. 1.000 0.000
1.000
0.000
1 1.000 + 0.000
0.00
.388 .015
1.000 1: 0.000
.122 .045
.039
1.000 0.000
.466 .22
.040
.478
1.000
.603 .034
1.000 + 0.000
.595 .040
+ .28
1.000 0.000
.607 + .052
0.10
.800 + .021
1.000 + 0.000
.797 4 .023
.773 = 043
1.000 0.000
.783 + .211
ConCone
0.15
.933 + .004
1.000 + 0.000
.908 .02
.924 = .211
1.000
.908 4 .015
0.20
.981
1.000 0.000
.911
- .011
1.000 0.000
+: .008
.997 .006
.999
.008
0.30
1.000
0.000
1.000 + 0.000
1.000
0.000
.99 + .001
1.000
0.000
1.000 + 0.000
0.35
1.000 + 0.000
1.000 + 0.000
1,000
0.000
1.000 + 01.00
1.000
0.000
1. 1.000 + 0.000
0.40
1.000 + 0.000
1.000 + 0.000
1,000
0.000
1.000 0.000
1.000 0.000
1.000 0.000
0.45
1.000 0.000
1.000 1 0.000
1.000
0.000
1.000 0.000
1.000 0.000
1.000 0.000
1.000
1.000 0.000
1.000
0.000
1.000 - 0.000
1.000 0.000
1.000 0.000
32

---

Table 14: Results for the completeness dataset when not allowing for shared parameters with joint training using ASM, and considering human60 task expert and human80 concept expert. We report avg std and highlight the best baseline in bold.
Metric
DCBM-LS
DCBM-NC-LS DCBN-XT CONN-XT-BE XT-L8
DCBM-NLS
OCBN-NC-XIL
DCBN-XT- CORN-XT-HIL
0.00
.800 + 0055
.25 + .009
.834 .013
822 = .077
.824 =2 .014
.41 + .0.9
0.01
.825 + 0015
.800 + .005
.820 .022
.825 = .009
.822 .008
826 + .012
0.05
.19 0022
.266 008
.831 .23
.866 + .066
.832 .014
833 + .099
825
+: 004
.222 .011
+ 0066
.007
+: 009
0.15
800 + .016
.28 + 008
8333 .006
823 - 007
830 013
829 + 009
AnrFasi
0.20
816 1
.825 : 0009
.823 .014
826 = 0111
830 =:0=20 015
826 + .05
0.25
816 .002
.333 + .008
.831 .013
.333 = .014
.824 004
.28 4 .025
0.30
.822
.815
.009
.277
.87 002
+ 021
.300
= 018
+ .013
0.40
.815 + .011
823 008
837 .018
828 = 013
833 0188
.24 + 0014
0.45
.821 + .015
.813 000
.823 = 007
833 = .009
.22 2 .015
824 :: .012
0.50
.817 + .028
.818 + .017
.816 .017
.827 .006
.32 .013
.825 + .020
0.00
.899 + .007
837 +: OU7
876 .05
877 + 005
832 007
882 +: .012 0012
0.01
878 + 0004
835 + 009
869 003
874 # .005
842 014
869 + 006
0.05
.84 .004
.87 - .011
.85 + .005
.873 = .012
.838 = 000
868 + .004
.856
.39
864
= .110
.800
+ .006
AncCome
846 .004
006
= .007
.011
847 + .005
0.20
831 + .012
.39 011
833 .006
833 = 006
843 = 009
.83 + 000
0.25
832 + 007
839 0011
831 009
831 = .015
844 000
832 : .00
0.30
.833 + .009
.844 .008
.826 .009
.27 + .008
.81 .009
.834 + .009
.011
.007
0.40
.826 .005
841 + +011
.119 .010
831 + 0066
838 .033
.855 + .011
0.45
.877 + .005
.37 010
.23 1 .001
.35 = .006
840 007
.824 +: .006
0.50
.827 = .005
.835 + 011
.820 .009
.24 = .005
.835 .006
.821 4: .007
1.000 0.000
1.000 + 0.000
1.000
0.000
1.000 = 0.000
1.000
0.000
1.000 0.000
0.01
1.000 + 0.000
1.000 +: 0.000
1.000 0.000
1.000 = 0.000
1.000 0.000
1.000 + 0.000
0.05
1.000 0.000
1.000 + 0.000
1,000 0.000
1.000 = 0.000
1.000:00000
1. 1.000 + 0.000
0.10
0.0000
1.000 0.000
1,000 0.000
1.000 0.000
1.000
1.000 + 0.000
ConTask
1.000 0.000
1.000 +: 0.000
1.000 0.000
1.000
0.000
1.000 0.000
1.000 + 0.000
0,25
1.000 0.000
1.000 = 0.000
1.000 0.000
1.000 = 0.000
1.000 0.000
1.000 + 000
0.30
1.000 0.000
1.000 = 0.000
1.000 0.000
1.000 = 0 0.000
1.000
1.000 + 0.000
0.35
1.000 0.000
1.000 1 0.000
1,000 0.000
0.000
-.000
1.000 0.000
0.45
1.000
0.000
1.000 + 0.000
1.000
0.000
1.000
0.000
1.000
0.000
1.000 + 0.000
0,50
1.000 + 0.000
1.000 + 0.000
0.000 1,000
1.000
0.000
1.000
0.000
1. 1.000 + 0.000
0.00
.21
1.000
.490
1.000 0.000
.523 .026
1.000 + 0.000
.500 .041
.507 + .081
1.000
0.000
.487 + .033
0.05
.653 + .052
1.000 + 0.000
.631 .028
.573 + .063
1.000 = 0.000
.656 + .019
0.10
816 + .011
1.000 + 0. 0.000
7788 + 038
838 + .035
1.000 = 0.000
.855 + .0199
CocCone
0.15
.999
1.000 0.000
.933
.027
1.000 0.000
1: .023
.986 .011
1.000 1 0.000
.981 .034
1 .011
1.000 1 0.000
.983 4 .022
0.25
.999
.001
1.000 + 0.000
1.000 0.000
.998 + .003
1.000 0.000
.999 + .001
0.30
1.000 + 0.000
1.000 + 0.000
1.000
0.000
1.000 + 001
1.000 + 0.000
1.000 + 0.000
0.35
1.000 0.000
1.000 0.000
1,000
0.000
1.000 0.000
1,000 0.000
1.000
0.40
0.0000
0.000
1,000
0.000
1.000 0.000
1.000
0.000
1.000 0.000
1.000 +
1.000
0.000
1.000
0.000
1.000 - 0.000
1.000 1:000:000 0.000
1.000 0.000
0.50
1.000 0.000
1.000 0.000
1.000
0.000
1. 1.000 - 0.000
1.000 0.000
1.000 0.000
33

---

Table 15: Results for the completenee dataset when not allowing for shared parameters with independent training using ASM, and considering human80 task expert and human80 concept expert. LS refers to the label-smoothing- free implementation, while NLS to the one with label smoothing. We report avg + std and highlight the best baseline in bold.
Metric
DCBM-IS
DCBM-NO -C-ISS
DCBM-
DCBN-XT MINN-NT-MI..
.846 022
.838 + 019
.200
.859 = .012
.815 =
+ .013
0.01
.839 + .019
847 = 0133
.17 .023
810 .155
881 .009
824 + .007
0.05
.829 + .014
.42 = .019
.838 .021
.832 = .019
.833 .014
.877 + .010
0.10
.82 0044
.812 .015
.835 .006
.27 + .010
.32 024
.800 +: .023
.015
022
.244
+ .111
.022
0100
AnTTus
0.20
826 + .012
824 010
832 .018
821 + 011
831 :014
836 + .015
0,25
823 2018
833 008
.244 + .020
825 = .019
817
821 + .010
0.30
.199
.855 0044
.835 .010
817 = .012
.22 0014
.01
0.35
.822 : .01
012
.826
837 017
.32
.877 :: .018
.36 .166
834 016
.821
= 014
832 .188
885 .016
0.45
833 + 007
831 = 014
842 = .008
822 = 021
837 2 .114
.25 + .00
0,50
.25 + .199
823 + .111
.226 .122
805 + .015
834 + .010
827 + .088
+ 008
.055
4
0.01
884 006
843 -10
885 .022
885 + .055
843 4 005
883 + 003
0.05
881 + .005
$45
881 + .004
.81 = .011
844 0003
.86 + .005
0.10
.699 + .003
843 .010
.800 = .007
.864 = .003
846 .006
.833 + .007
AvcCCnn
.846
847
.850
= .007
.877
.003
.35
844 006
.388 .008
837 = 000
848 .07
+ .00
0.25
830 + .006
$43 003
832 + .005
833 = 008
843 .007
.39 + 0015
0.30
836 + 007
849 008
.31 .001
832 = .008
811 .00
881 + .005
0.35
.830 + .008
.845 + .004
.824 .013
.37 + .002
8844 + .012
.831 + .008
.011
.833
+ = 010
.832 + .009
.845 012
.829 # .011
844 .012
.332 + .007
0,50
.826 + .003
.511 = 007
821 + .008
827 = 0044
811 111 .008
.831 + .008
0.00
.901 .023
.961
1,000 0.000
.027
.943
0.000
.400 .05
.43 023
1.000 0.000
+ .039
963 022
1.
0.05
982 + .012
.95 019
1.000 0.000
.961 = 028
.96 007
1 1.000 + 0.000
0.10
.933 + .011
.999 002
1.000 0.000
.993 = .013
1.000 - 0.000
1. 1.000 + 0.000
ConTask
0.15
1.000 0.000
.997 + .007
1,000 0.000
.971 = .037
1.000 0.000
1.000 + 0.000
0.25
1.000 0.000
1.000 + 0.000
1.000 0.000
1.000 = 0.000
1.000 0.000
1.000 + 0.000
0,30
1.000 0.000
1.000 0.000
1.000 0.000
1.000 = 0.000
1.000:00000
1.000 1 0.000
0.35
1.000 0.0000
1.000 = 0.000
1.000 0.000
1.000 = 0.000
1.000 0 0.000
1.000 1 0.000
0.40
1.000 0.0000
1,000 0.000
0.000
1.000 0.000
0.50
1.000
0.000
1. 1.000 +: 0.000
1.000 0.000
1. 1.000 0.000
1.000
0.000
1 1.000 + 0.000
0.00
.388 .015
1.000 1: 0.000
.122 .045
.039
1.000 0.000
.466 .22
.040
.478
1.000
.603 .034
1.000 + 0.000
.595 .040
+ .28
1.000 0.000
.607 + .052
0.10
.800 + .021
1.000 + 0.000
.797 4 .023
.773 = 043
1.000 0.000
.783 + .211
ConCone
0.15
.933 + .004
1.000 + 0.000
.908 .02
.924 = .211
1.000
.908 4 .015
0.20
.981
1.000 0.000
.911
- .011
1.000 0.000
+: .008
.997 .006
.999
.008
0.30
1.000
0.000
1.000
0.000
1.000
0.000
.99 + .001
1.000
0.000
1.000 + 0.000
0.35
1.000 + 0.000
1.000 + 0.000
1,000
0.000
1.000 + 01.00
1.000
0.000
1. 1.000 + 0.000
0.40
1.000 + 0.000
1.000 + 0.000
1,000
0.000
1.000 0.000
1.000 0.000
1.000 0.000
0.45
1.000 0.000
1.000 1 0.000
1.000
0.000
- 0.000
1.000 0.000
1.000 0.000
1.000
1.000 0.000
1.000
0.000
1.000 - 0.000
0.000000000
1.000 0.000
34

---

Table 16: Results for the completeness dataset when not allowing for shared parameters with joint training using ASM, and considering human80 task expert and human80 concept expert. We report avg std and highlight the best baseline in bold.
Metric
DCBM-LS
OCEM-XC-LS
DCBN-XT CORN-XT-BEE XT-L8
DCBM-NLS
OCBM-NC-XIL
OCBN-XT-HI
0.00
.833 + .099
.811 + .001
.834 .013
.533 = .055
.866 .019
841 + .019
0.01
.50 + .018
.815 = .010
.820 .02
.833 - .010
.851 .008
826 + .012
0.05
.831 009
.811 .013
.831 .23
.499 .066
.838 .014
.83 + .019
+ 014
.222
+ 0111
000
0.15
812 + .013
.28 008
833 .066
824 + 011
834 = .000
829 + 009
AnrFasi
0.20
881 .009
.823 = .014
816 007
827 = .10
826 + .015
0.25
=20
.888 .006
.831 .013
.32 = .014
822 .013
.28 4 .025
0.30
.82
.899
.828
.30 .02
.32 021
.30
= 010
+ .013
0.40
.888 + .018
830 012
837 .018
831 = .15
833 .088
.24 + 0014
0.45
.820 - .166
.899 = 007
.823 = 007
835 = .111
833 016
824 :: .012
0.50
.824 + .026
.800 .018
.816 .017
.26 .011
.31 .007
.825 + .020
0.00
.811 .007
839 + OUT
876 .05
877 + 005
834 008
882 0012
0.01
879 + 005
839 + 008
869 + 003
875 # .004
844 + .011
869 + 006
0.05
.876 + 0004
840 0077
.875 .005
.77 1011
.811 811 .008
.68 + .004
.858
8644
=
.829
.006
AncCone
.46 .002
+ 006
= .009
834 600
847 + .005
0.20
830 + .013
839 .009
833 .006
832 = 007
844 011
.836 + 000
0.25
.31 + 0088
838 0001
.31 009
831 = 0014
843 = .006
832 + .00
0.30
.832 + .011
.845 + .008
.826 .009
.27 # .006
.83 000
.834 + .009
0.40
.825 .002
840 -01
.819 .010
829 + 000
837 .004
885 + .011
0.45
.827 + .002
.35 : 007
.23 + .001
.31 = 008
837 = 006
.824 + .006
0.50
.829 = .007
.836 + 0013
.820 .009
825 = .005
836 .007
.821 4: .007
.804 -211
.018
1.000 0.000
= .200
.93 .037
1.000 0.000
0.01
888 + .660
$45 188
1.000 0.000
.055 = 015
916 025
1 1.000 + 0.000
0.05
.967 = .009
.688 + .012
1,000 0.000
.938 = .030
.988 .012
1. 1.000 + 0.000
0.10
.009
1,000 0.000
= .100
1.000 0.000
ConTask
1.000
0.000
1.000 +: 0.000
1.000 0.000
1.000 0.000
1.000 0.000
1.000 + 0.000
0,25
1.000 + 0.000
1.000 0.000
1.000 0.000
1.000 1.000000000 0.000
1.000 0.000
1.000 + 0.0
0.30
1.000 0.000
1.000 0.000
1.000 0.000
1.000 - 0.000
1.000
1.000 + 0.000
0.35
1.000 0.000
1.000 1 0.000
1,000 0.000
-.000
1.000 0.000
0.45
1.000
0.000
1.000 + 0.000
1.000
0.000
1.000
0.000
1.000
0.000
1.000 + 0.000
0,50
1.000 + 0.000
1.000 + 0.000
1,000 0.000
1.000
0.000
1.000
0.000
1. 1.000 + 0.000
0.00
.500
1.000
.490
1.000 0.000
.526 .024
1.000 + 0.000
.500 .041
.488 + .099
1.000 0.000
.487 + .033
0.05
.649 + .045
1.000 + 0.000
.631 .028
.583 + .550
1.000 = 0.000
.656 + .019
0.10
817 + .046
1.000 0 0.000
7788 + 038
831 = .038
1.000 = 0.000
.855 + .0199
CocCone
0.15
.9664
1.000 0.000
.933
.027
1.000 0.000
1: .023
.987 .010
1.000 1 0.000
.981 .034
.970 - .038
1.000 0.000
.983 4 .022
0.25
.999
.001
1.000 + 0.000
1,000 0.000
.999 + .002
1.000
0.000
.999 + .001
0.30
1.000 + 0.000
1.000 + 0.000
1.000
0.000
1.000 + .001
1.000 + 0.000
1.000 + 0.000
0.35
1.000
0.000
1.000 0.000
1,000
0.000
1.000 0.000
1,000
0.000
1.000
0.40
1.000 0.000
0.000
1,000
0.000
1.000 0.000
1.000
0.000
1: 0.000
1.000 - 0.000
1.000
0.000
1.000
0.000
1.000 - 0.000
1.000 0.000
1.000 0.000
0.50
1.000 - 0.000
1.000
0.000
1.000
0.000
1. 1.000 - 0.000
1.000 0.000
1.000 0.000
35

---

Table 17: Results for the completeness dataset when not allowing for shared parameters with independent training using ASM, and considering oracle task expert and human80 concept expert. LS refers to the subell-moothhin--fe free implementation, while NLS to the one with label smoothing. We report avg = std and highlight the best baseline in bold.
Metric
DCBM-I-
DCBM- -I-M-
DCBM-XT
DCBM-NC-XLLS
DCBN-XT
1.000 0.000
1.000 + 0.000
.220
1.000 = 0.000
1.000 0.000
+ .013
0.01
1.000 - 0.000
1.000 + 0.000
817 .023
.909 = .002
.999 .002
824 + .007
0.05
.967 = .019
.976 + 027
.838 0011
.975 = .014
.986 .012
.27 + .010
0.10
904 .022
.888 + .018
.835 .066
+ .010
921 .038
.80 +: .023
888 +
.24
+ .044
.899 .014
0100
AnTTus
0.20
846 + .015
842 001
832 .018
845 = 009
862 015
836 + 0015
0,25
826
.39 0013
.244 + .00
828 = 014
822 .018
821 + .010
0.30
$19
.888 .02
.835 .010
.29 = .100
.819 008
.816 -: .011
0.35
.811 009
832 015
.226
.188
837 0055
.87 :: .018
.39 2000 .015
832 013
.821
844 012
828 .19
885 .016
0.45
884 .000
833 012
842 .008
821 = 021
830 .055
825 + .00
0,50
.23 + .012
831 + 0100
.826 .122
805 + .017
830 000
827 + .018
+
.005
0.01
884 006
843 .100
885 .022
885 + .05
843 4 005
883 + 003
0.05
881 + .00
+45
881 + .004
.81 = ,011
844 0003
.76 + .005
0.10
.699 + .003
.43 .010
.800 = .007
.864 = .003
846 .006
.833 + .007
AvcCCnn
.846
.008
.850
= .007
.87
.003
.35
844 006
.388 .008
837 = 000
848 .077
+ .00
0.25
830 + .006
$43 003
832 + .005
833 = 008
843 .007
.39 + 0015
0.30
836 + 007
849 008
.31 .001
832 = .008
811 = .007
881 + .005
0.35
.830 + .008
.845 + .004
.824 .013
.37 + .002
8844 + .012
.831 + .008
.833
+ = .100
.007
,832 .009
.845 012
.829 # .011
844 .012
.322 + .007
0,50
.826 + .003
.511 = 007
.21 + .008
827 = .044
811 .008
.811 + .008
0.00
0.000 0.000
0.000 0.000
1,000 0.000
0.000 0.000
0.000
.005 .007
0.000
1.000 0.000
.017
.005
1
0.05
.32 + .140
.327 + 205
1.000 0.000
205 = .117
.19 100
1 1.000 + 0.000
0.10
.768 + 117
814 003
1,000 0,000
.670 = .092
684 279
1. 1.000 + 0.000
ConTask
0.15
.891 + .015
.833 + 006
1.000 0.000
.870 = .041
.96 .20
1. 1.000 + 0.000
0.25
.988 .010
.90 .011
1.000 0.000
.991 + .013
.98 004
1. 1.000 + 0.000
0,30
.997 + .007
.95 006
1.000 0.000
.987 = .019
1000 1:00:::0 0.000
1.000 0 0.000
0.35
1.000 = 0.000
.999 000
1.000 0.000
1.000 = 1.000000000 0.000
1.000 0:00=0000 0.000
1.000 + 0.000
0.40
1.000 0.000
1.000
1,000 0.000
1.000
1.000 0.000
0.50
1.000
0.000
1.000 +: 0.000
1.000
0.000
1. 1.000 0.000
1.000
0.000
1 1.000 + 0.000
0.00
.388 .015
1.000 +: 0.000
.112 .045
.039
1.000 0.000
.466 .22
.480 + .040
.478
1.000 0.000
.603 .034
1.000 + 0.000
.595 .040
+ .28
1.000 0.000
.607 + .052
0.10
.800 + .021
1.000 + 0.000
.797 4 .023
.773 = 043
1.000 0.000
.783 + .211
ConCone
0.15
.933 + .004
1.000 + 0.000
.908 .02
.924 = .211
1.000
.908 4 .015
0.20
.981
1.000 0.000
.911
- .011
1.000 0.000
+: .008
.997 .006
.999
.008
0.30
1.000
0.000
1.000 + 0.000
1.000
0.000
.99 + .001
1.000
0.000
1.000 + 0.000
0.35
1.000 + 0.000
1.000 + 0.000
1,000
0.000
1.000 + 01.00
1.000
0.000
1. 1.000 + 0.000
0.40
1.000 + 0.000
1.000 + 0.000
1,000
0.000
1.000 0.000
1.000 0.000
1.000 0.000
0.45
1.000 0.000
1.000 1 0.000
1.000
0.000
1.000 - 0.000
1.000 0.000
1.000 0.000
1.000
1.000 0.000
1.000
0.000
1.000 - 0.000
0.000000000
1.000 0.000
36

---

Table 18: Results for the complet dataset when not allowing for shared parameters with joint training using ASM, and considering oracle task expert and human80 concept expert. We report avg std and highlight the best baseline in bold.
Metric
DCBM-LS
DCBM-NC-LS
DCBN-XT- XT-
DCBM-NLSS
DCBN-NC-NLLS
OCBN-XT-HI -T-S
0.00
1.000 = 0.000
1.000 + 0.000
.844 .013
1.000 = 0.000
1.000 0.000
841 + .019
0.01
1.000 + 0.000
1.000 = 0.000
.200 002
1.000 = 0.000
1.000 0.000
826 + .012
0.05
988 0008
.911 .012
.831 .23
.992 .012
.957 .018
883 + .019
+ 025
.22
+ 021
J26
+ 009
0.15
886 + .010
887 + 018
833 .066
885 = 017
882 = 027
829 + 009
AnrFast
0.20
854 .019
.69 + 0111
.823 .014
854 = 016
864 .012
826 + .055
0.25
.44 + .013
.833 + .088
.831 .013
850 = .016
852 = .010
.28 4 .025
0.30
.800
.828
-013
843
+ 010
812 = 012
+ .013
0.40
.87 + .012
841 = 007
837 : .018
831 = 013
833 013
824 + 0014
0.45
.826 + 000
.819 = 007
.823 = 017
.37 = .114
.37 = 016
824 :: .012
0.50
.830 + .018
.823 + .017
.816 .017
.299 + .002
.24 .009
.825 + .020
0.00
.833 .008
839 + U10
876 .05
877 + 006
836 006
.82 0012
0.01
878 + 006
840 + 007
869 + 003
876 + 0004
842 009
.69 + 006
0.05
.874 0004
.40 0006
.875 .005
875 = 2013
.43 008
.68 + .004
.858
.644
=
831
+ .006
AncCome
.46 .006
007
= .10
835 = .114
847 + .005
0.20
830 + .011
836 .055
833 .006
834 006
843 = 009
.836 + 000
0.25
.31 + 0088
841 + .005
.311 000
833 = 005
846 .00
832 + .007
0.30
.831 + .011
.844 .008
.826 .009
.29 + .005
.899 008
.834 + .009
0.40
.823 .006
841 + .009
.819 .010
829 + 006
.87 .004
.855 + .011
0.45
.844 .005
.36 + 007
.223 .001
.35 = .008
839 = 005
.824 + .006
0.50
.825 + .001
.837 + 012
.820 .009
823 = .003
.834 .007
.821 4: .007
0.000
0.000
0.000 + 0.000
1.000
0.000
0.000 = 0.000
0.000 0.000
1.000 + 0.000
0.01
0.000 + 0.000
0.000 + 0.000
1.000 0.000
0.000 + 0.000
0.000 0.000
1 1.000 + 0.000
0.05
.129 + 0888
.246 + 138
1,000 0.000
.071 = .066
.167 1164
1. 1.000 + 0.000
0.10
.466 .336
576 188
1,000 0.000
.585 = .611
.637
1.000 0.000
ConTask
.872
.988 .018
.909 .020
1.000 0.000
.932 + .244
.299 .008
1. 1.000 + 0.000
0,25
.959 + .019
.65 022
1.000 0.000
.973 = .013
.62 16 .023
1. 1.000 + 0.000
0.30
1.000 = 0.000
.989 019
1.000 0.000
.986 = .016
.986 .014
1.000 1 0.000
0.35
.994 .007
1.000 1 0.000
1,000 0.000
.002
1.000 0.000
1.000 0.000
+
.999
0.45
1.000
0.000
1.000 + 0.000
1.000 0.000
1.000 0.000
1.000 0.000
1 1.000 + 0.000
0,50
1.000 + 0.000
1.000 + 0.000
0.000 1,000
1.000
0.000
1.000 0.000
1 1.000 + 0.000
0.00
.522
1.000
.490
1.000 0.000
.058
.549 .022
1.000 + 0.000
.500 .041
.524 + .038
1.000 0.000
.487 + .033
0.05
.667 + .044
1.000 + 0.000
.631 .028
.604 + .053
1.000 = 0.000
.656 + .019
0.10
.877 + .012
1.000 0. 0.000
778 + 038
813 = .040
1.000 = 0.000
.855 + .0199
CocCone
0.15
.926 .015
1.000 0.000
.953
.023
1.000 0.000
1: .023
.990 .008
1.000 1 0.000
.981 .034
.972 - .036
1.000 1:000:000 0.000
.983 4 .022
0.25
.999
.001
1.000 + 0.000
1,000 0.000
1,000 + 0.000
1.000
0.000
.999 + .001
0.30
1.000 + 0.000
1.000 + 0.000
1.000
0.000
1.000 + .001
1.000 + 0.000
1.000 + 0.000
0.35
1.000
0.000
1.000
0.000
1,000
0.000
1.000 + 0.000
1,000
0.000
1.000
0.40
1.000 0.000
0.000
1,000
0.000
1.000 0.000
1.000
0.000
1.000 - 0.000
1.000
0.000
1.000
0.000
1.000 - 0.000
1.000 1 0.000
1.000 0.000
0.50
1.000 - 0.000
1.000
0.000
1.000 0.000
1. 1.000 - 0.000
1.000 0.000
1.000 0.000
37

---

Table 19: Results for the completenes, dataset when not allowing shared parameters with independent training using ASM, and considering human60 task expert and oracle concept expert. LS refers to the labol-smoothing- free implementation, while NLS to the one with label smoothing. We report avg = std and highlight the best baseline in bold.
Metric
DCBM-IS
DCBM- -C-I-
DCBN-XT
DCBM-NT
.881 0200
.813 + 0008
.906
= .032
.828 .010
+ .010
0.01
.17 = .001
.825 + 015
.914 = .019
.87 = .266
.831 = .017
.19 + .02
0.05
.933 + .019
.836 .013
.914 = .200
.884 = .029
.833 .014
.904 + .025
0.10
885 0033
.14 .014
.921 .200
+ .033
.844 .014
+ .044
873 038
016
+ .035
.10 006
AnTTus
0.20
864 .29
.825 014
884 007
.82 = 014
834 .166
.86 + 002
0,25
845 + .027
.36 011
.851 .031
829 = .188
817
.871 + .255
0.30
.388 + .066
+
.823 .019
.31 = .018
821
.830 + .018
0.35
810 .015
014
.833
027
+ .017
.34 + .017
824 019
.311 .013
.31 .009
831 .166
+ 0011
0.45
.366 + .015
.835 014
.83 .011
.31 = .04
827 = .016
.26 + .018
0,50
.822 + .018
.818 + .012
.834 .011
807 + .010
828 .006
832 + .013
0.000
+ 0008
0.01
1.000 + 0.000
1.000 0.000
1.000 + 0.000
843 + 005
1 1.000 + 0.000
0.05
.999 0000
$45
.999 + .001
.999 = .001
.844 .003
.998 + .001
0.10
.955 + .001
843 .010
.996 .001
.996 = .001
.846 .006
.966 + .001
AvcConn
847
.877
+ .001
975
844 .00
.975
.973 003
848 .007
+
0,25
942 = .005
$43 .003
944 .004
.40 = 007
815 .007
.933 + 000
0.30
890 = 008
819 .088
.992 = 006
9011 = .006
841 .007
898 + 000
0.35
.864 + .005
.845 .004
.863 .005
867 .002
884 + .012
.81 + .004
.843
.42
.846
849 0088
845
.812 .010
= .100
844 .012
.837 + .005
0,50
.840 + .001
.511 = 007
.366 .008
839 = .005
811 = .008
.810 + .006
0.00
1.000 0.000
1.000 0.000
1,000
0.000
1.000 0.000
0.000
1.000 0.000
1.000
1.000 0.000
1.000 0.000
0.05
1.000 0.000
1.000 - 0.000
1.000 0.000
1.000 = 0.000
1.000
1.000 + 0.000
0.10
1,000 0.000
1.000 = 0.000
1,000 0,000
1.000 = 0.000
1.000=0.000
1. 1.000 + 0.000
ConTask
0.15
1.000 0.000
1.000 + 0.000
1,000 0.000
1.000 0.000
=
1.000 + 0.000
0.25
1.000 0.000
1.000 +: 0.000
1.000 0.000
1.000 = 0.000
0.000 1.000
1.000 + 0.000
0,30
1.000 0.000
1.000 : 0.000
1.000 0.000
1.000 = 0.000
1.000 0.00000000
1.000 0 0.000
0.35
1.000 0.000
1.000 = 0.000
1.000 0.000
1.000 0 0.000
1.000 0.000
1.000 1 0.000
0.40
1.000 0.000
1.000 0.000
1,000 0.000
0.000
1.000 0.000
0.50
1.000
0.000
1. 1.000 +: 0.000
1.000 0.000
1. 1.000 0.000
1.000
0.000
1 1.000 + 0.000
0.00
0.000 + .001
1.000 1: 0.000
0.000 0.000
+ .001
1.000 0.000
+ 000
.009
.019
0.000
.082 .001
1.000 + 0.000
.096 .028
.089 + .26
1.000 0.000
.107 + .019
0.10
.411 + .035
1.000 + 0.000
.25 4 .025
215 027
1.000 0.000
.214 # .00
ConCone
0.15
.356 + .011
1.000 - 0.000
.330 1 031
375 = .211
1.000
.363 + .046
0.20
.481 .027
1.000 0.000
.966
.519 .02
1.000 0.000
.485 1 .025
.031
.652
.046
0.30
.800 + J032
1.000 + 0.000
.842 J016
815 + 028
1.000
0.000
.824 + .030
0.35
943 + .008
1.000 + 0.000
920 .009
928 + .017
1.000 + 0.000
935 + 0099
0.40
.994 + .001
1.000 + 0.000
.994 0005
.993 - .007
1.000
.988 4 .010
0.45
1.000
1.000 1 0.000
1.000
0.000
1.000
0.000
- .001
1.000 1.000:00000
1.000 0.000
1.000 0.000
1.000 - 0.000
1.000
1.000
38

---

Table 20: Results for the completenent dataset when not allowing for shared parameters with joint training using ASM, and considering human60 task expert and oracle concept expert. We report avg std and highlight the best baseline in bold.
Metric
DCBM-LS
DCBM-NC-LS DCEN-XT-BE XT-
DCBM-NLS
DCBM-NC-NIA.
DCBN-XT-HH XT-NNS
0.00
.877 + 0311
.825 + .009
.906 .026
899 = .03
.828 = .012
.944 + .026
0.01
886 + .019
.899 - .012
.930 .013
883 = .014
.800 = .010
.909 + .024
0.05
888 027
.866 + .014
.896
.834 .015
.977 + .003
902
0155
.903
+ .009
0.15
867 .006
.877 022
898 006
885 = 023
818 0077
913 + .018
AnrFasi
0.20
.633 + .020
.825 .01
.889 .005
.77 = 028
.32 008
.892 + .025
0.25
.599 + .028
.835 .009
.878 .018
.62 + .006
825 = 0155
.822 + .016
0.30
.845 .017
.82
.849
+310
: .023
.888 .019
.25 015
.500
= 013
831 .088
0.40
.825 + .006
825 015
840 .055
830 = .155
826 0011
835 + 002
0.45
.833 + .013
.822 010
.831 = 006
843 = .114
.83 = .022
883 : .013
0.50
8400 + .014
.825 + .U10
.826 .008
.37 + .009
.820 +017
.829 + .012
0.00
1.000
0.000
839 + OU7
1.000
0.000
1. 1.000 + 0.000
838 004
1 1.000 + 0.000
0.01
1.000 + 0.000
840 + 007
1,000 0.000
1.000 = 0.000
844 .100
1 1.000 + 0.000
0.05
1.000 = 0.000
.800 .009
.999 .001
1.000 = 0.000
.42 .00
.999 + .001
.997
.40
=
.31
+ .011
AncCone
005
.987 .004
.999 = .00
.10
+ .004
0.20
973 + .00
.40 .008
.973 + .002
976 000
815 008
977 + .033
0.25
945 + .006
813 007
.988 = 006
949 = .008
846 # 000
.966 + .00T
0.30
.955 .003
.846 + 1006
.903 .006
.100 + .007
.835 + .009
.905 + .008
0.40
.837 .007
.800 + .009
841 .008
846 + .010
.38 .006
.899 + .004
0.45
.836 + .003
.888 + .008
.333 + .008
843 = .055
842 .006
.831 + .007
0.50
.836 + .006
.837 + 0013
.835 .008
.33 = .007
.34 = .007
.83 : .007
1.000 0.000
1.000 +: 0.000
1.000
0.000
1.000 = 0.000
1.000 0.000
1.000 0.000
0.01
1.000 + 0.000
1.000 +: 0.000
1.000 0.000
1.000 = 0.000
1.000 0.000
1.000 + 0.000
0.05
1.000 0.000
1.000 + 0.000
1,000 0.000
1.000 = 0.000
1.000=0.000
1. 1.000 + 0.000
0.10
1.000 0.0000
1.000 0.000
1,000 0.000
1.000 0.000
1.000
1.000 + 0.000
ConTask
1.000 0.000
1.000 +: 0.000
1.000 0.000
1.000
0.000
1.000 0.000
1.000 + 0.000
0,25
1.000 0.000
1.000 : 0.000
1.000 0.000
1.000 = 0.000
1.000 0.000
1.000 + 000
0.30
1.000 0.000
1.000 = 0.000
1.000
0.000
1.000 = 0 0.000
1.000
1.000 + 0.000
0.35
1.000 0.000
1.000 1 0.000
1,000 0.000
-.000
1.000 0.000
1.000
1.000
0.45
1.000
0.000
1.000 + 0.000
1.000
0.000
1.000
0.000
1.000 0.000
1.000 + 0.000
0,50
1.000 + 0.000
1.000 + 0.000
1,000 0.000
1.000 + 0.000
1.000
0.000
1. 1.000 0.000+00.00 0.000
0.00
1.000
0.000
1.000 0.000
0.000
.009 .007
1.000 + 0.000
.009 .009
.005 = .003
1.000 0.000
.009 + .006
0.05
.086 + .026
1.000 + 0.000
.083 .028
.059 = .014
1.000 = 0.000
.083 + .005
0.10
.170 + .010
1.000 + 0.000
.174 .23
188 - .20
1.000 0.000
.193 + .027
CocCone
0.15
.366
1.000 0.000
.322
=
1.000 0.000
1: .012
.471 .037
1.000 1 +.000..000 0.000
.144 .088
412 = .009
1.000 0.000
.449 + .026
0.25
.628 .045
1.000 + 0.000
.589 J034
.618 + .036
1.000 0.000
.628 + .038
0.30
.745 + .027
1.000 + 0.000
.773 .015
.767 + J037
1.000 + 0.000
.741 + .023
0.35
.927 .019
1.000 0.000
.227 .U88
+ .025
1,000 0.000
.96 + .029
0.40
.982 .012
0.000
.970 2
= .02
1.000 0.000
4: .014
.998 .002
1.000 0.000
.996 .004
.999 0.000
1.000 1:000:000 0.000
.996 4 .004
0.50
1.000 -: 0.000
1.000 -: 0.000
1.000 0.000
1.000 - 0.000
1.000 0.000
1.000 + 0.000
39

---

Table 21: Results for the completeness dataset when not allowing shared parameters with independent training using ASM, and considering human80 task expert and oracle concept expert. LS refers to the -mwol-smoothiig----- free implementation, while NLS to the one with label smoothing. We report avg = std and highlight the best baseline in bold.
Metric
DCBM-IS
DCBM- -C-I-S
DCBN-XT
DCBM-NT
.921 0211
.838 + 019
.906
.919 = .019
.815 -
.925 + .010
0.01
.902 = .017
.817 = 013
.914 1 .099
.903 = .300
811 = .009
.19 + .02
0.05
.906 + .019
.812 = .019
.914 = .200
.897 = .033
.833 .014
.904 + .025
0.10
.77 .055
.812 + .015
.921 .200
+ .044
.82 022
+ .034
863 .016
022
+ .030
006
AnTTus
0.20
866 026
.24 010
884 007
.80 = 013
831 0011:0144
.86 + .02
0,25
853 + .028
.33 0008
.851 .031
833 = .088
817
.871 + .255 .025
0.30
842 + .019
.835 .014
.823 .099
.31 000
822 2014
.830 + .018
0.35
.855 .016
.28 + 012
.833
837 020
.32
+ .017
.835 .019
834 016
.31 .013
= .100
832 .188
+ .011
0.45
.39 + 014
831 # 014
.37 .011
831 = 007
837 .014
.22 + .018
0,50
.821 + .014
.23 + .111
.844 .011
805 + .013
834 + .010
882 + .013
0.000
+ 008
0.01
1.000 + 0.000
843 .110
1.000 0.000
1.000 + 0.000
843 005
1 1.000 + 0.000
0.05
.999 001
$45+.007
.999 + .001
.999 = .001
.84 .003
.998 + .001
0.10
.955 + .001
843 .010
.996 = .001
.996 = .001
.846 .006
.966 + .001
AvcConn
847
.877
+ .00
.75
844 .06
.975
.973 003
848 .007
+ .002
0,25
942 = .055
$43 .003
944 .004
.40 = 007
815 .007
.93 + 000
0.30
890 = 008
819 0088
.992 = 006
9011 = .006
841 0007
898 + 000
0.35
.864 + .005
.845 .004
.863 .005
867 .002
884 + .012
.81 + .004
.42
.846
849 .088
841 012
.812 .010
= .100
844 .012
.837 + .005
0,50
.840 + .001
.811 007
.36 .008
.39 = .005
811 .008
8810 + .006
0.00
.917
1,000 0.000
= .015
.943
0.000
.49 .014
.43 023
1.000 0.000
.946 + 046
.63 022
1.
0.05
.74 + .022
.85 019
1.000 0.000
.999 = 030
.96 1 007
1. 1.000 = 0.000
0.10
.91 + .013
.99 002
1,000 0,000
.987 = 020
1.000 - 0.000
1. 1.000 + 0.000
ConTask
0.15
.999 + .002
.997 +: .007
1,000 0.000
.975 = .050
1.000 0.000
1.000 + 0.000
0.25
1.000 0.000
1.000 + 0.000
1.000 0.000
1.000 = 0.000
1.000 0.000
1.000 + 0.000
0,30
1.000 0.000
1.000 0.000
1.000 0.000
1.000 = 0.000
0.000:00000
1.000 + 0.000
0.35
1.000 0.000
1.000 : 0.000
1.000 0.000
1.000 = 0.000
1.000 10000:00000 0.000
1.000 1 0.000
0.40
1.000 0.000
1,000 0.000
0.0000
1.000 0.000
0.50
1.000
0.000
1. 1.000 +: 0.000
1.000 0.000
1. 1.000 0.000
1.000
0.000
1 1.000 + 0.000
0.00
0.000 + .001
1.000 1: 0.000
0.000 0.000
+ .001
1.000 0.000
+ 000
.009
.019
0.000
+:
.082 .001
1.000 + 0.000
.096 .028
.089 + 026
1.000 0.000
.107 + .019
0.10
.411 + .035
1.000 + 0.000
.25 4 .025
215 027
1.000 0.000
.214 # .00
ConCone
0.15
.356 + .011
1.000 - 0.000
.330 1 031
375 = .211
1.000
.363 + .046
0.20
.481 .027
1.000 0.000
.966
.519 .02
1.000 0.000
.485 1 .025
.031
.652
.046
0.30
.800 + J032
1.000 + 0.000
.842 J016
.15 + 028
1.000 0.000
.824 + .030
0.35
943 + .008
1.000 + 0.000
920 .009
928 + .017
1.000 0.000
935 + 0099
0.40
.994 + .001
1.000 + 0.000
.994 0005
.993 - .007
1.000
.988 4 .010
0.45
1.000
1.000 1 0.000
1.000
0.000
1.000
1.000=00000
- .001
1.000 1
1.000 0.000
1.000 0.000
1.000 - 0.000
1.000
1.000
40

---

Table 22: Results for the completeness dataset when not allowing for shared parameters with joint training using ASM, and considering human80 task expert and oracle concept expert. We report avg std and highlight the best baseline in bold.
Metric
DCBM-LS
OC-M-N
DCBN-XT-BE XT-
DCBM-NLS
DCEM-NC-MIL
DCBN-XT-HM XT-NIS
0.00
.892 + 021
.838 + 010
.906 .026
.30 = .004
.848 .010
.924 + .22
0.01
.933 + .028
.816 = 016
.930 .013
.902 = .023
.818 000
.909 + .024
0.05
.897 022
.800 .221
.866
.019
.839
.977 +: .003
.90
017
.903
+ .007
0.15
877 + .034
.22 + 021
898 006
888 = 026
824 .019
913 + 0018
AnrFasi
0.20
.688 + .016
.82 .019
.889 .005
.033 = .188
.87 = .21
.892 + .025
0.25
.66 + .021
.843 0111
.878 .018
.888 = .020
.19
.82 + .016
0.30
.850
830
.849
+:
836 .011
015
.500
= 013
835
+
0.40
.344 + 000
830 010
840 .055
831 = .16
827 2 0114
835 + 002
0.45
.34 + .010
.819 011
.831 = 006
.388 = .11
.85 = 009
883 : .013
0.50
.835 .008
.827 + .009
.826 .008
.35 .008
.824 .00
.829 + .012
0.00
1.000 0.000
840 + D0S
1.000
0.000
1. 1.000 + 0.000
835 005
1 1.000 + 0.000
0.01
1.000 + 0.000
840 + 006
1,000 0.000
1.000 = 0.000
845 = 002
1 1.000 + 0.000
0.05
.999 0.000
.81 .010
.999 .001
1.000 = 0.000
844 006
.999 + .001
.996
=
.833
+ .001
AncCConc
$29 007
.977 .004
.90 = .002
= 000
+ .004
0.20
973 + .006
840 .009
.975 + .002
975 = 006
816 000
977 + .033
0.25
.48 + 000
813 000
.48 = 006
949 = .008
846 = 002
.966 + .00T
0.30
.944 + .006
.866 + .007
.903 .006
.006
.80 + 000
.905 + .008
0.40
.836 .007
.80 + .009
841 .008
846 + .009
838 006
.839 + .004
0.45
.834 + .003
.36 007
.33 + .008
.812 = .004
842 = 008
.831 + .007
0.50
.837 = .007
.838 + 011
.835 .008
832 = .007
.86 .006
.832 : .007
.838 210
.943 0044
1.000
0.000
.919 = .031
.939 .022
1.000 0.000
0.01
.300 + .024
953 014
1.000 0.000
.946 = .41
942 .017
1 1.000 + 0.000
0.05
.973 + .300
.89 016
1,000 0.000
.983 = .014
.986 :: .019
1. 1.000 + 0.000
0.10
.002
.001
1,000 0.000
1.000 0.000
1.000 + 0.000
ConTask
+
1.000 0.000
1.000 +: 0.000
1.000 0.000
1.000 0.000
1.000 0.000
1.000 + 0.000
0,25
1.000 - 0.000
1.000 0.000
1.000 0.000
1.000 - 0.000
1.000 0.000
1.000 + 000
0.30
1.000 0.000
1.000 1 0.000
1.000 0.000
1.000 - 0.000
1.000
1.000 + 0.000
0.35
1.000 0.000
1.000 0.000
1,000 0.000
0.000
1.000 0.000
1.000
0.45
1.000
0.000
1.000 + 0.000
1.000
0.000
1.000 0.000
1.000
0.000
1.000 + 0.000
0,50
1.000 + 0.000
1.000 + 0.000
1,000 0.000
1.000 0.000
1.000
0.000
1. 1.000 0.00000.0000 0.000
0.00
.000
1.000
0.000
0.000
1.000
0.000
0.000
.012 .009
1.000 + 0.000
.009 .009
.005 = .003
1.000 0.000
.009 + .006
0.05
.086 + .025
1.000 + 0.000
.083 .028
J061 + .055
1.000 = 0.000
.083 + .005
0.10
.170 + .037
1.000 + 0.00
.174 .23
.10 + .22
1.000 = 0.000
.193 + .027
CocCone
0.15
.304 .033
1.000 0.000
.322
.035
1.000 0.000
1: .012
.470 .032
1.000 + 0.000
.144 .088
445 = .009
1.000 0.000
.449 + .026
0.25
.620
.149
1.000 + 0.000
.589 J034
.616 + .03
1.000 0.000
.628 + .038
0.30
.747 + .034
1.000 + 0.000
.773 .015
.774 + J037
1.000 + 0.000
.741 + .023
0.35
.924 .119
1.000 0.000
.227 .U88
+ .025
1,000 0.000
.966 0029
0.40
.985 .007
0.000
.970 2
1.000 0.000
4: .014
.998 .003
1.000 0.000
.996 .004
.999 - .001
1.000 0.000
.996 4 .004
0.50
1.000 + 001
1.000 -: 0.000
1.000 0.000
1. 1.000 - 0.000
1.000 : 0.000
1.000 + 0.000
41

---

Table 23: Results for the completenese dataset when not allowing shared parameters with
independent training using ASM, and considering oracle task expert and oracle concept expert. LS refers to the - free implementation, while NLS to the one with label smoothing. We report avg + std and highlight the best baseline in bold.
Metric
DCBM-I-S
DCBM-NC-I.S
DCBN-XT -T-LS
DCBK-NC-XLS
DCBM-XT MN-N--T-LLSS
1.000 0.000
1.000 + 0.000
.906
1.000 = 0.000
1.000 = 0.000
+ .010
0.01
1.000 - 0.000
1.000 + 0.000
914 = .099
1.000 = 0.0000000 0.000
.999 .002
.919 + .022
0.05
.999 .002
.976 + 007
.914 = .000
.997 = .004
.986 .012
.904 + .025
0.10
.79 004
.888 + .118
.21 020
.922 .008
U21 008
+ .034
+ 022
.933
.955 .122
.89 .014
006
AnTTus
0.20
890 + .010
.842 .111
884 007
890 = 008
862 0055
.86 + 0002
0,25
.59 + 030
.39 .013
.851 .031
853 = .008
822 .018
.871 + .025
0.30
.41 + .088
.838 .012
.823 .019
842 = .018
.819 008
.830 + .018
0.35
.877 .016
.32
.833
.35 .088
837 0055
+: .017
.32 .013
.32 013
.31 .013
= 014
828 .19
+ .011
0.45
842 + 015
833 012
.37 .011
831 = 007
830 .015
.26 + .018
0,50
.833 + .010
.81 + .110
.834 .011
808 + .017
.30 .008
882 + .013
0.000
+ 008
0.01
1.000 + 0.000
843 .110
1.000 0.000
1.000 + 0.000
843 + 005
1 1.000 + 0.000
0.05
.999 0000
$45+.007
.999 + .001
.999 = 0001
.84 .003
.988 + .001
0.10
.955 + .001
.43 .010
.996 .001
.996 = .001
.846 .006
.996 + .001
AvcConn
847
.877
+ .01
975
844 .06
.975
.973 003
848 .007
+
0,25
942 = .055
$43 .003
944 .004
.40 = 007
815 .007
.933 + 000
0.30
890 = 008
819 .088
.992 = 006
9011 = .006
841 .007
898 + 000
0.35
.864 + .005
.84 .004
.863 .005
867 .002
884 + .012
.81 + .004
.843
.846
849 0088
845
.812
= .100
841 .012
.837 + .005
0,50
.840 + .001
.811 = 007
.366 .008
839 = .00
811 = .008
8810 + .006
0.00
0.000 0,000
0.000 0.000
1,000 0.000
0.000 0.000
0.000
0.000
.004 .007
0.000 0.000
1.000 0.000
.188
.005 .011
1
0.05
305 .344
.327 + 2055
1.000 0.000
212 = .108
.59 :: 103
1 1.000 = 0.000
0.10
650 + .16
.14 003
1,000 0,000
.01 = .056
684 279
1. 1.000 + 0.000
ConTask
0.15
.865 + .030
.833 + 0062
1.000 0.000
.888 = .059
896 .20
1. 1.000 + 0.000
0.25
.84 .014
+90
1.000 0.000
.980 + .024
.98 004
1. 1.000 + 0.000
0,30
.992 + .013
.95 006
1.000 0.000
.988 = .014
1000 1:00:::0 0.000
1.000 + 0.000
0.35
1.000
= 0.000
.999 002
1.000 0.000
1.000 0 0.000
1.000 0.000
1.000 + 0.000
0.40
1.000
1.000 0.000
1,000 0.000
1.000
1.000 0.000
0.50
1.000
0.000
1.000 +: 0.000
1.000 0.000
1. 1.000 0.000
1.000
0.000
1 1.000 + 0.000
0.00
0.000 + .001
1.000 1: 0.000
0.000 0.000
.000
1.000 0.000
+ 000
.009
0.000
.082 .001
1.000 + 0.000
.096 .028
.089 + 026
1.000 0.000
.107 + .019
0.10
.411 + .035
1.000 : 0.000
.25 4 .025
215 027
1.000 0.000
.214 .00
ConCone
0.15
.356 + .011
1.000 - 0.000
.330 031
375 = .211
1.000
.363 + .046
0.20
.481 .027
1.000 0.000
.966
.519 .02
1.000 0.000
.485 1 .025
.031
.652
.046
0.30
.800 + J032
1.000 + 0.000
.842 J016
815 + 028
1.000 0.000
.824 + .030
0.35
943 + .008
1.000 + 0.000
920 .009
928 + .017
1.000 0.000
935 + 0099
0.40
.994 + .001
1.000 + 0.000
.994 000
.993 - .007
.988 4 .010
0.45
1.000
1.000 0.000
1.000
0.000
1.000
-.000
- .001
1.000 1.000:00000
1.000 0.000
1.000 0.000
1.000 - 0.000
1.000
1.000
42

---

Table 24: Results for the compltteness dataset when not allowing for shared parameters with joint training using ASM, and considering oracle task expert and oracle concept expert. We report aug + std and highlight the best baseline bold.
Metric
DCBM-LS
DCBM-NC-LS
DCBN-XT-BE XS-
DCBM-NLS
DCBN-NC-NLL
DCBN-XT- CORN-XT-NILL
0.00
1.000 = 0.000
1.000 + 0.000
.906 .026
1.000 = 0.000
1.000 0.000
.924 + .02
0.01
1.000 0.000
1.000 + 0.000
.930 0033
1.000 = 0.000
1.000 0.000
.909 + .024
0.05
1.000 0.000
.979 + .133
.866 023
.044
.986 000
+ 0003
+ 022
.903
.077
900
.05
0.15
948 + .010
869 + 0155
898 006
949 = 011
861 .017
.13 + 0088
AndFast
0.20
.92 .022
.56 026
.889 .005
.920 = 024
876 .013
.892 + .025
.025
0.25
862 + 020
.845 .011
.878 .018
.888 = .200
.866 .012
.82 + .016
0.30
.899
834
.849
+: .023
830 .088
013
.550
= 021
+
0.40
.33 + .016
829 010
810 .055
828 = .188
841 = 011
.35 + 002
0.45
840 + 020
.822 = 010
.831 = 006
.39 = .011
.833 .009
.83 : .013
0.50
.840 + .016
.824 .014
.826 .008
845 .005
.824 + .011
.829 + .012
0.00
1.000 + 0.000
839 + OU7
1.000
0.000
1. 1.000 + 0.000
837 004
1 10000 + 0.000
0.01
1.000 0.000
841 + 008
1,000 0.000
1.000 + 0.000
844 + .012
1 1.000 + 0.000
0.05
.999 = .001
.844 .008
.999 .001
1.000 = 0.000
.846 = .00
.999 + .001
.996
+
+
.834
+ .01
AncCome
.003
.099
.987 .004
.991 = .00
836 = 600
+ .004
0.20
972 + 007
.41 + .008
.975 + .002
975 = 000
818 = .110
977 + .033
0.25
945 + .006
812 + .006
.988 .006
.918 = .008
846 = 000
.966 + .00
0.30
.922 .004
.85 + .005
.903 .006
.006
.88 0008
.905 + .008
.007
.00
0.40
.837 0008
.39 + .110
841 .008
846 + .012
.36 .006
.899 + .004
0.45
.833 + .000
.37 = 005
.333 + .008
8110 = .005
842 = 008
.831 + .007
0.50
.836 = .005
.837 + 011
.835 .008
832 = .008
.836 .010
.83 : .007
0.000 0.000
0.000 + 0.000
1.000
0.000
0.000 = 0.000
0.000 0.000
1.000 0.000
0.01
.144 + .031
0.000 - 0.000
1.000 0.000
.003 = .07
0.000 0.000
1 1.000 + 0.000
0.05
.215 + .993
.288 .00
1,000 0.000
.136 = 108
.178 = 009
1. 1.000 + 0.000
0.10
.609 .132
764 .076
1,000 0.000
=
.771 =
1.000 0.000.000 0.000
ConTask
1.000 0.000
.003
0.20
.936 .027
.977 + .033
1.000 0.000
.902 + .065
921 .031
1. 1.000 + 0.000
0.25
.922 + .013
.80 018
1.000 0.000
.982 = .225
.788 27 .017
1.000 + 000
0.30
1.000 = 0.000
.998 + .003
1.000 0.000
.977 = .041
.996 .005
1.000 + 0.000
0.35
1.000 0.000
1,000 0.000
1.000
1.000 0.000
0.45
1.000
0.000
1.000 + 0.000
1.000 0.000
1.000 0.000
1.000 0.000
1.000 + 0.000
0,50
1.000 + 0.000
1.000 + 0.000
1,000 0.000
1. 1.000 0.000
1.000 0.000
1. 1.000 0.000 0.000
0.00
.001
1.000
0.000
1.000 0.000
0.000
.011 .007
1.000 + 0.000
.009 .009
.006 = .005
1.000 0.000
.009 + .006
0.05
.090 + .025
1.000 + 0.000
.083 .028
.065 = .16
1.000 = 0.000
.083 + .005
0.10
.173 + .037
1.000 + 0.000
.174 .23
197 = .033
1.000 = 0.000
.193 + .027
CocCone
0.15
.111 .035
1.000 0.000
.322
.038
1.000 0.000
1: .012
.473 .030
1.000 + 0.000
.744 .088
451 = .100
1.000 0.000
.449 + .026
0.25
.630
.049
1.000 + 0.000
.599 J034
620 + .00
1.000
0.000
.628 + .038
0.30
.748 + .037
1.000 + 0.000
.773 .015
.774 + 0036
1.000 + 0.000
.741 + .023
0.35
.122 .223
1.000 0.000
.227 .U88
+ 0225
1,000 0.000
.96 + 0029
0.40
.977
0.000
.970 2
1.000 0.000
4: .014
.997 .004
1.000 0.000
.996 .004
.999 - .001
1.000 0.000
.996 4 .004
0.50
1.000 +: 0.000
1.000 -: 0.000
1.000 0.000
1.000 + 0.000
1.000 0.000
1.000 + 0.000
43

---

Table 25: Results for the completeness dataset when allowing for shared parameters with indepen- dent training using ASM, and considering oracle task expert and oracle concept expert. LS refers to the soaeel-moothing-freee implementation, while NLS to the one with label smoothing. We report arg + std and highlight the best baseline in bold.
Metric
DCBM-I.
DCBM-NC-I.
DCBN-XT:
DCBM-NLS
DCBM-NC-NL.S
DCBN-NT -T-N-
1.000
0.000
1.000 + 0.000
.855
1.000 = 0.000
1.000 0.000
+ .023
0.01
1.000 = 0.000
1.000 + 0.000
914 = .026
1.000 = 0.000
1.000 0.000
.935 + .001
0.05
.996 :004
.974 = 0011
.922 .015
.992 = .006
.958 .032
.923 + .008
0.10
.78 .004
.26 + .047
.23
+ .022
914 .014
+ .024
032
.988
.95 + .07
866
+: 023
AnTTus
0.20
900 + 021
.80 013
.58 0088
.81 = 024
860 .100
.86 + 000
0,25
.47 + .024
846 006
852 + .016
.80 = .088
=30
841 10.0000 .017
0.30
.399 + 024
.831 .010
.838 .023
.22 = .010
.844 000
.833 4 .011
0.35
.823
.27 + .088
.829
= 010
843 013
:: 003
.233 .018
014
# .012
011
.29
+: .016
0.45
.833 + .014
827 = 0066
822 .001
821 = 014
838 = 015
.28 : .012
0,50
.199 + .009
.81 + .066
.823 .014
821 + .009
829 .007
.32 + .008
0.000
+ 007
1.000
869
0.01
1.000 + 0.000
869 + OU7
1.000 0.000
1.000 + 0.000
871 .02
1 10000 + 0.000
0.05
.988 + 0000
870 0044
.998 + .001
.977 = .001
872 .009
.998 + .001
0.10
.944 = .001
867 + .006
.995 .002
.994 = .002
866 = .014
.994 + .001
AvcConn
863 010
.988
= .002
.82
+
974 .002
863 010
.973 .003
.970 .00
862 = .011
.044
0,25
941 + .008
866 008
.988 = .009
949 = .003
869 000
.40 000
0.30
911 + 006
861 008
.911 = .001
.905 = .007
869 006
.905 : .00
0.35
.883 + .009
.899 + .008
.882 .004
.87 .001
.871 - .008
+ +010
.010
.868
.863 007
860 014
.872 .014
.80 = .100
.872 .010
+ .010
0,50
.800 + .008
866 012
.869 .005
862 = .013
.688 = .012
.864 + .008
0.00
0.000
0.000 0.000
1,000 0.000
0.000 0.000
0.000
0.000
.015 - .034
0.000
1.000 0.000
0.000 0.000
1.
0.05
.257 .172
.277 + 151
1.000 0.000
.65 .332
360 - 253
1 1.000 + 0.000
0.10
.633 + .064
607 +: 3349
1,000 0.000
633 = .277
659 # 155
1. 1.000 + 0.000
ConTask
0.15
.819 + .067
.077 + .182
1,000 0.000
.60 = .017
.924 022
1. 1.000 + 0.000
0.25
.990 + .010
.980 .018
1.000 0.000
.994 + .008
.93 006
1. 1.000 + 0.000
0,30
1.000 0.000
.998 .003
1.000 0.000
1.000 = 0.000
1.000 0.000
1.000 0 0.000
0.35
1.000 0.000
.999 000
1.000 0.000
.997 = .007
.997 .007
1.000 + 0.000
0.40
1.000 0.0000
1,000 0.000
1.000 0.000
1.000 0.000
0.50
1.000
0.000
1.000 +: 0.000
1.000 0.000
1. 1.000 0.000
1.000 0.000
1 1.000 + 0.000
0.00
.001 002
1.000 1: 0.000
0.000
0.000
.001
1.000 0.000
0.000 + 0.000
.019
=
.144 + .016
1.000 + 0.000
.123 .055
.162 .044
1.000 0.000
.157 + .030
0.10
.255 + .019
1.000 + 000
258 .022
270 041
1.000 0.000
.299 + .014
ConCone
0.15
.406 + .018
1.000 - 0.000
.110 011
438 = .30
1.000=0.000
.411 4: .015
0.20
.567 .23
1.000 0.000
.562
= .016
1.000 0.000
.39 4: .0133
.019
.706
.027
.022
0.30
.899 + .019
1.000 + 0.000
.556 J034
.69 + .017
1.000 0.000
.854 + .015
0.35
.555 + .004
1.000 + 0.000
.977 .012
969 + .008
1.000 0.000
968 + .014
0.40
.993 + .001
1.000 1 0.000
.996 0003
.994 + .003
1.000=0.000
.995 + .004
0.45
0.000
1.000 1 0.000
1.000 0.000
1.000 0.000
-.000000000
1.000 1 0.000
1.000 - 0.000
1.000 + 0.000
1.000 0.000
1.000 - 0.000
1.000
1.000 0.000
44

---

Table 26: Results for the eempleteness dataset when allowing for shared parameters with joint training using ASM, and considering oracle task expert and oracle concepi expert. We report std and highlight the best baseline bold.
Metric
DCBM-LS
DCBM-MC-LS
DCBN-XT-
DCBM-NISS
DCBN-NC-XLS
DCBN-XT- CEAN-XT-XILL
0.00
1.000 = 0.000
1.000 + 0.000
.93 .010
1.000 = 0.000
1.000 0.000
.919 + .017
0.01
1.000 + 0.000
1.000 + 0.000
.922 .008
1.000 = 0.000
1.000 0.000
.925 + .020
0.05
1.000 0.000
.63 .356
.905 0455
.044
.957 .010
+: .009
.633
+: 038
.900
.911 + .07
J14
+ 0100
0.15
945 - 0011
.76 + 021
.999 0011
937 = .033
.65 006
893 + 031
AnrFas
0.20
9011 + .27
.48 014
.90 .020
.111 = .025
840
.81 + .028
0.25
.844 + .355
.837 .008
.666 .017
.688 = .014
.455 = 028
883 + .029
0.30
.844
.843
.800
833 .011
016
.366
= 013
822
+ .018
0.40
.81 + .007
833 006
821 .018
838 = .122
812 .012
.828 + .017
0.45
.829 + .009
.823 017
.832 009
.32 = .014
.822 = 000
.333 +: 0014
0.50
.818 .024
.816 + .015
.828 .018
836 .014
.223 + .013
.829 + .025
0.00
1.000
0.000
866 + U10
1.000 0.000
1. 1.000 + 0.000
856 010
1 1.000 + 0.000
0.01
1.000 + 0.000
859 + 005
1.000 6.000
1.000 + 0.000
869 + 004
1 1.000 + 0.000
0.05
.997 = .001
.61 .009
.999 .001
.998 = .001
853 = .08
.999 + .001
.993
+
.996
.877 .015
+
AncCome
983 .006
864 015
.987 .006
.984 = .003
862 0011
+
0.20
967 + .008
.39 007
.972 = .006
974 .004
853 2: 000
.970 + 000
0.25
942 + 004
862 010
.911 .001
944 .009
856 = 008
.900 : .006
0.30
.933 .007
.888 .008
.906 .009
+ .010
.80 + 000
.909 + .007
.005
0.40
.800 .010
863 + .009
.863 .010
858 + .009
.59 +011
.854 + 0009
0.45
.856 + .009
.65 : 007
.852 + .007
855 = .07
858 .009
.859 + .00
0.50
.877 = .004
.851 + .001
.844 = .001
855 = .009
858 .007
861 + .006
0.000 0.000
0.000 + 0.000
1.000 0.000
0.000 = 0.000
0.000 0.000
1.000 + 0.000
0.01
0.000 + 0.000
0.000 + 0.000
1.000 0.000
.007 = .16
0.000 0.000
1 1.000 + 0.000
0.05
.119 + .995
.292 + .264
1,000 0.000
304 = .181
.33 = .114
1. 1.000 + 0.000
0.10
.10 .00
.741 + .80
1,000 0.000
= .178
.661 1644
1.000 0.000
ConTask
+116
.088
.100 + .077
.950 + .018
1.000 0.000
.910 + .053
.655 .026
1. 1.000 + 0.000
0,25
.93 + .003
.97 = 016
016
1.000 0.000
.952 = .300
971 .012
1.000 + 0.000
0.30
.988 + .013
1.000 0.000
1.000 0.000
.988 = .018
.992 .008
1. 1.000 + 0.000
0.35
1.000 1 0.000
1,000 0.000
1.000 0.000
1.000 - 0.000
1.000
0.45
1.000
0.000
1.000 + 0.000
1.000
0.000
1.000 0.000
1.000 0.000
1.000 + 0.000
0,50
1.000 + 0.000
1.000 + 0.000
1,000 0.000
1. 1.000 0.000
1.000 0.000
1. 1.000 0.000 0.000
0.00
.001
1.000
.001
1.000 0.000
.002
.017 .007
1.000 + 0.000
.021 .009
.017 = .007
1.000 0.000
.019 + .008
0.05
.146 + .020
1.000 + 0.000
.113 +: .025
.143 .015
1.000 = 0.000
.122 0019
0.10
.261 + .015
1.000 + 0.000
.259 007
.271 = .019
1.000 = 0.000
.273 4 .031
CocCone
0.15
.422 .055
1.000 0.000
.398
.036
1.000 0.000
.395 1: .0113
.528 .024
1.000 1 1.0000000 0.000
.527
- .29
1.000 1:000000 0.000
4: .020
0.25
.715 + J025
1.000 + 0.000
.682 J026
688 + 024
1.000 0.000
.687 + .019
0.30
.155 + .034
1.000 + 0.000
821 .121
843 020
1.000 + 0.000
825 + .22
0.35
946 .014
1.000 + 0.000
.40 .U13
+ .013
1,000 0.000
.24 + .00
0.40
.982 2.10 .010
1.000 0.000
.984
1.000 0.000
.009
1.000 0.000
1.000 1 0.000
.997 .003
.997 = .002
1.000 1:000:000 0.000
.998 .001
0.50
1.000 -: 0.000
1.000 - 0.000
1.000 0.000
1.000 - 0.000
1.000 0.000
1 1.000 0.000
45

---

Table 27: Results for the completeness dataset when not allowing for shared parameters with
independent training using CE, and considering oracle task expert and oracle concept expert. L.S
refers to the label-smoothing frree implementation, while NL.S to the one with label smoothing. We report arg + std and highlight the best baseline in bold.
Metric
A
DCBM-LS
DCBM-NC-LS
DCBM-XT-LS
DCBM-NLS
DCBM-HC-NLS
OCBM-NI-TN NLS
0.00
.988 + .008
.933 + .020
.893 .033
.900 .005
.943 + .014
.911 .007
0.01
.987 +: .006
.26 = .019
.909
.004
.934
.908
0.05
989 + .007
.917 + .009
.999 + .024
.911 + .008
.21 + .014
.894 + .027
0.10
.975 + .008
= .015
.905 020
.006
.877 .028
0.15
.976 + .077
.918 + .018
.905 .018
.979 .007
.891 .015
.96
AocTask
0.20
.988 + 028
889 = .020
.907 + 008
.988 + .02
.905 = .018
.909 + .009
0.25
.600 + .110
= .007
.886 -
.911 .007
.905 .008
0.30
.955 + .012
.933 + .014
.877 .018
.953 .014
.877 .020
.884 .014
0.35
952 +: .009
875 = .008
.76 + .20
941 .005
.86 = .018
.886 1 .015
0.40
.017
+
+022
.879
0.45
.977 +: .009
867 = .012
.83 + .200
.96 .02
.87 = .016
.873 1 .017
0.50
.900 + .009
868 + .006
.865 + .010
.904 _010
.864 + .021
.661 .015
0.00
.992 + .002
845 = .008
.911 + .004
.91 + .002
842 = .008
.90 = .002
0.01
.990 + .002
843 + .010
.988 .003
.899 + .003
844 + .005
.90 .002
0.05
.97 + .003
845 .007
.957 .002
.985 + .005
844 .003
0.10
.900 + .007
843 + .010
.980 + .006
.980 + -007
846 + -006
986 + .004
AncCone
0.15
.975 + .005
847 = .008
.977 .007
.971
.972 .003
0.20
.966 + .003
844 + .006
.968 .004
.970 .006
.48 + .007
.96 .002
0.25
.959 + .007
845 = .003
.999 + .004
.961 + .005
.845 = .007
.977 .007
0.30
.960 .009
.008
.955
0.35
.949 + .006
845 0044
.966 + 0012
.533 + .005
844 .012
.944 .008
0.40
.937 + -003
842 .016
.937 .003
.941 + .010
.847 + .007
.942 .011
0.45
.929 + .008
.845 = .012
.928 .00
.30 .007
844 .012
.922 1 .007
0.50
.912 + .008
851 + .007
.910 .004
.077 + .007
841 + .008
.910 .006
0.00
.472 + .777
461 = .082
1.000 0.000
.110 .024
.059
1.000 0.000
0.01
.500 + .064
569 + .158
1. 1.000 + 0.000
.485 + .041
.499 + .092
1.000 + 0.000
0.05
.541 + .33
684
1.000 0.000
.551
1.000 0.000
0.10
.635 + .042
.72 + .027
1 1.000 0.000
.614 .041
.899 .053
1.000 0.000
0.15
.675 + .079
.738 = 0069
1. 1,000 0.000
.589 + .166
.791 = .021
1.000 0.000
ConTo
0.20
.059
.049
0.25
.793 + .037
8644 + .00
1 1.000 0.000
.803 .017
.845 .034
1.000 0.000
0.30
827 +: .027
884 = .031
1. 1,000 = 0.000
.009 + .064
.66 = .041
1.000 0.000
0.35
.014
.854
0.40
.878 + .033
884 .049
1.000 - 0.000
.867 + .026
898 = .031
1.000 1 0.000
0.45
.871 + .024
.905 + .027
1. 1.000 0.000
.888 + .076
.977 + .020
1.000 0.000
0.50
.9200 + .20
.22 = .009
1.000 0.000
.811 .022
.928 .018
1.000 0.000
0.00
.255 + .012
1.000 0.000
.254
.299
1.000 0.000
.276 .012
0.01
.274 .012
1.000 0.000
.253 000
.274 .007
1.000 0.000
.279 + .006
0.05
.309 + 0016
1.000 + 0.000
.311 311 .018
.366 +: .013
1.000 + 0.000
.310 .017
0.10
.375 +
1.000 0.000
.388
362
1.000 0.000
.371 .012
ConCone
0.15
.439 + .006
1.000 + 0.000
.416 .014
.438 1: .017
1.000 0.000
.429 1: .012
0.20
.699 + .009
1.000 = 0.000
477 .02
.92 .025
1.000 0.000
.491 .014
0.25
.544
.532
.524
0.30
.582 + .016
1.000 - 0.000
.592 .013
580 .014
1.000 = 0.000
.596 .013
0.35
.647 .017
1.000 0.000
648
1.000
0.000
.655 -110
0.40
+ .009
1.000 0.000
.766 .020
.688 .009
1.000 0.000
.697 .013
0.45
.764 + 014
1.000 + 0.000
.762 .015
.760 .077
1.000 + 0.000
.764 + .144
0.50
4: 013
1.000 0.000
.25 006
.32
1.000 0.000
.825 .005
46

---

Table 28: Results for the compl eteness dataset when not allowing for shared parameters with joint training using CE, and considering oracle task expert and oracle concept expert. We report aug + std and highlight the best baseline in bold.
Metric
DCBM-LS
DCBM-NC- NC-LS
DCBM-NT-L3
DCEM-NLS
DCBM-NC-NLS NC-NLS
DCBM-NT-NHE
0.00
.999 +
.022
.019
.800
0.01
985 + 0015
.955 = .013
.906 + 003
.983 + .015
.956 = .019
.899 = .025
0.05
.81 + .015
.999 + .019
.871 + .026
.800 .013
.933 + .014
.903 .008
0.10
+ .007
.921 = .011
.869 .014
977 .000
.915 .013
.899 +
0.15
.600 + .015
.902 = .013
.884 + .013
966 + .016
.911 .019
.99 = .011
0.20
.46 + .024
895 + .019
.889 + .015
.566 + .018
.896 + .007
.885 .026
0.25
+ .013
890 = .009
.888 .011
.017
.877 .014
.822 .227
0.30
.488 + -007
.81 .020
.883 + .021
.41 + .002
883 + .017
.887 .022
0.35
.43 -
= .055
.883 027
.999
.88 .010
0.40
.920 .010
.870 + .015
.80 .006
.955 .020
871 + .017
.874 .00T
0.45
.911 +: .016
867 = .021
.800 + .012
.923 + .014
.609 = .012
.870 = .018
0.50
.915
-014
.865
.10
.014
.867 .016
0.00
.90 + .002
840 = .009
.988 + .002
988 + .005
.838 = .006
.900 .003
0.01
.007
.005
.011
.987
0.05
.986 + .033
.43 = 0008
987 = .002
.887 = .005
844 .004
.82 .004
0.10
.977 + .004
841 + .007
.981 .004
.800 .004
.834 + .006
.978 .002
AccConc
0.15
+ .007
.811 = .007
.973
.972 + .006
.009
.922 004
0.20
969 + .005
841 + .009
.967 + .004
.966 + .005
844 + .007
.944 + .005
0.25
.911 +
= .006
.60
.958 .003
0.30
.933 + .007
.844 + .006
.953 .005
.956 + .002
.838 .008
.55 .010
0.35
.966 + 0004
832 = .008
.948 + .004
.44 .006
.838 = .006
.946 1 .005
0.40
+ .008
.011
0.45
.977 + .006
.877 + .006
.926 .006
.980 + .005
.843 .008
.926 .001
0.50
.909 + .006
836 = .012
.100 .010
.900 + .007
.338 = .009
.910 .010
0.00
.397 + 144
487 = .18
1.000 + 0.000
.933 + .075
.503 = .16
1.000 + 0.000
0.01
.466 + .109
.424 + .131
1. 1.000 0.000
.119 + .093
.488 + .110
1.000 0.000
0.05
.505 + .788
= 119
1.000 0.000
.552
.580 = .093
1.000 0.000
0.10
.704 + .046
.19 + .50
1 1.000 + 0.000
.616 003
.683 + .064
1.000 + 0.000
CouTask
0.15
.732 + .045
.759 = 0053
1.000 0.000
.736
1.000 0.000
0.20
.790 + .040
.33 = 002
1.000 0.000
.804 .011
.34 = .047
1.000 0.000
0.25
.32 + -015
853 + .008
1. 1.000 + 0.000
.785 + .054
844 + .038
1.000 + 0.000
0.30
+ .034
029
1.000 0.000
.826
1.000 0.000
0.35
.838 + .021
864 + .032
1 1.000 0.000
.866 + .039
.977 + .019
1.000 0.000
0.40
.877 + .026
.917 = .035
1.000 0.000
.854 .24
.905 = .023
1.000 0.000
0.45
+ .016
.033
1.000
.876
0.50
.913 + .059
.877 = .007
1.000 + 0.000
.896 + .030
.900 + .022
1.000 1 0.000
0.00
.015
.267
.245
0.01
.266 + .000
1.000 = 0.000
.244 4 .019
.251 .010
1.000 - 0.000
.200 .009
0.05
.301 + .019
1.000 1 0.000
.10 .024
.01 +: .019
1. 1.000 0.000
.309 .012
0.10
.363 + .014
1.000 0.000
.300 .019
.357 .014
1.000 0.000
.359 .011
ConCone
0.15
.396 + .018
1.000 + 0.000
.409 +: .023
.11 +: .028
1.000 + 0.000
.122 .015
0.20
466 + .010
1.000 0.000
.472
.622
0.000
.464 .021
0.25
.16 + .027
1.000 + 0.000
.507 .021
.528 .015
1.000 0.000
.522 .024
0.30
.67 + .20
1.000 = 0.000
.566 .012
563 + .034
1.000 = 0.000
.574 .015
0.35
.19
.63
.618
0.40
.600 + .22
1.000 - 0.000
.679 0011
.666 .25
1.000 - 0.000
.674 012
0.45
.745 + 0166
1.000 0.000
.732 + 006
.742 0008
1.000 + 0.000
.741 .013
0.50
+ 0007
1.000
.795
0.000
.806 .07
47

---

Table 29: Results for the completeness dataset when not allowing shared parameters with independent training using OVA, and considering oracle task expert and oracle concept expert. LS refers to the label-smoothing- free implementation, while NLS to the one with label smoothing. We report avg + std and highlight the best baseline in bold.
Metric
DCBM-I-S
DCBM-NC-IS
DCBK-NC-
DCBM-XT
1.000 0.000
+ 0077
.906
1.000 = 0.000
.999 .002
+ .010
0.01
1.000 0.000
.977 = 013
.914 1 .099
1.000 = 0.000
.999 = .002
.918 + .021
0.05
.998 + .011
.965 = 021
.914 = .000
.999 = .002
.909 .017
.901 + .024
0.10
.73 .019
+ .007
.14
.987 .004
.10 026
+: .029
.61
.X2 + 0155
.907
.699 + .188
880 .017
009
AnTTush
0.20
931 .066
.81 + 0111
893 0009
J26 = 023
883 .100
.96 + 000
0,25
.088 + .009
866 0100
.855 002
915 = .012
867 .017
887 + .010
0.30
.922 + .018
.863 .016
.875 + .044
.89 = .010
860 =000
.677 + .01
0.35
885
819 014
.622
.008
.555
+ .018
.800 .017
816 018
.888 .055
874 012
816 .16
+ .013
0.45
859 + 013
.36 014
.44 .010
817 = .009
840 .014
841 .016
0,50
.800 + .014
.82 + 0008
.844 .016
813 + .014
832 .006
824 + .010
0.000
+ 0088
0.01
1.000 + 0.000
843 .110
1.000 0.000
1.000 + 0.000
843 005
1.000 + 0.000
0.05
.998 + .001
$45+.007
.999 .001
.98 + 0.000
844 = .003
.977 + .001
0.10
.933 = .001
.43 .010
.994 + 0.000
.993 = .002
8446 0006
.994 + .002
AvcConn
847
.984
847
+ .003
.60
844 .00
.999 .004
= .05
848 .007
+ .044
0,25
950 + .004
$43 .003
.99 .066
919 = 003
815 = .007
954 + .009
0.30
.37 = 009
819 .088
.931 006
.35 = .005
841 = .007
.931 + .00
0.35
.933 + .003
.845 .004
.914 + .005
.977 = .005
884 + .012
911 + .005
.415 012
.006 .007
= .07
844 .012
+ .005
0,50
.837 + .005
.811 007
.37 .007
.36 = .044
8111 =1 .008
.339 + 000
0.00
0.000 1.00
0058
1.000 0.000
0.000 0.000
.009
0.000
.077 .070
.110 002
1.000 0.000
.071 = .81
.19 044
1
0.05
.355 + 241
449 .71
1.000 0.000
300 = 040
407 127
1 1.000 = 0.000
0.10
.688 + .011
.753 072
1,000 0.000
.608 = .076
.773 .77
1. 1.000 + 0.000
ConTask
0.15
.728 + .012
.819 0036
1,000 0.000
.680 = .137
872
1. 1.000-2 + 0.000
+
0.25
.977 .024
.910 .022
1.000 0.000
.966 + .099
.99 003
1. .000 + 0.000
0,30
.10 + .012
924 021
1.000 0.000
.918 .012
.19 .019
1.000 - 0.000
0.35
.930 = .029
.961 + .009
1.000 0.000
.929 = .012
.977 .020
1. 1.000 1 0.000
0.40
.957
.900
1,000 0.000
1.000 0.000
.970
.UT0
.799
0.50
.997 004
997 + JU7
1.000 0.000
996 + 0077
M97 004
1 1.000 + 0.000
0.00
.002 .002
1.000 +: 0.000
.004
+ .006
1.000 0.000
.005 + 000
.029
1.000 0.000
.117 .016
1.000 + 0.000
.129 .043
.138 + .200
1.000 0.000
.144 .017 .017
0.10
.244 :: .013
1.000 0.000
.257 4 .026
.50 - .005
1.000 0.000
.253 + .034
ConCone
0.15
.347 + .035
1.000 - 0.000
.357 036
384 = .06
1.000
.363 4: .013
0.20
.470 .025
1.000 0.000
.111
.031
1.000 0.000
1: .034
A023
.605
.030
0.30
.677 .035
1.000 + 0.000
.711 J016
.693 + .015
1.000
0.000
.695 .22
0.35
.789 + .017
1.000 + 0.000
.778 .U15
.782 0008
1.000 0.000
.700 0009 009
0.40
.862 + .006
1.000 + 0.000
.64 # .009
862 = .007
1.000=0.000
853 4 .00
0.45
.930 + .009
1.000 0.000
.932
- .005
0.000
4 007
.986 + .004
1.000 - 0.000
.90 0002
- .001
1.000
.989 4 .002
48

---

Table 30: Results for the completenent dataset when not allowing for shared parameters with joint training using OVA, and considering oracle task expert and oracle concept expert. We report avg + std and highlight the best baseline bold.
Metric
DCBM-LS
DCBM-NC-LS
DCEN-XT-BE XT-
DCBM-NLS
DCBN-NC-XIA.
DCBN-XT- XT-NXL
0.00
1.000 = 0.000
.997 = .001
.907 .028
1.000 = 0.000
.998 = .004
.927 + .22
0.01
1.000 + 0.000
1.000 = 0.000
.933 .008
1.000 = 0.000
1.000 0.000
.913 : .024
0.05
.998 :004
.84 .110
.903
.044
.968 021
+: .006
+ 013
.907
+ 029
.155 2
.009
0.15
958 .015
.79 + .017
899 .016
.956 = 008
877 0017
.10 + 009
AneFasi
0.20
.27 + .018
.879 0018
.893 .022
942 = .008
883 = .015
.877 + .020
0.25
.96 + .016
.870 .011
.893 .015
.988 = .012
861 .01
.894 + .016
.880
=
849
.16 010
469 019
.576
= .199
813 002
: .014
0.40
.88 + .014
847 017
874 = .017
895 = 018
818 0012
.73 + 0088
0.45
.871 + .016
.32 0088
.881 007
883 = .009
.811 000
850 + .001
0.50
.861 + .016
.825 + .009
.851 .017
865 .016
.829 + .008
.862 + .015
0.00
1.000
0.000
836 +: 00S
1.000 0.000
1.000 + 0.000
832 006
1 1.000 + 0.000
0.01
1.000 0.000
835 + 008
1,000 0.000
1.000 + 0.000
840 =4 013
1 1.000 + 0.000
0.05
.998 = .001
.40 .008
.998 .001
1.000 = 0.000
.839 = 003
.996 + .001
0004
=
.80
+ .001
AncCome
982 .006
008
.985 .004
.984 = .003
+ .005
0.20
964 + .003
.37 = 0088
.967 + .003
.944 = 000
839 = 008
.69 + .005
0.25
947 = 003
836 0005
.949 = 008
.950 = ,008
840 006
.952 + .002
0.30
.36 + .007
.833 + .006
.938 .006
.97 = .005
.84 334+.000 000
.937 + .004
.914
0.40
.891 .005
.82 + .112
.996 .005
.901 + .008
831 .006
.900 .007
0.45
.872 + .006
.835 = .003
.873 + .003
875 = .008
837 .011
.875 + 000
0.50
855 = 004
.830 + .008
.853 .005
855 = .008
.834 .004
.819 :: .008
0.000 0.000
.014 .013
1.000 0.000
= .046
.010 .022
1.000 + 0.000
0.01
0.000 = 0.000
.011 + .002
1.000 0.000
.019 .442
0.000 0.000
1 1.000 + 0.000
0.05
.377 = .076
.233 + 095
1,000 0.000
.249 = 1522
.30 = 166
1. 1.000 + 0.000
0.10
.678
657 - 061
1,000 0.000
.683 = .669
.716
0.000
ConTask
.783
+
.833 .018
.885 + .03
1.000 0.000
.544 .055
876 .026
1 .000 + 0.000
0,25
.866 - .029
.199 025
1.000 0.000
877 = .29
912 .02
1. 1.000 + 0.000
0.30
.844 + .019
.921 .022
1.000 0.000
.901 = .044
.915 .016
1. 1.000 + 0.000
0.35
.924
.21 .015
1,000 0.000
.969
1.000 0.000
.016
+
+ .022
960
0.45
.999 + .014
.75 + .U19
1.000 0.000
.988 + .016
U89 .088
1 1.000 + 0.000
0,50
1.000 + 0.000
.955 + 009
1,000 0.000
998 + .004
996 000
1 1.000 0.000
0.00
.007 .004
1.000
.004
1.000
0.000
+ .002
.022 .003
1.000 + 0.000
.027 .010
.018 = .008
1.000
0.000
.023 + .007
0.05
.131 + .033
1.000 +: 0.000
.134 .026
.108 = .037
1.000 : 0.000
.137 0018
0.10
.230 + .021
1.000 + 0.000
.229 007
.36 - .036
1.000 = 0.000
.244 :: .018
CocCone
0.15
.411
1.000 0.000
.338
1.000 0.000
.377 1: .018
.480 .011
1.000 1 0.000
.69
453 - .057
1.000 0.000
.457 4: .021
0.25
.589
0039
1.000 + 0.000
.587 J126
568 + .22
1.000 0.000
.575 + .020
0.30
.600 + .028
1.000 + 0.000
.661 J126
.648 + 0033
1.000 + 0.000
680 + 002
0.35
.741 .015
1.000 0.000
.761 .014
.745 + .048
1,000 0.000
.764 + .013
0.40
..19
1.000 0.000
.299
1.000 0.000
1: .025
.882 .014
1.000 0.000
.96 .013
.882 = .006
1.000 0.000
.833 4 .015
0.50
.932 4: 007
1.000 -: 0.000
.911 .012
.931 + 017
1.000 - 0.000
.933 + 011
49

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
We present Deferring Concept Bottleneck Models, as method to bridge Learning Defer (L2D) and Concept Bottleneck Models (CBMs). We provide theoretical results and we empirically show the benefits of our approach in addressing open issues in CBMs and L2D.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section 6 we discuss how treating concepts to be independent without considering interactions among them might not be realistic. While this limitation is shared with standard CBMs, it also affects DCBMs.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
50

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Yes, all the proofs are reported in Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All experimental details are reported in Appendix B and documented in the code attached to the submission.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
51

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
The code is available attttpss//ggcuub..c////aaapppppp
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
public/guides
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/eso/euuusd//eeeuuubbiiiii 7/nips cc/ wtc/publiic/ gwwice/gg/e///eeeeeeeeooiiieiees Codeeobmiiiieeee SanSSuubissss iobbbmisssiioi ssisssoicccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Yes, the experimental details are reported in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report the number of repetitions and plot standard deviation for our methods.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
52

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report such details in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We test our methods on synthetic data or widely known datasets.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Section 6 we discuss how having interpretable models might help audit deferring systems.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
53

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release novel datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite all the authors of used assets.
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
54

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification:
We do release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We did not perform crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human participants involved in our study.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
55

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [No]
Justification: No, the introduced method does not interact with LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy for what should or should not be described.
56


Reviews:
[{"review_id": "DSKmUdH2uy", "reviewer": "Reviewer_YD1r", "summary": "CBM are a family of models that are interpretable by design by conditioning on human-understandable concepts. This paper introduces: Deferring CBM, a type of CBM that allows deferring of uncertain concepts to humans.  Inspired by Learning to Defer, DCBM is trained for the cases when human intervention seems necessary to the model.  The authors perform experiments on CUB and CIFAR-h to show the effectiveness of deffering to humans.", "questions": "- I didnt understand why task accuracy reduces when there is more defering? L222\n\n- In Fig 5, it seems that some of the concepts are not easy to grasp, and realistically if a human cannot identify the concepts, defering is futile. Whats the genuine application of DCBM?\n\n- Why do you defer tasks as well?", "limitations": "Limitations are not mentioned. \n\n- Human study to validate", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strength\n\n- This is a great work that attempts to add human-in-loop setting for CBMs.\n\n- The paper is generally well-organized.\n\n- The authors theoretically motivate and bridge the gap between L2D and CBMs.\n\n# Weakness\n\n\n- I am struggling to understand the reason to defer to humans for the task prediction. The original and many of the following works of CBMs were motivated by the fact that concepts might be easier to intervene on than the task prediction. Hence, your setting to defer concepts is well motivated, but I dont understand, what is the motivation of intervene on the task? This is one of my main concerns. \n\n- There are already many works that suggest that concepts aren't independent (and there is leakage between them), and they are also cited in the paper. I assume concept independence is, hence a strong assumption. \n\n- I suppose deferring is another way of intervention on concepts. It would be natural to ask for the same about of interventions, how does the performance fair?", "was_revised": true, "final_justification": "I would like to thank the authors for their detailed answers. I will update my score to 5 since the authors have answered some of my concerns. My concern regarding interventions was to compare against other intervention strategies, which I believe the authors have already performed for another reviewer. I appreciate them.", "rebuttal": "We thank the reviewer for their useful feedback. We are glad to see that the reviewer states ours to be “*a great work that attempts to add human-in-loop setting for CBMs*”, appreciates the overall organization of the paper, and praises the soundness of our theoretical results.\n\nFirst, we address the weaknesses highlighted by the reviewer:\n\n**Defer on Task.** Deferring on the final task is a useful strategy to mitigate risk in incomplete scenarios where concepts alone are insufficient to determine the task label. For instance, in the CIFAR-10h dataset, concepts do not allow distinguishing cats from deer, since they have the same concept-level representation. Clearly, standard CBMs with no leakage would fail in such classification task. Conversely, in these cases, incomplete concept combinations trigger the defer option of a DCBM on the task, highlighting instances that can **not** be classified by looking at concepts only. We show this experimentally on CIFAR-10h, where we can see that deferring on the final task improves the final accuracy (Figure 4). Instead, the accuracy of the ablated DCBM-NoTask (i.e., a DCBM with no possibility to defer on the final task) plateaus at around 90%. This is due to DCBM-NoTask randomly guessing between deer and cats, while correctly classifying other classes. Furthermore, we report the following additional table showing how, whenever defer is not too costly, a DCBM correctly defers only instances of cats and deer to a human. Coherently with our formulation, when the cost increases, the model instead prefers to take a guess instead of deferring. The table reports the coverage on all task labels for increasing cost $\\lambda$.\n\n|Cost $\\lambda$|airplane|automobile|bird|**cat**|**deer**|dog|frog|horse|ship|truck|\n|-|-|-|-|-|-|-|-|-|-|-|\n|0.0|$1.0\\pm 0.0$|$1.0\\pm 0.0$|$1.0\\pm 0.0$|$0.0\\pm 0.0$|$0.0\\pm 0.0$|$1.0\\pm 0.0$|$1.0\\pm 0.0$|$1.0\\pm 0.0$|$1.0\\pm 0.0$|$1.0\\pm 0.0$|\n|0.01|$1.0\\pm 0.0$|$1.0\\pm 0.0$|$1.0\\pm 0.0$|$0.0\\pm 0.0$|$0.0\\pm 0.0$|$1.0\\pm 0.0$|$1.0\\pm 0.0$|$1.0\\pm 0.0$|$1.0\\pm 0.0$|$1.0\\pm 0.0$|\n|0.05|$0.988\\pm 0.003$|$0.998\\pm 0.003$|$0.982\\pm 0.003$|$0.032\\pm 0.014$|$0.009\\pm 0.003$|$0.999\\pm 0.003$|$0.984\\pm 0.007$|$0.979\\pm 0.007$|$0.987\\pm 0.003$|$0.989\\pm 0.003$|\n|0.1|$0.978\\pm 0.01$|$0.998\\pm 0.003$|$0.982\\pm 0.006$|$0.047\\pm 0.009$|$0.015\\pm 0.003$|$0.981\\pm 0.011$|$0.978\\pm 0.005$|$0.977\\pm 0.007$|$0.977\\pm 0.018$|$0.984\\pm 0.008$|\n|0.2|$0.97\\pm 0.023$|$0.998\\pm 0.003$|$0.98\\pm 0.022$|$0.126\\pm 0.014$|$0.037\\pm 0.017$|$0.957\\pm 0.025$|$0.984\\pm 0.003$|$0.982\\pm 0.01$|$0.986\\pm 0.005$|$0.97\\pm 0.0$|\n|0.3|$0.983\\pm 0.003$|$0.996\\pm 0.003$|$0.983\\pm 0.012$|$0.181\\pm 0.013$|$0.064\\pm 0.003$|$0.961\\pm 0.007$|$0.989\\pm 0.01$|$0.981\\pm 0.005$|$0.99\\pm 0.0$|$0.98\\pm 0.009$|\n|0.4|$0.987\\pm 0.01$|$1.0\\pm 0.0$|$0.987\\pm 0.006$|$0.739\\pm 0.394$|$0.676\\pm 0.478$|$0.978\\pm 0.02$|$0.986\\pm 0.016$|$0.992\\pm 0.01$|$0.998\\pm 0.003$|$0.995\\pm 0.005$|\n|0.5|$0.998\\pm 0.003$|$1.0\\pm 0.0$|$0.995\\pm 0.0$|$0.975\\pm 0.011$|$0.979\\pm 0.018$|$0.997\\pm 0.005$|$1.0\\pm 0.0$|$1.0\\pm 0.0$|$1.0\\pm 0.0$|$0.997\\pm 0.003$|\n\n**Concept Independence.** We agree that assuming independence among concepts can be thought of as a strong assumption. Nevertheless, we stress that (i.) this is a fundamental and widely accepted assumption in the CBM literature [1,2,3]; (ii.) assuming to know the relationship between concepts might be an even stronger assumption than independence for openly available data; (iii) our characterization of DCBMs can be straightforwardly extended to group together sets of mutually exclusive binary concepts (thus not independent) into a single multi-class concept, on which one could easily apply the defer option. As we mentioned in our conclusive section, extending the L2D paradigm to non-independent concepts and acknowledging their interactions is an interesting and challenging research question left for future work.\n\n**Concept Interventions.** We thank the reviewer for this question, which we however might have not fully understood. The primary objective of our work is to equip CBMs with the capability to autonomously call for human intervention when appropriate. Therefore, the amount of required interventions is chosen by the model itself, depending on an application dependent hyperparameter that represents the cost of deferring. All results show how decreasing the cost of deferring, reduces the coverage of the machine learning model by increasing the number of defers and hence improves the performance of the overall DCBM.\n\nHaving already addressed the question on task deferrals, we answer the remaining questions:\n\n**Accuracy Trends.** We are sorry if Figures 3 and 4 were not clear enough. The figure reports the contrary of what puzzles the reviewer: since on the x-axis we have the cost of deferring (larger cost, means more coverage/less defer), we can see a positive effect of defer on both Task accuracy and Concept Accuracy: in all the Figures indeed we have the largest accuracy at $\\lambda=0$, i.e., when deferring is not costly, while increasing the defer cost (i.e., more coverage of the ML model) reduces the number of defers and the accuracy.\n\n**Concept Interpretability.** We fully agree that whenever the concepts are difficult to predict and a human is likely to make a mistake, performing an intervention is counterproductive. We argue that **this is precisely why our DCBMs can be useful in practice compared to existing CBMs**, which often assume the human expert is an oracle, limiting their applicability in real-world contexts. Conversely, by explicitly taking into account human performance, DCBMs avoid asking for a human intervention when the expected human performance is worse than that of the ML model. Moreover, letting the model call for human intervention autonomously improves the Human-AI collaboration by partially offloading human oversight to the machine.\n\n**Human Study.** We thank the reviewer for the suggestion of performing human studies. However, in this paper, we focused on methodological aspects to define a novel method to combine CBMs and the L2D literature. User studies are still rare in the L2D literature: to the best of our knowledge, only a couple of works [4,5] implement human validation for learning-to-defer. In particular, these works featuring user studies *only focus* on the user study itself due to the non-trivial effort in defining and executing them over L2D peculiarities. As they are definitely not immediate, they constitute a promising future work. We will discuss the feasibility of human studies when discussing limitations in the conclusive section of our work.\n\nWe thank the reviewer again for their constructive comments. We will incorporate these clarifications, additional experimental results, and not-previously-mentioned limitations in the final version of the paper. Please let us know if there are some remaining concerns the reviewer would like us to discuss.\n\n\n1. Tuomas P. Oikarinen, Subhro Das, Lam M. Nguyen, and Tsui-Wei Weng. Label-free concept bottleneck models. In ICLR. OpenReview.net, 2023.\n2. Danis Alukaev, Semen Kiselev, Ilya Pershin, Bulat Ibragimov, Vladimir Ivanov, Alexey Kornaev, and Ivan Titov. 2023. Cross-Modal Conceptualization in Bottleneck Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5241–5253, Singapore. Association for Computational Linguistics.\n3. Rémi Kazmierczak, Eloïse Berthier, Goran Frehse, Gianni Franchi: CLIP-QDA: An Explainable Concept Bottleneck Model. Trans. Mach. Learn. Res. 2024 (2024)\n4. Bondi, Elizabeth, Raphael Koster, Hannah Sheahan, Martin Chadwick, Yoram Bachrach, Taylan Cemgil, Ulrich Paquet, and Krishnamurthy Dvijotham. \"Role of human-AI interaction in selective prediction.\" In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 5, pp. 5286-5294. 2022.\n5. Hemmer, Patrick, Monika Westphal, Max Schemmer, Sebastian Vetter, Michael Vössing, and Gerhard Satzger. \"Human-AI collaboration: the effect of AI delegation on human task performance and task satisfaction.\" In Proceedings of the 28th International Conference on Intelligent User Interfaces, pp. 453-463. 2023."}, {"review_id": "IUse9bRdpi", "reviewer": "Reviewer_bcXH", "summary": "The paper proposes a framework for addressing concept bottlenecks in deferring systems. The core contribution is a method to improve interpretability and robustness in such systems by explicitly modeling deferral mechanisms. The authors formalize the problem using theoretical guarantees and validate their approach on synthetic and benchmark datasets. The work bridges technical rigor with practical considerations for deploying deferring systems in real-world scenarios.", "questions": "- In the paper, the expert error is assumed to follow a fixed distribution, but real-world expert errors may correlate with input data complexity (e.g., higher misjudgment rates for rare cases). How can we validate the robustness of DCBM under dynamic error scenarios? Additionally, if human experts exhibit malicious or strategic behavior, does DCBM’s deferral mechanism collapse?\n- The delay cost (λ) is treated as a global hyperparameter, but in practical scenarios, the delay costs for different concepts or tasks may vary significantly, for example, correcting \"tumor size\" in medical diagnosis is more urgent than revising \"blood pressure measurements\". Can we design a locally adaptive cost function that accounts for task-specific urgency or contextual priorities?", "limitations": "- The experiments only measure coverage but do not quantify decision latency. In real-world systems, human response delays could cause prediction chain blocking in DCBM. Perhaps an asynchronous deferral mechanism could be introduced: allowing the model to emit provisional predictions while awaiting human responses.\n- To increase coverage, DCBM must reduce deferrals, but deferred concepts are often critical explanatory factors with high model uncertainty. This leads to degraded explanation quality at high coverage rates.\n- The computational overhead of the joint training strategy remains unquantified.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- Strengths\n  - The paper is clear and easy to follow. And the writing is concise.\n  - First work to combine deferral with concept bottlenecks, enabling interpretable L2D. The coverage-interpretability-accuracy trade-off is a novel perspective.\n\n- Weakness\n  - The multi-variable loss assumes concept independence, violating real-world hierarchical/causal structures. Independent deferral decisions may propagate errors when parent-child concept relationships exist.", "was_revised": true, "final_justification": "The authors have satisfactorily addressed key concerns raised in the initial reviews, particularly regarding experimental clarity and theoretical justification.", "rebuttal": "We are glad the reviewer appreciated the novelty in being the “*first work to combine deferral with concept bottlenecks*” and introducing coverage into the interpretability-accuracy trade-off. We now address the weaknesses and the question posed by the reviewer.\n\n**Concept Independence.** We agree with the reviewer that assuming concept independence might be a strong assumption in some scenarios, which however we share with most of the literature on CBMs [1,2,3]. The reviewer is correct also in stating that errors might be propagated and would require particular attention, which is precisely the reason we discussed in our conclusive section this intuition as a future work. Still, assuming to know the relationship between concepts might be an even stronger assumption than independence for openly available data. Therefore, in this paper, we focus instead on a two-layer structure where concepts only influence the final task label and not each other. Finally, our characterization of DCBMs can be straightforwardly extended to group together sets of mutually exclusive binary concepts (thus not independent) into a single multi-class concept, on which one could easily apply the defer option. As we mentioned in our conclusive section, extending the L2D paradigm to non-independent concepts and acknowledging their interactions is an interesting and challenging research question left for future work.\n\n**Human-Error Distribution Shift.** We thank the reviewer for highlighting an interesting point on the robustness of DCBMs. To the best of our knowledge, the problem of learning to defer under distribution shifts has not yet been studied in the literature. A plausible solution to that would be to use continual learning strategies, i.e., adapting the training process to a stream of data with possible distribution shifts. Learning to defer systems, therefore, are thought to be used, in a real-world context, with humans that have similar expertise to the ones that provided the additional feedback for training. Concerning the strategic behavior, attacks would be possible from experts resulting to be more accurate than the ML model in the training set, but that strategically choose to misclassify instances at inference time. Detecting this behavior or other distribution shifts is out of the scope of the current work, but we thank the reviewer for the suggestion, which we will mention in the future work references of the paper. To the best of our knowledge, there are no works investigating this specific adversary model in the literature.\n\n**Adaptable Defer Cost.**  Our current framework covers theoretically this scenario. Our formulation separates the deferral loss for each concept, i.e., we have one separate deferral system for each concept and task variable. Therefore, it is possible to specify different costs for each concept depending on how important or difficult that given concept is expected to be. However, the cost must be fixed at training time.\n\n**Latency and Provisional Decisions.** We thank the reviewer for the interesting suggestion, which provides a straightforward extension in our formulation. In cases where a prediction needs to be given without waiting for human intervention, the machine’s response to concepts and tasks can always be evaluated to formulate an immediate prediction, albeit at a high risk of error. The model predictions are produced as the log-probability of each class, including the additional special class for deferral $\\bot$. If the model chose to defer to the human, the DCBM could select the class corresponding to the second-largest value and return that as a provisional response with a warning. This approach is solid as the model is trained jointly to increase the activations of the ground-truth labels and, when the human is correct, of the deferral label $\\bot$.\n\n**Coverage and Explanations.** Indeed, higher coverage rates are reached when defer cost is high and therefore the system is trained to rely less on human experts. This is an instance of the trade-off between coverage, interpretability, and accuracy that underlies our work. The choice of the cost of deferring a given concept or the final task is application dependent and should be a main design choice when deploying a DCBM. We will improve the discussion on this point by adding a subsection on the limitations of DCBMs in the conclusion of our paper, where we already hinted at the problem of having different costs.\n\n**Computational Overhead.** The cost of training a DCBM against a CBM is negligible, as they have essentially the same architecture, apart for an additional feature on each concept or task classifier. In detail, on our hardware, for a training epoch it takes $\\approx 9.0$ seconds on CUB for a DCBM and $\\approx 7.5$ for a standard CBM. For CIFAR-10h, given the smaller number of concepts, the difference is even more negligible, with both taking approximately one second per epoch. We report epoch time as, due to early stopping strategies, the overall training time might vary across runs. We will improve Appendix B, which already reports training architecture and hardware details, by reporting training times of DCBMs and CBMs.\n\n\nWe thank the reviewer again for their positive feedback. We will incorporate these clarifications, suggested limitations and additional discussion in the final version of the paper. Please let us know if there are some other concerns the reviewer would like us to discuss.\n\n\n\n1. Tuomas P. Oikarinen, Subhro Das, Lam M. Nguyen, and Tsui-Wei Weng. Label-free concept bottleneck models. In ICLR. OpenReview.net, 2023.\n2. Danis Alukaev, Semen Kiselev, Ilya Pershin, Bulat Ibragimov, Vladimir Ivanov, Alexey Kornaev, and Ivan Titov. 2023. Cross-Modal Conceptualization in Bottleneck Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5241–5253, Singapore. Association for Computational Linguistics.\n3. Rémi Kazmierczak, Eloïse Berthier, Goran Frehse, Gianni Franchi: CLIP-QDA: An Explainable Concept Bottleneck Model. Trans. Mach. Learn. Res. 2024 (2024)"}, {"review_id": "MbowFLvkgR", "reviewer": "Reviewer_7H8p", "summary": "This works introduces a Learning to Defer (L2D) framework into Concept Bottleneck Models, proposing Deferring CBMs that can learn when an intervention is needed through a composition of deferring systems. The authors show that intelligent deferring offers not only improved performance, but also paves the way for future work on explaining deferrals for better human-machine interaction.", "questions": "- The accuracy-interpretability claim seems unsubstantiated by the cited paper (CEM), which states that \"embedding-based CBMs (i.e., Hybrid-CBM and CEM) can achieve competitive or better downstream accuracy than DNNs that do not provide any form of concept-based explanations\". Could the authors explain what they mean by accuracy-interpretability trade-off in this case?\n- There seems to be a distinction that exists between the initial concept annotations being inaccurate, and an actual human-in-the-loop being inaccurate, but they seemed to be conflated in the text. For example, the difference between an Oracle and Uncertain human in Figure 3 is unclear, and the authors only state that \"we use them to produce random human concept labels as done by Collins et al. [2023].\" Could the authors elaborate this experimental setup further?\n- Similarly, do DCBMs utilize this concept uncertainty information directly? It seems the human predictions utilize this inherent uncertainty, but it is unclear how this extends to real-world tasks.\n- Could the authors compare to recent work on informed intervention strategies or explain how their work cannot be compared?", "limitations": "Yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The paper is clearly written.\n- Exploring human-in-the-loop interactions with interpretable systems is highly useful for real-world deployment of these models.\n\nWeaknesses\n- Lack of baselines. The paper presents a Learning to Defer approach, training a model to autonomously defer on its intermediate concepts. This seems like it could be easily compared to other intervention strategies [1], to show how well the model can improve task/concept performance across a consistent number of interventions. However, the authors claim that \"works on concept interventions fundamentally differ from our L2D-based approach in that they assume that experts themselves trigger a correction in a model’s concept predictions.\" It is unclear what the authors mean by this.\n- Overall the results do not seem to adequately support the main claims of the paper. Both the main claim that human experts may introduce errors that \"muddle the effects and dynamics of the human-AI collaboration expected when using CBMs\" (harm performance? harm something else?) or that their proposed DCBMs adequately handle this human uncertainty, especially compared to other intervention strategies.\n\nTypos\n- Line 14? \"DCBMs can explain why defer occurs on the final task\"\n- Line 198 \"syntethic\"\n\n[1] Shin, Sungbin, et al. \"A closer look at the intervention procedure of concept bottleneck models.\" International Conference on Machine Learning. PMLR, 2023.", "was_revised": true, "final_justification": "I appreciate the novelty of the work bringing the learning to defer framework to concept bottleneck models, and can understand that there exists a distinction between the fundamental goals of intervention and deferral. While I understand that other reviewers are satisfied with the experimental evaluation, I believe that a more robust comparison against intervention strategies would still be necessary to support the main claims of the paper.\n\nEven if intervention strategies are less informed than deferral strategies, they share the same goal of improving the aggregate human-machine team, and DCBMs require additional information (\"one human expert prediction per instance at training time\") so should ideally outperform these intervention strategies in real-world scenarios. The result comparing to UCP, as well as the detailed toy examples, provide a great step in this direction, and I would love to see more results in this vein for the camera-ready version of the paper.\n\nOverall, I will keep my score.", "rebuttal": "We thank the reviewer for their feedback. We appreciate the reviewer recognizes that \"*exploring human-in-the-loop interactions with interpretable systems is highly useful for real-world deployment of these models*\" and having appreciated the clarity of our writing.\n\nIn the following, we address the reviewer’s concerns on the weaknesses of our work:\n\n**Baselines.** We thank the reviewer for the opportunity to clarify this point. Intervention strategies, such as those proposed by Shin et al. (2023), allocate a number of admissible interventions and then choose for each instance, or for a batch of instances, on which concepts to intervene. Using learning to defer methodologies, we instead equip a CBM with the capability to (i.) *autonomously* ask for human intervention and (ii.) acknowledge the capabilities of the expert, i.e., we consider *fallible human beings* with variable performance. Intervention selection strategies instead only consider the uncertainty of the model, which however might disregard that a human would not be better than the ML predictor in classifying a particular instance. Practically, a DCBM autonomously calls for human intervention on the concepts on which there is the higher probability that the human would outperform the model itself. We report the following additional results where we compare DCBM-NoTask (i.e., without the option to defer on the final task) and the UCP strategy from Shin et al. (2023) on the CUB dataset with uncertain humans.\n\n| Cost | ConcCov | UCP | DCBM-NoTask |\n|-|-|-|-|\n| 0.0 | $0.185 \\pm 0.002$ | $0.665 \\pm 0.015$ | $0.800 \\pm 0.004$ |\n| 0.01 | $0.188 \\pm 0.002$ | $0.666 \\pm 0.015$ | $0.803 \\pm 0.000$ |\n| 0.02 | $0.271 \\pm 0.029$ | $0.685 \\pm 0.010$ | $0.796 \\pm 0.001$ |\n| 0.0225 | $0.424 \\pm 0.049$ | $0.715 \\pm 0.024$ | $0.782 \\pm 0.003$ |\n| 0.025 | $0.601 \\pm 0.033$ | $0.746 \\pm 0.012$ | $0.756 \\pm 0.002$ |\n| 0.0275 | $0.700 \\pm 0.057$ | $0.761 \\pm 0.015$ | $0.743 \\pm 0.007$ |\n| 0.03 | $0.808 \\pm 0.016$ | $0.771 \\pm 0.002$ | $0.738 \\pm 0.004$ |\n| 0.05 | $0.931 \\pm 0.001$ | $0.745 \\pm 0.003$ | $0.723 \\pm 0.004$ |\n| 0.1 | $0.963 \\pm 0.002$ | $0.719 \\pm 0.003$ | $0.704 \\pm 0.004$ |\n| 0.5 | $1.000 \\pm 0.000$ | $0.679 \\pm 0.002$ | $0.676 \\pm 0.003$ |\n\nThe results show that when we allow for a large number of interventions ($\\lambda \\leq .025$), UCP underperforms because it asks for interventions based solely on model uncertainty, without accounting for whether the human is likely to be more accurate. As a result, it defers to the human on instances where the ML model predictions would have been a better option. On the other hand, when the budget of interventions is limited, DCBMs have a more conservative approach and tend to prefer the use of the ML model, leading to slightly worse outcomes. We thank the reviewer for the opportunity to add this additional comparison, which we will discuss in our paper.\n\n**Claims on Human Uncertainty.** In our experiments with the CUB dataset with uncertain labels, we evaluate our claims that (i.) humans might introduce errors when intervening and that (ii.) DCBMs correctly handle this scenario. Concepts annotated by humans do not necessarily correspond to the ground truth, as we discussed in the previous question and whose construction we report in the following “Concept Annotations” paragraph of the rebuttal. As shown in Figure 3b, we can see that even when the cost of deferring is equal to zero, the model still directly classifies 19% of the concepts at inference time, i.e., $CovConc\\approx0.185$. This is because the DCBM is able to detect instances where the human will provide an inaccurate answer. Compared to intervention strategies, cost zero of intervening would correspond to allocating an intervention budget on all concepts. Without knowing the human distribution, this strategy would intervene also on concepts where it would have been better to use the ML classifier, as it is also visible in the additional results provided in the previous answer.\n\nHaving already tackled the question on intervention strategies when discussing the reported weaknesses of our work, we now address the remaining questions of the reviewer.\n\n**Accuracy-interpretability trade-off.** The CEM paper shows that there is an inherent trade-off between the interpretability of concepts and the accuracy of the task: if one is willing to sacrifice part of the interpretability of concepts (e.g., by resorting to concept embeddings), the performance of the concept-based model gets closer to the one of a black-box model. In other words, using embeddings is generally less interpretable than a plain-vanilla concept-bottleneck model, but still more interpretable than a black box model.\n\n**Concept Annotations.** The learning-to-defer paradigm assumes to model the predictive distribution of a human expert. Therefore, the annotations made by the human expert reported in the dataset and the decisions made at inference time should *approximately* follow the same distribution. Intuitively, they should be inaccurate in a similar way. Concerning the procedure to account for human uncertainty in the CUB concepts (lines 195‒197), we adopt the following strategy. Let $c$ be the ground-truth label of a concept for a given sample and $u$ be the corresponding label of uncertainty, as provided by Koh et al. (2020). Uncertainty labels have the following semantics: not visible ($u=1$), guessing ($u=2$), probably ($u=3$), and definitely ($u=4$). Koh et al. (2020) translate the uncertainty labels in the following probabilities, which we use to sample the value $\\hat{c}$ of the concept provided by a human practitioner. We will improve the discussion of this procedure, mentioned in lines 195‒197, by extending our section on the experimental details in Appendix B.\n\n| $c$ | $u$ | $p_H(\\hat{c} \\mid c, u)$ | \n|-|-|-|\n| 1 | 1 | 0.00 |\n| 1 | 2 | 0.50 |\n| 1 | 3 | 0.75 |\n| 1 | 4 | 1.00 |\n| 0 | 1 | 0.00 |\n| 0 | 2 | 0.50 |\n| 0 | 3 | 0.25 |\n| 0 | 4 | 0.00 |\n\n**Concept Uncertainty.** No, a DCBM does not use concept uncertainty as an estimate to whether it should ask for intervention or not. Instead, it models human uncertainty by training over a dataset containing both ground-truth values and human predictions, as done in standard learning-to-defer. In this way, a DCBM models whether a human will be able to provide a correct answer or not. Depending on the defer cost, it then determines whether to defer the answer to the human or directly answer the classification problem.\n\n\nWe thank the reviewer again for their constructive feedback. We will incorporate these clarifications, additional experimental results, and revised explanations in the final version of the paper. Please let us know if there are some other concerns the reviewer would like us to discuss."}, {"review_id": "3stLZV6Wxl", "reviewer": "Reviewer_Nfpd", "summary": "The paper introduces deferring concept bottleneck models (DCBMs). This method combines the idea of learning to defer with the intervention procedure of concept bottleneck models. In that, DCBMs improve upon default interventions in two ways: Estimating when interventions are required and estimating whether deferring to the human would actually be helpful. To achieve this, DCBM introduces a new loss to train a CBM with deferring in mind.", "questions": "Clarifications for the experimental evaluations:\n- Setup:\n\t- How are the uncertain concept annotations for CUB constructed? Specifically, what is the randomness in the human concept labels?\n\t- What is the task accuracy of the human for cifar10-h?\n\t- Are the black-box baselines the same as the models used to compute the concept embeddings?\n- RQ2:\n\t- In my understanding, deferring of concepts cannot help to mitigate concept completeness. Thus, the only possible deferring to handle concept completeness is deferring on the task. However, when the task is deferred to the human, than the concept space does not matter (i.e. also whether it is complete or not). Maybe I am missing the point here, but I do not really understand why the concept of completeness is particularly interesting in this context. I would appreciate if the authors could elaborate more on why it is not expected that task deferring should be independent of the concept space.\n\nGeneral questions for the proposed method:\n- Deferral cost of concepts and task labels: In the conclusion, it is stated that the same costs are assumed for deferring at a concept or task level. I agree that different costs should probably be explored in the future, but I am also curious how this affects the current results: Is the deferring of concepts dependent on the number of concepts compared to the (singular) task label? Or is this irrelevant, even given the large differences in numbers of concepts between Cifar10 and CUB?\n- Coverage of the concepts is the percentage of concept predictions done by the model. However, particularly in CUB there is a large number of concepts the model predicts correctly (~95%). A concept coverage of 80% thus means that the model queries at least four correct concepts for one incorrect concept in the best case, is this correct? And given that, the model still performs quite badly (in Fig 3a) in these situations compared to previous strategies that investigate interventions (e.g. LCP in [1]), when comparing only interventions (or deferring) on the concepts. Can the authors discuss this case (and the relation to previous work on interventions) in more detail?\n- As DCBM allows for deferring of both the task label and the concept label, is there any accounting for the fact that deferring at the concept level could already be sufficient and make deferring on the task label for this sample unnecessary?\n- Is there an argument for not training the concept predictor fully based on the input when using DCBM training (as e.g. the original independent scheme from Koh et al.)?\n\nLimitations:\nThe limitations of this work are not sufficiently discussed in the paper. I recommend either the creation of a separate limitations subsection or having them clearer and structured as part of the conclusion.\nI see the following limitations/discussion points:\n- (As briefly mentioned in the paper): Deferring costs are the same for concepts and tasks. While task-level deferring might be in line with the previous learning to defer setup, the main focus of deferring on CBMs should be concept-level, as task-level deferring is assumed to be substantially more expensive. A different approach would be showing that the predicted concepts make task-level deferring easier for humans than doing the same for a black-box model and thus more applicable, but this would require more experiments.\n- The approach only works reliably as long as the expert performs the same between training and testing. This also implies that the model cannot necessarily adapt to a different human with other strengths. While I understand the argumentation why and where learning to defer improves upon interventions, the current method loses many of its benefits when this assumption is not given anymore.\n\n\nMinor points:\n- Eq. 4: Large braces would help to clarify what the sum covers, and the final ',' seems unnecessary\n- Clarity of figures and captions:\n\t- As Figs. 3 and 4 are quite information dense; it might be helpful to mention the key messages in the caption to improve reader understanding. Additionally, the lines overlap a lot, which makes it quite difficult to follow the results sometimes. \n\t- The caption of Fig. 5 and the in-figure descriptions could be improved to make the figures easier to understand: In particular, the in-figure header of b could better reflect the content of the plot (i.e., the probability for deferral.) Additionally, I recommend not using red and green to improve readability for color-blind people.\n\n\n[1] Shin, Sungbin, et al. \"A closer look at the intervention procedure of concept bottleneck models.\" _International Conference on Machine Learning_. PMLR, 2023.", "limitations": "The limitations are not sufficiently discussed (see above).", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper introduces a novel method to incorporate deferring to humans into CBM training, tackling an important problem of practicability of interventions.\n- The introduced loss is formally derived and justified\n\nWeaknesses:\n- The implications of some of the results is not clear and missing some discussions.", "was_revised": true, "final_justification": "During the rebuttal, my main concerns with the paper have been addressed. In particular, the authors have provided more details about their experimental evaluations and the discussion about concept completeness.\nThey have also performed an additional experiment which better shows the strengths of the method when compared to other semi-automated intervention approaches, i.e. the explicit modeling of human uncertainty.\nIn that, I think that the paper addresses one important problem of the current intervention approaches on CBMs and has a good value to the community.\nThus, I recommend accepting the paper.", "rebuttal": "We thank the reviewer for their detailed comments and the insightful feedback on the novelty of our method and the formality in deriving and justifying our loss function for learning to defer on Concept Bottleneck Models. In the following, we hope to address and clarify the questions raised by the reviewer.\n\n**CUB Concept Annotations.** The CUB dataset reports a measure of label uncertainty (Koh et al., 2020), which we use to simulate possibly-faulty human experts. In detail, let $c$ be the label of a concept for a given sample and $u$ be the corresponding label of uncertainty. The uncertainty labels have the following semantics: not visible ($u=1$), guessing ($u=2$), probably ($u=3$), and definitely ($u=4$). Koh et al. (2020) translate the uncertainty labels in the following probabilities, which we use to sample the value $\\hat{c}$ of the concept provided by a human expert. We will improve the discussion of this procedure, mentioned in lines 195‒197, by extending our section on the experimental details in Appendix B.\n\n| $c$ | $u$ | $p_H(\\hat{c}\\mid c, u)$ | \n|-|-|-|\n| 1 | 1 | 0.00 |\n| 1 | 2 | 0.50 |\n| 1 | 3 | 0.75 |\n| 1 | 4 | 1.00 |\n| 0 | 1 | 0.00 |\n| 0 | 2 | 0.50 |\n| 0 | 3 | 0.25 |\n| 0 | 4 | 0.00 |\n\n**CIFAR-10h**. The performance of the human expert labels against the ground truth in CIFAR-10h is $\\approx 0.95$.\n\n**Concept Embeddings.** Yes, the black-box baselines and the CBM models, including our DCBM, adopt the same architecture and the same common frozen representation. Then, the task and concept classifiers are trained independently for the black box, the standard CBM, our DCBM, and its ablations (DCBM-NoTask and DCBM-NoConcepts). We will remark this information in Appendix B, where we already discuss the architectures and the training procedures for the CUB and CIFAR-10h datasets.\n\n**Concept Completeness.** In a concept-incomplete scenario, when concepts are not sufficient to distinguish between two or more classes, we cannot train a task classifier over those with the same concept-level representation — such as “cat” and “deer” in the CIFAR-10h dataset. The right choice would then be to defer to a human, which can distinguish between the two classes by also employing input data or additional information. By implementing this idea, DCBMs provide a mechanism to deploy a model in an incomplete setting, addressing the risks that might arise from deploying a classifier that arbitrarily chooses one of the two classes. We also report additional results on the coverage of the machine learning model on all the task labels of the CIFAR-10h dataset. The results show how, whenever defer is not too costly, a DCBM only defers instances of cats and deer to a human. Coherently with our formulation, when deferring becomes too costly, the model instead prefers to take a guess instead of deferring, hence increasing the coverage.\n\n|defer_cost|airplane|automobile|bird|cat|deer|dog|frog|horse|ship|truck|\n|-|-|-|-|-|-|-|-|-|-|-|\n|0.0|$1.0\\pm 0.0$|$1.0\\pm 0.0$|$1.0\\pm 0.0$|$0.0\\pm 0.0$|$0.0\\pm 0.0$|$1.0\\pm 0.0$|$1.0\\pm 0.0$|$1.0\\pm 0.0$|$1.0\\pm 0.0$|$1.0\\pm 0.0$|\n|0.01|$1.0\\pm 0.0$|$1.0\\pm 0.0$|$1.0\\pm 0.0$|$0.0\\pm 0.0$|$0.0\\pm 0.0$|$1.0\\pm 0.0$|$1.0\\pm 0.0$|$1.0\\pm 0.0$|$1.0\\pm 0.0$|$1.0\\pm 0.0$|\n|0.05|$0.988\\pm 0.003$|$0.998\\pm 0.003$|$0.982\\pm 0.003$|$0.032\\pm 0.014$|$0.009\\pm 0.003$|$0.999\\pm 0.003$|$0.984\\pm 0.007$|$0.979\\pm 0.007$|$0.987\\pm 0.003$|$0.989\\pm 0.003$|\n|0.1|$0.978\\pm 0.01$|$0.998\\pm 0.003$|$0.982\\pm 0.006$|$0.047\\pm 0.009$|$0.015\\pm 0.003$|$0.981\\pm 0.011$|$0.978\\pm 0.005$|$0.977\\pm 0.007$|$0.977\\pm 0.018$|$0.984\\pm 0.008$|\n|0.2|$0.97\\pm 0.023$|$0.998\\pm 0.003$|$0.98\\pm 0.022$|$0.126\\pm 0.014$|$0.037\\pm 0.017$|$0.957\\pm 0.025$|$0.984\\pm 0.003$|$0.982\\pm 0.01$|$0.986\\pm 0.005$|$0.97\\pm 0.0$|\n|0.3|$0.983\\pm 0.003$|$0.996\\pm 0.003$|$0.983\\pm 0.012$|$0.181\\pm 0.013$|$0.064\\pm 0.003$|$0.961\\pm 0.007$|$0.989\\pm 0.01$|$0.981\\pm 0.005$|$0.99\\pm 0.0$|$0.98\\pm 0.009$|\n|0.4|$0.987\\pm 0.01$|$1.0\\pm 0.0$|$0.987\\pm 0.006$|$0.739\\pm 0.394$|$0.676\\pm 0.478$|$0.978\\pm 0.02$|$0.986\\pm 0.016$|$0.992\\pm 0.01$|$0.998\\pm 0.003$|$0.995\\pm 0.005$|\n|0.5|$0.998\\pm 0.003$|$1.0\\pm 0.0$|$0.995\\pm 0.0$|$0.975\\pm 0.011$|$0.979\\pm 0.018$|$0.997\\pm 0.005$|$1.0\\pm 0.0$|$1.0\\pm 0.0$|$1.0\\pm 0.0$|$0.997\\pm 0.003$|\n\n\n\n**Costs and Number of Deferrals.** The likelihood of a concept or task classifier to ask for a human is regulated at training time by the cost parameter $\\lambda$ and is independent of the probability of deferring of the remaining classifiers. Consequently, at parity of cost, we can expect the *cumulative* number of defers to be larger for larger sets of concepts.\n\n**Comparison with UCP on CUB.** A concept coverage of 80% means that the model asks for a human intervention one out of five times. When deferring only on the concepts (DCBM-NoTask), the model performs better than our black box and CBM baselines, until the cost gets too high to intervene and converges to the CBM performance. We report the following additional results, where we compare a DCBM-NoTask (i.e. a DCBM deferring on concepts only) and the UCP strategy from Shin et al. (2023) on the CUB dataset with uncertain humans. We compare with UCP instead of LCP since LCP uses information not available at inference time, and thus it constitutes only a theoretical upper bound as discussed in Shin et al. (2023).\n\n| Cost | ConcCov | UCP | DCBM |\n|-|-|-|-|\n| 0.0 | $0.185 \\pm 0.002$ | $0.665 \\pm 0.015$ | $0.800 \\pm 0.004$ |\n| 0.01 | $0.188 \\pm 0.002$ | $0.666 \\pm 0.015$ | $0.803 \\pm 0.000$ |\n| 0.02 | $0.271 \\pm 0.029$ | $0.685 \\pm 0.010$ | $0.796 \\pm 0.001$ |\n| 0.0225 | $0.424 \\pm 0.049$ | $0.715 \\pm 0.024$ | $0.782 \\pm 0.003$ |\n| 0.025 | $0.601 \\pm 0.033$ | $0.746 \\pm 0.012$ | $0.756 \\pm 0.002$ |\n| 0.0275 | $0.700 \\pm 0.057$ | $0.761 \\pm 0.015$ | $0.743 \\pm 0.007$ |\n| 0.03 | $0.808 \\pm 0.016$ | $0.771 \\pm 0.002$ | $0.738 \\pm 0.004$ |\n| 0.04 | $0.916 \\pm 0.005$ | $0.751 \\pm 0.001$ | $0.724 \\pm 0.002$ |\n| 0.05 | $0.931 \\pm 0.001$ | $0.745 \\pm 0.003$ | $0.723 \\pm 0.004$ |\n| 0.1 | $0.963 \\pm 0.002$ | $0.719 \\pm 0.003$ | $0.704 \\pm 0.004$ |\n| 0.2 | $0.990 \\pm 0.002$ | $0.690 \\pm 0.001$ | $0.685 \\pm 0.001$ |\n| 0.3 | $0.998 \\pm 0.001$ | $0.682 \\pm 0.002$ | $0.677 \\pm 0.002$ |\n| 0.5 | $1.000 \\pm 0.000$ | $0.679 \\pm 0.002$ | $0.676 \\pm 0.003$ |\n\nThe results show that when we allow for a large number of interventions ($\\lambda\\leq.025$),  UCP underperforms because it asks for interventions based solely on model uncertainty, without accounting for whether the human is likely to be more accurate. As a result, it defers to the human on instances where the ML model  predictions would have been a better option. On the other hand, when the budget of interventions is limited, DCBMs have a more conservative approach and tend to prefer the use of the ML model, leading to slightly worse outcomes. We thank the reviewer for the opportunity to add this additional comparison, which we will discuss in our paper.\n\n**Task Defer.** We thank the reviewer for the opportunity to clarify this point. Yes, deferring at the concepts level might be sufficient to avoid deferring at the task level. In particular, the classification of the task occurs after eventual defers are solved in concept-space. The task is then predicted once a value has been assigned to all concepts. This can be clearly seen once again on the results from the CIFAR10h, when cost is $\\lambda=0$ and concepts are perfectly predicted. In this case, as discussed previously, defer occurs only for deer and cat, the only two classes that the ML model cannot distinguish when looking at concepts. For all the other classes, coverage is one, meaning that correcting concepts suffices to avoid defer on the final task, even when it would not be costly to do so. We will add this discussion in the final version of the paper.\n\n**Independent Scheme.** We train all our DCBM models using the independent scheme, i.e., training the concept predictors from the ground-truth inputs and the task predictor on the ground-truth concepts, as illustrated in Koh et al. (2020). As the only difference, we pre-train an image encoder to avoid parameter sharing among concept predictors (Lines 173–181).\n\n**Limitations.** We thank the reviewer for the suggestion, and we will add a paragraph dedicated to the limitations of our method. In particular, we will remark that particular care should be taken in assigning deferral costs depending on the application and that the advantages of learning-to-defer hold only when the human distribution is similar enough between training and testing. On this last point, we would like to remark that, to the best of our knowledge, the problem of learning to defer under distribution shifts has not yet been studied in the literature. A plausible solution to that would be to use continual learning strategies, i.e., adapting the training process to a stream of data with possible distribution shifts. The integration of learning-to-defer and continual learning constitutes a promising direction that we will report as future work. We will also take into account the minor points on formatting and readability, for which we thank the reviewer.\n\n\nWe thank the reviewer again for their constructive and helpful suggestions. We will incorporate this discussion, additional experimental results, and not-previously-mentioned limitations in the final version of the paper. Please let us know if there are some remaining concerns the reviewer would like us to discuss."}]

Author rebuttal:

Meta-review: This paper proposes Deferring Concept Bottleneck Models (DCBMs), a novel framework that integrates Learning to Defer (L2D) into Concept Bottleneck Models (CBMs). Traditional CBMs rely on human interventions to correct intermediate concepts but assume humans are always correct and available—an unrealistic assumption. DCBMs address this by allowing the model to autonomously decide when to defer predictions (either at the concept or task level) to a human expert, especially when the expert is likely to be more accurate. This paper also provides theoretical guarantees, extensive experiments, and comparisons with baselines and ablated models, demonstrating that DCBMs achieve a better trade-off between accuracy, interpretability, and human reliance.

After the rebuttal process, this paper finally receives the scores of 5, 5, 5, 3, where all the reviewers acknowledge the novelty of this paper, while Reviewer 7H8p gives the score of 3. The main concern of  Reviewer 7H8p lies in the insufficient experimental validation:

>I appreciate the novelty of the work bringing the learning to defer framework to concept bottleneck models, and can understand that there exists a distinction between the fundamental goals of intervention and deferral. While I understand that other reviewers are satisfied with the experimental evaluation, I believe that a more robust comparison against intervention strategies would still be necessary to support the main claims of the paper.
Even if intervention strategies are less informed than deferral strategies, they share the same goal of improving the aggregate human-machine team, and DCBMs require additional information ("one human expert prediction per instance at training time") so should ideally outperform these intervention strategies in real-world scenarios. The result comparing to UCP, as well as the detailed toy examples, provide a great step in this direction, and I would love to see more results in this vein for the camera-ready version of the paper.
Overall, I will keep my (borderline) overall negative score.

Given that the merits of this paper significantly overwhelm shortcomings, I tend to accept this paper.
Decision: Accept (poster)