Concept Embedding Models: Beyond the Accuracy-Explainability Trade-Off
Authors: Mateo Espinosa Zarlenga
Pietro Barbiero
Gabriele Ciravegna
Giuseppe Marra
Francesco Giannini
Michelangelo Diligenti
Zohreh Shams
Frederic Precioso
Stefano Melacci
Adrian Weller
Pietro Lio
Mateja Jamnik
Venue: NeurIPS 2022 Accept (2022)
Keywords: Explainable Artificial Intelligence
Concept Bottleneck Models
Concept-based Explainability
Interpretability
XAI

Abstract: Deploying AI-powered systems requires trustworthy models supporting effective human interactions, going beyond raw prediction accuracy. Concept bottleneck models promote trustworthiness by conditioning classification tasks on an intermediate level of human-like concepts. This enables human interventions which can correct mispredicted concepts to improve the model's performance. However, existing concept bottleneck models are unable to find optimal compromises between high task accuracy, robust concept-based explanations, and effective interventions on concepts---particularly in real-world conditions where complete and accurate concept supervisions are scarce. To address this, we propose Concept Embedding Models, a novel family of concept bottleneck models which goes beyond the current accuracy-vs-interpretability trade-off by learning interpretable high-dimensional concept representations. Our experiments demonstrate that Concept Embedding Models  (1) attain better or competitive task accuracy w.r.t. standard neural models without concepts, (2) provide concept representations capturing meaningful semantics including and beyond their ground truth labels, (3) support test-time concept interventions whose effect in test accuracy surpasses that in standard concept bottleneck models, and (4) scale to real-world conditions where complete concept supervisions are scarce.

Concept Embedding Models: Beyond the Accuracy-Explainablly Trade-Off
Mateo Espinosa Zarlenga' University of Cambridge me4660cam.. ac uk
Pietro Barbiero' University of Cambridge pb7370cam.. ac. uk
Gabriele Ciravegna Université Côte d' Azur, Inria, CNRS,
I3S, Maasai, Nice, France
fr
Giuseppe Marra
Francesco Giannini
KU Leuven
University Siena
giuseppe.. marra@kuleuven. be
francesco wwwooogggnnnnnn giannini bunisi.ii
Michelangelo Diligenti University of Siena dili diligmic@din unisi.it
Zohreh Shams Babylon Health University of Cambridge zs3150cam. ac. uk
Frederic Precioso
Stefano Melacci University of Siena
Adrian Weller University of Cambridge Alan Turing Institute aw665@cam ac
Université Côte Azur, Inria, CNRS, I3S, Maasai, Nice, France fprecioson wwweeiiissuuiic....mm
mela@diism unisi.it
Pietro Lio University of Cambridge p12190cam. ac. uk
Mateja Jamnik
University of Cambridge mateja. jateeja.jamiin jamnikēcl, cam. ac uk
Abstract
Deploying AI Al-powered systems requires trustworthy models supporting effective human interactions, going beyond raw prediction accuracy. Concept bottleneck models promote trustworthiness by conditioning classification tasks interme- diate level of human-like concepts. This enables human interventions which can correct mispredicted concepts improve the model's performance. However, ex- isting concept bottleneck models are unable to find optimal compromises between high task accuracy, robust concept-based explanations, and effective interventions concepts panittsually in real-world conditions where complete and accurate concept supervisions are scarce. To address this, we propose Concept Embed- ding Models, novel family of concept bottleneck models which goes beyond the current accuracy-vs securacy-vs-interpee ininrre-ttnttterretaabittyty trade-off by learning interpretable high- dimensional concept representations. Our experiments demonstrate that Concept Embedding Models (1) attain better or competitive task accuracy w.r.t. standard neural models without concepts, (2) provide concept representations capturing meaningful semantics including and beyond their ground truth labels, (3) support test-time concept interventions whose effect in test accuracy surpasses that in standard concept bottleneck models, and (4) scale to real-world conditions where complete concept supervisions scarce.
"Equal contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Introduction
What is an apple? While any child can explain what an "apple" is by enumerating its characteristics, deep neural networks (DNNs) fail to explain what they learn in human understadddee terms despite their high prediction accuracy M. This cesuraccy---intepretaable
trade-off has become a major concern as high-perffee nigh-performing DNNs become commonplace in practice 24, thus questioning the ethical [ and legal 7 g ramifications of their deployment.
Concept bottleneck models (CBMs, (9]. Figure Tap aim at replacing "black-box" DNNs by first learning predict set of concepts, that is, "interpretable" high-level units of information (e.g.,
"color" or "shape") [T]] provided at training time [1 12), and then using these concepts to learn a downstream classification task. Predicting tasks
function of concepts engenders user trust [T3 by allowing predictions to be explained in terms of concepts and by supporting human interventions, where at test-time an expert can correct a mispredicted concept, possibly changing the CBM's output. That said, concept bottlenecks may impair task accuracy [4] especially when concept labels do not contain all the necessary information for accurately predicting downstream task (i.e.. they form an "incomplete" representation of the task [15), as seen in Figure [b] In principle, extending a CBM's bottleneck with a set of unsupervised neurons may improve task accuracy, as observed by Mahinpei et al. [14]. However, as we will demonstrate in this work, such a hybrid approach not only significantly hinders the performance of concept interventions, but it also affects the interpretabilityy of the learnt bottleneck, thus undermining user trust [33 Therefore, we argue that novel concept-basee architectures are required overcome the current accuracyinterpretabliiy pitfalls of CBMs, thus enabling their deployment in real-world settings where concept annotations are likely to be incomplete.
MA8K
Accuracy. Explainability
Y
Apple T(ce)
CBM Concept Efficiency (CUB)
Trade-Gr Off
100
Task Accuracy (96)
Teak Accurasy (96)
18 8
90
OURS
C
Shape: sphere
80
Coloc: rod
70
8
[pix)
i 88
60
C2V FAAty
50
X
50
60
70
80
90 100
Concepts Training (94)
Concept Alignment (96)
(a)
(b)
(c)
Figure 1: (a) A concept bottleneck model, (b) task accuracy after using only a fraction of total concept annotations to train a CBM on CUB [16] and, (c) the securacy-v-iinteppetabbl trade-off (corner star represents the optimal trade-off).
In this paper, we propose Concept Embedding Models (CEMs), a novel concept bottleneck model (described in Section 3) which overcomes the current accuracy-vs -s-interpretaailiity trade-off found in concept-incomplet settings (as shown in Figure Tc Furthermore, we introduce two new metrics for evaluating concept representations (Section 4 and use them to help understand why our approach circumvents the limits found in the current state-of-the-art CBMs (Section 5). Our experiments show that CEM (1) attains better or competitive task accuracy w.r.t. standard DNNs trained without concept supervision, (2) learns concept representations that capture meaningful semantics at least as well as vanilla CBMs, and (3) supports effective test-time concept interventions.
2 Background
Concept bottleneck models (CBMs, 9
A concept bottleneck model learns a mapping from
samples x E X labels y Y by means of: (i) concept encoder function g : X + C which maps samples from the input space x E X C R to an intermediate space è E C C R formed by k concepts, and (ii) a label predictor function : C - Y which maps samples from the concept space è E C' to downstream task space ÿ Y C R A CBM requires a dataset composed of tuples in X C x y, where each sample consists of input features x (e.g., an image's pixels), ground truth concept vector E {0, 1 (i.c., a binary vector where each entry represents whether concept is

---

active or not) and a task label y (e.g., an image's 's class). During training, a CBM is encouraged to align è = ((x) and ÿ f(g(x)) to x's corresponding ground truth concepts с and task labels y-- respectively. This can be done by (i) sequentially training first the concept encoder and then using its output to train the label predictor, (ii) independently training the concept encoder and label predictor and then combining them to form CBM, or (iii) jointly training the concept encoder and label predictor via a weighted sum of cross entropy losses.
Concept representations in CBMs For each sample x X, the concept encoder g learns k different scalar concept representations ći...., CR. Boolean and Fuzzy CBMs [9] assume that each dimension of è, which we describe by & = s(O) e [0. is aligned with single ground truth concept and represents a probability of that concept being active. The glement-wise activation function s R + [o, 1] can be either a thresholding function s(x) - 1220.5 (Boolean CBM) or sigmoidal function s(x) & 1/[1+
(Fuzzy CBM) A natural extension of this framework is a Hybrid CBM [14), where è E R (k++) contains unsupervised dimensions and k supervised concept dimensions which, when concatenated, form shared concept vector (i.e., an "embeeding"").
Concept Interventions in CBMs Interventions are one of the core motivations behind CBMs [9]. Through interventions, concept bottleneck models allow experts to improve a CBM's task performance by rectifying mispredicted concepts by setting, at test-time, é := (where c; is the ground truth value of the i-th concept). Such interventions can significantly improve CBMs performance within a -n-n--iintttooooo setting 99. Furthermore, interventions enable the construction of meaningful concept-basee counterfactuals 77. For example, intervening on a CBM trained to predict bird types from images can determine that when the size of a "black" bird with "black" beak changes from "medium" to "large", while other concepts remain constant, then one may classify the bird as a "raven" rather than a "crow".
3 Concept Embedding Models
In real-world settings, where complete concept annotations are costly and rare, vanilla CBMs may need to compromise their task performance order to preserve their interpretability (9. While Hybrid CBMs are able to overcome this issue by adding extra capacity in their bottlenecks, this comes at the cost of their interpretability and their responsiveness to concept interventions, thus undermining user trust 133. To overcome these pitfalls, we propose Concept Embedding Models (CEMs), concept- based architecture which represents each concept as a supervised vector. Intuitively, using high-dimensional embeddings represent each concept allows for extra supervised learning capacity, as opposed to Hybrid models where the information flowing through their unsupervised bottleneck activations is concept-agnostic. In the following section, we introduce our architecture and describe how it learns a mixture of two semantic embeddings for each concept (Figure[) 2}. We then discuss how interventions are performed in CEMs and introduce RandInt, train-time regularisation mechanism that incentivises our model to positively react to interventions at test-time.
3.1 Architecture
For each concept, CEM learns mixture of two embeddings with explicit semantics representing the concept's 's uctivity. Such design allows our model to construct evidence both in favour of and against concept being active, and supports simple concept interventions one switch between the two embedding states at intervention time.
We represent concept C; with two embeddings et C, E Rm each with a specific semanties: ot represents its active state (concept is true) while é, represents its inactive state (concept is false). To this aim, a DNN (x() learns a latent representation h E R Thatte which is the input to CEM's embedding generators. CEM then feeds h into two concept-specific fully connected layers, which learn two concept embeddings in R" namely Jt¹ = o, (h) a(W, h + bx²) and è, = o (h) a(W, h + b, F Notice that while more complicated models can be used to parameterise our concept embedding generators of (h) and o; (h). opted for simple lne-layer neural network to constrain
In practice (e.g., e. ( one may use logits rather than sigmoidal activations to improve gradient flow In practice, we Paky-ReLU for the activation a(-).
3

---

Label Predictor
Input
Latent Code
Embedding Generators
Bottleneck
è
fx(x)
f(ō)
(2)
Figure 2: Concept Embedding Model: from intermediate latent code h, we learn two embeddings per concept, one for when it is active (i.e., ?, ), and another when it is inactive (i.e., c,), Each concept embedding (shown this example as a vector with m = 2 activations) is then aligned to its corresponding ground truth concept through the scoring function s(-), which learns to assign activation probabilities P. for each concept. These probabilities are used to output an embedding for each concept via weighted mixture of each concept's 's positive and negative embedding.
parameter growth in models with large bottlenecks. Our architecture encourages embeddings e and 6, to be aligned with ground-truth concept 6; via learnable and differentiable scoring function R -> [0, 1]. trained to predict the probability i - s((,,= ,, a oW, \T + b.) of concept e; being active from the embeddings' joint space. For the sake of parameter efficiency, parameters W. and b, are shared across all concepts. Once both semantic embeddings are computed, we construct the final concept embedding è; for 6₁ as a weighted mixture 67" and c,
é A (p:C, + - -
Intuitively, this serves a two-fold purpose: (i) forces the model depend only on ह when the i-th concept is active, that is, c; (and only on é, when inactive), leading two different semantically meaningful latent spaces, and (ii) it enables a clear intervention strategy where one switches the embedding states when correcting mispredicted concept, as discussed below. Finally, all mixed concept embeddings are concatenated, resulting in bottleneck g(x) = è with k m units (see end of Figure 2.. This is passed the label predictor obtain downstream task label. In practice. following Koh et al. [9] we use an interpretable label predictor f parameterised by a simple linear layer, though more complex functions could be explored too. Notice that as in vanilla CBMs, CEM provides concept-based explanation for the output of f through its concept probability vector p(x) & [pt. : p indicating the predicted concept activity. This architecture can be trained in an end-to-end fashion by jointly minimising via stochastic gradient descent weighted sum of the cross entropy loss both task prediction and concept predictions:
C E(x.y.)) (uusk (v. + aLCrossEorr (c.
(c. p(x))] (c.p(x))]
(1)
where hyperparameter E R+ controls the relative importance of concept and task accuracy.
3.2 Intervening with Concept Embeddings
As in vanilla CBMs. CEMs support test-time concept interventions. To intervene on concept c, one can update è; by swapping the output concept embedding for the one semantically aligned with the concept ground truth label. For instance, if for some sample x and concept e; CEM predicted ú = 0.1 while a human expert knows that concept G; is active (c; = 1), they can perform the intervention 9. := 1. This operation updates CEM's bottleneck by setting è; to é rather than (0.1ēt + 0.9ē,) Such an update allows the downstream label predictor to act on information related to the corrected concept. In addition, we introduce RandIni, a regularisation strategy exposing CEMs to concept interventions during training to improve the effectiveness of such actions test-time. RandInt randomly performs independent concept interventions during training with probability Pint

---

(i.e.. Ps is set to P := for concept o; with probability Pin).. In other words, for all concepts C,,
during training we compute embedding é; as:
(Gét (1)c)),) - (p.c) +1 p.),,) with probability (1 (1-Pinl
with probability Pint
while at test-time we always use the predicted probabilities for performing the mixing. During backpropagation, this strategy forces feedback from the downstream task to update only the correct concept embedding (e.g. C if c 1) while feedback from concept predictions updates both c and 6.. Under this view, RandInt can be thought of as learning an average over an exponentially large family of CEM models (similarly to dropout (18) where some of the concept representations are trained using only feedback from their concept label while others receive training feedback from both their concept and task labels.
4 Evaluating concept bottlenecks
To the best of our knowledge, while great deal of attention has been paid to concept- based explainability in recent years, existing work still fails provide methods that can be used to evaluate the interpretability of a concept embedding or to explain why certain CBMs underperform in their task predictions. With this in mind, we propose (i) new metric for evaluating concept quality in multidimensional representations and (ii) an informationn-hhorree information flow in concept bottlenecks, can help understand why CBM may underperform in a downstream task.
method which, by analysing the
Concept Alignment Score (CAS) The Concept Alignment Score (CAS) aims measure how much learnt concept representations can be trusted as faithful representations of their ground truth concept labels. Intuitively, CAS generalises concept accuracy by considering the homogeneity of predicted concept labels within groups of samples which are close in a concept subspace. More specifically, for each concept a the CAS applies a clustering algorithm
find p > clusters, assigning to each sample x(5) cluster label o. E {1.....pp .P}. We compute this label by clustering samples using their i-th concept representations (e()) _e(0) ...
...... Given N test samples, the homogeneity score (
hx()) 19 then computes the conditional entropy H of ground truth labels C; -c,-)) J==1 w.r.t. cluster
labels [I, p) ( 1 that is, h = 1 when H(C,, II;) = O and h = 1 H(C,, H)) otherwise. The higher the homogeneity, the more learnt concept representation is "aligned" with its labels, and can thus be trusted as faithful representation. CAS averages homogeneity scores over all concepts and number of clusters p. providing a normalised score [0, 1]:
CAS(C), ... 24)
Σ M(C,, (..,p)
2=2
To tractably compute CAS in practice, we sum homogeneity scores by varying p across p E {2, 2 5, 2 + 26, ...,, . for some 8 > 1 (details in Appendix A.1). A.1.. Furthermore, we use k- k-Medoids [20] for cluster discovery, as used in Ghorbani al. [10 and Magister
(2T), and use concept logits when computing the CAS for Boolean and Fuzzy CBMs. For Hybrid CBMs, we use è; & [C]e:A+>]] 1]](++1)]] as the concept representation for given that the extra capacity is a shared embedding across all concepts.
Information bottleneck The relationship between the quality of concept representations w.r.t. the input distribution remains widely unexplored. Here we propose analyse this relationship using information theory methods for DNNs developed in Tishby et al. [22) and Tishby and Zaslavsky [23]. In particular, we compare concept bottlenecks using the Information Plane method [22] study the information flow at concept level. To this end, we measure the evolution of the Mutual Information (I(.,)) of concept representations w.r.t. the input and output distributions across training epochs. We conjecture that embedding based CBMs circumvent the information bottleneck by preserving more information than vanilla CBMs from the input distribution as part their nigh-dimensional activations. If true, such effect should be captured by Information Planes in the form of positively correlated evolution of I [(X, Ć'), the Mutual Information (MI) between inputs X and learnt concept
5

---

representations Č, and I(Ć, Y), the MI between learnt concept representations C and task labels Y. In contrast, we anticipate that scalar-based concept representations (e.g.. Fuzzy and Bool CBMs), will be forced to compress the information from the input data at concept level, leading to a compromise between the [(X, C)) and I(Č, Y). Further details our implementation are in Appendix A.2
5 Experiments
In this section, we address the following research questions:
Task accuracy - What is the impact of concept embeddings on a CBM's downstream task performance? Are models based on concept embeddings still subject an information bottleneck [22)?
Interpretability- - Are CEM concept-based explanations aligned with ground truth con- cepts? Do they offer interpretability beyond simple concept prediction and alignment?
Interventions - Do CEMs allow meaningful concept interventions when compared to Hybrid or vanilla CBMs?
5.1
Setup
Datasets For our evaluation, we propose three simple benchmark datasets of increasing con- cept complexity (from Boolean to vector-based concepts): (1) XOR (inspired by (24)) in which tuples (I1, E [0, 1]2 are annotated with two Boolean concepts {1c22005) and labeled as y y= C1 XOR C2; (2) Trigonometric (inspired by (14)) in which three latent normal random variables {br are used to generate a 7-dimensional input whose three concept annotations are a Boolcan and task label is a linear function of the same; (3) Dot in which four latent random vectors V1, V2, W1, W2 E R² are used to generate two concept annotations, representing whether latent vectors v; point in the same direction of reference vectors w,, and task labels, representing whether the two latent vectors V1 and V2 point in the same direction. Furthermore, we evaluate our methods on two real-world image tasks: the Caltech-UCSS. Birds-200-201 dataset (CUB, (I6)), preprocessed as in [9, and the Large-scale CelebFaces Attributes dataset (CelebA, (25). In our CUB task we have 112 complete concept annotations and 200 task labels while in our CelebA task we construct 6 balanced incomplete concept annotations and each image can be one of 256 classes. Therefore, we use CUB test each model in real-world task where concept annotations are numerous and they form a complete description of their downstream task. In contrast, our CclebA task is used to evaluate the behaviour of each method in scenarios where the concept annotations are scarce and incomplete w.r.t. their downstream task. Further details on these datasets and their properties are provided in Appendix A.3
function of {b, )
Baselines We compare CEMs against Bool, Fuzzy, and Hybrid Joint-CBMs as they all provide concept-based explanations for their predictions and allow concept interventions at test-time. Note that this set excludes architectures such as Self-Explainable Neural Networks (26) and Concept Whitening [12] as they do not offer clear mechanism for intervening on their concept bottlenecks. To ensure fair comparison, we use the same architecture capacity across all models. We empirically justify this decision in Appendix A.4 by showing that the results discussed in this section do not change if one modifies the underlying model architecture. Similarly, we use the same values of and m within a dataset for all models trained on that dataset and set Piut 0.25 when using CEM (see Appendix A.5 for ablation study of Pint showing how intervention improvement plateaus around this value). When using Hybrid CBMs, we include as many activations in their bottlenecks as their CEM counterparts (so that they both end up with bottleneck with km activations) and use Leaky-Rel. U activation for unsupervised activations. Finally, our evaluation we include a DNN without concept supervision with the same capacity as its CEM counterpart to measure the effect of concept supervision in our model's 's performance. For further details on the model architectures and training hyperparameters, please refer to Appendix A.6
Metrics We measure a model's performance based on four metrics. First, we measure task and concept classification performance in terms of both iask and mean concept accuracy. Second, we evaluate the interpretability of learnt concept representations via our concept alignment score. To casily visualise the accuracy- initeeerttit trade-off, we plot our results in a two-dimensional

---

XOR
Trigonometry
Dot
CUB
CelebA
Task. Accurecy (96)
100
85
90
Mydei
80
75
70
80
Sorean CBM
Facdy CSN
60
53
50
65
50
50
80
85
70 Concept Alignment (9))
Concept Aligrment (9)
Concept Alignment (9)
Concapt Algnment (9)
Concept Alignment (9))
No concepts
Bookean-CBM
FBZZy-CBM
Hybrid-CBM
CEM (ours)
Figure 3: sesuuaccyy--iteeprttaalll trade-off in terms of task accuracy and concept alignment score for different concept bottleneck models. In CelebA, our most constrained task, we show the top-1 accuracy for consistency with other datasets.
plane showing both task accuracy and concept alignment. Third, we study the information bottleneck in our models via mutual information (MI) and the Information Plane technique [27]. Finally, we quantify user trust [13] by evaluating a model's task performance after concept interventions. All metrics in evaluation, across all experiments, are computed on test sets using 5 random seeds, from which we compute a metric's s mean and 95% confidence interval using the Box-Cox transformation for non-normal distributions.
5.2 Task Accuracy
CEM improves generalisation accuracy (y-axis of Figure Our evaluation shows that embedding-basee CBMs (i.c., Hybrid-CBM and CEM) can achieve competitive or better down- stream accuracy than DNNs that do not provide any form of concept-baset explanations, and can easily outperform Boolean and Fuzzy CBMs by a large margin (up to +45% on Dot). This effect is emphasised when the downstream task is not linear function of the concepts (e.g.. XOR and Trigonometry) or when concept annotations are incomplete (e.g., Dot and CelebA). At the same time, observe that all concept-based models achieve similar high mean concept accuracy across all datasets (see Appendix (.7)). This suggests that, as hypothesised, the trade-off between con- cept. accuracy and task performance in concept-incomplete tasks significantly alleviated by the introduction of concept embeddings training our baselines using only a fraction of the available concepts in CUB seen in Appendix A.88 Finally, notice that CelebA showcases how including concept supervision during training (as in CEM) lead to even higher task accuracy than the one obtained by vanilla end-to-end model (+5% compared to "No concepts" model). This result further suggests that concept embedding representations enable high levels of interpretability without sacrificing performance.
CBM's bottleneck. Similar results can be observed when
CEM overcomes the information bottleneck (Figure The Information Plane method indicates, as hypothesised, that embedding-basee -based CBMs (i.e., Hybrid-CBM and CEM) do not compress input data information, with [X, C') monotonically increasing during training epochs. On the other hand, Boolean and Fuzzy CBMs. well vanilla end-to-end models, tend to "forget" [27] input data information in their attempt to balance competing objective functions. Such a result constitutes plausible explanation to why embedding- based representations are able maintain both high task accuracy and mean concept accuracy compared to CBMss with scalar concept representations. In fact, the extra capacity allows CBMs maximise concept accuracy without werrcconttaiiing concept representations, thus allowing useful input information to pass by. In CEMs all input information flows through concepts, as they supervise the whole concept embedding. In contrast with Hybrid models, this makes the downstream tasks completely dependent on concepts, which explains the higher concept alignment scores obtained by CEM (as discussed the next subsection).
5.3 Interpretability
CEM learns more interpretable concept representations (x-axis of Figure Using the proposed CAS metric, we show that concept representations learnt by CEMs have alignment scores competitive or even better (e.g., on CelebA) than the ones of Boolean and Fuzzy CBMs. The alignment score also shows, hypothesised, that hybrid concept embeddings are the least faithful representations with
7

---

XOR
Trigonometry
Dot
CUB
CelebA
1.0
5.0
0.5
0.5
2.5
0.0
0
0.0
0
2.5
5.0 [(X, C)
7.5
2.5
5.0
7.5
0
1
0
5
10
1(X,
C) No concepts
1(X, Ç)
1XX č)
I(X, Ć)
Boolean-CBM
FuzZyy-CMM
HybridCBMM
CEM (ours)
Figure 4: Mutual Information (MI) of concept representations (C) w.r.t input distribution (X) and ground truth labels (Y) during training. The size of the points is proportional the training epoch.
Sampp
Newrer beughters
His White Wings (CEMM
(1lyhei))
letve Conveg traeive
(b)
Figure 5: Qualitative results: (a and b) t-SNE visualisations of has white wings" concept embedding learnt in CUB with sample points coloured red if the concept active in that sample, (c) top-5 test neighbours of CEM's embedding for the concept "has white wings" across 5 random test samples.
alignment scores up to 25% lower than CEM in the Dot dataset. This due to their unsupervised activations containing information which may not be necessarily relevant to a given concept. This result is a further evidence for why we expect interventions be ineffective Hybrid models (as we show shortly).
CEM captures meaningful concept semantics (Figure Our concept alignment results hint at the possibility that concept embeddings learnt by CEM may be able to offer more than simple concept prediction. In fact, we hypothesise that their seemingly high alignment may lead these embeddings forming more interpretable representations than Hybrid embeddings, which can lead to these embeddings serving as better representations for different tasks. To explore this, we train a Hybrid-CBA) and a CEM, both with m = 16, using a variation of CUB with only 25% of its concept annotations randomly selected before training, resulting in a bottleneck with 28 concepts (see Appendix A.9 for details). Once these models have been trained to convergence, we use their learnt bottleneck representations to predict the remaining 75% of the concept annotations in CUB using a simple logistic linear model. The model trained using the Hybrid bottleneck notably underperfoms when compared to the model trained using the CEM bottleneck (Hybrid-trained model has a mean concept accuracy of 91.83% + 0.51% % while the CEM-trained model's concept accuracy is 94.33% + 0.88%). This corroborates our CAS results by suggesting that the bottlenecks learnt by CEMs are considerably more interpretable and can therefore serve as powerful feature extractors.
We can further explore this phenomena qualitatively by visualising the embeddings learnt for single concept using its 2-dimensional t-SNE (28) plot. As shown in colour in Figure 5al we can see that the embedding space learnt for a concept é; (here show the concept "has white wings") forms two clear clusters of samples, one for points in which the concept is active and one for points in which the concept inactive. When performing similar analysis for the same concept in the Hybrid CBM (Figure 5b), where we use the entire extra capacity Čja.--] as the concept's 's embedding representation, we see that this latent space not as clearly separable as that CEM's embeddings, suggesting this latent space is unable to capture sonnntt-semantics as clearly CEM's latent space. Notice that CEM's t-SNE seems also show smaller subclusters within the activated and inactivated clusters. As Figure 5c shows, by looking at the nearest Euclidean neighbours in concept's 's embedding's 's space,

---

CelebA
CelebA
8 8
23
Groups
Concepts intervered
Groups incorrecty
Insorrecty (ours)
Back-CBM
FBZZy-CBM
(ours)
Figure 6: Effects of performing positive random concept interventions (left and center left) and incorrect random interventions (center right and right) for different models in CUB and CelebA. As in [9] when intervening CUB we jointly set groups of mutually exclusive concepts.
we see that CEM concepts clearly capture a concept's 's activation, as well as exhibit high class-wise coherence by mapping same-type birds close to each other (explaining the observed subclusters). These results, and similar qualitative results Appendix [A.10] suggest that CEM is learning a hierarchy in its latent space where embeddings separated with respect their concept activation and, within the set of embeddings that have the same activation, embeddings are clustered according their task label.
5.4 Interventions
CEM supports effective concept interventions and is more robust to incorrect interventions (Figure (6) When describing our CEM architecture, we argued in favour of using a mixture of two semantic embeddings for each concept this would permit test-time interventions which can meaningfully affect entire concept embeddings. In Figure left and center-left, we observe, hypothesised, that using a mixture of embeddings allows CEMs be highly responsive to random concept interventions in their bottlenecks. Notice that although all models have a similar concept accuracy (see Appendix (A.7), observe that Hybrid CBMs, while highly accurate without interventions, quickly fall short against even scalar-based CBMs once several concepts are intervened in their bottlenecks. In fact, we observe that interventions in Hybrid CBM bottlenecks have little effect on their predictive accuracy, something that did not change if logit concept probabilities were used instead of sigmoidal probabilities. However, even interventions performed by human experts quite rarely perfect. For this reason, we simulate incorrect interventions (where a concept is set to the wrong value) measure how robust the model is such errors. We observe (Figure(jcennter-o centter-iight and right) that CEM's performance deteriorates as more concepts are incorrectly intervened on (as opposed hybrid-CBMs, while it can withstand few errors without losing much performance (as opposed to Bool and Fuzzy-CBMs). We suggest that this is consequence of CEM's "incorrect" embeddings still carrying important task saec-fpec information which can then be used by the label predictor produce more accurate task labels, something worth exploring in future work. As result, users can trust CEMs to better handle a small number of accidental mistakes made by human experts when intervening in its concept activations. Finally, by comparing the effect of interventions both CEMs and CEMs trained without RandInt, we observe that RandInt fact leads to a model that is not just significantly more receptive to interventions, but is also able to outperform even scalar- based CBMs when large portions of their bottleneck are artificially set by experts (c.g., as in CclebA). This, shown in Appendix A.II comes without a significant computational training costs for CEM. Interestingly, such a positive effect in concept interventions is not observed if RandInt is used when training other baselines (see Appendix A..2 for explanation). This suggests that our proposed architecture can not only be trusted in terms of its downstream predictions and concept explanations, as seen above, but can also be a highly effective model when used along with experts that can correct mistakes in their concept predictions. For further details, including an exploration of performing interventions with Sequential and Independent CBMs, please refer Appendix A.13
Discussion
Relations with the state-of-the-art Concept bottleneck models engender user trust by (i) forcing information flow through concept-aligned activations during training and (ii) by supporting

---

human interventions in their concept predictions. This allows CBMs to circumvent the well- unreliability of post-hoc methods [29 30 T0 such as saliency maps B1 and invites their use in settings where input features are naturally hard to reason about (e.g., raw image pixels). In addition, CBMs encourage human interactions allowing experts to improve task performance by rectifying mispredicted concepts, which contrasts other concept-based interpretable architectures such as Self-Explainable Neural Networks [26) and Concept Whitening [12]. However, our experiments show that all existing CBMs are limited to significant
trade-offs. In this respect, our work reconciles theoretical results with empirical observations: while theoretical results suggest that explicit per-concept supervisions should improve generalisation bounds [34], in contrast Koh et al. [9]. Chen et al. [12], and Mahinpei et al. [4] empirically show how learning with intermediate concepts may impair task performance in practice. The Information Plane method [27] reveals that the higher generalisation error of existing concept bottleneck models might be explained as compression in the input information flow caused by narrow architectures of Boolean and Fuzzy CBMs. In contrast, CEM represents the first concept-based model which does not need to compromise between task accuracy, concept interpretability gap in the literature. Furthermore, through an ablation study on CEM's embedding size shown in Appendix A.14 we see that one does not need increase the embedding size drastically to begin to see the benefits of using a CEM over a vanilla CBM or an end-to-end black box DNN. We note that as stronger interpretable models are deployed, there are risks of societal harm which we must be vigilant to avoid.
intervention power, thus filling this
Conclusion Our experiments provide significant evidence in favour of CEM's accu- racy/interpeetaiiiity and, consequently, in favour its real-world deployment. In particular, CEMs offer: (i) state-of-the-arr task aceuracy, (ii) interpretable concept representations aligned with human ground truths, (iii) effective interventions on learnt concepts, and (iv) robustness to incorrect concept interventions. While in practice CBMs require carefully selected concept annotations during training, which can be as expensive as task labels to obtain, our results suggest that CEM is more efficient in concept-iccopplrr settings, requiring less concept annotations and being more applicable to real-world tasks. While there is room for improvement in both concept alignment and task accuracy in challenging benchmarks such as CUB or CelebA, as well as in resource utilisation during infer- ence/training (see Appendix (. our results indicate that CEM advances the state-of-the-art for the aceuracy-vs-inted intteeereaty off making progress on crucial concern in explainable AI.
Acknowledgments and Disclosure of Funding
The authors would like to thank Carl Henrik Ek, Alberto Tonda, Andrei Margeloiu, Fabrizio Silvestri, and Maria Sofia Bucarelli for their insightful comments on carlier versions of this manuscript. MEZ acknowledges support from the Gates Cambridge Trust via a Gates Cambridge Scholarship. PB acknowledges support from the European Union's 's Horizon 2020 research and innovation programme under grant agreement No 848077. GC and FP acknowledges support from the EU Horizon 2020 project A14Media, under contract no. 951911 1 and by the French government, through Investments in the Future projects managed by the National Research Agency (ANR), 31A Cote d' A'ur with the AW acknowledges support from a Turing AI Fellowship
reference number ANR-19-P3IA--0N under grant FP/VV22529/11 The Alan Turing Institute, and the Leverhulme Trust via CFI. GM is funded by the Research maundation-Flandes by TAILOR, a project funded by EU Horizon 2020 research and innovation programme under GA No 952215. This work was also partially supported by HumanE-AI- AI-Net a project funded by EU Horizon 2020 research and innovation programme under GA 952026. MJ is supported by the EPSRC grant EP/T019603/1.
Wlanddean,,,,,, GA No 1239422N). FG is supported
References
[1] Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al. Explainable Artificial Intelligence (XAI): Concepts. taxonomies, opportunities and challenges toward responsible AI. Information Fusion, 58:82-115, 2020.
[2] Michelle Goddard. The EU General Data Protection Regulation (GDPR): European regulation that has global impact. International Journal of Market Research, 59(6):703-705, 2017.
10

---

[3] Gabrielle Ras, Marcel van Gerven, and Pim Haselager. Explanation methods in deep learning: Users, values, concerns and challenges. Explainable and Interpretable Models Computer Vision and Machine Learning, pages 19-36. Springer, 2018
[4] Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong. Interpretable machine learning: Fundamental principles and 10 grand challenges. arXiv preprint 1.kv:22003.1255/1 2021.
[5] Juan Manuel Durán and Karin Rolanda Jongsma. Who is afraid of black box algorithms? On the epistemological and ethical basis of trust in medical AI. Journal of Medical Ethics, 47(5): 329-335, 2021.
[6] Samuele Lo Piano. Ethical principles in machine learning and artificial intelligence: cases from the field and possible ways forward. Humanities and Social Sciences Communications, 7(1):
1-7, 2020.
[7] Sandra Wachter, Brent Mittelstadt, and Chris Russell. Counterfactual explanations without opening the black box: Automated decisions and the GDPR. Harv. JL & Tech., 31:841, 2017. 2017.
[8] EUGDPR. GDPR. General data protection regulation, 2017.
[9] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning, pages 3338-334.. PMLR. 2020.
[10] Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. In Proceedings of the AAAI conference artificial intelligence, volume 33, pages 3681-3688, 2019.
[11] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pages 2668 2677. PMLR, 2018.
[12] Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2(12):772-782, 2020.
[13] Max W Shen. Trust AI: Interpretability not necessary or sufficient, while black-box interaction is necessary and sufficient. arXiv preprint arXiv:2202.05302. 2022.
[14] Anita Mahinpei, Justin Clark, Isaac Lage, Finale Doshi-Velez, and Weiwei Pan. Promises and pitfalls black-box concept learning models. arXiv preprint arXiv:2106.13314, 2021.
[15] Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On completeness-aware concept-based explanations in deep neural networks. Advances in Neural Information Processing Systems, 33:20554-20565 2020.
[16] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset, 2011.
[17] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference artificial intelligence and statistics, pages 315-323, JMLR Workshop and Conference Proceedings, 2011.
[18) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov, Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 1919:119291195 2014.
[19] Andrew Rosenberg and Julia Hirschberg. V-measure: A conditional entropy-based external cluster evaluation measure. In Proceedings of the 2007 joint conference on empirical methods in natural language processing and computational natural language learning (EMNLP-CoNLL) pages 410-420, 2007.
[20] Leonard Kaufman and Peter J Rousseeuw. Partitioning around medoids (program pam). Finding groups in data: introduction to cluster analysis, 344:68-125 1990.
11

---

[21] Lucie Charlotte Magister, Dmitry Kazhdan, Vikash Singh, and Pietro Liò. Gcexplainer: Humma-in-the-loop concept-bued based explanations for graph neural networks. arXiv preprint arXiv:2107.1 1889, 2021.
[22] Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprini physics/0004057, 2000.
[23] Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE information theory workshop (ITW), pages 1-5. IEEE, 2015.
[24]
Marvin Minsky and Seymour A Papert. Perceptrons: An introduction to computational geomerry. MIT press, 1969.
[25] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference computer vision, pages 3730-3738, 2015.
[26] David Alvarez-Melis and Tommi S Jaakkola. Towards robust interpretability with self-explaining neural networks. arXiv preprint arXiv:/806.07553 2018.
[27] Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information, arXiv preprint arXiv:1703.00812 2017.
[28] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learming research, 9111), 2008
[29] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in neural information processing systems, 31, 2018.
[30] Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un) reliability of saliency methods. In Explainable Al: Interpreting, Explaining and Visualizing Deep Learning, pages 267-280. Springer, 2019.
[31] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pages 3319-3328, PMLR, 2017.
[32] Picter-Jan Kindermans, Kristof T Schütt, Maximilian Alber, Klaus-Robert Müller, Dumitru Erhan, Been Kim, and Sven Dähne. Learning how to explain neural networks: Patternnet and patternattribution. arXiv preprint arXiv: 1705.0599, 2017.
[33] Ramprasaath R Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Why did you say that? arXiv preprint 11. 2016.
[34] Chi Li, M Zeeshan Zia, Quoc-Huy Tran, Xiang Yu, Gregory D Hager, and Manmohan Chan- draker. Deep supervision with intermediate concepts. IEEE transactions on pattern analysis and machine intelligence, 41(8):1828-1884 2018.
[35] Artemy Kolchinsky, Brendan D Tracey, and David H Wolpert. Nonlinear information bottleneck. Entropy, 21(12):118) 2019.
[36] Andrew Michael Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan Daniel Tracey, and David Daniel Cox. On the information bottleneck theory of deep learning. In International Conference on Learning Representations, 2018. URL https wwwaareeiiee.net/oouueeeeeeeeeee WPG-A
[37] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[38] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv.1412.6980, 2014.
12

---

[39] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga. al. Pytorch: An imperative style, high-perfrrmance deep learning library. arXiv preprint arXiv: 1912.0/703, 2019.
[40] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit- learn: Machine learning in python. the Journal of machine Learning research, 12:2825-288 2830,, 2011.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] Section 3 and Section reflect our paper's contributions
and scope indicated in introduction
(b) Did you describe the limitations of your work? [Yes] See our conclusion in Section for a discussion on some of the limitations of our approach.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We discuss this in Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We have read the ethics review guidelines and ensured that your paper conforms to them.
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] We uploaded a zip file with our code and documentation in the supplemental material and made our code available in a public repository
(b) Did you specify all the training details (e.g., data splits, byperparameters, how they were chosen)? [Yes] All training details are provided in Appendix [A.6
(c) Did you report error bars (c.g., w.r.t. the random seed after running experiments multiple times)? [Yes] Error bars reported for all metrics and represent 95% confidence intervals of the mean, unless specified otherwise.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] Computational resources we used are specified in Appendix 15
4. If you are using existing assets (e.g., code, data, models) or curating/rellusing new assets...
(a) your work uses existing assets, did you cite the creators? [Yess] References we used are listed in Appendix A.odde. Appendix A.6 (models), and Appendix A.3 (data).
(b) Did you mention the license of the assets? [Yes] We specified the licence of our assets in Appendix 15
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] We included our new assets (code, data, and models) the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] All new datasets are synthetic.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] All new datasets are synthetic.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A] All new datasets are synthetic.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A] All new datasets are synthetic.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] All new datasets synthetic.
https///gthhbbcommmmmeccsssssssp
14


Reviews:
[{"review_id": "bcx3OWyfRFS", "reviewer": "Reviewer_rNZX", "summary": "This paper tackles the trustworthiness of concept bottleneck models (CBM) by improving the accuracy-interpretability tradeoff using a concept-based architecture (Concept Embedding Model or CEM) which represents each concept as a supervised vector. Furthermore, the authors propose two metrics for evaluating concept representations.", "questions": "See weaknesses.", "limitations": "The authors discussed the limitations of the proposed method in Section 6.", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) The paper tackles an important problem of trustworthiness and accuracy-interpretability tradeoff.\n2) The paper is well-organized and easy to follow.\n3) The authors performed multiple experiments demonstrating that the proposed approach would work reasonably well in different scenarios and improves the accuracy-interpretability.\n4) Scalable to real-world cases with lacking complete concept supervision.\n\nWeaknesses:\n1)  Lack of necessary statistical analysis. Some improvements do not seem to be significant. It would be better to provide task and concept accuracy as well as statistical significance, e.g., error bars, in a Table. \n2) Does RandInt regularizer increase the training time and cost? I think providing and comparison of the training cost (or model sizes) can also be helpful.\n3) Have you tried applying RandInt regularizer to Bool-CBM, Fuzzy-CBM, or other baselines? Based on the provided results in Figure 6, it seems that a lot of improvements are due to adding the RandInt regularizer. For a fair comparison, I would like to see if adding that to the existing methods can improve their performance as well. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "F6MNtMUlTGE", "reviewer": "Reviewer_6on2", "summary": "This paper introduces a model named concept embedding model (CEM) based on concept bottleneck model (CBM) architecture. Compared to CBM, CEM contains an embedding generator layer that considers two embedding representations (one for activate and one for inactivate) and then produces an embedding representation for one concept. Results show that the model produces high task accuracy and interpretability at the same time compared to CBM-family models. ", "questions": "(1)\tLine 163-164, p should be rho? \n\n(2)\tSec 5.4, line 312-314 suggests that CEM is robust to incorrect concepts in intervention. But is it desirable in the practice? Do authors consider the scenario where changing the important concept that can change the final decision?  If so, how it can truly reflex the user trust?\n", "limitations": "This paper does not emphasize the advantage of using two separate concept representations (c_hat+ and c_hat-), which is the novelty of the CEM, and does not evaluate the interpretability and user trust thoroughly. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: This paper tries to solve a challenging research question: to design XAI models which are good at task performance and interpretability. The authors conduct experiments on multiple datasets and evaluate different models using different metrics. \n\nWeakness: \n(1) The proposed model is not thoroughly studied. For instance, the relationship between c_hat+ and c_hat−. This is the novelty of the proposed model compared to CBM. For instance, for one concept, do these two embeddings represent opposite concepts? \n\n(2) There is some unclearness about the baseline models in the paper. For example, the CEM uses m=16 to represent one concept in c_hat (bottleneck). What is the dimension of c_hat for CBMs? In Appendix A.5, it says “γ = k · (m − 1)” for Hybrid-CBM. Does it mean that the dimension of c_hat is k · (m − 1) dimension? Why not k*m as in CEM?\n\n(3) The lack of justification for the proposed CAS. In Fig.3, the baseline model “no concepts” has an even better score than Boolean-CBM and Fuzzy-CBM. \n\n(4) The qualitative results are not very convincing. Fig.5 c and Appendix Fig. 5 show the samples and their nearest neighbors for one concept. However, it does not reflex information about the concept and for a well-trained classifier, it should find out the visually similar samples based on Euclidean distance in the embedding space.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DtgfwGlD8Oz", "reviewer": "Reviewer_JCLo", "summary": "The paper extends the concept bottleneck method by generating two embedding vectors for each concept; one representing the embedding when the concept is active while the other representing when the concept is inactive. These two embedding are linearly combined through a scoring function (similar to the gating mechanism) the produces a probability score of which embedding is to be used. The concept embedding vectors are then used in the downstream task for prediction. This extension increase the model capacity to encode more information about concepts. \n", "questions": "Read my listed points in the weakness section. ", "limitations": "No. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n- easy to read \n- simple extension \n\nWeakness\n- constraining the architecture of all other baselines as similar to the architecture of the proposed method seems to be unfair. \n- I still can not see how this model can be used in scenarios where incomplete concepts exist. In particular, in eq 1, how can the model be trained without a full supervision of concepts?\n- the information bottleneck metric in section 4 is a bit unclear. more detailed explanation would be preferred. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1rl95z5gdlz", "reviewer": "Reviewer_GkfX", "summary": "Concept bottleneck models implicitly learn to explain the downstream tasks in addition to learning how to perform them. However, these models forgo predictive performance on the downstream tasks. The authors propose Concept Embedding Models (CEMs), a novel family of concept bottleneck models that address this issue.", "questions": "- How robust are the concepts learned? Are they robust to changes in backgrounds?\n- Is it possible to relax assumptions that datasets should contain concept annotations? Maybe train the model with a contrastive loss for concepts instead of a supervised loss.\n- How does the underlying architecture of the model change the concepts generated?\n- How much is the additional computational overhead incurred while training CEMs as opposed to CBMs?\n\nI would encourage the authors to include results on OAI & AwA2 too.", "limitations": "I would encourage the authors to list the limitations of the proposed approach.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The proposed techniques perform better than CBMs and their variants on downstream tasks.\n- Concept representation learned by CEMs is more aligned to the ground truth concepts and successfully captures the semantics of the images.\nWeaknesses:\n- CEMs assume that the datasets contain annotations of concepts which is not valid in practice and are often quite expensive to obtain.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept