Do Concept Bottleneck Models Respect Localities?
Authors: Naveen Janaki Raman
Mateo Espinosa Zarlenga
Juyeon Heo
Mateja Jamnik
Venue: Accepted by TMLR ()

Abstract: Concept-based explainability methods use human-understandable intermediaries to produce explanations for machine learning models. These methods assume concept predictions can help understand a model's internal reasoning.  In this work, we assess the degree to which such an assumption is true by analyzing whether concept predictors leverage "relevant" features to make predictions, a term we call locality.  Concept-based models that fail to respect localities also fail to be explainable because concept predictions are based on spurious features, making the interpretation of the concept predictions vacuous. To assess whether concept-based models respect localities, we construct and use three metrics to characterize when models respect localities, complementing our analysis with theoretical results. Each of our metrics captures a different notion of perturbation and assess whether perturbing "irrelevant" features impacts the predictions made by a concept predictors. We find that many concept-based models used in practice fail to respect localities because concept predictors cannot always clearly distinguish distinct concepts. Based on these findings, we propose suggestions for alleviating this issue.

Published in Transactions on Machine Learning Research (06/2025)
Do Concept Bottleneck Models Respect Localities?
Naveen Raman Carnegie Mellon University
navernr@ermuu edu
Mateo Espinosa Zarlenga University of Cumbridge
me4660cma.ccm ac. uk
Juycon Heo University of Cambridge
jh2324 Rcam. ac. uk
Mateja Jamnik University of Cambridge
mateje.jammakBBBlcoo..oo uk
Reviewed on OpenReview: https: openreview. net/ forum? id= 1d=4mXkbUUX0f
Abstract
Concept-based explainability methods use human-understandlbble intermediaries to produce explanations for machine learning models. These methods assume concept predictions can help understand a model's internal reasoning. In this work, we assess the degree to which such an assumption true by analyzing whether concept predictors leverage "relevant" features to make predictions, a term we call locality. Concept-bascd models that fail to ro- spect localities also fail to be explainable because concept predictions based on spurious features, making the interpretation of the concept predictions vacuous. To assess whether concept-based models respect localities, construct and use three metrics to characterize when models respect localities, complementing our analysis with thcoretical results. Each of our metrics captures a different notion of perturbation and assess whether perturbing "ir- relevant" features impacts the predictions made by a concept predictors. We find that many concept-basede models used in practice fail to respect localities because concept predictors cannot always clearly distinguish distinct concepts. Based on these findings, we propose suggestions for alleviating this issue.
1 Introduction
Concept-based explainability provides insights into black-box machine learning models by constructing planations via hunan-understandalle "concepts" (Kim et al., 2018 Ghorbani et al. 2019 Koh et al. 2020). For example, we can explain that an image was predicted to be a cherry because we predicted that the con- cepts "Red Fruit" and "Cireular Fruit" were present in the image. Within this paradigm, Concept Bottleneck Models (CBMs) (Koh et al.) 2020) are family of models that first. predict. concepts from an input (via a concept predictor) and then predict a downstream label from these concepts (via a label predictor). This design allows CBMs provide concept-based explanations via their predicted concepts (Chauhan et al. 2022
Shin et al. 2023 Espinosa Zarlenga et al. 2023b).
Models within the concept-based paradigm roly on the assumption that model's concept predictions serve as an explanation for its predictions (Koh et al.) 2020). For example, if the concept predictor for "Red Fruit" mistakenly learns to predict the "Circular Fruit" concept instead, then explanations based on the "Red Fruit" concopt predictor would be incorrect as this predictor fails to reflect the presence of "Red Fruit." Therefore, the faithfulness of concept predictors to their underlying concepts is critical because these models operate under the strong assumption that concept predictions align with their corresponding semantic concepts.

---

Published in Transactions on Machine Learning Research (06/2025)
Locality Leakage
Concepts
Semantics
Value
Locelity
Coneept
Input Sample
Downetream Label
jeg. scd nose apple unddlerdddddddd ploels)
Perurr Out loqullty Featurus
X
Locality Masking
Ounep
Hiss Apple?
Conoept
g(x)
f(e)
fruits
Apple unchanged)
sbx)
CBM
Masi Locality Featuree (o zero out applererated placts)
Hos Plato?
Locality Intervention
Doncept
Apple was
Hae Orange?
508)
Perturb External Concepts
(e.g. charging Oranges to lamone))
Figure 1: We investigate whether concept-based models properly capture known concept "localitics", For example, we train a CBM to predict whether an image has a fruit (left panel A), with concepts such as "Has Bowl" and "Has Apple". Here, concepts are localised to image subregions (middle panel B). We study whether CBMs learn these localities by analysing how changes to input features, both within and outside a concept's locality, impact CBM's predictions (right panel C).
Our work analyzes the faithfulness of concept predictors by assessing the degree to which model's concept predictions are a proxy for model's internal reasoning. We do so to understand whether explanations from concept-based models are trustworthy, that is whether we can rely on the explanations from concept-based models to gain insights (Vashney) 2022]. We analyze the relationship between concept predictors and a model's internal reasoning through locality, which refers the idea that oftentimes only a subset of features are needed to predict concept (Figure []. For example, we assess the degree to which the prediction of the concept "Red Fruit" is influenced by background features irrelevant to that concept. We construct set of experiments to assess whether suucept-based models respect locality, this can give insight into their trustworthiness. Our results reveal that concept- based models often fail to respect localities, a significant shortcoming that casts doubl on their interpretability.
Our contributions are as follows: we (1) construct three metrics to quantify whether models respect localities found in datasets, (2) conduct thorough experiments across varioty of architectural and training choices to characterize when models respect localities, and (3) propose theoretical models to better understand the impact of dataset construction locality.
2 Related Works
Explainability and Locality Our work is broadly situated within the eXplainable Artificial Intelligence (XAI) field, which aims to elucidate the reasons behind black-box model's predictions. Developing XAI methods is difficult because of the variety of desiderata for XAI methods (Rudin et al. 2022), including trustworthiness (Shen 2022) and accuracy (Lipton, 2018). While a variety XAI methods are used in practice, such as saliency maps (Bachrens et al. 2010) and SHAP [undbbegg Lee 2017). prior work has demonstrated the fragility of those methods by analyzing the patterns these methods learn (Adebayo et al., 2018). Our work follows a similar perspective by studying the fragility of concept- based explanations through the lens of locality.
Concept-based Explainability Our investigation into locality touches upon various other phenomena in concept-based models. -oneept-based models use human- manannnndersttddable concepts to explain machine 2018). Methods vary from fully concept-supervised (Koh et al. 2020
learning model's prediction (Kim et al.
Chen et al. 2020 Yuksekgonul et al. 2022
Kim et al. 2023 where one has access to task and concept
labels, to concopt-unsupervise [Ghorbani
2019
Ych al. 2020 Oikarinen 2023 Pham et al.
We include our code and dataset. construction details here: httes//gtttt...ccmaaaaaat/////o. Concepts
2

---

Published in Transactions on Machine Learning Research (06/2025)
2024 where only task labels are available. In this paper, we explore CBMs (ooh al. 2020 due to their ubiquity as building blocks for state-of-the-art methods (Espinosa Zarlenga et al. 2022 Yuksekgonul et al.
2022
Kim et al., 2023 Oikarinen et al. 2023 Espinosa Zarlenga et al. 2023c).
Prior work in concept-based explainability has shown that concept- based models are prone to " "concept lenkage", where label-specific information undesirably encoded into concept predictions (Mahinpei et al.
2021
Havasi al., 2022). Follow-up works studied how leakage may be (1) detected at inference (Marconato
ct al.
2022), (2) measured at test-time (Espinosa Zarlenga et al., 2023a Marconato et al. 2024), (3) affected
by inter-concept correlations (Heidemann al. 2023), and (4) exploited for adversarial attacks (Sinha
ct al. 2023). Our work builds on these by studying the related but separate phenomena of locality, which captures the robustness of concept predictors to perturbations. We build on similar work on spurious concept
correlations (Margeloiu et al. 2021 Furby et al. 2023 2024 by introducing metrics to rigorously understand the origin of this phenomenon in different architectural choices. Our work additionally builds on literature based models by analyzing how incorrectly learning such relationships can decrease the robustness of concept predictors (Raman et. al. 2024). Finally, related to our work is prior work which develops metrics for understanding concepts (Espinosa Zarlenga et al., 2023a Marconato et al. 2024); however, in contrast to these works, we focus on quantifying the ability of concept predictors to rely on the relevant sets of features, rather than capturing whether they encode impurities or leakages. We include a further comparison of our work with other metrics in Appendix B.7
studying inter-concept relationships in
Spurious Correlations and Locality Our work investigates whether concept predictors rely only on rel- evant, features while ignoring irrelevant features. Similar to our emphasis on whether concept-based models rely on irrelevant features, prior work has investigated whether deep learning models exploit spurious corre- lations (Geirhos al. 2020 Ye ct al. 2024). Previous work in spurious correlations investigated how factors such as model size (Sagawa al. 2020), loss function (Levy et al. 2020). and data augmentation (Srivastava) et al. 2020) impact spurious correlations. In this work, similarly aim to systematically understand which factors impact localities in concept-based models. However, differentiate ourselves from prior studies of spurious correlations because we (a) demonstrate that locality may fail to hold even in simple situations, and (b) study how a failure of locality can lead to the breakdown interpretability for popular concept-based methods.
3
Why do we need locality?
We first formally define the problem of concept-based learning, then explain why we can use locality to understand the relationship between model predictions and model reasoning.
3.1 Defining Sancept-Based Learning
Concept-based learning is a supervised learning task where, given a set of input features, models produce a label prediction and a corresponding concept-based explanation. In concept-based learning we are given a set of features X (x(+) E Th"" " task labels Y y== (2 E (1,.... T)] and concepts C = {c(() e {0, 13 J==1- Concepts are humman-undesstnndalle attributes distilled from the raw input features x. For example, if is image, then concept could correspond to the attribute "Red Fruit". Each c( is 0-1 vector of length k, so that (0) indicates the presence of the j-th concept in x()
We can use concept-based learning construct interpretable architectures. A popular example of concept- based architectures is concept bottleneck models (CBMs) (Koh et al., 2020). CBMs are two-stage ar- chitecture consisting of a concept predictor g + é and a subsequent label predictor f è + g. The concept predictor maps an input x to a set of interpretable concepts é, and then the label predictor uses these interpretable concepts to predict a label y. During inference, for any input x, a CBM predicts both a set of concepts = gxx and a label ÿ = f(xxx)..
The explainability of concept-based models arises from the concept predictions è. Typically, the label predictor f shallow neural network (e.g., linear layer), mispredicted labels be traced back
3

---

Published in Transactions on Machine Learning Research (06/2025)
to concept prediction issues. For example, if an image is incorrectly predicted to be "apple", then the incorrect prediction can be traced back to an incorrect prediction for the "Red Fruit" concept.
3.2 Locality and Concept-Based Explainability
To understand the assumptions underpinning concept-based models, we first introduce the idea of locality, We define locality as a dataset property where concepts can be fully predicted using some subset of features. For any concept and data point combination, we define its local region as a minimal set of features neoded to fully predict the concept. Formally, for each concept j and data point x(0, we define its local region as (x(x() S [1...m), where m is the number of features in x(9, This corresponds to minimum set of features that can completely predict the corresponding concept value. For example, the "Red Fruit" concept can be fully predicted using all the pixels from the foreground and none from the background. In Figure Db.
we highlight examples of local regions for different concepts.
In this work, we claim that concept-based models should make concept predictions based only on the features in a concept's local region. We believe that concept- based models can assist with interpretability, but only when the underlying concept predictors respect localities. When concept-based models fail respect localities, they rely on features outside a concept's local region to make predictions. Because features outside of concept's 's local region have no impact on the true presence of a concept, concept-based models may rely spurious correlations to predict concepts. Concept prediction based spurious correlations jeopardizes the use of concepts as an explainability mechanism because the presence of a concept can no longer reliably provide insight into a model's reasoning. For example, if the "Red Fruit" concept is predicted using irrelevant elements such as the presence of a "Fruit Bowl" rather than the presence of red fruit, then we cannot explain predictions based on the "Red Fruit" concept. A better understanding of localities allows for a better understanding of the trustworthiness of concept- based models.
We answer some common misconceptions about locality:
Misconception #1: "Respecting localities and avoiding spurious correlations are the same problem" Spurious correlations and respecting localities are concerned with learning salient patterns. However, the motivations behind each are different; respecting localities concerned with the explainability of concept-based models, while spurious correlations are concerned with the generalizability of models. Moreover, as we demonstrate in Section 5.1 even in datasets without shortcuts, and therefore, no motivations to learn spurious correlations, concept-based based models can still fail to respect localitics.
2. Misconception #2: "Global concepts negate locality" If a concept is global and not confined to a local subset of features, then the local region is all the features; (x(()) = [m]. ). In this scenario, all concept predictors respect localities because no features exist outside a concept's local region. In this work, we do not focus on such global concepts because they do not yield much insight into the nature of locality.
3. Misconception #3: "Not respecting localities is a benefit, not drawback of concept- based models" - While leveraging correlations between features outside of the local region and the presence of a concept could lead to spurious yet accurate concept predictors, they endanger the separation between concepts and labels the concept-based model. Combining label and concept predictions should reflect the presence of a concept rather than correlations. Otherwise, generating trustworthy explanations from concept predictions becomes infeasible and concept predictors become inherently less robust to changes to out-of-locality features.
4. Misconception #4: "Concept leakage and locality refer to the same phenomenon" Concept leakage refers to the phenomenon where information about downstream labels or overall data distribution are "leaked" into concept predictions, which can reduce the purity of concept predictions (Mahinpei et al.) 2021). While locality is also concerned with concept predictors, its focus on the robustness of concept predictors to feature perturbations outside of a concept's local region. Locality assesses whether concept predictions truly reflect model's reasoning about a concept., while leakage assess whether downstream labels leak into concept predictions.
4

---

Published in Transactions on Machine Learning Research (06/2025)
4 Quantifying Locality
We assess the reliability of concept-based models by analyzing whether these models respect localities in a dataset. As noted in Section 3 concept-based models failing to respect localities jeopardizes the assumption that concept predictions are indicators of a model's reasoning behind a prediction. For example, if the prediction for the concept "Red Fruit" is not based on predicting red fruit but rather some correlated concept, then it becomes difficult to explain model predictions based on concept predictions. We study the extent to which models respect localities understand whether we can trust the explanations and concept predictions from concept-based models.
We construct three metrics to quantify locality because each metric captures notion of perturbation that concept-based models must be invariant to. When concept-based models respect localities, they are invariant to perturbations outside of a concept's local region. Each of our metrics for locality corresponds to a different notion of perturbation; locality leakage corresponds to adversarial perturbations, locality intervention cor- responds to in-distribution perturbations, and locality masking corresponds to out-of-distribution masking perturbations. We present pictorial example of each metric in Figure
4.1 Locality Leakage
The locality leakage metric assesses a concept-based model's invariance to adversarial distortion outside of a concept's local region. Prior work has demonstrated that adversarial perturbations can impact a model's predictions [Goodfellow al., 2014). However, it is unknown whether adversarial perturbations outside of a concept's local region impact their predictions, especially in scenarios without spurious correlations between features.
To formalize the impact of adversarial perturbations on concept's predictions, we define the locality leakage metric as follows:
&
hi g, 1 =
max x' ER
19(x'), - )yx
(Locality Leakage)
(x x" x(()
Here, x' is adversarial data point, which perturbs features outside of the local region for concept. j. (x)x))), For this data point, measure the degree to which can alter the prediction of concept j (i.e., 19(x)) -xx())) simply through adversarial perturbation of features outside of a concept's local region (i.c., Va E (,) x,, = xx").. Taken together, the locality leakage metric averages across all concepts j, the maximum perturbation in concept prediction,
-xx))))(()))) by selecting an alternate data point x' that agrees the local region Ly(x()). That is, x' leverages features outside of its local region to modify the prediction of concept j. A large locality leakage implies that modifications to irrelevant features significantly impact a concept's prediction. In Appendix B.1 we consider an alternative formulation of the locality leakage metric which constrains the magnitude of adversarial perturbations and demonstrates similar results up to certain levels of constraints.
4.2 Locality Intervention
We introduce the locality intervention metric to assess concept-based model's invariance to in-distribution shifts. The locality leakage metric potentially suffers from perturbations leading to significantly out-of- distribution features never seen in practice. To address this, we perturb features outside the local region while ensuring that the resulting feature vector x' is still in distribution and the underlying ground-truth concept unchanged. To perturb a data point x()), we find an alternative data point x' x() X, so that concept j is unaltered, Cj
(8) = S while the prediction for concept j is maximally altered, 19(x)) (1) (x)()) Because g(x!)), E [0. 1] rather than being (0, 1}. the locality intervention metric allows us quantify the magnitude of impact that out-of-local region perturbations in more fine-grained manner compared to
5

---

Published in Transactions on Machine Learning Research (06/2025)
concept prediction accuracy. Formally, we define the locality intervention metric follows:
hi g. (x Ji-1 (c())
max 10) ex =
)g(x )5x
(Locality Intervention)
Here, we take the average perturbation lx(x(1), g( (x) across concepts j. by seeing how much an alternate data point x() with the same groudd-ruuh concept value disagrees in concept prediction. In essence, this measures a concept predictors susceptibility to in-distribution perturbations across other concepts outside of concept j's local region.
4.3 Locality Masking
We introduce a third metric that assesses a concept-based model's invariance to a masking perturbation outside a concept's local region. The locality masking metric resides between the locality leakage and intervention metrics in terms of perturbation severity. The locality leakage metric adversarially perturbs non-local region features, which can lead to out-of-distribution data points, while the locality intervention metric ensures that perturbations are in distribution, but requires large dataset C to find in-distribution perturbations. To get around the limitations of locality leakage and intervention, we assess the impact masking concept j' upon the prediction for concept j. By relying on masking to perturb features, our resulting features are not significantly out-of-distributioo especially because we only mask a fraction of the total features. All we require access to the localities for each concept, through, for example, bounding boxers for concepts.
We assess two components for the locality masking metric: the locality relevant masking metric and the locality irrelevant masking metric. For concept. j, the locality relevant masking metric computes the impact of masking relevant features (x(x))), while the locality irrelevant masking metric computes the impact of masking features relevant to an unrelated concept j', (x))) Let. M(x(0,)) ,) be the replacement of all features in (x))) with a constant mask value n, Additionally, let S(x( ,) {i' (Ly))))))))) which corresponds
0}, the set of concepts such that the local region of j' is non-overlapping with the local region for j. This way, masking concept j' should not impact the prediction for concept j. Then, we define the locality relevant masking metric
A Σ
hrm g. px M
[a(M((e -00x
(Relevant Masking)
k
Similarly, we define the locality irrelevant masking metric as:
him g. k. (1) 11=1. M
1x(M(x()) -x)x(")))) (Irrelevant Masking) ((x())) ,) ((
Σ
Here, M (j) defines a masking function that masks concept j data point ()) We analyze the impact of masking upon the concept prediction, both when masking concept j (where we compute (x())))))) g(x(")) Iy),, and when masking concept j' (where we compute g (x(1), (j))) g(x()), We note that we divide by 5(x(9),) .j) to account for the number of masked concepts j'.
Concept-based models that respect localities should be unimpacted by masking outside of the local region and only impacted by perturbations to features within their local region. That is, such models should have a high locality relevant masking metric and a low locality irrelevant masking metric.
5 Experiments
We construct experiments assess whether CBMs respect localitics in both (a) controlled testbed where we can understand the impact of various factors and (b) real-world datasets where we can understand the performance of concept-based models in practice.

---

Published in Transactions on Machine Learning Research (06/2025)
Synthetic
Synthetic
COCO
Figure 2: We plot examples from our 2-object and 4-object synthetic datasets, along with examples from CUB and COCO. For the synthetic datasets, concepts refer to the positions of each shape. For CUB, concepts are the colours and shapes of different parts of the bird, and for COCO, concepts are the presence of objects such as bowls and spoons.
5.1 Experimental Setup
Synthetic Datasets We construct a synthetic dataset to understand locality in a controlled euvironment (sec Figure 2 Our synthetic datasot consists of concepts and k object locations. Each location consists of triangle or a square, corresponding to two concepts: "Is Square" and "Is Triangle", for total of k concepts. For example, in the synthetic 2-object dataset, have four concepts and two object locations. We construct datasets with 1, 2, 4, and 8 objects. For the 1- and 2-object datasets, construct 256 training examples, while for the 4- and 8-object datasets we construct 1024 training examples due to the increased number of concept combinations. We construct our dataset using hetic objects so concepts for different object locations are independent; the presence of a triangle on the left side gives no information on the shape of the right. This construction implicitly should discourage models from picking up spurious correlations due the absence of shortcuts.
2
Non-Synthetic Datasets To understand locality in more realistic scenarios, we evaluate concept- basod models on two non-synthetic datasets: Caltech- UCSD Birds (CUB) (Wah al. 2011] and Common Objects in Context (COCO) [Lnn et al. 2014 We select these datasets because both ubiquitous within concept- based explainability [Espinosa Zarlenga et al. 2022 Choi et al. 2023). The CUB dataset consists of images annotated with a bird species (the label) and properties, such as wing colour and beak length. This results in 112 attributes, selected by Koh et al. (2020), which can be partitioned into 28 concept groups based on the body part (c.g. wing). The COCO datasct consists of scenes with objects. The original COCO dataset consists of 100 objects, and we randomly downsample this to 10 objects so that COCO and CUB have comparable dataset sizes. Concepts refer to the presence of objects and the task label is function constructed from this concept that indicates whether any object in some fixed subset is present. We construct lask in this manner following prior work (Espinosa Zarlenga et al. 2022
Other Experimental Details We vary architecture and training procedure across experiments and detail these choices for each experiment individually. We run all experiments for three seeds on an NVIDIA TITAN Xp GPU, with the total number of hours ranging from 100 to 200. We run experiments on a Debian Linux platform with CPUs and 1 GPU. We train our synthetic modols for 50 epochs, our COCO model for 25 epochs, and our CUB model for 100 epochs, selecting each based on the time at which accuracy and loss reach convergence. For CUB and COCO we select learning rate of 0.005, while for the synthetic experiments, select 0.05, selecting this through manual inspection of model performance.
5.2 Impact of Architecture
We analyze how architect choice impacts whether concept-based models respect locality. We perform our experiments across the different synthetic object datasets (see Section 5.2); we do so because we can quickly evaluate architectures without tuning parameters due to the simplicity of the datasct. Additionally, even smaller models can learn the synthetic dataset, while the same not true for CUB and COCO. Here, we first evaluate models based on locality leakage metric. To instantiate locality leakage, we employ projected

---

Published in Transactions on Machine Learning Research (06/2025)
1.0
1.0
Localiyy Leakage
0.8
0.8
0.8
Metric
Metric
0.6-
0.6
0.6
S-bbject catasel
0.4
0.4
2-object cataset
Accuracy Leakage
0.4
Accuracy Leakage
object cataset
0.2
3
5
Layers
Model Depth
Model Depth
(a) Model Depth
(b) Fixed Parameters
(c) Fixed Receptive Field
Figure 3: We find that deep concept predictors lead to higher locality leakage, implying that deep concept predictors leverage features beyond concept's local region. Such trend occurs even when accounting for the number of parameters (b) or the receptive field size(c).
gradient descent to find the data point x' that maximizes lg(x'); g(x("))) for each concept j. Projected gradient descent allows us to quickly find x', while ensuring that concepts within the local region ()(x!))
remain constant. In Section 5.3
we consider our locality intervention and masking metrics.
Prior work in spurious correlations by Sagawa et all (2020) demonstrates that larger models learn more spurious correlations, so we naturally investigate whether such a pattern holds for localities. We construct a set of concept predictors which vary in depth from to 7 layers, where deeper models have more parameters. We train all models for 50 epochs; at 50 epochs, all models achieve perfect task and concept accuracy.
We compare the locality leakage for our models across varying model depths and find in Figure D that deeper models tend to have higher locality leakages. Across all models and dataset combinations, the locality leakage is at least 0.4, implying that on average, concept predictions can be changed by at least 40% through adversarial perturbations. We also find that increasing model depth leads to higher locality leakages, with 7-layer models having locality leakages over 0.9 for all synthetic datasets. This mimics the trend found in the spurious correlation literature where deeper models tend to produce more spurious correlations (Sagawa et al. 2020 Comparing across synthetic datasets, with the highest leakage for all models except for 5-layer models. Our experiments reveal that deeper models, which also correspond to models with more parameters, tend to respect locality less.
find that the 8-object dataset produces models
While deeper models lead to higher levels of locality leakage, is unclear whether this is due to (a) the concept predictors being deeper, (b) the concept predictors having more parameters, or (c) the concept predictors having a larger receptive field. To distinguish between these scenarios, we construct concept predictors that hold either (a) the number of parameters or (b) the receptive field size constant. We again vary the model depth wetween 3 and 7 layers for both settings and compare both the concept accuracy and the locality Ieakage metric for these models.
We find that depth, rather than the number of parameters or receptive field size after all convolutions, is primarily responsible for the increase locality leakage. In Figure 3 and Figure 3b, we find that even when holding the number of parameters or receptive field size constant, depth-7 models still cxhibit a locality leakage of 1.0. Between depth 4 and depth we find that the locality leakage is non-decreasing, increasing depth either increases locality leakage keeps it at 1.0. Our results imply that overly deep models could be partially responsible for concept-based models failing to respect localities.
5.3 Dataset Choice
Locality Intervention To understand how concept-based models perform beyond synthetic datasets, we assess robustness to in-distribution (through locality intervention) and masking) perturbations real-world datasets. For in-distribution perturbations, we conjecture that train- ing diversity plays a large role because more diverse training datasets allow concept predictors to better differentiate between concepts. To quantify this, we analyze the impact of train-time concept combinations
(through locality
8

---

Published in Transactions on Machine Learning Research (06/2025)
30%
Locality Intervention
Locality Intervention
CUB COCO
50%
20%
25%
10%
0%
0%
0%
25% 50% 75% 100%
0% 25% 50% 75% 100% Concept Combinations
% Concept Combinations
(a) Synthetic
(b) CUB and COCO
Figure 4: We find that datasets with a larger number of concept. combinations can reduce the locality intervention metric synthetic (a) and real-world scenarios (b). Therefore, training with a diverse set of concept combinations can help improve the in-distribution robustness of concept predictors.
15%
Relevant Mask
Locality Masking
Irrelevant Mask
10%
5%
0%
CUB
COCO
Dataset
(a) Locality Masking
(b) Masking Example
Figure 5: To assess robustness to out-of-distribution perturbations, we compare the relevant and irrelevant masks for the CUB and COCO datasets. For CUB, masking relevant or irrelevant concepts similarly impacts predictions, indicating that concept predictors leverage features beyond a concept's local region to make prodictions. For example, masking the tail of the bird (b) docs not lead to changes in prediction for the concept of "Tail Colour."
test-time locality intervention. We compare this trend for three datascts: the 8-object synthetic dataset, CUB, and COCO. For each dataset vary the fraction of concept combinations from 25% to 100%. For example, 25% corresponds to sampling 25% of the values for e(t) seen in the training dataset and filtering the training dataset to only use tuples (x(1) c ()) with corresponding c()) 100% corresponds to using the original training dataset. We ensure that perturbations do not impact concept labels by creating a set of data points for each concept. combination c. By doing so, we can always find data point where concept held constant, while other concepts j' are variod.
We analyze the relationship between the diversity of the training concept and the locality intervention in Figure 4 and find that more combinations of concepts decrease the locality intervention. In the synthetic dataset, observing 50% of concept combinations reduces the locality intervention to 0.06. We see similar inflexion points for CUB and COCO as using 50% of concept combinations reduces locality intervention to 0.14. For all three data sets, increasing the diversity of the training set reduces locality intervention, demonstrating the importance of constructing data sets with large number of combinations of concepts. Our results reflect that dataset-related changes can impact in-distribution perturbations, while architectural changes impact out-of-distrrbutiin perturbations.

---

Published in Transactions on Machine Learning Research (06/2025)
Num O ects
1.0
1.0
adversarial
Locality Leakage
1.0
control
0.9
0.9
0.8
Metric
0.8
0.8
Locality
0.7
0.7
0.6
0.6
Accuracy Leakage
0.6
Acouracy Leakage
0.4
0.5
0.5
10
20
30
40
50
0.0004
0.004
0.04
Epochs
Weight Decay
Layers
(a) Epochs
(b) &g Regularization
(c) Adversarial Training
Figure 6: We find that training time modifications such as (a) carly stopping, (b) lg regularization, and (c) adversarial training cannot significantly reduce locality leakage without also impacting accuracy. Applying adversarial training leads to locality leakage values between 0.6 and 0.9 across all concept predictor depths, all concept predictors exhibit some locality leakage.
Locality Masking Next, we investigate the performance of concept- based models with real-world datasets through the locality masking metric. Locality masking captures more realistic out-of-distribution perturba- tions than locality leakage because the perturbations are not as dramatic, making it more appropriate for use with real-world datasets. For CUB, we have access to the centroid each concept, so we mask all concepts within radius of 0.31, where is the image width (we evaluate the impact of varying in Appendix [B.6). To avoid masking unrclated features, we mask only fcatures closer to the centre of the masked concopt than the centre of any other concept. For COCO select such an for CUB because it masks each concept without covering adjacent concepts; experiments with other values of reveal similar trends. We fill masked values with zero (black) but confirm our results using mean-masks in Appendix B.4
provided the bounding box, we use that instead. We
In Figure we assess locality masking for CUB and COCO. For CUB, we find only a 4.7% difference between relevant and irrelevant masks, indicating that concept predictors leverage features beyond a concept's local region to make predictions. In both COCO and CUB, the locality masking metric is most 15%; intuitively, even when fully masking the concept, concept predietors can make predictions with only 15% loss in accuracy. For example, masking the bird's tail in Figure 5p does not hinder tail colour prediction. This occurs because saacept-based models infer a tail's colour from its body or head. Such behaviour is undesirable because concept predictors become detached from the concepts themselves, so predictions for the tail of a bird reflect propertics of the body and head.
5.4 Training Modifications
We analyze the impact of modifications to training procedures, including the number of epochs and loss function, upon the locality leakage metric. Our results in Section 5.2 demonstrated that the selection of architecture impacted locality leakage, so it would be natural to understand whether training modifications also impact locality leakage. Prior work dernonstrated that early stopping, which occurs when training stops after the loss converges or increases for a fixed number of epochs, can prevent overfitting (Caruana et al. 2000). We investigate whether such phenomena can impact locality, so we analyze the impact of training iterations upon locality. We vary the number of epochs from 5 to 50 and analyze the impact upon accuracy and leakage.
In Figure 0 we find that increasing the number of epochs leads to increased accuracy while also increasing locality leakage. When we train models for 30 or more epochs, they achieve perfect concept accuracy but also achieve a locality leakage of at. least 0.9. Moreover, there exists no training setting where the model achieves perfect accuracy while having a small locality leakage.
We additionally investigate whether modifications to the training loss can improve the locality properties of concept-based models. We consider two types of training modifications: la regularization (weight decay)
10

---

Published in Transactions on Machine Learning Research (06/2025)
Table 1: Recent iterations of CBMs such as CEM and ProbCBM can reduce locality leakage for synthetic and real-world datasets due to their increased representational capacity. However, VLM-based approaches such as label-free CBMs fail to differentiate between relevant and irrelevant masking, showing that this model fails to address locality. We refrain from evaluating locality masking on the synthetic datasct because it would be redundant with the locality intervention metric.
Datasct
CBM Variant
Accuracy
Leakage
Relevant Masking
Irrelevant Masking
Synthetic
CBM Baseline (7-Layer)
100.0% = 0.0%
1,000 + 0.000
0.500 = 0.000
0.052 + 0.037
CEM
100.0% 0.0%
0.724 1 0.028
0.380 1 0.002
0.355 1 0.012
ProbCBM
100.0% 0.0%
0.635
0.014
0.384 0.000
0.329 + 0.002
CUB
CBM Baseline (7-Layer)
67.1% + 0.4%
0.113 + 0.002
0.107 + 0.002
Label-Free CBM
74.6% = 0.0%
0.053 0.001
0.052 + 0.001
CEM
68.1% + 1.0%
0.035 + 0.001
0.034 + 0.001
ProbCBM
68.2% = 0.8%
0.008 + 0.001
0.007 + 0.000
coCO
CBM Baseline (7-Layer)
83.9% = 0.3%
0.144 = 0.014
0.095 + 0.004
CEM
83.6% 1 1.1%
0.024 T 0.009
0.020 1 0.012
ProbCBM
83.3% = 0.4%
0.024 0.003
0.008 + 0.001
and adversarial training. We select & rogularization because prior work showed that it can potentially reduce spurious correlations (Sagawa et al.) 2019), while we select adversarial training because adversarially trained models are potentially more robust (Jin et al. 2020).. For f regularization, we vary the weight decay parameter in {0.0004, 0.004, 0.04] and evaluate on the 2-object synthetic dataset with the 7-layer model. For adversarial training, we incorporate an adversarial loss and analyze the impact across different model depths in the 2-object synthetic dataset.
In Figure G and Figure Gp, we find that neither l regularisation nor adversarial training can reduce locality leakage beyond 0.6 without impacting accuracy. While large amounts of & regularization can reduce the locality leakage down to 0.5, we find that this also leads reduction in accuracy. When concept predictors have few layers, adversarial training leads to more locality Icakage, while even for 6 and 7-layer concept predictors, the locality leakage is at least 0.8. This demonstrates that adversarial training fails to enforce that concept-based models respect localities.
5.5
CBM Variants
Newer CBM variants have alleviated issues ranging from representational power (Espinosa Zarlenga et al. 2022 to uncertainty (Kim et al. 2023), so we investigate the impact of these advances on our locality Icakage metric. We evaluated the performance of three CBM variants according to accuracy and leakage (on the synthetic dataset) or masking (on the CUB and COCO datasets):
1. Concept Embedding Models address the low representational power of CBMs by allowing each concept to be represented through a vector (Espinosa Zarlenga et al., 2022).
2. Probabilistic concept bottleneck models (ProbCBMs) enable more accurate uncertainty estimates by modelling concept embeddings as normally distributed vectors (Kim et al. 2023
3. Label Free CBMs incorporate large language models CBMs can be trained without needing ground-truth labels for concepts. It does so by generating concept set for a given list of labels and using the embeddings of this concept set to train label predictors (Oikarinen et al. 2023). We evaluate label-free CBMs only on the CUB dataset because they rely on human-understandaal descriptions for each label, which are not present for the synthetic or COCO datasets.
We compare metrics across these CBM variants in Table I We find that CEMs and ProbCBMs reduce the leakage the synthotic dataset, although leakage 0.6 still indicates that concept predictions can be modified by 60% of the original value. Moreover, ProbCBMs have a higher gap between relevant and irrelevant masking compared to the baseline CBM in the COCO dataset. Meanwhile, in the CUB dataset.
11

---

Published in Transactions on Machine Learning Research (06/2025)
label-free CBMs only tighten the gap between relevant and irrelevant masking by factor of 6. CEMs and ProbCBMs represent potential panacea to address locality because they better model the embedding space for each concept, which assists concept predictors in differentiating between concepts. For example, ProbCBMs allow concept prodictors to incorporate uncertainty when making concept predictions. With label-free CBMs, we find that the difference belween relevant. and irrelevant masking almost nonexistent, implying that concept predictors leverage features beyond the local region. Our results with other variants of CBMs demonstrate that better-trained concept predictors, with better representational capacities, have the potential to reduce locality-related issues in some situations, though locality-related issues still persist with such models.
6 Theoretical Insights into Locality
To understand whether our empirical takeaways from Section | are an artefact of our experimental setup, we investigate the fundamental relationships between concept-based models and locality. We aim to formalize our study of locality intervention by analyzing how dataset construction impacts concept predictors.
Datasets with many concepts and few data points make it difficult to learn good concept predictors. For example, if dataset only contains images of cherries and no images of strawberries, then the "Red Fruit" concept would be perfectly correlated with the "Circular Fruit" concept. This makes it difficult or impossible to distinguish these two concepts, which means that the underlying concept predictors do not correspond to the true concept. We formalize this idea through the theorem below:
Theorem 6.1. Suppose out of k total concepts, we can predict m, {n: (2: 72.-- ... 7m C (1,2... kj, with perfect accuracy, PIC, * c,] =V E ?...,...... Let Mjaar = Plc, = g/o; = r] be a correlation matris, where g.r E 10, For any concept whose value we do not know (j & (.. 72... %m))) ) if for s different triplets (g, with E (mI "rn:?2...7inl we have (a) Mjqúr 2
7mm.
a and (b) Pfc, = r] 2 3, then there exists a concept predictor that achieves error e = FlC; + cj] S a (1 3)", using only information about the concepts (7,72... 7mm: g(x), w(g(x),, 9 (xx)) (x) rm)) for the concept predictor g and for some function w.
We present some intuition and interpretation for our Theorem. In the theorem, 71, 72,....................... ........... 7m E [k] represent the m concepts that are perfectly known and a. [0, 1] represents the probability that concept j [k] has value of [0, 1], given that concept i E [k] has value of r [0, 1]. If concept j is correlated sulficiently many other concepts i then we can train a concept predictor with low error e. The error rate, is dictated by two factors: a, which represents the correlation rate between concept i and concept j (that is, Pfc, = c/ = r] 2 1 - a); and 3, which represents the probability that concept i is r (that is P(c, r]] 2 3). Our bound on e is strict when s is large, as in that scenario, (1 - 3) is small, while the a term becomes tight. Our bound can additionally be modified even if concepts in (71,72,.... 72...., 7m) are known up to error 5. In that scenario, our bound then becomes e a 8 3) so we only pay linear cost in the error rate.
Theorem 6. highlights the interaction between architecture, dataset, and locality. Intuitively, the theorem implies there exists concept predictors with low error that only leverage a subset of concepts for prediction. These concept predictors make predictions irrespective of the true presence of a concept. For example, this implies there exists low-error concept. predictors for "Red Fruit" " and "Circular Fruit" that only leverage the "Red Fruit" concept, so the concept predictor for "Circular Fruit" " is made irrelevant of whether the image contains a circular fruit. We prove such a thcorem by explicitly constructing the low error concept prodictor and analyzing this predictor. Here, a corresponds to the level of independence bet concepts, low a corresponds to situations where concepts are highly correlated, and is the frequency at which concept is r. In essence, datasets with highly correlated concepts can lead to concept-bused models failing to respect localities, as they fail to distinguish between concepts. Such results mirror the trends in Section 5 where increasingly diverse datasets can improve robustness to in-distribution perturbations.
7 Discussion
Avoiding Locality in Practice While vanilla CBM models generally fail to respect localities, we describe architectural and dataset choices that can help alleviate this issue. Both our empirical (Section 5) and
12

---

Published in Transactions on Machine Learning Research (06/2025)
theoretical (Section 6 results demonstrate that larger and more diverse datasets can improve the adherence of concept-based models to localities found in the dataset. Concept-based models require datasets with different combinations of concepts so that concept predictors can distinguish between concepts. Moreover, our empirical results demonstrate that smaller models tend to exhibit less locality; this is partially because these models are less prone to overfitting the spurious correlations found datasets. Finally, models with improved representational capacity can better distinguish between concepts, as shown through the improved performance of ProbCBM and CEM models. Taken together, we recommend the following: (a) practitioners ought to be cautious about interpreting concept predictions as the arbiter of truth, (b) when possible, the datasets used by practitioners should be large and contain diverse concept combinations so concept predictors can distinguish between concepts, and (c) practitioners should use smaller models and models with greater representational capacily (e.g., CEMs or ProbCBMs) when possible.
Limitations and Future Work Our work is an initial inquiry into the relationship between locality and concept-based models which naturally leads to future work. We look at two real-world datasets for concept- based models; we select these datasets because of their ubiquity and use across concept-based learning tasks. Future work could build on this by considering more datasets including across other modalities, such as those for tabular data (Espinosa Zarlenga et al. 2023c). Additionally, future work could consider the application of our metrics to both the regression and generation settings. We can extend our metrics to the regression setting in straightforward manner by modifying the label predictor to predict continuous values, while extension to the generative setting would require analogous metrics modifications to concepts only modify relevant generated features. Our work considers set of architectural and training-related modifications to assess their impact on locality. Our study is not exhaustive; there are always new training techniques and modifications to architectures that could potentially alleviate issues related to locality. However, we aimed construct a thorough study that studied a variety of modifications to better understand locality and its relationship to the assumptions made by concept-based models.
8 Conclusion
Concept-based models rely on the assumption that concept predictions can give insight into a model's rea- soning. We test whether this assumption holds by analyzing whether concept-based models respect. locality. We construct three metrics quantify locality and apply these across architectural and training configu- rations. We find that lack of respect for locality is pervasive problem that is partially alleviated through diverse training data and smaller architectures. We complement our empirical results with a theorelical analysis that demonstrates how concept correlations can lead to concept-based models which fail to respect locality. Our study sheds light into the assumptions underlying concept-based models and gives insight into how concept- based models can be constructed with trustworthy explanations.
Broader Impacts
Our paper analyses the robustness of concept predictors, which underpin the explainability of CBMs. Un- derstanding and improving explainability allows for safer machine learning model deployment, and our work aims to improve such deployments through bedteroumderstood interpretability algorithms.
Acknowledgements
The authors would like to thank Katic Collins, Andrei Margeloiu, Tennison Liu, and Xiangjian Jiang for their suggestions and discussions on the paper. Additionally, the authors would like to thank Zohreh Shams for their comments on the paper, and from the NeurIPS XAI Workshop reviewers for their insightful comments. During the time of this work, NR was supported by a Churchill Scholarship, and NR additionally acknowl- odges support from the NSF GRFP Fellowship. MEZ acknowiedges support from the Gates Cambridge Trust via Gates Cambridge Scholarship. JH thanks support from Cambridge Trust Scholarship.
13

---

Published in Transactions on Machine Learning Research (06/2025)
References
Julius Adobayo, Justin Gilmer, Michael Muclly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in newral information processing systems, 31, 2018.
David Bachrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Glaus-Robert Müller. How to explain individual classification decisions. The Journal of Machine Learning Research, 11: 1803-1831, 2010.
Rich Caruana, Steve Lawrence, and C Giles. Overfitting in neural nets: Backpropagation, conjugate gradient, and early stopping. Advances in neural information processing systems, 13, 2000.
Kushal Chauhan, Rishabh Tiwari, Jan Freyberg, Pradeep Shenoy, and Krishnamurthy Dvijotham. Interac- tive concept bottleneck models. arXiv preprint arXiv:2212.07430 2022.
Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2(12):772-782, 2020.
Jihye Choi, Jayaram Raghuram, Ryan Feng, Jiefeng Chen, Somesh Jha, and Atul Prakash. Concept-based explanations for out-of-distribution detectors. Tn International Conference on Machine Learning, pp- 5817 5837. PMLR, 2023.
Matco Espinosa Zarlenga, Pictro Barbicro, Gabricle Ciravegna, Giuseppe Marra, Franccsco Giannini, Michelangelo Diligenti, Zohreh Shams, Frederic Precioso, Stefano Melacci, Adrian Weller, al. Con- cept embedding models: Beyond the accuracy-oxpaannabltty trade-off. Advances in Neural Information Processing Systems, 35:21400 21413, 2022.
Mateo Espinosa Zarlenga, Pictro Barbiero, Zohreh Shams, Dmitry Kazhdan, Umang Bhatt, Adrian Weller, and Mateja Jamnik. Towards Robust Metrics for Concept Representation Evaluation. Proceedings of the AAAT Conference on Artificial Intelligence, 37(10): June 2023a. doi: vn..///11..3..
26392. URL https://ojs aaai www.oodex...ioeooooooooooooeee.e index. phtp/AAA//arriille//ee///233
Matco Espinosa Zarlenga, Katherine M Collins, Krishnamurthy Dvijotham, Adrian Weller, Zohreh Shams, and Mateja Jamnik. Learning to receive help: Intervention-awan concept embedding models. arXiv preprint ar Xrxv:2330 16928, 2023b.
Mateo Espinosa Zarlenga, Zohreh Shams, Michael Edward Nelson, Been Kim, and Mateja Jamnik. Tabebm: Concept-based interpretable neural networks for tabular data. Transactions on Machine Learning Re- search, 2023c.
Jack Furby, Daniel Cunnington, Dave Braines, and Almn Preece. Towards a deeper understanding of concept bottleneck models through end-to-end explanation. arXiv preprint arXiv:2302.03572 2023.
Jack Furby, Daniel Cunnington, Dave Braincs, and Alun Preece. Can we constrain concept bottleneck models learn semantically meaningful input features? arXiv preprint arXiv:2402.000 000111 2024.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shorteut learning in deep neural networks. Nature Machine Intelli- gence, 2(11):665-673, 2020.
Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept-based expla- nations. Advances in Neural Information Processing Systems, 32, 2019.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint Xrxi::1412.66:5 2014.
Marton Havasi, Sonali Parbhoo, and Finale Doshi- Velez. Addressing leakage in concept bottleneck models.
In Advances in Newral Information Processing Systems, 2022.
14

---

Published in Transactions on Machine Learning Research (06/2025)
Lena Heidemann, Maureen Monnet, and Karsten Roscher. Concept correlation and its effects concept- based models. In Proceedings of the TEEE/CVE Winter Conference on Applications of Computer Vision, pp. 4780 4788, 2023.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. bert really robust? a strong bascline for natural language attack on text classification and entailment. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 8018 8018-8025, 2020.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Inter- pretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pp. 2668 2677. PMLR, 2018.
Eunji Kim, Dahuin Jung, Sangha Park, Siwon Kim, and Sungroh Yoon. Probabilistic concept bottleneck models. arXiv preprint orXim:2306.01574 2023.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning, pp. 5338-5348. PMLR, 2020.
Daniel Levy, Yair Carmon, John C Duchi, and Aaron Sidford. Large-scale methods for distributionally robust optimization. Advances in Neural Information Processing Systems, 33:8847-8860, 2020.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 18th European Conference, Zurich, Swilzerland, September 6-12, 2014, Proceedings, Part 13, pp. 740-755. Springer, 2014.
Zachary C Lipton. The mythos of model interprotability: In machine learning, the concept of interpretability is both important and slippery. Queue, 16(3):31-57, 2018.
Scolt M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
Anita Mahimpei, Justin Clark, Isaac Lage, Finale Doshi-Velez, and Weiwei Pan. Promises and pitfalls of black-box concept learning models. arxiv preprint arXiv:2106.133 13314, 2021.
Emanuele Marconato, Andrea Passcrini, and Stefano Teso. Glancenets: Interpretable, leak-proof concept- based models. Advances in Neural Information Processing Systems, 35:21212 21227, 2022.
Emanuele Marconato, Stefano Teso, Antonio Vergari, and Andrea Passerini. Not all neuro-symbolic con- cepts are created equal: Analysis and mitigation of reasoning shortcuts. Advances in Neural Information Processing Systems, 36, 2024.
Andrei Margeloit, Matthow Ashman, Umang Bhatt, Yanzhi Chen, Mateja Jamnik, and Adrian Weller. Do concept bottleneck models learn as intended? arXie preprint arXiv:2105.0442 2021.
Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and Tsui- Wei Weng. Label-free concept bottleneck models. arXiv preprint arXio:2304.611 2023.
Thang M Pham, Peijie Chen, Tin Nguyen, Seunghyun Yoon, Trung Bui, and Anh Totti Nguyen. Peeb: Part-based image classifiers with an explainable and editable language bottleneck. arXiv preprinl arXiv:2403.0540 2024.
Naveen Raman, Mateo Espinosa Zarlenga, and Mateja Jamnik. Understanding inter-concept relationships in concept-based models. arXie preprint urXiv:2405. 188.1777,,11 2024.
Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong. Interpretable machine learning: Fundamental principles and 10 grand challenges. Statistic Surveys, 16:1-80 85, 2022.
15

---

Published in Transactions on Machine Learning Research (06/2025)
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Perey Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arxiv preprinl wrXXi:1911.08791, 2019.
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overpa- ramcterization exacerbates spurious correlations. In International Conference Machine Learning, pp. 8346 8356. PMLR, 2020.
Max W Shen. Trust in ai: Interpretability is not necessary sufficient, while black-box interaction is necessary and sufficient. arXiv preprint arXio:2202. 05302, 2022.
Sungbin Shin, Yohan Jo, Sungsoo Ahn, and Namhoon Lee. A closer look at the intervention procedure of concept bottleneck models. arXiv preprint 2422:2200.11260, 2023.
Sanchit Sinha, Mengdi Huai, Jianhui Sun, and Aidong Zhang. Understanding and enhancing robustness of concept-based based models, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 15127 15135, 2023.
Megha Srivastava, Tatsunori Hashimoto, and Percy Liang. Robustness to spurious correlations via human annolations. In International Conference on Machine Learnöng, pp. 9109 9119. PMLR, 2020.
Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method, 2000.
Kush R Vashney. Trustworthy machine learning. Independently published, 2022.
C. Wah, S. Branson, Welinder, P. Perona, and S. Belongie. The caltech-uesd birds 200s-200020011 dataset. Tochnical Report CNS-TR-2011-001, California Institute of Technology, 2011.
Wenqian Ye, Guangtao Zheng, Xu Cao, Yunshong Ma, and Aidong Zhang. Spurious correlations in machine learning: A survey. arXiv preprial arXiv:2402. 12715, 2024.
Chih-Kuan Ych, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On completeness-awae concept-based explanations in deep neural networks. Advances in Neural Information Processing Systems, 33:20554 20565, 2020.
Mert Yuksekgomul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. orXiv preprint arXiv:2205.15.4 2022.
16

---

Published in Transactions on Machine Learning Research (06/2025)
1.0
Locality Leakage
0.8
0.6
0.4
0.2
0
12 Regularization Penalty
8
16
(a) Regularization Weightage
(b) Example Image
Figure 7: We introduce a regularization penalty to encourage perturbed images in locality leakage to remain similar to the original data point. While increasing the weight of this penalty decreases the locality leakage (a), we can find slightly perturbed examples that exhibit high locality lcakage (b).
A Architecture Selection Details
In Section 5 we construct concept predictors of various depths and widths to analyze the impact of concept predictor architecture upon leakage. Our models increase in width from 64, for the 3-depth models, to 512, for the 7-depth modcls. For models which hold the number of parameters constant, we construct model layers that shallower models are also wide; for example, the 3-layer model has widths 64, 64, and 32, while the 7-layer model has widths 64, 64, 64, 32, 28, 24, and 20. We similarly construct models to hold the receptive field size constant; details can be found in our code.
B Additional Ablation Results
B.1 Constrained Locality Leakage
Because our locality leakage metric can lead to out-of-distributio inputs, we consider a variant which finds a perturbed datapoint x' while minimizing the distance from the original datapoint x. We incorporate this as a penalty the loss function, weighted by hyperparameter X. Here, balancos between significantly perturbed data points, and those with less perturbations but still achieving high locality leakage (Figure 7p). As example Figure 7p is data point with less perturbation but with locality leakage near 1. More generally, we find that even when locality leakage outputs are highly constrained (large A), there still exists perturbed data points that achieve high locality leakage.
B.2 MLP Concept Predictors
To get more insight into the impact of concept predictor width and depth on locality leakage, we use multi- layer perceptrons (MLPs) concept predictors. We vary the depth and width of MLPs, varying the depth from to 3 and width from 5 to 15. In Table 2 we see that, when holding the depth constant at I 3, increasing width lcads to higher locality Icakage. However, interestingly, holding the width constant and increasing depth generally leads to less locality leakage. Such a trend polentially occurs because of information bottlenecking (Tishby et al.] 2000); increasingly deep models that are not sufficiently wide might make it difficult to train CBMs because information gets "stuck" between layers. The experiments with MLPs highlight that locality leakage can be an issue with other architectures, though the septh-widtte patterns from CNNs do not necessarily carry over to MI.Ps.
17

---

Published in Transactions on Machine Learning Research (06/2025)
Table With MLPs as concept predictors, wider MLPs lead to more locality leakage. However, deeper MLPs do not lead more locality lcakage, contradicting the pattern found in Section [ This trend potentially
occurs because deeper models lead information bottlenecks which prevent information from travelling between layers Tiishyy et al. 2000
Depth
Width
Locality Leakage
1
0.63 + 0.32
0.80 = 0.28
0.97 0.05
2
0.59 = 0.30
10
0.47 + 0.08
15
0.57 = 0.32
3
5
0.49 = 0.40
10
0.74 + 0.19
0.74 = 0.09
1.0
Locality Leakage
0.8
0.6
0.4
1-object dataset
2-object dataset
4-object dataset
0.2
3
4
Layers
Figure 8: We reconstruct the experiment from Section and adding noise to the synthetic datasets. We see that adding noise does not change the main result that deeper models lead to more locality leakage, which implies that our results are robust to noise in the input.
18

---

Published in Transactions on Machine Learning Research (06/2025)
Back
Back
038
0.89
Belly
Belly
Breast
0.22
Breast
Crown
Main Part
0.07
-019
Forehead
Forehead
Nape
-0.8
Nape
Tail
Tai
0.0
Throat
0.0
Back
Belly
Breast
Crown
Fornhead
Nape
Tail
Throat
Back
Belly
Breast.
Crown
Forehead
Nape
Tall
Throat
Masking Part
Masking Part
(a) Zero Masks
(b) Mean Masks
Figure 9: We compare the impact of zero masks (a) and mean masks (b) in the CUB dataset. We compute the change in prediction for one part (row) when masking another part (column). For both zero and mean masks, masking the belly has the same effect for predicting the belly as it does for predicting unrelated parts such as the back and forehcad. Morcover, the absence strong diagonal pattern (whore the same part is masked and analyzed) implies that both zero and mean masks have similar ellects for relevant and irrelevant masking.
B.3 Noisy Synthetic Datasets
We construct a variant of the syuthetic dataset with additional noise to understand whether locality leakage persists when slightly perturbing data features. We construct such a dataset by adding Gaussian noise the 1, 2, and 4-object datasets at train and test time, computing locality leakage across different layered concept predictors.
In Figure 8 we see that incrcasing the number of layers similarly leads to increased locality leakage. The trend matches the patterns seen in Section 5 where deeper concept predictors led to higher locality leakages. Similar results with or without noise imply that our results are not sensitive to small dataset perturbations.
B.4 Selection of Masking Colour
We assess whother our locality masking experiments Scction 5 are a type of mask used. We originally used zero mask, where features are modified to zero, and we whether similar results apply when using mean masks instead. The mean masks replace the data point features with the average value of the feature computed throughout the training set. By analyzing mean masks, we are able to see whether our choice of the masking colour has a big impact on our results.
We compare the results using zero and mean masks for the CUB dataset Figure 9 We find that both zero and mean masks exhibit similar trends, where masking parts such as the back and belly can lead to performance decreases for unrelated parts. For example, masking the belly of bird leads to locality masking score of 0.10 for the back. Moreover, we shown by the absence of any pattern the diagonal. Therefore, our locality masking results from Section are not sensitive to the type of masks used, whether that is zero or mean masks.
difference between relevant and irrelevant masking,
B.5 Residual Connections
Introducing residual connections CBMs can alleviate concept leakage, so we explore whether similar choices can improve locality (Havasi et al. 2022 Residual connections introduce a connection between the input x and the output y, avoiding the bottleneck layer c. We introduce a similar connection and assess the impact locality leakage in CBMs in a synthetic dataset.
19

---

Published in Transactions on Machine Learning Research (06/2025)
1.0
0.8
Metric
0.6
Accuracy
0.4
Leakage
3
6
7
Model Depth
Figure 10: We incorporate residual connections following prior work Havasi et al. 2002 to understand whether techniques used to address concept leakage can also improve locality. Across model sizes, we find that models trained with residual connections still exhibit high levels of locality leakage, with this best seen with 7epth-7 models.
40%
Change in Pred.
30%
20%
10%
0%
0.1 0.2
Mask Radius (e)
We vary the mask radius for the tail concept in the CUB dataset and find that, as expected, increasing mask radius leads increasing change concept prediction. However, even large masks that lead to significantly distorted images, hinting that predictors may use correlations to predict tail colour.
Figure 11:
In Figure 10 we find that residual connections fail to impact locality leakage. Concept predictors with depths and still exhibit a locality leakage near when using residual connections. The result demonstrrtts the difference between locality and concept leakage; solutions concept leakage such as residual connections fail to improve the locality properties of CBMs.
B.6 Impact of Mask Size
To better understand locality masking, analyse the impact of mask size concept predictions. We evaluate the impact of mask predictions in CUB for the concepts of tail colour and tail shape, selecting these since tails are visually distinct from other parts in this task. We reformulate masking in CUB so all features within e the part's centre are masked, even if the features are closer to another concept, and vary 6 from 1% to 30% 1. Intuitively, we expect that increasing e should increase changes concept prodiction, yet it is unclear how large c needs be so that models are significantly less confident. We compute the change in concept prediction across concept values that this as threshold indicative of high-confidence prediction, with the expectation that masking should decrease model confidence so models are uncertain (around 50%).
least. 75% confident, selecting
In Figure I we find that, as expected, increasing increases changed predictions, yet even large values of fail to change high-confidence concept predictions significantly. This implies that models are exploiting correlations between, for example, "chest colour" and "toil colour" predict the "white tail colour" concept. Such phenomenon might occur because concepts are occluded for some samples CUB, For example, predicting " lail colour" when the tail occluded requires leveraging " "chest colour".
20

---

Published in Transactions on Machine Learning Research (06/2025)
B.7 Comparison with Other Metrics
We present a discussion of the relationship between our work and a of related but distinct metrics proposed by Espinosa Zarlenga et al. (2023a). In that work, the authors propose two metrics: the oracle impurity score (QIS) and niche impurity score (NIS). OIS captures the ability of concept predictions for concept j to predict an unrelated concept J'. Higher values for OIS indicate that more information is leaked between concepts, or that concept predictors are encoding impurities. NIS extends this to consider sets of concepts that encode impurities by investigating the predictive power of such sets. NIS values near 1/2 indicate that no impurities are encoded, while larger NIS values imply that higher levels of impurity are encoded, For further details and a rigorous definition, we refer readers to prior work (Espinosa Zarlengal et al. )
We demonstrate that the OIS and NIS metrics fail to distinguish between models with low and high levels of locality leakage. We do so by comparing the OIS and NIS metrics across models of varying depth trained on the 2-object dataset. Across all model depths, we find that the OIS values range between 0.05 and 0. 0010, while the NIS values range between 0.41 and 0.47. Both metrics indicate that models do not leak impuritics into concept predictors. Despite this, we find a large variation locality leakage between models with 3 layers and those with 7. The reason for this discrepancy is because the OIS and NIS metrics are computed by comparing concept predictions to groundtruth concepts, which ignores whether concept predictors leverage the appropriate set of features in concept's 's local region. In contrast, our metrics evaluate concept predictors across perturbed data points to understand whether these concept. predictors respect locality. Our locality metrics are complementary to the OIS and NIS metrics, and they each measure different desirable properties of concept predictors.
C Proofs
This section presents our proof for Theorem 6.1 discussed in Section For to present our proof, begin by showing two lemmas that will become useful later on. The first lemmn, shown below, introduces simple identity on sets of real numbers in [0, 1J:
Lemma C.1. Consider a set of n real numbers {p) ... p.., where Vi, 0 0 P. S 1. Then, the following identity must hold:
Σ P. (1--;) p2) =1 -1- - p.) #=1
Proof. To simplify this proof, we will show the equivalent but more accessible statement that:
I pd) =1
Σ P [ pj)
We proceed to do this by induction on the number of real numbers n. Our base case n = trivially follows by noting that both sides of the equation above resolve to 1 - Pi when 1.
Now, assume that the claim above holds for all positive integers up some value i E + Consider what
happens when k + 1:
21

---

Published in Transactions on Machine Learning Research (06/2025)
k+1
R
(1-p:) = (1 Pk+1)
II - pp))
2m1
k
p.)
-P II (1 pt)
1=1
i=1
(1
Ps
(1 - pj))
Ph+1
- pi)
j=1
i=1
pk+1 II (1 Pj)
Pi
j=1
k+1
Σ 111
(1
j=1
where the inductive hypothesis was applied in our second step. This shows that statement. holds for = k and therefore holds for all positive integers
The second lemma directly builds on the identity introduced in Lemma C.1 to produce simple upper bound for sum of products involving the same list of reals in [0, 1]: Lemma C.2. Consider a set of real numbers {pr ...... "Pp,, where Vi, 0 S S 1. Then
P
$1
Proof- We begin by noticing that, because all (pi,... terms in the product IT (1-p:) must be non-negative. This trivially implies that the product IT; p.) itself must be non-uegative. Therefore, can conclude that:
p Po) are reals such that Vi, 0 ps S 1, each of the
0 0
(1 p.) 1=1
P.) S
P:
p,) 1
(by Lemma C.1
which is exactly what we wanted to show.
Equipped with these two lemmas, we now present proof for the key theoretical result introduced in Section
Theorem 6.1. Suppose out of k total concepts, we know m, {71, 72, 72. .mm C {1, 11,2...A). with perfect accuracy. Let Migsir = PPcc = g/c; = r] be correlation matrix, where q.r E {0.1}. For any concept & {11,72 ..-mTmm whose value we do not know, if for different triplets iq,i,)), with i E 1...72.--700 we have i) Mjg.rr 2 1 a and si) P(c, = r] 2 3, then there exists a concept predictor that achieves error P cc, S aa + (1 (1-9) 3)* using only information about the concepts (rI, 721.72..-700 }.
Proof. We propose an algorithm which achieves error rate = P(c, = r + -9).. To predict Cj. we leverage the correlation between cj and an s different values of Mj where E {%, 72 m....mmm 7m). By leveraging this, we't re able to accurately predict the presence of cj
22

---

Published in Transactions on Machine Learning Research (06/2025)
For this, let the triplets satisfying (i) and (ii) be B (Br) Bi (1) Furthermore, for any triplet B₁ (gt, (n) in this set, let be Plca, = mi].
Here, we propose an algorithm that takes as an input the set of triplets B, along with their associated probabilities {pili= and returns a label ê for an unknown concept j. Our algorithm proceeds as follows: We will iterate with an index variable u starting with u and finishing with u = 8. At each iteration, we select the triplet B, (qu, dug Tu). As stated above, we know in (2 Tmh and therefore we have perfect knowledge of c, Therefore, we check if C. Tu- If that is the case, then our algorithm predicts C qu- Otherwise, simply guess that č = 0.
continue to the next iteration of the loop. If reach the end of the loop, then we
We note that our algorithm will terminate at step with probability Pu I1= pi). This is because to terminate at step Additionally, we must not terminate in any previous steps I < u, each event happening with probability Pfc., # rı] = 1 Pfcs, ri] = 11 pt- If we terminate at step then we know that c, = ru and we predict qu. This prediction fails with probability Pic, & qulc,, = =ra] = 1-P(c, = qu Ci,, = =ru] = 1- M du..u. For the sake of simplifying notation, and because j remains constant and implicitly known throughout the algorithm, we use MB. to represent Miquukurr for triplet B, (qu Ta).
must encounter the event Ci2 Tu, which occurs with probability P(c:, ru] Pu
The error rate Plcc c,] of this algorithm is then given by
Σ
MB. Pa
II
(1
pi) + Flc, + 0 Pw)
n-1
< ==1
MB. Pu
II (1
pt)
+ - pp)
1=1
==1
Plc, + < Then, we note that Pa 3, implying that II- (1
Pw)) (1 - 33". Finally, note that
MB,, a, so Ex=1 MB.Pu Pu I[ pı) S E.=1 Pu [T=1
(1 pt) S a by Lemma C.2 Combining all
of the above gives error cs (1 3)
23


Reviews:
[{"review_id": "idFVhfA4Ek", "reviewer": "Reviewer_APxv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. The faithfulness of the concept predictor plays an important role in concept-based explanations, and this work provides a valuable, new way to assess it. \n2. The method is solid, and the presentation is generally clear. \n3. The experiments on different model architectures, training strategies, and hyperparameters are comprehensive.\n\n**Weakness**\n1. Some details concerning methods are missing.\n- 1.1 When measuring the locality leakage, how are adversarial perturbations conducted?\n- 1.2 When measuring the locality intervention, how to make sure the perturbations do not alter the concept labels?\n2. Some experimental settings need further clarification and adjustment.\n- 2.1 In the synthetic experiment, it’s not clear to me why there are k concepts corresponding to k/2 regions. It seems to imply that concepts are region-specific and enforce the concept predictor to only focus on a designated region (left or right in the 2-object case). In the general image classification set, objects concerning a certain concept are usually not present in a designated region.\n- 2.2 In the locality masking experiments on CUB, I’m not sure the mask with a radius of 0.3l is reasonable for all the concepts. Different concepts intrinsically have different sizes (e.g., belly vs. beak). The bird region also has various sizes in different images. This rough masking strategy might mask relevant (irrelevant) regions unintentionally.\n3. Lack of discussion and comparison with other methods\n- 3.1 What are other methods/metrics assessing the concept of faithfulness, and how does locality compare with these metrics", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes a new perspective on assessing the faithfulness of concept predictors in the context of concept bottleneck models, named \"locality\", which evaluates whether concept predictors rely only on relevant regions in the input image. It proposes three metrics to quantify locality -- locality leakage, locality intervention, and locality masking. They demonstrate these metrics in various experiments and provide insights on how correlations of concepts affect locality.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Adjustments that are critical to secure my recommendation for acceptance.\n- 1.1 Clarify the details concerning the methods as listed in the weakness part.\n- 1.2 Justify or adjust the experiment settings listed in the weakness part.\n2. Adjustment that will further strengthen the work, in my view\n- 2.1 Discuss and compare other metrics for assessing concept faithfulness.\n- 2.2  Make a connection between locality and intervention by discussing how locality affects the efficiency of the model intervention.", "broader_impact_concerns": "There are no concerns about the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Yj2G63VFdc", "reviewer": "Reviewer_Bbsu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The paper introduces three novel and well-motivated metrics—locality leakage, locality intervention, and locality masking—that provide a way to evaluate whether concept-based models base their predictions on the intended, relevant features.\n- It features an extensive experimental evaluation spanning synthetic and real-world datasets (such as CUB and COCO). This thorough evaluation covers different architectural choices (e.g., model depth, receptive field, parameter count) and training modifications (like early stopping, adversarial training, and regularization).\n- The inclusion of theoretical insights supports the empirical findings and offers a deeper understanding of how dataset correlations can lead to models relying on spurious features.\n- The manuscript offers practical guidance by recommending that practitioners use diverse datasets and models with appropriate capacity (e.g., smaller models or those with improved representational power) to obtain more trustworthy concept explanations.\n\nOverall, the work addresses a timely and important challenge in explainable AI by questioning the reliability of concept-based explanations, which is critical for model trustworthiness.\n\n##", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates whether concept-based models, which explain their predictions using human-understandable intermediaries (or \"concepts\"), actually base these predictions on the intended, relevant features.\nThe authors make three key contributions: i) they introduce three novel metrics—locality leakage, locality intervention, and locality masking—to quantitatively assess if and how much a model’s concept predictions change when features outside the concept’s relevant region are perturbed, ii) they conduct an experimental study on both synthetic and real-world datasets (including CUB and COCO) to evaluate how factors like model depth, architectural choices, training procedures, and dataset diversity affect these locality metrics, and iii) they provide theoretical insights that explain how correlations between concepts in a dataset can lead models to depend on only a subset of features, even when this does not align with the true concept definition.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Explicitly define all variables and conditions. For example, clarify that the “perfect accuracy” assumption for the subset {$\\gamma_1,\\dots,\\gamma_m$} means that for those concepts, the predictor has zero error.\n- Clearly state the domain and range of each variable and what the probability terms represent.\n\nIn addition, the following discussion would improve the manuscript and strengthen their contributions, if possible (but not necessary).\n\n### Discuss tightness and optimality of the bound\n- Elaborate on whether the bound $\\epsilon \\leq \\alpha + (1 − \\beta)^s$ is tight under realistic conditions or if it can be improved further.\n- Consider discussing conditions or examples where the bound becomes loose or where additional constraints (e.g., independence assumptions) could yield a sharper bound.\n\n### Relax the “perfect accuracy” requirement\n- Investigate how the theorem might extend if the known concepts are predicted with high—but not perfect—accuracy.\n- For example, include an error term $\\delta$ for the prediction of the known concepts and derive an adjusted bound", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nrJxWLJeUX", "reviewer": "Reviewer_6Tsj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The main body of this paper is well-developed, with standardized figures and tables. Theoretical validation has been conducted for the core contributions.\n2. The motivation and innovation of this paper are acceptable.\n\nWeaknesses:\n1. The writing of the abstract is extremely difficult to comprehend. It extensively discusses the research background but only briefly mentions the methodology in a single sentence. There is no mention of methodological innovation or how the experiments validate the findings. The issues raised in the background section are also quite peculiar. The author argues that conceptual-level explanations are inferior because they are \"virtual,\" but isn't this precisely the advantage of conceptual-level explanations? They transform abstract features into concepts that are easier for humans to understand.\n\n2. The author's writing and description are inadequate, leading to many logical doubts. The formulas are not numbered, and many symbols are not explained. Additionally, many formulas do not seem to be used later in the text. It is recommended that the author further organizes and clarifies these aspects.\n\n3. The derivation of formulas in the article is relatively brief, lacking detailed intermediate steps. To enhance readability, the authors should provide more detailed derivation steps, especially for the intermediate processes of key formulas. The assumptions of Theorem 6.1 are quite stringent, which may be difficult to satisfy in practical applications, as real-world data often contains noise and incompleteness. Additionally, the specific form of the data distribution is not discussed in the assumptions of the theorem. The application of metrics such as \"locality leakage\" and \"locality intervention,\" defined in the formulas, is not thoroughly explained in the experimental section.\n\n4. This paper has shortcomings in the comparative validation of interpretability methods, which mainly include two aspects: 1）The paper should compare with more interpretability methods and adopt a broader range of evaluation metrics. 2）More experiments should be designed to verify the effectiveness of the proposed method. The current experimental setup is insufficient to fully validate the claims. It is recommended that the authors refer to other interpretability method papers to enrich the experiments. The paper primarily utilizes synthetic datasets and two real-world datasets for experiments. Although these datasets are widely used in the study of CBM, they do not comprehensively represent all possible application scenarios. The experiments in the paper mainly focus on variations in model architecture and training methods, lacking in-depth exploration of other factors that may influence model interpretability. Additionally, the experiments do not consider the impact of different task types (such as classification, regression, and generative tasks) on model interpretability. The absence of these factors limits the applicability of the experimental results, making it difficult to fully reflect the model's performance in real-world applications.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper designs three metrics to quantify locality and evaluates them across different architectural and training configurations. The results indicate that the neglect of locality is a widespread issue, but this can be partially mitigated through the use of diverse training data and smaller model architectures. Additionally, the paper provides a theoretical analysis that reveals how concept correlations can lead to concept-based models failing to meet locality requirements.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. It is recommended that the authors reorganize the abstract and introduction, adding more details about the methodology to highlight its features and innovations.\n\n2. It is suggested that the authors include more comparative methods, evaluation metrics, and dataset validation.\n\n3. The article involves a large number of formulas and symbols, with particularly confusing use of superscripts and subscripts. It is recommended that the authors add a notation section or provide a table to clarify these elements. For instance,  Mj,q,i,r......\n\n4. The description of the methodology in the article is not detailed enough. It is recommended to add a framework diagram of the method and pseudocode for the core steps.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)