Editable Concept Bottleneck Models
Authors: Lijie Hu
Chenyang Ren
Zhengyu Hu
Hongbin Lin
Cheng-Long Wang
Zhen Tan
Weimin Lyu
Jingfeng Zhang
Hui Xiong
Di Wang
Venue: ICML 2025 poster (2025)
Keywords: Concept-based Models
Interpretability
Editing
Primary area: social_aspects->accountability_transparency_and_interpretability

Abstract: Concept Bottleneck Models (CBMs) have garnered much attention for their ability to elucidate the prediction process through a human-understandable concept layer. However, most previous studies focused on cases where the data, including concepts, are clean. In many scenarios, we always need to remove/insert some training data or new concepts from trained CBMs due to different reasons, such as privacy concerns, data mislabelling, spurious concepts, and concept annotation errors. Thus, the challenge of deriving efficient editable CBMs without retraining from scratch persists, particularly in large-scale applications. To address these challenges, we propose Editable Concept Bottleneck Models (ECBMs). Specifically, ECBMs support three different levels of data removal: concept-label-level, concept-level, and data-level. ECBMs enjoy mathematically rigorous closed-form approximations derived from influence functions that obviate the need for re-training. Experimental results demonstrate the efficiency and effectiveness of our ECBMs, affirming their adaptability within the realm of CBMs.

Editable Concept Bottleneck Models
Lijie Hu 12 Chenyang Ren 123 Zhengyu Hu 141 Hongbin Lin
Cheng-Long Wang" 12 Zhen Tan
Weimin Lyu Jingfeng Zhang? Hui Xiong* Di Wang 12
Abstract
non-linear architectures, posing challenges for end-users seeking to comprehend and trust their decisions. This lack of interpretability presents significant barrier to adoption, particularly in critical domains such as healthcare (Ahmad et al., 2018; Yu et al., 2018) and finance (Cao, 2022), where transparency is paramount. To address this demand, explain- able artificial intelligence (XAI) models (Das & Rad, 2020; Hu al., 2023b;a) have emerged, offering explanations for their behavior and insights into their internal mecha- nisms. Among these, Concept Bottleneck Models (CBMs) (Koh et al., 2020) have gained prominence for explaining the prediction process of end-to-end Al models. CBMs add bottleneck layer for placing human-underssandable concepts. In the prediction process, CBMs first predict the concept labels using the original input and then predict the final classification label using the predicted concept in the bottleneck layer, which provides a self-explained decision to users.
Concept Bottleneck Models (CBMs) have gar- nered much attention for their ability to eluci- date the prediction process through a human- understandable concept layer. However, most previous studies focused on cases where the data, including concepts, are clean. In many scenar- ios, we often need to remove/insert some training data or new concepts from trained CBMs for rea- sons such as privacy concerns, data mislabelling, spurious concepts, and concept annotation errors. Thus, deriving efficient editable CBMs without re- training from scratch remains challenge, particu- larly in large-scale applications. To address these challenges, we propose Editable Concept Bottle- neck Models (ECBMs). Specifically, ECBMs support three different levels of data removal: concept-label-level, concept-level, and data-level. ECBMs enjoy mathematically rigorous closed- form approximations derived from influence func- tions that obviate the need for retraining, Ex- perimental results demonstrate the efficiency and adaptability of our ECBMs, affirming their practi- cal value in CBMs. Code is available on ht tps: ///ittull htg.tabb.cc//tuuuttraaaaaa lab/ ECBM
Concast Inbel-ovvl
Consrge-bee bvail
Date (oud
Imput
(04-) 3o
Innel
Concept
1. Introduction
Editad
Dataset (1Ac:.0-1142
Modern deep learning models, such as large language mod- els (Zhao et al., 2023; Yang et al., 2024a;b; Xu et al., 2023;
Figure 1: An illustration of Editable Concept Bottleneck Models with three settings.
Yang et al., 2024c) and large multimodal (Yin et al., 2023;
Ali et al., 2024; Cheng et al., 2024), often exhibit intricate
Existing research on CBMs predominantly addresses two primary concerns: Firstly, CBMs heavily rely on laborious dataset annotation. Researchers have explored solutions to these challenges in unlabeled settings (Oikarinen et al., 2023; Yuksekgonul al., 2023; Lai al., 2023). Secondly, the performance of CBMs often lags behind that of origi- nal models lacking the concept bottleneck layer, attributed to incomplete information extraction from original data to bottleneck features. Researchers aim to bridge this utility gap (Sheth & Ebrahimi Kahou, 2023; Yuksekgonul et al., 2023; Espinosa Zarlenga et al., 2022). However, few of them considered the adaptivity or editability of CBMs, cru-
"Equal contribution 'rrovbbee Responsible Al and Data Ana- lytics (PRADA) Lah *King Abdullah University of Science and Technology 'Shanghai Jiao Tong University "Thrust of Artificial Intelligence, The Hong Kong University Science and Technol- ogy (Guangzhou), China Department of Computer Science and Engineering, The Hong Kong University of Science and Technol- ogy, Hong Kong SAR, China Ariizna State University "Stony Brook University 'The University of Auckland. Correspondence to: Di Wang < wang @ kaust.edu.sa>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Editable Concept Bottleneck Models
cial aspects encompassing annotation errors, data privacy considerations, concept updates. Actually, these demands are increasingly pertinent in the era of large models. We de- lineate the editable setting into three key aspects (illustrated in Figure l):
in CBMs remains largely unexplored due to their composite structure, i.e., the intermediate representation layer.
To the best of our knowledge, we are the first to work to fill this gap by demonstrating the effectiveness of influence functions in elucidating the behavior of CBMs, especially in identifying mislabeled data and discerning the data influ- ence. Comprehensive experiments on benchmark datasets show that our ECBMs are efficient and effective. Our con- tributions are summarized follows.
Concept-label-lveel: In most scenarios, concept labels are annotated by humans or experts. Thus, is unavoid- able that there are some annotation errors, indicating that there is need to correct some concept labels in a trained CBM.
We delineate three different settings that need various levels of data or concept removal in CBMs: concept- label-level, concept-level, and data-level. To the best of our knowledge, our research marks the first exploration of data removal issues within CBMs.
In CBMs, the concept set is pre-defined by LLMs experts. However, in many cases, evoly- ing situations demand concept updates, evidenced by discoveries such as chronic obstructive pulmonary disease as a risk factor for lung cancer, and doctors have the requirements to add related concepts. For another example, recent research found a new factor, obesity (Sattar et al., 2020) are risky for severe COVID- 19 and factors (e.g.. older age, male gender, Asian race) are risk associated with COVID-19 infection (Rozen- feld et al., 2020). On the other hand, one may also want to remove some spurious or unrelated concepts for the task. This demand is even more urgent in some rapidly evolving domains like the pandemic.
Concept-level:
To make CBMs able to remove data concept in- fluence without retraining. we propose the Editable Concept Bottleneck Models (ECBMs). Our approach in ECBMs offers a mathematically rigorous closed- form approximation. Furthermore, to improve com- putational efficiency, we present streamlined versions integrating Eigenvalue-correcteed Kronecker-Factored Approximate Curvature (EK-FAC).
To showcase the effectiveness and efficiency of our ECBMs, we conduct comprehensive experiments across various benchmark datasets to demonstrate our superior performance.
Data issues can arise in CBMs when train- ing data is erroneous or poisoned. For example, if a doctor identifies case as erroneous or poisoned, this data sample becomes unsuitable for training. There- fore, it is essential to have the capability to completely delete such data from the Iearned models. We need such an editable model that can interact effectively with doctors.
Data-level:
2. Related Work
Concept Bottleneck Models. CBM (Koh et al., 2020) stands out as an innovative deep-learning approach for im- age classification and visual reasoning. It introduces con- cept bottleneck layer into deep neural networks, enhanc- ing model generalization and interpretability by learning specific concepts. However, CBM faces two primary chal- lenges: its performance often lags behind that of original models lacking the concept bottleneck layer, attributed to incomplete information extraction from the original data to bottleneck features. Additionally, CBM relies on laborious dataset annotation. Researchers have explored solutions to these challenges. Chauhan et al. (2023) extend CBM into interactive prediction settings, introducing an interaction policy to determine which concepts to label, thereby improv- ing final predictions. Oikarinen et al. (2023) address CBM limitations and propose a novel framework called Label-free CBM. This innovative approach enables the transformation of any neural network into interpretable CBM without requiring labeled concept data, all while maintaining high accuracy. Post-hoc Concept Bottleneck models (Yuksek- gonul et al., 2023) can be applied to various neural networks without compromising model performance, preserving in- terpretability advantages. CBMs work on the image field
The most direct way to address the above three problems is retraining from scratch on the data after correction. How- ever, retraining models in such cases prove prohibitively expensive, especially in large models, which is resource- intensive and time-consuming. Therefore, developing efficient method to approximate prediction changes becomes paramount. Providing users with an adaptive and editable CBM is both crucial and urgent.
We propose Editable Concept Bottleneck Models (ECBMs) to tackle these challenges. Specifically, compared to retrain- ing. ECBMs provide a mathematically rigorous closed-form approximation for the above three settings to address ed- itability within CBMs efficiently. Leveraging the influence function (Cook, 2000; Cook & Weisberg, 1980), we quan- tify the impact of individual data points, individual concept labels, and the concept for all data on model parameters. De- spite the growing attention and utility of influence functions in machine learning (Koh & Liang, 2017), their application
2

---

Editable Concept Bottleneck Models
also includes the works of Havasi al. (2022),Kim et al. (2023).Keser et al. (awaa), & Nakamura (2022) and Sheth & Kahou (2023). Despite many works on CBMs, we are the first to investigate the interactive influence between concepts through influence functions. Our research endeav- ors to bridge this gap by utilizing influence functions in CBMs, thereby deciphering the interaction of concept mod- els and providing an adaptive solution to concept editing. For more related work, please refer to Appendix I.
ual point in the sample. Consider a neural network Ô arg ming Σi=1 [(z;; 0) with loss function é and dataset D= - If we remove 2m from the training dataset, the parameters become i_ arg ming Σigm (zz 0). The influence function provides efficient model approx- imation by defining series of e-parameterized models as - = argmin Σi=1 {(z; + cl(2mis 0). By perform- ing a first-order Taylor expansion on the gradient of the objective function corresponding to the arg min process, the influence function is defined as:
3. Preliminaries
Iø (2m) a ddd.-
Concept Bottleneck Models. In this paper, we consider the original CBM, and we adopt the notations used by Koh et al. (2020). We consider a classification task with a concept set denoted as {PI,... , pk with each pt being a concept given by experts or LLMs, and training dataset represented as D {= where
Ho! 1 Vol(zmid);
de
l==0
where - (z;;;) is the Hessian matrix. When the loss function L is twice-differentiable and strongly convex in 0, the Hessian Ha is positive definite and thus the influence function is well-defined, For non-convex loss functions, Bartlett (1953) proposed replacing the Hessian Ha with i G3 + 51, where G3 is the Fisher information matrix defined as D Vol(z; 0) V (2,; f), and 5 is the damping term used to ensure the positive definiteness of H. We can employ the eneenvauuecoorrecte Factored Approximate Curvature (EK-FAC) method to fur- ther accelerate the computation. See Appendix C for addi- tional details.
(Ii, Vi, C)). Here, for i E [n]] I; E IRd represents the input feature vector, Vi E Rd denotes the label (with d, corresponding to the number of classes) and G = (ct..... .t) E IR represents the concept vector. In this context, cf represents the label of the concept Py of the i-th data. In CBMs, our goal is to learn two representations: one called concept predictor that transforms the input space into the concept space, denoted as g : IR R² and the other called label predictor which maps the concept space to the prediction space, denoted as f: R + Rdo. Usually, here the map f is linear. For each training sample 2/ = (Iz, V/. 4t), we consider two empirical loss functions: concept predictor و and label predictor f:
Kronecker-
4. Editable Concept Bottleneck Models
In this section, we introduce our EBCMs for the three set- tings mentioned in the introduction, leveraging the influence function. Specifically, at the concept-label level, we calcu- late the influence of a set of data samples' individual concept labels; at the concept level, we calculate the influence of multiple concepts; and at the data level, we calculate the influence of multiple samples.
g arg min
ΣΣ (c:) log(e)) =1
(1)
where (0()) (+) is the predicted j-th concept. For brevity, write the loss function as Lc(g(z,),c) c)) Σÿ= (g(x)) a))
for data (z;, c). Once we obtain the concept predictor g. the label predictor is defined as:
4.1. Concept Label-level Editable CBM
f= = arg min Ly ())
(2)
In many cases, certain data samples contain erroneous an- notations for specific concepts, yet their other information remains valuable. This is particularly relevant in domains such as medical imaging, where acquiring data is often costly and time-consuming. In such scenarios, it is common to correct the erroneous concept annotations rather than removing the entire data from the dataset. Estimating the re- trained model parameter is crucial in this context. We refer to this scenario the concept label-level editable CBM.
where Ly represents the cross-entropy loss, similar to (1). CBMs enforce dual precision predicting interpretable concept vectors é = g(x) (matching concept e) and final outputs y = f()) (matching label y), ensuring transparent reasoning through explicit concept mediation. Furthermore, in this paper, we focus primarily on the scenarios in which the label predictor f is linear transformation, motivated by their interpretability advantages in tracing concept-to-label relationships. For details on the symbols used, please refer to the notation table in Appendix 2.
Mathematically, we have a set of erroneous data D. and its associated index set S. C [r]] x [k] such that for each (w, r) E Se, (In, Bu: C.n) E D. with cf is mislabeled and Tye is corrected concept label. Our goal is to estimate the retrained CBM. The retrained concept predictor and label
Influence Function. The influence function measures the dependence of an estimator on the value of individ-
3

---

Editable Concept Bottleneck Models
predictor are represented follows:
CBM framework, only can test-time interventions be performed, but the concept predictor of the CBM can also be further refined based on test data that repeatedly produces errors. Our ECBM method incorporates the corrected test data into the training dataset without requiring full retraining. This approach extends the rectification process from the data level to the model level.
ge arg min
Σ
Li (o())))))
(1J)45.
+
Σ
L'o (g(a))))),
(iJ)ES.
fe arg min
Σ Ly (Ôe (z))).
(4)
4.2. Concept-level Editable CBM
In this case, a set of concepts is removed due to incorrect attribution or spurious concepts, termed concept-level edit. Specifically., for the concept set, denote the erroneous con- cept index set as M C [k], we aim to delete these concept labels in all training samples. We aim to investigate the im- pact of updating the concept set within the training data on the model's 's predictions. It is notable that compared to the above concept label case, the dimension of output (input) of the retrained concept predictor (label predictor) will change. If we delete concepts from the dataset, then g becomes g' Rdd + k- and f becomes f" TR& - Rdd, More specifically, if retrain the CBM with the revised dataset, the corresponding concept predictor becomes:
For simple neural networks, we can use the influence func- tion approach directly to estimate the retrained model. How- ever, for CBM architecture, if we intervene with the true concepts. the concept predictor 9 fluctuates to ge accord- ingly. Observe that the input data of the label predictor comes from the output of the concept predictor, which is also subject to change. Therefore, we need to adopt two- stage editing approach. Here we consider the influence function for (3) and (4) separately. We first edit the concept predictor from I to ger and then edit from f to Se based on our approximated concept predictor. To begin, we provide the following definitions:
Definition 4.1. Define the gradient of the j-th concept pre- dictor and the label predictor for the i-th data point x, as:
9-PM PM arg min
ΣΣ JEM
Lool' (1)).
(5)
Go A T (1) Gy Gy - v ((((()))...)
The variation of the parameters in dimension renders the application of influence function-based editing challenging for the concept predictor. This is because the influence function implements the editorial predictor by approximate parameter change from the original base after c-weighting the corresponding loss for given sample, and thus, it is unable deal with changes in parameter dimensions.
Theorem 4.2. The retrained concept predictor de defined by (3) can be approximated by ge, defined by:
9 - -H77
Σ
(Go Twi 9) GT Cwi g))
(w.r)ES.
To overcome the challenge, our strategy is to develop some transformations that need to be performed on Ò-pxr to align its dimension with 9 so that we can apply the influence function to edit the CBM. We achieve this by mapping I-par to 80pxx A P(g-px)) which has the same amount of parameters as 9 and has the same predicted concepts 9- (i) g (j) for all j E [d.] M. We achieve this effect by inserting zero row vector into the r-th row of the matrix in the final layer of ÿ-pux for r E M. Thus, we can see that the mapping P is one-to-one. Moreover, assume the parameter space of 9 is and that of "'pa To is the subset of T. Noting that g0pxx the optimal model of the following objective function:
where Hg 1 V En, Golzi, ()) is the Hessian matrix of the loss function with respect to 9.
Theorem 4.3. The retrained label predictor fe defined by (4) can be approximated by fer defined by:
Σ (Gy(In). - -Cy(:ieerr)) 1=1
where H V Gy ( f) the Hessian matrix, and Ge is given in Theorem 4.2.
Difference from Test-Time Intervention. The ability to intervene in CBMs allows human users interact with the model during the prediction process. For example, medical expert can directly replace an erroneously predicted concept value è and observe its impact on the final prediction ÿ. However, the underlying flaws in the concept predictor remain unaddressed, meaning similar errors may persist when applied to new test data. In contrast, under the editable
9- 9-pM arg min ΣΣ (olod((x;))
(6)
g'ETo
JEM
i.e., it is the optimal model of the concept predictor loss on the remaining concepts under the constraint To. Now we
For convenience, in this paper, we only consider concept removal; our method can directly extend to concept insertion.

---

Editable Concept Bottleneck Models
can apply the influence function edit I approximate 0"pu with the restriction the value of 0 for rows indexed by MM with the last layer of the neural network, denoted as --pM: After that, we remove from T TPp the parameters initially inserted to fill in the dimensional difference, which always equals O because of the restriction we applied in the editing stage, thus approximating the true edited concept predictor g-px-- We now detail the editing process from 9 to 0'pas using the following theorem.
where Hj = VV Gy(mii--xx f) is the Hessian matrix. Deleting the r-th dimension of fo E M, then we can map it to f-pm- which is the approximation of the final edited label predictor f-pms under concept level.
4.3. Data-level Editable CBM
In this scenario, we more concerned about fully remov- ing the influence of data samples on CBMs due to different reasons, such as the training data involving poisoned or er- roneous issues. Specifically, we have a set of samples to be removed {(z,, Vr, ki) with C [n]. Then, we define the retrained concept predictor as
Theorem 4.4. For the retrained concept predictor 9-pMr defined in (5), we map it to I-pu as (6). And we can edit the initial i to I-pme defined as:
.- PM
H JEM 1=1
ΣV
G ((()))
k
9-zG arg min
E Llc(o( g j-
Σ
(8)
where - V DjEM Ei= Co oz.)). Then, by re- moving all zero rows inserted during the mapping phase, we can naturally approximate 9-PM- = P- (p-pa)
which can be evaluated by the following theorem:
Theorem 4.7. For datasel D = {(xI Vi, oi))=r given
set of data 2, = (Zr: Vr, Cr), E G to be removed. Suppose the updated concept predictor 9-zo is defined by (8), then we have the following approximation for 9-z0
For the second stage of training, assume we aim to remove concept pr for r E M and the new optimal model is f-pae- We will encounter the same difficulty as in the first stage, i.e., the number of parameters of the label predictor will change. To address the issue, our key observation is that in the exist- ing literature on CBMs, we always use linear transformation for the label predictor, meaning that the dimensions of the input with values of 0 will have no contribution to the final prediction. To leverage this property, we fill the missing values in the input of the updated predictor with O, that is, replacing 8-px with papa and consider fpm=o defined by
M +.H rEG
--20 = 9-2G H
Ge(on.cr:)). (9)
where Hg =V, Eig Go oa g) is the Hessian matrix of the loss function with respect to 9.
Based on -- the label predictor becomes s-ao which defined by
Spx arg min Σ LY (i-pp vv)
(7)
f_Ro = arg min
Σ AE|n]-G
Ly (f(ō=zc (2:), .1). (10)
1=1
In total, we have the following lemma: Lemma 4.5. the CBM, if the label predictor utilizes linear transformations the form f with input c, then, for each y E M, we remove the r-th concept from and denote the new input as d'; set the r-th concept to 0 and denote the new input as c.. Then we have f-pu" c' = Spw== c for any input
Compared with the original loss before unlearning in (2), can observe two changes in (10). First, we remove |G] data points in the loss function Ly. Secondly, the input for the loss is also changed from ((v,) to 9- Therefore, it is difficult to estimate directly with an influence function. Here we introduce an intermediate label predictor as
j-zc arg min
Σ Lr((()(s)),(, (1I) *Ein]-G
Lemma 4.5 demonstrates that the retrained f-pas and fpu=o. when given inputs p-PM (x) and g"per (z) respec- tively, yield identical outputs. Consequently, we can utilize fo as the editing target in place of f-pms
and split the estimate of f-zo - into f-zo - f-zc and f-2o- ff.
Theorem 4.6. For the revised retrained label predictor fp defined by (7), we can edit the initial label predictor to Fpu=D by the following equation as a substitute for fpu=0:
Theorem 4.8. For dataset D = ((x;, Vi, Ci) given a set of data 2, = (Ir, yr. Cx), r G to be removed. The intermediate label predictor f- za is defined in (11). Then we have
fpa=D 2 Fpu=D= 2 f-H;'
Σ
Gy (z 9" f)
j-zo - ff H
Σ iEIn]-G
(r & Ac-
5

---

Editable Concept Bottleneck Models
We denote the edited version of f-zo as l=20= - f + Ag. Define BG
describe detailed visual features of birds. CelebA comprises 202,599 celebrity images, each annotated with 40 binary attributes that detail facial features, such as hair color, eye- glasses, and smiling. As the dataset lacks predefined clas- sification tasks, following (Espinosa Zarlenga et al., 2022), we designate 8 attributes as labels and the remaining 32 attributes as concepts. For all the above datasets, we follow the same network architecture and settings outlined in (Koh et al., 2020).
-H
Σ
Gr -r Gr (I
F. 50 1E(m|-G
where up - VV Gy f-za) is the Hes- sian matrix concerning ''=2a' Then f-zo can be estimated by i-zo= BG- Combining the above two-stage approxi- mation, then, the final edited label predictor f-za can be obtained by
Ground Truth and Baselines. We use retrain as the ground truth method. Retrain: We retrain the CBM from scratch by removing the samples, concept labels, or concepts from the training set. We employ two baseline methods: CBM-IF, and ECBM. CBM-IF: This method is a direct implemen- tation of our previous theorems of model updates in the three settings. See Algorithms 1-3 in Appendix for details. ECBM: As we discussed above, all of our model updates can be further accelerated via EK-FAC, ECBM corresponds to the EK-FAC accelerated version of Algorithms 1-3 (refer to Algorithms 6-8 in Appendix).
I- = --=0a + BG f + Aa + Bg.
(12)
Acceleration via EK-FAC. As mentioned Section 3, the loss function in CBMs is non-convex, meaning the Hessian matrices in all our theorems may not be well-defined. To address this, we adopt the EK-FAC approach, where the Hessian is approximated as Hg Go + 57. Here, Go rep- resents the Fisher information matrix of the model 0, and is a small damping term introduced to ensure positive definiteness. For details on applying EK-FAC to CBMs, see Appendix C.1. Additionally, refer to Algorithms 6-8 in the Appendix for the EK-FAC-based algorithms correspond- ing to our three levels. with their original (Hessian based) versions provided in Algorithms 1-3, respectively.
Evaluation Metric. We utilize two primary evaluation met- ries to assess our models: the Fl score and runtime (RT), Fl score measures the model performance by balancing pre- cision and recall. Runtime, measured in minutes, evaluates the total running time of each method to update the model.
Theoretical Bounds. We provide error bounds for the con- cept predictor between retraining and ECBM across all three levels; see Appendix D.2, E.2 and F.2 for details. We show that under certain scenarios, the approximation error be- comes tolerable theoretically when leveraging some damp- ing term 8 regularized in the Hessian matrix.
Implementation Details. Our experiments utilized an Intel Xeon CPU and an RTX 3090 GPU. For utility evaluation, at the concept level, one concept was randomly removed for the OAT dataset and repeated while ten concepts were randomly removed for the CUB dataset, with five different seeds. At the data level, 3% of the data points were randomly deleted and repeated 10 times with different seeds. At the concept-label level, we randomly selected 3% of the data points and modified one concept of each data randomly, repeating this 10 times for consistency across iterations.
5. Experiments
In this section, we demonstrate our main experimental results on utility evaluation, edition efficiency, and inter- pretability evaluation. Details and additional results are in Appendix H due to space limit.
5.2. Evaluation of Utility and Editing Efficiency
Our experimental results, as illustrated in Table 1, demon- strate the effectiveness of ECBMs compared to traditional retraining and CBM-IF, particularly emphasizing computa- tional efficiency without compromising accuracy. Specif- ically, ECBMs achieved Fl scores close to those of re- training (0.8808 vs. 0,8825) while significantly reducing the runtime from 297.77 minutes to 2.36 minutes. This pattern is consistent in the CUB dataset, where the run- time was decreased from 85.56 minutes for retraining to 0.65 minutes for ECBMs, with a negligible difference in the F1 score (0.7971 to 0.7963). These results highlight the potential of ECBMs to provide substantial time sav- approxppxxillately 22-30% of the computational time required for retraining- while maintaining comparable ac- curacy. Compared to CBM-IF, ECBM also showed a slight
5.1. Experimental Settings
Dataset.We We utilize three datasets: X-ray Grading (OAI) (Nevitt al., 2006), Bird Identification (CUB) (Wah et al., 2011), and the Large-scale CelebFaces Attributes Dataset (CelebA) (Liu et al., 2015). OAI is a multi-center ob- servational study of knee osteoarthritis, comprising 36,369 data points. Specifically, we configure n=10 concepts that characterize crucial osteoarthritis indicators such as joint space narrowing, osteophytes, and calcification. Bird iden- tification (CUB)² consists of 1 11,788 data points, which be- long to 200 classes and include 112 binary attributes to
TThe original dataset is processed. Detailed explanation can be found in
6

---

Editable Concept Bottleneck Models
Table I: Performance comparison of different methods on the three datasets.
OAI
CUB
CelebA
Edit Level
Method
Fl score
RT (minute)
Fl score
RT (minute)
Fl score
RT (minute)
Retrain
0.8825=0.0054
297.77433.01
0.7971=0.0066
85.56:14.22
0.3827:10.0722
304.71=35.15
Concept Label
CRM-IF(Ours)
0.0039+000003
4.63++0.89
0.7699+0.0003
1.33++0.09
0..561++0.134
5.84+0.82
ECBM(Ours)
0.8008=0000339
2.36=0.54
0.7963==00005
0.65=10.08
0.3845=0.0327
0.59
Retrain
0.8448.1..0191
258. 44.14444
0,7811 10.0047
87.21 1.262
0.377610.0335
355 85 _2,399
Concept
CBM-IF(OUrs)
0.8214=000071
4,94++0911
0.7579++0.006
1.45+0008
0.3609++0.0200
5.51+0.889
ECBM(Ours)
0.8403-10.0000
2.36110.60
0.7787:10.0058
0.59110.17
0.076110.0280
0.58
Retrain
0.8811=0.0005
319.37+-31.0 1.08
1.7838++0.005
86,20++7.74
0.3797
325 62 05.52950.59
Data
CBM-IF(Qurs)
0.8472=20.0046
5.07:00.75
0,7623++0.00 0,031
1.46=00008
0.3536+10.0166
0.97
ECur()urs)
0.8797=0000038
2.50=:0.49
0.7827=000088
0.65=0019
0.3748==0.0347
0.49=0061
reduction in runtime and significant improvement in Fl score. The former verifies the effective acceleration of our algorithm by EK-FAC This efficiency is particularly crucial in scenarios where frequent updates to model annotations are needed, confirming the utility of ECBMs in dynamic environments where running time and accuracy are critical.
first select 1-10 most influential and 1-10 least influential concepts by our influence function. Then, we will remove these concepts and update the model via retraining or our ECBM and analyze the change (F1 Score Difference) w.r.t. the original CBM before removal.
The results in Figure 3(a) demonstrate that when we remove the 1-10 most influential concepts identified by the ECBM method, the Fl score decreases by more than 0.025 com- pared to the CBM before removal. In contrast, Figure 3(b) shows that the change in the F1 score remains consistently below 0.005 when removing the least influential concepts. These findings strongly indicate that the influence function in ECBM can successfully determine the importance of con- cepts. Furthermore, observe that the gap between the Fl score of retraining and ECBM is consistently smaller than 0.005, and even smaller in the case of least important con- cepts. This further suggests that when ECBM edits various concepts, its performance is very close to the ground truth.
We can also see that the original version of ECBM, i.e., CBM-IF, also has lower runtime than retraining but lower F1 score than ECBM. Such results may be due to dif- ferent reasons. For example, our original theorems depend on the inverse of the Hessian matrices, which may not be well-defined for non-convex loss. Moreover, these Hessian matrices may be ill-conditioned or singular, which makes calculating their inverse imprecise and unstable.
Editing Multiple Samples. To comprehensively evalu- ate the editing capabilities of ECBM in various scenarios, we conducted experiments on the performance with mul- tiple samples that need to be removed. Specifically, for the concept label/data levels, we consider the different ra- tios of samples (1-10%) for edit, while for the concept level, we consider removing different numbers of concepts E {2, 4, 4, .................... 20). We compared the performance of re- training, CBM-IF, and ECBM methods. As shown in Figure 2. except for certain cases at the concept level, the Fl score of the ECBM method is generally around 0.0025 lower than that of the retrain method, which is significantly better than the corresponding results of the CBM-IF method. Recall- ing Table 1, the speed of ECBM is more than three times faster than that of retraining. Consequently, ECBM is editing method that achieves trade-off between speed and effectiveness.
ECBMs can erase data influence. For the data level, ECBMs aim facilitate an efficient removal of samples. We perform membership inference attacks (MIAs) to provide direct evidence that ECBMs can indeed erase data influence. MIA is privacy attack that aims to infer whether specific data sample was part of the training dataset used to train a model. The attacker exploits the model's 's behavior, such as overconfidence or overfitting, to distinguish between train- ing (member) and non-training (non-member) data points. In MIAs, the attacker typically queries the model with a data sample and observes its prediction confidence or loss values, which tend to be higher for members of the training set than non-members (Shokri et al., 2017).
To quantify the success of these edits, we calculate the RMIA (Removed Membership Inference Attack) score for each category. The RMIA score is defined as the model's confidence in classifying whether given sample belongs to the training set. Lower RMIA values indicate that the sample behaves more like a test set (non-member) sam- ple (Zarifzadeh et al., 2024). This metric is especially cru- cial for edited samples, as a successful ECBM should make the removed members behave similarly to non-members,
5.3. Results on Interpretability
ECBM can measure concepts importance. The origi- nal motivation of the influence function is to calculate the importance score of each sample. Here, we will show that the influence function for the concept level in Theorem 4.4 can be used to calculate the importance of each concept in CBMs, which provides an explainable tool for CBMs. In detail, we conduct experiments on the CUB dataset. We
7

---

Editable Concept Bottleneck Models
Concept-labelleeel
Concept-level
Data-level
01150
0.781
Datrah
6/85
0.782
0.703
F1 Score
8,900
Score
0.700
0.735
0.775
0.90
6.365
6.75
25
115 50 75
32,5 150
22.6
Ratlo of Edition
Number of Deletion
Ratio of Deletion
Figure 2: Impact of edition ratio on three settings on CUB dataset.
F1 1.oorre Difference
F1 Scooe Difference
Number of Deletions
Number of Deletions
(a) Results on the 1-10 most influential concepts
(b) Results on the 1-10 least influential concepts
Figure 3: F1 score difference after removing most and least influential concepts given by ECBM.
Before Editing
After Editing
182
Noms arnn Memoort APEA
1.0
Nammal decribatioon of foonaved meriannd AMAA
ret-menbor
nor innntars
82
30
62-
60-
33
RMIA
RMIA
(a) RMIA Score Before Editing
(b) RMIA Score After Editing
Figure 4: RMIA scores of data before and after removal.
reducing their membership vulnerability. See Appendix H for its definition.
closer to the non-members' mean score. This shift in RMIA values demonstrates the effectiveness of ECBMs in editing the model, as the removed members now exhibit behavior closer to that of non-members. The post-editing RMIA score distributions are shown in Figure 4(b). These results provide evidence of the effectiveness of ECBMs in editing the model's 's knowledge about specific samples.
We conducted experiments by randomly selecting 200 sam- ples from the training set (members) and 200 samples from the test set (non-ment) (non-members) of the CUB dataset. We calcu- lated the RMIA scores for these samples and plotted their frequency distributions, as shown in Figure 4(a). The mean RMIA score for non-menne memberr was 0.049465, while mem- bers had a mean score of 0.063505. Subsequently. we ap- plied ECBMs to remove the 200 training samples from the model, updated the model parameters, and then recalculated the RMIA scores. After editing, the mean RMIA score for the removed members decreased to 05, significantly
6. Conclusion
In this paper, we propose Editable Concept Bottleneck Models (ECBMs). ECBMs can address issues of remov- ing/inserting some training data or new concepts from
8

---

Editable Concept Bottleneck Models
trained CBMs for different reasons, such privacy con- cerns, data mislabelling, spurious concepts, and concept annotation errors retraining from scratch. Furthermore, to improve computational efficiency, we present streamlined versions integrating EK-FAC. Experimental results show our ECBMs are efficient and effective.
Bourtoule, L.., Chandrasekaran, V., Choquette- C. A.,
Jia, H., Travers, A., Zhang, B., Lie, D., and Papernot, N. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pp. 141-159, IEEE, 2021.
Brophy, J. and Lowd, D. Machine unlearning for random forests. In International Conference on Machine Learn- ing, pp. 1092-1104, PMLR, 2021.
Impact Statement
Brunet, M.-E.,
Houlihan, C., Anderson, A., and Zemel, R. Understanding the origins of bias in word embeddings. In International conference machine learning, pp. 803-811. PMLR, 2019.
This research propose editable concept bottleneck models. While we acknowledge the importance of evaluating societal impacts, our analysis suggests that there are no immediate ethical risks requiring specific mitigation measures beyond standard practices in machine learning.
Cao, L. Ai in finance: challenges, techniques, and oppor- tunities. ACM Computing Surveys (CSUR), 55(3):1-38, 2022.
Acknowledgements
Lijie Hu, Chenyang Ren, Cheng-Long Wang, and Di Wang are supported in part by the funding BAS/I/1689-01-00 URFF1/44633001001 REI/1/5232-01-00 REW/1/5332-01-01 and URE/1/5508-01-01 from KAUST, and funding from KAUST Center of Excellence for Generative AI, un- der award number 5940. Zhengyu Hu, Hongbin Lin, and Hui Xiong are supported by the National Key R&D Pro- gram of China (Grant No. 2023YFF0725001) the Na- tional Natural Science Foundation of China (Grant No. 92370204), the Guangdong Basic and Applied Basic Re- search Foundation (Grant No. 2023B1515 11531151100057) the Guangzhou-HHKUSTCCZ Joint Funding Program (Grant No. 2023A03J0008), and the Education Bureau of Guangzhou Municipality.
Cao, Y. and Yang. J. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy, pp. 463-480. IEEE, 2015.
Chauhan, K., Tiwari, R., Freyberg, J., Shenoy, P., and Dvi- jotham, K. Interactive concept bottleneck models. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 37(5), pp. 3948-5955, 2023.
Chen, C.,, Sun, F., Zhang, M., and Ding, B. Recommenda- tion unlearning. In Proceedings of the ACM Web Confer- ence 2022, pp. 2768-2777, 2022a.
Chen, H., Si, S., Li, Y.. Chelba, C., Kumar, S., Boning, D., and Hsieh, C.-J. Multi-stage influence function. Advances in Neural Information Processing Systems, 33:12732- 12742, 2020.
References
Agarwal, N., Bullins, B., and Hazan, E. Second-order stochastic optimization for machine learning in linear time. Journal of Machine Learning Research, 18(11)): 1-40, 2017.
Chen, M., Zhang, Z., Wang, T., Backes, M., Humbert, M., and Zhang, Y. Graph unlearning. In Proceedings of the 2022 ACM SIGSAC conference on computer and commu- nications security, pp. 499 503,51113100 2022b.
Ahmad, M. A., Eckert, C., and Teredesai, A. Interpretable machine learning in healthcare. In Proceedings of the 2018 ACM international conference on bioinformatics, computational biology. and health informatics, pp. 559- 560, 2018.
Cheng, K., Lin, G., Fei, H., Yu, L.,, Ali, M. A., Hu, I.., Wang, D., et al. Multi-hop question answering under temporal knowledge editing. arXiv preprint urXiv:2404.00492, 2024.
Chowdhury, S. B. R., Choromanski, K., Sehanobish, A., Dubey, A., and Chaturvedi, S. Towards scalable exact machine unlearning using parameter-efficien fine-tuning. arXiv preprint arXiv:2406.162577 2024.
Ali, M. A., Li, Z., Yang. S., Cheng, K., Cao, Y., Huang, T., Hu, L., Yu, L., and Wang, D. Prompt-saw: Leveraging relation-aware graphs for textual prompt compression. arXiv preprint arXiv:2404.00 00449904999..... 2024.
Bartlett, M. Approximate confidence intervals. Biometrika, 40(1/2):12-19, 1953.
Cook, R. D. Detection of influential observation in linear regression. Technometrics, 42(1):65-68, 2000.
Basu, S., Pope, P., and Feizi, S. Influence functions in deep learning are fragile. In International Conference on Learning Representations (ICLR), 2021.
Cook, R. D. and Weisberg, S. Characterizations of an em- pirical influence function for detecting influential cases regression. Technometrics, 22(4):495-508 1980.

---

Editable Concept Bottleneck Models
Das, A. and Rad, P. Opportunities and challenges in explain- able artificial intelligence (xai): A survey. arXiv preprint arXiv:2006. 11371, 2020.
the AAAI Conference Artificial Intelligence, volume 37(11), pp. 12907-12915, 2023b.
Izzo, Z., Smart, M. A., Chaudhuri, K., and Zou, J. Approx- imate data deletion from machine learning models. In International Conference Artificial Intelligence and Statistics, pp.
Zarlena, M., Barbiero, P., Ciravegna, G., Marra, G., Giannini, F., Diligenti, M., Shams, Z., Precioso, F., Melacci, S., Weller, A., Lió, P., and Jamnik, M. Concept embedding models: Beyond the accuracy-explaine trade-off. In Advances in Neural Information Processing Systems, volume 35, 2022.
Zarlenga,
PMLR, 2021.
Keser, M., Schwalbe, G., Nowzad, A., and Knoll, A. Inter- pretable model-agnostic plausibility verification for 2d object detectors using neck models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 3890-3899, 2023.
concept bottle-
George, T., Laurent, C., Bouthillier, X., Ballas, N., and Vincent, P. Fast approximate natural gradient descent in kronecker factored eigenbasis. Advances in Neural Information Processing Systems, 31, 2018.
Kim, E., Jung, D., Park, S., Kim, S., and Yoon, S. Prob- abilistic concept bottleneck models. arXiv preprint arXiv:2306.01574, 2023.
Golatkar, A., Achille, A., and Soatto, S. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 9304-93003 9312, 2020a.
Koh, P. W. and Liang, Understanding black-box predic- tions via influence functions. In International conference machine learning, pp. 1885-1894. PMLR, 2017.
Golatkar, A., Achille, A., and Soatto, S. Forgetting out- side the box: Scrubbing deep networks of information accessible from input-output observations. In Computer Vision-ECCV 2020: 16th European Conference, Glas- gow, UK, August 23-28, 2020, Proceedings, Part XXIX 16, pp. 383-398. Springer, 2020b.
Koh, P. W., Nguyen, T., Tang, Y. Mussmann, S., Pierson, E., Kim, B., and Liang, P. Concept bottleneck models. In International conference machine learning, pp. 5338- 5348. PMLR, 2020.
Golatkar, A., Achille, A., Ravichandran, A., Polito, M., and Soatto, S. Mixed-privacy forgetting in deep networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 792-801, 2021.
Kwon, Y., Wu, E., Wu, K., and Zou, J. Datainf: Efficiently estimating data influence in lora-tuned Iims and diffu- sion models. In The Twelfth International Conference on Learning Representations, 2023.
Guo, C., Goldstein, T., Hannun, A., and Van Der Maaten, L. Certified data removal from machine learning models. arXiv preprint arXiv: 1911.99030, 2019.
Lai, S., Hu, L., Wang, J., Berti-Equille, L., and Wang, D. Faithful lision-language interpretation via concept bot- tleneck models. The Twelfth International Conference on Learning Representations, 2023.
Guo, H., Rajani, N., Hase, P., Bansal, M., and Xiong, C. Fastif: Scalable influence functions for efficient model interpretation and debugging. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 10350, 2021.
Liu, J., Lou, J., Qin, Z., and Ren, K. Certified minimax unlearning with generalization rates and deletion capacity, Advances in Neural Information Processing Systems, 36, 2024.
Han, X., Wallace, B. C., and Tsvetkov, Y. Explaining black box predictions and unveiling data artifacts through influ- ence functions. arXiv preprint arXiv:2005.0666 06676600000600..0. 2020.
Liu, 7.,, Luo, P., Wang, X., and Tang, X. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference computer vision, pp. 3730- 3738, 2015.
Havasi, M., Parbhoo, S., and Doshi- Velez, Addressing leakage in concept bottleneck models. Advances in Neu- ral Information Processing Systems, 35:23386-23397, 2022.
Mehta, R., Pal, S., Singh, V., and Ravi, S. N. Deep unlearn- ing via randomized conditionally independent hessians. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 10422-10431, 2022.
Hu, L., Liu, Liu, N., Huai, M., Sun, L., and Wang, D. Improving faithfulness for vision transformers. arXiv preprint arXiv:2311.1799 2023a.
Hu, L., Liu, Y., Liu, N., Huai, M., Sun, L., and Wang, D. Seat: stable and explainable attention. In Proceedings of
Nevitt, M., Felson, D., and Lester, G. The osteoarthritis initiative. Protocol for the cohort study, 1:2, 2006.
10

---

Editable Concept Bottleneck Models
Oikarinen, T., Das, S., Nguyen, L., and Weng. L. Label-free concept bottleneck models. In International Conference on Learning Representations, 2023.
Wu, Y., Dobriban, E., and Davidson, S. Deltagrad: Rapid re- training of machine learning models. In International Conference on Machine Learning, pp. 10355-10366. PMLR, 2020.
Rozenfeld, Y., Beam, J., Maier, H., Haggerson, W., Boudreau, K., Carlson, J., and Medows, R. A model of disparities: risk factors associated with covid-19 infec- tion. International journal for equity in health, 19(1):126, 2020.
Xu, X., Kong, K., Liu, N., Cui, L., Wang, D., Zhang, J., and Kankanhalli, M. An Ilm can fool itself: A prompt-based adversarial attack. arXiv preprint arXiv:2310.13345, 2023.
Sattar, N., McInnes, B., and McMurray, J. J. Obesity is risk factor for severe covid-19 infection: multiple potential mechanisms, Circulation, 142(1):4-6, 2020.
Yang, S., Ali, M. A., Wang, C.-L., Hu, L., and Wang, D. Moral: Moe augmented lora for 1lms lifelong learning. arXiv preprint arXiv:2402. 1260, 2024a.
Yang, S., Hu, L., Yu, L.., Ali, M. A., and Wang, D. Human-ai interactions in the communication era: Autophagy makes large models achieving local optima. arXiv preprint arXiv: 11271, 2024b.
Sawada, Y. and Nakamura, K. Concept bottleneck model with additional unsupervised concepts. IEEE Access, 41758-41765, 2022.
Sckhari, A., Acharya, J., Kamath, G., and Suresh, A. T. Re- member what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Processing Systems, 34:18075-18086 2021.
Yang. S., Su, J., Jiang, H., Li, M., Cheng, K., Ali, M. A., Hu, L., and Wang, D. Dialectical alignment: Resolving the tension 3h and security threats of Ilms. arXiv preprint arXiv:2404.000 2024c.
Sheth, I. and Ebrahimi Kahou, S. Auxiliary losses for learn- ing generalizable concept- based models. In Oh, A., Nau- mann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Pro- cessing Systems, volume 36, pp. 16966-26990, 2023.
Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., and Chen, A survey on multimodal large language models. arXiv preprint arXiv:2306. 13549, 2023.
Yu, K.-H., Beam, A. L.,, and Kohane, I. S. Artificial intel- ligence healthcare. Nature biomedical engineering, (10):719-731, 2018.
Sheth, I. and Kahou, S. E. Auxiliary losses for learning generalizable concept-based models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Yuksekgonul, M., Wang, M., and Zou, J. Post-hoc con- cept bottleneck models. In The Eleventh International Conference on Learning Representations, 2023.
Shokri, R., Stronati, M., Song, C., and Shmatikov, V. Mem- bership inference attacks against machine learning mod- cls. In 2017 IEEE symposium on security and privacy (SP), pp. 3-18. IEEE, 2017.
Zarifzadeh, S., Liu, P., and Shokri, R. Low-cost high-power membership inference attacks. In Forty-first International Conference on Machine Learning, 2024.
Thudi, A., Deza, G., Chandrasekaran, V., and Papernot, N. Unrolling sgd: Understanding factors influencing ma- chine unlearning. In 2022 IEEE 7th European Symposium Security and Privacy (EuroS&P), pp. 303-313-330 319. IEEE, 2022.
Zhao, W. X., Zhou, K., Li, Li, Tang, T., Wang, X., Hou, Y.,
Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. A survey of large language models. arXiv preprint arXiv: 18223, 2023.
Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. The caltech-uesd birds-200-2011 dataset. California Institute of Technology, 2.1
Wang, H., Ustun, B., and Calmon, F. Repairing without retraining: Avoiding disparate impact with counterfactual distributions. In International Conference on Machine Learning, pp.
PMLR, 2019.
Warnecke, A., Pirch, L., Wressnegger, C., and Rieck, K. Machine unlearning of features and labels. Network and Distributed System Security (NDSS) Symposium, 2023.

---

Editable Concept Bottleneck Models
A. Notation Table
Table 2: Notation Table.
Symbol
Description
c (pi,...,PH
Set of concepts provided by experts or LLMs.
D {=
Training dataset, where 2; (Ira; Vir
E Rd,
Feature vector for the 2-th sample.
M1 E Rd
Label for the i-th sample, with d2 being the number of classes.
= (c)...., E R
Concept vector for the i-th sample.
Corrected concept label for the w-th sample and r-th concept.
c
Weight of the concept P/ in the concept vector C;-
9: R + R
Concept predictor mapping input space to concept space. Label predictor mapping concept space to prediction space.
f: Rdd
(g((()))))
Loss function for the j-th concept predictor.
Ly((((z))),
Loss function from concept space to output space. Gradient of the loss (g(x)),)) c.).
Co(ai,
Gy (I(,
Gradient of the loss Ly (f(g(z))).c)
Hg
Hessian matrix of the loss function with respect to 0.
G
Fisher information matrix of model Å.
6
Damping term for ensuring positive definiteness of the Hessian. Estimated concept predictor.
9
f
Estimated label predictor.
Retrained concept predictor after correcting erroneous data.
fe.
Retrained label predictor after correcting erroneous data. Retrained concept predictor after removing concepts indexed by M. Mapped concept predictor with the same dimensionality g.
9-PM
9 8-pM
9 8-PM
Approximation of the retrained concept predictor 9-pas
Label predictor after setting the r-th concept zero for E M.
5PM==
Approximation of the label predictor fpw=0- Hessian matrix of the loss function with respect to ğ.
Hg
H
Hessian matrix of the loss function with respect to f. Set of erroneous concept indices to be removed. Set of indices of samples to be removed from the dataset. Data sample to be removed, where r E G.
M C [k]]
G C [n]
2T (Fr: Vr: C.)
9-z0
Retrained concept predictor after removing samples indexed by G.
--20
Approximation of the retrained concept predictor -- Intermediate label predictor.
f-10
f f-s0
Final edited label predictor after removing samples indexed by G.
B. Influence Function
Consider neural network i = arg ming Ei= f(z;, 0) with loss function L and dataset D = {2;) That is minimize the empirical risk
R(0)= Σ L(22, 8)
Assume is strongly convex in Then uniquely defined. If we remove a point 2n from training dataset, the parameters become θ_ =m arg ming Σism L(z,, O). Up-weighting 2m by e small enough, then the revised risk
R(0) = Σn L(z;; 0) + <zimm 0) is still strongly convex. Then the response function θ.-z... R(O) is also uniquely defined. The parameter change is denoted as A. 0 - ô. Since i--z.. is the minimizer of R(0)', we have the
12

---

Editable Concept Bottleneck Models
first-order optimization condition
R(0) + c.
0,,- L(zm: -e.-
Since - base - perform Taylor expansion of the right-hand side:
VR(0) ( 0)] + v2 + +
+
A. R
Noting c <V2I A. is (((l term, which smaller than other parts, we drop the following analysis. Then the Taylor expansion equation becomes
VR(0) VV( V2R(O) A 00
Solving for A., we obtain:
A. V [²)( + +VV(2,)
Remember 0 minimizes R. then VR(O) 0. Dropping o(e) term, we have
A -
db dûc. 2m
dA. =-H,-VI(2,0) Tup paramas (2)-
dc
dc
c=0
l==0
Besides, we obtain the approximation of i
am directly by 0_am= 0 + Iup.params (2).
C. Acceleration for Influence Function
EK-FAC. EK-FAC method relies on two approximations to the Fisher information matrix, equivalent to G3 in our setting, which makes it feasible to compute the inverse of the matrix.
Firstly, assume that the derivatives of the weights in different layers are uncorrelated, which implies that Gj has a block- diagonal structure. Suppose ğo can be denoted by go (x) gez o.... ga, o...o go, (I)) where E [L]. We fold the bias into the weights and vectorize the parameters in the (-th layer into vector 0G Rd di N is the number of I-th layer
parameters. Then Gj can be reaplcaed by (G)) GL(()) where Gi(D) _ n Ei= Vo
Denote h1, as
the output and prtvvaciveeed output of 1-th layer. Then G)( can be approximated by
G((0) ~ ())
A n
1-1 (x,) h1 (r.)
4 Tr.
n
Furthermore, in order to accelerate transpose operation and introduce the damping term, perform eigenvalue decomposition of matrix Q4-1 and I and obtain the corresponding decomposition results as QaARQ and QrArQT. Then the inverse of (f) can be obtained by
(1, x (Gi + >:da.)) = -Qa, & Qr,) (Ao,,, @ Ar, + +vla,)) (Q @ Qr.)"
Besides, George et al. (2018) proposed new method that corrects the error in equation 13 which sets the i-th diagonal element of AQ_,, & Ar, as A; n Σ/= ((Qa.. @ Qr.) Voll)
C.I. EK-FAC for CBMs
In our CBM model, the label predictor is single linear layer, and Hessian computing costs are affordable. However, the concept predictor is based on Resnet-18, 18, which has many parameters. Therefore, perform EK-FAC for ğ.
9 = arg min
Lc, = arg min
Lold (x))..))
j=1
13

---

Editable Concept Bottleneck Models
define H3 E,, Lc, (g(2)))))) (e) the Hessian matrix of the loss function with respect to the parameters.
To this end, consider the (-th layer of 9 which takes input layer of activations {ag,} where j {1, ...,...... J indexes the input map and t E T indexes the spatial location which is typically 2-D grid. This layer is parameterized by a set of weights W = (w),).5) and biases = (b,). where i E {i...., } indexes the output map, and & A indexes the spatial offset (from the center of the filter).
The convolution layer computes a set of pre-activations as
[Si], Si, Σ w. + bç-
Denote the loss derivative with respect to as
0 3Σ LC 08it
Dsa
which can be computed during backpropagation.
The activations actually stored A of dimension [7] J. Similarly, the weights are stored / | J array Wi. The straightforward implementation of convolution, though highly parallel in theory, suffers from poor memory access patterns. Instead, efficient implementations typically leverage what is known as the expansion operator [-] For instance, [Ai-1] is [T] x J|A| matrix, defined as
1A-111 +5 [ALille+a) 0j,4+8
In order to fold the bias into the weights, we need to add homogeneous coordinate (i.e. a column of all I's) to expanded activations [At-1] and denote this as [A--1]H. Concatenating the bias vector to the weights matrix, then we have = (bi, Wi).
Then, the approximation for Ha is given as:
(> aastDDDDs )
(O (g) =E [Dws. DWi
E
Σ
VEET
1 E [DS] D.i & Ω 1r.
EE [|AI-I
T
Estimate the expectation using the mean of the training set,
GO
Σ
DS DS) á oñ.
Furthermore, if the factors Q and i have eigen decomposition QnAnQd and Qr respectively, then the eigen decomposition of Ω i can be written as:
Ω @ f, = QnAnQ @ QrArQ
(Qn s Qr) (Aa @ Ar) (Qn Qr)
Since subsequent inverse operations required and the current approximation for (() is PSD, actually use damped version as
(i()) = (Gi (g) + A:da;) (Qu.., & Qr.) (Am., & Ar, + Araa)) (Qa.,, & Qr.)
(13)
Besides, (George et al., 2018) proposed a new method that corrects the error in equation 13 which sets the i-th diagonal element of Agr & Ar, as
A n- - 1
Σ (QQr. @ @r.) Vo,e),
14

---

Editable Concept Bottleneck Models
D. Concept-label-leve Influence
D.I. Proof Concept-label--eve Influence Function
We have a set of erroncous data De and its associated index set Sg S [n] x [k] such that for each (w, E Se, we have (Tess Ywu Cw) De with en mislabeled and e its corrected concept label. Thus, our goal is to approximate the new CBM without retraining.
Proof Sketch. Our goal edit 9 and f to de and fe- (i) First, we introduce new parameters Ic,e that minimize modified loss function with small perturbation c. (ii) Then, perform Newton step around and obtain estimate for ge- (iii) Then, we consider changing the concept predictor at one data point (Fi, Migr Ci.) and retraining the model to obtain new label predictor fi., obtain approximation for ft.- (iv) Next, we iterate ie over 1,2, n, sum all the equations together, and perform Newton step around obtain an approximation for fe. (v) Finally, we bring the estimate of ğ into the equation for fe to obtain the final approximation.
Theorem D.1. Define the gradient of the j-th concept predictor of the i-th data xa as
Go( (o(
(14)
then the retrained concept predictor ge defined by
ge = arg min
Σ cg(z))))c) (1,))
E Le (o(a)).)))
(15)
(i.)es.
can be approximated by Ie, defined by:
0 - --H
Σ
(Goldim (e (a Cwi 9))
(16)
(1.r)).8
where H-= = Vg (z;)), c.) is the Hessian matrix of the loss function with respect to g.
Proof. For the index r) E Se, indicating the r-th concept of the with data is wrong, we correct this concept c to T.
Rewrite ge as
ge arg min
Σ La (o(x),,)) (w,r)eS.
Σ
Le Cw)
E
Le (o(o))
(17)
(w,r)ES.
To approximate this effect, define new parameters Ge,e as
De.e A arg min
Σ a)) (w,r)
Σ
e L. (g(sm);
Σ (wrreS.
e L. (g(2m))
(18)
i
Then, because Ie,e minimizes equation 18, we have
9 Σ L' (z)) (w.res.
Σ e V.VgLe (u)
Σ (w.reS.
e.
= 0.
Perform Taylor expansion of the above equation at ğ.
Σ 1/ (o(x.),))+
Σ
V.VgLe ( Σ
VgV' (f(#w),
9
(w.rleS.
(w,r)s.
?
Σ
1/ (0(z;), ci) (gc. 9) ~ 0.
(19)
15

---

Editable Concept Bottleneck Models
Because of equation 15, the first term of equation 19 equals 0. Then have
ge,e -
Σ e. .. (V&Le (0(Iw), VoLic (I(Iw).
(wir)ES,
where
H1 = Σ
Le (0(z.).
Then, we do a Newton step around 9 and obtain
Je
9c
High
Σ
Vi (i(Tw), Em) - V Volic (3(#)w) Gw))
(20)
(s,,))S..
Define the gradient of the j-th concept predictor of the i-th data
Go c59) - v (o(a),
(21)
then the retrained concept predictor de defined by
ge arg min
Σ (1,)ES
L'C cc(
E L' (g(as). (k,))S,
(22)
can be approximated by Ge, defined by:
H-
Σ (w.r)ES,
(Go Twigg (Tu Cwi g))
(23)
Theorem D.2. Define the gradient of the label predictor of the i-th data 2; as
Gy(zag. f) & V (f(g(z:)). v1)
(24)
then the retrained label predictor fe defined by
fe arg min
Ly (f (āc (xx)) Σ
can be approximated by:
H Σ (Crloen). - Gy(asiee)))
where H v Ei= Gy(ag f) is the Hessian matrix, and Ge is given in Theorem D.I.
Proof. Now we come to deduce the edited label predictor towards fe.
First, we consider only changing the concept predictor at one data point (xi Bi. ci.) and retrain the model to obtain a new label predictor fr..
f. = arg min
Σ
n
Ly (I + Ly (s (ga (xz.)) .wn))
5=1.8$1e
We rewrite the above equation as follows:
f.. arg min
Ly (f(g(z))) ve) + Ly (f (ge (zi.)) vo.)) Ly (f (9 (xx,))) (et.)))uu ve.)
16

---

Editable Concept Bottleneck Models
We define Le,se as:
fes arg arg min Σ Ly (x))) m.) +e- Ly (f (9e (xi.)) , - e. Ly (f (Ixr)) Mao)
Derive with respect to f at both sides of the above equation. we have
Lr (feie (0 (2;)), vi) te vjLr (fese (āe (Iie)) : Vio) c. viLrr (fese (0 (zi.)) vi.) =0
Perform Taylor expansion of the above equation f.
Ly (Z((())) 31 te. + (f (as.). vh.)
Vilr (f(( v3 Σ Ly (f(( (fese -f) =0
Then we have
Se, - f ~ - H v (Ir (1 (xi))) Ly
where 11 11 = Ei= -y Ly (zx)) v1)
Iterate ie over 1, 1,2, ... n, and sum all the equations together, we can obtain:
f= c. cc Hj' Σ v [Ly (f (m:)) - Ly (f(())))
Perform Newton step around f and we have
n ie f
11 f- Ev v (Lx ([e Lr (I:)))))
(25)
Bringing the edited (20) of into equation (25), we have
fe zf Σv v LL (())) Ly (f(( & fe.
Define the gradient of the label predictor of the i-th data x; as
(r(gg. & V (f(g(a:)),u),
(26)
then the retrained label predictor fa defined by (4) can be approximated by fer defined by:
f H Σ (Gr (r
where H; El Gy 9. f) is the Hessian matrix, and ge is given in Theorem D.1.
17

---

Editable Concept Bottleneck Models
D.2. Theoretical Bound for the Influence Function
Consider the dataset D [z ca. Mi 1", the loss function of the concept predictor g is defined as:
k Σ 2m 1m1
LTeaal (D; (D;
Σ Lc(g(z)).)) + || ΣΣ (c( + ||911
(a,) log(ee) + (
|1911
Mathematically, we have a set of erroneous data D. and its associated index set S. S [n] x [k] such that for each (w, r) E Se, we have (Iar: Vue Cw) E De with d mislabeled and T is corrected concept label. Thus, our goal is to estimate the retrained CBM. The retrained concept predictor and label predictor will be represented in the following manner.
ge = arg min
Σ (L,)E
L' (i(
L' (g(2,),)) + |Ial? (k,))S.
E
(27)
Define the corrected dataset as D Then the loss function with the influence of erroneous data D. removed becomes
(T(D";g)
Σ ce( (1,)S
E (c(, 2 -11111
(28)
(K.)ES
Assume ÿ - arg min Toxal (Dgg) is the original model parameter, and (D*) is the minimizer of (D ;g), which is obtained from retraining. Denote ge (D*) as the updated model with the influence of erroneous data De removed and obtained by the influence function method in theorem D.1, which an estimation for 9. (D*).
To simplify the problem, we concentrate the removal of erroneous data D. and neglect the process of adding the corrected data back. Once obtain the bound for (D)) ge (D*) under this circumstance, the bound for the case where the corrected data is added back can naturally be derived using a similar approach. For brevity, we use the same notations.
Then, the loss function L (-("";g) becomes
8 (l(:))
I. (D)
Σ (1))s
||oll = Troaal (D:g)
Σ (1,))ES.
The (g(2s),
(29)
And the definition of (D() becomes
+ H.y
Σ (w.r)ES.
GolLww Cwi
(30)
where H3 = - c)) - 8 is the Hessian matrix of the loss function with respect to ÿ. Here 8 1 is a small damping term for ensuring positive definiteness the Hessian. Introducing the damping term into the Hessian essentially equivalent to adding regularization term to the initial loss function. Consequently, can also be interpreted as the regularization strength.
In this part, will study the error between the estimated influence given by the theorem D.1 method and retraining. We the parameter changes as the evaluation metric:
[g 9) (ge -0)1 = rel
(31)
Assumption D.3. The loss Le(x,c;g)
Le(z,eg;1) =L (g(x))
convex and twice-differeeniable in g. with positive regularization > There exists CH E IR such that
Lc,,, V S Cnllgi 92l12
for all (x. E [(m, [-) j E [k] and 91.92 E T.
18

---

Editable Concept Bottleneck Models
Then the function (D, Se:g);
L'(D, (e:)) = Σ ()) =
E
(65
(K.)ES.
(1.))..
is convex and twice-differenniable in with some positive regularization, Then we have
( S [8 Ch 1 92ll2
for 91, 92 E T. Corollary D.4.
(D" :9)) (D' ( (D': ((nk + (Sel) Ch) ll 922l
Define C₁ - (nk [Sel) Cи
Definition D.5. Define (D) as the number of pairs
C = LVD,,,
A'in smallest singular value of I. ;D)
o min smallest singular value of v y Total (D: g).
Based above corollaries and assumptions, derive the following theorem. Theorem D.6., We obtain the error between the actual influence and our predicted influence as follows:
( -0a(D")
26 + mmin + o'min
+
C₁,
2 +033
(6 + )mmin) min)
Proof.
We will use the one-step Newton approximation intermediate step. Define Agne(D")
Agn)(D") A H5² (.,
where H1 = 6- is the regularized empirical Hessian at Ò but reweighed after removing the influence of wrong data. Then the one-step Newton approximation for (D") ) defined as gvv & Agnr(D)) + 9.
In the following, will separate the error between (D*) and å. (D*) into the following two parts:
(DD - Oe(" = ge("D) (D" (ONi(DD) - 9) (D") 9)
xx
(D=..a(D)))
Firstly, in Step 1, we will derive the bound for Newton-actual error ErING acI Since L (g) is strongly convex with parameter o'mm + and minimized by (DD),, we can bound the distance (lā - (Nv( Il2 terms of the norm of the gradient gNv.
(D') - gN: (D))l2 o'in + Vg (Dx)()
(32)
Therefore, the problem reduces to bounding I minimizes L - L', that is,
Noting that VgL'(g) = V,L This because g
LG + ['()) 0.
19

---

Editable Concept Bottleneck Models
Recall that AgNi" H V (D - H1g VgL-(D' Given the above conditions, we can have this bound for Erryi, 2 --D))
VoL (DN((D'))12
VBLL (0+ - -DNI(D"))ll
VoL (0 - AgN, - (V,
(0) V (0) AgN.(D 1/" ( AgN: 3 (0) Agr( dt
(33)
Ch 2
V - (0) ?
H (D")
[22 (0)
2
CF
C
V V (0)1
2( min +8)²
= 2( min
+ 6)2
CHC C22L²
2(0'min + +8)²
Now we come to Step 2 to bound EITNI .r( -DD),,,, and we will bound the difference in parameter change between Newton and our ECBM method.
(Dg) - -) (X' 9)ll = ||[(6. 1 V2L (o)) (0)) + (6.I+ Lo Toma (1) ()) VLL'(D,
I+
For simplification, we use matrix A, B for the following substitutions:
A = 1 (g)
B = 6. 1 V3 T (g)
And A and B are positive definite matrices with the following properties
8 + 0 d'ninn
X A x 5 + d'Exx
8 + 0min
- B - 8 + max
Therefore, we have
((gNe - 9) (g. (D') 9)ll
(A-1 +B 1) V (D"
[A-1 + B- 1| L (D":
26 + Omin + J'nnn
(34)
(D",
(6 d'min) (6 mmin)
26 + amin + o
C
(6- d'iin) - (6 mmin)
By combining the conclusions from Step I and Step II in Equations 32, 33 and 34, we obtain the error between the actual influence and our predicted influence as follows:
(D"))l
CH.
28 + minn + O'min
+
C%
20 min + +5)3
(0 + ) (0 + miin)
20

---

Editable Concept Bottleneck Models
Remark D.7. Theorem D.6 reveals one key finding about influence function estimation: The estimation error scales inversely with the regularization parameter 5 (O(1) /5),, indicating that increased regularization improves approximation accuracy. Remark D.8. In CBM, retraining the most accurate way to handle the removal of training data point. For the concept predictor. derive theoretical error bound for an influence aunction-bused approximation. However, the label predictor differs. As single-layer linear model, the label predictor is computationally inexpensive retrain. However, its input depends on the concept predictor, making theoretical analysis challenging due to: (1) Input dependency: Changes in the concept predictor affect the label predictor's 's input, coupling their updates. (2) Error propagation: Errors from the concept predictor propagate to the label predictor, introducing complex interactions. Given the label predictor's low retraining cost, direct retraining is more practical and accurate. Thus, we focus our theoretical analysis on the concept predictor.
E. Concept-level Influence
E.1. Proof of Concept-level Influence Function
We address situations that delete p, for M concept removed dataset. Our goal is estimate 9-pre- f-pne- which is the concept and label predictor trained on the pr for E M concept removed dataset.
Proof Sketch. The main ideas are as follows: (i) First, we define a new predictor Byss which has the same dimension as 9 and the same output as --p.e- Then deduce an approximation for 8pm. (ii) Then, we consider setting p, = instead of removing
get fpw= which is equivalent to f-pms according to lemma E. We estimate this new predictor substitute. (iii) Next, assume we only use the updated concept predictor 97>, for one data (Zi,, y and obtain new label predictor fir. and obtain a one-step Newtonian iterative approximation of für with respect to f. (iv) Finally, we repeat the above process for all data points and combine the estimate of g in Theorem E.2, we obtain a closed-form solution the influence function for
First, we introduce our following lemma:
Lemma E.1. For the concept bottleneck model, if the label predictor utilizes linear transformations of the form f with imput c, then, for each r E M, we remove the r-th concept from c and denole the new input as c'. Set the r-th concept to 0 and denote the new input as e. Then we have f-pu- c' = =p e for any c.
Proof. Assume the parameter space of f-pux and fpu=o are I and To respectively, then there exists surjection P: To- For any 0 E TT, P(0) is the operation that removes the r-th row of 0 for r E M. Then we have:
P(O) d' Σ 0bi -d
Σ O(elltt M) = 0 c.
MEM
Thus, the loss function Ly (0, Ly (P(O), of both models is the same for every sample in the second stage. Besides, by formula derivation, we have, for 0' E To. for any 0 in p-'()')
aLy (0.c )) aLy (P(0).c)
00
00'
Thus, if the same initialization is performed, fp c' fpm=0 d for any in the dataset.
Theorem E.2. For the retrained concept predictor g-PM
pm defined by
9-pM = arg min
Σ
g'
JAM 1=1
map it to p'
/- PM arg min
Σ
ETE
jØM 1=1
And we can edit the initial 9 to g- pm defined as:
--pms 9 -1111
D (a,,
(35)
JEM
21

---

Editable Concept Bottleneck Models
where H3 V EjEM Ei=1 ( (())))
9). Then, by removing all zero rows inserted during the mapping phase, we can naturally approximate 8-Pm- = P (p (0-
Proof. At this level, we consider the scenario that removes a set of mislabeled concepts or introduces new ones. Because after removing concepts from all the data, the new concept predictor has a different dimension from the original. We denote (xxx) as the j-th concept predictor with 24, and cf as the j-th concept in data 24. For simplicity, we treat g as a collection of l concept predictors and separate different columns vector g (2%). Actually, the neural network gets g whole.
For the comparative purpose, we introduce new notation 0 pms Specifically, define weights of and --px for the last layer of the neural network as follows.
C1
W12
widd
xt
8-pM PM (x) =
W121
W22
W2d,
x²
Gr-1
Cr+1
w(k-1)1
Wick 1)2
(kk- 1)d,
1) x
d:
Ck
(x-1)x1
1011
W12
W1d
x.
C1
W(r-1)1
W(r-112 1y2
W(r-1)d 1)d
x=11
c
9- pas (x)
0
x
x++1
1(++1)11
W1r++1122
1)++
Cr+1
:
Wk1
Wk2
Wkd.
Ck
kxd;
dg x
Rx
where is an index from the index set M.
Firstly, we want to edit to j'-par E To [Winal = 0} C T based on 9, where Winnal is the parameter of the final layer of neural network. Let us take a look at the definition of 9'pM'
9 9-pM arg min
Σ> Tlol'
g'ETo
jEM i=1
Then, we separate the r-th concept-related item from the rest and rewrite ý as the following form:
Σ Σ cl)
ΣΣ (e(((())).)))
arg min g
3ET
JEM t=1
TEM 1=
Then, if the r-th concept part is up-weighted by some small e, this gives us the new parameters per which we will abbreviate as ge below.
Σ Σ (olola))c)+c tc. ΣΣ Lolo(z)).))
Pe.pm arg min gET
i=Mi
PEM i=
Obviously, when e - 0, 9. + 8" par
We can obtain the minimization conditions from the definitions above.
ΣΣ
10
(c)),.)))
(36)
PM
J
22

---

Editable Concept Bottleneck Models
"
n +e B.. Σ
Σ
Σ
c)) 0.
JEM
PEM
Perform first-order Taylor expansion of equation 36 with respect to 9c, then get
(t))) (0-pu dc) R 0.
JEM 1=
JEM 1=1
Then we have
9 --px - Dc
H
de
JEM t=1
Where Hg. = DESEMM Ei= (x))
We can see that:
When e = 0,
9. = g' pM
When = 1, -e -9.
9 9-ps
H
g
j JEM t=1
where V jØM En c). c.
Then, an approximation 0 is obtained.
g PM
2d 0
H-1
V
(37)
JØM 1=1
Recalling the definition of the gradient:
Co(o,,c;)) =L)(((:)). (x:) log(c))
Then the approximation 0- par becomes
9 9-PM
4
H
JEM
Theorem E.3. For the retrained label predictor f-pxe defined as
f-pas = arg min
Ly = arg min
Ly (f)))
f
We can consider its equivalent version fom== as:
fPM=D = arg min
Ly
which can be edited by
n Ppuzoo Fpm-zo 2 f H.² Σ Go pm ffrf
where H = - E. Gzn par f) is the Hessian matrix. Deleting the r-th dimension of fox:=o can map it to f-pxe which the approximation of the final edited label predictor f-ps under concept level.
M, then we
23

---

Editable Concept Bottleneck Models
Proof. Now, we come the approximation of f-pax- Noticing that the input dimension of decreases to A: J... We consider setting Pr = for all data points in the training phase the label predictor and get another optimal model fPM==- From lemma 1, we know that for the same input æ, oox= = f-ps- And the values of the corresponding parameters in f and f-pas are equal.
Now, let us consider how to edit the initial f to fou==-- Firstly, assume we only use the updated concept predictor g'pse for data (Ii,, Vi,, Ci,)) and obtain the following fir, which is denoted as
for arg min
+ Ly Ly [)(((()))).
(Eir)), Ly Ly(((((a))))
Then up-we weightt the i, data by some small and have the following new parameters:
fe, arg min
Σ Lr(((((()))))) +e- (trr)) Vir) Lr((((((i))))..)))
Deduce the minimized condition subsequently,
by (() - VyIr(J (ttr)) - (f -0.
If we expand first term of f. which fir.e f(e + 0). then
Ly (f(((z))..l)) te. v (f(0" PM - (f us.)))
1m1
Ly (f(0(2)).01) (frr.e - f)= =
Note that V Ei Ly Lx(f(((z))) = 0. Thus we have
fine - f= f-11² ( (crr)) - (fdrr((()))
We conclude that
dfc de
= - (f( ((y()))())((((
l==0
Perform one-step Newtonian iteration at ff and we get the approximation of f...
f > f (VyLr (f - L (x+r)) vir)
Reconsider the definition of ir,, we use the updated concept predictor 3 Pm for one data (Ii.. Uh, Ci Now we carry out this operation for all the other data and estimate fpm==0- Combining the minimization condition from the definition of we have
n
Soy=o x + ~
V
Ly (f((()))) V
Lr(f(e PMs (s:)). w.)
=
+
-
Ly (f(0" (e))).v))
-
V Lr (2))),
(38)
24

---

Editable Concept Bottleneck Models
Theorem E.2 gives the edited version of 9-pM- Substitute into equation 38, and we get the final closed-form edited label predictor under concept level:
ppu=o= 2o 1pm=0=1 H
Vi Ly (f.
where = (f) is the Hessian matrix of the loss function respect is the Hessian matrix of the loss function respect to Recalling the definition of the gradient:
Gy V yLr (f(0-pm((x)) .
then the approximation becomes
Ppu== fp - f- H Σ Gy pas f).
E.2. Theoretical Bound for the Influence Function
Consider the dataset D = {2 co. -- the loss function of the concept predictor is defined as:
LTmaa (D;g) Σ Lo(g(a)) + ||911
8
ΣΣ
(e(()())) +
||911
n
g (x:) log(ee") + 8 ||a11
==1
i=1j=1
Mathematicallyy have set of erroneous concepts need to be removed, which are denoted p. for M. Then the retrained concept predictor becomes
9 J-PM- arg min
ΣΣ i=1
(o)) es) ||g1
We map it to 9r PMr as ÿ g-pac par to g PM concepts papa (j) as g-pxr (j) for all j E [d]] M. We achieve this effect by inserting a zero row vector into the r-th row of the matrix in the final layer of --pa. for E M. Thus, we can see that the mapping P is one-to-one, Moreover, assume the parameter space of ğ T and that of Ø- To is the subset of T. Noting that g" is the optimal model of the following objective function:
A P( par ). which has the same amount of parameters as 9 and has the same predicted
n
g
Ppp
arg min g'ETo
Σ Σ JEM
[llo (x)), ||g12
Then the loss function with the influence of erroneous concepts removed becomes
a (D ΣΣ JAMEL
||al| + 8 (z))
Locual (Dig)
ΣΣ JEM
Le (o(2))
(39)
Assume 9 arg min LTmaa (D:g) the original model parameter. --pas (D) and -- (D) is the minimizer (D= (D:g), g). which is obtained from retraining different parameter space. 9mp,, (D) shares the same dimensionality as the original model. Because --pax (D) and pmc (D) produces identical outputs given identical inputs, to simplify the proof, use 8mpm (D) as the retrained model.
Denote J-PMr as the updated model with the influence of erroneous concepts removed and is obtained by the influence function method in theorem E.2, which is an estimation for p'p (D).
9-pza (D) 2
H
Σ
Σ
C
JÆM
25

---

Editable Concept Bottleneck Models
In this part, will study the error between the estimated influence given by the theorem E.2 method and D'Par (D). We the parameter changes the evaluation metric:
(0-PM - 9) (9 PM - -g) -19-v p-pae
(40)
Assumption E.4. The loss Lo(x, c; g:
Lc(D:g:))= Σ [l
is convex and twice-differentiable in g. with positive regularization 8 > 0. There exists Cи E IR such that
(Diggs)) - Cnllg1 g2ll2
for all j [k] and 91, 92 r. Definition E.5.
Cj = max (Vgg:x/(VL((
3
0' min smallest singular value of V2 (D;
smallest singular value of V2 Looal (D:))
T min
L'(D, = Σ Lo(;gg:))
(41)
JEM
Corollary E.6.
I.- (D:g) Traai (D; - [D, M:9)
(42)
l2 S ((k + MM) CH) ll ID: - (D:
92ll
Define C A (ke + Ml) Cи
Based on above corollaries and assumptions, we derive the following theorem. Theorem E.7. We obtain the error between the actual influence and our predicted influence as follows:
110 (D) - 9 pMs (D)ll
C#CLI M
28 + Fmin + I'min
C.IMI.
2( min + 8)3
(5 + o'min)) ) (6 amin))
Proof.
We will use the one-step Newton approximation as an intermediate step. Define Agne(D) as
Agvi(D) Hg (D:
where = 5. 5 13 is the regularized empirical Hessian at ğ but reweighed after removing the influence of wrong data. Then the one-step Newton approximation for 8m (D) is defined as gNv - (g ğ.
In the following, will separate the error between 8-par (D) and pas (D) into the following two parts:
9" pm (D) I-pN (D) 9 pM (D) (N)(D) (DN)(D) 9) (g-par (D) 9)
Er (D)
Ems.a(P) (D)
Firstly, in Step 1, we will derive the bound for Newton-actual error ErTyL act (D). Since L-(g) is strongly convex with parameter o'min + and minimized by 3rpm.. (D), we can bound the distance || PM (D) - (D)]l, in terms of the norm of the gradient at gvv.
2
| pm (D) (Dr < o' 5 V
(ox:
(43)
26

---

Editable Concept Bottleneck Models
Therefore, the problem reduces bounding , - (DNc(D)))l2 Noting that V ( -Vg L This is because minimizes I,- T', that is,
V 1 (g) + V V))'(0) = O. H55 VV (D:g). Given the above conditions, we can have this bound
Recall that A for Erryt, a (DD)
= H (DD
V (oN((D))122
L (0 + AgNi(D))l2
VoL (6+ + AgN, (D)) - V,L (0) V (0) Ag),(D)
J' (V (g (g)) L (0) (D) dt
(44)
= Gi
H
(D
V 1. (0)
9 1. (0)
2
2
Tn
g = 2( min + 6)2
| T,' (1)1
2iii + +o)²
20 + +8)2
Now we come to Step 2 to bound EITyL. ir( (DD), and we will bound the difference in parameter change between Newton and our ECBM method.
(D) 9) - (0-ps (D) - 9
[[(66- [(66-- 1+ V2L (o) + (6-I+ I+ Li (0)) "I. VoL'(D,
For simplification, we use matrix A, B for the following substitutions:
A 8 1 V2 (g)
B= 8 I I (g)
And A and B are positive definite matrices with the following properties
8 + 0 min
XA- - 8 + o max
8 + min
- B x 8 + 0 max
Therefore, we have
[[ (D) - g) - (g-px - 9)ll
[1 + +B-1) (P;
|--1 + B
|| (D:
26 + mmn + J'inn
(45)
(D:
(6 + dminn) (6 + minn)
28 + mmin J'inn
CLIM
(6 )'in) (8 + min))
By combining the conclusions from Step I and Step II in Equations 43, 44 and 45, we obtain the error between the actual influence and our predicted influence as follows:
|/ا" par (D) - -p-pMr (D)ll
28 + Jmin + d'min M.
< 2( min + +6)3 +
(6 + =(in)) (8 + mmia)
27

---

Editable Concept Bottleneck Models
Remark E.8. Theorem E.7 reveals one key finding about influence function estimation: The estimation emror scales inversely with the regularization parameter å (O)1/8), indicating that increased regularization improves approximation accuracy. Besides, the error bound is linearly increasing with the number of removed concepts |M. This implies that the estimation eтror increases with the number of erroneous concepts removed.
F. Data-level Influence
F.1. Proof of Data-level Influence Function
We address situations that for dataset D {z,, Vi, Hi given set of data 2, (rr, Vr, cr), r G to be removed. Our goal is to estimate ÿ-za. f-za.f2ac which is the concept and label predictor trained on the 2, for r E G removed dataset.
Proof Sketch. (i First, we estimate the retrained concept predictor --sc- (ii) Then, we define a new label predictor f-ec and estimate f-ro- - f. (iii) Next, in order to reduce computational complexity, use the lemma method to obtain the approximation of the Hessian matrix of f-zo. (iv) Next, we compute the difference f-2o f-zo
(x,,)), v..) (x H. f-1G (V (f-zo
viLr (f-za (0(xi.)).
(v) Finally, we divide f-ro f, which we actually concerned with, into (f-co- -ooo----oh (f-za- f).
Theorem F.1. For dataset D {(1 ys, H given a set of data 2, (Ir, yr: cr), r G to be removed. Suppose the updated concept predictor 9-20 is defined by
--zc = arg min
Σ
Σ
La, ()())))))
JEIN iE(nI-G
where Lc(g(2;),,;) A Ez= (2(()(()),)) c.) Then have the following approximation for 9-z0
M
g-zc =c R g
A 9 H
(46)
J=1
where VV=Va Eig Polz, 02 is the Hessian matrix of the loss function with respect to 9.
Proof. Firstly, we rewrite g- as
n
Lo( - Elols(er).co)
-_z0 = arg min
g
rEG
Then we up-weighted the r-th data by some and have new predictor 9-za.e> which is abbreviated as 9.:
g. e arg min Σ Lo(g(()))c) Elolg(e)).c))
(47)
rEG
Because 9. minimizes the right side of equation 47, we have
Ly (x:)),)) - Va. Σ
(x) =0
rEG
When 0. ge g. G. So we can perform first-order Taylor expansion with respect to ÿ. and we have
V, Lo(G(2;).c) - V. > I V2 Σ Le(o(zs),c)) (g. - ~ 0.
(48)
rEG
28

---

Editable Concept Bottleneck Models
Recap the definition of g:
9 - arg min Ly(
Then, the first term of equation 48 equals
Let
0, then we have
dg. de
Σ ()Lo(((Z)),c),
c=0
FEG
where H3 = -
Remember when + 0, ge + z0 Perform a Newton step at 9. then we obtain the method to edit the original concept predictor under concept level:
g-sa R g-sa 9 11 Σ VVLc(((())...)
Recall the definition of (2 czig). 9), then can edit the initial I to g-pp- defined as:
M ΣΣ PEG
8-2a = --za - g H
(49)
where H3 V. Eig Geloi, ()) is the Hessian matrix of the loss function with respect to
Theorem F.2. For dataset {1 Ys c.)li= given a set of data 2, = (Ir, Mry Cr., r E G to removed. The label
predictor f-zo trained on the revised dataset becomes
f-zo = arg min
Σ iE(n)-G
Ly.(f.-za).
(50)
The intermediate label predictor 8-zo defined by
f-sc arg min
Σ Lf.
iE|n]-G
Then f-2o f can be approximated by
j_zo = arg min
Σ Lr(((((zs)..)
(51)
sEIn|-G
We denote the edited version of f-zo f-zo A
Ac. Define Bū as
-17- Σ fel Gr
sEIn|-G
where H Hi-00 EiEni Gy (Z: F=zo) is the Hessian matrix concerning f= Then f-zo can be estimated by i_so Bg. Combining the above two-stage approximation, then, the final edited label predictor J-zo can be obtained by
f-2o = f-ze + BB = f Bg.
(52)
Proof. We can see that there is huge gap between f- and f. Thus, firstly, we define f-za as
8-zc arg min Σ Ly (f(g(z;)). vi)
L (f(((z,)).)))
f
TEG
29

---

Editable Concept Bottleneck Models
Then, define fe follows estimate f-za-
fe- -arg min
Ly (f(((=))).))) - by (f(G(z))))
TEG
From the minimization condition, we have
Ty (fc. (0(38))))) - Evphv (((e))) = 0.
rEG
Perform first-order Taylor expansion at f.
by (f(O(z)): - - V ΣIn by (f((2,)).
rEG
Ly (f(0(z))..81) (fa-za- -f) = -.
-))=00
Then f-zo can be approximated by
f H
1ULLY (f(0(e.)).u.) Ac.
(53)
rEG
Then the edit version of f-zo defined as
F= = Ag
(54)
Then we estimate the difference between f-zo and f-za- Rewrite J-zo
8-zo = arg min
Σ Ly (f()(x:))
(55)
where S [n] G.
Compare equation 50 with 55, still need to define an intermediary predictor f-zo, är
f-ze.rr arg min f
Σ
Lv. (f,g(z)) + Ly. (f. --))
iES
Ligtt
arg min
Σ
Lr, (J.g) Lr. (fiğ-ac) Lr, (1,9)
LiES
Up-weight the i, data by some e, define fo. fe,-20.5
fe-ra arg min
Σ Σ Ly, (f.g) +e Ly,, (f.ğ-zo) - Ly (f.g)
LiES
We denote fe. as fr in the following proof. Then, from the minimization condition, we have
V Σ Lv. (Fe.0) + V,IVe (fr - C.VLLya (f:.)(
(56)
iES
30

---

Editable Concept Bottleneck Models
When + 0, f² f-so-
Then we perform
Taylor expansion at f-za of equation 56 and have
Ly, (f-z.:0) + ViLren (f-z):
SES
V vilr (f=2010) } Σ Iyg (i-zo1) (f2 f=20] 0.
iE9
Organizing the above equation gives
ft f-zo -
-.....T (VyLr.. (f-zo:0-zc)
-ViLyn (f-zo:0)
where H ii-- = vi Eies Ly, (f-c.10.
When 1, = f-zairr Then we perform Newton iteration with step size 1 at f-zo-
f-zcrrr - -0 f.o (VpIrir ( (f-201820)) - Vylrs. (i (f-zo1
Iterate i, through and we have
8-zo f-zo = H
[ Ly, (f-zor - iES
V Σ Ly, (f-z..0)
(57)
iES
The edited version of 9-20 has been deduced as g-sa in theorem F.1, substituting this approximation into (57), then we have
f- - f-2o -11
8-20
V E Ly, (F- - ΣEE, Lr. (f-2::)
(58)
iES
Noting that we cannot obtain S- and H i- directly because we do not retrain the label predictor but edit it to f= as a substitute. Therefore, approximate f-zo with f-za and j with H which is defined by:
H FG
Lr, (f=
iES
Then define Bg
BG 2 1:
-11
Σ
Ly,
-V
(F-za:
(59)
iES
iES
Combining (54) and (59), then deduce the final closed-form edited label predictor
f-2o = f= + BG = Ag + BG.
Replace the definition of gradient of Le and Lc, then we obtain the final version of approximation.
F.2. Theoretical Bound for the Influence Function
Consider the dataset D = {2 cs y1 y. the loss function of the concept predictor g is defined as:
8
n k
9 (a:) log(a)) + 8
LToat (D;g) Σ Lo(o(e)) + ||
ΣΣ
Ll(((s)).
||
Σ
11/1122
==
i=1 j=
Mathematically, we have a set of erroneous data 2, = (xr, yr, Cr), r E need to removed. Then the retrained concept predictor becomes
31

---

Editable Concept Bottleneck Models
8 c)) + - -19911
g-zG arg min
Σ Σ
1= AE
Define the new dataset as D C. then the loss function with the influence of erroneous data removed becomes
L (D(:'g))
Σ Σ [l) + 8 -1al12 Tonal (D:g) i=
ΣE'
To (o(
(60)
5=1iEG
Assume 9 arg min Total (D:g) is the original model parameter. タ-zo is the minimizer of L. (D" Denote g-zc the updated model with the influence of erroneous data removed and is obtained by the influence function method in theorem F.I, which is an estimation for 9 --
g- R g-20
- 9 + H
(61)
In this part, we will study the emror between the estimated influence given by the theorem F.1 method and --zo- We use the parameter changes the evaluation metric:
K0-za - 9) (g- -9)1 = =1 -azal
(62)
Assumption F.3. The loss (o(x,cgg:)))
k Lo(x,cg)) Σ (o()))).). j=1
is convex and twice-differentiable in with positive regularization 5 > 0. There exists Cn E IR such that
Lo c2 S Chlla: 92l12
for all (z, c) E D and g1. 92 r. Definition F.4.
C (VLCc(D)))
O'nnn smallest singular value of V (D:
mmin smallest singular value of V 9 I (D: (D:
L'(D,Gig) Σ Lo (9g)
(63)
iEG
Corollary F.S.
(D: = L (D: (D:g) L'D,
(64)
IV - V2 (D: $ ((n + |G]) Cи) 1lg1 - 92l
Define C# (n+ [G|) Cи
Based above corollaries and assumptions, derive the following theorem. Theorem F.6. We obtain the error between the actual influence and our predicted influence as follows:
_ 20 (D) (D)]
C C2
25 + amin + o'
+
2 min + +8)3
(8 + )'inn) + O min
32

---

Editable Concept Bottleneck Models
Proof. We will use the one-step Newton approximation as an intermediate step. Define (g)
(g) A H (DL
where H5 = 8. V is the regularized empirical Hessian at ÿ but reweighed after removing the influence of wrong data. Then the one-step Newton approximation for 0= (D) is defined as gNv (D) - Agvi(D) + 9.
In the following, we will separate the error between 8- (D) and 9- (D) into the following two parts:
--s0 (D) - - (D) = -- (D) (D) + [gNi (D) - 9) - (ā- so (D) - 9)
Errsa, (D)
(r)),
Firstly, in Step 1, we will derive the bound for Newton-actual error ErTNI, act (D). Since L.- (g) is strongly convex with parameter mii d and minimized by g- (D), we can bound the distance ||ğ- (D) (Nvx(D)ll in terms of the norm of the gradient at INz:
2
-0= ( (D) - (D)l22 S
m + Ligh (DNr
(65)
Therefore, the problem reduces to bounding 9 minimizes L L', that is,
|| V (oNx((D)) Noting that V V,L'(D,G;)) = V-L". This is because
V L (g) + (D, G: =0
Recall that AgN:= = H V,L'(D,G;)) H (D; Given the above conditions, we can have this bound for EITNI, aCTNt,ac(-11 act -D).
L (DN((D))l2
1o (g + AgNi(D))l12
V (6+ + AgN. (D)) - V,L (0) - V-L (0) (D)112
!'' (V (0+1- AgN: 1 (0) (D)
(66)
C# 2
h
(Agn((D"))
[ ( V L (0)
2
T
V[ (1)1
11 GDD
20 + +572
2( min + 8)}
C H G2
20 min + +8)2
Now we come to Step 2 to bound EITNL ir( ((D)), and we will bound the difference in parameter change between Newton and our ECBM method.
(D) - 9) - -0- (D) - 1
= =1[[(0- - I- V2L- (o) + (6- I V Lreal (0)) H) VgL'(D,G:9)
For simplificatinn, we use matrix A, B for the following substitutions:
A = 6 (g)
B 5- I+ I (g)
And A and B are positive definite matrices with the following properties
8 + O'nin
X A x 8 + o
8 + o min
B 8 + 0 max
33

---

Editable Concept Bottleneck Models
Therefore, have
([ - 9) - (7- (D) - 9)ll
[1AA) + +B=1) V V
(D: (D:))
A-- + BBB V (D; [)
20 miin min
(67)
V
(ò + min) (6 + minn)
20 min + o min
CLG
(ō + min) (6 + miin)
By combining the conclusions from Step I and Step in Equations 65, 66 and 67, obtain the error between the actual influence and our predicted influence follows:
110 (D) --
20 + min o min
+
CL
2 min + 6)3
(6 + d'inn) . (5 + miin)
Remark F.7. The error bound linearly increasing with the number of removed data |G]. This implies that the estimation emror increases with the number of erroneous data removed.
G. Algorithm
Algorithm Concept-label-level ECBM
1: Input: Dataset D {(x;, Vi, =)) original concept predictor f. and label predictor 9. a set of erroneous data D, and its associated index set S..
2: For the index (w,r) in S., correct ct to the right label cy'' for the w-th data (I Vios Car).
3: Compute the Hessian matrix of the loss function respect to
Hg = v Σ Lo, (6' (z)),
4: Update concept predictor g:
H
Σ
(VgLe, (I'(zw), Vale, (zw),
(w.r)ES.
5: Compute the Hessian matrix of the loss function respect to f:
H;=V² v Σ Ty, (f.9)
6: Update label predictor f:
n V Σ Ly (f)( - H.¹ v, Σ
n
i f=f+H
Ly ((((a)))))))
1=1
[=1
7: Return: f.g.
Algorithm Concept-level ECBM
1: Input: Dataset D {(,,, yr, 4)) original concept predictor label predictor 9 and the to be removed concept index set M.
34

---

Editable Concept Bottleneck Models
2: For r M, set pr 0 for all the data = E D.
3: Compute the Hessian matrix of the loss function respect to
H3 H-= = V Σ
Lo, Lo,(o'(r)),.))
3 M im
4: Update concept predictor g":
n Lc, (z)).
g =
JEM
5: Compute the Hessian matrix of the loss function respect to f:
- V
La (f((
6: Update label predictor f:
f-f-
Ly (f()
1=1
7: Map 9" to ÿ by removing the r-th row of the matrix the final layer of g for E M.
8: Return: g.
Algorithm 3 Data-level ECBM
1: Input: Dataset D = {(r; Us, Ci) I original concept predictor f. label predictor 9. and the to be removed data index set G.
2: For r É G, remove the r-th data (Iy, Yr. 6)) from and define the new dataset as S.
3: Compute the Hessian matrix of the loss function with respect to g:
H = V
Lo, (o' (x)),
4: Update concept predictor ğ:
g 9 + H [g)
TEG
5: Update label predictor j. Compute the Hessian matrix of the loss function with respect to f:
H1 -1= V La (f()())).)) w.).
6: Compute A as:
A= A= Σ
1 (f((
ic(n]
7: Obtain ʃ as
j f- A
8: Compute the Hessian matrix of the loss function concerning f:
H Σ (F(((=)))) iEIn]-G
9: Compute B as
B= B= Σ V (I Ly (f(((()))) vz)
G
35

---

Editable Concept Bottleneck Models
10: Update the label predictor f as: f A+ B.
11: Return: 9-
Algorithm 4 EK-FAC for Concept Predictor
1: Input: Dataset D { Me. c))]i= original concept predictor g.
2: for the 1-th convolution layer of g: do
3: Define the input activations {a weights W (Wij,o), and biases b (b,) of this layer;
4: Obtain the expanded activations [A as:
[A-ilwJAIH = [AL-1((+8855 = aj.tt.
5: Compute the pre-activations:
Siibe = Si,t
Σ
Wi,5,505,488 + bb.
SED
6: During the backpropagation process, obtain the Dst,t
0 0 Ei Lc,
184,
Osi,t
7: Compute Ω and F':
0 Σ
([A)- IH 11[4i-1l
i=1
r, n
DSj DS
8: Perform eigenvalue decomposition of Q and Fú, obtain Qn, An. Qr, Ar, which satisfies
Ô411 QnAaQ ñ, = Q
9: Define a diagonal matrix A and compute the diagonal element as
12
A
(19m. @Qr) Lc,
10: Compute i as
01 72'=(Qmm Qr.) (A Xela)) (Qn (Qn Qr.)²
(Qam
11: end for
12: Splice H, sequentially into large diagonal matrices
A
0
Ag
HJl
where is the number of the convolution layer of the concept predictor. 13: Return: the inverse Hessian matrix H
Algorithm EK-FAC for Label Predictor f
Input: Dataset ((*,, Vr. N. original label predictor f.
36

---

Editable Concept Bottleneck Models
2: Denote the pre-activated output of / as f', Compute A
a A
1
Σ T
g(z.) g(x.)"
3: Comput B as:
B= n Σ
v
(f (f (0(x)) .v)
4: Perform eigenvalue decomposition of AA and BB, obtain QA A. QB, AB, which satisfies
A = Q4A4QA
B = QBA
5: Define a diagonal matrix A and compute the diagonal element as
= Σ (QAA@1)VV
6: Compute |
as
(Qa QB)
7: Return: the inverse Hessian matrix 1171"
Algorithm EK-FAC Concept-label-level ECBM
1: Input: Dataset D [(x,, V/, c.)) ) original concept predictor f. label predictor g. and the to be removed data index set G, and damping parameter X.
2: For r E G, remove the r-th data (I,, Wr. ()) from D and define the new dataset as S.
3: Use EK-FAC method in algorithm 4 to accelerate iHVP problem for 9 and obtain the inverse Hessian matrix A,²
4: Update concept predictor g:
g = 9 H
Σ
(VgLe, (g' (Iw), VgLc, (0" (xww), (w))
(w.r)ES.
5: Use EK-FAC method in algorithm 5 to accelerate iHVP problem for f and obtain i1
6: Update label predictor f:
n j 8= V Σ Ly (f(((x))...) - H v, Σ Ly (f((t)).. 1=1
n
1=1
7: Return: 1.9.
Algorithm 7 EK-FAC Concept-level ECBM
Input: Dataset D {(2,, yr, == original concept predictor label predictor and the to be removed concept index set M, and damping parameter X.
2: For r E M, set Pr 0 for all the data E D.
3: Use EK-FAC method in algorithm 4 to accelerate iHVP problem for ý and obtain the inverse Hessian matrix Ar
4: Update concept predictor ÿ:
g -'=- H
V
Lc, (o)
JEM
5: Use EK-FAC method in algorithm 5 to accelerate iHVP problem for f and obtain A
37

---

Editable Concept Bottleneck Models
6: Update label predictor f:
f Σ Ly (f(((
E=1
7: Map 9" to g by removing the r-th row of the matrix the final layer of g" for E M.
8: Return: 9-9-
Algorithm EK-FAC Data-level ECBM
1: Input: Dataset D { ye, ==) original concept predictor f, and label predictor 9, a set of erroneous data De
and its associated index set Se, and damping parameter X.
2: For the index (w, rr) in S., correct c to the right label ct' for the w-th data (Tar ywe Ca
3: Use EK-FAC method in algorithm 4 to accelerate iHVP problem for 9 and obtain the inverse Hessian matrix H3
4: Update concept predictor
-=9- Hg"
Σ (w.r)ES,
(VgLe, (zw) cio') VgLC, (0" (Iw), cw))
5: Use EK-FAC method in algorithm 5 to accelerate iHVP problem for f and obtain ; Compute A as:
A= #=
Σ ViIv (f(((z))...)
iE(n]-G
Obtain as
f=
6: Use EK-FAC method in algorithm to accelerate iHVP problem for f and obtain H Compute as
=²=
(y( Ly (f(((())))
<E(n]-G G
Update the label predictor ass f
7: Return: f, g.
H. Additional Experiments
H.1. Experimental Setting
Methodology for Processing CUB Dataset For CUB dataset, we follow the setting in (Koh et al., 2020). We aggregate instance-level concept annotations into class-level concepts via majority voting: e.g., if more than 50% crows have black wings in the data, then we set all crows to have black wings.
RMIA score. The RMIA score is computed as:
P((fo(e)) (2,))
(R, (x).
where fo(x) represents the model's output (logits) for the data point (N,, denotes a Gaussian distribution with mean and variance o², (z) is the mean of the model's outputs for z under the assumption that æ belongs to the training set, and (x) is the variance of the model's 's outputs for r. The likelihoods Pr(fe(z)[W) represent the probability that the model's 's output fe(x) follows the Gaussian distribution parameterized by a and o². under the two different hypotheses: x being a member of the training set versus not being a member.
38

---

Editable Concept Bottleneck Models
H.2. Improvement via Harmful Data Removal
We conducted additional experiments on CUB datasets with synthetically introduced noisy concepts labels. Firstly, introduce noises under three levels. At the concept level, choose 10% of the concepts and flip these concept labels for portion of the data. At the data level, we choose 10% of the data and flip their labels. At the concept-label level, we choose 10% of the total concepts and flip them. Then, we conduct the following experiments.
We introduce noises into the three levels and train the model. After that, we remove the noise and obtain the retrained model, which is the ground truth(gt) this harmful data removal task. In contrast, we use ECBM to remove the harmful data.
Concept Level
Data Level
Concept Label Level
Accassiyy
Acceret
Scare
Score
0.72
Scores
0.755
0.10
0.7
MXY
0.75
0373
072
0.7
6362
0.7
Before Removal
Remuval by Retrain
Removal by ECIB
Before Removal
Removal by Retrain
Removal by ECBB
Before Removal
Removal by Retrain
Removal by ECBM
Figure 5: Model performance after the removal of harmful data.
From Figure 5, it can be observed that the model performance improves across all three settings after noise removal and subsequent retraining or ECBM editing. This confirms that the performance of ECBM is nearly equivalent to retraining in various experimental scenarios, further providing evidence of the robustness of our method.
H.3. Periodic Editing Performance
ECBM can perform periodic editing. To evaluate the multiple editing performance of ECBM. we conduct the following experiments. Firstly, we introduce noises under three levels. At the concept level, we choose 10% of the concepts and flip these concept labels for a portion of the data. At the data level, we choose 10% of the data and flip their labels. At the concept-label level, we choose 10% of the total concepts and flip them. Then, we conduct the following experiments.
At the concept level, first remove 1% the concepts, then retrain or use ECBM to edit and repeat. In the data level, we first remove 1% of the data, then retrain from 1% of the data, then retrain use ECBM to edit. Note that when removing the next 1% of the concepts, ECBM edits the model based on the last editing result. The results at each level are shown in Figure 6, 7 and 8.
ECBM to edit. At the concept label level, first remove concept label
From the above three levels, we can find that with the mislabeled information removed, the retrained model achieves better performance in both accuracy and Fl score than the initial model. Furthermore, the performance of the ECBM-edited model is similar to that of the retrained model, even after 10 rounds of editing, which demonstrates the ability of our ECBM method to handle multiple edits.
H.4. More Visualization Results and Explanation
Visualization. Since CBM is an explainable model, we aim to evaluate the interpretability of our ECBM (compared to the retraining). We will present some visualization results for the concept-level edit. Figure presents the top 10 most influential concepts and their corresponding predicted concept labels obtained by ECBM and the retrain method after randomly deleting concepts for the CUB dataset. (Detailed explanation can be found in Appendix H.4.1.) Our ECBM can provide explanations for which concepts are crucial and how they assist the prediction. Specifically, among the top 10 most important concepts in the ground truth (retraining). ECBM can accurately recognize within them. For instance, we correctly identify "has "has_uppera" wwwwupppts.scommmmmm ppperpaats.coloooorrama. color:or.nge"" wwwssupppr.tail.comm toil.cooo:::eed, and "has.breast.co breast..cooo::bll.. as some the most important concepts when predicting categories. Additional visualization results under data level and concept-label
39

---

Editable Concept Bottleneck Models
Accuracy Comparison (Concept Level)
F1 Score Comparison (Concept Level)
Acsumey Dimingy
0,50
SSore Strilay
6768
Hozen Aet
ECBMAN
D03MP7
6.660
Accuracy
0.964
2.754
8.922
0,70
0.796
15
75
Mislabeled Concept
Mislabeled Concept
(a) The accuracy of edited model compared with retrained. (h) The score of edited model compared with retrained.
Figure 6: Accuracy and Fl score difference of the edited model compared with retrained at concept level.
Accuracy Comparison (Data Level)
F1 Score Comparison (Data Level)
Accrncy Simllgy
Score Brallgy
Raman
Ratan
6770
Accuracy
0.783
6.755
0.70
(%)
Data (%)
(a) The accuracy of the edited model compared with retrained. (b) he Fl score of the edited model compared with retrained.
Figure 7: Accuracy and Fl score difference of the edited model compared with retrained data level.
level on OAI and CUB datasets are included Appendix H.4.2.
H.4.1. EXPLANATION FOR VISUALIZATION RESULTS
At the concept level, we remove each concept one at a time, retrain the CBM, and subsequently evaluate the model performance. We rank the concepts in descending order based on the model performance loss. Concepts that, when removed, cause significant changes in model performance are considered influential concepts. The top 10 concepts are shown in the retrain column as illustrated in Figure 9. In contrast, we use our ECBM method instead the retrain method, as outlined in Algorithm and the top concepts are shown in the ECBM column of Figure 9.
To help readers connect the top 10 influential concepts with the input image, provide visualizations of the data and list the concept labels corresponding to the top 10 influential concepts, which are shown in Figure 9,10, 11.
For the other two levels and for additional datasets, we also conduct similar procedure, and the corresponding visualization results are presented in Figure 12, 13, 14, 15, and 16.
H.4.2. VISUALIZATION RESULTS
We provide our additional visualization results Figure 10, 11, 12, 13, 14, 15, and 16.
40

---

Editable Concept Bottleneck Models
Accuracy Comparison (Concept-label Level)
F1 Score Comparison (Conce Level)
6788
Acsumey Similgyy
0,700
Scare Stralagy
Hazen Aet
ECBMAC
203MP7
6.766
Accuracy
0.796
2.722
6.782
5%
7%
Concept Label (%)
Concept Label (%)
(a) The accuracy of edited model compared with retrained. (h) The score of the edited model compared with retrained.
Figure 8: Accuracy and Fl score difference of the edited model compared with retrained at concept-label level.
Label
Greeve Billed Ani
ECRM
Retrain
Label
Layan Albatross
ECRM
Ramain
celor caleried
under Mark
colerted
Mack
Mack
onlor Mack rel
coler
helly coler
prinary colorttacc
tllck parching heart colar
color:blcck
black paiciig like brcae color back pslan
Aupcrporiinggla tie
shapp--ppeee parchapr like
brast oder
his braat odler
back colerzed
colerned
buck celorzed
Hill plggecopaaareee
hill uhapccppttttss
hil whppecpppttiieee
Figure 9: Visualization of the Top 10 Most Influential Concepts for CBM(Identified by ECBM Retrain) Highlighted on an Extracted Image.
I. More Related Work
Influence Function. The influence function, initially a staple in robust statistics (Cook, 2000; Cook & Weisberg, 1980), has seen extensive adoption within machine learning since (Koh & Liang, 2017) introduced it to the field. Its versatility spans various applications, including detecting mislabeled data, interpreting models, addressing model bias, and facilitating machine unlearning tasks. Notable works in machine unlearning encompass unlearning features and labels (Warnecke et al., 2023), minimax unlearning (Liu et al., 2024), forgetting subset of image data for training deep neural networks (Golatkar et al., 2020a; 2021), graph unlearning involving nodes, edges, and features. Recent advancements, such as the LiSSA method (Agarwal et al., 2017; Kwon et al., 2023) and kNN-based techniques (Guo et al., 2021), have been proposed to enhance computational efficiency. Besides, various studies have applied influence functions to interpret models across different domains, including natural language processing (Han et al., 2020) and image classification (Basu et al., 2021), while also addressing biases classification models (Wang et al., 2019), word embeddings (Brunet et al., 2019), and finetuned models (Chen et al., 2020). Despite numerous studies influence functions, CBM. Moreover, compared traditional neural networks, CBMss are more complicated in their influence function. Because we only need to change the predicted output in the traditional influence function. While in CBMs, we should first remove the true concept, then we need to approximate the predicted concept in order to approximate the output. Bridging the gap between the true and predicted concepts poses significant theoretical challenge in our proof.
are the first utilize them construct the editable
Model Unlearning. Model unlearning has gained significant attention in recent years, with various methods (Bourtoule al., 2021; Brophy & Lowd, 2021; Cao & Yang. 2015: Chen et al., 2022a;b) proposed to efficiently remove the influence of certain data from trained machine learning models. Existing approaches can be broadly categorized into exact and approximate unlearning methods. Exact unlearning methods aim replicate the results of retraining by selectively updating only a portion of the dataset, thereby avoiding the computational expense of retraining on the entire dataset (Sekhari et al., 2021; Chowdhury et al., 2024). Approximate unlcarning methods, on the other hand, seek to adjust model parameters to approximately satisfy the optimality condition of the objective function on the remaining data (Golatkar et al., 2020a; Guo et al., 2019; Izzo et al., 2021). These methods are further divided into three subcategories: (1) Newton step- based updates that leverage Hessian-relatee related terms [22, 26, 31, 34, 40, 43, 49], often incorporating Gaussian noise mitigate

---

Editable Concept Bottleneck Models
residual data influence. To reduce computational costs, some works approximate the Hessian using the Fisher information matrix (Golatkar et al., 2020a) small Hessian blocks (Mehta al., 2022). (2) Neural tangent kernel unlearning approximates training linear process, either by treating it as single linear change (Golatkar et al., 2020b), (3) SGD path tracking methods. such as DeltaGrad (Wu et al., 2020) and unrolISGD (Thudi et al., 2022), reverse the optimization trajectory of stochastic gradient descent during training. Despite their advancements, these methods fail to handle the special architecture of CBMs. Moreover, given the high cost of obtaining data, we sometimes prefer to correct data rather than remove it, which model unlearning is unable achieve.
J. Limitations and Broader Impacts
It is important to acknowledge that the ECBM approach is essentially approximation of the model that would be obtained by retraining with the edited data. However, results indicate that this approximation is effective in real-world applications.
Concept Bottleneck Models (CBMs) have garnered much attention for their ability to elucidate the prediction process through a human-understandable concept layer. However, most previous studies focused on cases where the data, including concepts, are clean. In many scenarios, we always need remove/insert some training data or new concepts from trained CBMs due to different reasons, such data mislabeling, spurious concepts, and concept annotation errors. Thus, the challenge of deriving efficient editable CBMs without retraining from scratch persists, particularly in large-scale applications. To address these challenges, we propose Editable Concept Bottleneck Models (ECBMs). Specifically, ECBMs support three different levels of data removal: concept-label-level, concept-level, and data-level. ECBMs enjoy mathematically rigorous closed-forn approximations derived from influence functions that obviate the need for re-training. Experimental results demonstrate the efficiency and effectiveness our ECBMs, affirming their adaptability within the realm of CBMs. Our ECBM can be an interactive model with doctors the real world, which is an editable explanation tool.
42

---

Editable Concept Bottleneck Models
Label
Black Footed Albatross
ECBM
Retrain
has upperparts eoolo::orooggeeeeeee
has colirooooeccooooooaaaaeee
has has_leg_ color:orange
has maa__eeg_ee 0
colorr:orange
has __p___ppe___tae tail color:rel
0
has cess__ppe____e Lirt_taa___ge
cail_colorrred
has wtwp_brrccttlleeeeeeeeeee color:black
0
has breast cotoss:bolorbccckk
has belleellllyoo celly_color:
0
has wwwlleeellllllla__a_________aa__ color::red
has primary_color:bbacc color:black
has lalis_prmary_eeee color:black
has shape::perching-in has wwweebeeellorrrreeeeeeeeeeeeteat color:grey has back color::red has bill shape:spa
has shape:perchinn-liii has breast color:grey has back coak__colrr:red has bill shape:spatulate
0
0
Label
Crested Auklet
ECBM
Retrain
has upperparts coolr has leg__ll_____elleee color:oorange has upper tail color:red has bweestt_colorbbbbeeeeeteeee color:blacck has color:reed bel has pwipprrrrcoooiie color:black has shape::erchiin-IIi has breast color:grey has back color:redd has bill spape:spatulate
has upperparts color:orange
has la___ll___ollle
has Bass__uppe_____ sppee__aii__ color:red
0
has wwwpebbrreloorrreeeeeeeeeee color:black
0
has belll__cloo::red
0
has chas_primary_colle color:black has shape::prching-like has breast color:grcy has back baa__bbccccccaaaeeee color:red has bill shape:spatulate
Label
Least Auklet
ECBM
Retrain
has upperparts color:orangee has leg. coloo::oroeeee
has upperparts color:orange 0
1
has__eg lass__ee____oolo
color::ornnge
has_uppes tail color:rred
0
has haas__pper_att taiie__alll___ color:red
has wwweebreellorrrittae color:blacck
has breastt color:bllack has belly selly_coooor:red has pllppprrrmmopllpee color:blacck
has has_ brs__elly_cooorrree color::red has has_ color::black has shape::prrching like
0
0
0
has wwwps::errchiigllik..... has breast color:grey has back color:rred
has breast color:grey has rel has bill shape:spatulate
0
has bill spape:spatuuatt
Label
Rhinoceros Auklet
ECBM
Retrain
has upperparts color:orangee has leg. color:oraage has se___pper__aae tail color:rel
has upperparts color:orange 0
has leg_lle______leeee__ee color:ooaagge
0
has upper_taill taiie_talllleee coll-reee
has www.bettcooorrtiittttteeeeeeeeee color:blacck has_belly belbblyylyoooorrrddd
0
has wwwpstttcloorrbbblllleeeee color::black
0
has_ becllbelllyyaeelleeeeeeeee color:rred beuolllllyoooorreeddd
has lllppprmra__ooooo color:blackk
0
has haa__rimarry colorrrbbcoo
has shape::perching-iii has breast wwweebrerelorrrriiaeeeeeeeeeeeet color:grry has back color:red
has has www.shhppperrcinng-liiie
0
has breast colerrgre has back colo_ccol has bill shape:spatulate
0
has bill spape::patuultte
0
Figure 10: Visualization of the top- 10 most influential concepts for different classes in CUB.
43

---

Editable Concept Bottleneck Models
Label
Brewer Blackbird
ECBM
Retrain
has upperparts coooo::oroooooeeeeee
has colirooooeccooooooaaaaeee
has has_leg_ color:orange
has maa_leeg_ee 0
1
colorr:orange
has ______ppe___ae_ tail color:red
has_ caas__pper_aae List_aaa___ge
cail_colorrred
has wtw__rrrrctrrrrtii_erre___teeeee color:black
0
has breast cottst:oolorbccckk
has (eeely__clooraan
0
has wwwleeelllllllaaa_aa__e_a color::red
has primary_color:bbacc color:black has shape::ercchin-Iie has breastreeoolll color:grey has back color::red has bill shape:spa
has falis_prmary_cooe color:black
has shape:perchiin-liiin has breast color:grey has back coak__colrr:red has bill shape:spatulate
0
0
Label
Red Winged Blackbird
ECBM
Retrain
has upperparts coolr has leg__ll_____elleee color:oorange has upper tail color:red has coansttcolorrbllck has beguboeelycooorrreed has color:black has shape::erchiin-IIi has breast_ color:grey has back color:redd has bill spape:spatulate
has upperparts color:orange
has la___llg__ollle
has cep_puppp_r____ee color:red 0
0
has wwwpebbrreloorrreeeeeeeeeea color:black
color:reed
0
has belll__cllor:red
0
has chas_primary_colle color:black has shape::prching-like has breast color:grcy has back baa__bbccccccaeaeee color:red has bill shape:spatulate
Label
Rusty Blackbird
ECBM
Retrain
has upperparts color:orangee has leg. coloo::oooge
has upperparts color:orange 1
has__eg__r lass__ee___ooooo
color::ornnge
has_upper tail color:rred
has hlas__pper_ttt -aiit_talll____ee color:red 0
has breastreeoolllloeleeeeeeeeeeeete color:blacck
has breast coaast
has ses__blll__ooorrrne color::redd
has belly_coloo::red
has has shape::prrching like
color:
0
has pllppprrrrmmoollee color:blacck has wwwps::ercchingllik..... has breast color:grey has back color:rred
0
has breast color:grey has back rol has bill shape:spatulate
0
0
has bill spape:spatuuatt
Label
Yellow Headed Blackbird
ECBM
Retrain
has upperparts color:orangee has leg. color:oraage has se__upperr_aa tail color:red
has upperparts color:orange
has leg_lle______eeee color:oorange
0 has upper_tail taiie_ttllleeee coll-reee
has www.beeteooorrttiiterteeeeteeete color:blacck beleellllyoolorrredd
0
has wwwpsttccloorrbbllllleeeee color::black
0
has_belll wecl_belllyyyeelleeeeeeeee color:rred beu_lllllyoooorreeddd
has lllpprimra__ooooe color:blackk
has has_primaryy colorrrbbco
shape::perching-iin
has wwwpsshpprcciignllliiiii
has breast wwweebeerelorrrrrieeeeeeee color:grey has back color:red
1
has breast colerrgre
has back colocccolr 0
has bill spape::patuultte
has bill shape:spatulate
Figure 11: Visualization of the top- 10 most influential concepts for different classes in CUB.
44

---

Editable Concept Bottleneck Models
Label
Pine Warbler
Concept Name
Influence Score
has hals_belly_____ wwwelellllllllgggeeeeeeeeeeeeeee coloragrey
0.037985
has underparts color:grey
0.037982
has breast color:grey
0.03798
has bill Lenggng:gonggnrrteeeee than bead
0.037946
has throat color:grey
0.037901
has back color:grcy
0.037894
has wwwp_rrooooohoeeeeeeee grooo__cooor:rrey
0.037868
has cotss_rrmrr___cm pwwparyyycoorrggrreer
0.037866
has shape::sallowwlike
0.037811
Kendal
has nape copp_ooorrrrrey
0.037763
Label
Bewick Wren
Concept Name has color::blue wing_color::blue
Influence Score
0,04231
has wtwpwcoowwooocceeee color::ble
0.042196
has forehead color::blue
0.042055
has bill Lecl_billllsshll shape::patullte has under tail color:blue
0.041994
0.041622
has head patterm:uniqqe pattern
0,041412
has _es__pppe___ae tail color:blue
0,041179
nape color:bbuee
0.040844
has shape::salloww-ike has tail pattem::sottedd
0.040686
0.040507
Label
Song Sparrow
Concept Name
Influence Score
has upperparts color::blue upporparts_cooorbbuue
0.036309
has wiwpwwigg_ooooe color::bo
0.036304
has caass_pimary____ wwtp:paaycoloorbbllee color:blue
0.036271
has back color::blue
0.036261
has wrwwerrownmcmmmmm wtt__coo_ooooobbbleee color::blue
0.036219
has breast color::buue
0.036178
underparts color:bblue euederparts_cloorbblee
has nape color:blue
0.036104
has upper tail color::blue
0.036083
has foreheud wow.ehedd_cllrrrbbllll color:blue
0.035959
Figure 12: Visualization of the most influential concept label related to different data in CUB.
45

---

Editable Concept Bottleneck Models
Label
Brewer Blackbird
Concept Name
Influence Score
has forchcad color:orange
0.042692
has breast color:orange
0.042647
has moss_rron_ccomm coloooooor has throat color:orange
0.042646
0.042588
has cuss_uppe__age tail color:orange
0.042574
has upperpappsssssssc...........ssss _uupperpartt_ooooororanag
0.042569
has primary_color:cnooo color:orange has back color:oorange
0.042546
0.042543
has fass_nape_comm crap__ooor:
0.042484
beelleeellllllleelleeeeeeeeeeeee color:orange
0.042463
Label
Frigatebird
Concept Name
Influence Score
has tail pattemm:multi-olooeed
0.053243
has bill shape::nodle
0.053006
has ltccbbbattcrcenaeeeeeeeeeeeeeeee pateeee:ttttmil-oooooddddd
0.052768
has Batussprimary_aee hllpppaaay_oooooorrrrgge color:orangge
0.052117
has underparts color:orange
0.051954
has under tail color:sorange
0.051617
ctt__croommmoooee color:bufff
0.050712
head pyttien::eyrring
0.049705
has shape::perching-llkc
0,049511
has foreheud color:orange
0.049194
Label
Philadelphia Vireo
Concept Name
Influence Score
has _el__cye___oooeeeeee cooooo:oooeegeee
0.04754
has shape:swalloowlike
0.047265
has bill spape:spatulate
0,047145
has crown color:rufoous
0,047015
has tail pattem::multi-cooore
0.046809
has forehead color:rrfous
0.046604
has back color:rufous
0.046068
has lize:laage (16__332 in)
0.045287
has aass_nape_oom coap._color
0.044857
has upperparts www.erparts_oooorrruuueeo color:rufous
0.043474
Figure 13: Visualization of the most influential concept label related to different data in CUB.
46

---

Editable Concept Bottleneck Models
Label
Seaside Sparrow
Concept
Influence Score
bill,
0.083579
has shape:swallow-Be
0.08343
has nape color:rufous
0.08309
has tail seateem-multi-coleeee has bill longth than head
0.081434
0,079826
has size::lange (16___32__)) buullbbllyyyyollll color:buff
0.078111
0.069643
back color:bliue
0.067222
has eye wetpeeeeeeerreee color:orange has upperparis color::blue
0.063228
0.057842
Label
Vesper Vesper_Sparroie Sparrow
Concept.Nams
Influence Score
has color:red
0.035092
patterncppttedd
0.035083
has bill colortred
0.035072
has, breast. partemsspotted
0.035042
has bill shape:all-purgee purpose
0.03491
has upper tail color:bblue
0.034787
has putlem:cputteed
0.034754
has colon:red
0.034625
has pat__aae___eeee color:reed
0.034427 has threat color:black
Figure 14: Visualization of the most influential concept label related to different data in CUB.
47

---

Editable Concept Bottleneck Models
Label
Vermilion Flycatcker
Concept Name
Influence Score
has bill length:llnggr, than head
0.082524
has size:iizzgeeeeeco.. (16_________________
0.082308
patter--mulii-oooodd
0,079543
leg, color:orange
has shape:swallow-Bee like
0.078894
has back pattern:mmuii-ooorrd
0.074584
has underparts_ color::bbf
0.073978
has bill shape::all-prrpse
0.063468
has rounder tail
0.059044
shape:perching-lkke
0.053268
Label
Fox Sparrow
Concept Name
Influence Score
breast
has underparts hettepartsccolorrbbuee color:bluc
0.04173
has belly color:blue
0.041652
has upper tail, color:bblue
0.041646
has breast pattern:spotted
0.041567
has color:blue
0.041521
tat__rpp_cccccee color::blue has back color::buue
0.041307
has forehead color::blu has under tail, color:bbuue
0.041287
0.041208
Figure 15: Visualization of the most influential concept label related to different data in CUB.
48

---

Editable Concept Bottleneck Models
Concept Name
Influence Score
Joint space narrowing
0.3358
Joint space narrowing lateral
0.1622
Sclerosis femur medial
0.1161
Sclerosis femur lateral
0.0993
Sclerosis tibia lateral
0.0878
Osteophytes tibia medial
0.0724
Osteophytes femur lateral
0.047
Osteophytes tibia lateral
0.031
Osteophytes femur medial
0.0271
Sclerosis tibia medial
0.0213
Concept Name
Influence Score
Joint space narrowing
0.3506
Osteophytes femur medial
0.1698
Osteophytes tibia medial
0.0991
Osteophytes tibia lateral
0.0824
Joint space narrowing lateral
0.0728
Sclerosis tibia lateral
0.0674
Osteophytes femur lateral
0.0595
Sclerosis femur lateral
0.0467
Sclerosis femur medial
0.0272
Sclerosis tibia medial
0.0245
Concept Name
Influence Score
Joint space narrowing
0.2978
Joint space narrowing lateral
0.2018
Osteophytes femur lateral
0.1247
Sclerosis tibia lateral
0.0949
Sclerosis tibia medial
0.0892
Osteophytes femur medial
0.055
Sclerosis femur medial
0.0463
Osteophytes tibia medial
0.0387
Sclerosis femur lateral
0.0321
Osteophytes tibia lateral
0.0195
Figure 16: Visualization of the most influential concept label related to different data in OAI.


Reviews:
[{"review_id": "WmrJ3usaaz", "reviewer": "Reviewer_8zrQ", "summary": "This paper improves Concept Bottleneck Models (CBMs) by proposing how to update or “edit” a well-trained CBM. The issues arise when the concept-label level annotations need to be updated, concepts themselves need to be removed and certain data samples used in the training of the model themselves need to be removed. Rather than retraining a CBM from scratch which is computationally expensive, Editable CBM proposes approaches inspired by influence functions to update model parameters on the aforementioned challenges. Theoretical and empirical results demonstrate the effectiveness of the method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "-*Response to W2: the authors have only considered sequential setting (probably the joint setting as it gives the best performance)*\n\nWe sincerely thank the reviewer for highlighting the importance of the jointly training mode in CBM. We agree that joint training sometimes leads to higher accuracy in both label and concept predictions.\n\nHowever, model performance is not the sole priority: \n1. Compared to joint training, sequential training is more robust under limited data conditions. \n2. Joint training requires balancing concept loss and task loss, which may result in suboptimal performance for both. Sequential training avoids this trade-off. \n3. The modular architecture of sequential training allows for easy post-hoc interventions. \n\nGiven these advantages, we focus on the editable CBM with sequential training in this work. Our goal is to explore model editing, which represents the unique perspective and theoretical contribution of our study. We believe this approach complements, rather than replaces, CBM performance optimization research.\n\nFinally, due to the complexity and workload of designing algorithms for the three editing levels, it is not feasible to analyze both sequential and jointly training methods within a single paper. Therefore, in this work, we focus on developing editing algorithms for sequentially trained CBMs across three levels and provide theoretical guarantees. In fact, editing jointly trained CBMs using influence functions is also achievable and will be considered in our future work.\n\n\n\n-*Response to W3: Theorem 4.4, the authors insert 0 valued rows*\n\nThank you for your suggestion.\n\nWhen a concept is removed, the output dimension of the concept predictor $g$ decreases accordingly. To facilitate estimation, we modify $g$ into $g'$ by inserting a zero-parameter row into its final layer. These parameters remain fixed during training and thus stay zero, ensuring that the model's effective parameter space is strictly a subset of the original space.\n\nIn Theorem 4.4, we approximate $g'$ using influence functions, assuming $g'$ continues training within the original parameter space. Consequently, the algorithm's implementation, including parameter updates, remains unaffected regardless of whether the inserted rows are set to 0, 1, or any constant.\n\n-*Response to W4: Error bars*\n\nWe apologize for the omission and we have addressed it in the camera-ready version.\n\n-*Response to W5: Concept-level metrics*\n\nThanks for your suggestion. We will add experiments about concept-level metrics. We perform experiments on CUB and test the concept accuracy of ECBM and retrain. The results are listed below.\n\n\n\n|         | ECBM(%) | Retrain(%) |\n| ------- | ---- | ---------- |\n| Concept   |  93.7112    |     95.1705      |\n| Data        |  94.5184    |    95.2801|\n| Concept-label  | 95.0219 | 95.1407       |\n\n*Table A: Concept Accuracy of ECBM and Retrain under Three Levels\n\nThe results show that the accuracy of Retrain and ECBM is very close, with differences generally within a small margin. For example, in the Data and Concept-label levels, the accuracy gap is less than 0.5%. The results show that ECBM not only approximates the accuracy of the retrain method's labels on the test set, but also has a similar performance in terms of concept accuracy.\n\n\n-*Response to W6: The related work section*\n\nWe will include more related works in the camera-ready version.\n\n\n## Questions:\n\n-*Response to Q1: What is $R^{d_i}$?*\n\nHere $R^{d_i}$ represents the space of all \n$d_i$-dimensional real vectors.\n\n-*Response to Q2: Why are the concepts in the log?*\n\nThis is the definition of cross-entropy, identical to that in the original CBM, as described in the second paragraph of page 15. The activation function used is the sigmoid function. Note that this is distinct from the definition of the loss function.\n\n-*Response to Q3: ff problem*\n\nThanks for your correction. It should be f.\n\n-*Response to Q4: In lines 167-169 ... Why?*\n\nThis is because if we correct the concept and then retrain the CBM, the concept predictor $\\hat{g}$ will be updated to $\\hat{g}_e$, which differs from the scenario at test time intervention where the concept predictor remains unchanged. This distinction serves as the key motivation for editing CBMs.\n\n-*Response to Q5: Why are no other concept-level metrics utilized?*\n\nSee Response to W5 for reference.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes. They are absolutely correct.", "theoretical_claims": "Yes. I have detailed my reservations in the sections below.", "experimental_designs_or_analyses": "Yes. I have detailed my reservations in the sections below.", "relation_to_broader_scientific_literature": "The paper improves CBMs through Influence functions, an important problem never tried before.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\n1. A very important problem studying the impact of various concept editing mechanisms on trained CBMs without retraining the entire models.\n2. Theoretically sound submission with appropriate proofs and justifications\n\nWeakness:\n1. The overall paper suffers from confusing notations (refer to Questions) and some unusual choice of variable names which hinder understanding. Specifically, \n2. A very important property of CBMs in the original paper [1], was the joint and sequential training paradigm of these models. However, it looks like the authors have only considered one of the settings (probably the joint setting as it gives the best performance). The results should also be demonstrated on the sequential setting to make it truly generalize to all CBMs.\n2.1. In addition to 2, I am still not sure about the theoretical basis of \"correcting\" a jointly trained function $f$ and $g$ would work. If working with a sequential setting, it is easy to see that editing $f$ and $g$ with Hessians is straight-forward, but with joint network training, many of the assumptions are invalid as information has flowed from $f$ to $g$ during training. As an example, in Theorem 4.3, how are we measuring the impact using $\\hat{g}$, which should actually be $\\hat{g} - H_{\\hat{g}}^{-1}$ as the changed params in the concept predictor should influence label predictor as well.\n3. In Theorem 4.4, the authors insert 0 valued rows to make up for the dimensional inconsistency and then remove the same rows after the edit to achieve their desired result. This process is very uncertain, with no claims as to if the 0 valued rows lose important information or not. As a suggestion, the authors can perform a small ablation on the before and after effect of packing these rows with numbers - 0, -1, +1, etc. or report Mutual Information-type metrics.\n4. (Very minor) Error bars on the Time column are not present - why is that?\n5. Why would no concept-level metrics be reported? I understand why F-1 score is used, but editing the model can make a difference in the concept performance as well. In addition, intervention performance as done by [1] are also not reported.\n6. (minor) Lastly, the related work section can be expanded to include other approaches to improve CBMs.\n\n[1] CBMs, Koh et al, ICML '19", "other_comments_or_suggestions": "Refer Weakness.", "questions_for_authors": "1. What is $R^{d_i}$ in Line 118? It is not defined.\n2. In Equation-1, why are the concepts $c_i^j$ in the $log$? This is not consistent with actual CBM, where they are only utilized as a sigmoid.\n3. Typo in Line 149 - ff should be $f$ or $\\hat{f}$ (unclear).\n4. In lines 167-169 what do the authors mean by \"if we intervene with the true concepts, the concept predictor $\\hat{g}$ fluctuates to $\\hat{g_e}$ accordingly\". Why is this the case?\n5. Why are no other concept-level metrics utilized?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "70cQTjY4Ky", "reviewer": "Reviewer_tX1Q", "summary": "The authors present Editable CBMs, where they consider _editability_ from the lens of retraining CBMs at three different levels: 1) Concept Label-level, i.e. when there's label noise in the concept space, 2) Concept level, i.e. removing spurious concepts from the bottleneck predictions and 3) Data-level, i.e. final label noise. For 1), the authors use influence function to estimate the retrain approximation for concept prediction and label predictor sequentially; 2) A similar strategy is applied except for adding a zero-row for the empty concept in the new model; 3) Additional steps to remove the influence of examples with label-noise. The authors use EK-FAC for the second-order approximations and iHVP based algorithms to speed up the compute.\n\nResults are presented on three datasets and additional (extensive) proofs and details are provided in the Appendix.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weakness:\n\n-*Response to W2: the authors have only considered sequential setting (probably the joint setting as it gives the best performance)*\n\nWe sincerely thank the reviewer for highlighting the importance of the jointly training mode in CBM. We agree that joint training sometimes leads to higher accuracy in both label and concept predictions.\n\nHowever, model performance is not the sole priority: \n1. Compared to joint training, sequential training is more robust under limited data conditions. \n2. Joint training requires balancing concept loss and task loss, which may result in suboptimal performance for both. Sequential training avoids this trade-off. \n3. The modular architecture of sequential training allows for easy post-hoc interventions. \n\nGiven these advantages, we focus on the editable CBM with sequential training in this work. Our goal is to explore model editing, which represents the unique perspective and theoretical contribution of our study. We believe this approach complements, rather than replaces, CBM performance optimization research.\n\nFinally, due to the complexity and workload of designing algorithms for the three editing levels, it is not feasible to analyze both sequential and jointly training methods within a single paper. Therefore, in this work, we focus on developing editing algorithms for sequentially trained CBMs across three levels and provide theoretical guarantees. In fact, editing jointly trained CBMs using influence functions is also achievable and will be considered in our future work.\n\n\n\n-*Response to W3: Theorem 4.4, the authors insert 0 valued rows*\n\nThank you for your suggestion.\n\nWhen a concept is removed, the output dimension of the concept predictor $g$ decreases accordingly. To facilitate estimation, we modify $g$ into $g'$ by inserting a zero-parameter row into its final layer. These parameters remain fixed during training and thus stay zero, ensuring that the model's effective parameter space is strictly a subset of the original space.\n\nIn Theorem 4.4, we approximate $g'$ using influence functions, assuming $g'$ continues training within the original parameter space. Consequently, the algorithm's implementation, including parameter updates, remains unaffected regardless of whether the inserted rows are set to 0, 1, or any constant.\n\n-*Response to W4: Error bars*\n\nWe apologize for the omission and we have addressed it in the camera-ready version.\n\n-*Response to W5: Concept-level metrics*\n\nThanks for your suggestion. We will add experiments about concept-level metrics. We perform experiments on CUB and test the concept accuracy of ECBM and retrain. The results are listed below.\n\n\n\n|         | ECBM(%) | Retrain(%) |\n| ------- | ---- | ---------- |\n| Concept   |  93.7112    |     95.1705      |\n| Data        |  94.5184    |    95.2801|\n| Concept-label  | 95.0219 | 95.1407       |\n\n*Table A: Concept Accuracy of ECBM and Retrain under Three Levels\n\nThe results show that the accuracy of Retrain and ECBM is very close, with differences generally within a small margin. For example, in the Data and Concept-label levels, the accuracy gap is less than 0.5%. The results show that ECBM not only approximates the accuracy of the retrain method's labels on the test set, but also has a similar performance in terms of concept accuracy.\n\n\n-*Response to W6: The related work section*\n\nWe will include more related works in the camera-ready version.\n\n\n## Questions:\n\n-*Response to Q1: What is $R^{d_i}$?*\n\nHere $R^{d_i}$ represents the space of all \n$d_i$-dimensional real vectors.\n\n-*Response to Q2: Why are the concepts in the log?*\n\nThis is the definition of cross-entropy, identical to that in the original CBM, as described in the second paragraph of page 15. The activation function used is the sigmoid function. Note that this is distinct from the definition of the loss function.\n\n-*Response to Q3: ff problem*\n\nThanks for your correction. It should be f.\n\n-*Response to Q4: In lines 167-169 ... Why?*\n\nThis is because if we correct the concept and then retrain the CBM, the concept predictor $\\hat{g}$ will be updated to $\\hat{g}_e$, which differs from the scenario at test time intervention where the concept predictor remains unchanged. This distinction serves as the key motivation for editing CBMs.\n\n-*Response to Q5: Why are no other concept-level metrics utilized?*\n\nSee Response to W5 for reference.\n\n\n# Reviewer tX1Q\n\n-*Response to Other Comments Or Suggestions*\n\nThanks for your comments. We will check the paper, fix all the errors and define editable CBM explicitly in the camera-ready version.", "claims_and_evidence": "1. The claims of editability on the three levels are sufficiently shown in the results and appendix.\n2. My problem is with the general presentation of results: while the algorithm and results certainly show that this algorithm provides pretty close results to retraining, the results and the language do not contain the original essence of CBMs which is to provide humans the ability to _intervene_ on the intermediate concepts to _better_ the final prediction. In all of the theory to prove that the modifications made by authors to fit the CBM framework, this seems to have been lost. The authors do not compare how the concept intervention behavior changes / remains same for different levels of intervention - based on the language, it seems like final accuracy is the only focus of the paper. \n3. Furthermore, a lot of comparisons with modern CBM architectures are missing - for example, the simple choice of soft concepts vs hard concept choices are not acknowledged.(Mahinpei et al) or a note to the ubiquity / lack thereof of the proposed algorithm.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria barring the critique above makes sense.", "theoretical_claims": "Not fully (only the main parts)", "experimental_designs_or_analyses": "Refer to point 2 and 3 in claims and evidence", "relation_to_broader_scientific_literature": "CBMs are gaining popularity, and fast re-training is important; this work certainly furthers improving adoption.", "essential_references_not_discussed": "CBMs:\n[1] Mahinpei, Anita, et al. \"Promises and pitfalls of black-box concept learning models.\" arXiv preprint arXiv:2106.13314 (2021).\nLabel noise\n[2] Thulasidasan, Sunil, et al. \"Combating label noise in deep learning using abstention.\" arXiv preprint arXiv:1905.10964 (2019).\n[3] Rolnick, David, et al. \"Deep learning is robust to massive label noise.\" arXiv preprint arXiv:1705.10694 (2017)\n[4] Balloli, Vaibhav, Sara Beery, and Elizabeth Bondi-Kelly. \"Are they the same picture? adapting concept bottleneck models for human-AI collaboration in image retrieval.\" arXiv preprint arXiv:2407.08908 (2024).", "other_strengths_and_weaknesses": "The works is original, has signficant impact and novel. \n\nA key weakness is the focus of results which only focuses on final accuracy while other things that could be shown is test-time intervention performance of retrain vs ECBM, robustness to intervention noise, etc.\nAnother weakness is the limitation and broader impact section which is poorly written - no real limitation except stating it is an approximation and broader impact vaguely mentions doctors and that's it. It is okay to simply state cost saving instead of vaguely writing \"ECBM can be an interactive model with doctors in the real world, which is an editable explanation tool.\" Regarding limitation, the authors can choose to address how many more modifications are required to other architectures like CEM (Zarlenga et al), etc.", "other_comments_or_suggestions": "1. The authors are encouraged to proof-read grammar in a lot of places (missing oxford commas, unnecessarily lengthy sentences, etc.)\n2. Define editability early on in the introduction to better prime the readers on what to expect (see also adaptivity -> adaptability in Section 1)", "questions_for_authors": "The authors are requested to address/clarify all the weaknesses and concerns stated above - with sufficient clarification, I'm willing to raise the score to WA/A.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aw32V9SPwD", "reviewer": "Reviewer_mn5G", "summary": "The paper introduces Editable Concept Bottleneck Models (ECBMs), an extension of Concept Bottleneck Models (CBMs) that allows efficient data and concept removal without full retraining. Using influence functions and Hessian-based approximations, ECBMs support three levels of editability: concept-label, concept, and data-level. Experiments on multiple datasets show that ECBMs achieve similar performance to retraining while being 20-30x faster, making them highly efficient for real-world applications. The work enhances CBMs' adaptability and interpretability but could further explore concept addition and real-world deployment.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "-*Response to Weaknesses 1*\n\nThanks for your invaluable advice. We will add this part in the revision. Here, we provide the analysis for algorithm 1.\n\nThe time complexity of the algorithm is \\( O(n \\cdot (m^2 + d^2) + s_e \\cdot m^2 + d^3) \\), where \\( n \\) is the number of data points, \\( m \\) is the dimensionality of \\( \\hat{g} \\), \\( d \\) is the dimensionality of \\( \\hat{f} \\), and \\( s_e \\) is the size of the erroneous data set \\( S_e \\). Computing the Hessian matrices for \\( \\hat{g} \\) and \\( \\hat{f} \\) takes \\( O(n \\cdot m^2) \\) and \\( O(n \\cdot d^2) \\), respectively, while the updates for \\( \\hat{g} \\) and \\( \\hat{f} \\) contribute \\( O(s_e \\cdot m^2) \\) and \\( O(n \\cdot d^2 + d^3) \\). The space complexity is \\( O(m^2 + d^2 + n \\cdot (m + d)) \\), dominated by storing the Hessian matrices and the required gradients across all data points.\n\n-*Response to Weaknesses 2 and 3*\n\nWe will modify Figure 1 and improve our notation in the revision.\n\n-*Response to Question 1 and 2*\nThank you for the valuable suggestion. We will include the time and complexity part in the revision.\n\nWe agree that influence function may not be familiar to all readers, and we appreciate the opportunity to make our work more accessible. In response, we propose to include an additional appendix section that provides a high-level, intuitive explanation of influence functions.\n\nThank you very much for your valuable feedback and recognition of our article. If we have addressed all of your concerns, we kindly ask you to consider giving a higher rating.", "claims_and_evidence": "Yes, the experiments and analyses support the claims.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria are appropriate.", "theoretical_claims": "The paper provides a closed-form solution for the model approximation of CBMs, which includes theoretical claims. I have checked the correctness of the derivation and did not identify any issues.", "experimental_designs_or_analyses": "Yes. They design experiments to evaluate the utility and efficiency of ECBMs, comparing them with retraining and CBM-IF. They analyze the impact of different edit settings, concept importance, and data removal. They further validate ECBMs using membership inference attacks. These designs and analyses are valid.", "relation_to_broader_scientific_literature": "The paper extends CBM by introducing efficient editability using influence functions, building on prior CBM research and influence function applications.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths \n- The paper extends CBMs by incorporating editable capabilities, addressing practical issues such as privacy concerns, annotation errors, and dataset corrections. This is particularly useful for dynamic datasets that require frequent updates.\n\n- The paper evaluates ECBMs on multiple datasets (OAI, CUB, and CelebA), demonstrating that ECBMs achieve near-identical performance to retraining while reducing computation time by up to 30x.\n\n- The paper presents closed-form solutions using influence functions, avoiding costly retraining while maintaining accuracy.\n\n- ECBMs provide an efficient way to remove concept biases and erase data influences, addressing model privacy and fairness concerns.\n\n- The incorporation of EK-FAC further accelerates computation, making ECBMs scalable.\n\n\n\nWeaknesses\n- While the paper discusses the computational advantages of ECBMs, a more detailed analysis of time and space complexity would strengthen the scalability argument.\n\n- The font size in Figure 1 is too small, making it difficult to read. Given its importance, it would be beneficial to adjust the layout (e.g., expanding it to a two-column format).\n\n- Some mathematical notation could be better explained, particularly for readers unfamiliar with influence functions. A more intuitive explanation or additional background material would improve clarity.", "other_comments_or_suggestions": "Please refer to the weaknesses section above.", "questions_for_authors": "- Could the authors provide a more detailed analysis of the time and space complexity of ECBMs? While the empirical results demonstrate efficiency, a formal complexity discussion would further support the claims regarding scalability.\n- Some derivations rely on influence functions, which may be unfamiliar to many readers. Could the authors provide a more accessible explanation or an appendix section with a high-level intuition behind these functions?\n\n\nOverall, I did not find any significant weaknesses in the paper. I may change my score based on the authors’ responses regarding weaknesses and above questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6LQk6zm0uv", "reviewer": "Reviewer_iqcY", "summary": "Editable CBMs provide the ability to edit a trained CBM to account for issues in annotation errors, concept set changes, and problems with specific data points. That is done with the help of influence functions that approximate the model.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "-*Response to Claims And Evidence*\n\nWe respectfully disagree with your opinion. The primary goal of CBM is to explicitly decompose the model's intermediate representation into a set of interpretable concepts, typically predefined by domain experts or specific task requirements before training. However, when task requirements change, new concepts may need to be added to the existing concept set, making such scenarios possible in practice. Most CBM methods, however, fix the concept set after training, with the model's structure and parameters tightly bound to these concepts. As a result, directly adding new concepts post-training is challenging and often requires retraining the model.\n\n-*Response to Relation To Broader Scientific Literature*\n\nThank you for sharing your perspective. I understand your concerns and would like to highlight why CBM editing is important:\n\n1. Correcting Labeling Errors: In fields like healthcare, training data is valuable, and discarding mislabeled data isn’t ideal. CBM editing allows targeted corrections without costly retraining.\n\n2. Updating Concepts: During deployment, missing or irrelevant concepts may arise. CBM editing enables efficient updates.\n\n3. Privacy Constraints: CBM editing allows data removal requests to be handled accurately without full model retraining.\n\nAs you mentioned, retraining CBMs can be computationally expensive due to their bottleneck structure. This makes CBM editing a practical alternative. \n\nIn summary, model editing effectively addresses practical challenges such as correcting data errors, ensuring privacy, and adapting to new requirements, all without the high cost of retraining. These factors highlight the importance of CBM editing in real-world applications, as acknowledged by the other reviewers.\n\n\n-*Response to Essential References Not Discussed, Other Strengths And Weaknesses and Questions For Authors 2*\n\nThank you for your thoughtful perspective. While we acknowledge that editing CBM within the context of privacy constraints does share some similarities with machine unlearning, this work is not solely focused on CBM and machine unlearning. Our primary objective is to enable the flexible editing of CBM across three levels: data, concept, and concept-label. This process goes beyond unlearning to encompass modification and optimization as well. Ultimately, the core goal of this work is to enhance the applicability and adaptability of CBM, rather than to design a new machine unlearning algorithm specifically for CBM.\n\n-*Response to Questions For Authors 1*\n\nThanks for your information. [1] proposes a method for intervening on the intermediate representations of neural networks, but its network architecture is not based on CBMs, thereby diverging from the goal of editing the original CBM framework.\n\nIn contrast, our work focuses on the specific application scenarios of CBMs. By addressing model editing at the levels of data, concepts, and concept labels, we systematically formulate this problem mathematically for the first time and develop editing algorithms with theoretical guarantees. Our research fills a critical gap in this area.\n\n\n-*Response to Questions For Authors 3*\n\nYes. Our code can be found here.\nhttps://anonymous.4open.science/r/ECBM-4B14\n\n\nThank you very much for your valuable feedback and recognition of our article. If we have addressed all of your concerns, we kindly ask you to consider giving a higher rating.", "claims_and_evidence": "The claims made are largely clear and supported by evidence. There is one exception, which is that in the motivation, the concept-level editing is motivated by the fact that oftentimes one wants to *add* concepts to the concept set post-hoc. As far as I understand this work, only the removal of a concept is possible, but not the addition thereof. Thus, I suggest changing this framing.", "methods_and_evaluation_criteria": "The proposed method is well-motivated and sound. The datasets used, while simplistic, are established benchmarks in the field.", "theoretical_claims": "I did not check the proofs in the Appendix, however, the equations in the main text intuitively do not contradict my intuition of how they should be.", "experimental_designs_or_analyses": "The empirical evaluation is sound and thorough, i.e. the authors measure the traits that ECBMs are supposed to fulfill. It is impressive that ECBMs' performance is close to \"Retrain\", which functions as an Oracle.", "relation_to_broader_scientific_literature": "This work contributes to the literature in CBMs. To the best of my knowledge, in the context of CBMs, model editing of this sort has not been explored. Personally, I am not convinced that model editing is such an important task in CBMs, as their concept bottleneck prevents them anyways from being too large to be retrained.", "essential_references_not_discussed": "I recommend moving the related work section on Machine Unlearning into the main text. It would help framing the paper correctly and understanding the contribution of this work.", "other_strengths_and_weaknesses": "I am not well-read in the domain of machine unlearning, but I am very sure that they are highly relevant to this work. That is, my intuition tells me that this work is essentially CBM + Machine Unlearning, and I am unsure how much novelty there is with respect to the existing methods in the field of Machine Unlearning.", "other_comments_or_suggestions": "The abbreviation ECBM is already used by the published energy-based concept bottleneck models", "questions_for_authors": "My main reasons for not giving a higher score are \n1. that model editing in the context of CBMs is not a big problem in my opinion. I think existing methods such as [1] could easily be adapted to quickly retrain with the adapted dataset. As such, the significance is limited in my opinion. \n\n2. I am unsure of the novelty with respect to existing methods in Machine Unlearning, as the usage of Influence functions for this purpose appears quite standard, and I would imagine that the editing of encoder and predictor of the CBM can be \"mapped\" to some existing tasks in that field. \n\nWhile these are not questions, I invite the authors to comment on these opinions.\n\n3. Can the authors provide code for reproducibility?\n\n\n[1] Laguna, Sonia, et al. \"Beyond Concept Bottleneck Models: How to Make Black Boxes Intervenable?.\" The Thirty-eighth Annual Conference on Neural Information Processing Systems.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a practical and well-motivated method for editing Concept Bottleneck Models (CBMs) without retraining, using influence function-inspired techniques. The approach is theoretically grounded and empirically validated across key scenarios such as concept and data removal. The rebuttal satisfactorily addressed most concerns, particularly around scalability and effectiveness. In the final version, the authors may consider including empirical results for test-time interventions and broader applicability to newer CBM variants (e.g., CEM, retrieval-based CBMs).
Decision: Accept (poster)