Editable Concept Bottleneck Models

## Abstract

Abstract Concept Bottleneck Models (CBMs) have garnered much attention for their ability to elucidate the prediction process through a human-understandable concept layer. However, most previous studies focused on cases where the data, including concepts, are clean. In many scenarios, we always need to remove/insert some training data or new concepts from trained CBMs due to different reasons, such as privacy concerns, data mislabelling, spurious concepts, and concept annotation errors. Thus, the challenge of deriving efficient editable CBMs without retraining from scratch persists, particularly in large-scale applications. To address these challenges, we propose Editable Concept Bottleneck Models (ECBMs). Specifically, ECBMs support three different levels of data removal: concept-label-level, concept-level, and data-level. ECBMs enjoy mathematically rigorous closed-form approximations derived from influence functions that obviate the need for re-training. Experimental results demonstrate the efficiency and effectiveness of our ECBMs, affirming their adaptability within the realm of CBMs.

## 1 Introduction

Modern deep learning models, such as large language models (Zhao et al., ; Yang et al., ; ; Xu et al., ; Yang et al., ) and large multimodal (Yin et al., ; Ali et al., ; Cheng et al., ), often exhibit intricate non-linear architectures, posing challenges for end-users seeking to comprehend and trust their decisions. This lack of interpretability presents a significant barrier to adoption, particularly in critical domains such as healthcare (Ahmad et al., ; Yu et al., ) and finance (Cao, ), where transparency is paramount. To address this demand, explainable artificial intelligence (XAI) models (Das & Rad, ; Hu et al., ; ) have emerged, offering explanations for their behavior and insights into their internal mechanisms. Among these, Concept Bottleneck Models (CBMs) (Koh et al., ) have gained prominence for explaining the prediction process of end-to-end AI models. CBMs add a bottleneck layer for placing human-understandable concepts. In the prediction process, CBMs first predict the concept labels using the original input and then predict the final classification label using the predicted concept in the bottleneck layer, which provides a self-explained decision to users.

Figure: Figure 1: An illustration of Editable Concept Bottleneck Models with three settings.
Refer to caption: x1.png

Existing research on CBMs predominantly addresses two primary concerns: Firstly, CBMs heavily rely on laborious dataset annotation. Researchers have explored solutions to these challenges in unlabeled settings (Oikarinen et al., ; Yuksekgonul et al., ; Lai et al., ). Secondly, the performance of CBMs often lags behind that of original models lacking the concept bottleneck layer, attributed to incomplete information extraction from original data to bottleneck features. Researchers aim to bridge this utility gap (Sheth & Ebrahimi Kahou, ; Yuksekgonul et al., ; Espinosa Zarlenga et al., ). However, few of them considered the adaptivity or editability of CBMs, crucial aspects encompassing annotation errors, data privacy considerations, or concept updates. Actually, these demands are increasingly pertinent in the era of large models. We delineate the editable setting into three key aspects (illustrated in Figure 1):

- •
*Concept-label-level:*
In most scenarios, concept labels
are annotated by humans or experts. Thus, it is unavoidable that there are some annotation errors, indicating that there is a need to correct some concept labels in a trained CBM.
- •
*Concept-level:*
In CBMs, the concept set is pre-defined by LLMs or experts. However, in many cases, evolving situations demand concept updates, as evidenced by discoveries such as chronic obstructive pulmonary disease as a risk factor for lung cancer, and doctors have the requirements to add related concepts. For another example, recent research found a new factor, obesity (Sattar et al., ) are risky for severe COVID-19 and factors (e.g., older age, male gender, Asian race) are risk associated with COVID-19 infection (Rozenfeld et al., ).
On the other hand, one may also want to remove some spurious or unrelated concepts for the task. This demand is even more urgent in some rapidly evolving domains like the pandemic.
- •
*Data-level:*
Data issues can arise in CBMs when training data is erroneous or poisoned. For example, if a doctor identifies a case as erroneous or poisoned, this data sample becomes unsuitable for training. Therefore, it is essential to have the capability to completely delete such data from the learning models. We need such an editable model that can interact effectively with doctors.

The most direct way to address the above three problems is retraining from scratch on the data after correction. However, retraining models in such cases prove prohibitively expensive, especially in large models, which is resource-intensive and time-consuming. Therefore, developing an efficient method to approximate prediction changes becomes paramount. Providing users with an adaptive and editable CBM is both crucial and urgent.

We propose Editable Concept Bottleneck Models (ECBMs) to tackle these challenges. Specifically, compared to retraining, ECBMs provide a mathematically rigorous closed-form approximation for the above three settings to address editability within CBMs efficiently. Leveraging the influence function (Cook, ; Cook & Weisberg, ), we quantify the impact of individual data points, individual concept labels, and the concept for all data on model parameters.
Despite the growing attention and utility of influence functions in machine learning (Koh & Liang, ), their application in CBMs remains largely unexplored due to their composite structure, i.e., the intermediate representation layer.

To the best of our knowledge, we are the first to work to fill this gap by demonstrating the effectiveness of influence functions in elucidating the behavior of CBMs, especially in identifying mislabeled data and discerning the data influence. Comprehensive experiments on benchmark datasets show that our ECBMs are efficient and effective. Our contributions are summarized as follows.

- •
We delineate three different settings that need various levels of data or concept removal in CBMs: concept-label-level, concept-level, and data-level. To the best of our knowledge, our research marks the first exploration of data removal issues within CBMs.
- •
To make CBMs able to remove data or concept influence without retraining,
we propose the Editable Concept Bottleneck Models (ECBMs). Our approach in ECBMs offers a mathematically rigorous closed-form approximation. Furthermore, to improve computational efficiency, we present streamlined versions integrating Eigenvalue-corrected Kronecker-Factored Approximate Curvature (EK-FAC).
- •
To showcase the effectiveness and efficiency of our ECBMs, we conduct comprehensive experiments across various benchmark datasets to demonstrate our superior performance.

## 2 Related Work

Concept Bottleneck Models.
CBM (Koh et al., ) stands out as an innovative deep-learning approach for image classification and visual reasoning. It introduces a concept bottleneck layer into deep neural networks, enhancing model generalization and interpretability by learning specific concepts. However, CBM faces two primary challenges: its performance often lags behind that of original models lacking the concept bottleneck layer, attributed to incomplete information extraction from the original data to bottleneck features. Additionally, CBM relies on laborious dataset annotation. Researchers have explored solutions to these challenges. Chauhan et al. () extend CBM into interactive prediction settings, introducing an interaction policy to determine which concepts to label, thereby improving final predictions. Oikarinen et al. () address CBM limitations and propose a novel framework called Label-free CBM. This innovative approach enables the transformation of any neural network into an interpretable CBM without requiring labeled concept data, all while maintaining high accuracy. Post-hoc Concept Bottleneck models (Yuksekgonul et al., ) can be applied to various neural networks without compromising model performance, preserving interpretability advantages. CBMs work on the image field also includes (Havasi et al., ; Kim et al., ; Keser et al., ; Sawada & Nakamura, ; Sheth & Kahou, ). Despite many works on CBMs, we are the first to investigate the interactive influence between concepts through influence functions. Our research endeavors to bridge this gap by utilizing influence functions in CBMs, thereby deciphering the interaction of concept models and providing an adaptive solution to concept editing. For more related work, please refer to Appendix H.

## 3 Preliminaries

Concept Bottleneck Models. In this paper, we consider the original CBM, and we adopt the notations used by Koh et al. (). We consider a classification task with a concept set denoted as $\bm{c}=\{p_{1},\cdots,p_{k}\}$ with each $p_{i}$ is a concept given by experts or LLMs, and a training dataset represented as $\mathcal{D}=\{z_{i}\}_{i=1}^{n}$, where $z_{i}=(x_{i},y_{i},c_{i})$. Here, for $i\in[n]$, $x_{i}\in\mathbb{R}^{m}$ represents the feature vector, $y_{i}\in\mathbb{R}^{d_{z}}$ denotes the label (with $d_{z}$ corresponding to the number of classes), and $c_{i}=(c_{i}^{1},\cdots,c_{i}^{k})\in\mathbb{R}^{k}$ represents the concept vector. In this context, $c_{i}^{j}$ represents the weight of the concept $p_{j}$. In CBMs, our goal is to learn two representations: one called concept predictor that transforms the input space to the concept space, denoted as $g:\mathbb{R}^{m}\to\mathbb{R}^{k}$, and another called label predictor that maps the concept space to the prediction space, denoted as $f:\mathbb{R}^{k}\to\mathbb{R}^{d_{z}}$. Usually, here the map $f$ is linear. For each training sample $z_{i}=(x_{i},y_{i},c_{i})$, we consider two empirical loss functions: one is from the input space to concept space, and the other is from concept space to output space:

$$ $\displaystyle\hat{g}=\operatorname*{arg\,min}_{g}\sum_{j=1}^{k}\sum_{i=1}^{n}L _{C}(g^{j}(x_{i}),c_{i}^{j})=\operatorname*{arg\,min}_{g}\sum_{j=1}^{k}\sum_{i =1}^{n}L_{C_{j}}(g(x_{i}),c_{i}),$ (1) $\displaystyle\hat{f}=\operatorname*{arg\,min}_{f}\sum_{i=1}^{n}L_{Y}(f(\hat{g} (x_{i})),y_{i})=\operatorname*{arg\,min}_{f}\sum_{i=1}^{n}L_{Y_{i}}(f,\hat{g}),$ (2) $$

where $g^{j}(x_{i})$ is the $j$-th concept predictor with $x_{i}$, $L_{C}$ and $L_{Y}$ are loss functions. We treat $g$ as a collection of $k$ concept predictors and separate different columns as a vector $g^{j}(x_{i})$ for simplicity. Furthermore, in this paper, we primarily focus on the scenarios where the label predictor $f$ is a linear transformation.

For any input $x$, we aim to ensure that its predicted concept vector $\hat{c}=g(x)$ and prediction $\hat{y}=f(\hat{g}(x))$ are close to their underlying counterparts, thus capturing the essence of the original CBMs.

Influence function. The influence function is a measure of the dependence of the estimator on the value of any one of the points in the sample. Consider a neural network $\hat{\theta}=\arg\min_{\theta}\sum_{i=1}^{n}\ell(z_{i};\theta)$ with loss function $\ell$ and dataset $D=\{z_{i}\}_{i=1}^{n}$. If we remove a point $z_{m}$ from the training dataset, the parameters become $\hat{\theta}_{-z_{m}}=\arg\min_{\theta}\sum_{i\neq m}\ell(z_{i};\theta)$. Influence function gives us an efficient approximation for $\hat{\theta}_{-z_{m}}$ by defining a response function as $\hat{\theta}_{\epsilon,-z_{m}}=\underset{\theta\in\Theta}{\operatorname{argmin
}}\frac{1}{n}\sum_{i=1}^{n}L(z_{i};\theta)+\epsilon L(z_{m};\theta).$ Perform first-order Taylor expansion on the gradient of the objective function corresponding to the $\operatorname*{arg\,min}$ process of the response function, we can obtain the influence function defined by

$$ $\mathcal{I}_{\hat{\theta}}\left(z_{m}\right)\triangleq\left.\frac{\mathrm{d}{ \hat{\theta}}}{\mathrm{d}\epsilon}\right|_{\epsilon=0}=-H^{-1}_{\hat{\theta}} \cdot\nabla_{\theta}\ell(z_{m};{\hat{\theta}}),$ $$

which can evaluate the influence of $z_{m}$ on the parameters.
When the loss function $\ell$ is twice-differentiable and strongly convex in $\theta$, the Hessian $H_{{\hat{\theta}}}$ is positive definite and thus the influence function is well-defined. For non-convex loss, Bartlett () proposed that the Hessian $H_{{\hat{\theta}}}$ can be replaced by $\hat{H}=G_{\hat{\theta}}+\lambda I$
where $G_{\hat{\theta}}$ is the Fisher information matrix defined by $n^{-1}\sum_{i=1}^{n}\nabla_{\theta}\ell(z_{i};\theta)\nabla_{\theta}\ell(z_{i}
;\theta)^{\mathrm{T}}$, $\lambda$ is a small damping term used to ensure the positive definiteness of $\hat{H}$. We can employ the Eigenvalue-corrected Kronecker-Factored Approximate Curvature (EK-FAC) method to further accelerate the computation. See Appendix B for details.

## 4 Editable Concept Bottleneck Models

In this section, we introduce our EBCMs for the three settings mentioned in the introduction by leveraging the influence function. Specifically, for the concept-label level, we will calculate the influence of a set of data sample’s different concept labels; for the concept level, we will calculate the influence of several concepts; for the data level, we will calculate the influence of several samples.

### 4.1 Concept Label-level Editable CBM

In many cases, several data samples possess erroneous annotations for certain concepts, we may opt to preserve its other information, particularly considering the high cost associated with acquiring data in specific domains like medical imaging.
In such scenarios, it is common practice to correct such erroneous concepts instead of removing the whole data point from the dataset. Estimating the changes in the parameters of the retraining model holds significance in this context.
We name this case as concept label-level editable CBM.

Mathematically, we have a set of erroneous data $D_{e}$ and its associated index set $S_{e}\subseteq[n]\times[k]$ such that for each $(w,r)\in S_{e}$, we have $(x_{w},y_{w},c_{w})\in D_{e}$ with $c_{w}^{r}$ is mislabeled and $\tilde{c}_{w}^{r}$ is its corrected concept label. Thus, our goal is to approximate the new CBM without retraining. The retrained concept predictor label predictor will be represented in the following manner.

$$ $\displaystyle\hat{g}_{e}=\operatorname*{arg\,min}_{g}\left[\sum_{(i,j)\notin S _{e}}L_{C}\left(g^{j}(x_{i}),c_{i}^{j}\right)+\sum_{(i,j)\in S_{e}}L_{C}\left( g^{j}(x_{i}),\tilde{c}_{i}^{j}\right)\right],$ (3) $\displaystyle\hat{f}_{e}=\operatorname*{arg\,min}_{f}\sum_{i=1}^{n}L_{Y}\left( f\left(\hat{g}_{e}\left(x_{i}\right)\right),y_{i}\right).$ (4) $$

For simple neural networks, we can use the influence function approach directly to estimate the retrained model. However, for CBM architecture, if we intervene with the true concepts, the concept predictor $\hat{g}$ fluctuates to $\hat{g}_{e}$ accordingly. Observing that the input data of the label predictor is the output of the concept predictor, which is also changing. Therefore, we need to adopt a two-stage edition approach. Here we consider the influence function for equation 3 and equation 4 separately. We first edit the concept predictor from $\hat{g}$ to $\bar{g}_{e}$, and then edit from $\hat{f}$ to $\bar{f}_{e}$ based on our approximated concept predictor, by the following two theorems.

###### Theorem 4.1 .

The retrained concept predictor $\hat{g}_{e}$ defined by (3) can be approximated by:

$$ $\hat{g}_{e}\approx\bar{g}_{e}\triangleq\hat{g}-H^{-1}_{\hat{g}}\cdot\sum_{(w,r )\in S_{e}}\left(\nabla_{\hat{g}}L_{C}\left(\hat{g}^{r}(x_{w}),\tilde{c}_{w}^{ r}\right)-\nabla_{\hat{g}}L_{C}\left(\hat{g}^{r}(x_{w}),c_{w}^{r}\right)\right),$ (5) $$

where $H_{\hat{g}}=\nabla^{2}_{\hat{g}}\sum_{i,j}L_{C}(\hat{g}^{j}(x_{i}),c_{i}^{j})$ is the Hessian matrix of the loss function respect to $\hat{g}$.

###### Theorem 4.2 .

The retrained label predictor $\hat{f}_{e}$ defined by equation 4 can be approximated by:

$$ $\displaystyle\hat{f}_{e}$ $\displaystyle\approx\bar{f}_{e}=\hat{f}+H^{-1}_{\hat{f}}\cdot\nabla_{f}\sum_{i =1}^{n}L_{Y_{i}}\left(\hat{f},\hat{g}\right)-H^{-1}_{\hat{f}}\cdot\nabla_{f} \sum_{i=1}^{n}L_{Y_{i}}\left(\hat{f},\bar{g}_{e}\right),$ $$

where $H_{\hat{f}}=\nabla^{2}_{\hat{f}}\sum_{i=1}^{n}L_{Y_{i}}(\hat{f},\hat{g})$ is the Hessian matrix of the loss function respect to $\hat{f}$, $L_{Y_{i}}(\hat{f},\hat{g})\triangleq L_{Y}(\hat{f}(\hat{g}(x_{i})),y_{i})$, and $\bar{g}_{e}$ is given in Theorem 4.1.

Difference with test-time intervention.
The ability to intervene in CBMs enables human users to interact with the model in the prediction process, for example, a medical expert can substitute the erroneous predicted concept value $\hat{c}$ directly, and then observe its effect on the final prediction $\hat{y}$. However, the fundamental flaws in the concept predictor have not been thoroughly rectified, and similar errors may persist when applied to new test data. While under the editable CBM framework, not only can test-time intervention be performed, but the concept predictor of the CBM can also undergo secondary editing based on the test data that repeatedly yields errors. This process extends the rectification from the data level to the model level.

###### Theorem 4.1 .

###### Theorem 4.2 .

### 4.2 Concept-level Editable CBM

In this case, a set of concepts is removed due to incorrect attribution or spurious concepts, termed concept-level edit. (^1^11For convenience, in this paper, we only consider concept removal; our method can directly extend to concept insertion.)Specifically, for the concept set, denote the erroneous concept index set as $M\subset[k]$, we aim to delete these concept labels in all training samples. We aim to investigate the impact of updating the concept set within the training data on the model’s predictions.
It is notable that compared to the above concept label case, the dimension of output (input) of the retrained concept predictor (label predictor) will change. If we delete $t$ concepts from the dataset, then ${g}$ becomes ${g}^{\prime}:\mathbb{R}^{m}\rightarrow\mathbb{R}^{k-t}$ and ${f}$ becomes ${f}^{\prime}:\mathbb{R}^{k-t}\rightarrow\mathbb{R}^{d_{z}}$.
More specifically, if we retrain the CBM with the revised dataset, the corresponding concept predictor becomes:

$$ $\hat{g}_{-p_{M}}=\operatorname*{arg\,min}_{g^{\prime}}\sum_{j\notin M}L_{C_{j} }=\operatorname*{arg\,min}_{g^{\prime}}\sum_{j\notin M}\sum_{i=1}^{n}L_{C}({g^ {\prime}}^{j}(x_{i}),c_{i}^{j}).$ (6) $$

The variation of the parameters in dimension renders the application of influence function-based editing challenging for the concept predictor. This is because the influence function implements the editorial predictor by approximate parameter change from the original base after $\epsilon$-weighting the corresponding loss for a given sample, and thus, it is unable to deal with changes in parameter dimensions.

To overcome the challenge, our strategy is to develop some transformations that need to be performed on $\hat{g}_{-p_{M}}$ to align its dimension with $\hat{g}$ so that we can apply the influence function to edit the CBM. We achieve this by mapping $\hat{g}_{-p_{M}}$ to $\hat{g}^{*}_{-p_{M}}\triangleq\mathrm{P}(\hat{g}_{-p_{M}})$, which has the same amount of parameters as $\hat{g}$ and has the same predicted concepts $\hat{g}^{*}_{-p_{M}}(j)$ as $\hat{g}_{-p_{M}}(j)$ for all $j\in[k]-M$. We achieve this effect by inserting a zero row vector into the $r$-th row of the matrix in the final layer of $\hat{g}_{-p_{M}}$ for $r\in M$. Thus, we can see that the mapping $P$ is one-to-one. Moreover, assume the parameter space of $\hat{g}$ is $T$ and that of $\hat{g}^{*}_{-p_{M}}$, $T_{0}$ is the subset of is the subset of $T$. Noting that $\hat{g}^{*}_{-p_{M}}$ is the optimal model of the following objective function:

$$ $\hat{g}^{*}_{-p_{M}}=\operatorname*{arg\,min}_{g^{\prime}\in T_{0}}\sum_{j \notin M}^{k}\sum_{i=1}^{n}L_{C_{j}}(g^{\prime j}(x_{i}),c_{i}^{j}),$ (7) $$

i.e., it is the optimal model of the concept predictor loss on the remaining concepts under the constraint $T_{0}$. Now we can apply the influence function to edit $\hat{g}$ to approximate $\hat{g}^{*}_{-p_{M}}$ with the restriction on the value of 00 for rows indexed by MM with the last layer of the neural network, denoted as $\bar{g}^{*}_{-p_{M}}$. After that, we remove from $\bar{g}^{*}_{-p_{M}}$ the parameters initially inserted to fill in the dimensional difference, which always equals 00 because of the restriction we applied in the editing stage, thus approximating the true edited concept predictor $\hat{g}_{-p_{M}}$. We now detail the editing process from $\hat{g}$ to $\hat{g}^{*}_{-p_{M}}$ using the following theorem.

###### Theorem 4.3 .

For the retrained concept predictor $\hat{g}_{-p_{M}}$ defined in equation 6, we map it to $\hat{g}^{*}_{-p_{M}}$ as equation 7. And we can edit the initial $\hat{g}$ to $\hat{g}^{*}_{-p_{M}}$ as:

$$ $\hat{g}^{*}_{-p_{M}}\approx\bar{g}^{*}_{-p_{M}}\triangleq\hat{g}-H^{-1}_{\hat{ g}}\cdot\nabla_{\hat{g}}\sum_{j\notin M}\sum_{i=1}^{n}L_{C_{j}}(\hat{g}(x_{i}) ,c_{i}),$ $$

where $H_{\hat{g}}=\nabla^{2}_{g}\sum_{j\notin M}\sum_{i=1}^{n}L_{C_{j}}(\hat{g}(x_{i
}),c_{i})$.
Then, by removing all zero rows inserted during the mapping phase, we can naturally approximate $\hat{g}_{-p_{M}}\approx\mathrm{P}^{-1}(\hat{g}^{*}_{-p_{M}})$.

For the second stage of training, assume we aim to remove concept $p_{r}$ for $r\in M$ and the new optimal model is $\hat{f}_{-p_{M}}$.
We will encounter the same difficulty as in the first stage, i.e., the number of parameters of the label predictor will change. To address the issue, our key observation is that in the existing literature on CBMs, we always use linear transformation for the label predictor, meaning that the dimensions of the input with values of 0 0 will have no contribution to the final prediction.
To leverage this property, we fill the missing values in the input of the updated predictor with 0 0, that is, replacing $\hat{g}_{-p_{M}}$ with $\hat{g}^{*}_{-p_{M}}$ and consider $\hat{f}_{p_{M}=0}$ defined by

$$ $\hat{f}_{p_{M}=0}=\operatorname*{arg\,min}_{f}\sum_{i=1}^{n}L_{Y_{i}}\left(f, \hat{g}^{*}_{-p_{M}}\right).$ (8) $$

In total, we have the following lemma:

###### Lemma 4.4 .

In the CBM, if the label predictor utilizes linear transformations of the form $\hat{f}\cdot c$ with input $c$.
For $r\in M$, we remove the $r$-th concept from c and denote the new input as $c^{\prime}$; set the $r$-th concept to 0 0 and denote the new input as $c^{0}$. Then we have $\hat{f}_{-p_{M}}\cdot c^{\prime}=\hat{f}_{p_{M}=0}\cdot c^{0}$ for any input $c$.

Lemma 4.4 demonstrates that the retrained $\hat{f}_{-p_{M}}$ and $\hat{f}_{p_{M}=0}$, when given inputs $\hat{g}_{-p_{M}}(x)$ and $\hat{g}^{*}_{-p_{M}}(x)$ respectively, yield identical outputs. Consequently, we can utilize $\hat{f}_{p_{M}=0}$ as the editing target in place of $\hat{f}_{-p_{M}}$.

###### Theorem 4.5 .

For the revised retrained label predictor $\hat{f}_{p_{M}=0}$ defined by equation 8, we can edit the initial label predictor $\hat{f}$ to $\bar{f}_{p_{M}=0}$ by the following equation as a substitute for $\hat{f}_{p_{M}=0}$:

$$ $\hat{f}_{p_{M}=0}\approx\bar{f}_{p_{M}=0}\triangleq\hat{f}-H_{\hat{f}}^{-1} \cdot\nabla_{\hat{f}}\sum_{l=1}^{n}L_{Y_{l}}\left(\hat{f},\bar{g}^{*}_{-p_{M}} \right),$ $$

where $H_{\hat{f}}=\nabla^{2}_{\hat{f}}\sum_{i=1}^{n}L_{Y_{i}}(\hat{f},\hat{g})$. Deleting the $r$-th dimension of $\bar{f}_{p_{M}=0}$ for $r\in M$, then we can map it to $\bar{f}_{-p_{M}}$, which is the approximation of the final edited label predictor $\hat{f}_{-p_{M}}$ under concept level.

###### Theorem 4.3 .

###### Lemma 4.4 .

###### Theorem 4.5 .

### 4.3 Data-level Editable CBM

In this scenario, we are more concerned about fully removing the influence of data samples on CBMs due to different reasons, such as the training data involving poisoned or erroneous issues. Specifically, we have a set of samples to be removed $\{(x_{i},y_{i},c_{i})\}_{i\in G}$ with $G\subset[n]$. Then, we define the retrained concept predictor as

$$ $\hat{g}_{-z_{G}}=\operatorname*{arg\,min}_{g}\sum_{j=1}^{k}\sum_{i\in[n]-G}L_{ C_{j}}(g(x_{i}),c_{i})$ (9) $$

which can be evaluated by the following theorem:

###### Theorem 4.6 .

For dataset $\mathcal{D}=\{(x_{i},y_{i},c_{i})\}_{i=1}^{n}$, given a set of data $z_{r}=(x_{r},y_{r},c_{r})$, $r\in G$ to be removed. Suppose the updated concept predictor $\hat{g}_{-z_{G}}$ is defined by equation 9, then we have the following approximation for $\hat{g}_{-z_{G}}$

$$ $\hat{g}_{-z_{G}}\approx\bar{g}_{-z_{G}}\triangleq\hat{g}+H^{-1}_{\hat{g}}\cdot \sum_{r\in G}\nabla_{g}L_{C_{r}}(\hat{g}(x_{r}),c_{r}),$ (10) $$

where $H_{\hat{g}}=\nabla^{2}_{\hat{g}}\sum_{i,j}L_{C_{j}}(\hat{g}^{j}(x_{i}),c_{i}^{
j})$ is the Hessian matrix of the loss function respect to $\hat{g}$.

Based on $\hat{g}_{-z_{G}}$, the label predictor becomes $\hat{f}_{-z_{G}}$ which is defined by

$$ $\hat{f}_{-z_{G}}=\operatorname*{arg\,min}_{f}\sum_{i\in[n]-G}L_{Y_{i}}(f,\hat{ g}_{-z_{G}}).$ (11) $$

Compared with the original loss before unlearning
in equation 2, we can observe two changes in equation 11. First, we remove $|G|$ data points in the loss function $L_{Y}$. Secondly, the input for the loss is also changed from $\hat{g}(x_{i})$ to $\hat{g}_{-z_{G}}$. Therefore, it is difficult to estimate directly with an influence function. Here we introduce an intermediate label predictor as

$$ $\tilde{f}_{-z_{G}}=\operatorname*{arg\,min}\sum_{i\in[n]-G}L_{Y_{i}}(f,\hat{g}),$ (12) $$

and split the estimate of $\hat{f}_{-z_{G}}-\hat{f}$ into $\hat{f}_{-z_{G}}-\tilde{f}_{-z_{G}}$ and $\tilde{f}_{-z_{G}}-\hat{f}$.

###### Theorem 4.7 .

For dataset $\mathcal{D}=\{(x_{i},y_{i},c_{i})\}_{i=1}^{n}$, given a set of data $z_{r}=(x_{r},y_{r},c_{r})$, $r\in G$ to be removed. The intermediate label predictor $\tilde{f}_{-z_{G}}$ is defined in equation 12. Then we have

$$ $\tilde{f}_{-z_{G}}-\hat{f}\approx H^{-1}_{\hat{f}}\cdot\sum_{i\in[n]-G}\nabla_ {\hat{f}}L_{Y_{i}}(\hat{f},\hat{g})\triangleq A_{G}.$ $$

We denote the edited version of $\tilde{f}_{-z_{G}}$ as $\bar{f}^{*}_{-z_{G}}\triangleq\hat{f}+A_{G}$.
Define $B_{G}$ as

$$ $\hat{f}_{-z_{G}}-\tilde{f}_{-z_{G}}\approx-H^{-1}_{\bar{f}^{*}_{-z_{G}}}\cdot \nabla_{\hat{f}}\sum_{i\in[n]-G}\left(L_{Y_{i}}\left(\bar{f}^{*}_{-z_{G}},\bar {g}_{-z_{G}}\right)-L_{Y_{i}}\left(\bar{f}^{*}_{-z_{G}},\hat{g}\right)\right) \triangleq B_{G},$ $$

where $H_{\bar{f}^{*}_{-z_{G}}}=\nabla_{\bar{f}}\sum_{i\in[n]-G}L_{Y_{i}}\left(\bar{f
}^{*}_{-z_{G}},\hat{g}\right)$ is the Hessian matrix concerning $\bar{f}^{*}_{-z_{G}}$.
Combining the above two-stage approximation, then, the final edited label predictor $\bar{f}_{-z_{G}}$ can be obtained by

$$ $\bar{f}_{-z_{G}}=\bar{f}^{*}_{-z_{G}}+B_{G}=\hat{f}+A_{G}+B_{G}.$ (13) $$

Acceleration via EK-FAC. As we mentioned in Section 3, as the loss function in CBMs is non-convex, the Hessian matrices in all our theorems may not be well-defined. We can use the EK-FAC approach, i.e., using $\hat{H}=G+\lambda I$ to approximate the Hessian,
where $G$ is the Fisher information matrix, and $\lambda$ is a small damping term used to ensure the positive definiteness. See Appendix B.1 for using EK-FAC to CBMs. Also, see Algorithm F-F in the Appendix for the detailed EK-FAC-based algorithms for our three levels, whose original (Hessian) versions are in Algorithm F-F, respectively.

###### Theorem 4.6 .

###### Theorem 4.7 .

## 5 Experiments

In this section, we demonstrate our main experimental results on utility evaluation, edition efficiency, and interpretability evaluation. Details and additional results are in Appendix G due to space limit.

### 5.1 Experimental Settings

Dataset. We utilize three datasets: X-ray grading (OAI) (Nevitt et al., ), Bird identification (CUB) (Wah et al., ) and Large-scale CelebFaces Attributes dataset (CelebA) (Liu et al., ). OAI is a multi-center observational study of knee osteoarthritis, which comprises 36,369 data points. Specifically, we configure $n=10$ concepts that characterize crucial osteoarthritis indicators such as joint space narrowing, osteophytes, and calcification. Bird identification (CUB) consists of 11,788 data points, which belong to 200 classes and include 112 binary attributes to describe detailed visual features of birds. CelebA comprises over 200,000 celebrity images, each annotated with 40 binary attributes that detail facial features, such as hair color, eyeglasses, and smiling. As the dataset lacks predefined classification tasks, following Espinosa Zarlenga et al. (), we designate 8 attributes as labels and the remaining 32 attributes as concepts. For all the above datasets, we follow the same network architecture and settings outlined in Koh et al. ().

Baselines. We employ three baseline methods: Retrain, CBM-IF, and ECBM. Retrain: We retrain the CBM from scratch by removing the samples, concept labels, or concepts from the training set. CBM-IF: This method is a direct implementation of our previous theorems of model updates in the three settings. See Algorithms F-F in Appendix for details. ECBM:
As we discussed above, all of our model updates can be further accelerated via EK-FAC, ECBM corresponds to the EK-FAC accelerated version of Algorithms F-F (refer to Algorithms F-F in Appendix).

Evaluation Metric. We utilize two primary evaluation metrics to assess our models: the F1 score and runtime (RT). The F1 score measures the model’s performance by balancing precision and recall. Runtime, measured in seconds, evaluates the running time of each method to update the model.

Implementation Details. Our experiments utilized an Intel Xeon CPU and an RTX 3090 GPU. For utility evaluation, at the concept level, one concept was randomly removed for the OAI dataset and repeated while ten concepts were randomly removed for the CUB dataset, with five different seeds. At the data level, 3% of the data points were randomly deleted and repeated 10 times with different seeds. At the concept-label level, we randomly selected 3% of the data points and modified one concept of each data randomly, repeating this 10 times for consistency across iterations.

### 5.2 Evaluation of Utility and Editing Efficiency

Our experimental results, as illustrated in Table 1, demonstrate the effectiveness of ECBMs compared to traditional retraining and CBM-IF, particularly emphasizing computational efficiency without compromising accuracy. Specifically, ECBMs achieved F1 scores close to those of retraining (0.8808 vs. 0.8825) while significantly reducing the runtime from 31.44 seconds to 8.29 seconds. This pattern is consistent in the CUB dataset, where the runtime was decreased from 27.88 seconds for retraining to 7.03 seconds for ECBMs, with a negligible difference in the F1 score (0.7971 to 0.7963). These results highlight the potential of ECBMs to provide substantial time savings—approximately 22-30% of the computational time required for retraining—while maintaining comparable accuracy. Compared to CBM-IF, ECBM also showed a slight reduction in runtime and a significant improvement in F1 score. The former verifies the effective acceleration of our algorithm by EK-FAC. This efficiency is particularly crucial in scenarios where frequent updates to model annotations are needed, confirming the utility of ECBMs in dynamic environments where running time and accuracy are critical.

We can also see that the original version of ECBM, i.e., CBM-IF, also has a lower runtime than retraining but a lower F1 score than ECBM. Such results may be due to different reasons. For example,
our original theorems depend on the inverse of the Hessian matrices, which may not be well-defined for non-convex loss. Moreover, these Hessian matrices may be ill-conditioned or singular, which makes calculating their inverse imprecise and unstable.

**Table 1: Performance comparison of different methods on the three datasets.**
| Edit Level | Method | OAI | CUB | CelebA |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| F1 score | RT (second) | F1 score | RT (second) | F1 score | RT (second) |  |  |
| Concept Label | Retrain | 0.8825$\pm$0.0054 | 31.44 | 0.7971$\pm$0.0066 | 27.88 | 0.3827$\pm$0.0272 | 57.60 |
| CBM-IF(Ours) | 0.8639$\pm$0.0033 | 16.31 | 0.7699$\pm$0.0035 | 14.39 | 0.3561$\pm$0.0134 | 34.93 |  |
| ECBM(Ours) | 0.8808$\pm$0.0039 | 8.29 | 0.7963$\pm$0.0050 | 7.03 | 0.3845$\pm$0.0327 | 15.67 |  |
| Concept | Retrain | 0.8448$\pm$0.0191 | 27.33 | 0.7811$\pm$0.0047 | 28.41 | 0.3776$\pm$0.0350 | 68.94 |
| CBM-IF(Ours) | 0.8214$\pm$0.0071 | 17.38 | 0.7579$\pm$0.0065 | 15.70 | 0.3609$\pm$0.0202 | 35.56 |  |
| ECBM(Ours) | 0.8403$\pm$0.0090 | 8.30 | 0.7787$\pm$0.0058 | 6.43 | 0.3761$\pm$0.0280 | 15.99 |  |
| Data | Retrain | 0.8811$\pm$0.0065 | 33.72 | 0.7838$\pm$0.0051 | 28.08 | 0.3797$\pm$0.0375 | 65.60 |
| CBM-IF(Ours) | 0.8472$\pm$0.0046 | 17.84 | 0.7623$\pm$0.0031 | 15.86 | 0.3536$\pm$0.0166 | 40.08 |  |
| ECBM(Ours) | 0.8797$\pm$0.0038 | 8.81 | 0.7827$\pm$0.0088 | 7.11 | 0.3748$\pm$0.0347 | 16.75 |  |

Editing Multiple Samples.
To comprehensively evaluate the editing capabilities of ECBM in various scenarios, we conducted experiments on the performance with multiple samples that need to be removed. Specifically, for the concept label/data levels, we consider the different ratios of samples (1-10%) for edit, while for the concept level, we consider removing different numbers of concepts $\in\{2,4,6,\cdots,20\}$. We compared the performance of retraining, CBM-IF, and ECBM methods. As shown in Figure 2, except for certain cases at the concept level, the F1 score of the ECBM method is generally around $0.0025$ lower than that of the retrain method, which is significantly better than the corresponding results of the CBM-IF method. Recalling Table 1, the speed of ECBM is more than three times faster than that of retraining. Consequently, ECBM is an editing method that achieves a trade-off between speed and effectiveness.

Figure: Figure 2: Impact of edition ratio on three settings on CUB dataset.
Refer to caption: x2.png

### 5.3 Results on Interpretability

Influence function in ECBM can measure the importance of concepts.
The original motivation of the influence function is to calculate the importance score of each sample. Here, we will show that the influence function for the concept level in Theorem 4.3 can be used to calculate the importance of each concept in CBMs, which provides an explainable tool for CBMs. In detail, we conduct our experiments on the CUB dataset. We first select 1-10 most influential and 1-10 least influential concepts by our influence function. Then, we will remove these concepts and update the model via retraining or our ECBM and analyze the change (F1 Score Difference) w.r.t. the original CBM before removal.

The results in Figure 3(a) demonstrate that when we remove the 11-1010 most influential concepts identified by the ECBM method, the F1 score decreases by more than 0.0250.025 compared to the CBM before removal. In contrast, Figure 3(b) shows that the change in the F1 score remains consistently below $0.005$ when removing the least influential concepts. These findings strongly indicate that the influence function in ECBM can successfully determine the importance of concepts. Furthermore, we observe that the gap between the F1 score of retraining and ECBM is consistently smaller than $0.005$, and even smaller in the case of least important concepts. This further suggests that when ECBM edits various concepts, its performance is very close to the ground truth.

Figure: (a) Results on the 1-10 most influential concepts
Refer to caption: x5.png

ECBMs can erase data influence.
For the data level, ECBMs aim to facilitate an efficient removal of samples. We perform membership inference attacks (MIAs) to provide direct evidence that ECBMs can indeed erase data influence. MIA is a privacy attack that aims to infer whether a specific data sample was part of the training dataset used to train a model. The attacker exploits the model’s behavior, such as overconfidence or overfitting, to distinguish between training (member) and non-training (non-member) data points. In MIAs, the attacker typically queries the model with a data sample and observes its prediction confidence or loss values, which tend to be higher for members of the training set than non-members (Shokri et al., ).

To quantify the success of these edits, we calculate the RMIA (Removed Membership Inference Attack) score for each category. The RMIA score is defined as the model’s confidence in classifying whether a given sample belongs to the training set. Lower RMIA values indicate that the sample behaves more like a test set (non-member) sample Zarifzadeh et al. (). This metric is especially crucial for edited samples, as a successful ECBM should make the removed members behave similarly to non-members, reducing their membership vulnerability. See Appendix G for its definition.

We conducted experiments by randomly selecting 200 samples from the training set (members) and 200 samples from the test set (non-members) of the CUB dataset. We calculated the RMIA scores for these samples and plotted their frequency distributions, as shown in Figure 4(a). The mean RMIA score for non-members was 0.049465, while members had a mean score of 0.063505. Subsequently, we applied ECBMs to remove the 200 training samples from the model, updated the model parameters, and then recalculated the RMIA scores. After editing, the mean RMIA score for the removed-members decreased to 0.052105, significantly closer to the non-members’ mean score. This shift in RMIA values demonstrates the effectiveness of ECBMs in editing the model, as the removed members now exhibit behavior closer to that of non-members. The post-editing RMIA score distributions are shown in Figure 4(b). These results provide evidence of the effectiveness of ECBMs in editing the model’s knowledge about specific samples.

Figure: (a) RMIA Score Before Editing
Refer to caption: x7.png

Visualization. Since CBM is an explainable model, we aim to evaluate the interpretability of our ECBM (compared to the retraining). We will present some visualization results for the concept-level edit. Figure 5 presents the top 1010 most influential concepts and their corresponding predicted concept labels obtained by our ECBM and the retrain method after randomly deleting concepts for the CUB dataset.
Our ECBM can provide explanations for which concepts are crucial and how they determine the predicted label.
Specifically, among the top $10$ most important concepts in the ground truth (retraining), ECBM can accurately recognize $9$ with their correct concept labels.
For instance, we correctly identify "has_upperparts_color::orange", "has_upper_tail_color::red", and "has_breast_color::black" as some of the most important concepts when predicting categories. Additional visualization results under data level and concept-label level on OAI and CUB datasets are included in Appendix G.2.

Figure: Figure 5: Visualization of the top-10 most influential concepts and their predicted concept labels.
Refer to caption: extracted/5901906/images/visual-1.png

## 6 Conclusion

In this paper, we propose Editable Concept Bottleneck Models (ECBMs). ECBMs can address issues of removing/inserting some training data or new concepts from trained CBMs for different reasons, such as privacy concerns, data mislabelling, spurious concepts, and concept annotation errors retraining from scratch. Furthermore, to improve computational efficiency, we present streamlined versions integrating EK-FAC. Experimental results show our ECBMs are efficient and effective.

## Appendix A Influence Function

Consider a neural network $\hat{\theta}=\arg\min_{\theta}\sum_{i=1}^{n}\ell(z_{i},\theta)$ with loss function $L$ and dataset $D=\{z_{i}\}_{i=1}^{n}$. That is $\hat{\theta}$ minimize the empirical risk

$$ $R(\theta)=\sum_{i=1}^{n}L(z_{i},\theta)$ $$

Assume $R$ is strongly convex in $\theta$. Then $\theta$ is uniquely defined. If we remove a point $z_{m}$ from the training dataset, the parameters become $\hat{\theta}_{-z_{m}}=\arg\min_{\theta}\sum_{i\neq m}L(z_{i},\theta)$. Up-weighting $z_{m}$ by $\epsilon$ small enough, then the revised risk ${R(\theta)}^{{}^{\prime}}=\frac{1}{n}\sum_{i=1}^{n}L(z_{i};\theta)+\epsilon L(
z_{m};\theta)$ is still strongly convex.
Then the response function $\hat{\theta}_{\epsilon,-z_{m}}={R(\theta)}^{{}^{\prime}}$ is also uniquely defined. The parameter change is denoted as $\Delta_{\epsilon}=\hat{\theta}_{\epsilon,-z_{m}}-\hat{\theta}$.
Since $\hat{\theta}_{\epsilon,-z_{m}}$ is the minimizer of ${R(\theta)}^{{}^{\prime}}$, we have the first-order optimization condition as

$$ $\nabla_{\hat{\theta}_{\epsilon,-z_{m}}}R(\theta)+\epsilon\cdot\nabla_{\hat{ \theta}_{\epsilon,-z_{m}}}L(z_{m},\hat{\theta}_{\epsilon,-z_{m}})=0$ $$

Since $\hat{\theta}_{\epsilon,-z_{m}}\rightarrow\hat{\theta}as\epsilon\rightarrow 0$, we perform a Taylor expansion of the right-hand side:

$$ $\left[\nabla R(\hat{\theta})+\epsilon\nabla L(z_{m},\hat{\theta})\right]+\left [\nabla^{2}R(\hat{\theta})+\epsilon\nabla^{2}L(z_{m},\hat{\theta})\right] \Delta_{\epsilon}\approx 0$ $$

Noting $\epsilon\nabla^{2}L(z_{m},\hat{\theta})\Delta_{\epsilon}$ is $o(\|\Delta_{\epsilon}\|)$ term, which is smaller than other parts, we drop it in the following analysis. Then the Taylor expansion equation becomes

$$ $\left[\nabla R(\hat{\theta})+\epsilon\nabla L(z_{m},\hat{\theta})\right]+ \nabla^{2}R(\hat{\theta})\cdot\Delta_{\epsilon}\approx 0$ $$

Solving for $\Delta_{\epsilon}$, we obtain:

$$ $\Delta_{\epsilon}=-\left[\nabla^{2}R(\hat{\theta})+\epsilon\nabla^{2}L(z,\hat{ \theta})\right]^{-1}\left[\nabla R(\hat{\theta})+\epsilon\nabla L(z,\hat{ \theta})\right].$ $$

Remember $\theta$ minimizes $R$, then $\nabla R(\hat{\theta})=0$. Dropping $o(\epsilon)$ term, we have

$$ $\Delta_{\epsilon}=-\epsilon\nabla^{2}R(\hat{\theta})^{-1}\nabla L(z,\hat{ \theta}).$ $$

$$ $\left.\frac{d\hat{\theta}_{\epsilon,-z_{m}}}{d\epsilon}\right|_{\epsilon=0}= \left.\frac{d\Delta_{\epsilon}}{d\epsilon}\right|_{\epsilon=0}=-H_{\hat{\theta }}^{-1}\nabla L(z,\hat{\theta})\equiv\mathcal{I}_{up,params}(z).$ $$

Besides, we can obtain the approximation of $\hat{\theta}_{-z_{m}}$ directly by $\hat{\theta}_{-z_{m}}\approx\hat{\theta}+\mathcal{I}_{up,params}(z)$.

## Appendix B Acceleration for Influence Function

#### EK-FAC.

EK-FAC method relies on two approximations to the Fisher information matrix, equivalent to $G_{\hat{\theta}}$ in our setting, which makes it feasible to compute the inverse of the matrix.

Firstly, assume that the derivatives of the weights in different layers are uncorrelated, which implies that $G_{\hat{\theta}}$ has a block-diagonal structure.
Suppose $\hat{g}_{\theta}$ can be denoted by $\hat{g}_{\theta}(x)=g_{\theta_{L}}\circ\cdots\circ g_{\theta_{l}}\circ\cdots
\circ g_{\theta_{1}}(x)$ where $l\in[L]$. We fold the bias into the weights and vectorize the parameters in the $l$-th layer into a vector $\theta_{l}\in\mathbb{R}^{d_{l}}$, $d_{l}\in\mathbb{N}$ is the number of $l$-th layer parameters.
Then $G_{\hat{\theta}}$ can be reaplcaed by $\left(G_{1}(\hat{\theta}),\cdots,G_{L}(\hat{\theta})\right)$, where $G_{l}(\hat{\theta})\triangleq n^{-1}\sum_{i=1}^{n}\nabla_{\hat{\theta}_{l}}
\ell_{i}\nabla_{\theta_{l}}\ell_{i}^{\mathrm{T}}$. Denote $h_{l}$, $o_{l}$ as the output and pre-activated output of $l$-th layer. Then $G_{l}(\theta)$ can be approximated by

$$ ${G}_{l}(\theta)\approx\hat{G}_{l}(\theta)\triangleq\frac{1}{n}\sum_{i=1}^{n}h_ {l-1}\left(x_{i}\right)h_{l-1}\left(x_{i}\right)^{T}\otimes\frac{1}{n}\sum_{i= 1}^{n}\nabla_{o_{l}}\ell_{i}\nabla_{o_{l}}\ell_{i}^{T}\triangleq{\Omega}_{l-1} \otimes{\Gamma_{l}}.$ $$

Furthermore, in order to accelerate transpose operation and introduce the damping term, perform eigenvalue decomposition of matrix ${\Omega}_{l-1}$ and ${\Gamma_{l}}$ and obtain the corresponding decomposition results as $Q_{\Omega}{\Lambda}_{{\Omega}}{Q}_{{\Omega}}^{\top}$ and ${Q}_{\Gamma}{\Lambda}_{\Gamma}{Q}_{\Gamma}^{\top}$. Then the inverse of $\hat{H}_{l}(\theta)$ can be obtained by

$$ $\hat{H}_{l}(\theta)^{-1}\approx\left(\hat{G}_{l}\left(\hat{g}\right)+\lambda_{ l}I_{d_{l}}\right)^{-1}=\left(Q_{\Omega_{l-1}}\otimes Q_{\Gamma_{l}}\right) \left(\Lambda_{\Omega_{l-1}}\otimes\Lambda_{\Gamma_{l}}+\lambda_{l}I_{d_{l}} \right)^{-1}\left(Q_{\Omega_{l-1}}\otimes Q_{\Gamma_{l}}\right)^{\mathrm{T}}.$ $$

Besides, George et al. () proposed a new method that corrects the error in equation 14 which sets the $i$-th diagonal element of $\Lambda_{\Omega_{l-1}}\otimes\Lambda_{\Gamma_{l}}$ as $\Lambda_{ii}^{*}=n^{-1}\sum_{j=1}^{n}\left(\left(Q_{\Omega_{l-1}}\otimes Q_{
\Gamma_{l}}\right)\nabla_{\theta_{l}}\ell_{j}\right)^{2}_{i}.$

### B.1 EK-FAC for CBMs

In our CBM model, the label predictor is a single linear layer, and Hessian computing costs are affordable. However, the concept predictor is based on Resnet-18, which has many parameters. Therefore, we perform EK-FAC for $\hat{g}$.

$$ $\hat{g}=\operatorname*{arg\,min}_{g}\sum_{j=1}^{k}L_{C_{j}}=\operatorname*{arg \,min}_{g}\sum_{j=1}^{k}\sum_{i=1}^{n}L_{C}(g^{j}(x_{i}),c_{i}^{j}),$ $$

we define $H_{\hat{g}}=\nabla^{2}_{\hat{g}}\sum_{i,j}L_{C_{j}}(g(x_{i}),c_{i})$ as the Hessian matrix of the loss function with respect to the parameters.

To this end, consider the $l$-th layer of $\hat{g}$ which takes as input a layer of activations $\{a_{j,t}\}$ where $j\in\{1,2,\ldots,J\}$ indexes the input map and $t\in\mathcal{T}$ indexes the spatial location which is typically a 2-D grid. This layer is parameterized by a set of weights $W=\left(w_{i,j,\delta}\right)$ and biases $b=\left(b_{i}\right)$, where $i\in\{1,\ldots,I\}$ indexes the output map, and $\delta\in\Delta$ indexes the spatial offset (from the center of the filter).

The convolution layer computes a set of pre-activations as

$$ $[S_{l}]_{i,t}=s_{i,t}=\sum_{\delta\in\Delta}w_{i,j,\delta}a_{j,t+\delta}+b_{i}.$ $$

Denote the loss derivative with respect to $s_{i,t}$ as

$$ $\mathcal{D}s_{i,t}=\frac{\partial\sum L_{C_{j}}}{\partial s_{i,t}},$ $$

which can be computed during backpropagation.

The activations are actually stored as $A_{l-1}$ of dimension $|\mathcal{T}|\times J$. Similarly, the weights are stored as an $I\times|\Delta|J$ array $W_{l}$. The straightforward implementation of convolution, though highly parallel in theory, suffers from poor memory access patterns. Instead, efficient implementations typically leverage what is known as the expansion operator $\llbracket\cdot\rrbracket$. For instance,
$\llbracket A_{l-1}\rrbracket$ is a $|\mathcal{T}|\times J|\Delta|$ matrix, defined as

$$ $\llbracket{{A}}_{l-1}\rrbracket_{t,j|\Delta|+\delta}=\left[{{A}}_{l-1}\right]_ {(t+\delta),j}=a_{j,t+\delta},$ $$

In order to fold the bias into the weights, we need to add a homogeneous coordinate (i.e. a column of all $1$’s) to the expanded activations $\llbracket A_{l-1}\rrbracket$ and denote this as $\llbracket A_{l-1}\rrbracket_{\mathrm{H}}$. Concatenating the bias vector to the weights matrix, then we have $\theta_{l}=(b_{l},W_{l})$.

Then, the approximation for $H_{\hat{g}}$ is given as:

$$ $\displaystyle G^{(l)}(\hat{g})=$ $\displaystyle\mathbb{E}\left[\mathcal{D}w_{i,j,\delta}\mathcal{D}w_{i^{\prime} ,j^{\prime},\delta^{\prime}}\right]=\mathbb{E}\left[\left(\sum_{t\in\mathcal{T }}a_{j,t+\delta}\mathcal{D}s_{i,t}\right)\left(\sum_{t^{\prime}\in\mathcal{T}} a_{j^{\prime},t^{\prime}+\delta^{\prime}}\mathcal{D}s_{i^{\prime},t^{\prime}} \right)\right]$ $\displaystyle\approx$ $\displaystyle\mathbb{E}\left[\llbracket{A}_{l-1}\rrbracket_{\mathrm{H}}^{\top} \llbracket{A}_{l-1}\rrbracket_{\mathrm{H}}\right]\otimes\frac{1}{|\mathcal{T}| }\mathbb{E}\left[\mathcal{D}{S}_{l}^{\top}\mathcal{D}{S}_{l}\right]\triangleq \Omega_{l-1}\otimes\Gamma_{l}.$ $$

Estimate the expectation using the mean of the training set,

$$ $\displaystyle G^{(l)}(\hat{g})\approx$ $\displaystyle\frac{1}{n}\sum_{i=1}^{n}\left(\llbracket{A}^{i}_{l-1}\rrbracket_ {\mathrm{H}}^{\top}\llbracket{A}^{i}_{l-1}\rrbracket_{\mathrm{H}}\right) \otimes\frac{1}{n}\sum_{i=1}^{n}\left(\frac{1}{|\mathcal{T}|}\mathcal{D}{{S}^{ i}_{l}}^{\top}\mathcal{D}{S}^{i}_{l}\right)\triangleq\hat{\Omega}_{l-1}\otimes \hat{\Gamma_{l}}.$ $$

Furthermore, if the factors $\hat{\Omega}_{l-1}$ and $\hat{\Gamma}_{l}$ have eigen decomposition $Q_{\Omega}{\Lambda}_{{\Omega}}{Q}_{{\Omega}}^{\top}$ and ${Q}_{\Gamma}{\Lambda}_{\Gamma}{Q}_{\Gamma}^{\top}$, respectively, then the eigen decomposition of $\hat{\Omega}_{l-1}\otimes\hat{\Gamma_{l}}$ can be written as:

$$ $\displaystyle\hat{\Omega}_{l-1}\otimes\hat{\Gamma}_{l}$ $\displaystyle={Q}_{{\Omega}}{\Lambda}_{{\Omega}}{Q}_{{\Omega}}^{\top}\otimes{Q }_{{\Gamma}}{\Lambda}_{{\Gamma}}{Q}_{{\Gamma}}^{\top}$ $\displaystyle=\left({Q}_{{\Omega}}\otimes{Q}_{{\Gamma}}\right)\left({\Lambda}_ {{\Omega}}\otimes{\Lambda}_{{\Gamma}}\right)\left({Q}_{{\Omega}}\otimes{Q}_{{ \Gamma}}\right)^{\top}.$ $$

Since subsequent inverse operations are required and the current approximation for $G^{(l)}(\hat{g})$ is PSD, we actually use a damped version as

$$ ${\hat{G^{l}}(\hat{g})}^{-1}=\left(G_{l}\left(\hat{g}\right)+\lambda_{l}I_{d_{l }}\right)^{-1}=\left(Q_{\Omega_{l-1}}\otimes Q_{\Gamma_{l}}\right)\left( \Lambda_{\Omega_{l-1}}\otimes\Lambda_{\Gamma_{l}}+\lambda_{l}I_{d_{l}}\right)^ {-1}\left(Q_{\Omega_{l-1}}\otimes Q_{\Gamma_{l}}\right)^{\mathrm{T}}.$ (14) $$

Besides, George et al. () proposed a new method that corrects the error in equation 14 which sets the $i$-th diagonal element of $\Lambda_{\Omega_{l-1}}\otimes\Lambda_{\Gamma_{l}}$ as

$$ $\Lambda_{ii}^{*}=n^{-1}\sum_{j=1}^{n}\left(\left(Q_{\Omega_{l-1}}\otimes Q_{ \Gamma_{l}}\right)\nabla_{\theta_{l}}\ell_{j}\right)^{2}_{i}.$ $$

## Appendix C Proof of Concept-label-level Influence

We have a set of erroneous data $D_{e}$ and its associated index set $S_{e}\subseteq[n]\times[k]$ such that for each $(w,r)\in S_{e}$, we have $(x_{w},y_{w},c_{w})\in D_{e}$ with $c_{w}^{r}$ is mislabeled and $\tilde{c}_{w}^{r}$ is its corrected concept label. Thus, our goal is to approximate the new CBM without retraining.

#### Proof Sketch.

Our goal is to edit $\hat{g}$ and $\hat{f}$ to $\hat{g}_{e}$ and $\hat{f}_{e}$. (i) First, we introduce new parameters $\hat{g}_{\epsilon,e}$ that minimize a modified loss function with a small perturbation $\epsilon$. (ii) Then, we perform a Newton step around $\hat{g}$ and obtain an estimate for $\hat{g}_{e}$. (iii) Then, we consider changing the concept predictor at one data point $(x_{i_{c}},y_{i_{c}},c_{i_{c}})$ and retraining the model to obtain a new label predictor $\hat{f}_{i_{c}}$, obtain an approximation for $\hat{f}_{i_{c}}$. (iv) Next, we iterate $i_{c}$ over $1,2,\cdots,n$, sum all the equations together, and perform a Newton step around $\hat{f}$ to obtain an approximation for $\hat{f}_{e}$. (v) Finally, we bring the estimate of $\hat{g}$ into the equation for $\hat{f}_{e}$ to obtain the final approximation.

###### Theorem C.1 .

The retrained concept predictor $\hat{g}_{e}$ defined by

$$ $\hat{g}_{e}=\operatorname*{arg\,min}\left[\sum_{(i,j)\notin S_{e}}L_{C}\left(g ^{j}(x_{i}),c_{i}^{j}\right)+\sum_{(i,j)\in S_{e}}L_{C}\left(g^{j}(x_{i}), \tilde{c}_{i}^{j}\right)\right],$ (15) $$

can be approximated by:

$$ $\hat{g}_{e}\approx\bar{g}_{e}\triangleq\hat{g}-H^{-1}_{\hat{g}}\cdot\sum_{(w,r )\in S_{e}}\left(\nabla_{\hat{g}}L_{C}\left(\hat{g}^{r}(x_{w}),\tilde{c}_{w}^{ r}\right)-\nabla_{\hat{g}}L_{C}\left(\hat{g}^{r}(x_{w}),c_{w}^{r}\right)\right),$ (16) $$

where $H_{\hat{g}}=\nabla^{2}_{\hat{g}}\sum_{i,j}L_{C}(\hat{g}^{j}(x_{i}),c_{i}^{j})$ is the Hessian matrix of the loss function respect to $\hat{g}$.

###### Proof.

For the index $(w,r)\in S_{e}$, indicating the $r$-th concept of the $w$-th data is wrong, we correct this concept $c_{w}^{r}$ to $\tilde{c}_{w}^{r}$. Rewrite $\hat{g}_{e}$ as

$$ $\hat{g}_{e}=\operatorname*{arg\,min}\left[\sum_{i,j}L_{C}\left(g^{j}(x_{i}),c_ {i}^{j}\right)+\sum_{(w,r)\in S_{e}}L_{C}\left(g^{r}(x_{w}),\tilde{c}_{w}^{r} \right)-\sum_{(w,r)\in S_{e}}L_{C}\left(g^{r}(x_{w}),c_{w}^{r}\right)\right].$ (17) $$

To approximate this effect, define new parameters $\hat{g}_{\epsilon,e}$ as

$$ $\hat{g}_{\epsilon,e}\triangleq\operatorname*{arg\,min}\left[\sum_{i,j}L_{C} \left(g^{j}(x_{i}),c_{i}^{j}\right)+\sum_{(w,r)\in S_{e}}\epsilon\cdot L_{C} \left(g^{r}(x_{w}),\tilde{c}_{w}^{r}\right)-\sum_{(w,r)\in S_{e}}\epsilon\cdot L _{C}\left(g^{r}(x_{w}),c_{w}^{r}\right)\right].$ (18) $$

Then, because $\hat{g}_{\epsilon,e}$ minimizes equation 18, we have

$$ $\nabla_{\hat{g}}\sum_{i,j}L_{C}\left(\hat{g}^{j}_{\epsilon,e}(x_{i}),c_{i}^{j} \right)+\sum_{(w,r)\in S_{e}}\epsilon\cdot\nabla_{\hat{g}}L_{C}\left(\hat{g}_{ \epsilon,e}^{r}(x_{w}),\tilde{c}_{w}^{r}\right)-\sum_{(w,r)\in S_{e}}\epsilon \cdot\nabla_{\hat{g}}L_{C}\left(\hat{g}_{\epsilon,e}^{r}(x_{w}),c_{w}^{r} \right)=0.$ $$

Perform a Taylor expansion of the above equation at $\hat{g}$,

$$ $\displaystyle\nabla_{\hat{g}}\sum_{i,j}L_{C}\left(\hat{g}^{j}(x_{i}),c_{i}^{j}\right)$ $\displaystyle+\sum_{(w,r)\in S_{e}}\epsilon\cdot\nabla_{\hat{g}}L_{C}\left( \hat{g}^{r}(x_{w}),\tilde{c}_{w}^{r}\right)-\sum_{(w,r)\in S_{e}}\epsilon\cdot \nabla_{\hat{g}}L_{C}\left(\hat{g}^{r}(x_{w}),c_{w}^{r}\right)$ $\displaystyle+\nabla^{2}_{\hat{g}}\sum_{i,j}L_{C}\left(\hat{g}^{j}(x_{i}),c_{i }^{j}\right)\cdot(\hat{g}_{\epsilon,e}-\hat{g})\approx 0.$ (19) $$

Because of equation 15, the first term of equation C equals 0 0. Then we have

$$ $\hat{g}_{\epsilon,e}-\hat{g}=-\sum_{(w,r)\in S_{e}}\epsilon\cdot H^{-1}_{\hat{ g}}\cdot\left(\nabla_{\hat{g}}L_{C}\left(\hat{g}^{r}(x_{w}),\tilde{c}_{w}^{r} \right)-\nabla_{\hat{g}}L_{C}\left(\hat{g}^{r}(x_{w}),c_{w}^{r}\right)\right),$ $$

where

$$ $H_{\hat{g}}=\nabla^{2}_{\hat{g}}\sum_{i,j}L_{C}\left(\hat{g}^{j}(x_{i}),c_{i}^ {j}\right).$ $$

Then, we do a Newton step around $\hat{g}$ and obtain

$$ $\hat{g}_{e}\approx\bar{g}_{e}\triangleq\hat{g}-H^{-1}_{\hat{g}}\cdot\sum_{(w,r )\in S_{e}}\left(\nabla_{\hat{g}}L_{C}\left(\hat{g}^{r}(x_{w}),\tilde{c}_{w}^{ r}\right)-\nabla_{\hat{g}}L_{C}\left(\hat{g}^{r}(x_{w}),c_{w}^{r}\right)\right).$ (20) $$

∎

###### Theorem C.2 .

The retrained label predictor $\hat{f}_{e}$ defined by

$$ $\hat{f}_{e}=\operatorname*{arg\,min}\left[\sum_{i=1}^{n}L_{Y}\left(f\left(\hat {g}_{e}\left(x_{i}\right)\right),y_{i}\right)\right]$ $$

can be approximated by:

$$ $\displaystyle\hat{f}_{e}$ $\displaystyle\approx\bar{f}_{e}=\hat{f}+H^{-1}_{\hat{f}}\cdot\nabla_{f}\sum_{i =1}^{n}L_{Y_{i}}\left(\hat{f},\hat{g}\right)-H^{-1}_{\hat{f}}\cdot\nabla_{f} \sum_{i=1}^{n}L_{Y_{i}}\left(\hat{f},\bar{g}_{e}\right),$ $$

where $H_{\hat{f}}=\nabla^{2}_{\hat{f}}\sum_{i=1}^{n}L_{Y_{i}}(\hat{f},\hat{g})$ is the Hessian matrix of the loss function respect to $\hat{f}$, $L_{Y_{i}}(\hat{f},\hat{g})\triangleq L_{Y}(\hat{f}(\hat{g}(x_{i})),y_{i})$, and $\bar{g}_{e}$ is given in Theorem C.1.

###### Proof.

Now we come to deduce the edited label predictor towards $\hat{f}_{e}$.

First, we consider only changing the concept predictor at one data point $(x_{i_{c}},y_{i_{c}},c_{i_{c}})$ and retrain the model to obtain a new label predictor $\hat{f}_{i_{c}}$.

$$ $\hat{f}_{i_{c}}=\operatorname*{arg\,min}\left[\sum_{\begin{subarray}{c}i=1,i \neq i_{c}\end{subarray}}^{n}L_{Y}\left(f\left(\hat{g}\left(x_{i}\right)\right ),y_{i}\right)+L_{Y}\left(f\left(\hat{g}_{e}\left(x_{i_{c}}\right)\right),y_{i _{c}}\right)\right].$ $$

We rewrite the above equation as follows:

$$ $\hat{f}_{i_{c}}=\operatorname*{arg\,min}\left[\sum_{i=1}^{n}L_{Y}\left(f\left( \hat{g}\left(x_{i}\right)\right),y_{i}\right)+L_{Y}\left(f\left(\hat{g}_{e} \left(x_{i_{c}}\right)\right),y_{i_{c}}\right)-L_{Y}\left(f\left(\hat{g}\left( x_{i_{c}}\right)\right),y_{i_{c}}\right)\right].$ $$

We define $\hat{f}_{\epsilon,i_{c}}$ as:

$$ $\hat{f}_{\epsilon,i_{c}}=\operatorname*{arg\,min}\left[\sum_{i=1}^{n}L_{Y} \left(f\left(\hat{g}\left(x_{i}\right)\right),y_{i}\right)+\epsilon\cdot L_{Y} \left(f\left(\hat{g}_{e}\left(x_{i_{c}}\right)\right),y_{i_{c}}\right)- \epsilon\cdot L_{Y}\left(f\left(\hat{g}\left(x_{i_{c}}\right)\right),y_{i_{c}} \right)\right].$ $$

Derive with respect to $f$ at both sides of the above equation. we have

$$ $\nabla_{\hat{f}}\sum_{i=1}^{n}L_{Y}\left(\hat{f}_{\epsilon,i_{c}}\left(\hat{g} \left(x_{i}\right)\right),y_{i}\right)+\epsilon\cdot\nabla_{\hat{f}}L_{Y}\left (\hat{f}_{\epsilon,i_{c}}\left(\hat{g}_{e}\left(x_{i_{c}}\right)\right),y_{i_{ c}}\right)-\epsilon\cdot\nabla_{\hat{f}}L_{Y}\left(\hat{f}_{\epsilon,i_{c}} \left(\hat{g}\left(x_{i_{c}}\right)\right),y_{i_{c}}\right)=0$ $$

Perform a Taylor expansion of the above equation at $\hat{f}$,

$$ $\displaystyle\nabla_{\hat{f}}\sum_{i=1}^{n}L_{Y}\left(\hat{f}\left(\hat{g} \left(x_{i}\right)\right),y_{i}\right)+\epsilon\cdot\nabla_{\hat{f}}L_{Y}\left (\hat{f}\left(\hat{g}_{e}\left(x_{i_{c}}\right)\right),y_{i_{c}}\right)$ $\displaystyle-\epsilon\cdot\nabla_{\hat{f}}L_{Y}\left(\hat{f}\left(\hat{g} \left(x_{i_{c}}\right)\right),y_{i_{c}}\right)+\nabla^{2}_{\hat{f}}\sum_{i=1}^ {n}L_{Y}\left(\hat{f}\left(\hat{g}\left(x_{i}\right)\right),y_{i}\right)\cdot \left(\hat{f}_{\epsilon,i_{c}}-\hat{f}\right)=0$ $$

Then we have

$$ $\hat{f}_{\epsilon,i_{c}}-\hat{f}\approx-\epsilon\cdot H^{-1}_{\hat{f}}\cdot \nabla_{f}\left(L_{Y}\left(\hat{f}\left(\hat{g}_{e}\left(x_{i_{c}}\right) \right),y_{i_{c}}\right)-L_{Y}\left(\hat{f}\left(\hat{g}\left(x_{i_{c}}\right) \right),y_{i_{c}}\right)\right),$ $$

where $H^{-1}_{\hat{f}}=\nabla^{2}_{\hat{f}}\sum_{i=1}^{n}L_{Y}\left(\hat{f}\left(
\hat{g}\left(x_{i}\right)\right),y_{i}\right)$.

Iterate $i_{c}$ over $1,2,\cdots,n$, and sum all the equations together, we can obtain:

$$ $\hat{f}_{\epsilon,e}-\hat{f}\approx-\epsilon\cdot H^{-1}_{\hat{f}}\cdot\sum_{i =1}^{n}\nabla_{f}\left(L_{Y}\left(\hat{f}\left(\hat{g}_{e}\left(x_{i}\right) \right),y_{i}\right)-L_{Y}\left(\hat{f}\left(\hat{g}\left(x_{i}\right)\right), y_{i}\right)\right).$ $$

Perform a Newton step around $\hat{f}$ and we have

$$ $\hat{f}_{e}\approx\hat{f}-H^{-1}_{\hat{f}}\cdot\sum_{i=1}^{n}\nabla_{f}\left(L _{Y}\left(\hat{f}\left(\hat{g}_{e}\left(x_{i}\right)\right),y_{i}\right)-L_{Y} \left(\hat{f}\left(\hat{g}\left(x_{i}\right)\right),y_{i}\right)\right).$ (21) $$

Bringing the edited 20 of $g$ into equation 21, we have

$$ $\displaystyle\hat{f}_{e}\approx$ $\displaystyle\hat{f}-H^{-1}_{\hat{f}}\cdot\sum_{i=1}^{n}\nabla_{f}\left(L_{Y} \left(\hat{f}\left(\bar{g}_{e}\left(x_{i}\right)\right),y_{i}\right)-L_{Y} \left(\hat{f}\left(\hat{g}\left(x_{i}\right)\right),y_{i}\right)\right)$ $\displaystyle=$ $\displaystyle\hat{f}-H^{-1}_{\hat{f}}\cdot\sum_{i=1}^{n}\nabla_{f}\left(L_{Y_{ i}}\left(\hat{f},\bar{g}_{e}\right)-L_{Y_{i}}\left(\hat{f},\hat{g}\right)\right)$ $\displaystyle=$ $\displaystyle\hat{f}+H^{-1}_{\hat{f}}\cdot\nabla_{f}\sum_{i=1}^{n}L_{Y_{i}} \left(\hat{f},\hat{g}\right)-H^{-1}_{\hat{f}}\cdot\nabla_{f}\sum_{i=1}^{n}L_{Y _{i}}\left(\hat{f},\bar{g}_{e}\right)\triangleq\bar{f}_{e}.$ $$

∎

###### Theorem C.1 .

###### Proof.

###### Theorem C.2 .

###### Proof.

## Appendix D Proof of Concept-level Influence

We address situations that delete $p_{r}$ for $r\in M$ concept removed dataset. Our goal is to estimate $\hat{g}_{-p_{M}}$, $\hat{f}_{-p_{M}}$, which is the concept and label predictor trained on the $p_{r}$ for $r\in M$ concept removed dataset.

#### Proof Sketch.

The main ideas are as follows: (i) First, we define a new predictor $\hat{g}^{*}_{p_{M}}$, which has the same dimension as $\hat{g}$ and the same output as $\hat{g}_{-p_{M}}$. Then deduce an approximation for $\hat{g}^{*}_{p_{M}}$. (ii) Then, we consider setting $p_{r}=0$ instead of removing it, we get $\hat{f}_{p_{M}=0}$, which is equivalent to $\hat{f}_{-p_{M}}$ according to lemma D.1. We estimate this new predictor as a substitute. (iii) Next, we assume we only use the updated concept predictor $\hat{g}_{p_{M}}^{*}$ for one data $(x_{i_{r}},y_{i_{r}},c_{i_{r}})$ and obtain a new label predictor $\hat{f}_{ir}$, and obtain a one-step Newtonian iterative approximation of $\hat{f}_{ir}$ with respect to $\hat{f}$. (iv) Finally, we repeat the above process for all data points and combine the estimate of $\hat{g}$ in Theorem D.2, we obtain a closed-form solution of the influence function for $\hat{f}$.

First, we introduce our following lemma:

###### Lemma D.1 .

For the concept bottleneck model, if the label predictor utilizes linear transformations of the form $\hat{f}\cdot c$ with input $c$. For r in M, we remove the $r$-th concept from c and denote the new input as $c^{\prime}$. Set the $r$-th concept to 0 and denote the new input as $c^{0}$. Then we have $\hat{f}_{-p_{M}}\cdot c^{\prime}=\hat{f}_{p_{M}=0}\cdot c^{0}$ for any c.

###### Proof.

Assume the parameter space of $\hat{f}_{-p_{M}}$ and $\hat{f}_{p_{M}=0}$ are $\Gamma$ and $\Gamma_{0}$, respectively, then there exists a surjection $P:\Gamma\rightarrow\Gamma_{0}$. For any $\theta\in\Gamma$, $P(\theta)$ is the operation that removes the $r$-th row of $\theta$ for $r\in M$. Then we have:

$$ $P(\theta)\cdot c^{\prime}=\sum_{t\notin M}\theta[j]\cdot c^{\prime}[j]=\sum_{t }\theta[t]\mathbb{I}\{t\notin M\}c[t]=\theta\cdot c^{0}.$ $$

Thus, the loss function $L_{Y}(\theta,c^{0})=L_{Y}(P(\theta),c^{\prime})$ of both models is the same for every sample in the second stage. Besides, by formula derivation, we have, for $\theta^{\prime}\in\Gamma_{0}$, for any $\theta$ in $P^{-1}(\theta^{\prime})$,

$$ $\frac{\partial L_{Y}(\theta,c^{0})}{\partial\theta}=\frac{\partial L_{Y}(P( \theta),c^{\prime})}{\partial\theta^{\prime}}$ $$

Thus, if the same initialization is performed, $\hat{f}_{-p_{M}}\cdot c^{\prime}=\hat{f}_{p_{M}=0}\cdot c^{0}$ for any $c$ in the dataset.
∎

###### Theorem D.2 .

For the retrained concept predictor $\hat{g}_{-p_{M}}$ defined by

$$ $\hat{g}_{-p_{M}}=\operatorname*{arg\,min}_{g^{\prime}}\sum_{j\notin M}L_{C_{j} }=\operatorname*{arg\,min}_{g^{\prime}}\sum_{j\notin M}\sum_{i=1}^{n}L_{C}({g^ {\prime}}^{j}(x_{i}),c_{i}^{j}),$ $$

we map it to $\hat{g}^{*}_{-p_{M}}$ as

$$ $\hat{g}^{*}_{-p_{M}}=\operatorname*{arg\,min}_{g^{\prime}\in\Gamma_{0}}\sum_{j \notin M}^{k}\sum_{i=1}^{n}L_{C}(g^{\prime j}(x_{i}),c_{i}^{j}).$ $$

And we can edit the initial $\hat{g}$ to $\hat{g}^{*}_{-p_{M}}$ as:

$$ $\hat{g}^{*}_{-p_{M}}\approx\bar{g}^{*}_{-p_{M}}\triangleq\hat{g}-H^{-1}_{\hat{ g}}\cdot\nabla_{\hat{g}}\sum_{j\notin M}\sum_{i=1}^{n}L_{C}(\hat{g}^{j}(x_{i}) ,c^{j}_{i}),$ (22) $$

where $H_{\hat{g}}=\nabla^{2}_{g}\sum_{j\notin M}\sum_{i=1}^{n}L_{C}(\hat{g}^{j}(x_{i
}),c^{j}_{i})$.

Then, by removing all zero rows inserted during the mapping phase, we can naturally approximate $\hat{g}_{-p_{M}}\approx\mathrm{P}^{-1}(\hat{g}^{*}_{-p_{M}})$.

###### Proof.

At this level, we consider the scenario that removes a set of mislabeled concepts or introduces new ones. Because after removing concepts from all the data, the new concept predictor has a different dimension from the original. We denote $g^{j}(x_{i})$ as the $j$-th concept predictor with $x_{i}$, and $c_{i}^{j}$ as the $j$-th concept in data $z_{i}$. For simplicity, we treat $g$ as a collection of $k$ concept predictors and separate different columns as a vector $g^{j}(x_{i})$. Actually, the neural network gets $g$ as a whole.

For the comparative purpose, we introduce a new notation $\hat{g}_{-p_{M}}^{*}$. Specifically, we define weights of $\hat{g}$ and $\hat{g}_{-p_{M}}^{*}$ for the last layer of the neural network as follows.

$$ $\hat{g}_{-p_{M}}(x)=\underbrace{\left(\begin{array}[]{cccc}w_{11}&w_{12}& \cdots&w_{1m}\\ w_{21}&w_{22}&\cdots&w_{2m}\\ \vdots&\vdots&&\vdots\\ w_{(k-1)1}&w_{(k-1)2}&\cdots&w_{(k-1)m}\end{array}\right)}_{{(k-1)}\times m} \cdot\underbrace{\left(\begin{array}[]{c}x^{1}\\ x^{2}\\ \vdots\\ x^{m}\end{array}\right)}_{m\times 1}=\underbrace{\left(\begin{array}[]{c}c_{1} \\ \vdots\\ c_{r-1}\\ c_{r+1}\\ \vdots\\ c_{k}\end{array}\right)}_{(k-1)\times 1}$ $$

$$ $\hat{g}^{*}_{-p_{M}}(x)=\underbrace{\left(\begin{array}[]{cccc}w_{11}&w_{12}& \cdots&w_{1m}\\ \vdots&\vdots&&\vdots\\ w_{(r-1)1}&w_{(r-1)2}&\cdots&w_{(r-1)m}\\ 0&0&\cdots&0\\ w_{(r+1)1}&w_{(r+1)2}&\cdots&w_{(r+1)m}\\ \vdots&\vdots&&\vdots\\ w_{k1}&w_{k2}&\cdots&w_{km}\end{array}\right)}_{k\times m}\cdot\underbrace{ \left(\begin{array}[]{c}x^{1}\\ \vdots\\ x^{r-1}\\ x^{r}\\ x^{r+1}\\ \vdots\\ x^{m}\end{array}\right)}_{m\times 1}=\underbrace{\left(\begin{array}[]{c}c_{1} \\ \vdots\\ c_{r-1}\\ 0\\ c_{r+1}\\ \vdots\\ c_{k}\end{array}\right)}_{k\times 1},$ $$

where $r$ is an index from the index set $M$.

Firstly, we want to edit to $\hat{g}^{*}_{-p_{M}}\in T_{0}=\{w_{\text{final}}=0\}\subseteq T$ based on $\hat{g}$, where $w_{\text{final}}$ is the parameter of the final layer of neural network. Let us take a look at the definition of $\hat{g}^{*}_{-p_{M}}$:

$$ $\hat{g}^{*}_{-p_{M}}=\operatorname*{arg\,min}_{g^{\prime}\in T_{0}}\sum_{j \notin M}\sum_{i=1}^{n}L_{C}(g^{\prime j}(x_{i}),c_{i}^{j}).$ $$

Then, we separate the $r$-th concept-related item from the rest and rewrite $\hat{g}$ as the following form:

$$ $\hat{g}=\operatorname*{arg\,min}_{g\in T}\left[\sum_{j=1,j\neq r}^{k}\sum_{i=1 }^{n}L_{C}(g^{j}(x_{i}),c^{j}_{i})+\sum_{r\in M}\sum_{i=1}^{n}L_{C}(g^{r}(x_{i }),c^{r}_{i})\right].$ $$

Then, if the $r$-th concept part is up-weighted by some small $\epsilon$, this gives us the new parameters
$\hat{g}_{\epsilon,p_{M}}$, which we will abbreviate as $\hat{g}_{\epsilon}$ below.

$$ $\hat{g}_{\epsilon,p_{M}}\triangleq\operatorname*{arg\,min}_{g\in T}\left[\sum_ {j\notin M}\sum_{i=1}^{n}L_{C}(g^{j}(x_{i}),c^{j}_{i})+\epsilon\cdot\sum_{r\in M }\sum_{i=1}^{n}L_{C}(g^{r}(x_{i}),c^{r}_{i})\right].$ $$

Obviously, when $\epsilon\rightarrow 0$, $\hat{g}_{\epsilon}\to\hat{g}^{*}_{-p_{M}}$. We can obtain the minimization conditions from the definitions above.

$$ $\nabla_{\hat{g}^{*}_{-p_{M}}}\sum_{j\notin M}\sum_{i=1}^{n}L_{C_{j}}(\hat{g}^{ *}_{-p_{M}}(x_{i}),c_{i})=0.$ (23) $$

$$ $\nabla_{\hat{g}_{\epsilon}}\sum_{j\notin M}\sum_{i=1}^{n}L_{C_{j}}(\hat{g}_{ \epsilon}(x_{i}),c_{i})+\epsilon\cdot\nabla_{\hat{g}_{\epsilon}}\sum_{r\in M} \sum_{i=1}^{n}L_{C_{r}}(\hat{g}_{\epsilon}(x_{i}),c_{i})=0.$ $$

Perform a first-order Taylor expansion of equation 23 with respect to $\hat{g}_{\epsilon}$, then we get

$$ $\nabla_{g}\sum_{j\notin M}\sum_{i=1}^{n}L_{C_{j}}(\hat{g}_{\epsilon}(x_{i}),c_ {i})+\nabla^{2}_{g}\sum_{j\notin M}\sum_{i=1}^{n}L_{C_{j}}(\hat{g}_{\epsilon}( x_{i}),c_{i})\cdot(\hat{g}^{*}_{-p_{M}}-\hat{g}_{\epsilon})\approx 0.$ $$

Then we have

$$ $\hat{g}^{*}_{-p_{M}}-\hat{g}_{\epsilon}=-H^{-1}_{\hat{g}_{\epsilon}}\cdot \nabla_{g}\sum_{j\notin M}\sum_{i=1}^{n}L_{C_{j}}(\hat{g}_{\epsilon}(x_{i}),c_ {i}).$ $$

Where $H_{\hat{g}_{\epsilon}}=\nabla^{2}_{g}\sum_{j\notin M}\sum_{i=1}^{n}L_{C_{j}}(
\hat{g}_{\epsilon}(x_{i}),c_{i})$.

We can see that:

When $\epsilon=0$,

$$ $\hat{g}_{\epsilon}=\hat{g}^{*}_{-p_{M}},$ $$

When $\epsilon=1$, $\hat{g}_{\epsilon}=\hat{g}$,

$$ $\hat{g}^{*}_{-p_{M}}-\hat{g}\approx-H^{-1}_{\hat{g}}\cdot\nabla_{g}\sum_{j \notin M}\sum_{i=1}^{n}L_{C_{j}}(\hat{g}(x_{i}),c_{i}),$ $$

where $H_{\hat{g}}=\nabla^{2}_{g}\sum_{j\notin M}\sum_{i=1}^{n}L_{C_{j}}(\hat{g}(x_{i
}),c_{i})$.

Then, an approximation of $\hat{g}^{*}_{-p_{M}}$ is obtained.

$$ $\hat{g}^{*}_{-p_{M}}\approx\hat{g}-H^{-1}_{\hat{g}}\cdot\nabla_{g}\sum_{j \notin M}\sum_{i=1}^{n}L_{C_{j}}(\hat{g}(x_{i}),c_{i}).$ (24) $$

∎

###### Theorem D.3 .

For the retrained label predictor $\hat{f}_{-p_{M}}$ defined as

$$ $\hat{f}_{-p_{M}}=\operatorname*{arg\,min}_{f^{\prime}}\sum_{i=1}^{n}L_{Y}= \operatorname*{arg\,min}_{f^{\prime}}\sum_{i=1}^{n}L_{Y}(f^{\prime}(\hat{g}_{- p_{M}}(x_{i})),y_{i}),$ $$

We can consider its equivalent version $\hat{f}_{p_{M}=0}$ as:

$$ $\hat{f}_{p_{M}=0}=\operatorname*{arg\,min}_{f}\sum_{i=1}^{n}L_{Y_{i}}\left(f, \hat{g}^{*}_{-p_{M}}\right),$ $$

which can be edited by

$$ $\hat{f}_{p_{M}=0}\approx\bar{f}_{p_{M}=0}\triangleq\hat{f}-H_{\hat{f}}^{-1} \cdot\nabla_{\hat{f}}\sum_{l=1}^{n}L_{Y_{l}}\left(\hat{f},\bar{g}^{*}_{-p_{M}} \right),$ $$

where $H_{\hat{f}}=\nabla^{2}_{\hat{f}}\sum_{i=1}^{n}L_{Y_{i}}(\hat{f},\hat{g})$ is the Hessian matrix of the loss function respect to is the Hessian matrix of the loss function respect to $\hat{f}$.

###### Proof.

Now, we come to the approximation of $\hat{f}_{-p_{M}}$. Noticing that the input dimension of $f$ decreases to $k-|M|$. We consider setting $p_{r}=0$ for all data points in the training phase of the label predictor and get another optimal model $\hat{f}_{p_{M}=0}$. From lemma D.1, we know that for the same input $x$, $\hat{f}_{p_{M}=0}(x)=\hat{f}_{-p_{M}}$. And the values of the corresponding parameters in $\hat{f}_{p_{M}=0}$ and $\hat{f}_{-p_{M}}$ are equal.

Now, let us consider how to edit the initial $\hat{f}$ to $\hat{f}_{p_{M}=0}$.
Firstly, assume we only use the updated concept predictor $\hat{g}^{*}_{-p_{M}}$ for one data $(x_{i_{r}},y_{i_{r}},c_{i_{r}})$ and obtain the following $\hat{f}_{ir}$, which is denoted as

$$ $\hat{f}_{ir}=\operatorname*{arg\,min}_{f}\left[\sum_{i=1}^{n}L_{Y}(f(\hat{g}(x _{i})),y_{i})+L_{Y}(f(\hat{g}^{*}_{-p_{M}}(x_{ir})),y_{ir})-L_{Y}(f(\hat{g}(x_ {ir})),y_{ir})\right].$ $$

Then up-weight the $i_{r}$-th data by some small $\epsilon$ and have the following new parameters:

$$ $\hat{f}_{\epsilon,ir}=\operatorname*{arg\,min}_{f}\left[\sum_{i=1}^{n}L_{Y}(f( \hat{g}(x_{i})),y_{i})+\epsilon\cdot L_{Y}(f(\hat{g}^{*}_{-p_{M}}(x_{ir})),y_{ ir})-\epsilon\cdot L_{Y}(f(\hat{g}(x_{ir})),y_{ir})\right].$ $$

Deduce the minimized condition subsequently,

$$ $\nabla_{f}\sum_{i=1}^{n}L_{Y}(\hat{f}_{ir}(\hat{g}(x_{i})),y_{i})+\epsilon \cdot\nabla_{f}L_{Y}(\hat{f}_{ir}(\hat{g}^{*}_{-p_{M}}(x_{ir})),y_{ir})- \epsilon\cdot\nabla_{f}L_{Y}(\hat{f}_{ir}(\hat{g}(x_{ir})),y_{ir})=0.$ $$

If we expand first term of $\hat{f}$, which $\hat{f}_{ir,\epsilon}\to\hat{f}(\epsilon\to 0)$, then

$$ $\displaystyle\nabla_{f}\sum_{i=1}^{n}L_{Y}\left(\hat{f}(\hat{g}(x_{i})),y_{i}\right)$ $\displaystyle+\epsilon\cdot\nabla_{f}L_{Y}(\hat{f}(\hat{g}^{*}_{-p_{M}}(x_{ir} )),y_{ir})-\epsilon\cdot\nabla_{f}L_{Y}(\hat{f}(\hat{g}(x_{ir})),y_{ir})$ $\displaystyle+\left(\nabla^{2}_{f}\sum_{i=1}^{n}L_{Y}\left(\hat{f}(\hat{g}(x_{ i})),y_{i}\right)\right)\cdot(\hat{f}_{ir,\epsilon}-\hat{f})=0.$ $$

Note that $\nabla_{f}\sum_{i=1}^{n}L_{Y}(\hat{f}(\hat{g}(x_{i})),y_{i})=0$. Thus we have

$$ $\hat{f}_{ir,\epsilon}-\hat{f}=H^{-1}_{\hat{f}}\cdot\epsilon\left(\nabla_{f}L_{ Y}(\hat{f}(\hat{g}^{*}_{-p_{M}}(x_{ir})),y_{ir})-\nabla_{f}L_{Y}(\hat{f}(\hat{ g}(x_{ir})),y_{ir})\right).$ $$

We conclude that

$$ $\left.\frac{\mathrm{d}\hat{f}_{\epsilon,ir}}{\mathrm{d}\epsilon}\right|_{ \epsilon=0}=H^{-1}_{\hat{f}}\cdot\left(\nabla_{\hat{f}}L_{Y}(\hat{f}(\hat{g}^{ *}_{-p_{M}}(x_{ir})),y_{ir})-\nabla_{\hat{f}}L_{Y}(\hat{f}(\hat{g}(x_{ir})),y_ {ir})\right).$ $$

Perform a one-step Newtonian iteration at $\hat{f}$ and we get the approximation of $\hat{f}_{i_{r}}$.

$$ $\hat{f}_{ir}\approx\hat{f}+H^{-1}_{\hat{f}}\cdot\left(\nabla_{\hat{f}}L_{Y}( \hat{f}(\hat{g}(x_{ir})),y_{ir})-\nabla_{\hat{f}}L_{Y}(\hat{f}(\hat{g}^{*}_{-p _{M}}(x_{ir})),y_{ir})\right).$ $$

Reconsider the definition of $\hat{f}_{i_{r}}$, we use the updated concept predictor $\hat{g}_{-p_{M}}^{*}$ for one data $(x_{i_{r}},y_{i_{r}},c_{i_{r}})$. Now we carry out this operation for all the other data and estimate $\hat{f}_{p_{M}=0}$. Combining the minimization condition from the definition of $\hat{f}$, we have

$$ $\displaystyle\hat{f}_{p_{M}=0}\approx$ $\displaystyle\hat{f}+H^{-1}_{\hat{f}}\cdot\left(\nabla_{\hat{f}}\sum_{i=1}^{n} L_{Y}(\hat{f}(\hat{g}(x_{i})),y_{i})-\nabla_{\hat{f}}\sum_{i=1}^{n}L_{Y}(\hat{ f}(\hat{g}^{*}_{-p_{M}}(x_{i})),y_{i})\right)$ $\displaystyle=$ $\displaystyle\hat{f}+H^{-1}_{\hat{f}}\cdot\left(-\nabla_{\hat{f}}\sum_{i=1}^{n }L_{Y}(\hat{f}(\hat{g}^{*}_{-p_{M}}(x_{i})),y_{i})\right)$ $\displaystyle=$ $\displaystyle\hat{f}-H^{-1}_{\hat{f}}\sum_{l=1}^{n}\nabla_{\hat{f}}L_{Y}(\hat{ f}(\hat{g}^{*}_{-p_{M}}(x_{l})),y_{l}).$ (25) $$

Theorem D.2 gives us the edited version of $\hat{g}^{*}_{-p_{M}}$. Substitute it into equation 25, and we get the final closed-form edited label predictor under concept level:

$$ $\hat{f}_{p_{M}=0}\approx\bar{f}_{p_{M}=0}\triangleq\hat{f}-H_{\hat{f}}^{-1} \cdot\nabla_{\hat{f}}\sum_{l=1}^{n}L_{Y_{l}}\left(\hat{f},\bar{g}^{*}_{-p_{M}} \right),$ $$

where $H_{\hat{f}}=\nabla^{2}_{\hat{f}}\sum_{i=1}^{n}L_{Y_{i}}(\hat{f},\hat{g})$ is the Hessian matrix of the loss function respect to is the Hessian matrix of the loss function respect to $\hat{f}$.
∎

###### Lemma D.1 .

###### Proof.

###### Theorem D.2 .

###### Proof.

###### Theorem D.3 .

###### Proof.

## Appendix E Proof of Data-level Influence

We address situations that for dataset $\mathcal{D}=\{(x_{i},y_{i},c_{i})\}_{i=1}^{n}$, given a set of data $z_{r}=(x_{r},y_{r},c_{r})$, $r\in G$ to be removed. Our goal is to estimate $\hat{g}_{-z_{G}}$, $\hat{f}_{-z_{G}}$, which is the concept and label predictor trained on the $z_{r}$ for $r\in G$ removed dataset.

#### Proof Sketch.

(i) First, we estimate the retrained concept predictor $\hat{g}_{-z_{G}}$. (ii) Then, we define a new label predictor $\tilde{f}_{-z_{G}}$ and estimate $\tilde{f}_{-z_{G}}-\hat{f}$. (iii) Next, in order to reduce computational complexity, use the lemma method to obtain the approximation of the Hessian matrix of $\tilde{f}_{-z_{G}}$. (iv) Next, we compute the difference $\hat{f}_{-z_{G}}-\tilde{f}_{-z_{G}}$ as

$$ $-H^{-1}_{\tilde{f}_{-z_{G}}}\cdot\left(\nabla_{\hat{f}}L_{Y}\left(\tilde{f}_{- z_{G}}(\hat{g}_{-z_{G}}(x_{i_{r}})),y_{i_{r}}\right)-\nabla_{\hat{f}}L_{Y} \left(\tilde{f}_{-z_{G}}(\hat{g}(x_{i_{r}})),y_{i_{r}}\right)\right).$ $$

(v) Finally, we divide $\hat{f}_{-z_{G}}-\hat{f}$, which we actually concerned with, into $\left(\hat{f}_{-z_{G}}-\tilde{f}_{-z_{G}}\right)+\left(\tilde{f}_{-z_{G}}-\hat
{f}\right)$.

###### Theorem E.1 .

For dataset $\mathcal{D}=\{(x_{i},y_{i},c_{i})\}_{i=1}^{n}$, given a set of data $z_{r}=(x_{r},y_{r},c_{r})$, $r\in G$ to be removed. Suppose the updated concept predictor $\hat{g}_{-z_{G}}$ is defined by

$$ $\hat{g}_{-z_{G}}=\operatorname*{arg\,min}_{g}\sum_{j\in[k]}\sum_{i\in[n]-G}L_{ C_{j}}(\hat{g}(x_{i}),c_{i})$ $$

where $L_{C}(\hat{g}(x_{i}),c_{i})\triangleq\sum_{j=1}^{k}L_{C_{j}}(\hat{g}(x_{i}),c_
{i})$.
Then we have the following approximation for $\hat{g}_{-z_{G}}$

$$ $\hat{g}_{-z_{G}}\approx\bar{g}_{-z_{G}}\triangleq\hat{g}+H^{-1}_{\hat{g}}\cdot \sum_{r\in G}\nabla_{g}L_{C}(\hat{g}(x_{r}),c_{r}),$ (26) $$

where $H_{\hat{g}}=\nabla^{2}_{\hat{g}}\sum_{i,j}L_{C}(\hat{g}^{j}(x_{i}),c_{i}^{j})$ is the Hessian matrix of the loss function respect to $\hat{g}$.

###### Proof.

Firstly, we rewrite $\hat{g}_{-z_{G}}$ as

$$ $\hat{g}_{-z_{G}}=\operatorname*{arg\,min}_{g}\left[\sum_{\begin{subarray}{c}i= 1\end{subarray}}^{n}L_{C}(\hat{g}(x_{i}),c_{i})-\sum_{r\in G}L_{C}(g(x_{r}),c_ {r})\right],$ $$

Then we up-weighted the $r$-th data by some $\epsilon$ and have a new predictor $\hat{g}_{-z_{G},\epsilon}$, which is abbreviated as $\hat{g}_{\epsilon}$:

$$ $\hat{g}_{\epsilon}\triangleq\operatorname*{arg\,min}_{g}\left[\sum_{i=1}^{n}L_ {C}(g(x_{i}),c_{i})-\epsilon\cdot\sum_{r\in G}L_{C}(g(x_{r}),c_{r})\right].$ (27) $$

Because $\hat{g}_{\epsilon}$ minimizes the right side of equation 27, we have

$$ $\nabla_{\hat{g}_{\epsilon}}\sum_{i=1}^{n}L_{Y}(\hat{g}_{\epsilon}(x_{i}),c_{i} )-\epsilon\cdot\nabla_{\hat{g}_{\epsilon}}\sum_{r\in G}L_{Y}(\hat{g}_{\epsilon }(x_{r}),c_{r})=0.$ $$

When $\epsilon\rightarrow 0$, $\hat{g}_{\epsilon}\rightarrow\hat{g}$.
So we can perform a first-order Taylor expansion with respect to $\hat{g}$, and we have

$$ $\nabla_{g}\sum_{i=1}^{n}L_{C}(\hat{g}(x_{i}),c_{i})-\epsilon\cdot\nabla_{g} \sum_{r\in G}L_{C}(\hat{g}(x_{r}),c_{r})+\nabla^{2}_{g}\sum_{i=1}^{n}L_{C}( \hat{g}(x_{i}),c_{i})\cdot(\hat{g}_{\epsilon}-\hat{g})\approx 0.$ (28) $$

Recap the definition of $\hat{g}$:

$$ $\hat{g}=\operatorname*{arg\,min}_{g}\sum_{i=1}^{n}L_{Y}(g(x_{i}),c_{i}),$ $$

Then, the first term of equation 28 equals 0 0. Let $\epsilon\rightarrow 0$, then we have

$$ $\left.\frac{\mathrm{d}\hat{g}_{\epsilon}}{\mathrm{d}\epsilon}\right|_{\epsilon =0}=H^{-1}_{\hat{g}}\cdot\sum_{r\in G}\nabla_{g}L_{C}(\hat{g}(x_{r}),c_{r}),$ $$

where $H^{-1}_{\hat{g}}=\nabla^{2}_{g}\sum_{i=1}^{n}\ell(\hat{g}(x_{i}),c_{i})$.

Remember when $\epsilon\rightarrow 0$, $\hat{g}_{\epsilon}\rightarrow\hat{g}_{-z_{G}}$.
Perform a Newton step at $\hat{g}$, then we obtain the method to edit the original concept predictor under concept level:

$$ $\hat{g}_{-z_{G}}\approx\bar{g}_{-z_{G}}\triangleq\hat{g}+H^{-1}_{\hat{g}}\cdot \sum_{r\in G}\nabla_{g}L_{C}(\hat{g}(x_{r}),c_{r}).$ $$

∎

###### Theorem E.2 .

For dataset $\mathcal{D}=\{(x_{i},y_{i},c_{i})\}_{i=1}^{n}$, given a set of data $z_{r}=(x_{r},y_{r},c_{r})$, $r\in G$ to be removed. The label predictor $\hat{f}_{-z_{G}}$ trained on the revised dataset becomes

$$ $\hat{f}_{-z_{G}}=\operatorname*{arg\,min}_{f}\sum_{i\in[n]-G}L_{Y_{i}}(f,\hat{ g}_{-z_{G}}).$ (29) $$

The intermediate label predictor $\tilde{f}_{-z_{G}}$ is defined by

$$ $\tilde{f}_{-z_{G}}=\operatorname*{arg\,min}\sum_{i\in[n]-G}L_{Y_{i}}(f,\hat{g}),$ $$

Then $\tilde{f}_{-z_{G}}-\hat{f}$ can be approximated by

$$ $\tilde{f}_{-z_{G}}-\hat{f}\approx H^{-1}_{\hat{f}}\cdot\sum_{i\in[n]-G}\nabla_ {\hat{f}}L_{Y_{i}}(\hat{f},\hat{g})\triangleq A_{G}.$ (30) $$

We denote the edited version of $\tilde{f}_{-z_{G}}$ as $\bar{f}^{*}_{-z_{G}}\triangleq\hat{f}+A_{G}$. And $\hat{f}_{-z_{G}}-\tilde{f}_{-z_{G}}$ can be approximated by

$$ $\begin{split}\hat{f}_{-z_{G}}-\tilde{f}_{-z_{G}}\approx&-H^{-1}_{\bar{f}^{*}_{ -z_{G}}}\cdot\left(\nabla_{\hat{f}}\sum_{i\in[n]-G}L_{Y_{i}}\left(\bar{f}^{*}_ {-z_{G}},\bar{g}_{-z_{G}}\right)-\nabla_{\hat{f}}\sum_{i\in[n]-G}L_{Y_{i}} \left(\bar{f}^{*}_{-z_{G}},\hat{g}\right)\right)\triangleq B_{G},\end{split}$ (31) $$

where $H_{\bar{f}^{*}_{-z_{G}}}=\nabla_{\bar{f}}\sum_{i\in[n]-G}L_{Y_{i}}\left(\bar{f
}^{*}_{-z_{G}},\hat{g}\right)$ is the Hessian matrix of the loss function on the intermediate dataset concerning $\bar{f}^{*}_{-z_{G}}$. Then, the final edited label predictor $\bar{f}_{-z_{G}}$ can be obtained by

$$ $\bar{f}_{-z_{G}}=\bar{f}^{*}_{-z_{G}}+B_{G}=\hat{f}+A_{G}+B_{G}.$ (32) $$

###### Proof.

We can see that there is a huge gap between $\hat{f}_{-z_{G}}$ and $\hat{f}$. Thus, firstly, we define $\tilde{f}_{-z_{G}}$ as

$$ $\tilde{f}_{-z_{G}}=\operatorname*{arg\,min}_{f}\sum_{i=1}^{n}L_{Y}\left(f(\hat {g}(x_{i})),y_{i}\right)-\sum_{r\in G}L_{Y}\left(f(\hat{g}(x_{r})),y_{r}\right).$ $$

Then, we define $\tilde{f}_{\epsilon,-z_{G}}$ as follows to estimate $\tilde{f}_{-z_{G}}$.

$$ $\tilde{f}_{\epsilon,-z_{G}}=\operatorname*{arg\,min}_{f}\sum_{i=1}^{n}L_{Y} \left(f(\hat{g}(x_{i})),y_{i}\right)-\epsilon\cdot\sum_{r\in G}L_{Y}\left(f( \hat{g}(x_{r})),y_{r}\right).$ $$

From the minimization condition, we have

$$ $\nabla_{\tilde{f}}\sum_{i=1}^{n}L_{Y}\left(\tilde{f}_{\epsilon,-z_{G}}(\hat{g} (x_{i})),y_{i}\right)-\epsilon\cdot\sum_{r\in G}\nabla_{\tilde{f}}L_{Y}\left( \tilde{f}_{\epsilon,-z_{G}}(\hat{g}(x_{r})),y_{r}\right)=0.$ $$

Perform a first-order Taylor expansion at $\hat{f}$,

$$ $\displaystyle\nabla_{\hat{f}}\sum_{i=1}^{n}L_{Y}\left(\hat{f}(\hat{g}(x_{i})), y_{i}\right)-\epsilon\cdot\nabla_{\hat{f}}\sum_{r\in G}L_{Y}\left(\hat{f}(\hat {g}(x_{r})),y_{r}\right)$ $\displaystyle+\nabla^{2}_{\hat{f}}\sum_{i=1}^{n}L_{Y}\left(\hat{f}(\hat{g}(x_{ i})),y_{i}\right)\cdot\left(\tilde{f}_{\epsilon,-z_{G}}-\hat{f}\right)=0.$ $$

Then $\tilde{f}_{-z_{G}}$ can be approximated by

$$ $\tilde{f}_{-z_{G}}\approx\hat{f}+H^{-1}_{\hat{f}}\cdot\sum_{r\in G}\nabla_{ \hat{f}}L_{Y}\left(\hat{f}(\hat{g}(x_{r})),y_{r}\right)\triangleq A_{G}.$ (33) $$

Then the edit version of $\tilde{f}_{-z_{G}}$ is defined as

$$ $\bar{f}^{*}_{-z_{G}}=\hat{f}+A_{G}$ (34) $$

Then we estimate the difference between $\hat{f}_{-z_{G}}$ and $\tilde{f}_{-z_{G}}$. Rewrite $\tilde{f}_{-z_{G}}$ as

$$ $\tilde{f}_{-z_{G}}=\operatorname*{arg\,min}_{f}\sum_{i\in S}^{n}L_{Y}\left(f( \hat{g}(x_{i})),y_{i}\right),$ (35) $$

where $S\triangleq[n]-G$.

Compare equation 29 with 35, we still need to define an intermediary predictor $\hat{f}_{-z_{G},ir}$ as

$$ $\displaystyle\hat{f}_{-z_{G},ir}$ $\displaystyle=\operatorname*{arg\,min}_{f}\left[\sum_{\begin{subarray}{c}i\in S \\ i\neq i_{r}\end{subarray}}L_{Y_{i}}\left(f,\hat{g}(x_{i})\right)+L_{Y_{ir}} \left(f,\hat{g}_{-z_{G}}\right)\right]$ $\displaystyle=\operatorname*{arg\,min}_{f}\left[\sum_{\begin{subarray}{c}i\in S \end{subarray}}L_{Y_{i}}\left(f,\hat{g}\right)+L_{Y_{ir}}\left(f,\hat{g}_{-z_{ G}}\right)-L_{Y_{ir}}\left(f,\hat{g}\right)\right].$ $$

Up-weight the $i_{r}$ data by some $\epsilon$, we define $\hat{f}_{\epsilon,-z_{G},i_{r}}$ as

$$ $\hat{f}_{\epsilon,-z_{G},ir}=\operatorname*{arg\,min}_{f}\left[\sum_{\begin{ subarray}{c}i\in S\end{subarray}}L_{Y_{i}}\left(f,\hat{g}\right)+\epsilon\cdot L _{Y_{ir}}\left(f,\hat{g}_{-z_{G}}\right)-\epsilon\cdot L_{Y_{ir}}\left(f,\hat{ g}\right)\right].$ $$

We denote $\hat{f}_{\epsilon,-z_{G},ir}$ as $\hat{f}_{\epsilon}^{*}$ in the following proof. Then, from the minimization condition, we have

$$ $\nabla_{\hat{f}}\sum_{\begin{subarray}{c}i\in S\end{subarray}}L_{Y_{i}}\left( \hat{f}_{\epsilon}^{*},\hat{g}\right)+\epsilon\cdot\nabla_{\hat{f}}L_{Y_{ir}} \left(\hat{f}_{\epsilon}^{*},\hat{g}_{-z_{G}}\right)-\epsilon\cdot\nabla_{\hat {f}}L_{Y_{ir}}\left(\hat{f}_{\epsilon}^{*},\hat{g}(x_{i_{r}}\right).$ (36) $$

When $\epsilon\rightarrow 0$, $\hat{f}_{\epsilon}^{*}\rightarrow\tilde{f}_{-z_{G}}$.
Then we perform a Taylor expansion at $\tilde{f}_{-z_{G}}$ of equation 36 and have

$$ $\displaystyle\nabla_{\hat{f}}\sum_{\begin{subarray}{c}i\in S\end{subarray}}L_{ Y_{i}}\left(\tilde{f}_{-z_{G}},\hat{g}\right)+\epsilon\cdot\nabla_{\hat{f}}L_{ Y_{ir}}\left(\tilde{f}_{-z_{G}},\hat{g}_{-z_{G}}\right)$ $\displaystyle-\epsilon\cdot\nabla_{\hat{f}}L_{Y_{ir}}\left(\tilde{f}_{-z_{G}}, \hat{g}\right)+\nabla^{2}_{\hat{f}}\sum_{\begin{subarray}{c}i\in S\end{ subarray}}L_{Y_{i}}\left(\tilde{f}_{-z_{G}},\hat{g}\right)\cdot(\hat{f}_{ \epsilon}^{*}-\tilde{f}_{-z_{G}})\approx 0.$ $$

Organizing the above equation gives

$$ $\hat{f}_{\epsilon}^{*}-\tilde{f}_{-z_{G}}\approx-\epsilon\cdot H^{-1}_{\tilde{ f}_{-z_{G}}}\cdot\left(\nabla_{\hat{f}}L_{Y_{ir}}\left(\tilde{f}_{-z_{G}},\hat {g}_{-z_{G}}\right)-\nabla_{\hat{f}}L_{Y_{ir}}\left(\tilde{f}_{-z_{G}},\hat{g} \right)\right),$ $$

where $H_{\tilde{f}_{-z_{G}}}=\nabla^{2}_{\hat{f}}\sum_{\begin{subarray}{c}i\in S\end
{subarray}}L_{Y_{i}}\left(\tilde{f}_{-z_{G}},\hat{g}\right)$.

When $\epsilon=1$, $\hat{f}_{\epsilon}^{*}=\hat{f}_{-z_{G},ir}$. Then we perform a Newton iteration with step size $1$ at $\tilde{f}_{-z_{G}}$,

$$ $\hat{f}_{-z_{G},ir}-\tilde{f}_{-z_{G}}\approx-H^{-1}_{\tilde{f}_{-z_{G}}}\cdot \left(\nabla_{\hat{f}}L_{Y_{ir}}\left(\tilde{f}_{-z_{G}},\hat{g}_{-z_{G}} \right)-\nabla_{\hat{f}}L_{Y_{ir}}\left(\tilde{f}_{-z_{G}},\hat{g}\right)\right)$ $$

Iterate $i_{r}$ through set $S$, and we have

$$ $\hat{f}_{-z_{G}}-\tilde{f}_{-z_{G}}\approx-H^{-1}_{\tilde{f}_{-z_{G}}}\cdot \left(\nabla_{\hat{f}}\sum_{i\in S}L_{Y_{i}}\left(\tilde{f}_{-z_{G}},\hat{g}_{ -z_{G}}\right)-\nabla_{\hat{f}}\sum_{i\in S}L_{Y_{i}}\left(\tilde{f}_{-z_{G}}, \hat{g}\right)\right)$ (37) $$

The edited version of $\hat{g}_{-z_{G}}$ has been deduced as $\bar{g}_{-z_{G}}$ in theorem E.1, substituting this approximation into equation 37, then we have

$$ $\hat{f}_{-z_{G}}-\tilde{f}_{-z_{G}}\approx-H^{-1}_{\tilde{f}_{-z_{G}}}\cdot \left(\nabla_{\hat{f}}\sum_{i\in S}L_{Y_{i}}\left(\tilde{f}_{-z_{G}},\bar{g}_{ -z_{G}}\right)-\nabla_{\hat{f}}\sum_{i\in S}L_{Y_{i}}\left(\tilde{f}_{-z_{G}}, \hat{g}\right)\right).$ (38) $$

Noting that we cannot obtain $\hat{f}_{-z_{G}}$ and $H_{\tilde{f}_{-z_{G}}}$ directly because we do not retrain the label predictor but edit it to $\bar{f}^{*}_{-z_{G}}$ as a substitute. Therefore, we approximate $\hat{f}_{-z_{G}}$ with $\bar{f}^{*}_{-z_{G}}$ and $H_{\tilde{f}_{-z_{G}}}$ with $H_{\bar{f}^{*}_{-z_{G}}}$ which is defined by:

$$ $H_{\bar{f}^{*}_{-z_{G}}}=\nabla^{2}_{\hat{f}}\sum_{\begin{subarray}{c}i\in S \end{subarray}}L_{Y_{i}}\left(\bar{f}^{*}_{-z_{G}},\hat{g}\right)$ $$

Then we define $B_{G}$ as

$$ $B_{G}\triangleq-H^{-1}_{\bar{f}^{*}_{-z_{G}}}\cdot\left(\nabla_{\hat{f}}\sum_{ i\in S}L_{Y_{i}}\left(\bar{f}^{*}_{-z_{G}},\bar{g}_{-z_{G}}\right)-\nabla_{ \hat{f}}\sum_{i\in S}L_{Y_{i}}\left(\bar{f}^{*}_{-z_{G}},\hat{g}\right)\right)$ (39) $$

Combining equation 34 and equation 39, then we deduce the final closed-form edited label predictor as

$$ $\bar{f}_{-z_{G}}=\bar{f}^{*}_{-z_{G}}+B_{G}=\hat{f}+A_{G}+B_{G}.$ $$

∎

###### Theorem E.1 .

###### Proof.

###### Theorem E.2 .

###### Proof.

## Appendix F Algorithm

Algorithm 1 Concept-label-level ECBM

$$ $H_{\hat{g}}=\nabla^{2}_{\hat{g}}\sum_{i,j}L_{C_{j}}(\hat{g}^{j}(x_{i}),c_{i}^{ j}).$ $$

$$ $\tilde{g}=\hat{g}-H^{-1}_{\hat{g}}\cdot\sum_{(w,r)\in S_{e}}\left(\nabla_{\hat {g}}L_{C_{r}}\left(\hat{g}^{r}(x_{w}),{c_{w}^{r}}^{\prime}\right)-\nabla_{\hat {g}}L_{C_{r}}\left(\hat{g}^{r}(x_{w}),c_{w}^{r}\right)\right).$ $$

$$ $H_{\hat{f}}=\nabla^{2}_{\hat{f}}\sum_{i=1}^{n}L_{Y_{i}}(\hat{f},\hat{g}).$ $$

$$ $\displaystyle\tilde{f}$ $\displaystyle=\hat{f}+H^{-1}_{\hat{f}}\cdot\nabla_{f}\sum_{i=1}^{n}L_{Y}\left( \hat{f}\left(\hat{g}(x_{i})\right),y_{i}\right)-H^{-1}_{\hat{f}}\cdot\nabla_{f }\sum_{l=1}^{n}\left(L_{Y}\left(\hat{f}\left(\tilde{g}(x_{l})\right),y_{l} \right)\right).$ $$

Algorithm 2 Concept-level ECBM

$$ $H_{\hat{g}}=\nabla^{2}_{\hat{g}}\sum_{j\notin M}\sum_{i=1}^{n}L_{C_{j}}(\hat{g }^{j}(x_{i}),c_{i}^{j}).$ $$

$$ $\tilde{g}^{*}=\hat{g}-H^{-1}_{\hat{g}}\cdot\nabla_{\hat{g}}\sum_{j\notin M} \sum_{i=1}^{n}L_{C_{j}}(\hat{g}^{j}(x_{i}),c_{i}^{j}).$ $$

$$ $H_{\hat{f}}=\nabla^{2}_{\hat{f}}\sum_{i=1}^{n}L_{Y}(\hat{f}(\hat{g}(x_{i}),y_{ i}).$ $$

$$ $\tilde{f}=\hat{f}-H_{\hat{f}}^{-1}\cdot\nabla_{\hat{f}}\sum_{l=1}^{n}L_{Y} \left(\hat{f}\left(\tilde{g}^{*}(x_{l})\right),y_{l}\right).$ $$

Algorithm 3 Data-level ECBM

$$ $H_{\hat{g}}=\nabla^{2}_{\hat{g}}\sum_{i,j}L_{C_{j}}(\hat{g}^{j}(x_{i}),c_{i}^{ j}).$ $$

$$ $\tilde{g}=\hat{g}+H^{-1}_{\hat{g}}\cdot\sum_{r\in G}\nabla_{g}L_{C}(\hat{g}(x_ {r}),c_{r})$ $$

$$ $H_{\hat{f}}=\nabla^{2}_{\hat{f}}\sum_{i=1}^{n}L_{Y}(\hat{f}(\hat{g}(x_{i}),y_{ i}).$ $$

$$ $A=H^{-1}_{\hat{f}}\cdot\sum_{i\in[n]-G}\nabla_{\hat{f}}L_{Y}\left(\hat{f}(\hat {g}(x_{i})),y_{i}\right)$ $$

$$ $\bar{f}=\hat{f}+A$ $$

$$ $H_{\bar{f}}=\nabla^{2}_{\bar{f}}\sum_{i\in[n]-G}L_{Y}(\bar{f}(\hat{g}(x_{i})), y_{i}).$ $$

$$ $B=-H^{-1}_{\bar{f}}\cdot\sum_{i\in[n]-G}\nabla_{\hat{f}}\left(L_{Y}(\bar{f}( \tilde{g}(x_{i})),y_{i})-L_{Y}(\bar{f}(\hat{g}(x_{i})),y_{i})\right)$ $$

Algorithm 4 EK-FAC for Concept Predictor $g$

$$ $\llbracket{{A}}_{l-1}\rrbracket_{t,j|\Delta|+\delta}=\left[{{A}}_{l-1}\right]_ {(t+\delta),j}=a_{j,t+\delta},$ $$

$$ $[S_{l}]_{i,t}=s_{i,t}=\sum_{\delta\in\Delta}w_{i,j,\delta}a_{j,t+\delta}+b_{i}.$ $$

$$ $\mathcal{D}s_{i,t}=\frac{\partial\sum_{j=1}^{k}\sum_{i=1}^{n}L_{C_{j}}}{ \partial s_{i,t}}$ $$

$$ $\displaystyle\hat{\Omega}_{l-1}=$ $\displaystyle\frac{1}{n}\sum_{i=1}^{n}\left(\llbracket{A}^{i}_{l-1}\rrbracket_ {\mathrm{H}}^{\top}\llbracket{A}^{i}_{l-1}\rrbracket_{\mathrm{H}}\right)$ $\displaystyle\hat{\Gamma_{l}}=$ $\displaystyle\frac{1}{n}\sum_{i=1}^{n}\left(\frac{1}{|\mathcal{T}|}\mathcal{D} {{S}^{i}_{l}}^{\top}\mathcal{D}{S}^{i}_{l}\right)$ $$

$$ $\displaystyle\hat{\Omega}_{l-1}$ $\displaystyle=Q_{\Omega}{\Lambda}_{{\Omega}}{Q}_{{\Omega}}^{\top}$ $\displaystyle\hat{\Gamma}_{l}$ $\displaystyle={Q}_{\Gamma}{\Lambda}_{\Gamma}{Q}_{\Gamma}^{\top}$ $$

$$ $\Lambda_{ii}^{*}=n^{-1}\sum_{j=1}^{n}\left(\left(Q_{\Omega_{l-1}}\otimes Q_{ \Gamma_{l}}\right)\nabla_{\theta_{l}}L_{C_{j}}\right)^{2}_{i}.$ $$

$$ $\hat{H}_{l}^{-1}=\left(Q_{\Omega_{l-1}}\otimes Q_{\Gamma_{l}}\right)\left( \Lambda+\lambda_{l}I_{d_{l}}\right)^{-1}\left(Q_{\Omega_{l-1}}\otimes Q_{ \Gamma_{l}}\right)^{\mathrm{T}}$ $$

$$ $\hat{H}_{\hat{g}}^{-1}=\left(\begin{array}[]{ccc}\hat{H}_{1}^{-1}&&\mathbf{0} \\ &\ddots&\\ \mathbf{0}&&\hat{H}_{d}^{-1}\end{array}\right)$ $$

Algorithm 5 EK-FAC for Label Predictor $f$

$$ $A=\frac{1}{n}\cdot\sum_{i=1}^{n}\hat{g}(x_{i})\cdot\hat{g}(x_{i})^{\mathrm{T}}$ $$

$$ $B=\frac{1}{n}\cdot\sum_{i=1}^{n}\nabla_{f^{\prime}}L_{Y}(\hat{f}\left(\hat{g}( x_{i})\right),y_{i})\cdot{\nabla_{f^{\prime}}L_{Y}(\hat{f}\left(\hat{g}(x_{i}) \right),y_{i})}^{\mathrm{T}}$ $$

$$ $\displaystyle A$ $\displaystyle=Q_{A}{\Lambda}_{{A}}{Q}_{{A}}^{\top}$ $\displaystyle B$ $\displaystyle={Q}_{B}{\Lambda}_{B}{Q}_{B}^{\top}$ $$

$$ $\Lambda_{ii}^{*}=n^{-1}\sum_{j=1}^{n}\left(\left(Q_{A}\otimes Q_{B}\right) \nabla_{\hat{f}}L_{Y_{j}}\right)^{2}_{i}.$ $$

$$ $\hat{H}_{\hat{f}}^{-1}=\left(Q_{A}\otimes Q_{B}\right)\left(\Lambda+\lambda I_ {d}\right)^{-1}\left(Q_{A}\otimes Q_{B}\right)^{\mathrm{T}}$ $$

Algorithm 6 EK-FAC Concept-label-level ECBM

$$ $\tilde{g}=\hat{g}-H^{-1}_{\hat{g}}\cdot\sum_{(w,r)\in S_{e}}\left(\nabla_{\hat {g}}L_{C_{r}}\left(\hat{g}^{r}(x_{w}),{c_{w}^{r}}^{\prime}\right)-\nabla_{\hat {g}}L_{C_{r}}\left(\hat{g}^{r}(x_{w}),c_{w}^{r}\right)\right).$ $$

$$ $\displaystyle\tilde{f}$ $\displaystyle=\hat{f}+H^{-1}_{\hat{f}}\cdot\nabla_{f}\sum_{i=1}^{n}L_{Y}\left( \hat{f}\left(\hat{g}(x_{i})\right),y_{i}\right)-H^{-1}_{\hat{f}}\cdot\nabla_{f }\sum_{l=1}^{n}\left(L_{Y}\left(\hat{f}\left(\tilde{g}(x_{l})\right),y_{l} \right)\right).$ $$

Algorithm 7 EK-FAC Concept-level ECBM

$$ $\tilde{g}^{*}=\hat{g}-H^{-1}_{\hat{g}}\cdot\nabla_{\hat{g}}\sum_{j\notin M} \sum_{i=1}^{n}L_{C_{j}}(\hat{g}^{j}(x_{i}),c_{i}^{j}).$ $$

$$ $\tilde{f}=\hat{f}-H_{\hat{f}}^{-1}\cdot\nabla_{\hat{f}}\sum_{l=1}^{n}L_{Y} \left(\hat{f}\left(\tilde{g}^{*}(x_{l})\right),y_{l}\right).$ $$

Algorithm 8 EK-FAC Data-level ECBM

$$ $\tilde{g}=\hat{g}-H^{-1}_{\hat{g}}\cdot\sum_{(w,r)\in S_{e}}\left(\nabla_{\hat {g}}L_{C_{r}}\left(\hat{g}^{r}(x_{w}),{c_{w}^{r}}^{\prime}\right)-\nabla_{\hat {g}}L_{C_{r}}\left(\hat{g}^{r}(x_{w}),c_{w}^{r}\right)\right).$ $$

$$ $A=H^{-1}_{\hat{f}}\cdot\sum_{i\in[n]-G}\nabla_{\hat{f}}L_{Y}\left(\hat{f}(\hat {g}(x_{i})),y_{i}\right)$ $$

$$ $\bar{f}=\hat{f}+A$ $$

$$ $B^{\prime}=-H^{-1}_{\bar{f}}\cdot\sum_{i\in[n]-G}\nabla_{\hat{f}}\left(L_{Y}( \bar{f}(\tilde{g}(x_{i})),y_{i})-L_{Y}(\bar{f}(\hat{g}(x_{i})),y_{i})\right)$ $$

## Appendix G Additional Experiments

### G.1 Experimental Setting

RMIA score.
The RMIA score is computed as:

$$ $LR_{\theta}(x,z)\approx\frac{\text{Pr}(f_{\theta}(x)|\mathcal{N}(\mu_{x,\bar{z }}(x),\sigma^{2}_{x,\bar{z}}(x)))}{\text{Pr}(f_{\theta}(x)|\mathcal{N}(\mu_{ \bar{x},z}(x),\sigma^{2}_{\bar{x},z}(x)))}\times\frac{\text{Pr}(f_{\theta}(z)| \mathcal{N}(\mu_{x,\bar{z}}(z),\sigma^{2}_{x,\bar{z}}(z)))}{\text{Pr}(f_{ \theta}(z)|\mathcal{N}(\mu_{\bar{x},z}(z),\sigma^{2}_{\bar{x},z}(z)))}$ $$

where $f_{\theta}(x)$ represents the model’s output (logits) for the data point $x$, $\mathcal{N}(\mu,\sigma^{2})$ denotes a Gaussian distribution with mean $\mu$ and variance $\sigma^{2}$, $\mu_{x,\bar{z}}(x)$ is the mean of the model’s outputs for $x$ under the assumption that $x$ belongs to the training set, and $\sigma^{2}_{x,\bar{z}}(x)$ is the variance of the model’s outputs for $x$. The likelihoods $\text{Pr}(f_{\theta}(x)|\mathcal{N})$ represent the probability that the model’s output $f_{\theta}(x)$ follows the Gaussian distribution parameterized by $\mu$ and $\sigma^{2}$, under the two different hypotheses: $x$ being a member of the training set versus not being a member.

### G.2 More Visualization Results

We provide our additional visualization results in Figure 6, 7, 8, 9, 10, 11, and 12.

Figure: Figure 6: Visualization of the top-10 most influential concepts for different classes in CUB.
Refer to caption: extracted/5901906/images/visual-3.png

Figure: Figure 7: Visualization of the top-10 most influential concepts for different classes in CUB.
Refer to caption: extracted/5901906/images/visual-7.png

Figure: Figure 8: Visualization of the most influential concept label related to different data in CUB.
Refer to caption: extracted/5901906/images/app_CUB/CUB-1.png

Figure: Figure 9: Visualization of the most influential concept label related to different data in CUB.
Refer to caption: extracted/5901906/images/app_CUB/CUB-5.png

Figure: Figure 10: Visualization of the most influential concept label related to different data in CUB.
Refer to caption: extracted/5901906/images/app_CUB/CUB-9.png

Figure: Figure 11: Visualization of the most influential concept label related to different data in CUB.
Refer to caption: extracted/5901906/images/app_CUB/CUB-4.png

Figure: Figure 12: Visualization of the most influential concept label related to different data in OAI.
Refer to caption: extracted/5901906/images/app_OAI/OAI-1.png

## Appendix H More Related Work

Influence Function. The influence function, initially a staple in robust statistics Cook (); Cook & Weisberg (), has seen extensive adoption within machine learning since Koh & Liang () introduced it to the field. Its versatility spans various applications, including detecting mislabeled data, interpreting models, addressing model bias, and facilitating machine unlearning tasks. Notable works in machine unlearning encompass unlearning features and labels Warnecke et al. (), minimax unlearning Liu et al. (), forgetting a subset of image data for training deep neural networks Golatkar et al. (; ), graph unlearning involving nodes, edges, and features. Recent advancements, such as the LiSSA method Agarwal et al. (); Kwon et al. () and kNN-based techniques Guo et al. (), have been proposed to enhance computational efficiency. Besides, various studies have applied influence functions to interpret models across different domains, including natural language processing Han et al. () and image classification Basu et al. (), while also addressing biases in classification models Wang et al. (), word embeddings Brunet et al. (), and finetuned models Chen et al. (). Despite numerous studies on influence functions, we are the first to utilize them to construct the editable CBM. Moreover, compared to traditional neural networks, CBMs are more complicated in their influence function. Because we only need to change the predicted output in the traditional influence function. While in CBMs, we should first remove the true concept, then we need to approximate the predicted concept in order to approximate the output. Bridging the gap between the true and predicted concepts poses a significant theoretical challenge in our proof.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. How frequently are such editing updates expected to be made in a practical setting? After training the CBM, one may find some concept labeling errors and/or the need to remove some training samples or concepts. So applying the ECBM method once after training makes sense, but are there scenarios where the editing is done periodically? In situations where the distribution of the test data can change over time, it makes sense to adapt the CBM efficiently, instead of retraining. 

1. Referring to Figure 2, the F1 score decreases as more edits are made to the CBM. This could be because the experiments consider a contrived scenario where a random subset of concepts or data points are removed. In reality, if there are concept/data labeling errors or some spurious concepts, then I think the performance of the (edited) CBM is expected to improve?

1. In Table 1, all the values corresponding to the ECBM method are bold-faced. It is more useful to highlight the best (and perhaps second-best) performing methods.

1. Lines 455–456: note the errors or typos in the values. On line 517, it says the top 1010 most influential concepts. I think it should be top 10.
soundness: 3
strengths: Being able to edit or adapt a CBM model under the scenarios described in the paper is an important practical problem, with applications in domains such as medical imaging. I like how the paper formally addresses the three scenarios of concept-level, data-level, and concept label-level editing in a theoretical way. 

Addressing data removal issues in a CBM, due to mis-labeling, poisoning, or privacy issues and the adoption of influence functions for this is a novel contribution. Using ECBMs to effectively edit the model’s knowledge about specific samples, and the connection to membership inference attacks is interesting.
summary: The paper proposes editable concept bottleneck models (ECBMs), that allow a CBM model to be updated efficiently without a full retraining. They consider three different types of editing: concept-level, data-level, and concept-label level, which cover practical scenarios such as data mis-labeling, spurious concepts, and concept annotation errors. 

Using the idea of influence functions, they propose closed-form approximations to update the CBM parameters (both concept predictor and label predictor) under the three types of editing scenarios. They utilize the EK-FAC method to accelerate the computation of Hessian inverse and non-convex loss functions.
weaknesses: 1. Lots of issues with inconsistent and non-intuitive notation which makes it hard to follow the theory. I think it needs to be proof-read carefully and if possible use simpler notations. The loss functions should be defined so that the method can be understood better (are the loss functions convex?). In multiple places, it is not clear how the gradient and Hessian are defined with respect to a function, e.g., is $H_{\hat{g}}$ (last line of page 4) the Hessian of the function w.r.t the parameters of $\hat{g}$ ?

1. The theorems need some explanations to convey the implications. The paper is currently like theorem after theorem and it may be hard for readers to see the connection. In general, the presentation of ideas could be improved. 

1. In the experiments, it is not clear what is the CBM architecture and the number of model parameters? How well does the proposed method scale to a large number of parameters, given that Hessian matrices or their approximations are involved?

1. Limitations of the proposed work are not addressed.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 1
questions: - I am curious if the edited model should have similar model parameters to the retrained model, as in principle both remove the impact of certain concepts/labels/samples in different ways.  
- Why does ECBM outperform CBM-IF in terms of F1 scores? Although the authors explained that Hessian matrices might be ill-defined, ill-conditioned, or singular, how is the inversion conducted exactly in these cases?
- In Figure 5, it looks like neither the ECBM nor the Retrain model generates reasonable explanations. For example, I can not see orange upperparts or legs in either figure. Moreover,  the explanations are exactly the same for the two very different figures. Is it a bug or feature?
- As shown in equations (1) and (2), the authors seem to focus on CBM models where the concept predictor and the label predator are trained separately. What about CBM models where these two predictors are trained jointly?
- In equation (1) and (2), what is L_{C_j} and L_{Y_i}?
- In Lines 65-67, what is the difference between l and capital L?
soundness: 2
strengths: The paper addresses an interesting problem by proposing a post-hoc approach to edit concept bottleneck models, providing a effective solution. Additionally, the authors extensively examine three different editing scenarios concerning different kinds of noises in training data.
summary: This paper introduces a method to edit a trained concept bottleneck model without retraining. They use the influence function to estimate the parameters of the edited model in three scenarios: (1) removing concept labels for some samples; (2) removing some concepts for all samples; and (3) removing some data points.
weaknesses: - The related work section could be expanded. Other existing methods for model editing without retraining (if available) should be discussed.
- The experiments do not fully support the claims made in earlier sections. They claimed that the motivation for editing a model is to remove the impacts of erroneous labels, spurious concepts, and incorrect concept annotations. However, the experiment design demonstrates none of these situations, but only randomly removing concepts, concept labels, or data points. The authors might consider intentionally introducing some noise (at different levels) to their training data, and then use their model editing strategy to remove their impacts and see if the model’s performance/interpretability benefits from their methods.
- The presentation needs further improvement. There are several inconsistent notations in the paper. Some terms are used before they are defined.  Some equations are numbered but some are not. See more details in the Questions section. Some important proof/deviations are in the appendix, but not referenced appropriately in the main text (e.g., Appendix A).

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: **General questions and comments**

- lines 145-146: please reword "usually, here the map $f$ is linear" because it is a misleading claim. For CBMs, the predictor $f$ **has** to be linear, otherwise the predictions of the model would not be interpretable by construction. Using a nonlinear $f$ on top of the concepts does not lead to interpretable results.

- notation of loss functions throughout the paper: could the authors clarify why the losses change notation throughout the paper, and with what rationale? For example, in Eq. (1), $L_C$ becomes $L_{C_j}$. In Eq. (2), $L_Y$ becomes $L_{Y_i}$. In appendix D, the loss then becomes $L_{Y_l}$, and later $L_{Y_{ir}}$. Does this mean that each concept is trained with a different loss? Furthermore, it is never stated which losses are used in practice.

- lines 170-172: the derivative should be of $\hat{\theta}_{\epsilon,-z_m}$, not $\hat{\theta}$ as it does not depend on $\epsilon$.

- difference with test-time intervention. I do not think I understand the message of this paragraph. An example is provided where the trained CBM wrongly **predicts** a concept $c$. Then, a user corrects the predictor to observe what the model would have predicted had it correctly classified the concept. But the focus of the submission is to edit an existing CBM with respects to errors/changes in the **training set**. Also, the predictor $f$ is linear, so it suffices to look at the weights of the learned predictor to know how changing the predicted concept would affect the downstream prediction. I do not understand what "secondary editing based on the test data" means, as it seems to boil down to training the model on more samples observed at test time. I also do not understand what "This process extends the rectification from the data level to the model level" means.

- lines 246-249: "This is because the ..." this sentence is very confusing. It is intuitive that gradients cannot change the number of parameters of a network.

- line 254: "by inserting a zero-row vector into the $r$-th row of the matrix in the final layer of". This sentence makes several implicit assumptions about the structure of the encoder $g$. They are reasonable, but they should be stated clearly in order not to confuse readers.

- line 264: I do not understand why "the value of 00" is written several times instead of simply 0. Also, what is "MM"?

- theorem 4.3: I do not understand where the mapping phase plays a role in this statement. The gradient is taken with respects to the original predictor $\hat{g}$. The loss is computed only on the concepts not in $M$. And we are assuming each concept is predicted independently, so why would the weights in the last matrix of $g$ that correspond to the deleted concepts change? Wouldn't the gradients be 0 by definition? Then, the final predictor $\hat{g}^*_{-p_M}$ is obtained by simply removing those weights. How is this different from removing the weights to begin with from $\hat{g}$, and the editing the remaining network?

- lemma 4.4: the sentence "if the label ... . For $r \in M$ ..." is not grammatical. It is unclear whether the statement of the Lemma applies **to one concept $r \in M$** or **for all concepts in $M$**. Also, I could not find the proof of this Lemma in the Appendix. Does the statement of the Lemma hold for any loss $L_Y$? Or are there any assumptions being made here. The way I interpret the Lemma is that padding with 0 does not change the solution of the problem in the remaining entries, and I am wondering whether there might be losses where this would not be true.

- general question about two-phase approach. If I understand correctly, the need to use a two-step editing approach is due to the fact that $f$ depends on the output of $g$, and that the optimization process is over two losses: $L_C$ that only involves $g$, and $L_Y$ that involves both. I agree that in the general setting, writing out the full influence function might be hard. But $f$ will always be a linear function of the output of $g$, so wouldn't it be possible to write out the influence function for the end-to-end task, at least in the data level setting? Could the authors expand on what limitations they encountered that informed the use of a two-step approach?

- evaluation metric: could the authors expand on what f1 score they are computing? Is this the average 1-vs-all classification f1 score computed on each class in the dataset? Could the authors expand on the choice of using this metric instead of classification accuracy?

- implementation details: "at the concept-label level ...". Am I understanding correctly that, in this experiment, ground-truth labels are randomly flipped to noisy labels? This process introduces more noise rather than "correcting" any existing noise, right? I was somewhat surprised to see that this process does not seem to affect the overall F1 score significantly, neither from Table 1 not Figure 2, where the drop in performance is less than 1%. Maybe a more compelling experiment could be to noise the ground-truth annotations to a level where performance degrades significantly, and see what percentage of labels need to be restored for ECBM to obtain a predictor that is close to the one trained on the full dataset without noisy labels. This would reflect the motivating example mentioned in the main text.

- table 1: bolding the ECBM line by default is misleading because the "Retrain" method always provides a higher F1 score. The runtime for the "Retrain" method is in the order of seconds, which is quite small. Am I understanding correctly that the "Retrain" method also retrains the CNN backbone as in Koh et al. (2020)? Or is this just retraining the linear layer $f$? Could the authors expand on the data splits/optimizers/hyperparameters/number of epochs used in the experiments to guarantee reproducibility of the results?

- line 451: "We first select 1-10 most influential and 1-10 least influential" is not grammatical, please rephrase.

- Visualization: it is stated that "ECBM can accurately recognize 9 with their correct concept labels", but Fig. 5 does not seem to report ground-truth annotations, only predictions? Could the authors clearly state in Fig. 5 what the 1/0 column represents? I would rephrase claims that state "correctly identify" because there is no ground-truth notion of what should or should not be used to classify a bird. Figure 5 does support the claim that ECBM retrieves the same concepts as the Retrain method, which may or may not be correct.

- Limitations: conclusions should include a discussion of the limitations of the proposed method, and future research directions.

**Writing comments**

- all equations should be numbered.
- proofs should be explicitly referenced after theorems.
- lines 35-36: ".. for placing human-understandable concepts. In the prediction process ...". This sentence is not grammatical.
- line 80: it is unclear what erroneous or poisoned data mean.
- line 82: it is not clear what "the learning models" means.
- line 94: the sentence "and the concept for all data on model parameters" is incomplete and should be rephrased.
- lines 96-97: it is unclear what "due to their composite structure, i.e. the intermediate representation layer" means. All neural networks have intermediate representation layers, why is this a specific limitation about CBMs?
- line 128: the sentence "CBMs work on the image field also includes" is incomplete and should be rephrased.
- line 131-132: "thereby deciphering the interaction of concept models and providing an adaptive solution to concept editing" is a confusing sentence. What does it mean to decipher an interaction?
- line 161: "thus capturing the essence of the original CBMs" should be rephrased. What does it mean to capture the essence of a model? Eqs. (1) and (2) are the standard definition of a CBM, what is the "original CBM" being referred to here?
- line 189: "for certain concepts, we may opt to" is incomplete. These two sentences should be linked by a connector.
- line 193: "estimating the changes in the parameters of the retraining model holds significance" is not grammatical and should be rephrased.
- line 197-198: "the retrained concept predictor label predictor" is not grammatical and should be rephrased.
- line 208-209: "a two-stage edition approach", typo in "edition" -> "editing".
- line 256: typo in "is the subset of is the subset of $T$".
- line 311: the sentence "... due to different reasons, such as the training data involving poisoned or erroneous issues" is not grammatical.
- line 312: $G$ is overloaded, it was already used to define the Fisher information matrix.
- line 328: I do not understand the sentence "with the original loss before unlearning in equation 2".
- line 424: "we conducted experiments ... removed." is unclear and should be rephrased.
- line 455-456: typos in "11-1010", "0.0250.025".
- line 493: There is no need to report results with 6 decimals.
- line 517: "1010" is a typo.
- line 1248: "We can see that there is a *huge gap* between $\hat{f}_{-z_G}$ and $\hat{f}$". Could the authors clarify in what sense they are characterizing the gap between the two functions as *huge*?

**Minor comments**

- abstract: "elucidate the prediction process". It is unclear what prediction process this sentence is referring to.
- abstract: "affirming their adaptability within the realm of CBMs" this sentence sounds very unnatural and should be rephrased.
- line 27: "large multimodal models" (include "models").
- line 323: typo in "loss function **with** respect to".
soundness: 2
strengths: Efficient and fast editing of CBMs is an interesting problem that is valuable to the research community.

Although influence functions are well-studied, the submission proposes novel methods to apply these ideas within the context of CBMs, which is a substantial contribution.

Experimental evidence in support of the proposed methods is convincing. In particular, I found the experiment on membership attacks compelling and clear.
summary: The submission proposes to apply ideas of influence functions to edit concept bottleneck models (CBMs) at three different levels: concept label level (i.e., changing the ground-truth annotation of a concept), concept level (i.e., removing a concept completely), and data level (i.e., removing a sample from the training set). The proposed edits can be computed efficiently and they can account for the non-convexity of the optimization problem by means of the  Eigenvalue-corrected Kronecker-Factored Approximate Curvature (EK-FAC) method.

Experiments are performed on three datasets to showcase the flexibility and effectiveness of the proposed editing methods. Results show that Editable Concept Bottleneck Models (ECBMs) can retain classification performance while being significantly cheaper than retraining the full CBM from scratch.
weaknesses: The main limitation of the submission, in its current form, is presentation. I believe this limitation to be major, and a significant rewriting of the submission is needed to meet the standards of scientific writing. 

There are several incomplete and unclear sentences, and hand-wavy claims throughout the main body of the submission. This makes it hard for a reader to follow the text and appreciate the contribution, which I believe to be interesting and valuable.

I will expand on these points below, and I am looking forward to discussing with the authors to clarify my questions!

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: * Could you provide more details on how runtime efficiency scales with larger datasets or models? While your experiments show improvements over retraining on smaller datasets, it would be helpful to understand how ECBM performs with larger models like Vision-Language Models.

* You identify influential samples or concepts using influence scores. How do these rankings compare with simpler methods like analyzing weights in a linear label predictor? What additional insights do influence functions provide?
soundness: 2
strengths: * The paper introduces a novel application of influence functions to CBMs, focusing on the editability of models without full retraining. This is a significant contribution in the context of real-world applications where models need frequent updates due to evolving data or concepts.

* The mathematical formulation is rigorous and well-supported by experiments. The use of influence functions for efficient model editing is a novel approach, and the paper provides detailed derivations and approximations for each editing scenario (concept-label, concept, and data-level).

* The presentation of a handful of real-world interpretability results of their approach is appealing and easy to understand.
summary: The paper proposes a framework to acquire adaptively edited concept bottleneck models (CBMs), without actually retraining the model. 
The authors propose using influence functions to approximate the effects of three types of edits: concept-label-level, concept-level, and data-level. These edits address real-world challenges such as correcting erroneous concept annotations, adding/removing concepts, or removing problematic data due to privacy or errors.
Under each setting, they provide the closed-form approximation to measure the influence of each entity--a set of data sample's different concept labels, several concepts, and several samples--based on the influence function. 
The experimental results support the effectiveness and the interpretability of their method--matching the performance of retraining-based correction while reducing the computation overhead.
weaknesses: * Scailability: Although the authors include a relatively larger-scaled dataset (e.g., CelebA) to their evaluation, the experiments still focus on relatively small datasets compared to large-scale models like Vision-Language Models (VLMs). It remains unclear how well ECBMs would scale to larger datasets or more complex CBM architectures.
Particularly, it is critical to discuss and run analysis on the scalability of the approach, in terms of the number of parameters in a model, as it could be a key limitation of applying ECBM to real-world large-scale applications; given the need for computationally expensive Hessian computations, or of heavy linear algebra operations for the EK-FAC approximation, or of the complexity of the EK-FAC variants.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: - I am doubtful with the running time in Table 1. How the running time can be less 1 minute even for retraining method.
soundness: 3
strengths: - The notation and problem definition is clear.
- The paper is easy to follow.
- The effect of influence function in Section 5.3 is insightful.
- The result visualization is clear.
summary: This paper delineates 3 levels of editable settings which are concept-label-level, concept-level and data-level. Based on the three settings, the author proposes a concept bottleneck model (CBM) to remove the data or concept influence without retraining based on influence function. Furthermore, the author proposes Editable Concept Bottleneck Model (ECBM) and the streamlined version EK-FAC. Moreover, the author provides the mathematical proof for the three settings and the methods. The extensive experiments shows the effectiveness of the method.
weaknesses: - The baseline methods are not complete. For example, there are still some methods for EBM like, probability EBM [1], etc.
- The influence function is only suitable for convex function. I wonder how the author applies it on the non-convex function, like deep neural network.
- The approximate equal sign is mixed with the equal sign, which is not rigorous, like in Theorem 4.7. Please correct me if I have some mis-understanding.


[1] Kim, Eunji, et al. "Probabilistic concept bottleneck models." arXiv preprint arXiv:2306.01574 (2023).

Meta-review: This work studies an approach, based on approximate influence functions, to edit Concept Bottleneck Models (CBM) to accommodate for three levels of interventions: modifying a (potentially erroneous) concept annotated for a given sample, removing a concept from the model altogether, or removing the influence of a training sample. The approach is empirically effective while circumventing the need for retraining the CMB from scratch. 

**Strengths**
* Innovative approach, and nice trick, of influence functions to edit CMBs.
* Experiments support the claims in the paper.
* The three levels of edits are interesting.

**Weaknesses**
* The annotation often is unclear, making it more difficult to understand the methodology fully.
* Lack of clarity in writing and typos.
* Despite the stress on the rigor of the (approximate) mathematical formulations, no guarantees on the level of approximations are given, nor evaluated empirically. Additional assumptions - like the independence of the network layers - are unrealistic and only mentioned in passing.

**Summary**

This paper received profuse discussions among reviewers and authors, resulting in a more comprehensive picture of the paper from the former and an improvement of the quality of the manuscript by the latter. All reviewers agree on the relevant nature of the problem, and on the innovative solution. At the same time, several reviewers noted that the quality of the presentation is far from where they would like it to be. After reviewing all of the comments, questions, and responses from the authors, I turned to review the paper myself. Unfortunately, I concur with the reviewers nBLD, KdDJ, and oeij, that the presentation of the content is still confusing and unclear. I won't list all of my concerns here, but a non-exhaustive list includes:
* line 158: ".. different columns as a vector $g^j(x_i)$", whereas notation indicates that this should be a scalar.
* line 167: loss L is undefined;
* The notation $m$ is used to denote the index of a sample as well as the dimension of the data,
* line 207: The sentence beginning "Observing.." doesn't have a subject nor predicate.

Moreover, while verifying the modifications done by the authors, I noted that they added a Limitations and Broader Impacts section, as suggested by nBLD. However, this section only has 2 simple sentences on the limitations of the approximation used; the rest of it is simply a repeated verbatim copy of the abstract. This is, naturally, not acceptable.

As a result, given the strengths and weaknesses above, as well as revising the discussions and revision, I cannot recommend acceptance. I encourage the authors to continue improving on this work to provide the best presentation for their innovative and nice ideas.

Decision: Reject