Title: Concept Distillation: Leveraging Human-Centered Explanations for Model Improvement
Venue: NeurIPS 2023
Accepted: yes
Keywords: Human Centered Concepts

ML interpretability

XAI based Model Improvement

Debiasing
Abstract: Humans use abstract *concepts* for understanding instead of hard features. Recent interpretability research has focused on human-centered concept explanations of neural networks. Concept Activation Vectors (CAVs) estimate a model's sensitivity and possible biases to a given concept. We extend CAVs from post-hoc analysis to ante-hoc training to reduce model bias through fine-tuning using an additional *Concept Loss*. Concepts are defined on the final layer of the network in the past. We generalize it to intermediate layers, including the last convolution layer. We also introduce *Concept Distillation*, a method to define rich and effective concepts using a pre-trained knowledgeable model as the teacher. Our method can sensitize or desensitize a model towards concepts. We show applications of concept-sensitive training to debias several classification problems. We also show a way to induce prior knowledge into a reconstruction problem. We show that concept-sensitive training can improve model interpretability, reduce biases, and induce prior knowledge.

Concept Distillation: Leveraging Human-Centered Explanations for Model Improvement

Abstract

Humans use abstract concepts for understanding instead of hard features. Recent interpretability research has focused on human-centered concept explanations of neural networks. Concept Activation Vectors (CAVs) estimate a model's sensitivity and possible biases to a given concept. In this paper, we extend CA Vs from posthoc analysis to ante-hoc training in order to reduce model bias through fine-tuning using an additional Concept Loss . Concepts were defined on the final layer of the network in the past. We generalize it to intermediate layers using class prototypes. This facilitates class learing in the last convolution layer which is known to be most informative. We also introduce Concept Distillation t create richer concepts using a pre-trained knowledgeable model as the teacher. Our method can sensitize or desensitize a model towards concepts. We show applications of concept-sensitive training to debias several classification problems. We also use concepts to induce prior knowledge into IID, a reconstruction problem. Concept-sensitive training can improve model interpretability, reduce biases, and induce prior knowledge. Please visit https://avani17101.github.io/Concept-Distilllation/ for code and more details.

1 Introduction

EXplainable Artificial Intelligence (XAI) methods are useful to understand a trained model's behavior [74]. They open the black box of Deep Neural Networks (DNNs) to enable post-training identification of unintended correlations or biases using similarity scores or saliency maps. Humans, however, think in terms of abstract concepts , defined as groupings of similar entities [27]. Recent efforts in XAI have focused on concept-based model explanations to make them more aligned with human cognition. Kim et al. [32] introduce Concept Activation Vectors (CAVs) using a concept classifier hyperplane to quantify the importance given by the model to a particular concept. For instance, CAVs can determine the model's sensitivity on 'striped-ness' or 'dotted-ness' to classify Zebra or Cheetah using user-provided concept samples. They measure the concept sensitivity of the model's final layer prediction with respect to intermediate layer activations (outputs). Such post-hoc analysis can evaluate the transparency, accountability, and reliability of a learned model [14] and can identify biases or unintended correlations acquired by the models via shortcut learning [32, 5].

The question we ask in this paper is: If CA Vs can identify and quantify sensitivity to concepts, can they also be used to improve the model? Can we learn less biased and more human-centered models? In this paper, we extend CAVs to ante-hoc model improvement through a novel concept loss to desensitize/sensitize against concepts. We also leverage the broader conceptual knowledge of a large pre-trained model as a teacher in a concept distillation framework for it.

Biased

Student

Learned Shortcut:

if (texture==n)

predict (class=k)

Random textures

Concept

Teacher

CAV Learning &

Concept sets

random set

Texture concept

Trainset

...

Class consistent textures

Testset

...

Local XAI :

sample specific

Global XAI :

class common

XAI

Few-shot: some test

distribution

information

Domain Adaptation:

Full test distribution

information

Post-hoc

Explanations

Ante-hoc

Improvement

Zero-shot: w/o test

distribution

information

XAI has been used for ante-hoc model improvement during training [35, 30, 4]. They typically make fundamental changes to the model architecture or need significant concept supervision, making extensions to other applications difficult. For example, Koh et al. [35] condition the model by first predicting the underlying concept and then using it for class prediction. Our method can sensitize or desensitize the model to user-defined concepts without modifications to the architecture or direct supervision.

Our approach relies on the sensitivity of a trained model to human-specified concepts. We want the model to be sensitive to relevant concepts and indifferent to others. For instance, a cow classifier might be focusing excessively on the grass associated with cow images. If we can estimate the sensitivities of the classifier to different concepts, we can steer it away from irrelevant concepts. We do that using a concept loss term L C and fine-tuning the trained base model with it. Since the base models could be small and biased in different ways, we use concept distillation using a large, pre-trained teacher model that understands common concepts better.

We also extend concepts to work effectively on intermediate layers of the model, where the sensitivity is more pronounced. Kim et al. [32] measure the final layer's sensitivity to any intermediate layer outputs. They ask the question: if any changes in activations are done in the intermediate layer, what is its effect on the final layer prediction? They used the final layer's loss/logit to estimate the model sensitivity as their interest was to study concept sensitivities for interpretable model prediction. We, on the other hand, aim to fine-tune a model by (de)sensitizing it towards a given concept which may be strongest in another layer [3]. Thus, it is crucial for us to measure the sensitivity in any layer by evaluating the effect of the changes in activations in one intermediate layer on another. We employ prototypes or average class representations in that layer for this purpose. Prototypes are estimated by clustering the class sample activations [12, 72, 41, 49, 48, 63, 43]. Our method, thus, allows intervention in any layer.

In this paper, we present a simple but powerful framework for model improvement using concept loss and concept distillation for a user-given concept defined in any layer of the network. We leverage ideas from post-hoc global explanation techniques and use them in an ante-hoc setting by encoding concepts as CAVs via a teacher model. Our method also admits sample-specific explanations via a local loss [52] along with the global concepts whenever possible. We improve state-of-the-art on classification problems like ColorMNIST and DecoyMNIST [51, 16, 52, 5], resulting in improved accuracies and generalization. We introduce and benchmark on a new and more challenging TextureMNIST dataset with texture bias associated with digits. We demonstrate concept distillation on two applications: (i) debiasing extreme biases on classification problems involving synthetic MNIST datasets [42, 16] and complex and sensitive agevs .-gender bias in the real-world gender classification on BFFHQ dataset [34] and (ii) prior induction by infusing domain knowledge in the reconstruction problem of Intrinsic Image Decomposition (IID) [38] by measuring and improving disentanglement of albedo and shading concepts. To summarize, we:

Extend CAVs from post-hoc explanations to ante-hoc model improvement method to sensitize/desensitize models on specific concepts without changing the base architecture.

Extend the model CAV sensitivity calculation from only final layer to any layer and enhance it by making it more global using prototypes.

Introduce concept distillation to exploit the inherent knowledge of large pretrained models as a teacher in concept definition.

Benchmark results on standard biased MNIST datasets and on a challenging TextureMNIST dataset that we introduce.

Show application on a severely biased classification problem involving age bias.

Show application beyond classification to the challenging multi-branch Intrinsic Image Decomposition problem by inducing human-centered concepts as priors. To the best of our knowledge, this is the first foray of concept-based techniques into non-classification problems.

2 Related Work

The related literature is categorized in Fig. 2. Post-hoc (after training) explainability methods include Activation Maps Visualization [59], Saliency Estimation [54], Model Simplification [69], Model Perturbation [17], Adversarial Exemplar Analysis [22], etc . See recent surveys for a comprehensive discussion [74, 15, 45, 65, 2]. Different concept-based interpretability methods are surveyed by Hitzler and Sarker [27], Schwalbe [58], Holmberg et al. [28].

The ante-hoc model improvement techniques can be divided into zero-shot [70], few-shot [67], and multi-shot (domain-adaptation [66]) categories based on the amount of test distribution needed. In zero-shot training, the model never sees the distribution of test samples [70], while in few-shot, the model has access to some examples from the distribution of the test set [67]. Our method works with abstract concept sets (different than any test sample), being essentially a zero-shot method but can take advantage of few-shot examples, if available, as shown in results on BFFHQ subsection 4.1. We restrict the discussion to the most relevant model improvement methods.

Ross et al. [52] penalize a model if it does not prefer the Right answers for Right Reasons (RRR) by using explanations as constraints. EG Erion et al. [16] augment gradients with a penalizing term to match with user-provided binary annotations. More details are available in relevant surveys [18, 68, 24, 8].

Based on the scope of explanation, the existing XAI methods can be divided into global and local methods [74]. Global methods [32, 20] provide explanations that are true for all samples of a class ( e.g . 'stripiness' concept is focused for the zebra class or 'red color' is focused on the prediction of zero) while local methods [1, 59, 60, 53] explain each of samples individually often indicating regions or patches in the image which lead the model to prediction (example, this particular patch (red) in this image was focused the most for prediction of zero). Kim et al. [32] quantify concept sensitivity using CA Vs followed by subsequent works [62, 73, 57]. Kim et al. [32] sample sensitivity is local (sample specific) while they aggregate class samples for class sample sensitivity estimation, which makes their method global (across classes). We enhance their local sample sensitivity to the global level via using prototypes that capture class-wide characteristics by definition [12].

Only a few concept-oriented deep learning methods train neural networks with the help of concepts [56, 55]. Concept Bottleneck Models (CBMs) [35], Concept-based Model Extraction (CME) [30], and Self-Explaining Neural Networks (SENNs) [4] predict concepts from inputs and use them to infer the output. CBMs [35] require concept supervision, but CME [30] can train in a partially supervised manner to extract concepts combining multiple layers. SENN [4] learns interpretable basis concepts by approximating a model with a linear classifier. These methods require architectural modifications of the base model to predict concepts. Their primary goal is interpretability, which differs from our model improvement goal using specific concepts without architectural changes. Closest to our work is ClArC [5], which leverages CAVs to manipulate model activations using a linear transformation to remove artifacts from the final results. Unlike them, our method trains the DNN to be sensitive to any specific concept, focusing on improving generalizability for multiple applications.

DFA [40] and EnD [64] use bias-conflicting or out-of-distribution (OOD) samples for debiasing using few-shot generic debiasing. DFA uses a small set of adversarial samples and separate encoders for learning disentangled features for intrinsic and biased attributes. EnD uses a regularization strategy to prevent the learning of unwanted biases by inserting an information bottleneck using pre-known bias types. It is to be noted that both DFA and EnD are neither zero-shot nor interpretability-based methods. In comparison, our method works in both known bias settings using abstract user-provided

classksamplex

f

f(

Lp

f(

f

V

f (

f(

decision

Train

Test

Concepts

Random

Concept

concept sets and unknown bias settings using the bias-conflicting samples. Our Concept Distillation method is a concept sensitive training method for induction of concepts into the model, with debiasing being an application. Intrinsic Image Decomposition (IID) [11, 19] involves decomposing an image into its constituent Reflectance ( R ) and Shading ( S ) components [38, 46] which are supposed to be disentangled [46, 23].

We use CAVs for inducing R and S priors in a pre-trained SOTA IID framework [44] with improved performance. To the best of our knowledge, we are the first to introduce prototypes for CAV sensitivity enhancement. However, prototypes have been used in the interpretability literature before to capture class-level characteristics [33, 71, 31] and also have been used as pseudo-class labels before [12, 72, 41, 49, 48, 63, 43]. Some recent approaches [29, 61] use generative models to generate bias-conflicting samples ( e.g . other colored zeros in ColorMNIST) and train the model on them to remove bias. Specifically, Jain et al. [29] use SVMs to find directions of bias in a shared image and language space of CLIP [50] and use Dall-E on discovered keywords of bias to generate bias-conflicting samples and train the model on them. Song et al. [61] use StyleGAN to generate bias-conflicting samples. Training on bias-conflicting samples might not always be feasible due to higher annotation and computation costs.

One significant difference between such methods [29, 61] is that they propose data augmentation as a debiasing strategy, whereas we directly manipulate the gradient vectors, which is more interpretable. Moayeri et al. [47] map the activation spaces of two models using the CLIP latent space similarity. Due to the generality of the CLIP latent space, this approach is helpful to encode certain concepts like 'cat, dog, man,' but it is not clear how it will work on abstract concepts with ambiguous definitions like 'shading' and 'reflectance' as seen in the IID problem described above.

3 Concept Guidance and Concept Distillation

Concepts have been used to explain model behavior in a post-hoc manner in the past. Response to abstract concepts can also demonstrate the model's intrinsic preferences, biases, etc. Can we use concepts to guide the behavior of a trained base model in desirable ways in an ante-hoc manner? We describe a method to add a concept loss to achieve this. We also present concept distillation as a way to take advantage of large foundational models with more exposure to a wide variety of images.

3.1 Concept Sensitivity to Concept Loss

Building on Kim et al. [32], we represent a concept C using a Concept Activation Vector (CAV) as the normal v l C to a linear decision boundary between concept samples C from others C ′ in a layer l of the model's activation space (Fig. 3). The model's sensitivity S C,l ( x ) = ∇ L o ( f l ( x )) · v l C to C is boundary

the directional derivative of final layer loss L o for samples x along v l C [32]. The sensitivity score quantifies the concept's influence on the model's prediction. A high sensitivity for color concept may indicate a color bias in the model.

These scores were used for post-hoc analysis before ([32]). We use them ante-hoc to desensitize or sensitize the base model to concepts by perturbing it away from or towards the CA V direction (Fig. 3). The gradient of loss indicates the direction of maximum change. Nudging the gradients away from the CAV direction encourages the model to be less sensitive to the concept and vice versa. For this, we define a concept loss L C as the absolute cosine of the angle between the loss gradient and the CAV direction

which is minimized when the CAV lies on the classifier hyperplane (Fig. 3). We use the absolute value to not introduce the opposite bias by pushing the loss gradient in the opposite direction. A loss of (1 -L C ( x )) will sensitize the model to C . We fine-tune the trained base model for a few epochs using a total loss of L = L o + λL C to desensitize it to concept C , where L o the base model loss.

3.2 Concepts using Prototypes

Concepts can be present in any layer l , though the above discussion focuses on the sensitivity calculation of the final layer using model loss L o . The final convolutional layer is proven to learn concepts better than other layers [3]. We can estimate the concept sensitivity of any layer using a loss for that layer. How do we get a loss for an intermediate layer, as no ground truth is available for it?

Class prototypes have been used as pseudo-labels in intermediate layers before [12, 72, 41]. We adapt prototypes to define a loss in intermediate layers. Let f l ( x ) be the activation of layer l for sample x . We group the f l ( x ) values of the samples from each class into K clusters. The cluster centers P i together form the prototype for that class. We then define prototype loss for each training sample x using the prototype corresponding to its class as

We use L p in place of L o in Eq. 1 to define the concept loss in layer l . The prototype loss facilitates the use of intermediate layers for concept (de)sensitization. Experiments reported in Tab. 1 confirm the effectiveness of L p . Prototypes also capture sample sensitivity at a global level using all samples of a class beyond the sample-specific levels. We update the prototypes after a few iterations as the activation space evolves. If P n is the prototype at Step n and P c the cluster centres using the current f l ( x ) values, the next prototype is P n +1 k = (1 -α ) P n k + αP c k for each cluster k .

3.3 Concept Distillation using a teacher

Concepts are learned from the base model in the above formulation. Base models may have wrong concept associations due to their training bias or limited exposure to concepts. Can we alleviate this problem using a larger model that has seen vast amounts of data as a teacher in a distillation framework?

We use the DINO [13], a self-supervised model trained on a large number of images, as the teacher and the base model as the student for concept distillation. The teacher and student models typically have different activation spaces. We map the teacher space to the student space before concept learning. The mapping uses an autoencoder [26] consisting of an encoder E M and a decoder D M (Fig. 5). As a first step, the autoencoder (Fig. 5) is trained to minimize the loss L D M + L E M . L D M is the pixel-wise L2 loss between the original ( f t ) and decoded ( ˆ f t ) teacher activations and L E M is the pixel-wise L2 loss between the mapped teacher ( ˆ f s ) and the student ( f s ) activations. The mapping is learned over the concept set of images C and C ′ . See the dashed purple lines in Fig. 5.

Next, we learn the CAVs in the distilled teacher space ˆ f s , keeping the teacher, student, and mapping modules fixed. This is computationally light as only a few (50-150) concept set images are involved. The learned CAV is used in concept loss given in Eq. 1. Please note that E M is used only to align the two spaces and can be a small capacity encoder, even a single layer trained in a few epochs.

f(    )

Pre-trained

Teacher

Student Classifier

E M

Classification

Head

logits

GT labels

Conventional

GT Loss ( L o  )

LC

C

Training Set D

f l

(    )

Concept

Distillation Loss

( L c  )

...

Proto P k

Prototype

loss (L p )

Mapping and CAV learning in

Teacher

Concept

Distillation in

Student

DM

f(    )

f(    )

f(    )

f(    )

f(    )

f(    )

C'

layer l

2. Mapping and CAV learning

3. Concept Learning in Student

1. Mapping autoencoder training

C'

decision

boundary

f(    ) C

class k sample x

Mapping

autoencoder

4. Testing

Algorithm 1 Concept Distillation Pipeline

Given: A pretrained student model which is to be fine-tuned for concept (de)sensitization and a pretrained teacher model which will be used for concept distillation. Known Concepts C and negative counterparts (or random samples) C ′ , student training dataset D , and student bottleneck layer l , #iterations to recalculate CA Vs cav_update_frequency, #iterations to update prototypes proto_update_frequency.

1: Concept Distillation :

2: For all class samples in D , estimate class prototypes P 0 k ∈{ 0 ,K } with K-means.

3: Current iteration n = 0 , initial prototypes P 0 = P c .

4: While not converge do :

5: If n = 0 or (update_cavs and n % cav_update_frequency = 0) then :

6:

7:

8: Learn the mapping module as autoencoders E M and D M

Learn Mapping module :

Forward pass x ∈ C ∪ C ′ from Teacher and Student to get their concept activations f t and f s . .

9: CAV learning in mapped teacher's space :

10: v l C learned by binary linear classifier as normal to decision boundary of E M ( f t ( x )) for x ∈ C vs E M ( f t ( x ′ )) for x ′ ∈ C ′ .

̸

11: If n % proto_update_frequency = 0 and n = 0 then :

12: Estimate new class prototypes P c k ∈{ 0 ,K } with K-means.

13: Weighted Proto-type P n +1 k = (1 -α ) P n k + αP c k .

14: Else :

15: P n +1 k = P n k

16: Train student with loss L C + L o .

17: n + = 1 .

4 Experiments

We demonstrate the impact of the concept distillation method on the debiasing of classification problems as well as improving the results of a real-world reconstruction problem. Classification experiments cover two categories of Fig. 2: the zero-shot scenario, where no unbiased data is seen, and the few-shot scenario, which sees a few unbiased data samples. We use pre-trained DINO ViT-B8 transformer [13] * as the teacher. We use the last convolution layer of the student model for concept distillation. The mapping module ( < 120 K parameters, < 10 MB), CAV estimations (logistic regression, < 1 MB), and prototype calculations all complete within a few iterations of training, taking 15-30 secs on a single 12GB Nvidia 1080 Ti GPU * . Our framework is computationally light. In our experimentation reported below, we fix the CA Vs as initial CA Vs and observe similar results as

varying them. Also, our concept sensitive fine-tuning method converges quickly, and hence updating CAVs in every few iterations does not help * . Where relevant, we report average accuracy over five training runs with random seeds ( ± indicates variance).

4.1 Concept Sensitive Debiasings

We show results on two standard biased datasets (ColorMNIST [42] and DecoyMNIST [16]) and introduce a more challenging TextureMNIST dataset for quantitative evaluations. We also experimented on a real-world gender classification dataset BFFHQ [34] that is biased based on age. We compare with other state-of-the-art interpretable model improvement methods [51, 52, 16, 64, 40, 5]. Fig. 4 summarizes the datasets, their biases, and the concept sets used.

Poisoned MNIST Datasets: ColorMNIST [42] has MNIST digit classes mapped to a particular color in the training set [51]. The colors are reversed in the test set. The baseline CNN model ( Base ) trained on ColorMNIST gets 0% accuracy on the 100% poisoned test set, indicating that the model learned color shortcuts instead of digit shapes. We debias the model using a concept loss for color using color patches vs . random shapes (negative concept) to estimate CAV for color (Fig. 4). We show results comparison on the usage of teacher and prototypes component of our method in Tab. 1. As can be seen from Tab. 1, our method of using intermediate layer sensitivity via prototypes as described in subsection 3.2 yields better results. Similarly, usage of teacher (described in subsection 3.3) facilitates better student concept learning (also shown in CA V comparisons in supplementary). Our concept sensitive training not only improves student accuracy but observes evident reduction in TCAV scores [32] of bias concept as seen from Tab. 2.

Tab. 3 compares our results with the best zero-shot interpretability based methods, i.e ., CDEP [51], RRR [52], and EG [16]. Our method improves the accuracy from 31% by CDEP to 41.83%, and further to 50.93% additionally with the local explanation loss from [52] * .

D

1.0

0.8

0.6

0.4

0.2

0.0

Base

CIArC

Ours

InputImage

Base

Ours

CDEP

InputImage

Base

Ours

CDEP

Figure 6: Left: Comparison of our method and ClArC [5] on 20% biased ColorMNIST. Right: GradCAM [59] visualizations (more red, more important) on (i) TextureMNIST (middle image) (ii) Extreme case when the color bias is in the background (bottom) instead of foreground (top). Our model focuses on the shape more than CDEP (which is more blue in the foreground).

GradCAM [59] visualizations in Fig. 6 show that our trained model focuses on highly relevant regions. Fig. 6 left compares our class-wise accuracy on 20% biased ColorMNIST with ClArC [5], a few-shot global method that uses CAVs for artifact removal. (ClArC results are directly reported from their main paper due to the unavailability of the public codebase.) ClArC learns separate CAVs for each class by separating the biased color digit images ( e.g . red zeros) from the unbiased images ( e.g . zeros in all colors). A feature space linear transformation is then applied to move input sample activations away/towards the learned CAV direction. Their class-specific CA Vs definition can not be generalized to test sets with multiple classes. This results in a higher test accuracy variance as seen in Fig. 6. Further, as they learn CA Vs in the biased model's activation space, a common concept set cannot be used across classes. DecoyMNIST [16] has class indicative gray patches on image boundary that biased models learn instead of the shape. We define concept sets as gray patches vs . random set (Fig. 4) and report them in Tab. 3 second row. All methods perform comparably on this task as the introduced bias does not directly corrupt the class intrinsic attributes (shape or color), making it easy to debias. TextureMNIST is a more challenging dataset that we have created for further research in the area * . Being an amalgam of colors and patterns, textures are more challenging as a biasing attribute. Our method improves the performance while others struggle on this task (Tab. 3 last row). Fig. 6 shows that our method can focus on the right aspects for this task.

Generalization to multiple biasing situations is another important aspect. Tab. 4 shows the performance on different types of biases by differently debiased models. Our method performs better than CDEP in all situations. Interestingly, our texture-debiased model performs better on color biases than CDEP debiased on the color bias! We believe this is because textures inherently contain color information, which our method can leverage efficiently. Our method effectively addresses an extreme case of color bias introduced in the background of ColorMNIST, a dataset originally biased in foreground color. By introducing the same color bias to the background and evaluating our ColorMNIST debiased models, we test the capability of our approach. In an ideal scenario where color debiasing is accurate, the model should prioritize shape over color, a result achieved by our method but not by CDEP as shown in Fig. 6.

BFFHQ [34] dataset is used for the gender classification problem. It consists of images of young women and old men. The model learns entangled age attributes along with gender and gets wrong predictions on the reversed test set i.e . old women and young men. We use the bias conflicting samples by Lee et al. [40] - specifically, old women vs . women and young men vs . men - as class-specific concept sets.We compare against recent debiasing methods EnD [64] and DFA [40]. Tab. 5 shows our method getting a comparable accuracy of 63%. We also tried other concept set combinations: (i) old vs . young (where both young and old concepts should not affect) - → 62.8% accuracy, (ii) old vs . mix (men and women of all ages) and young vs . mix - → 62.8% accuracy, (iii) old vs . random set (consisting of random internet images from [32]) and young vs . random - → 63% accuracy. These experiments indicate the stability of our method to the concept set definitions. This proves our method can also work with bias-conflicting samples [40] and does not necessarily require concept sets (Tab. 5). We also experimented by training class-wise (for all women removing young bias

followed by men removing old bias and vice versa) vs training for all classes together (both men, women removing age bias as described above) and observed similar results, suggesting that our concept sensitive training is robust to class-wise or all class agnostic training. Local interpretable improvement methods like CDEP, RRR, and EG are not reported here as they cannot capture complex concepts like age due to their pixel-wise loss or rule-based nature.

Discussion No Distillation Case: We additionally show our method without teacher (Our Concept loss with CAVs learned directly in student) in all experiments as "Ours w/o Teacher" and find inferior performance when compared to Our method with Distillation. Bias in Teacher: We check the effect of bias in teacher by training it with varying fractions of OOD samples to bias them. Specifically, we use the same student architecture for the teacher. The teacher is trained on the ColorMNIST dataset with 5, 10, 25, 50, 75, and 90% biased color samples in the trainset ( e.g . k % bias indicates k % red zeros). The resulting concept-distilled system is then tested on the standard 100% reverse color setting of ColorMNIST. Tab. 6 shows that concept distillation improves performance even with high teacher bias, though accuracy decreases with the increasing bias in teacher. Apparently, 100% bias in teacher in this setting of teacher with same architecture as the student is essentially CA V learning in same model case (No Distil). Here, the improvements are due to prototypes, and as can be seen, there is a slight degradation in performance of 100% bias vs No Distil (23.63 vs 26.97). This can be attributed to an error due to the mapping module.

4.2 Prior Knowledge Induction

Intrinsic Image Decomposition (IID) is an inverse rendering problem [7] based on the retinex theory [39], which suggests that an image I can be divided into Reflectance R and Shading S components such that I = R · S at each pixel. Here R represents the material color or albedo, and S represents the scene illumination at a point. As per definition, R and S are disentangled , with R invariant to illumination and S invariant to albedo. As good ground truth for IID is hard to create, IID algorithms are evaluated on synthetic data or using some sparse manual annotations [9, 36]. Gupta et al. [23] use CAV-based sensitivity scores to measure the disentanglement of R and S . They create concept sets of (i) varying albedo : wherein the material color of the scene is varied (ii) varying illumination : where illumination is varied. From the definition of IID, Reflectance shall only be affected by albedo variations and not by illumination variations and vice-versa for Shading. They defined Concept Sensitivity Metric (CSM) to measure R -S disentanglement and evaluate IID methods post-hoc. Using our concept distillation framework, we extend their post-hoc quality evaluation method to ante-hoc training of the IID network to increase disentanglement between R and S . We train in different experimental settings wherein we only train the R branch ( R only) and both R and S branches together ( R & S ) with our concept loss in addition to the original loss [44] and report results in Tab. 7.

We choose the state-of-the-art CGIID [44] network as the baseline. The last layer of both R and S branches is used for concept distillation, and hence no prototypes are used. Following Li and Snavely [44], we fine-tune over the CGIntrinsics [44], IIW [10], and SAW [37] datasets while we report results

Input Images

CGIID S

Ours R

Ours S

GT S

GT R

CGIID R

over ARAP dataset [11], which consists of realistic synthetic images. We also train the baseline model for additional epochs to get CGIID++ for a fair comparison. Tab. 7 shows different measures to compare two variations of our method with CGIID and CGIID++ on the ARAP dataset [11]. We see improvements in MSE and SSIM scores which compare dense-pixel wise correspondences CSM scores [23] which evaluate R-S disentanglement. Specifically, CSM S measures albedo invariance of S and CSM R measures illumination invariance of R predictions. We report CSM scores in two concept set settings of Synthetic vs Real-World from Gupta et al. [23]. The improvement in MSE and SSIM scores appear minor quantitatively, but our method performs significantly better in terms of CSM scores. It also observes superior performance qualitatively, as seen in (Fig. 7) * . Our R outputs are less sensitive to illumination, with most of the illumination effects captured in S .

Discussion and Limitations: Our concept distillation framework can work on different classification and reconstruction problems, as we demonstrated. Our method can work well in both zero-shot (with concept sets) and few-shot (with bias-conflicting samples) scenarios. Bias-conflicting samples may not be easy to obtain for many real-world applications. Our required user-provided concept samples can incur annotation costs, though concept samples are usually easier to obtain than bias-conflicting samples. When neither bias-conflicting samples nor user-provided concept sets are available, concept discovery methods like ACE [21] could be used. ACE discovers concepts used by the model by image super-pixel clustering. Automatic bias detection methods like Bahadori and Heckerman [6] can be used to discover or synthesize bias-conflicting samples for our method. Our method can also be used to induce prior knowledge into complex reconstruction/generation problems, as we demonstrated with IID. The dependence on the teacher for conceptual knowledge could be another drawback of our method, as with all distillation frameworks [25].

Societal Impact: Our method can be used to reduce or increase the bias by the appropriate use of the loss L C . Like all debiasing methods, it thus has the potential to be misused to introduce calculated bias into the models.

5 Conclusions

We presented a concept distillation framework that can leverage human-centered explanations and the conceptual knowledge of a pre-trained teacher to distill explanations into a student model. Our method can desensitize ML models to selected concepts by perturbing the activations away from the CAV direction without modifying its underlying architecture. We presented results on multiple classification problems. We also showed how prior knowledge can be induced into the real-world IID problem. In future, we would like to extend our work to exploit automatic bias detection and concept-set definition. Our approach also has potential to be applied to domain generalization and multitask learning problems.

Acknowledgements: Wethank Prof. Vineeth Balasubramanian of IIT Hyderabad for useful insights and discussion. We would also like to thank the four anonymous reviewers of Neurips, 2023 for detailed discussions and comments which helped us improve the paper.

References

C. Aditya, S. Anirban, D. Abhishek, and H. Prantik. Grad-cam++: improved visual explanations for deep convolutional networks. arxiv 2018. arXiv preprint arXiv:1710.11063 , 2018.

N. Akhtar. A survey of explainable ai in deep visual modeling: Methods and metrics. ArXiv , abs/2301.13445, 2023.

A. Akula, S. Wang, and S.-C. Zhu. Cocox: Generating conceptual and counterfactual explanations via fault-lines. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 2594-2601, 2020.

D. Alvarez Melis and T. Jaakkola. Towards robust interpretability with self-explaining neural networks. Adv. Neural Inform. Process. Syst. , 2018.

C. J. Anders, L. Weber, D. Neumann, W. Samek, K.-R. Müller, and S. Lapuschkin. Finding and removing clever hans: Using explanation methods to debug and improve deep models. Information Fusion , 77: 261-295, 2022.

M. T. Bahadori and D. E. Heckerman. Debiasing concept-based explanations with causal analysis. arXiv preprint arXiv:2007.11500 , 2020.

H. G. Barrow and J. M. Tenenbaum. Recovering intrinsic scene characteristics from images. 1978.

K. Beckh, S. Müller, M. Jakobs, V. Toborek, H. Tan, R. Fischer, P. Welke, S. Houben, and L. von Rueden. Sok: Harnessing prior knowledge for explainable machine learning: An overview. In First IEEE Conference on Secure and Trustworthy Machine Learning , 2023.

S. Bell, K. Bala, and N. Snavely. Intrinsic images in the wild. ACM Transactions on Graphics (TOG) , 33:1 - 12, 2014.

S. Bell, K. Bala, and N. Snavely. Intrinsic images in the wild. ACM Transactions on Graphics (TOG) , 33 (4):1-12, 2014.

N. Bonneel, B. Kovacs, S. Paris, and K. Bala. Intrinsic Decompositions for Image Editing. Computer Graphics Forum (Eurographics State of The Art Report) , 2017.

M. Caron, P. Bojanowski, A. Joulin, and M. Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European conference on computer vision (ECCV) , pages 132-149, 2018.

M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650-9660, 2021.

D. T. Chang. Concept-oriented deep learning: Generative concept representations. arXiv preprint arXiv:1811.06622 , 2018.

M. Du, N. Liu, and X. Hu. Techniques for interpretable machine learning. Communications of the ACM , 63(1):68-77, 2019.

G. Erion, J. D. Janizek, P. Sturmfels, S. M. Lundberg, and S.-I. Lee. Improving performance of deep learning models with axiomatic attribution priors and expected gradients. Nature machine intelligence , 3 (7):620-631, 2021.

R. C. Fong and A. Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In Proceedings of the IEEE international conference on computer vision , pages 3429-3437, 2017.

Y. Gao, S. Gu, J. Jiang, S. R. Hong, D. Yu, and L. Zhao. Going beyond xai: A systematic survey for explanation-guided learning. arXiv preprint arXiv:2212.03954 , 2022.

E. Garces, C. Rodriguez-Pardo, D. Casas, and J. Lopez-Moreno. A survey on intrinsic images: Delving deep into lambert and beyond. Int. J. Comput. Vision , 130(3):836-868, 2022.

A. Ghandeharioun, B. Kim, C.-L. Li, B. Jou, B. Eoff, and R. W. Picard. Dissect: Disentangled simultaneous explanations via concept traversals. arXiv preprint arXiv:2105.15164 , 2021.

A. Ghorbani, J. Wexler, J. Zou, and B. Kim. Towards automatic concept-based explanations. arXiv preprint arXiv:1902.03129 , 2019.

I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 , 2014.

A. Gupta, S. Saini, and P. J. Narayanan. Interpreting intrinsic image decomposition using concept activations. In Proceedings of the Thirteenth Indian Conference on Computer Vision, Graphics and Image Processing , ICVGIP '22, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9781450398220. doi: 10.1145/3571600.3571603.

P. Hase and M. Bansal. When can models learn from explanations? a formal framework for understanding the roles of explanation data. arXiv preprint arXiv:2102.02201 , 2021.

G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015.

G. E. Hinton and R. R. Salakhutdinov. Reducing the dimensionality of data with neural networks. science , 313(5786):504-507, 2006.

P. Hitzler and M. Sarker. Human-centered concept explanations for neural networks. Neuro-Symbolic Artificial Intelligence: The State of the Art , 342(337):2, 2022.

L. Holmberg, P. Davidsson, and P. Linde. Mapping knowledge representations to concepts: A review and new perspectives. ArXiv , abs/2301.00189, 2022.

S. Jain, H. Lawrence, A. Moitra, and A. Madry. Distilling model failures as directions in latent space. arXiv preprint arXiv:2206.14754 , 2022.

D. Kazhdan, B. Dimanov, M. Jamnik, P. Liò, and A. Weller. Now you see me (cme): concept-based model extraction. arXiv preprint arXiv:2010.13233 , 2020.

M. Keswani, S. Ramakrishnan, N. Reddy, and V. N. Balasubramanian. Proto2proto: Can you recognize the car, the way i do? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10233-10243, 2022.

B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler, F. Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning , pages 2668-2677. PMLR, 2018.

E. Kim, S. Kim, M. Seo, and S. Yoon. Xprotonet: Diagnosis in chest radiography with global and local explanations. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 15714-15723, 2021.

E. Kim, J. Lee, and J. Choo. Biaswap: Removing dataset bias with bias-tailored swapping augmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14992-15001, 2021.

P. W. Koh, T. Nguyen, Y. S. Tang, S. Mussmann, E. Pierson, B. Kim, and P. Liang. Concept bottleneck models. In International Conference on Machine Learning , pages 5338-5348. PMLR, 2020.

B. Kovacs, S. Bell, N. Snavely, and K. Bala. Shading annotations in the wild. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 850-859, 2017.

B. Kovacs, S. Bell, N. Snavely, and K. Bala. Shading annotations in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6998-7007, 2017.

M. L EH. Lightness and retinex theory. J. Opt. Soc. Am. , 61(1):1-11, 1971.

E. H. Land. The retinex theory of color vision. Scientific American , 237 6:108-28, 1977.

J. Lee, E. Kim, J. Lee, J. Lee, and J. Choo. Learning debiased representation via disentangled feature augmentation. Advances in Neural Information Processing Systems , 34:25123-25133, 2021.

J. Li, P. Zhou, C. Xiong, and S. C. Hoi. Prototypical contrastive learning of unsupervised representations. arXiv preprint arXiv:2005.04966 , 2020.

Y. Li and N. Vasconcelos. Repair: Removing representation bias by dataset resampling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9572-9581, 2019.

Y. Li, L. Guo, and Y. Ge. Pseudo labels for unsupervised domain adaptation: A review. Electronics , 12 (15):3325, 2023.

Z. Li and N. Snavely. Cgintrinsics: Better intrinsic image decomposition through physically-based rendering. In European Conference on Computer Vision (ECCV) , 2018.

P. Linardatos, V. Papastefanopoulos, and S. Kotsiantis. Explainable ai: A review of machine learning interpretability methods. Entropy , 23(1):18, 2020.

Y. Ma, X. Feng, X. Jiang, Z. Xia, and J. Peng. Intrinsic image decomposition: A comprehensive review. In Image and Graphics: 9th International Conference, ICIG 2017, Shanghai, China, September 13-15, 2017, Revised Selected Papers, Part I 9 , pages 626-638. Springer, 2017.

M. Moayeri, K. Rezaei, M. Sanjabi, and S. Feizi. Text-to-concept (and back) via cross-model alignment. arXiv preprint arXiv:2305.06386 , 2023.

I. Nassar, M. Hayat, E. Abbasnejad, H. Rezatofighi, and G. Haffari. Protocon: Pseudo-label refinement via online clustering and prototypical consistency for efficient semi-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11641-11650, 2023.

C. Niu, H. Shan, and G. Wang. Spice: Semantic pseudo-labeling for image clustering. IEEE Transactions on Image Processing , 31:7264-7278, 2022.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

L. Rieger, C. Singh, W. Murdoch, and B. Yu. Interpretations are useful: penalizing explanations to align neural networks with prior knowledge. In International conference on machine learning , pages 8116-8126. PMLR, 2020.

A. S. Ross, M. C. Hughes, and F. Doshi-Velez. Right for the right reasons: Training differentiable models by constraining their explanations. arXiv preprint arXiv:1703.03717 , 2017.

G. Saha and K. Roy. Saliency guided experience packing for replay in continual learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 5273-5283, 2023.

W. Samek, T. Wiegand, and K.-R. Müller. Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models. arXiv preprint arXiv:1708.08296 , 2017.

Y. Sawada and K. Nakamura. C-senn: Contrastive self-explaining neural network. ArXiv , abs/2206.09575, 2022.

Y. Sawada and K. Nakamura. Concept bottleneck model with additional unsupervised concepts. IEEE Access , 10:41758-41765, 2022.

J. Schrouff, S. Baur, S. Hou, D. Mincu, E. Loreaux, R. Blanes, J. Wexler, A. Karthikesalingam, and B. Kim. Best of both worlds: local and global explanations with human-understandable concepts. arXiv preprint arXiv:2106.08641 , 2021.

G. Schwalbe. Concept embedding analysis: A review. arXiv preprint arXiv:2203.13909 , 2022.

R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision , pages 618-626, 2017.

D. Smilkov, N. Thorat, B. Kim, F. Viégas, and M. Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825 , 2017.

Y. Song, S. K. Shyn, and K.-s. Kim. Img2tab: Automatic class relevant concept discovery from stylegan features for explainable image classification. arXiv preprint arXiv:2301.06324 , 2023.

R. Soni, N. Shah, C. T. Seng, and J. D. Moore. Adversarial tcav - robust and effective interpretation of intermediate layers in neural networks. ArXiv , abs/2002.03549, 2020.

K. Tanwisuth, X. Fan, H. Zheng, S. Zhang, H. Zhang, B. Chen, and M. Zhou. A prototype-oriented framework for unsupervised domain adaptation. Advances in Neural Information Processing Systems , 34: 17194-17208, 2021.

E. Tartaglione, C. A. Barbano, and M. Grangetto. End: Entangling and disentangling deep representations for bias correction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 13508-13517, 2021.

S. Vojíˇ r and T. Kliegr. Editable machine learning models? a rule-based framework for user studies of explainability. Advances in Data Analysis and Classification , 14(4):785-799, 2020.

M. Wang and W. Deng. Deep visual domain adaptation: A survey. Neurocomputing , 312:135-153, 2018.

Y. Wang, Q. Yao, J. T. Kwok, and L. M. Ni. Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur) , 53(3):1-34, 2020.

L. Weber, S. Lapuschkin, A. Binder, and W. Samek. Beyond explaining: Opportunities and challenges of xai-based model improvement. Information Fusion , 2022.

M. Wu, M. Hughes, S. Parbhoo, M. Zazzi, V. Roth, and F. Doshi-Velez. Beyond sparsity: Tree regularization of deep models for interpretability. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018.

Y. Xian, C. H. Lampert, B. Schiele, and Z. Akata. Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis and machine intelligence , 41(9): 2251-2265, 2018.

M. Xue, Q. Huang, H. Zhang, L. Cheng, J. Song, M. Wu, and M. Song. Protopformer: Concentrating on prototypical parts in vision transformers for interpretable image recognition. arXiv preprint arXiv:2208.10431 , 2022.

L. Yang, B. Huang, S. Guo, Y. Lin, and T. Zhao. A small-sample text classification model based on pseudo-label fusion clustering algorithm. Applied Sciences , 13(8):4716, 2023.

R. Zhang, P. Madumal, T. Miller, K. A. Ehinger, and B. I. Rubinstein. Invertible concept-based explanations for cnn models with non-negative concept activation vectors. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 11682-11690, 2021.

Y. Zhang, P. Tiˇ no, A. Leonardis, and K. Tang. A survey on neural network interpretability. IEEE Transactions on Emerging Topics in Computational Intelligence , 5(5):726-742, 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce an idea by which a student model can become (de) sensitive to some human-understandable concept in its decision-making process. They find CAVs in a teacher model and then transform those CAVs into the student model feature space and use orthogonal vectors to these CAVs to penalize/incentivize models to make decisions based on those concepts.\n\nThey improve over baselines on biassed datasets where a concept is known to be the underlying bias.\n\nSTRENGTHS:\n+ I like their idea of debiasing the student model based on human feedback on biased and unbiased samples.\n+ method shows superior improvement over existing results when evaluated on the biassed dataset.\n+ They visually validate that their model is focusing on more important parts of the images in decision-making.\n\nWEAKNESSES:\n+ I'd expect to see some more analysis of the student's model performance. For example, why does the model still get relatively bad accuracy on the color MNIST? is it still biased toward the colors? can we still debias this model and push to get around 100 accuracies?\nif not, what are the limitations?\n\n+ This is a minor weak point but I expect to see some recent but relevant related work in the paper. As the paper is considering an alignment between concepts in different models' features spaces, it reminded me of a recent paper \"Text-To-Concept (and Back) via Cross-Model Alignment\" where an alignment between features spaces is used to interpret a model's behavior in terms of human-concepts. The general framework is somehow similar to this paper.\nFurthermore, the idea of considering the normal vector orthogonal to the decision-making half plane is somehow relevant to this paper \"Distilling Model Failures as Directions in Latent Space\". I want the authors to consider those and other relevant papers.\n\n+ quality of figures should improve (Fig 4).\n\nQUESTIONS:\nMy questions are found in the weakness part.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a methodology for training a model to sensitize or desensitize a specific concept. Particularly, it introduces a concept distillation loss that utilizes Concept Activation Vectors (CAVs) derived from a high-performing teacher classifier with abundant knowledge of the concept, aiming to reduce concept activation in that direction. Additionally, it introduces a prediction loss term based on class prototypes to incorporate global features. The paper also employs an autoencoder to minimize the discrepancy between the latent spaces of the teacher and the student. Through experiments on various biased datasets, it demonstrates the robustness of the concept-distilled student model against biases.\n\nSTRENGTHS:\nTo the best of my knowledge, this paper is the first to propose the distillation of concept information from a well-trained teacher to a student model. The experimental results provide clear verification that the proposed approach effectively reduces the spurious correlations learned by deep neural networks (DNNs). One particularly interesting result is that even when the teacher model is biased by 100%, the student model still exhibits a reduction in bias. This finding highlights the effectiveness of the proposed methodology in mitigating bias in the student model.\n\nWEAKNESSES:\nIt seems that one limitation is the use of only simple datasets in the experiments and another is that the overall pipeline is somewhat unclear (please refer to the first question of below section), but other than that, there don't appear to be any specific weakness.\n\nQUESTIONS:\n1) The description of the training pipeline seems somewhat unclear. Has the student model already been trained to achieve a high classification accuracy, and is it now receiving concept knowledge through the prototype loss? If so, wouldn't the latent space of the student model change, requiring constant updates to the module that maps the latent space between the teacher and student models?\n\n2) Regarding the motivation and implementation of concept distillation loss, could it potentially increase the sensitivity to orthogonal concepts as an unexpected side effect?\n\n3) In experiments where there are multiple concepts, how is the loss defined when trying to eliminate bias for all these concepts? Is it defined as an average or debiased one by one?\n\n4) Is it possible to experiment with datasets that have a wider range of concepts? For example, using the Broden concept [1] with ImageNet?\n\n5) Is it feasible to conduct experiments where the activation in a trained teacher model itself is controlled rather than distillation to student?\n\n6) Can additional evaluation be provided to assess how well the autoencoder used for latent space mapping is performing?\n\n[1] Bau, David, et al. \"Network dissection: Quantifying interpretability of deep visual representations.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.\n\nLIMITATIONS:\nIt seems that there are no particular specific limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents the idea of concept distillation from a pretrained teacher to improve a student. They utilize the notion of concept activation vectors (CAV) as concept representations and adapt a pretrained student to sensitize or desensitize a student w.r.t a concept. They apply this idea for two applications to improve a model in ante-hoc fashion: (i) reducing biases in a student towards a concept, and (ii) incorporating prior knowledge\n\nSTRENGTHS:\n1. The paper is reasonably well written\n\n2. The selection of baselines is quite extensive and the proposed method generally performs well\n\n3. The core idea of using CAV as representations to improve a model indeed has wide applicability. This is shown quite well by authors to a good extent\n\nWEAKNESSES:\nI unfortunately have multiple methodological and experimental issues with the. Please find them detailed in the \"Questions\" sections. The paper has lot of the hallmarks to be a useful and effective work, but I simply didn't enjoy reading it much as I should have, being constantly riddled with doubts in my mind.\n\nQUESTIONS:\nDoubt #1\n\n1. (Methodological, major) I am still not convinced by the use of teacher for CAV estimation. The first supporting argument was in line 170-172 \"Direct CAV learning ... as the model may not have sufficiently rich comprehension of the relevant concepts\". But even when using teacher representations you map them to the student activation space. So if the mapping $M$ to student activation and back worked perfectly, would it defeat the point of the original argument? The only other supporting argument I could find was an experment on ColorMNIST (Tab. 5). It was nice to see but could it be just due to a specific student architecture. Does this observation generalize to other tasks and more complex students?\n\n2. Can the method be used to improve biases in the original teacher given that you know it has good representation for the concepts? I was assuming that to be able to perform self improvement without a teacher would be more useful. \n\n3. Does the students original activation space get disturbed due to the modifications in its updates or are its layers before the selected intermediate layer fixed? If yes, could it harm the concept distillation loss since the student representations change?\n\n4. (Experimental) Can the model be used to remove many biases simultaneously? This would really enhance the applicability and power of the method.\n\n5. (Minor) Are $M$ and $M^{-1}$ mappings really exact inverses? In that case, isn't $L_{M^{-1}}$ guaranteed to be 0? If yes, then what is the point of $L_{M^{-1}}$? If not, then the notation should be modified.\n\n6. (Minor) I am slightly dissappointed with only one non-MNIST data for debiasing and one for IID task. This is not the biggest concern for me since you do test your method in multiple different ways. And it certainly won't be the deciding factor in my score but nevertheless the impact would feel greater if one of the task was more extensive in its coverage of complex datasets. And I assume you want debiasing as the main task.\n\nLIMITATIONS:\nThe authors do discuss in the main paper their limitations. They are also upfront about potential societal impact. I am pretty much satisfied with the discussion.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to sensitize or desensitize a (smaller) student model with respect to user-provided high-level concepts, by leveraging a (larger) teacher model. Specifically, a supervised mapping model learns a bijection between some chosen latent space in the teacher model and the student model. Then, CAVs extracted from teacher model is mapped to the student model through the mapping model. Finally, the student model is trained to sensitize / desensitize with respect to the mapped CAVs by minimizing / maximizing the concept sensitivity score.\n\nSTRENGTHS:\n* The ideas in this paper are quite creative and have plenty of potential if explored properly.\n* Working with new real-world datasets (i.e. IID experiment) in the interpretability domain is always welcomed, which makes interpretability methods more application grounded.\n\nWEAKNESSES:\nOverall my impression is the paper explores too many ideas at once and thus unfortunately fails to cover each one in detail. Design choices are not well-justified and phenomena are not well-understood.\n\n* I find this paper extremely hard to read. Ideas are not elaborated clearly when first introduced and pieces of information are scattered all over the paper. It takes about 3x the time to read this draft compared to other ones. Perhaps the writing needs to be cleaned up. \n    * Example 1: In P8 the ablation for ColorMNIST is placed after IID experiments which is confusing.\n* The proposed method is too complex and has too many claims that are not well substantiated.\n    * Claims:\n        * Optimizing for concept distillation loss (Eq 2) can effectively sensitize / desensitize\n        * Prototype-based loss captures sensitivity better at a global level and \"facilitates the use of any intermediate layer by serving as pseudo-class label\"(P5L168) which frankly I have no idea what this means.\n        * Large models learns CAV better than smaller ones. (the entire premise for adopting a teacher-student framework)\n* The compared baselines may not necessary be fair since the proposed method is a fully-supervised one that requires training of entire models while baselines are either zero-shot or few-shot. Should compare with fairness/debiasing methods that requires full training of the entire model.\n\nQUESTIONS:\n* For each of the claims, there should be dedicated experiments to verify the claims, instead of bunching everything together and comparing this menagerie method with weaker 0-shot/few-shot baselines. Please justify the claims explicitly. \n* A proper ablation study on the design choices is a necessary component for such complex method. How does each component affect the result?\n* For the concept distillation loss, the entire student model is optimized for Eq (2) but the (mapped) CAV is kept constant. If layers prior to the latent space where the CAV lies is modified, CAV no longer represents the latent space because the input representation has changed. For example, suppose some concept happens to lie in the first dimension of the CAV latent space. If we optimize the entire model, the model could switch the concept and encode it in the second dimension instead. Thus, does it not make sense to only optimize for the model after the CAV layer?\n* There are no descriptions of what each entry in Table 6 represents. P9L288 states to \"See supplementary for detailed explanations\" but I haven't been able to find where them. Can the authors point me to the exact section for the ablation study?\n* Why exactly is $M^{-1}$ needed?\n* Can the authors elaborate on why concept distillation from a completely biased teacher AND the teacher shares the same architecture as the student model might still improve? The result is quite surprising considering the benefits of distillation has disappeared in this setting.\n\nLIMITATIONS:\nThere is a small section in P9L299-302. The only limitation stated is \"The dependence on the teacher for conceptual knowledge could be another drawback of our method as with all distillation frameworks\". One limitation might be when CAV extracted from the teacher model fails to translate to the student model. When, where, and why does this happen?", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}