Concept Distillation: Leveraging Human-Centered Explanations for Model Improvement
Authors: Avani Gupta
Saurabh Saini
P J Narayanan
Venue: NeurIPS 2023 poster (2023)
Keywords: Human Centered Concepts
ML interpretability
XAI based Model Improvement
Debiasing
TL;DR: Novel human centered concept based training of Neural Networks using distillation, concept vector based loss and proto-types

Abstract: Humans use abstract *concepts* for understanding instead of hard features. Recent interpretability research has focused on human-centered concept explanations of neural networks. Concept Activation Vectors (CAVs) estimate a model's sensitivity and possible biases to a given concept. We extend CAVs from post-hoc analysis to ante-hoc training to reduce model bias through fine-tuning using an additional *Concept Loss*. Concepts are defined on the final layer of the network in the past. We generalize it to intermediate layers, including the last convolution layer. We also introduce *Concept Distillation*, a method to define rich and effective concepts using a pre-trained knowledgeable model as the teacher. Our method can sensitize or desensitize a model towards concepts. We show applications of concept-sensitive training to debias several classification problems. We also show a way to induce prior knowledge into a reconstruction problem. We show that concept-sensitive training can improve model interpretability, reduce biases, and induce prior knowledge.

Concept Distillation: Leveraging Human-Centered Explanations for Model Improvement
Avani Gupta 122 Saurabh Saini² PJ Narayanan 2
MM4 UAE 'IITT Hyderabad, India
davani.guptal saurabh. waiiiia iiit. aiit.cc.in ac.in pjn@iiit.ac in
Abstract
Humans use abstract concepts for understanding instead of hard features. Recent interpretability research has focused human-centered concept explanations of neural networks. Concept Activation Vectors (CAVs) estimate a model's sensitivity and possible biases to given concept. In this paper, we extend CAVs from post- hoc analysis to ante-hoe training in order to reduce model bias through fine-tuning using an additional Concept Loss. Concepts were defined the final layer of the network in the past. We generalize it to intermediate layers using class prototypes. This facilitates class learing in the last convolution layer which is known to be most informative. We also introduce Concept Distillation t create richer concepts using a pre-trained knowledgeable model as the teacher. Our method can sensitize desensitize a model towards concepts. We show applications of concept- sensiiive training to debias several classification problems. We also use concepts to induce prior knowledge into IID), a reconstruction problem. Concept-sensitive training can improve model interpretability. reduce biases, and induce prior knowledge. Please visit https://avani1711 1100 1000111tt0u1...io... Saapauiiio/Connep-Diiitiii
for code and more details.
1 Introduction
EXplainable Artificial Intelligence (XAI) methods are useful understand a trained model's behavior [74]. They open the black box of Deep Neural Networks (DNNs) to enable post-training identification of unintended correlations or biases using similarity scores or saliency maps. Humans, however, think in terms of abstract concepts, defined as groupings of similar entities (27). Recent efforts in XAI have focused on concept-baus based model explanations to make them more aligned with human cognition. Kim et al. 32 introduce Concept Activation Vectors (CAVs) using concept classifier hyperplane to quantify the importance given by the model particular concept. For instance, CAVs can determine the model's 's sensitivity on "striped-ness' or "dotted-ness" to classify Zebra or Cheetah using user-provided concept samples. They measure the concept sensitivity of the model's final layer prediction with respect intermediate layer activations (outputs). Such post-hoc analysis can evaluate the transparency, accountability, and reliability biases unintended correlations acquired by the models via shortcut learning
learned model [4 and can identify
32 5).
The question we ask in this paper is: If CAVs can identify and quantify sensitivity to concepts, can they also be used to improve the model? Can we learn less biased and more kuman--cntered models? In this paper, we extend CAVs to ante-hoc model improvement through novel concept loss to desenssitzze/sensitize against concepts. We also leverage the broader conceptual knowledge of large pre-trained model teacher in a concept distillation framework for it.
Work done while HIT Hyderabad
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Class consistent TEXtUINGS
Lzarted Shotton: (texture--nn predict (cloes=ak)
Testure
Post-hoc Explanations
Trainset
2
Concept sets
Few-shntt: distribution information
Blased Student
2
Local semple specili
2
Concept Distillation
meddo
Improvement
distrbutiin informetion
CAY Leamning & Space Mapying
Concept Teacher
Global XAI: class common
Domein Adaplation
Random testures
Full test distribution
Figure 1: Overview of our approach: The generic conceptual knowledge of capable teacher can be distilled to a student for performance improve- ment through bias removal and prior induction.
Informatio
Figure 2: Categorization of related works with ours highlighted.
They typically make fundamental changes to the model architecture or need significant concept supervision, making extensions other applications difficult. For example, Koh et al. (35) condition the model by first predicting the underlying concept and then using it for class prediction. Our method can sensitize or desensitize the model to user-defined concepts without modifications to the architecture or direct supervision.
XAI has been used for ante-hoc model improvement during training 335 30 4
Our approach relies on the sensitivity of a trained model to human-specified concepts. We want the model to be sensitive relevant concepts and indifferent to others. For instance, a cow classifier might be focusing excessively on the grass associated with cow images. If we can estimate the sensitivities of the classifier to different concepts, we can steer it away from irrelevant concepts. We do that using a concept loss term Le and fine-tuning the trained base model with it. Since the base models could be small and biased in different ways, we use concept distillation using a large, pre-trained teacher model that understands common concepts better.
We also extend concepts to work effectively on intermediate layers of the model, where the sensitivity is more pronounced. Kim et al. (32 measure the final layer's 's sensitivity to any intermediate layer outputs. They ask the question: if any changes in activations are done in the intermediate layer, what is its effect on the final layer prediction? They used the final layer's s loss/logit to estimate the model sensitivity as their interest was to study concept sensitivities for interpretable model prediction. We, on the other hand, aim to fine-tune model by (de)senniti it towards given concept which may be strongest in another layer 3 Thus, crucial for us to measure the sensitivity any layer by evaluating the effect of the changes in activations in one intermediate layer on another. We employ prototypes or average class representations in that layer for this purpose. Prototypes are estimated Our method, thus, allows
by clustering the class sample activations 12
49 48 63
43
intervention in any layer.
In this paper, we present simple but powerful framework for model improvement using concept loss and concept distillation for user- user-given concept defined in any layer the network. We leverage ideas from post-hoc global explanation techniques and use them in ante-hoc setting by encoding concepts as CAVs via a teacher model. Our method also admits sample-specific explanations via a local loss (22) along with the global concepts whenever possible. We improve ttate-of-the-an on classification
problems like ColorMNIST and MNCST [5T 16 52, 5) resulting in improved accuracies and generalization. We introduce and benchmark on with texture bias associated with digits. We demonstrate concept distillation on two applications: (i) debiasing extreme biases on classification problems involving synthetic MNIST datasets 42 [6] and complex and sensitive age-vs.-gende bias in the real-world gender classification on BFFHQ dataset [34] and (ii) prior induction by infusing domain knowledge in the reconstruction problem of Intrinsic Image Decomposition (IID) B8 by measuring and improving disentanglement of albedo and shading concepts. To summarize, we:
and more challenging TeXTureMNIST dataset
Extend CAVs from post-hoe explanations to ante-hoc model improvement method to sensi- tize/desensitize models specific concepts without changing the base architecture.

---

Extend the model CAV sensitivity calculation from only final layer to any layer and enhance it by making it more global using prototypes.
Introduce concept distillation to exploit the inherent knowledge of large pretrained models as a teacher in concept definition.
Benchmark results on standard biased MNIST datasets and on a challenging TNXIT
dataset that we introduce.
Show application on a severely biased classification problem involving age bias.
Show application beyond classification to the challenging multi-branch Intrinsic Image Decomposition problem by inducing human-centers concepts priors. To the best of our knowledge, this is the first foray of sancept-based techniques into non-classficattin problems.
2 Related Work
The related literature is categorized in Fig. 2 Post-hoc (after training) explainability methods include Activation Maps Visualization [59]. Saliency Estimation (54), Model Simplification [69], Model
Perturbation [17], Adversarial Exemplar Analysis (22]. etc. See recent surveys for comprehensive
discussion 774 15
45 65 2]. Different concept-based interpretability methods are surveyed by
Hitzler and Sarker 277. Schwalbe (58), Holmberg et al. (28
The ante-hoc model improvement techniques can be divided into zero-shot [70), few-shot [67], and multi-shot
(66)) categories based on the amount of test distribution needed. In zero-shot training, the model never sees the distribution of test samples [70], while in few-shot, the model has access to some examples from the distribution of the test set [67]. Our method works with abstract concept sets (different than any test sample), being essentially a zero-shot method but can take advantage of few-shot examples, if available, as shown in results on BFFHQ subsection 4.1 We restrict the discussion to the most relevant model improvement methods.
Ross et al. [52 penalize model if it does not prefer the Right answers for Right Reasons (RRR) by using explanations as constraints. EG Erion et al. [16] augment gradients with a penalizing term to match with user-provided binary annotations. More details are available in relevant surveys
18 68 24 g
Based on the scope of explanation, the existing XAI methods can be divided into global and local methods (74). Global methods [32 20 provide explanations that are true for all samples of class
(e.g. 'stripiness' concept is focused for the zebra class or 'red color' is focused on the prediction of zero) while local methods 5
60, 53 explain each of samples individually often indicating regions or patches in the image which lead the model to prediction (example, this particular patch (red) in this image was focused the most for prediction of zero). Kim et al. (32 quantify concept
sensitivity using CAVs followed by subsequent works [62 73 57 Kim et al.
32 sample sensitivity
local (sample specific) while they aggregate class samples for class sample sensitivity estimation, which makes their method global (across classes). We enhance their local sample sensitivity to the global level via using prototypes that capture class-wide characteristics by definition [2].
Only few concept-oriented deep learning methods train neural networks with the help of concepts 56 55
Concept Bottleneck Models (CBMs) 33 Conceptt-bass based Model Extraction (CME) 30)
and Self- Explainii Neural Networks (SENNs)
4 predict concepts from inputs and use them to infer
the output. CBMs 355 require concept supervision, but CME 30 can train in a partially supervised
manner to extract concepts combining multiple layers. SENN 4 learns interpretable basis concepts by approximating model with a linear classifier. These methods require architectural modifications of the base model to predict concepts. Their primary goal is interpretability. which differs from our model improvement goal using specific concepts without architectural changes. Closest to our work is ClArC [5), which leverages CAVs to manipulate model activations using a linear transformation to remove artifacts from the final results. Unlike them, our method trains the DNN to be sensitive to any specific concept, focusing on improving generalizability for multiple applications.
DFA 40 and EnD 64 use bias-conflicting or few-shot generic debiasing. DEA uses small set of adversarial samples and separate encoders for learning disentangled features for intrinsic and biased attributes. EnD uses a regularization strategy to prevent the learning of unwanted biases by inserting an information bottleneck using pre-known bias types. is to be noted that both DFA and EnD are neither zero-shot nor interpretability- methods. In comparison, our method works in both known bias settings using abstract user-provided
(OOD) samples for debiasing using
3

---

dlass samplo
Train
Concepts
Sckk.(%))
fl
P
Vo
6
6
6
6
Random Concept
f
dectsfoon
boundary
Figure 3: ty is calculated as normal to the separat- ing hyperplane of concept activations (textures C' vs. random set C' here). A model biased to- wards C' will have its class samples's loss gradient v Lp along v' (measured by sensitivity Sc...I (z)) To desensitize the model for C, we perturb V L to be parallel to the decision boundary by mini- mizing the cosine of the projection angle.
Figure 4: Datasets used: ColorMNIST (top row), TextureMNIST (next row), DecoyMNIST (third row), and BFFHQ (bottom rows). Concepts used include color, textured and gray patches, and bias-conflicting samples shown on the right.
concept sets and unknown bias settings using the bias-conflicting samples. Our Concept Distillation method is concept sensitive training method for induction of concepts into the model, with debiasing being an application. Intrinsic Image Decomposition (IID) [1 19 involves decomposing an image
into its constituent Reflectance (R) and Shading (S) components [88 46 which supposed be
disentangled [46 23)
We use CAVs for inducing R and S priors in pre-trained SOTA IID framework 44 with improved performance. To the best of our knowledge, we are the first to introduce prototypes for CAV sensitivity enhancement. However, prototypes have been used in the interpretability literature before to capture class-level characteristics [33
31 and also have been used as pscudo-class labels 437. Some recent approaches [29 61 generative models to generate bias confloonnii samples (e.g. other colored zeros in ColorMNIST) and train the model on
7
before 12 72 49
48
63
them to remove bias. Specifically, Jain et al. [29] SVMs to find directions of bias in shared image and language space of CLIP (50) and use Dall-E on discovered keywords of bias to generate bias-conflicting samples and train the model them. Song al. [6] use StyleGAN to generate bias-conflicting samples. Training on bias-conflicting samples might not always be feasible due to higher annotation and computation costs.
61] is that they propose data augmentation as a debiasing strategy, whereas we directly manipulate the gradient vectors, which is more interpretable. Moayeri et al. [47] map the activation spaces of two models using the CLIP latent space similarity. Due to the generality of the CLIP latent space, this approach is helpful to encode certain concepts like cat, dog, man,' but it is not clear how it will work on abstract concepts with ambiguous definitions like 'shading' and 'reflectance' seen in the IID problem described above.
One significant difference between such methods [29]6]]
3 Concept Guidance and Concept Distillation
Concepts have been used explain model behavior in a post-hoc manner in the past. Response to abstract concepts can also demonstrate the model's intrinsic preferences, biases, etc. Can we use concepts guide the behavior of trained base model in desirable ways in an ante-hoc manner? We describe method to add a concept loss to achieve this. We also present concept distillation as a way take advantage of large foundational models with more exposure to wide variety of images.
3.1 Concept Sensitivity to Concept Loss
Building on Kim et al. 32], we represent a concept C' using a Concept Activation Vector (CAV) as the normal o/c to linear decision boundary between concept samples C from others C' in layer I of the model's activation space (Fig. 3 The model's sensitivity (x) VL. (fi(z)) of to C is

---

the directional derivative of final layer loss Lg for samples æ along 'ic [32. The sensitivity score quantifies the concept's influence on the model's prediction. high sensitivity for color concept may indicate color bias in the model.
These scores were used for post-hoc analysis before [22] sensitize the base model to concepts by perturbing it away from or towards the CAV direction (Fig. 3). The gradient of loss indicates the direction of maximum change. Nudging the gradients away from the CAV direction encourages the model to be less sensitive to the concept and vice versa. For this, we define concept loss Le as the absolute cosine of the angle between the loss gradient and the CAV direction
We use them ante-hoc desensitize or
Lc (a)) cos(VV, (fi(z)),
(I)
which is minimized when the CAV lies on the classifier hyperplane (Fig. 3
We use the absolute value to not introduce the opposite bias by pushing the loss gradient in the opposite direction. loss of (1 (x)) will sensitize the model to C. We fine-tune the trained base model for few epochs using a total loss of Lo + Ale to desensitize concept C, where L, the base model loss.
3.2 Concepts using Prototypes
Concepts can be present in any layer L, though the above discussion focuses on the sensitivity calculation of the final layer using model loss Lo- The final convolutional layer proven learn concepts better than other layers [3]. We can estimate the concept sensitivity of any layer using a loss for that layer. How do we get loss for an intermediate layer, as no ground truth available for it?
We adapt prototypes define a loss in intermediate layers. Let fi(x) be the activation of layer for sample I. We group the fi(x) values of the samples from each class into K clusters. The cluster centers P: together form the prototype for that class. We then define prototype loss for each training sample using the prototype corresponding to its class as
Class prototypes have been used as pseudo-labels intermediate layers before 12 72 41
K (x)x) = K Σ 1
We use Lp in place of L. in Eq. I to define the concept loss in layer The prototype loss facilitates the use of intermediate layers for concept (de)sensitization. Experiments reported in Tab. confirm the effectiveness of Lg. Prototypes also capture sample sensitivity at global level using all samples of a class beyond the sample-specific levels. We update the prototypes after few iterations as the activation space evolves. If p" the prototype at Step n and P the cluster centres using the current ft(::) values, the next prototype is ph+11 (1 a) Ph aP for each cluster k.
3.3 Concept Distillation using a teacher
Concepts are learned from the base model in the above formulation. Base models may have wrong concept associations due their training bias limited exposure to concepts. Can alleviate this problem using a larger model that has seen vast amounts data as teacher in a distillation framework?
We use the DINO [3], a self-supervised model trained on a large number of images, as the teacher and the base model the student for concept distillation. The teacher and student models typically have different activation spaces. We map the teacher space to the student space before concept learning. The mapping uses autoencoder (26) consisting of an encoder EM and a decoder DM (Fig. 5). As first step, the autoencoder (Fig. 5) is trained to minimize the loss LD + LE LD is the pixel-wise L2 loss between the original (fr) and decoded (fi) teacher activations and Le is the pixel-wise L2 loss between the mapped teacher (ff) and the student (fa) activations. The mapping is learned over the concept set images C' and C". See the dashed purple lines in Fig.
Next, we learn the CAVs in the distilled teacher space fa, keeping the teacher, student, and mapping modules fixed. This is computationally light as only few (50-150) concept set images are involved. The learned CAV is used in concept loss given in Eq. I Please note that FAr used only to align the two spaces and can be small capacity encoder, even single layer trained in few epochs.
5

---

Mapping
Testing
Mapping
Mapping autocncoder
ft
Pre-trained Teacher
EM
DM
Li
Concept Distillation in Student
cleas sample
Concapt
f.
Distilation
Prototype loss (Leg
Proto Ps
Student Classifler
layer
logits
tabelx
Training Set
Classiflcation
GT LORK (La)
Figure 5: Our framework comprises a concept teacher and a student classifier and has the follow- ing four steps: 1) Mapping teacher space to student space for concepts C. and C' by training an autoencoder Eιs and DM (dotted purple lines); 2) CAV (o'') learning in mapped teacher space via linear classifier LC (dashed blue lines); 3) Training the student model with Concept Distillation (solid orange lines): We use oy and class prototypes loss Lp to define our concept distillation loss Le and use it with the original training loss Lo to (de)sensitize the model for concept C; 4) Testing where the trained model is applied (dotted red lines)
Algorithm 1 Concept Distillation Pipeline
Given: A pretrained student model which is to be fine-tuned for concept (de)sensitization and pretrained teacher model which will be used for concept distillation. Known Concepts C and negative counterparts (or random samples) C", student training dataset D, and student bottleneck layer l, #iterations recalculate CAVs cav _oy_update_at frequency #iterations to update prototypes proto_ poto_u frequency.
Concept Distillation:
2: For all class samples in D, estimate class prototypes (o, with K-means.
3: Current iteration n = 0, initial prototypes po = p=.
4: While not converge do:
5: If = 0 or (update_caa) _eaittteevvseeeeeeeeeeeeeeeeeeee and % saw_update_frrquuee
0) then:
Learn Mapping module:
7:
Forward pass x E C'U C" from Teacher and Student to get their concept activations fe and fx. Learn the mapping module as autoencoders EM and Dy--
8:
9:
CAV learning in mapped teacher's 's space:
10:
ve learned binary linear classifier normal to decision boundary of Eas(f(()) for a C x' C"
Ex (x(((((a')
11:
v 56 proto_ proto_update, frequeeccccy = 0 and n o 0 then: Estimate new class prototypes (0.8) with K-means.
12:
13:
Weighted Proto-type ph+ = (1 a) th + aPe.
14:
Else:
15:
P pa+1 P.
Train student with Le + Lag.
17:
n+ =1.
4
Experiments
We demonstrate the impact of the concept distillation method on the debiasing of classification problems as well improving the results of real-world reconstruction problem. Classification experiments cover two categories of Fig. 2] the zero-shot scenario, where no unbiased data is seen, and the few-shot scenario, which sees
unbiased data samples. We use pre-trained DINO ViT-B8 transformer [I3 as the teacher. We use the last convolution layer of the student model for concept distillation. The mapping module 120K parameters, 10MB), CAV estimations (logistic regression, < IMB), and prototype calculations all complete within a few iterations of training, taking 15-30 sees on single 12GB Nvidia 1080 Ti GPL Our framework computationally light. In our experimentation reported below, we fix the CAVs as iitiil CAVs and observe similar results as
MMore details in supplementary

---

Table Main components of our method shown on ColoTMNIST dataset: Ours usage of teacher and proto-types yields the best performance. Teacher?
Table 2: TCAV scores of bias concept: Our con-
cept sensitive training significantly decreases the sensitivity of model towards bias.
Prototype? Accuracy
Dataset
Concept
Base Model Ours
X
9.96
ColorMNIST
Color
0.52
0.21
/
26.97
Decoy MNIST
Spatial patches
0.57
0.45
x
30.94
MeXIS
Textures
0.68
0.43
/
50.93
BFFHQ
Age
0.78
0.13
varying them. Also, our concept sensitive fine-tuning method converges quickly, and hence updating CAVs in every few iterations does not help Where relevant, report average accuracy over five training runs with random seeds (+ indicates variance).
4.1 Concept Sensitive Debiasings
We show results on two standard biased datasets (ColorMNIST 42 and DecoyMNIST (16] and introduce more challenging TextureMNIST dataset for quantitative evaluations. We also experimented on real-world gender classification dataset BFFHQ 34 that is biased based on age. We compare with other state-of-the-arr interpretable model improvement methods
51 52
5 Fig. 4 summarizes the datasets, their biases, and the concept sets used.
Poisoned MNIST Datasets: ColorMNIST 442 has MNIST digit classes mapped to par- ticular color in the training set [5]] The colors are reversed the test set. The baseline CNN model (Base) trained on ColorMNIST gets 0% accuracy on the 100% poisoned test set, indicating that the model learned color shortcuts instead of digit shapes. We debias the model using a concept loss for color using color patches random shapes (negative concept) to estimate CAV for color (Fig. 4). We show results comparison on the usage of teacher and prototypes component of our method in Tab. As can be seen from Tab. 1, our method of using intermediate layer sensitivity via prototypes as described in (subsection 3.2]yseIds better results. Similarly, usage of teacher (described in /subsection 3.3) facilitates better student concept learning (also shown CAV comparisons in supplementary). Our concept sensitive training not only improves student accuracy but observes evident reduction in TCAV scores 332 of bias concept seen from Tab.
Tab. 3 compares our results with the best zero-shot interpretability based methods, i.e., CDEP (51), RRR [52], and EG [16]. Our method improves the accuracy from 31% by CDEP 41.83%, and further to 50.93% additionally with the local explanation loss from
Table 3: Comparison of the accuracy of our method with other zero-shot interpretable model- improvement methods. All methods require user-level intervention: Our method requires concept sets, while others (CDEP, RRR, EG) require user- user-provided rules.
Dataset
Bias
Base C5EP RRR EG
Ours w/o Teacher Ours
Ours+L
ColorMNIST Digit color
0.1
31.0
0.1
10.0
26.97
41.83
50.93
DNCOyMNIST Spatial patches
52.84
97.2
99.0
87.49
98.58 98.98
TexIUreMNIST Digit textures
11.23
10.18
11.35
10.43
38.72
48.82 56.57
Accumnt
Base
CIIC
Oura
Image
Rase
Inzut Image Base
CDEP
Figure Left: Comparison of our method and CIArC 5 on 20% biased ColorMNIST. Right: GradCAM (59 visualizations (more red, more important) on (i) TextureMNIST (middle image) (ii)
Extreme case when the color bias is in the background (bottom) instead of foreground (top). Our model focuses on the shape more than CDEP (which is more blue in the foreground).
7

---

Table 4: Our method improves the model using human-centered concepts and shows better general- ization to different datasets, while CDEP, which uses pixel-wise color rules, cannot.
ColorMNIST Trained
TeXIureMNIST Trained
Test Dataset
Base
CDEP 51
Ours+I
Base
CDEP
Ours+I
Invert color
0.00
23.38
50.93
11.35
10.18
45.36
Random color
16.63
37.40
46.62
11.35
10.18
64.96
Random texture
15.76
28.66
32.30
11.35
10.18
56.57
Pixel-hard
15.87
33.11
38,88
11.35
10.18
61.29
visualizations in Fig. 6jshow that our trained model focuses on highly relevant regions. Fig. 6 left compares our class-wise accuracy on 20% biased ColorMNIST with ClArC [5], a few-shot global method that uses CAVs for artifact removal. (CIArC results are directly reported from their main paper due to the unavailability of the public codebase.) ClArC learns separate CAVs for each class by separating the biased color digit images (e.g. red zeros) from the unbiased images (e.g. zeros in all colors). A feature space linear transformation is then applied to move input sample activations away/towards the learned CAV direction. Their class-specific CAVs definition can not be generalized to test sets with multiple classes. This results in higher test accuracy variance as seen in |Fig. 6 Further, as they learn CAVs in the biased model's 's activation space, common concept set cannot be used across classes DNCOyMNIST [16] has class indicative gray patches on image boundary that biased models learn instead of the shape. We define concept sets as gray patches vs, random set (Fig. 4) and report them in Tab. 3) second row. All methods perform comparably on this task as the introduced bias does not directly corrupt the class intrinsic attributes (shape or color), making it easy to debias. MNII is more challenging dataset that we have created for further research in the area Being an amalgam of colors and patterns, textures are more challenging as a biasing attribute. Our method improves the performance while others struggle on this task Tab.. 3 last row).
GradCAM (59
Fig. 6 shows that our method can focus on the right aspects for this task.
Generalization to multiple biasing situations is another important aspect. Tab. 4 shows the per- formance on different types of biases by differently debiased models. Our method performs better than CDEP in all situations. Interestingly, our texture-debiased model performs better on color biases than CDEP debiased on the color bias! We believe this because textures inherently contain color information, which our method can leverage efficiently. Our method effectively addresses an extreme case of color bias introduced in the background of ColorMNIST, a dataset originally biased in foreground color. By introducing the same color bias to the background and evaluating our ColorMNIST debiased models, we test the capability of our approach. In an ideal scenario where color debiasing is accurate, the model should prioritize shape over color, result achieved by our method but not by CDEP as shown in Fig. 6
BFFHQ 34] dataset is used for the gender classification problem. It consists of images of young women and old men. The model learns entangled age attributes along with gender and gets wrong predictions on the reversed test set ie. old women and young men. We use the bias conflicting samples
by Lee al. 40 - specifically, old women vs. women and young men vs. men as class-specific
concept sets.We We compare against recent debiasing methods EnD (64] and DFA [40] Tab. 5 shows our method getting a comparable accuracy of 63%. We also tried other concept set combinations: (i) old vs. young (where both young and old concepts should not affect) - 62.8% accuracy, (ii) old vs. mix (men and women of all ages) and young vs. mix + 62.8% accuracy. (iii) old vs. random set (consisting of random internet images from (32) and young vs. random + 63% accuracy. These experiments indicate the stability of our method to the concept set definitions. This proves our method can also work with bias- conffiiiii samples [40] and does not necessarily require concept sets Tab. 5) We also experimented by training class-wise (for all women removing young bias
Table 5: Comparisons in few-shot setting: Our method is not limited to user-provided concept sets and can also work with bias-conflicting samples. We compare our accuracy over the BFFHQ dataset 34 with other few-shot debiasing methods.
Dataset
Bias
Base
EnD 64
DEA 44
Ours w/o Teacher
Ours
BFFHQ
Age
56.8742.69
56.87 1..4
61 11.27433.0 33.26
59.4
63.0 10.79
8

---

Table 6: Increasing the bias of the teacher on ColorMNIST reduces accuracy.
Teacher's Training Data Bias%
No Distil
5
10
25
75
90
100
Student Accuracy%
26.97
40.47
33.18
30.38
28.74
23,2
23.63
followed by men removing old bias and vice versa) vs training for all classes together (both men, women removing age bias as described above) and observed similar results, suggesting that our concept sensitive training is robust to class-wise or all class agnostic training. Local interpretable improvement methods like CDEP, RRR, and EG are not reported here as they cannot capture complex concepts like age due to their pixel-wise loss or rule-based nature.
We additionally show method without teacher (Our Concept loss with CAVs learned directly in student) in all experiments as "Ours w/o Teacher" and find inferior performance when compared to Our method with Distillation. Bias in Teacher: We check the effect of bias in teacher by training it with varying fractions of OOD samples to bias them. Specifically, the same student architecture for the teacher. The teacher is trained on the ColorMNIST dataset with 5, 10, 25, 50, 75, and 90% biased color samples in the trainset (e.g. g. k% bias indicates k% red zeros). The resulting concept-distilled system is then tested the standard 100% reverse color setting of ColorMNIST, Tab. shows that concept distillation improves performance even with high teacher bias, though accuracy decreases with the increasing bias in teacher. Apparently, 100% bias in teacher in this setting of teacher with same architecture as the student is essentially CAV learning in same model case (No Distil). Here, the improvements are due prototypes, and as can be seen, there is slight degradation in performance of 100% bias vs No Distil (23.63 vs 26.97). This can be attributed to an error due the mapping module.
Discussion No Distillation Case:
4.2 Prior Knowledge Induction
Intrinsic Image Decomposition (IID) is an inverse rendering problem 7 based on the retinex theory (3), which suggests that an image 7 can be divided into Reflectance R and Shading S components such that I R S at each pixel. Here R represents the material color or albedo, and S represents the scene illumination illumination and S invariant to albedo. As good ground truth for IID is hard to create, IID algorithms evaluated on synthetic data or using some sparse manual annotations 9 36). Gupta et al. ( use CAV-based sensitivity scores to measure the disentanglement of R and S. They create concept sets of (i) varying albedo: wherein the material color of the scene is varied (ii) varying illumination: where illumination is varied. From the definition of IID, Reflectance shall only be affected by albedo variations and not by illumination variations and vice-versa for Shading. They defined Concept Sensitivity Metric (CSM) to measure R-5 disentanglement and evaluate III methods post-hoc. Using our concept distillation framework, we extend their post-hoc quality evaluation method to ante-hoc training of the IID network to increase disentanglement between R and S. We train in different experimental settings wherein we only train the R branch (R only) and both and branches together (R&:S) with our concept loss in addition to the original loss [44] and report results in Tab. 7
point. As per definition, R and S are disentangled, with R invariant to
We choose the state-of -tthhartt CGIID 44 network as the baseline. The last layer of both R and S branches is used for concept distillation, and hence no prototypes are used. Following Li and Snavely 10 and SAW 37 datasets while we report results
44 we fine-tune over the CGIntrinsies 47 IIW
Table 7: IID performance on ARAP dataset: Inducing human-centered concepts like albedo-invariance of S and illumination invariance of R results in improved IID performance.
Model
MSE R L MSE S SSIM R
SSIM
Synthetic
Real-World
CSM R 1 CSM f
CSM R
CGIID 44
0.066
0.027
0.536
0.581
1.790
0.930
5,431
CGIID++
0.080
0.032
0.520
0.552
0.860
0.401
3,421
Ours (R only)
0.052
0.027
0.54
0.581
1.889
1.260
4.89
Ours (R & S)
0.059
0.028
0.538
0.586
6.040
1.023
78.46

---

GTR
Input Images Figure 7: Qualitative IID results: Our method uses complex concepts like albedo and illumination to enhance R and $ predictions that are illumination and albedo invariant respectively.
CGIID R
CGIID S
Ours R
Ours GT
over ARAP dataset which consists of realistic synthetic images. We also train the baseline model for additional epochs to get CGIID++ for fair comparison, Tab. shows different measures to compare two variations of our method with CGIID and CGIID++ on the ARAP dataset [[]. We see improvements in MSE and SSIM scores which compare dense-pixel wise correspondences CSM scores [23] which evaluate R-S disentanglement. Specifically, C'SMss measures albedo invariance of S and CSMR measures illumination invariance of R predictions. We report CSM scores in two concept set settings of Synthetic vs Real-World from Gupta et al. [23]. The improvement in MSE and SSIM scores appear minor quantitatively, but method performs significantly better in terms of CSM scores. It also observes superior performance qualitatively, as seen in (Fig. 7 Our R outputs less sensitive to illumination, with most of the illumination effects captured in S.
Discussion and Limitations: Our concept distillation framework can work on different classi- fication and reconstruction problems, as demonstrated. Our method work well in both zero-shot (with concept sets) and few-shot (with bias-conflicting samples) scenarios. Bias-confficting samples may not be easy to obtain for many real-world applications. Our required user-pun gver-provided concept samples can incur annotation costs, though concept samples are usually casier to obtain than bias-conflicting samples. When neither bias-conflicting samples user-provided concept sets available, concept discovery methods like ACE [21] could be used, ACE discovers concepts used by the model by image super-pixel clustering. Automatic bias detection methods like Bahadori and Heckerman [6] can be used to discover or synthesize bias-conflicting samples for our method. Our method can also be used induce prior knowledge into complex reconsiruction/generaioo problems, as we demonstrated with IID. The dependence on the teacher for conceptual knowledge could be another drawback of our method, as with all distillation frameworks (25
Societal Impact: Our method can be used to reduce or increase the bias by the appropriate use the loss Lc. Like all debiasing methods, it thus has the potential to be misused to introduce calculated bias into the models.
Conclusions
We presented concept distillation framework that can leverage human-centered explanations and the conceptual knowledge pre-trained teacher distill explanations into a student model. Our method can desensitize MI. models to selected concepts by perturbing the activations away from the CAV direction without modifying its underlying architecture. We presented results on multiple classification problems. We also showed how prior knowledge can be induced into the real-world IID problem. In future, we would like to extend our work to exploit automatic bias detection and concept-set definition. Our approach also has potential to be applied to domain generalization and multitask learning problems.
We thank Prof. Vineeth Balasubramanian of IIT Hyderabad for useful insights and discussion. We would also like to thank the four anonymous reviewers of Neurips, 2023 for detailed discussions and comments which helped us improve the paper.
Acknowledgements:
10

---

References
C. Aditya, S. Anirban, D. Abhishek, and H. Prantik. Grad-cam++: improved visual explanations for deep convolutional networks. arxiv 2018, arXiv preprint urX6: 2018.
[2] Akhtar. A survey of explainable in deep visual modeling: Methods and metrics. ArXiv, 02x/2301.13445, 2023.
[3] A. Akula, S. Wang. and S.-C. Zhu. Cocox: Generating conceptual and counterfactual explanations via fault-lines. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 2594-26011 2601, 2020.
[4] D. Alvarez Melis and T. Jaakkola. Towards robust interpretability with self-cxplaining neural networks. Adv. Neural Inform. Process. Syst., 2018
[5] C. J. Anders, Weber, D. Neumann, W. Samek, K.-R. Müller, and S. Lapuschkin. Finding and removing clever hans: Using explanation methods to debug and improve deep models. Information Fusion, 77: 261-295, 2022.
[6] M. T. Bahadori and D. E. Heckerman. Debiasing concept-based explanations with causal analysis. arXiv preprint 20Xi: 1500, 2020.
[7] H. G. Barrow and J. M. Tenenbaum. Recovering intrinsic scene characteristics from images. 1978.
[8] K. Beckh, S. Müller, M. Jakobs, V. Toborek, H. Tan, R. Fischer, P. Welke, S. Houben, and L. von Rueden. Sok: Harnessing prior knowledge for explainable machine learning: An overview. In First IEEE Conference on Secure and Trustworthy Machine Learning, 2023.
[9] S. Bell, K. Bala, and N. Snaveey.. Intrinsic images in the wild. ACM Transactions on Graphics (TOG), 33:1 12, 2014.
[10] S. Bell, K. Bala, and N. Snavely. Intrinsic images in the wild. ACM Transactions on Graphics (TOG), 33 (4):1-12, 2014.
[11] N. Bonneel, B. Kovacs, S. Paris, and K. Bala. Intrinsic Decompositions for Image Editing. Computer Graphics Forum (Furographics State of The Art Report), 2017.
[12] M. Caron, Bojanowski, A. Joulin, and M. Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European conference on computer vision (ECCV), pages 132-149, 2018.
[13] Caron, H. Touvron, Misra, H. Jégou, Mairal, Bojanowski, and Joulin. Emerging properties in self-supervised vision transformers. In Proceedings the IEFF/CVF international conference on computer vision. pages 6660-9660, 2021.
[14] D. T. Chang. 1rXiv.1811.06622. 2018
Concept-oriented deep learning: Generative concept representations. arXiv preprint
[15] M. Du, N. Liu, and X. Hu. Techniques for interpretable machine learning. Communications of the ACM. 63(1):68-77 2019.
[16] G. Erion, D. Janizek, P. Sturmfels, S. M. Lundberg, and S.-I. Lee. Improving performance of deep learning models with axiomatic altribution priors and expected gradients. Nature machine intelligence, 3 (7):620-6) 631, 2021.
[17] R. C. Fong and A. Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In Proceedings of the IEEE international conference computer vision. pages 3429-3437, 2017.
[18] Y. Gao, S. Gu, J. Jiang, S. R. Hong, D. Yu, and L. Zhao. Going beyond xai: A systematic survey for explanation-guia guided learning. arXiv preprint orXiv:2212.03954, 2022.
[19] E. Garces, C. Rodriguez- Paadig, D. Casas, and Lopez- Moreno. survey on intrinsic images: Delving deep into lambert and beyond. Int. J. Comput. Vision, 130(3):836-866 2022.
[20] A. Ghandeharioun, B. Kim, C.-L. Li, B. Jou, B. Eoff, and R. W. Picard. Dissect: Disentangled simultancous explanations via concept traversals. arXiv preprint 105. 2021.
[21] A. Ghorbani, J. Wexler, J. Zou, and B. Kim. Towards automatic concept- explanations. arXiv preprint arXiv: 100V:1902.031299 2019.

---

[22] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv preprini arXiv:1412.6572 2014.
[23] A. Gupta, S. Saini, and P. J. Narayanan. Interpreting intrinsic image decomposition using concept activations. In Proceedings of the Thirteenth Indian Conference on Computer Vision, Graphics and Image Processing, ICVGIP '22, New York, NY. USA. 2023. Association for Computing Machinery. ISBN 9781 9781450398220. doi: 0111145//571600355116
[24] P. Hasc and M. Bansal. When can models learn from explanations? a formal framework for understanding the roles of explanation data, arXiv preprint arXiv:2102.02201, 2021.
[25] G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in neural network. arXiv preprint 2015.
[26] G. Hinton and R. R. Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 31305778):500-550,2220 2006.
[27] P. Hitzler and Sarker. Artificial Intelligence: The State of the Art, (42(337):2,2022)
concept explanations for neural networks. Neuro-Symbolic
[28] L. Holmberg, P. Davidsson, and P. Linde, Mapping knowledge representations to concepts: A review and new perspectives. ArXiv, abs/2301 ,00189,,10000 2022.
[29] S. Jain, H. Lawrence, A. Moitra, and A. Madry. Distilling model failures as directions in latent space. arXiv preprint arXiv:2206, 14754, 2022.
[30] D. Kazhdan, B. Dimanov, M. Jamnik, P. Liö, and A. Weller. Now you (eme): concept-based model extraction. arXiv preprint 200.:::0000.223333 2020.
[31] M. Keswani, S. Ramakrishnan, N. Reddy, and V. N. Balasubramanian. Proto2proto: Can you recognize the car, the way do? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10233-10243, 2022.
[32] B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler, F. Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pages 2668 2677. PMLR, 2018
[33] F. Kim, S. Kim, M. Seo, and S. Yoon. Xprotonet: Diagnosis in chest radiography with global and local explanations. 2021 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15714-15723, 2021.
[34] E. Kim, Lee, and J. Choo. Biaswap: Removing dataset bias with bias-tailored swapping augmentation. In Proceedingy of the IEEE/CVF International Conference on Computer Vision, pages 14992-15001, 2021.
[35] P. W. Koh, T. Nguyen, Y. S. Tang. S. Mussmann, E. Pierson, B. Kim, and P. Liang. Concept bottleneck models. In International Conference on Machine Learning, pages 5338-5348. PMLR, 2020.
[36] B. Kovacs, S. Bell, N. Snavely, and K. Bala. Shading annotations in the wild. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 850-859, 2017.
[37] B. Kovacs, S. Bell, N. Snavely, and K. Bala. Shading annotations in the wild. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 6998-7007, 2017.
[38] M. L EH. Lightness and retinex theory. Opt. Soc. Am., 61()):1-11, 1971
[39] F. H. Land. The retinex theory of color vision. Scientific American, 237 6:108-28, 1977.
[40] J. Lee, E. Kim, J. Lee, Lee, and Choo. Learning debiased representation via disentangled feature augmentation. Advances Neural Information Processing Systems, 34:25123-25133, 2021.
[41] Li, Zhou, C. Xiong, and S. C. Hoi. Prototypical contrastive learning of unsupervised representations. arXiv preprint arXiv:2005.04990 049006 2020.
[42] Y. Li and N. Vasconcelos. Repair: Removing representation bias by dataset resampling. In Proceedings of
the JEEE/CVF conference on computer vision and pattern recognition, pages 9581, 2019.
[43] Li, L. Guo, and Y. Ge. Pseudo labels for unsupervised domain adaptation: A review. Electronics, 12 (15):3325, 2023.
12

---

[44] Z. Li and N. Snavely. Cgintrinsies: Better intrinsic image decomposition through physically-based rendering. In European Conference on Computer Vision (ECCV). 2018.
[45] P. Linardatos, V. Papastefanopoulos. and S. Kotsiantis. Explainable ai: A review of machine learning interpretability methods. Entropy, 13:1): 2020.
[46] Y. Ma, X. Feng, X. Jiang, Z. Xia, and J. Peng. Intrinsic image decomposition: A comprehensive review. In Image and Graphics: 9th International Conference, ICIG 2017, Shanghai, China, September 13-15, 2017, Revised Selected Papers, Part 9. pages 626-638. Springer, 2017.
[47] M. Moayeri. K. Rezaei, M. Sanjabi, and S. Feizi. Text-to-concept (and back) via cross-model alignment. arXiv preprint 230x:23300006866. 2023.
[48] I. Nassar, M. Hayat, E. Abbasnejad, H. Rezatofighi, and G. Haffari. Protocon: Pseudo-label refinement via online clustering and prototypical consistency for efficient semi-supervised learning. In Proceedings of the IEEE/CVF Conferencee on Computer Vision and Puttern Recognition, pages 11641-11650, 2023.
[49] C. Niu, H. Shan, and G. Wang- Spice: Semantic pseudo-labeling for image clustering. IEEE Transactions on Image Processing, 31:7264-7278, 2022.
[50] A. Radford, J. W. Kim, C. Hallacy, A, Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8788 8763. PMLR, 2021.
[51] L. Rieger, C. Singh, W. Murdoch, and B. Yu. Interpretations are useful: penalizing explanations to align neural networks with prior knowledge. In International conference on machine learning, pages 81 $116-8126. PMLR, 2020.
[52] A. S. Ross, M. C. Hughes, and F. Doshi-Veke Velez. Right for the right reasons: Training differentiable models by constraining their explanations. arXiv preprint 170v::7703.3717,, 2017.
[53] G. Saha and K. Roy. Saliency guided experience packing for replay in continual learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5273-5283, 2023.
[54] W. Samek, T. Wiegand, and K.-R. Müller. Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models. arXiv preprint arXiv:/708$ 08296, 2017.
[55] Y. Sawada and K. Nakamura, C-senn: Contrastive self-cxplaining neural network. ArXiv, abs/2206.09575, 2022.
[56] Y. Sawada and K. Nakamura. Concept bottleneck model with additional unsupervised concepts. IEFE Access, 10:41758-41766 2022.
[57] J. Schrouff, S. Baur, S. Hou, D. Mineu, E. Loreaux, R. Blanes, J. Wexler, A. Karthikesalingam, and B. Kim. Best of both worlds: local and global explanations with auman-understandable concepts. arXiv preprint vrXiiv2206..0 106.08641, 2021.
[58] G. Schwalbe. Concept embedding analysis: review, arXiv preprint arXiv:2203. 13909,, 2022.
[59] R. R. Selvaraju. M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explanations from deep networks via gradient based localization. In Proceedings of the IEEE international conference on computer vision, pages 618 8-626, 2017.
[60] D. Smilkov, N. Thorat, B. Kim, F. Viégas, and M. Wattenberg. Smoothgrad: removing noise by adding noise, arXiv preprint Xiv: / 706.03825, 2017.
[61] Y. Song, K. Shyn. and K.-s. Kim. Img2tab: Automatic class relevant concept discovery from stylegan features for explainable image classification. arXiv preprint arXiv:2301.00 003344, 2023.
[62] R. Soni, N. Shah, T. Seng, and J. D. Moore. Adversarial tcav robust and effective interpretation of intermediate layers in neural networks. ArXiv, abs/2002.03549, 2020.
[63] K. Tanwisuth, X. Fan, H. Zheng, S. Zhang, H. Zhang, B. Chen, and M. Zhou. A prototype-orientee framework for unsupervised domain adaptation. Advances in Neural Information Processing Systems, 34: 17194-17208, 2021.
[64] E. Tartaglione, C. A. Barbano, and M. Grangetto. End: Entangling and disentangling deep representations for bias correction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 13508 13508-13517, 2021.
13

---

[65] S. Vojif and Т. Kliegr. Editable machine learning models? rule-based framework for user studies of explainability. Advances in Data Analysis and Classification, 14(4))785-79, 2020.
[66] M. Wang and W. Deng. Deep visual domain adaptation: A survey. Neurocoputting. 312:135-153, 2018.
[67] Y. Wang, Q. Yao, J. T. Kwok, and L. M. Ni. Generalizing from few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3):1334, 2020.
[68] L.. Weber, S. Lapuschkin, A. Binder, and W. Samek. Beyond explaining: Opportunities and challenges of xai sai-based model improvement. Information Fusion, 2022.
[69] M. Wu, M. Hughes, S. Parbhoo, M. Zazzi, V. Roth, and F, Doshi-Velez. Beyond sparsity: Tree regularization of deep models for interpretability. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018
[70] Y. Xian, C. H. Lampert, B. Schiele, and Z. Akata. Zero-shot learning comprehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysts and machine intelligence, 41(9): 2251-2265, 2018.
[71] M. Xue, Q Huang, H. Zhang, I.. Cheng, Song, M. Wu, and M. Song. Protopformer: Concentrating on pro- totypical parts vision transformers for interpretable image recognition. arXiv preprint arXiv:2208. 10431, 2022.
104311
[72] L. Yang, Huang, Guo, Y. Lin, and Zhao. A small-sample text classification model based on pseudo-label fusion clustering algorithm. Applied Sciences, 13(8):4716, 2023.
[73] R. Zhang, Madumal, Miller, K. Ehinger, and Rubinstein, Invertible saccpptbaseed explanations for cnn models with non-negative concept activation vectors. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1682-1 1690, 2021.
[74] Y. Zhang, P. Tiño, A. Lonardiss and K. Tang. A survey on neural network interpretahility. IEFE Transactions on Emerging Topics in Computational Intelligence, 5(5):726-742, 2021.
14


Reviews:
[{"review_id": "u7W3kDxtOe", "reviewer": "Reviewer_oT4y", "summary": "This paper aims to sensitize or desensitize a (smaller) student model with respect to user-provided high-level concepts, by leveraging a (larger) teacher model. Specifically, a supervised mapping model learns a bijection between some chosen latent space in the teacher model and the student model. Then, CAVs extracted from teacher model is mapped to the student model through the mapping model. Finally, the student model is trained to sensitize / desensitize with respect to the mapped CAVs by minimizing / maximizing the concept sensitivity score.", "questions": "* For each of the claims, there should be dedicated experiments to verify the claims, instead of bunching everything together and comparing this menagerie method with weaker 0-shot/few-shot baselines. Please justify the claims explicitly. \n* A proper ablation study on the design choices is a necessary component for such complex method. How does each component affect the result?\n* For the concept distillation loss, the entire student model is optimized for Eq (2) but the (mapped) CAV is kept constant. If layers prior to the latent space where the CAV lies is modified, CAV no longer represents the latent space because the input representation has changed. For example, suppose some concept happens to lie in the first dimension of the CAV latent space. If we optimize the entire model, the model could switch the concept and encode it in the second dimension instead. Thus, does it not make sense to only optimize for the model after the CAV layer?\n* There are no descriptions of what each entry in Table 6 represents. P9L288 states to \"See supplementary for detailed explanations\" but I haven't been able to find where them. Can the authors point me to the exact section for the ablation study?\n* Why exactly is $M^{-1}$ needed?\n* Can the authors elaborate on why concept distillation from a completely biased teacher AND the teacher shares the same architecture as the student model might still improve? The result is quite surprising considering the benefits of distillation has disappeared in this setting.", "limitations": "There is a small section in P9L299-302. The only limitation stated is \"The dependence on the teacher for conceptual knowledge could be another drawback of our method as with all distillation frameworks\". One limitation might be when CAV extracted from the teacher model fails to translate to the student model. When, where, and why does this happen?", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The ideas in this paper are quite creative and have plenty of potential if explored properly.\n* Working with new real-world datasets (i.e. IID experiment) in the interpretability domain is always welcomed, which makes interpretability methods more application grounded.", "weaknesses": "Overall my impression is the paper explores too many ideas at once and thus unfortunately fails to cover each one in detail. Design choices are not well-justified and phenomena are not well-understood.\n\n* I find this paper extremely hard to read. Ideas are not elaborated clearly when first introduced and pieces of information are scattered all over the paper. It takes about 3x the time to read this draft compared to other ones. Perhaps the writing needs to be cleaned up. \n    * Example 1: In P8 the ablation for ColorMNIST is placed after IID experiments which is confusing.\n* The proposed method is too complex and has too many claims that are not well substantiated.\n    * Claims:\n        * Optimizing for concept distillation loss (Eq 2) can effectively sensitize / desensitize\n        * Prototype-based loss captures sensitivity better at a global level and \"facilitates the use of any intermediate layer by serving as pseudo-class label\"(P5L168) which frankly I have no idea what this means.\n        * Large models learns CAV better than smaller ones. (the entire premise for adopting a teacher-student framework)\n* The compared baselines may not necessary be fair since the proposed method is a fully-supervised one that requires training of entire models while baselines are either zero-shot or few-shot. Should compare with fairness/debiasing methods that requires full training of the entire model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the appreciation of our idea’s potential and novelty including demonstration on real-world dataset on a reconstruction problem.\n\nThe primary goal of this work is to show that explainability ideas like concepts can be used in a loop to improve the models. We believe it is an important idea with many extensions as observed by all reviewers. We apologise if the writing is confusing and will improve it if given a chance. \n\nWe now address the concerns:\n* W1: We placed all experiments (the evidence) first followed with ablations at the end as done by most. We can include references earlier.\n * W2: We use concept distillation to desensitise a model to user desired concepts effectively. We introduced a novel concept loss and a prototype-based extension of CAVs. The loss can be used directly in the same model (no distil experiment in Tab 5) or using a teacher which has a richer knowledge of concepts. The prototypes transfer labels for loss computation in the chosen intermediate layers. These components, in our view, combine towards the core idea.\n     * W2.1: Effectiveness of concept distillation on different datasets and different tasks are shown. Improvement in accuracies of models in biased settings is substantial (Tables 1 column \"Ours+L\" vs Table 3 column \"Ours\"). The use of concepts leads to better generalization (Table 2, Figure 6). Finally, we demonstrated effectiveness in a challenging real world IID problem where none of the existing interpretability based model improvement methods could be applied ( Table 4, Figure 7)  \n     * W2.2 Prototypes have been used widely in the existing literature (L127, [Xue et al,  Keswani et al]) and have proved to capture class-wide characteristics well (L126, 127). They are a type of global method (L31 definition of proto-types, L64 for local vs global) by their very definition (as they are based on feature clusterings). Original CAV based sensitivity representations are limited to the last layer due to their loss based definition (explained in L131-138) and hence cannot be used to estimate the sensitivity to an intermediate layer. To extend CAV sensitivity to any layer $l$, we need a way to calculate model's sensitivity to it, which requires a loss to be calculated in layer $l$ (instead of the final layer where GT annotations are available). We circumvent this issue by estimating prototypes of the GT annotations at any layer (by forward passing and clustering the feature vectors based on labels in the chosen intermediate layer) (L168). This is a natural extension of prototype usage in literature.\n     * W2.3: We experimented with various architectures for teacher selection. Results in supplementary L27, L34 show larger model perform better. Please see no-distill in Table 5 as well as additional ablations in the common rebuttal and also in our response to Reviewer Gigg.\n* W3: *Zero-shot vs few-shot*\nWe are sorry we failed to clearly convey the experimental settings to the reviewer. Our method is zero-shot and not supervised. Zero-shot vs few shot are defined wrt training data (L195): zero-shot means the model has never seen the distribution of test samples (Xian et al.), while in few-shot, the model has access to some examples from the distribution of the test set (Wang et al.).  Our method works with abstract concept sets (different than any test sample) and is essentially zero-shot. Our comparison with other zero-shot interpretability methods like CDEP, RRR, EG (Table 1) is justified. Our method can also take advantage of few-shot examples if available as shown in results on BFFHQ (Table 3). In this experiment, our concept set comprises old/young men/women extracted from the test set (thus making it a few shot method as it has seen samples from test-distribution). We compare with few-shot methods EnD and DFA (Table 3) (L114-L117). This also makes our comparison fair in this category as well.\n *Full training of the entire model:* All of the above methods train the entire model with introduced losses/rules/samples. CDEP, RRR, EG have their loss terms added to the models like us while EnD and DFA train the entire model using their proposed algorithms using the ood samples as well. \n\n* RQ1: We hope we answered all the doubts of the reviewer in our responses above. We will try to rewrite better in the final version.\n* RQ2 and RQ4: We explored the impact of each design choice in several experiments in the ablations section table as well as in supplementary (Table 5, Table 6, Figure 8, L 278 and S:L34-45). We have provided a detailed explanation for the ablations of Table 6 in the common rebuttal above. We apologise this has not been explained more clearly in the paper.\n* RQ3: We will change the notation as mentioned in the common rebuttal too.\n* RQ5: When 100% biased teacher is used, the student gets 23% accuracy on ColorMNIST which is still low compared to 50.93% accuracy when a Dino ViTB/8 is used. \n* RL1: Theoretically, this can happen if the student's space for the chosen bottleneck layer does not have sufficient capacity to represent the concepts (e.g., Teacher's space is huge while students is very small for chosen bottleneck).  But this is unlikely to occur because the student classifier's activation space is expected to have the capacity to encode the detailed images from iid ood samples and perform classification or the task at hand. Concepts in general are supposed to be more abstract compared to these detailed images and should be encoded practically in the student space. This is what we found in our experiments.\n\nWe once again thank the reviewer for incisive comments and hope our explanation has made the reviewer positive about our work."}, {"review_id": "j5ckGRt9Oj", "reviewer": "Reviewer_gigg", "summary": "The paper presents the idea of concept distillation from a pretrained teacher to improve a student. They utilize the notion of concept activation vectors (CAV) as concept representations and adapt a pretrained student to sensitize or desensitize a student w.r.t a concept. They apply this idea for two applications to improve a model in ante-hoc fashion: (i) reducing biases in a student towards a concept, and (ii) incorporating prior knowledge", "questions": "Doubt #1\n\n1. (Methodological, major) I am still not convinced by the use of teacher for CAV estimation. The first supporting argument was in line 170-172 \"Direct CAV learning ... as the model may not have sufficiently rich comprehension of the relevant concepts\". But even when using teacher representations you map them to the student activation space. So if the mapping $M$ to student activation and back worked perfectly, would it defeat the point of the original argument? The only other supporting argument I could find was an experment on ColorMNIST (Tab. 5). It was nice to see but could it be just due to a specific student architecture. Does this observation generalize to other tasks and more complex students?\n\n2. Can the method be used to improve biases in the original teacher given that you know it has good representation for the concepts? I was assuming that to be able to perform self improvement without a teacher would be more useful. \n\n3. Does the students original activation space get disturbed due to the modifications in its updates or are its layers before the selected intermediate layer fixed? If yes, could it harm the concept distillation loss since the student representations change?\n\n4. (Experimental) Can the model be used to remove many biases simultaneously? This would really enhance the applicability and power of the method.\n\n5. (Minor) Are $M$ and $M^{-1}$ mappings really exact inverses? In that case, isn't $L_{M^{-1}}$ guaranteed to be 0? If yes, then what is the point of $L_{M^{-1}}$? If not, then the notation should be modified.\n\n6. (Minor) I am slightly dissappointed with only one non-MNIST data for debiasing and one for IID task. This is not the biggest concern for me since you do test your method in multiple different ways. And it certainly won't be the deciding factor in my score but nevertheless the impact would feel greater if one of the task was more extensive in its coverage of complex datasets. And I assume you want debiasing as the main task.", "limitations": "The authors do discuss in the main paper their limitations. They are also upfront about potential societal impact. I am pretty much satisfied with the discussion.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is reasonably well written\n\n2. The selection of baselines is quite extensive and the proposed method generally performs well\n\n3. The core idea of using CAV as representations to improve a model indeed has wide applicability. This is shown quite well by authors to a good extent", "weaknesses": "I unfortunately have multiple methodological and experimental issues with the. Please find them detailed in the \"Questions\" sections. The paper has lot of the hallmarks to be a useful and effective work, but I simply didn't enjoy reading it much as I should have, being constantly riddled with doubts in my mind. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad the reviewer appreciated our core idea and its wide applicability and about it having \"the hallmarks to be a useful and effective work.\" Our main focus is to explore the feasibility of using explainability ideas like concepts for model improvement in multiple use cases. We will improve the writing to convey this message more clearly in the revised draft and answer the specific questions below:\n* RQ1: In theory, it is true that if the mapping module ($M$) had a zero-loss, it could make the distillation case same as the case without distillation but this is not observed in our experiments due to two main reasons: 1) We use $M$ to map ONLY the conceptual knowledge as CAV and train it only for concept sets and not the training samples (L179, Algo 1 L2, Fig 3). 2) Due to major differences in the perceived notion of concepts in teacher and student networks and due to a simple $M$ (one upconv and downconv layer; S:52) the mse loss never goes to zero (e.g in colorMNIST it starts from ~11 and converges at ~5 for DiNO teacher to biased student alignment).  In our initial experiments, we had tried bigger architectures (resnet18+) for $M$ and found improved mapping losses but decreased student performances. We thank the reviewer for this question and will mention it in the paper.\n$M$ encodes an expert’s knowledge into the system via the provided concept sets,  quantifies this knowledge as CAV via a generalized teacher model trained on large amount of data, and thus helps in inducing it via distillation into the student model. This brings threefold advantages in our system: expert’s intuition, large model’s generality, and efficiency of distillation.\nThe advantage of distillation in colorMNIST is specifically highlighted in Table 5 with 'no-distill' ablation (also shown in additional ablations in RQ2.). Similar observation can be made for BFFHQ and IID (which have different student architectures, datasets biases, and tasks). Specifically, in MNIST variants we used a two layer convolution network (S:L63), while in BFFHQ, we experimented with ResNet18 as the student backbone (S:L65) while in IID we have a custom two-branch network (CGIID's network, L:269, S:L96). In all these cases, we observed distillation to be substantially more effective than the no-distill case.\n* RQ2: Yes, our method can be used to improve bias in the teacher network. We did the experiment where the distillation was not used, and the CAV's were learned in the same model and reported the result in no distill columns in Table 5 (also L285). This experiment was done with basic concept loss Lc and without the use of proto-types i.e. 'no-distll minus prototypes’. We report an additional ablation 'no-distill plus prototypes' accuracies of our model below.\n| Dataset       | Vanilla        | Without teacher | With teacher |\n|---------------|----------------|----------------|---------------------|\n| ColorMNIST    | 0.1            | 26.97          | 41.83               |\n| TextureMNIST  | 11.23          | 38.72          | 48.82               |\n| BFFHQ         |       56.87      | 59.4           | 63.00                 |\n\nThe experiment shows the effectiveness of having the teacher model and highlights the advantage of the rest of the framework, which occurs due to the threefold advantages described above.\n* RQ3: Yes, the student's original space might get changed but it does not impact the overall performance. \nTo verify this claim, we did an experiment by updating CAVs in the student after every few training iterations (50, 100, 200, etc.). We observed no significant performance changes due to this. We found that this is because the student's activation space changes gradually relative to the distillation which converges very fast (in less than an epoch i.e 200-500 iterations). \n\n* RQ4: Yes, this is possible. Theoretically, we see no reasons our method cannot be used to remove many biases simultaneously. Training for IID (both R-S training experiment, L265) we have two concepts: Reflectance's illumination-invariance and Shading's color-invariance, which are optimized simultaneously in the model. Exploring the same for more than two concepts and in multiple problems is one possible extension that we plan to explore in subsequent extensions of our work.\n* RQ5: No, they may not be the exact inverses. They were meant to denote the reversed mapping spaces of the encoder and decoder only (L177). Sorry for the confusion and we will change the notation to Encoder E and Decoder D.\n* RQ6: Instead of opting for multiple concepts for a relatively well-defined problem in classification, we focussed on more complex concepts like illumination-invariance in an ill-defined problem like IID to gauge the utility of the technique in a more real-world scenario.  \nWe would like to point our that MNIST datasets have been chosen because of two reasons: \n    1) Synthetic biases in MNIST datasets are designed to be extreme and hence more challenging than a real world scenario. Furthermore, the introduction of abstract biases in a well-understood dataset helps gauge the impact of the technique in a better way.\n    2) MNIST allows comparisons with other contemporary works, which all report using the same methodology. \n\n   Apart from this, we would ike to point that we have introduced a more challenging TextureMNIST dataset and also report results on BFFHQ, which has natural images. In case of IID, the IIW benchmark is the only large scale human annotated dataset (with rest of the datasets like Sintel, being synthetically rendered with known issues in the IID evaluation). As pointed out by the reviewer, we focused on gathering evidence for our method’s utility for multiple tasks and various types of biases rather than multiple datasets for a similar task. If needed, we can include additional results in the revision.\n\nWe once again thank the reviewer for the appreciation of the work and its utility and hope his/her reservations are addressed sufficiently.\n"}, {"review_id": "NSXAnIHtIf", "reviewer": "Reviewer_FYuP", "summary": "This paper proposes a methodology for training a model to sensitize or desensitize a specific concept. Particularly, it introduces a concept distillation loss that utilizes Concept Activation Vectors (CAVs) derived from a high-performing teacher classifier with abundant knowledge of the concept, aiming to reduce concept activation in that direction. Additionally, it introduces a prediction loss term based on class prototypes to incorporate global features. The paper also employs an autoencoder to minimize the discrepancy between the latent spaces of the teacher and the student. Through experiments on various biased datasets, it demonstrates the robustness of the concept-distilled student model against biases.", "questions": "1) The description of the training pipeline seems somewhat unclear. Has the student model already been trained to achieve a high classification accuracy, and is it now receiving concept knowledge through the prototype loss? If so, wouldn't the latent space of the student model change, requiring constant updates to the module that maps the latent space between the teacher and student models?\n\n2) Regarding the motivation and implementation of concept distillation loss, could it potentially increase the sensitivity to orthogonal concepts as an unexpected side effect?\n\n3) In experiments where there are multiple concepts, how is the loss defined when trying to eliminate bias for all these concepts? Is it defined as an average or debiased one by one?\n\n4) Is it possible to experiment with datasets that have a wider range of concepts? For example, using the Broden concept [1] with ImageNet?\n\n5) Is it feasible to conduct experiments where the activation in a trained teacher model itself is controlled rather than distillation to student?\n\n6) Can additional evaluation be provided to assess how well the autoencoder used for latent space mapping is performing?\n\n[1] Bau, David, et al. \"Network dissection: Quantifying interpretability of deep visual representations.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.", "limitations": "It seems that there are no particular specific limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "To the best of my knowledge, this paper is the first to propose the distillation of concept information from a well-trained teacher to a student model. The experimental results provide clear verification that the proposed approach effectively reduces the spurious correlations learned by deep neural networks (DNNs). One particularly interesting result is that even when the teacher model is biased by 100%, the student model still exhibits a reduction in bias. This finding highlights the effectiveness of the proposed methodology in mitigating bias in the student model.", "weaknesses": "It seems that one limitation is the use of only simple datasets in the experiments and another is that the overall pipeline is somewhat unclear (please refer to the first question of below section), but other than that, there don't appear to be any specific weakness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for his comments and efforts. The observation of the effectiveness of our method in 100% biased teacher is certainly an interesting point. As also explained in our response to Reviewer RoT4y,  we believe this happens as we are defining the concepts using our concept sets (different from the biased training samples) and explicitly making the gradients orthogonal to the CAV during learning. \n\n> Response to Weakness on usage of datasets:\n\nWe would like to point that the contemporary literature on debiasing (CDEP, RRR, EG, DFA, EnD, etc) also show results on such datasets (ColorMNIST, DecoyMNIST, BFFHQ). This is because synthetic datasets like ColorMNIST and DecoyMNIST, have extreme and well-defined biases, which are relatively difficult to remove and allow proper analysis of the de-baising method. We additionally introduced a new dataset (TextureMNIST) which is comparatively more challenging and has more real-world bias (textures). Textures being an amalgam of colors and shapes, encompass both scenarios, and CNNs tend to have this bias often as pointed by Geirhos et al.\nFurthermore, Prior-induction in IID using concepts is a novel task attempted in this field. This requires (de)sensitizing to ambiguous and subtle concepts like reflectance-invariance and illumination-invariance. Our method not only effectively (de)sensitizes the model towards the required concepts (Table 4, Figure 7), it does it in a very complex and large network: CGIID (L269, S:L96). \n\nGeirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F. A., & Brendel, W. (2018). ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv:1811.12231.\n\nWe will now answer all the questions one by one below:\n*  RQ 1: *Pipeline:* We start with a pretrained student network which is trained on the biased training set like ColorMNIST (namely Vanilla network) and thus performs extremely poorly on the unbiased test set with out-of-distribution samples like MNIST.\nEg, Vanilla network gets ~0% accuracy in ColorMNIST, 52.84% in DecoyMNIST, and 11.23% in TextureMNIST, as shown in Table 1.\nWe map the pretrained teacher’s activation space to this biased student’s space ONLY for the concept set samples. Mapping step is needed only for CAV projection into the student’s space.\n\nIn summary, our pipeline is:\n0: Mapping teacher space to student space by training the Auto-encoder.\n1: CAV Learning using distilled teacher outputs to define concepts, etc.\n2: Training for the main task with Concept Distillation. In this step the top part of the system is not used at all\n3: Testing or application where the trained model is applied. Only the bottom half of the bottom box is used.\nWe will clarify the same in our paper description and algorithm.\n\n*Latent Space Changes:* Yes, the latent space indeed changes gradually with the student’s training but as our distillation converges very fast (in less than an epoch i.e 200-500 iterations), this empirically never observed the drift in any of our experiments.  We tried updating CAV every few iterations (50, 100, 200, etc.), and it made no difference at all. Hence we chose constant CAV in all our experiments. \n\n* RQ 2: During the initial design phase of our method, we chose the concept and its opposite for CAV estimation (e.g. young women vs old women). We updated this to: concept vs random set, in order to specifically avoid the concern being raised here (e.g. young women vs women and men of all ages; see L244-249). This design choice addressed the possible issue of unexpected side-effects, and we empirically confirmed the same.\n* RQ 3: It can be defined either way. While we didn't experiment the effect of mutiple biases in classification problems explicitly, we did so in the case of IID. We tried two scenarios in our IID experiments:\n    1) Setting_1 (L265) wherein branch S was frozen while branch R was being trained for concept of illumination invariance and vice-versa. This represents the case of sequential bias removal.\n   2) In Setting_2 (L266) wherein both R and S branches were trained together along with their common backbone and losses were simply weighted and added.\nFor the IID problem, we did not observe significant performance differences. Exploring the same for more than two concepts and in multiple problems is one possible extension that we plan to explore in subsequent extension of our work.\n* RQ 4: We have tried to demonstrate the same with the IID problem (which represents a different task than classification and is trained using a large training set of CGIID with approx. 100k+images). Instead of opting for multiple concepts from [1] for a relatively well-defined problem in classification, we opted to focus on more ‘complex concepts’ like illumination-invariance in an ill-defined problem like IID to gauge the utility of the technique in a more real-world scenario.  If necessary, we can try to get additional results using the experiment suggested.\n* RQ 5: We performed a similar experiment where the distillation was not used and the CAV's were learned in the same model (without proto-types) and reported the result in no distill’ column in Table 5 (L285). We apologize for the unclear explanation and will improve the writing here.\nFor completion, we additionally report 'no distill' accuracies (with proto-types) of our model in our Common Response above.\n* RQ 6: It is unclear how mapping module can be standalone evaluated. We indirectly gauge its performance in the overall framework’s accuracy scores.\nOne probable way to evaluate the mapping module could be by observing it’s validation mse (we used a small validation set consisting of concept set samples for early stopping the training of mapping module autoencoder. The mse loss reduced by >50% after few epochs and then stabilized).\nWe welcome relevant suggestions and experiments for better evaluation."}, {"review_id": "QKLgv5UPAK", "reviewer": "Reviewer_YYKb", "summary": "The authors introduce an idea by which a student model can become (de) sensitive to some human-understandable concept in its decision-making process. They find CAVs in a teacher model and then transform those CAVs into the student model feature space and use orthogonal vectors to these CAVs to penalize/incentivize models to make decisions based on those concepts.\n\nThey improve over baselines on biassed datasets where a concept is known to be the underlying bias.\n", "questions": "My questions are found in the weakness part.", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ I like their idea of debiasing the student model based on human feedback on biased and unbiased samples.\n+ method shows superior improvement over existing results when evaluated on the biassed dataset.\n+ They visually validate that their model is focusing on more important parts of the images in decision-making.", "weaknesses": "+ I'd expect to see some more analysis of the student's model performance. For example, why does the model still get relatively bad accuracy on the color MNIST? is it still biased toward the colors? can we still debias this model and push to get around 100 accuracies?\nif not, what are the limitations?\n\n+ This is a minor weak point but I expect to see some recent but relevant related work in the paper. As the paper is considering an alignment between concepts in different models' features spaces, it reminded me of a recent paper \"Text-To-Concept (and Back) via Cross-Model Alignment\" where an alignment between features spaces is used to interpret a model's behavior in terms of human-concepts. The general framework is somehow similar to this paper.\nFurthermore, the idea of considering the normal vector orthogonal to the decision-making half plane is somehow relevant to this paper \"Distilling Model Failures as Directions in Latent Space\". I want the authors to consider those and other relevant papers.\n\n+ quality of figures should improve (Fig 4).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for very positive feedback. We will address the points below:\n\n>  Analysis on student model's performance \n\nThe model remains biased due to the severe bias (100%) in the training set, though we improve significantly over prior efforts. This demonstrates the effectiveness of using explainability measures like CAVs to improve a model. One way to completely debase a model on a chosen concept could be to re-destil iteratively, using results from previous iteration as input to the next. Bootstrapping models in such a way will be interesting and fruitful directions for the future.\n\n> Regarding Related work\n\nThank you for pointing to the recent papers. We will add them to final version if given a chance.\n\n\"Distilling Model Failures as Directions in Latent Space\" uses SVMs to find directions of bias in a shared image and language space (of CLIP), then trains using bias-conflicting samples generated by models like Dall-E. Training on bias conflicting samples might not always be feasible due to higher annotation and computation costs. One major difference is that their work proposes data augmentation as debiasing strategy whereas we directly manipulate the gradient vectors which is more interpretable.\n\n\"Text-To-Concept (and Back) via Cross-Model Alignment\" is a contemporary work (released after Neurips submission) and is very relevant. It maps the activation spaces of two different models using the CLIP latent space similarity. Due to the generality of the CLIP latent space, this approach is useful to encode certain concepts like ‘cat, dog, man’ etc, but it is not clear how it will work on abstract concepts with ambiguous definitions like ‘shading’ and ‘reflectance’ as seen in the IID problem. We are glad to see such current research and think our work will interest many more.\n\n> quality of figures should improve (Fig 4).\n\nWe will improve the resolution and clarify the details in all the figures. Thank you!\n\n\n"}]

Author rebuttal:
We thank reviewers for valuable suggestions and feedback. We are glad they acknowledged the potential of our work. Our primary goal is to show explainability ideas like concepts can be used in a loop to improve the models. This is an important idea with many future directions as observed by reviewers. The summary of our pipeline (Fig 3 and Alg 1):
* Step 0: Mapping teacher space to student space by training the autoencoder.
* Step 1: CAV Learning using distilled teacher outputs to define concepts.
* Step 2: Training the main task with Concept Distillation. The top part of Fig 3 is not used
* Step 3: Testing where the trained model is applied. Only the bottom half and classification head are used.

We address common concerns here. Detailed responses are included against reviews.

**Mapping module Notation & Training:**
* M and M-1 notation are meant to show spaces of encoder and decoder only (L177). They are not exact inverses.  We will change it to Encoder E and Decoder D. 
* We use the mapping module ONLY for the concepts while training for CAV. It has no role in later steps like concept distillation (L179, Algo 1 L2, Fig 3).

**Ablations:**
Several ablations to justify our design choices (Tabs 5, 6; Fig 8). We tried our concept loss formulation in no distillation setting wherein the CAVs are learned in the same student model (table 5, no distill). The improvement in accuracy goes from 0% in base vanilla network trained with Lo to 9.96% when Lc (eq 2, L147) is added during the training. Sensitivity used for Lc  is calculated as gradient of loss (eq 2 L147) in our experiments. We tried by calculating sensitivity as the gradient of logit (second column of table 6, "logit") and found the former works better.  We also ablate using a variant where we compute our concept distillation loss using last layers loss directly (the way Kim et al[28] calculated sensitivity) instead of using our proposed proto-type based loss, i.e. calculation of Lc, using Lo instead of Lp described in L155 (Lo column). This shows the effectiveness of using mean class representations (proto-types). 

We ablate using a fixed set of prototypes instead of varying them (Lp, fixed proto) and found the performance to reach 40.02%, by varying the proto-types according to L164 we achieve 41.83% accuracy which is a slight improvement over fixed proto-types case. Finally, we add a local loss (RRR loss term) and achieve 50.93% accuracy on colorMNIST. We also show results when i) KNN K in proto-type calculation is varied and found k=7 to work best (Figure 8 Right) ii) number of images (# imgs) in the concept set are varied and we observe a peak in # imgs = 150 which is chosen as the KNN K and # imgs in our experiments (L189, S:61, S:74)
We apologize for the unclear ablations and will add them to the paper to make them clear.

**No Distillation:**
We showed an ablation in main paper (Tab 5 L285)) where the distillation was not used and the CAV's were learned in the same model (without proto-types) and reported the result in 'no distill’ column in Tab 5 (L285) which is 9.96%. We additionally report 'no distill’ accuracies (with proto-types) of our model below:
| Dataset       | Vanilla        | Without teacher | With teacher |
|---------------|----------------|----------------|---------------------|
| ColorMNIST    | 0.1            | 26.97          | 41.83               |
| TextureMNIST  | 11.23          | 38.72          | 48.82               |
| BFFHQ         |       56.87      | 59.4           | 63.00                 |

These results are much better than Vanilla network even without teacher but much better with it.

**Latent Space:**
The student's original space might get changed with training updates but it does not impact the overall performance. To verify this claim, we did experiments by updating CAVs in the student after every few training iterations (50, 100, 200, etc.). We observed no significant performance changes due to this. This may be because the student's activation space changes gradually relative to the distillation which converges very fast (in less than an epoch i.e 200-500 iterations).

**Datasets:**
Instead of opting for multiple concepts for a relatively well-defined problem in classification, we opted to focus on more complex concepts like illumination-invariance in an ill-defined problem like IID to gauge the utility of technique in a more real-world scenario. MNIST datasets have been chosen because of two reasons: 
1) Synthetic biases in MNIST datasets are designed to be extreme and hence more challenging than a real world scenario. Introduction of abstract biases in a well-understood dataset helps gauge the impact of the technique in a better way.
2) MNIST allows comparisons with other contemporary works, which all report using the same methodology. 
Apart from this we introduced a more challenging TextureMNIST dataset and also report results on BFFHQ which has natural images. In case of IID, IIW benchmark is the only large scale human annotated dataset (with rest of the datasets like Sintel being synthetically rendered with known issues in the IID evaluation).

We will be making the above more clear in the paper. Additionally, we will improvise the figures and pipeline description. 

**Common Citations:**
* M. Xue, Q. Huang, H. Zhang, L. Cheng, J. Song, M. Wu, and M. Song. Protopformer: Concentrating on pro-totypical parts in vision transformers for interpretable image recognition. 
* M. Keswani, S. Ramakrishnan, N. Reddy, and V. N. Balasubramanian. Proto2proto: Can you recognize the car, the way i do?
* Xian, Y., Lampert, C. H., Schiele, B., & Akata, Z. (2018). Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly.
* Wang, Y., Yao, Q., Kwok, J. T., & Ni, L. M. (2020). Generalizing from a few examples: A survey on few-shot learning. 

(Note) Notations for rebuttal: S:LXX denotes supplementary Line XX while LXX denotes main paper lines XX.

Meta-review: After discussion, three of four reviewers recommend acceptance. The remaining reviewer has not updated their score after extensive discussion with the authors, but appears satisfied with the responses. The AC see no grounds to overturn this consensus. Authors are strongly encouraged to include the additional ablations and clarifications from the discussions in the final version.
Decision: Accept (poster)