SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We show results on two standard biased datasets (ColorMNIST [42] and DecoyMNIST [16]) and introduce a more challenging TextureMNIST dataset for quantitative evaluations. We also experimented on a real-world gender classification dataset BFFHQ [34] that is biased based on age. We compare with other state-of-the-art interpretable model improvement methods [51, 52, 16, 64, 40, 5]. Fig. 4 summarizes the datasets, their biases, and the concept sets used.
**Poisoned MNIST Datasets:** ColorMNIST [42] has MNIST digit classes mapped to a particular color in the training set [51]. The colors are reversed in the test set. The baseline CNN model (Base) trained on ColorMNIST gets  $0\%$  accuracy on the  $100\%$  poisoned test set, indicating that the model learned color shortcuts instead of digit shapes. We debias the model using a concept loss for color using color patches vs. random shapes (negative concept) to estimate CAV for color (Fig. 4). We show results comparison on the usage of teacher and prototypes component of our method in Tab. 1. As can be seen from Tab. 1, our method of using intermediate layer sensitivity via prototypes as described in subsection 3.2 yields better results. Similarly, usage of teacher (described in subsection 3.3) facilitates better student concept learning (also shown in CAV comparisons in supplementary). Our concept sensitive training not only improves student accuracy but observes evident reduction in TCAV scores [32] of bias concept as seen from Tab. 2.
Tab. 3 compares our results with the best zero-shot interpretability based methods, i.e., CDEP [51], RRR [52], and EG [16]. Our method improves the accuracy from  $31\%$  by CDEP to  $41.83\%$ , and further to  $50.93\%$  additionally with the local explanation loss from  $[52]^{*}$ .
Table 3: Comparison of the accuracy of our method with other zero-shot interpretable model-improvement methods. All methods require user-level intervention: Our method requires concept sets, while others (CDEP, RRR, EG) require user-provided rules.
Dataset	Bias	Base	CDEP[51]	RRR[52]	EG[16]	Ours w/o Teacher	Ours	Ours+L
ColorMNIST	Digit color	0.1	31.0	0.1	10.0	26.97	41.83	50.93±1.42
DecoyMNIST	Spatial patches	52.84	97.2	99.0	97.8	87.49	98.58	98.98±0.20
TextureMNIST	Digit textures	11.23	10.18	11.35	10.43	38.72	48.82	56.57±0.79
Figure 6: Left: Comparison of our method and ClArC [5] on  $20\%$  biased ColorMNIST. Right: GradCAM [59] visualizations (more red, more important) on (i) TextureMNIST (middle image) (ii) Extreme case when the color bias is in the background (bottom) instead of foreground (top). Our model focuses on the shape more than CDEP (which is more blue in the foreground).
Table 4: Our method improves the model using human-centered concepts and shows better generalization to different datasets, while CDEP, which uses pixel-wise color rules, cannot.
Test Dataset	ColorMNIST Trained	TextureMNIST Trained
Base	CDEP[51]	Ours+L	Base	CDEP[51]	Ours+L
Invert color	0.00	23.38	50.93	11.35	10.18	45.36
Random color	16.63	37.40	46.62	11.35	10.18	64.96
Random texture	15.76	28.66	32.30	11.35	10.18	56.57
Pixel-hard	15.87	33.11	38.88	11.35	10.18	61.29
GradCAM [59] visualizations in Fig. 6 show that our trained model focuses on highly relevant regions. Fig. 6 left compares our class-wise accuracy on  $20\%$  biased ColorMNIST with ClArC [5], a few-shot global method that uses CAVs for artifact removal. (ClArC results are directly reported from their main paper due to the unavailability of the public codebase.) ClArC learns separate CAVs for each class by separating the biased color digit images (e.g. red zeros) from the unbiased images (e.g. zeros in all colors). A feature space linear transformation is then applied to move input sample activations away/towards the learned CAV direction. Their class-specific CAVs definition can not be generalized to test sets with multiple classes. This results in a higher test accuracy variance as seen in Fig. 6. Further, as they learn CAVs in the biased model's activation space, a common concept set cannot be used across classes. DecoyMNIST [16] has class indicative gray patches on image boundary that biased models learn instead of the shape. We define concept sets as gray patches vs. random set (Fig. 4) and report them in Tab. 3 second row. All methods perform comparably on this task as the introduced bias does not directly corrupt the class intrinsic attributes (shape or color), making it easy to debias. TextureMNIST is a more challenging dataset that we have created for further research in the area*. Being an amalgam of colors and patterns, textures are more challenging as a biasing attribute. Our method improves the performance while others struggle on this task (Tab. 3 last row). Fig. 6 shows that our method can focus on the right aspects for this task.
Generalization to multiple biasing situations is another important aspect. Tab. 4 shows the performance on different types of biases by differently debiased models. Our method performs better than CDEP in all situations. Interestingly, our texture-debiased model performs better on color biases than CDEP debiased on the color bias! We believe this is because textures inherently contain color information, which our method can leverage efficiently. Our method effectively addresses an extreme case of color bias introduced in the background of ColorMNIST, a dataset originally biased in foreground color. By introducing the same color bias to the background and evaluating our ColorMNIST debiased models, we test the capability of our approach. In an ideal scenario where color debiasing is accurate, the model should prioritize shape over color, a result achieved by our method but not by CDEP as shown in Fig. 6.
BFFHQ [34] dataset is used for the gender classification problem. It consists of images of young women and old men. The model learns entangled age attributes along with gender and gets wrong predictions on the reversed test set i.e. old women and young men. We use the bias conflicting samples by Lee et al. [40] – specifically, old women vs. women and young men vs. men – as class-specific concept sets. We compare against recent debiasing methods EnD [64] and DFA [40]. Tab. 5 shows our method getting a comparable accuracy of  $63\%$ . We also tried other concept set combinations: (i) old vs. young (where both young and old concepts should not affect)  $\rightarrow$ $62.8\%$  accuracy, (ii) old vs. mix (men and women of all ages) and young vs. mix  $\rightarrow$ $62.8\%$  accuracy, (iii) old vs. random set (consisting of random internet images from [32]) and young vs. random  $\rightarrow$ $63\%$  accuracy. These experiments indicate the stability of our method to the concept set definitions. This proves our method can also work with bias-conflicting samples [40] and does not necessarily require concept sets (Tab. 5). We also experimented by training class-wise (for all women removing young bias
Table 5: Comparisons in few-shot setting: Our method is not limited to user-provided concept sets and can also work with bias-conflicting samples. We compare our accuracy over the BFFHQ dataset [34] with other few-shot debiasing methods.
Dataset	Bias	Base	EnD [64]	DFA [40]	Ours w/o Teacher	Ours
BFFHQ	Age	56.87±2.69	56.87±1.42	61.27±3.26	59.4	63±0.79
Table 6: Increasing the bias of the teacher on ColorMNIST reduces accuracy.
Teacher's Training Data Bias%	No Distil	5	10	25	75	90	100
Student Accuracy%	26.97	40.47	33.18	30.38	28.74	23.23	23.63
followed by men removing old bias and vice versa) vs training for all classes together (both men, women removing age bias as described above) and observed similar results, suggesting that our concept sensitive training is robust to class-wise or all class agnostic training. Local interpretable improvement methods like CDEP, RRR, and EG are not reported here as they cannot capture complex concepts like age due to their pixel-wise loss or rule-based nature.
Discussion No Distillation Case: We additionally show our method without teacher (Our Concept loss with CAVs learned directly in student) in all experiments as "Ours w/o Teacher" and find inferior performance when compared to Our method with Distillation. Bias in Teacher: We check the effect of bias in teacher by training it with varying fractions of OOD samples to bias them. Specifically, we use the same student architecture for the teacher. The teacher is trained on the ColorMNIST dataset with 5, 10, 25, 50, 75, and  $90\%$  biased color samples in the trainset (e.g.  $k\%$  bias indicates  $k\%$  red zeros). The resulting concept-distilled system is then tested on the standard  $100\%$  reverse color setting of ColorMNIST. Tab. 6 shows that concept distillation improves performance even with high teacher bias, though accuracy decreases with the increasing bias in teacher. Apparently,  $100\%$  bias in teacher in this setting of teacher with same architecture as the student is essentially CAV learning in same model case (No Distil). Here, the improvements are due to prototypes, and as can be seen, there is a slight degradation in performance of  $100\%$  bias vs No Distil (23.63 vs 26.97). This can be attributed to an error due to the mapping module.
4.2 Prior Knowledge Induction
Intrinsic Image Decomposition (IID) is an inverse rendering problem [7] based on the retinax theory [39], which suggests that an image  $I$  can be divided into Reflectance  $R$  and Shading  $S$  components such that  $I = R \cdot S$  at each pixel. Here  $R$  represents the material color or albedo, and  $S$  represents the scene illumination at a point. As per definition,  $R$  and  $S$  are disentangled, with  $R$  invariant to illumination and  $S$  invariant to albedo. As good ground truth for IID is hard to create, IID algorithms are evaluated on synthetic data or using some sparse manual annotations [9, 36]. Gupta et al. [23] use CAV-based sensitivity scores to measure the disentanglement of  $R$  and  $S$ . They create concept sets of  $(i)$  varying albedo: wherein the material color of the scene is varied  $(ii)$  varying illumination: where illumination is varied. From the definition of IID, Reflectance shall only be affected by albedo variations and not by illumination variations and vice-versa for Shading. They defined Concept Sensitivity Metric (CSM) to measure  $R-S$  disentanglement and evaluate IID methods post-hoc. Using our concept distillation framework, we extend their post-hoc quality evaluation method to ante-hoc training of the IID network to increase disentanglement between  $R$  and  $S$ . We train in different experimental settings wherein we only train the  $R$  branch  $(R$  only) and both  $R$  and  $S$  branches together  $(R&S)$  with our concept loss in addition to the original loss [44] and report results in Tab. 7.
We choose the state-of-the-art CGIID [44] network as the baseline. The last layer of both  $R$  and  $S$  branches is used for concept distillation, and hence no prototypes are used. Following Li and Snavely [44], we fine-tune over the CGIIntrinsics [44], IIW [10], and SAW [37] datasets while we report results
Table 7: IID performance on ARAP dataset: Inducing human-centered concepts like albedo-invariance of S and illumination invariance of R results in improved IID performance.
Model	MSE R ↓	MSE S ↓	SSIM R ↑	SSIM S ↑	Synthetic	Real-World CSM R ↑
CSM R ↑	CSM S ↑
CGIID [44]	0.066	0.027	0.536	0.581	1.790	0.930	5.431
CGIID++	0.080	0.032	0.520	0.552	0.860	0.401	3.421
Ours (R only)	0.052	0.027	0.54	0.581	1.889	1.260	4.89
Ours (R & S)	0.059	0.028	0.538	0.586	6.040	1.023	78.46
Figure 7: Qualitative IID results: Our method uses complex concepts like albedo and illumination to enhance  $\hat{R}$  and  $\hat{S}$  predictions that are illumination and albedo invariant respectively.
over ARAP dataset [11], which consists of realistic synthetic images. We also train the baseline model for additional epochs to get CGIID++ for a fair comparison. Tab. 7 shows different measures to compare two variations of our method with CGIID and CGIID++ on the ARAP dataset [11]. We see improvements in MSE and SSIM scores which compare dense-pixel wise correspondences CSM scores [23] which evaluate R-S disentanglement. Specifically,  $CSM_{S}$  measures albedo invariance of S and  $CSM_{R}$  measures illumination invariance of R predictions. We report CSM scores in two concept set settings of Synthetic vs Real-World from Gupta et al. [23]. The improvement in MSE and SSIM scores appear minor quantitatively, but our method performs significantly better in terms of CSM scores. It also observes superior performance qualitatively, as seen in (Fig. 7)*. Our  $R$  outputs are less sensitive to illumination, with most of the illumination effects captured in  $S$ .
Discussion and Limitations: Our concept distillation framework can work on different classification and reconstruction problems, as we demonstrated. Our method can work well in both zero-shot (with concept sets) and few-shot (with bias-conflicting samples) scenarios. Bias-conflicting samples may not be easy to obtain for many real-world applications. Our required user-provided concept samples can incur annotation costs, though concept samples are usually easier to obtain than bias-conflicting samples. When neither bias-conflicting samples nor user-provided concept sets are available, concept discovery methods like ACE [21] could be used. ACE discovers concepts used by the model by image super-pixel clustering. Automatic bias detection methods like Bahadori and Heckerman [6] can be used to discover or synthesize bias-conflicting samples for our method. Our method can also be used to induce prior knowledge into complex reconstruction/generation problems, as we demonstrated with IID. The dependence on the teacher for conceptual knowledge could be another drawback of our method, as with all distillation frameworks [25].
Societal Impact: Our method can be used to reduce or increase the bias by the appropriate use of the loss  $L_{C}$ . Like all debiasing methods, it thus has the potential to be misused to introduce calculated bias into the models.
We presented a concept distillation framework that can leverage human-centered explanations and the conceptual knowledge of a pre-trained teacher to distill explanations into a student model. Our method can desensitize ML models to selected concepts by perturbing the activations away from the CAV direction without modifying its underlying architecture. We presented results on multiple classification problems. We also showed how prior knowledge can be induced into the real-world IID problem. In future, we would like to extend our work to exploit automatic bias detection and concept-set definition. Our approach also has potential to be applied to domain generalization and multitask learning problems.
Acknowledgements: We thank Prof. Vineeth Balasubramanian of IIT Hyderabad for useful insights and discussion. We would also like to thank the four anonymous reviewers of Neurips, 2023 for detailed discussions and comments which helped us improve the paper.
[65] S. Vojíř and T. Kliegr. Editable machine learning models? a rule-based framework for user studies of explainability. Advances in Data Analysis and Classification, 14(4):785-799, 2020.
[66] M. Wang and W. Deng. Deep visual domain adaptation: A survey. Neurocomputing, 312:135-153, 2018.
[67] Y. Wang, Q. Yao, J. T. Kwok, and L. M. Ni. Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3):1-34, 2020.
[68] L. Weber, S. Lapuschkin, A. Binder, and W. Samek. Beyond explaining: Opportunities and challenges of xai-based model improvement. Information Fusion, 2022.
[69] M. Wu, M. Hughes, S. Parbhoo, M. Zazzi, V. Roth, and F. Doshi-Velez. Beyond sparsity: Tree regularization of deep models for interpretability. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[70] Y. Xian, C. H. Lampert, B. Schiele, and Z. Akata. Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis and machine intelligence, 41(9): 2251-2265, 2018.
[71] M. Xue, Q. Huang, H. Zhang, L. Cheng, J. Song, M. Wu, and M. Song. Protopformer: Concentrating on prototypical parts in vision transformers for interpretable image recognition. arXiv preprint arXiv:2208.10431, 2022.
[72] L. Yang, B. Huang, S. Guo, Y. Lin, and T. Zhao. A small-sample text classification model based on pseudo-label fusion clustering algorithm. Applied Sciences, 13(8):4716, 2023.
[73] R. Zhang, P. Madumal, T. Miller, K. A. Ehinger, and B. I. Rubinstein. Invertible concept-based explanations for cnn models with non-negative concept activation vectors. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11682-11690, 2021.
[74] Y. Zhang, P. Tiño, A. Leonardis, and K. Tang. A survey on neural network interpretability. IEEE Transactions on Emerging Topics in Computational Intelligence, 5(5):726-742, 2021.