# Advancing Model Explainability: Visual Concept Knowledge Distillation for Concept Bottleneck Models
## 1 Introduction  
Mission-critical systems, such as those in healthcare, autonomous driving, and defense, often demand reliable predictions. These systems also face hardware limitations, necessitating lightweight models rather than cumbersome ones. Knowledge distillation (KD) has emerged as a promising method for crafting lightweight models. By transferring knowledge from a cumbersome model (teacher) to a smaller, simpler model (student), KD empowers the student model to attain comparable performance with reduced computational complexity [1]. However, model compression using KD poses challenges in comprehending the network's decision-making process in relation to the teacher model [2]. While post hoc methods are commonly employed to grasp the decision-making process, they occasionally yield unintended explanations [3]. Despite the increasing volume of research literature on applying compressed models, discussions on the validity of these models in terms of explainability are rare. In contrast to post hoc methods, an ante hoc approach exists that learns using human-understandable domain knowledge. This approach mitigates misinterpretation and distortion of explanations by integrating human-understandable domain knowledge during learning. A notable interpretable model among ante hoc approaches is the concept bottleneck model (CBM) [4], which predicts human-understandable concepts upon receiving input and subsequently predicts the final task based on these inferred concepts. CBMs can be integrated with any end-to-end model, offering inherent interpretability that is absent in standard CNN models. The primary advantage of a CBM lies in its ability to provide intuitive interpretation through learning human-level concepts. In this study, we emphasize the inherent interpretability of CBM and integrate them with knowledge KD to address the limitations of post hoc interpretability methods, aiming for explainability, as illustrated in Figure 1. Specifically, we focus on image classification tasks to demonstrate the effectiveness of our approach. To our knowledge, it has not yet been demonstrated that KD can be used to train CBMs. Therefore, we investigate the effectiveness of KD in training three types of bottleneck models, as proposed by [4], when combined with KD. We introduce the representation of implicit knowledge through visual concept knowledge distillation (VICO-KD) to transfer the teacher's visual concept knowledge to the student while preserving the locality characteristics of the CBM. Our proposed VICO-KD applies to image classification models, leveraging explicit concepts as is traditional for CBMs and providing correct implicit concepts to users. This results in a transparent model that can accurately classify images while highlighting the visual evidence underlying its decision, presenting inferences in a user-friendly manner. This transparency is beneficial for understanding the model's learning process, diagnosing errors, and building trust in the model's predictions. We demonstrate improvements of 10.65%, 2.9%, and 0.3% on the CUB-200-2011 (CUB) [5], Animals with Attributes2 (AwA2) [6], and Large-scale CelebFaces Attributes (CelebA) [7] datasets, respectively, compared to Vanilla-KD, using benchmark datasets with concept annotations. When visualizing and comparing CBMs that have used Vanilla-KD for learning and saliency maps, we confirm that they learn visual concepts similar to the teacher CBM. This proves our method can teach simple, explicit, and complex visual concepts. We also validate that our approach is competitive with existing concept models. Finally, we perform an intervention, a critical function of CBMs, on the student trained by VICO-KD and verify that it performs even better than the teacher. The main contributions of this study are as follows:  
Figure 1: Enhancing interpretability and explainability through visual concept knowledge distillation in CBMs.