# Advancing Model Explainability: Visual Concept Knowledge Distillation for Concept Bottleneck Models
## 4 Experiments
### Experimental Results  
#### 4.2.1 Experimental Results on CUB-200-2011  
In this section, we conduct a comparative analysis to evaluate the performance of our proposed VICO-KD method against other concept models and Vanilla-KD (without VICO) on the CUB dataset. Furthermore, we explore the influence of varying the strength of the weight hyperparameter (\(w\)) in VICO-KD on the model's performance. **Performance comparison**. We employed the joint bottleneck in our proposed VICO-KD approach, which is considered the best-performing logit-level KD method. We conducted a comparative analysis, comparing the performance of our method with other concept models, including CEM [46], ProbCBM [13], PCBM [12], and CME [47]. Table 1 summarizes the results, indicating that VICO-KD achieves competitive performance across accuracy metrics. Particularly noteworthy is that VICO-KD outperforms other concept models regarding task accuracy (\(y\)), with improvements ranging from 0.004 to 0.152. These performance gains can be attributed to effectively incorporating visual concepts in the knowledge distillation process, enabling the student model to learn more discriminative and semantically meaningful features. In contrast, compared to the Vanilla-KD model trained without visual concepts, there is a notable difference of 0.098 in task accuracy and 0.014 in concept prediction accuracy. This signifies that our proposed method for training the CBM transfers knowledge at the concept and visual concept levels, effectively enhancing task performance. This improvement can be attributed to the explicit integration of visual concepts in the knowledge distillation process. Transferring knowledge at the class label and visual concept levels allows the student model to acquire more meaningful and discriminative features that align better with human-interpretable concepts. **Weight impact on visual concept transfer**. We investigated how the weight hyperparameter \(w\) influences the transfer of visual concepts from a teacher CBM to a student CBM, with \(w\) values ranging from 0.5 to 4.0, as depicted in Figure 3. Our results demonstrate that increasing \(w\) from 0.5 to 2.0 enhances task and concept accuracies, with peak task accuracy at 0.7865 and concept accuracy at 0.9619, both occurring at \(w=2.0\). This indicates optimal concept learning and model performance at this weight value. However, weights above 2.0 lead to declines in performance, indicating potential overfitting and reduced generalization when too much emphasis is placed on visual concept transfer. Nevertheless, concept accuracy remains consistently high, underscoring the model's robustness. Our findings emphasize the significance of tuning \(w\) to strike a balance between task performance and concept learning, recommending a weight of 2.0 for optimal results in visual concept transfer. **Visualization of saliency maps**. To gain deeper insights into the effectiveness of VICO-KD in transferring visual concepts from the teacher to the student model, we utilize the Grad-CAM technique to visualize and compare the saliency maps of the teacher CBM, VICO-KD, and Vanilla-KD, as depicted in Figure 4.  
Upon analyzing the saliency maps, a striking difference between the teacher CBM and Vanilla-KD becomes apparent. The teacher CBM consistently focuses on the most discriminative parts of the object, aligning well with human intuition. This indicates that the teacher model has learned to attend to the salient visual concepts crucial for accurate object classification. In contrast, Vanilla-KD often attends to regions less relevant to the object of interest, indicating that it may struggle to capture the teacher's decision-making process fully.