# Advancing Model Explainability: Visual Concept Knowledge Distillation for Concept Bottleneck Models
## 5 Discussion and Limitations  
**Performance preservation under model compression. ** Our VICO-KD method demonstrates the ability to maintain model performance while substantially reducing computational requirements. By applying the VICO-KD approach, we achieved significant model compression without compromising accuracy. In our Inception-V3 based experiments on the CUB and AwA2 datasets, we reduced the total number of parameters from 23.8 million to 5.95 million--a 75% reduction--by halving the channel count in each Inception module. In our ResNet-based experiments on the CelebA dataset, we distilled a ResNet50 teacher model (23.5 million parameters) into a ResNet18 student model (11.2 million parameters), achieving a 52% parameter reduction while preserving the model's core performance characteristics. **Spurious correlations in knowledge distillation. ** Recent studies have revealed that CBM concept predictors can exploit spurious correlations between concepts rather than  
Figure 5: Experimental results on the impact of weight on visual concept transfer. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**TS Pair** & **Model** & **Task (y)** & **Concept (c)** \\ \hline \multirow{2}{*}{TESE} & Vanilla-KD & 0.833 \(\pm\) 0.021 & 0.935 \(\pm\) 0.001 \\  & VICO-KD & 0.862 \(\pm\) 0.006 & 0.943 \(\pm\) 0.001 \\ \hline \multirow{2}{*}{THSE} & Vanilla-KD & 0.564 \(\pm\) 0.003 & 0.789 \(\pm\) 0.002 \\  & VICO-KD & 0.654 \(\pm\) 0.008 & 0.847 \(\pm\) 0.008 \\ \hline \multirow{2}{*}{TESH} & Vanilla-KD & 0.299 \(\pm\) 0.016 & 0.756 \(\pm\) 0.050 \\  & VICO-KD & 0.373 \(\pm\) 0.007 & 0.830 \(\pm\) 0.001 \\ \hline \multirow{2}{*}{THSH} & Vanilla-KD & 0.351 \(\pm\) 0.005 & 0.775 \(\pm\) 0.003 \\  & VICO-KD & 0.406 \(\pm\) 0.005 & 0.841 \(\pm\) 0.000 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: The experimental results of inexpressive concept predictors. The TS pair configurations encompass various combinations of concept set groups and teacher-student pairs. These configurations are delineated as teacher(easy) \(\rightarrow\) student(easy) (TESE), teacher(hard) \(\rightarrow\) student(hard) (THSH), teacher(hard) \(\rightarrow\) student(easy) (THSE), and teacher(easy) \(\rightarrow\) student(hard) (TESH). their intrinsic characteristics [11]. This issue can be exacerbated during knowledge distillation, particularly when transferring knowledge from teacher to student models. While VICO-KD attempts to address this by complementing concept relationships through visual features, there is room for improvement. Future research should explore whether alternative knowledge distillation methods, such as feature-based KD, could better mitigate spurious correlations. Additionally, studying the relationship between explicit and implicit concepts could lead to more robust knowledge transfer methods that reduce reliance on spurious correlations. The presence of spurious correlations raises fundamental questions about the interpretability and reliability of distilled knowledge in CBMs. Understanding and addressing these correlations is crucial for developing more trustworthy and interpretable models through knowledge distillation. **Alternative visual feature transfer methods. ** While our current approach uses saliency maps for visual concept transfer, attention maps could potentially offer an alternative mechanism. Attention maps, which directly capture where the model focuses during inference, might provide more direct insights into the model's decision-making process compared to gradient-based saliency maps.