# Advancing Model Explainability: Visual Concept Knowledge Distillation for Concept Bottleneck Models
## 3 Methodology
### Visual Concept Knowledge Distillation  
Incorporating explainability is crucial to ensuring a comprehensive understanding of the operation of compressed CBMs. When training CBMs with inherent interpretability using KD, the probability distributions of concept predictions between the teacher and student models tend to become highly similar. However, as discussed earlier, relying solely on post hoc explanations may result in discrepancies between the teacher and student models' explanations, leading to incomplete explanations presented to the user. To tackle this challenge, our proposed VICO-KD method encourages the student model to learn discriminative regions similar to the teacher model.  
In the general KD process, there can be a mixture of information about the foreground and background [41]. Therefore, guiding the student with attribute values the teacher deems important for prediction becomes necessary. To achieve this, we induce alignment of attribute values between the student and teacher based on saliency maps. Given the bottleneck structure of CBMs, concepts are predicted from the input, and task prediction is conducted based on the predicted concepts. From an interpretability perspective, identifying pixels in the image associated with essential concepts for task prediction is crucial. VICO-KD applies the rectified linear unit (ReLU) activation function to consider only the pixels that contribute to the prediction.  
Let \(A^{t}\in\mathbb{R}^{C\times H\times W}\) and \(A^{s}\in\mathbb{R}^{C\times H\times W}\) denote the activation maps of the teacher at layer \(l_{t}\) and the student model at layer \(l_{s}\) layer, respectively, where \(C\) is the number of channels, \(H\) is the height, and \(W\) is the width. For each class \(k\in 1,\dots,K\), the saliency maps are computed as follows:  
\[\begin{split} U^{t}_{c,k}&=\sum_{h=1}^{H}\sum_{v=1 }^{W}\frac{\partial y^{s}_{k}}{\partial A^{t}_{c,h,v}}\odot A^{t}_{c,h,v}\\ U^{s}_{c,k}&=\sum_{h=1}^{H}\sum_{v=1}^{W}\frac{ \partial y^{s}_{k}}{\partial A^{s}_{c,h,v}}\odot A^{s}_{c,h,v},\end{split} \tag{3}\]  
where \(y^{t}_{k}\) and \(y^{s}_{k}\) are the outputs for class \(k\) from the teacher and student models, respectively, and \(\odot\) denotes the element-wise multiplication. Grad-CAM is calculated for each channel \(c\in 1,\dots,C\).  
To consider only the important attributions, we apply the ReLU function:  
\[\begin{split} M^{t}_{c,k}&=\text{ReLU}\Big{(}U^{t }_{c,k}\Big{)}\\ M^{s}_{c,k}&=\text{ReLU}\Big{(}U^{s}_{c,k}\Big{)}. \end{split} \tag{4}\]  
Here, \(M^{t}_{c,k}\in\mathbb{R}^{H\times W}\) and \(M^{s}_{c,k}\in\mathbb{R}^{H\times W}\) represent the class-discriminative saliency maps [42] for the teacher and student models, respectively, highlighting the important regions for predicting class \(k\) in channel \(c\).  
To ensure that the student model learns similar attributions to the teacher model, we compute the L2 loss between the saliency maps:  
\[\mathcal{L}_{VC}=\frac{1}{C\cdot K}\sum_{k=1}^{K}\sum_{c=1}^{C}\Big{\|}M^{t}_ {c,k}-M^{s}_{c,k}\Big{\|}_{2}^{2}, \tag{5}\]  
This loss function encourages the student model to have similar saliency maps to the teacher model for each class and channel, effectively transferring the visual concept knowledge from the teacher to the student.