# Advancing Model Explainability: Visual Concept Knowledge Distillation for Concept Bottleneck Models
## 4 Experiments  
### Dataset and Experimental Setup  
To investigate the effectiveness of the KD method in CBM, we conducted experiments using benchmark datasets with concept annotations, namely, CUB, AwA2, and CelebA. All experiments were conducted on a single NVIDIA Tesla V100 GPU with 32 GB of memory, and the experiments were carried out using the PyTorch framework. **Dataset. ** We conducted experiments on the CUB, AwA2, and CelebA datasets. For the CUB dataset, we followed the preprocessing steps described in [4]. This dataset comprises 11,788 images of 200 bird species, with 312 attributes annotated for each image. These attributes represent concepts related to the visual characteristics of birds, such as wing color and beak size. We removed noisy attributes and retained only 112 attributes for our experiments. AwA2 contains 37,322 images of 85 animal species, with 85 attributes for each animal. We selected 50 animal species and 50 semantic attributes for our experiments. CelebA consists of 202,599 images of human faces, with 10,177 identities and 40 binary  
Figure 2: VICO-KD framework. The frameworks encompass three main streams: (1) logit-level KD stream: Illustrated in blue, this stream conducts logit-level KD to align the output distributions of the student and teacher. (2) The independent learning stream of the student: Represented in green, this stream focuses on the independent learning of the student. (3) The visual concept transferring stream: Illustrated in orange, this stream facilitates the transfer of visual explainability from the teacher to the student. attribute annotations per image. We utilized a subset of 256 identities and five binary attributes for our experiments on CelebA. **Model Architecture. ** For CUB and AwA2 datasets, we trained CBM using the bottleneck model training strategy and adopted the same hyperparameters as detailed in [4]. The architecture comprised an Inception-V3 [44] pretrained on ImageNet for the \(x\to c\) mapping and a 3-layer MLP for the \(c\to y\) mapping. Both teacher and student models share the same bottleneck structure, but the student model has a smaller capacity. Specifically, for the student model, we halved the channels in each inception module of the inception-V3 network. In contrast, for CelebA, we utilized the ResNet series to construct the teacher-student models, employing ResNet50 [45] for the teacher and ResNet18 for the student. For quantitative evaluation of the benchmark datasets, we utilize classification and 0-1 errors (for binary concepts) as task and concept errors, respectively. **Experimental setup. ** During training, both the teacher and student models underwent standard data augmentation techniques, including color jitter, random resized crop, and random horizontal flip. The teacher model underwent training for 1000 epochs, while the student model required a longer training duration of 3000 to ensure stable convergence. To determine the optimal hyperparameters, we conducted a grid search over the following values: optimizer \(\in\{\text{SGD},\text{Adam}\}\); learning rate \(\in\{0.001,0.01\}\); \(\mathcal{L}_{KL}\in\{0.5,1.0,2.0\}\); \(\mathcal{L}_{Aux-KL}\in\{0.5,1.0,2.0\}\); and \(w\) for \(\mathcal{L}_{VC}\), ranging from 0.5 to 4.0 in increments of 0.5. **Evaluation metric. ** The performance of VICO-KD is evaluated using two metrics: Task accuracy and concept accuracy. These metrics are defined as follows:  
\[\text{Task Accuracy}=\frac{1}{N}\sum_{i=1}^{N}1(\hat{y}_{i}=y_{i}), \tag{8}\]  
\[\text{Concept Accuracy}=\frac{1}{N}\sum_{i=1}^{N}\frac{1}{M}\sum_{j=1}^{M}1( \hat{c}_{i,j}=c_{i,j}), \tag{9}\]  
where \(N\) is the total number of test samples, and \(M\) denotes the number of concepts. Here, \(\hat{y}_{i}\) and \(y_{i}\) are the predicted and ground truth task labels for the \(i\)-th sample, respectively, while \(\hat{c}_{i,j}\) and \(c_{i,j}\) represent the predicted and ground truth concept labels for the \(j\)-th concept of the \(i\)-th sample.