Comprehensive Knowledge Distillation with Causal Intervention
Authors: Xiang Deng
Zhongfei Zhang
Venue: NeurIPS 2021 Poster (2021)
Keywords: knowledge distillation
sample and class representation distillation
causal inference

Abstract: Knowledge distillation (KD) addresses model compression by distilling knowledge from a large model (teacher) to a smaller one (student). The existing distillation approaches mainly focus on using different criteria to align the sample representations learned by the student and the teacher, while they fail to transfer the class representations. Good class representations can benefit the sample representation learning by shaping the sample representation distribution. On the other hand, the existing approaches enforce the student to fully imitate the teacher while ignoring the fact that the teacher is typically not perfect. Although the teacher has learned rich and powerful representations, it also contains unignorable bias knowledge which is usually induced by the context prior (e.g., background) in the training data. To address these two issues, in this paper, we propose comprehensive, interventional distillation (CID) that captures both sample and class representations from the teacher while removing the bias with causal intervention. Different from the existing literature that uses the softened logits of the teacher as the training targets, CID considers the softened logits as the context information of an image, which is further used to remove the biased knowledge based on causal inference. Keeping the good representations while removing the bad bias enables CID to have a better generalization ability on test data and a better transferability across different datasets against the existing state-of-the-art approaches, which is demonstrated by extensive experiments on several benchmark datasets.

Comprehensive Knowledge Distillation with Causal Intervention
Xiang Deng
Zhongfei Zhang
Computer Science Department
Computer Science Department
State University of New York at Binghamton
State University of New York at Binghamton zhongfei@cs binghamton. edu
edu
Abstract
Knowledge distillation (KD) addresses model compression by distilling knowledge from a large model (teacher) to smaller one (student). The existing distillation approaches mainly focus on using different criteria to align the sample represen- tations learned by the student and the teacher, while they fail to transfer the class representations. Good class representations can benefit the sample representation learning by shaping the sample representation distribution. On the other hand, the existing approaches enforce the student to fully imitate the teacher while ignoring the fact that the teacher is typically not perfect. Although the teacher has learned rich and powerful representations, it also contains unignorable bias knowledge which is usually induced by the context prior (e.g.. background) in the training data. To address these two issues, in this paper, we propose comprehensive, in- terventional distillation (CID) that captures both sample and class representations from the teacher while removing the bias with causal intervention. Different from the existing literature that uses the softened logits of the teacher the training targets, CID considers the softened logits as the context information of an image, which is further used remove the biased knowledge based on causal inference. Keeping the good representations while removing the bad bias enables CID to have better generalization ability on test data and a better transferability across different datasets against the existing statc-of-the-a arrt approaches. which is demonstrated by extensive experiments on several benchmark datasets
Introduction
The superior performances of deep neural networks (DNNs) are accompanied with large amounts of memory and computation requirements, which seriously restricts their deployment resource- limited devices. An effective and widely used solution to this issue is knowledge distillation [1]]7] 37 that compresses a large network (teacher) compact and fast network (student) by knowledge transfer. To this end, the student obtains significant performance boost.
The original knowledge distillation (KD) 19] uses the softened logits generated by teacher the targets to train a student. Ever since then, substantial efforts including [37 45) have been made on aligning the sample representations learned by the student with those learned by the teacher using different criteria. However, almost all the existing approaches 45 49 58) have overlooked the class representations. Good class representations beneficial to sample representation learning, since they can shape the sample representation distribution. To address this issue, we propose comprehensive distillation to incorporate the class representations learned by the teacher into the distillation process.
On the other hand, as the teacher has learned rich and powerful representations, the existing ap- proaches enforce the student to fully mimic the behavior of the teacher. However, fully imitating
'Code https//github..om/Xaagg -iin-annnnnn-nnLeu Deng-DL/CID
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Prediction:
Dogs:
Cats
Prediction:
Cats:
Dogs
Training dataset
Test dataset
Figure 1: Misclassification caused by context prior in the training dataset.
the representations of the teacher may not be optimal, since the bias contained in the teacher is also transferred to the student. The bias is usually caused by the context prior in the training data. For example, as shown in Figure [ the dogs in the training dataset are usually on green grasses and the cats in room, which misleads the trained classifier to classify the cats on green grasses in the test dataset as dogs and the dogs in a room as cats due to the bias induced by the context. Similar cases can also happen on the attributes of data samples, e.g., when the colors of the dogs in the training dataset are mostly black, the black cats in the test dataset may be wrongly classified as dogs. Transferring this kind of the bias contained in the pretrained teacher to the student hurts the student.
Since the biased knowledge in the teacher is caused by the training data, we assume that the training data used by the teacher and those used by the student from the same distribution. This is not strong assumption in knowledge distillation literature as almost all the existing work uses the same dataset when training teacher and student, which obviously satísfies the assumption. Contrary this assumption, when the training data for the teacher and the student are from different distributions, two issues arise. First, the teacher may not be able teach the student anymore due the data distribution discrepancy. Second, new biases will be introduced in the distillation process from the new training dataset. We leave these questions for the future work.
Under the above assumption, we formulate the causal relationships [32 among the pretrained teacher, the samples, and the prediction in causal graph shown in Figure 4(a) More details are given in Section 3.2.1] We then use the softened logits learned by the teacher as the context information of an image remove the biased knowledge based backdoor adjustment (14). To this end, propose a simple yet effective framework (i.e., CID) to achieve comprehensive distillation and bias removal.
We summarize our contributions and the differences from the existing approaches as follows:
We propose novel knowledge distillation framework, i.e., CID, which captures comprehen- sive representations from the teacher while removing the bias with causal intervention. To our best knowledge, this is the first work to study how to use causal inference to address KD-based model compression.
CID different from the existing approaches in two aspects. First, CID is able transfer the class representations which are largely ignored by the existing literature. Second, CID uses softened logits as sample context information to remove biases with causal intervention, which differs from the existing literature that uses the softened logits as the training targets to train student. Keeping the good knowledge while removing the bad bias enables CID to have better generalization test data and a better transferability new datasets.
Extensive experiments several benchmark datasets demonstrate that CID outperforms the state-of-the-an approaches significantly in terms of generalization and transferability.
2 Related Work
Knowledge Distillation. Hinton et al. 19 propose the original KD that trains student by using the softened logits of teacher as targets. Compared to one-hot labels, the logits provide extra information leamned by the teacher [19 [3]. However, KD fails to transfer the powerful representations learned by the teacher. Ever since then, many efforts have been made on aligning the sample representations learned by student and teacher. FitNet 37 aligns the sample representations learned by student with those learned by teacher through regressions. AT (56) distills sample feature attention from teacher into student. CRD [45] maximizes the mutual information between sample representations

---

200 220 learned by student and teacher. SRRL [21 aligns the sample representations of teacher and a student by using the teacher's classifier. CC 35 and SP [46] transfer the sample correlation over the whole dataset to the student, which may contain redundant and imelevant information as pointed
12 use different criteria to align the sample representations. We notice that almost all these approaches only transfer the sample representations while largely ignoring the class representations which can benefit the sample representation learning by shaping the sample representation distribution.
out in 77. Similarly, other approaches [2 20
23
45 49
38 aims to explore the cause-effect relationships between different variables. It can not only be used to interpret a particular phenomenon 6 26].
Causal Inference. Causal inference 33
but also serve as a tool address problems by determining and using the causal effects 3 5 29
Recently, it has been introduced to machine learning 4 and has been used in different applications,
including but not limited to domain adaptation [5 27 imitation learning [10], image captioning
S1.
scene graph generation 44] visaul dialog (36] few-shot learning [54], imbalance classification
45
semantic segmentation
53 57 VQA 28 and unsupervised learning [47]. this work, we
provide an interventional framework for knowledge distillation to remove the biased knowledge in
the teacher.
3 Comprehensive, Interventional Distillation
In this section, we first describe the comprehensive distillation which takes both the sample and class representations into account. We then present interventional distillation remove the biased knowledge with causal intervention and thus achieve comprehensive, interventional distillation (CID).
3.1 Comprehensive Representation Distillation
CID considers both sample and class representations and thus achieves comprchensive distillation.
Which layer's sample representations are transferred? Many approaches [37 45 transfer the intermediate or the last few layers' feature representations. In contrast, CID only distills the feature vectors in the last layer (before the classifier), since only these vectors are directly involved in making the final prediction. We empirically validate this point in Section 4.1
The criterion to align representations. We slightly modify the mean square error (MSE) as the criterion to align the sample representations. MSE is used in FitNet (37 for transferring the sample representations. However, tage that it biases towards the samples that have large-norm features. For example, for MSE(V2. Vo)= Va Voll2 where V. and V, are the variable and target, respectively. when V₂ [0.02, 0.02] and V, = [0.01. 0.011) the gradient is only 2(V. V/)= [0.02, 0.06]. It is observed that V. and V2, have totally different directions and are orthogonal, but the gradient is very small due to the small norms of the two vectors. In contrast, for large vectors V,, = [10, 20) and V,, = [9.5, 19.5), the gradient is [1. 1] which is much larger than [0.02, 0.06], even if the two vectors are very close terms of both direc- tions and norms. The inherent disadvantage of MSE makes the samples with different feature norms contribute differently to the student, which induces biases. As shown in Figure 2 the norms of the sample features learned by the teacher are across a wide range and have a noticeable variance. One natural idea to address this issue is to use the MSE of the normalized vectors, i.e., V.
find that MSE has disadvan-
1250
750
500
250
60
80
100
120
140
160
180
Figure 2: Feature norm distribution of ResNet32 on CIFAR-100.
V. 112. However, this loss only aligns the directions of the two vectors and the minimum point is not necessarily V. = V/2 anymore, e.g., Va [1, and V/2 [100, 100) are a solution to this loss due to the same vector direction although they are different substantially. To address this issue, we propose Normalized MSE (NM_ MSMM:
|| 22 ||allaa
M(V., Vo) MSE(Va, V6) IIV. VII
V|2211
V
where 11. denotes the square L. norm. NM MSE can be considered as sample-wise weighted MSE and the weights T for different samples are negatively related to their target feature vector
3

---

Class shape
Class shape
Class m shape
Cill1 J
C.lk 0
Cafk, 0
D
Crdk,
Coll D
Teacher:
C-(k. D
Cafk
Carlk, D
transfer
C:a.D
Cails o
Cafk. H
Call.
Coll
Crik
Student
CiK.
Cafk.
Cmffk, DI
Figure 3: Class representation transfer.
norms, which mitigates the bias and makes different samples equally contribute to the student. Although this a slight modification to the original MSE, we empirically find that it performs much better than MSE on benchmark datasets.
Class representations. The existing approaches only enable the student to capture the sample representations from the teacher while overlooking the class representations. CID incorporates this part into the distillation process. CID uses the class shapes to represent the class representations. As shown in Figure 3 the shape of class is graph with all the samples in the class as the nodes. The weight between two nodes defined as the representation similarity between the two nodes. We adopt the cosine similarity and thus the class representation of class is expressed as:
hi
M
C
=
(2)
1
|
|
1
where ht denotes the feature vector of sample in class k is the total number of samples in class superscript means transpose; |1-1|2 denotes L2 norm. The class representation is different from the sample-class relation defined in (7] which only captures
similarity while failing to capture sample-to-sample relations and thus cannot well represent class.
The objective for comprehensive representation distillation. CID transfers comprehensive knowl- edge consisting of both sample and class representations from teacher to student, and thus the objective for comprehensive distillation is written as:
(S, T) a.M(h W, h4) + 3 Σ II
1
1=1
where M the proposed NM MSE; hg E Rm and hy E R are the sample representations learned by student S and teacher T, respectively; W E mm is linear transformation for converting h, to the space with dimension m,; C and C are the ith class representations learned by and T, respectively; m is the total number of classes; and are balancing weights.
3.2 Interventional Distillation
Although teacher has learned good representations, it is typically not perfect. Comprehensive distillation enables the student to inherit the superior representations from the teacher while it also introduces the bias to the student. To address this issue, we use causal intervention remove the bias.
3.2.1 Structural Causal Model
In knowledge distillation, the pretrained teacher with the context information in training data can be considered as the prior knowledge for training the student. We illustrate the causalities among prior knowledge K, sample X, and prediction in Figure ((a) where A + B denotes that the causer of B. We describe the causal relationships among these variables in the following.
K + X: the context prior in determines where the object appears an image X, e.g., the context prior in the training dataset in Figure put the dog object in green grasses instead of rooms.
K + + X: J is the context-based representation of X by using the context bases in K. This relationship exists due to the fact that even for the same image, its context representation under different dataset contexts or with different pretrained teachers can differ substantially.

---

Teacher
Feature hr
Logits
Encoder !O
Classifier g/O
Transfer
Context
Input samples
Encoder f.O
&
Classifier g,0
P(Y)do(X)
Student
Feature
Figure 5: Interventional Distillation. A network can be represented as an encoder fQ followed by a linear classifier g() so that teacher T(X) = (x()((X) and student S(X) 9, (X)).
X Y + J: Besides the regular X + Y, the predic- tion is also affected by the prior knowledge K through mediation J. For example, in Figure [ the cats in the test dataset are misclassified to dogs, since the context prior in K misleads the model to focus on the grass feature in X.
Therefore, the prior knowledge K is confounder of X and Y. The existing approaches that directly learn P(Y) XX) from the teacher bring the bias to the student model. We propose to model P((Y (Y)) with causal intervention 33 to remove the bias.
Causal model
(b) Intervention
Figure 4: Causal model and intervention.
3.2.2 Interventional Distillation via Backdoor Adjustment
After determining the cause-effect relationships, we use causal intervention P(Y)d((X)) instead of P(Y X) as the classifier, which pursues the true causality from X to Y by removing the effects of confounder K. Physical intervention, i.e, collecting samples with objects in all possible contexts evenly, impossible
(57). Thanks backdoor adjustment, we can model P(Y ([Y]) by cutting off K + X, which is achieved by stratifying the confounder into pieces K = {k1 kg,,.., kij ki J so that is not a confounder of and Y anymore shown in Figure 4bb) The de-confounded student is expressed as:
[K| -
P(Y]do(X))
(4)
where g0 is a function which we define later for generating context representation J from X and k;-
different context items [54], set each item k; of the prior knowledge to a class c, i..., K = {c; [c The m context base vectors are set to the class centers. Since different samples in class have different probabilities of containing the object C, we use the the weighted average of sample features as the class center.
As there are classes which can be considered
E P(c|/,;)h
2 Σ5=1 P(g
where & is the total number of samples in class c, P(c)zz;) is set to the teacher learned probability.
With context base vectors, define the sample-specific context representation J linear com- bination the context base vectors. As the logits learned by the teacher contain sample-to-cle similarities, use the softened logits to approximate the context coefficients provide context infor- mation. The coefficient learned by the teacher for sample X on base ē; written as af o T( [. where 7 is temperature to soften the logits and a is the softmax function. J can thus be expressed as: J= g(X,,,, c.) = a,², Since the teacher has learned appropriate context information, we enforce the student to learn the context information from the teacher, which leads to the final interventional distillation objective:
m
Line = P(Y\do(()) + KC(a at)
Σ PPVVIAAXXXX(CC)))PP(() K(())a))
(6)
1=1
5

---

where & denotes the concatenation operation; /C is metric to force the student to learn the context information from teacher. The learned context information is then used in the first term (for causal intervention) to make the final prediction through línear classifier. we set K. to -i in this paper. We simply set the weight for KC) to as we find that works very well. P(c;) set to the percentage of the samples in class C₁2 e-g., in balanced datasets, P(c)) interventional distillation in F
We summarize the
The Σ operation in (6) makes the forward cost of the final linear classifier linearly increase with the number of classes. This issue can be addressed by adopting the normalized weighted geometric mean 50 as an approximation:
Lino s P(Y]s(()&& ==1
[) + ((a))a))
The complete objective of CID. CID aims to achieve comprehensive knowledge distillation while removing the bad bias with causal intervention. Thus, its final objective is written as:
LCID Line + aM(g? W, hc) +
10
of
(8)
4
Experiments
In this section, we first conduct ablation studies and then compare CID with SOTA approaches.
4.1 Ablation Studies
The ablation studies are conducted on CIFAR-100 by using WRN-40-2 and 1RN- as the teacher and the student, respectively.
Effects of different components of CID. We use w/o SR, w/o CR, and w/o INV to denote CID without the sample representation distillation, without the class representation distillation, and without the intervention P(Y P(Y|do( (X)) by using the regular XXY respectively. As shown in Figure 6 the performances drop significantly without any of these terms. Specifically, as expected, sample representations play the most important role with performance gain 1.51 %, since sample representa- tions are directly involved in making final predictions and are further used in class representation distillation and causal intervention for removing biases. The improvement of the class representation distillation is 0.65%, which suggests that good class presentations benefit the sample representation learning and thus benefit the performance. On the other hand, by using the proposed P(Y (X) instead of P(Y (X) to pursue the true causality from X to Y. the student obtains 0.70% improvement, which demonstrates the effectiveness and necessity of the interventional distillation.
Effects of NM MSE. The proposed NM MSE is modified from MSE by using personalized weight for each sample to remove the feature norm biases of the teacher so that each sample contributes equally to the student model. Despite its simplicity, it observed in Figure that NM MSE (i.e., CID) substantially outperforms MSE (i.e., CIDmnsc), which demonstrates the superiority of
Which layer's representations should be transferred? CID transfers the last layer's feature vectors of the teacher to the student with the motivation that these features are directly involved in making the final prediction. We check the effects of distilling the representations in different layers. We report the results in Figure 7 is observed that the sample representations in the last layer are more effective than those in the other layers, and even better than the combination of the representations in all the layers, The reason can be that as the representations in the intermediate layers are not directly used for the final prediction, enforcing the student to imitate these representations hurts the learning ability and flexibility of the student which has a small capacity.
4.2 Comparison Settings with SOTA Approaches
We compare CID with SOTA approacees across varieties of (a) benchmark datasets (i.e., CIFAR-10 100
24
24 Tiny ImageNet [ and ImageNet (II), (b) network architectures (i.e., ResNet
"httpss/tin--imaaeee htap:/inwiimmmnnnnnneheeuuuupppp

---

77.00
76,50
76.50
76.00
76.00
Acc (%)
75.50
75.50
75.00
75.00
74.50
CID
w/oSR w/CCR w/lNNV CID
Conv1
Conv2
Conv3
Last layer All convs (ie.CID)
Figure 6: Effects of different components of CID.
Figure 7: Effects of distilling the sample represen- tations in different layers.
Table 1: Test accuracies (%) on CIFAR-10.
Teacher
WRN-16-4
ResNet-56 (0.86M)
ResNet-56
WRN-16-4 (2.75M)
ResNet-110 (1.73M)
(#Params)
(2.75M)
(0.86M)
Student
WRN-16-1
ResNet-14 (0.18M)
ResNet-8 (0.08M)
ResNet-14 (0.18M)
WRN-16-1
(#Params)
(0.18M)
(0.18M)
Teacher
95.04
93.87
93.87
95.04
94.00
Vanilla Student
91.32
91.33
88.55
91.33
91.32
KD
92.55-10.10
91.88-10.22
88.7010.18
92.33
91.5211.116
FitNet
92.51:0.226
91.74:1.119
88.7410.15
92.55=0.14
91.46110117
CC
92.54-10.23
92.09-0.227
88.7110.16
92.5410.18
91.66:10.29
RKD
92.77=10.18
92.10:10.14
88.1440.08
92.60=0.299
91.81
AB
92.39=0.30
92.14=0..19
88.85
92.40=0.09
91.31
CRD
90.96-10.20
90.41=0.113
88.40=0.09
91.17=0.14
90.27=0.20
SRRL
92.56-10.14
91.87+0.12
88. 88.7610.20
92.30=0.25
91.83=0.15
CID
0.100
92.21
0.14
92.87+0.24
92.36
[6]. WRN 55,, VGG [41,, and MobileNet (39 (c) settings (i.e., the teacher and the student share
the architccture
different architectures), (d) cases (i.e., regular cases, across- across-dataset dataset cases, and
data-limited cases). The competitors include FitNet (37] AT [6). SP 46 CC
[35 PKT [31], AB 18] VID 2, RKD 30, CRD 45) SRRI. (21) and CSKD 7 Since CID uses the logits of teacher
to provide context information for each sample, for fair comparison, the KD [19] loss is added to all the competitors. We omit "+KD" for simplicity when denoting these competitors plus KD, e.g., "FiNNet<KD" is abbreviated to "FitNet". On the other hand, since CID does intervention with the assistance of a linear layer, we also try to add an extra linear layer the competitors, but find that it hurts their performances due to overfitting, which we report in Appendix.
4.3 Model Compression
CIFAR-10: Table reports the comparison results on CIFAR-10. We adopt the teacher and the student with the same architecture or different architectures. It is observed that CID consistently outperforms all the baselines significantly across different architectures on both settings, while there is no baseline consistently the second best, since these baselines show their advantages in different architectures. These results demonstrate the superiority of CID.
CIFAR-100: We further report the comparison results on CIFAR- Table 2 For a fair compari- son, we adopt the architectures from the SOTA approaches (CRD 45 and SRRL (2T). As shown in T for compressing the large models to the smaller ones, CID obtains the best performances in different settings, which demonstrates the effectiveness of CID for model compression. The superior performances of CID are due to its ability to distill comprehensive knowledge and remove biases.
Tiny ImageNet:
We further evaluate CID in challenging datatset Tiny ImageNet. Table shows that CID beats all the SOTA approaches substantially in terms of both Top-1 and Top-5 accuracies on the challenging dataset, which validates the usefulness and superiority of CID on different datasets.
ImageNet: To investigate the performance of CID on large scale datasets, we follow CRD by adopting ImageNet and using ResNet-34 34 and ResNet- 18 as the teacher and the student, respectively, As shown

---

Table 2: Test accuracies (%) on CIFAR-100.
Teacher (#Params)
WRN-40-4
WRN-40-2
ResNet-56
ResNet-50
ResNet-50
(8.97M)
(2.26M)
(0.86M)
(23.7IM)
(23.7IM)
Student
WRN-16-2
WRN-16-2
ResNer-20 (0.29M)
MobileNetV2 (IM)
VGG-8
(#Params)
(0.73M)
(0.73M)
(4M)
Teacher
79.50
75.61
72.34
79.34
79.34
Vanilla Student
73.26
73.26
69.06
64.60
70.36
KD
7..5
0..92++0.28
0.24
6..35+0.32
73.81+-0.13
FitNet
74.48=0.27
75.12++0.33
0..7
66.96+0.24
73.24++0.27
AT
74.70=00.13
75.32:10.15
71.08
66.13+0.23
7..01
SP
74.79-0031
74.98
70.66-0.12
68.5410.35
73.52.10.25
CC
1..1
75.09-10.23
71.30-1.31
68.95:10.15
1..29
VID
74.83=0.10
75.14=0.15
71.18=0.09
68.3420.11
73,46::0.25
RKD
74.66-0.26
74.89=10.20
70,93=0.25
68.66=0034
73.51
PKT
75.21-0.22
75.33+0.18
71.53=0.26
68.4120.14
73.61
CRD
75,49=0.28
75.6410.21
71.63=0.15
69.5420.39
7..5
SRRL
75.96=0.21
75.25
7..44
69.45+0.29
7..46++0.25
CSKD
74.56=0033
75.11
7..3
68.80+10.6
73.11
CID
76.40=0.15
7..95+0.19
7.29
09.68
7..75=10.17
Table 3: Test accuracies (%) on Tiny ImageNet.
Teacher: WRN-40-2, Student: WRN-16-2
Teacher: VGG-13, Student: VGG-8
Top-1 (%)
Top-5 (%)
Top-1 (%)
Top-5 (%)
Teacher
61.84
84.
61.62
81.71
Vanilla Student
56.13
79.96
55,46
78.15
KD
58.27:10.17
02.10.0..15
60.21110.19
81.61-10.28
FitNet
59.58-10224
82.59:10.18
60.11::0.13
82.11+0.116
SP
58.52+0.36
82.10:10.15
60.941:0.24
82.42-10.20
CC
60.12:0.12
83.08110.10
61.11:0.34
82.44+0.28
VID
59.91=0.110
83.16:10.17
61.351:0117
82.61+0.23
RKD
59.29=10233
82.9911007
60.541:.225
82.3940.16
CRD
59.8610.29
83.1810..15
61.9810.27
82.6440.19
SRRL
59.90
82.98:10.21
61.3011.21
82.3110.26
CID
00.91
83.52.10.20
0.18
83.8110.13
in Table CID outperforms these competitors significantly, which demonstrates the applicability and effectivness of CID large scale datasets.
4.4 Transferability Comparison
An important goal of representation learning is to learn general representations which can be trans- ferred to different datasets. We investigate the across-dataset generalization ability of CID. For fair comparison, we follow the settings of CRD. Specifically, we freeze the feature encoder the student and train linear classifier STL-10 [9] TinylmageNet. WRN-40-2 and WRN-16-2 adpoted as the teacher and the student, respectively.
The transferability comparison results are reported in Table 5 It is clearly observed that CID beats the prior work substantially on both datasets, which demonstrates its superior generalization ability new data. The reason is that when transferring the knowledge from dataset to another, the inherited bias from the teacher can be a disaster to the new dataset. The ability CID to remove the biased knowledge mitigates this issue, thus leading to better generalization new datasets.
4.5 Data-Limited Distillation Performances
In reality, happens that when powerful model released, only a few data samples are publicly accessible due to the privacy or confidentiality issues in various domains such as medical and industrial

---

Table 4: Comparison results on ImageNet.
Teacher
Vanilla Student
KD
OFD
AT
SRRL
CRD
CID
TOP-1 (%)
73.3
69.8
70.7
71.1
70.7
71.7
71.4
70.2
70.0
71.9
TOP-5 (%)
91.4
89.1
89.9
90.1
90.0
90.6
90.5
89.8
89.2
90.7
Table 5: Transferability performances.
Cross-datasel
Teacher
Student
KD
AT
FitNet
CRD
SRRL
CID
CIFAR-100 to STL-10
68.6
69.7
70.9
70.7
70.3
72.2
71.0
72.5
CIFAR-100 to Tiny ImageNet
31.5
33.7
33.9
34.2
33.5
35.5
34.3
35.9
Table 6: Comparison results in the data-limited scenario.
Training Data
Student
KD
FitNet
SP
CC
RKD
PKT
CRD
SRRL
CID
20%
52.50
59.14
58.41
60.35
58.60
58.95
59.48
59.07
59.30
62.13
40%
61.45
66.89
65.94
66.73
66.27
66.15
66.13
66.84
66.40
68.64
60%
65.57
69.90
69.21
69.70
69.38
69.74
70.18
70.53
70.01
70.85
domains. It is thus necessary for distillation approaches to work on these practical cases. We compare CID with the existing approaches in the data data-limited scenario on CIFAR-100 by using VGG-13 13 and VGG-8 as the teacher and the student, respectively. As shown Table CID outperforms all the baselines by a large margin in all the three cases with 20%, 40%, and 60% training data. We also notice that the advantage
CID more obvious when fewer training data are available, e.g., the improvement of CID over the second best method is about 2% 20% or 40% training data cases, which is much higher than the improvement on 60% training data. The reason is that when fewer data samples available, these samples severely inadequate to represent the real data distribution so that the biases become more serious. While the existing approaches fail to handle this issue, CID is able to address it with the interventional distillation, which leads better performance.
5 Conclusion, Limitations, and Broader Impact
Conclusion. In this paper, we have proposed comprehensive, interventional distillation (CID) that captures both sample and class representations while removing the bias by using softened logits as the context information based on causal intervention. To our best knowledge, CID is the first framework along the line of using causal inference to address KD-based model compression. To this end, CID able to keep the good representations and remove the bad bias. Extensive experiments demonstrate that CID has better generalization ability on test data and a better transferability across different datasets against the existing SOTA approaches.
Limitations. A major assumption in CID is that the training data used by the teacher and the student are from the same distribution. The assumption is typically satisfied in knowledge distillation literature as almost all the existing work uses the same data to train the teacher and the student. On the other hand, when the assumption is violated, new biases will be introduced from the new data. CID is not designed to solve this problem and we leave this question to the future work. Also, when the training data used by the student and the teacher differ substantially, the teacher may not be able to supervise the student anymore. The role of the teacher needs to changed this case, which we leave for the future work.
Broader Impact. There is an increasing interest in implementing DNNs on portable devices such as smart phones and watches, while DNNs need large amount of memory and computation, which highly limits their deployments on these eesource-lmmite devices. CID can be used to address this issue by compressing large models (teachers) to small and fast ones (students). The advantage of CID over the other distillation approaches is that not only enables the students to inherit comprehensive knowledge from the teachers but also removes the bad biased knowledge, which leads to better generalization and transferability. More essentially, in real world, collecting data is very expensive, while using sparse data points to train a student induces severe biases, which poses challenges to the existing distillation approaches. CID is able to address this problem with the interventional distillation. So far, no negative impact has been observed.

---

References
[1] Gustavo Aguilar, Yuan Ling, Yu Zhang, Benjamin Yao. Xing Fan, and Edward Guo. Knowledge distillation from internal representations. arXiv preprint arXiv: 191000332.. 2019.
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9163-91661 9171,2015
[3] Elias Bareinboim and Judea Pearl. Controlling selection bias causal inference. In Artificial Intelligence and Statistics, pages 100-108. PMLR, 2012.
[4] Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Rosemary Ke, Séhastien I Lachapelle, Olexa Bilaniuk, Anirudh Goyal, and Christopher Pal. A meta-transfer objective for learning to disentangle causal mecha- nisms. arXiv preprint #rXiv:/90..0011. 2019.
[5] Michel Besserve, Arash Mehrjou, Rémy Sun, and Bernhard Schölkopf. Counterfactuals uncover the modular structure of deep generative models. In International Conference on Learning Representations (ICLR), 2020.
[6] Krzyszrof Chalupka, Pictro Perona, and Frederick Eberhardt. Visual causal feature learning. In Uncertainty in Artificial Intelligence, 2015.
[7] Zailiang Chen, Xianxian Zheng. Hailan Shen, Ziyang Zeng, Yukun Zhou, and Rongchang Zhao. Improving knowledge distillation via category structure. In Eumpean Conference Computer Vision, pages 205-219. Springer, 2020.
[8] Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In Proceedings of the IEEE International Conference on Computer Vision, pages 4794 4802, 2019.
[9] Adam Coates, Andrew Ng, and Honglak Lee. analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 213-223. JMLR Workshop and Conference Proceedings, 2011.
[10] Pim de Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion in imitation learning. In Advances in Neural Information Processing Systems, 2019.
[11] J. Deng, W. Dong, R. Socher, L.-J. K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
[12] Xiang Deng and Zhongfei Zhang. Graph-free knowledge distillation for graph neural networks. In The 30th International Joint Conference Artificial Intelligence, 2021.
[13] Xiang Deng and Zhongfei Zhang. Learning with retrospection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7201-7209, 2021.
[14] Vanessa Didelez and Iris Pigeot. Judea pearl: Causality: Models. reasoning. and inference. Politische Vierteljahresschrij., 42(2):313-315 2001.
[15] Mingming Gong, Kun Zhang, Tongliang Liu, Dacheng Tao, Clark Glymour, and Bernhard Schölkopf. Domain adaptation with conditional transferable components. In International conference on machine learning, pages 2839 2848. PMLR, 2016.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[17] Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A compre- hensive overhaul of feature distillation. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 1921-1930,201
[18] Bycongho Hco, Minsik Lec, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3779-3787, 2019.
[19] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: (t..xv:55033025333 2015.
[20] Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selectivity transfer. arXiv preprint arXiv:1707.01219, 2017.
10

---

[21] Adrian Bulat Jing Yang. Brais Martinez and Georgios Tzimiropoulos. Knowledge distillation vis softmax regression representation learning. In International Conference on Learning Representations, 2021.
[22] Jangho Kim, SeongUk Uk Park, and Nojun Kwak. Paraphrasing complex network: Network compression via factor transfer. In Advances in Neural Information Processing Systems, pages 2760-2769, 2018.
[23] Animesh Koratana, Daniel Kang, Peter Bailis, and Matei Zaharia. Lit: Learned intermediate representation training for model compression. In International Conference on Machine Learning, pages 3509-3518, 2019.
[24] Alex Krizhevsky and Geolfrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
[25] Yufan Liu, Jiajiong Cao. Bing Li, Chunfeng Yuan, Weiming Hu, Yangxi Li, and Yunqiang Duan. Knowledge distillation via instance relationship graph. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7096-7104, 2019.
[26] David Lopez-Pate Paz, Robert Nishihara, Soumith Chintala, Bernhard Scholkopf, and Léon Bottou. Discovering causal signals in images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6979-6999999 01779-687,60 2017.
[27] Sara Magliacane, Thijs van Ommen, Tom Claassen, Stephan Bongers, Philip Versteeg, and Joris M Mooij. Domain adaptation by using causal inference predict invariant conditional distributions. In Advances in Neural Information Processing Systems, 2018.
[28] Yulci Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian Blan-Sheng Hua, and Ji-Rong Wen. Counterfactual vqa: A cause-effect look at language bias. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
[29] Giambattista Parascandolo, Niki Kilbertus, Mateo Rojas-Carulla, and Bernhard Schölkopf. Learning independent causal mechanisms. In International Conference on Machine Learning, pages 4036-4044. PMLR, 2018.
[30] Wonpyo Park, Dongju Kim, Yan Lai, and Minsu Cho. Relational knowledge distillation. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pages 3967-3976, 2019.
[31] Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In Proceedings of the European Conference Computer Vision (ECCV), pages 268 284, 2018.
[32] Judea Pearl. Interpretation and identification of causal mediation. Psychological methods, 19(4):559, 2014.
[33] Judea Pearl, Madelyn Glymour, and Nicholas Jewell. Causal inference in statistics: A primer. John Wiley Sons, 2016.
[34] Judea Pearl and Dana Mackenzie. The book of why: the new science of cause and effect. Basic books, 2018.
[35] Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang. Correlation congruence for knowledge distillation. In Proceedings of the IEEE International Conference on Computer Vision, pages 5007-5016, 2019.
[36] Jiaxin Qi. Yulei Niu, Jianqiang Huang, and Hanwang Zhang. Two causal principles for improving visual dialog. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10860-1066, 2020.
[37] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. International Conference on Learning Representations, 2015.
[38] Donald B Rubin. Essential concepts of causal inference: remarkable history and intriguing future. Biostatistics & Epidemiology, 3():14-1555 2019.
[39] Mark Sandler. Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4510 4520, 2018.
[40] Zhiqiang Shen and Marios Savvides. Meal v2: Boosting vanilla resnet-50 to 80%+ top-1 accuracy on imagenet without tricks. arXiv preprint arXiv:2009.0844 2020.
[41] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In International Conference Learning Representations, 2015.
11

---

[42] Suraj Srinivas and Francois Fleuret. Knowledge transfer with Jacobian matching. In Jennifer Dy and An- dreas Krause, editors, Proceedings the 35th International Conference Machine Learning. volume 80 of Proceedings of Machine Learning Research, pages 4723-4731, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018, PMLR.
[43] Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. Long-tailed classification by keeping the good and removing the bad momentum causal effect. In Advances in Neural Information Processing Systems, 2020.
[44] Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. Unbiased scene graph generation from biased training. In Proceedings of the IEEEJCVF Conference on Computer Vision and Pattern Recognition, pages 3716 3725, 2020.
[45] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In International Conference on Learning Representations, 2020.
[46] Frederick Tung and Greg Mori. Sireeang knowledge distillation. In Proceedings of the IEEE International Conference on Computer Vision, pages 1365-1374, 2019.
[47] Tan Wang, Jianqiang Huang, Hanwang Zhang, and Qianru Sun. Visual commonsense r-cnn. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10760- 10700, 2020.
[48] Xiaojie Wang, Rui Zhang. Yu Sun, and Jianzhong Qi. Kdgan: Knowledge distillation with generative adversarial networks. In Advances in Neural Information Processing Systems, pages 775-786, 2018.
[49] Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge distillation meets self- sulf-suupe In European Conference Computer Vision, pages 588 604, Springer, 2020.
[50] Kelvin Xu, Jimmy Ba. Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov. Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pages 2048-2057. PMLR, 2015.
[51] Xu Yang, Hanwang Zhang, and Jianfei Cai. Deconfounded image captioning: A causal retrospect. arXiv preprint arXiv:2003.03922 2020.
[52] Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 4133-4141, 2017.
[53] Fisher and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. In International Conference on Learning Representations (ICLR), 2016.
[54] Zhongqi Yue, Hanwang Zhang. Qianru Sun, and Xian- Sheng Hua. Interventional few-shot learning. In Advances in Neural Information Processing Systems, 2020.
[55] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
[56] Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. In International Conference on Learning Representations. 2017.
[57] Dong Zhang, Hanwang Zhang, Jinhui Tang, Xiansheng Hua, and Qianru Sun. Causal intervention for weakly-supervised semantic segmentation. In Advances in Neural Information Processing Systems, 2020.
[58] Helong Zhou, Liangchen Song. Jiajie Chen, Ye Zhou, Guoli Wang. Junsong Yuan, and Qian Zhang. Rethinking soft labels for knowledge distillation: A varianer tradeoff' perspective. In International Conference on Learning Representations, 2021.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] The limitations are given in Section 5
(c) Did you discuss any potential negative societal impacts of your work? [Yes] The societal impacts given in Section[5]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
12

---

2. If you including theoretical results...
(a) Did you state the set assumptions theoretical results? [Yes] The assumptions are stated in Paragraph 4 of Section I and Paragraph 2 of Section S
(b) Did you include complete proofs of all theoretical results? [Yss] This is illustrated with the
equations in the paper.
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as URL.) [Yes] Please refer to Appendix.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Yes] Please refer to the Appendix.
(c) Did you report enror bars (e.g.. with respect to the random seed after running experiments multiple times)? [Yes] Please refer Table [ and
(d) Did you include the total amount of computc and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] Please refer Appendix.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] Please refer to Appendix.
(b) Did you mention the license of the assets? [Yes] Please refer Appendix.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] Please refer Appendix.
(d) Did you discuss whether and how consent obtained from people whose data you' 'r us- ing/curating? [Yes] Please refer to Section 4.2
(c) Did you discuss whether the data you using/curating contains personally identifiable informa- tion or offensive content? [Yes] The adpoted datasets are public benckmark datasets.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include full text of instructions given to participants and screenshots, if applicable? [N/A] We did not use crowdsoureing or conducted research with human subjects.
(b) you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] We did not use crowdsourcing or conducted research with human subjects.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] We did not use crowdsourcing conducted research with human subjects.
13


Reviews:
[{"review_id": "oHqDdPgxxKl", "reviewer": "Reviewer_BQzY", "summary": "This paper focuses on comprehensive knowledge distillation to transfer the class representation. The proposed method aims to improve the performance of the student model by considering the transferring of class representation. It is a novel view, especially with a causal intervention approach. The experiments are conducted to demonstrate the advantages of the proposed method.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Considering class representation or category structure into knowledge distillation is not a novel approach to improve the performance of knowledge  distillation. There are some works done with the same idea such as Chen et.al.  But the introducing of causal intervention is novel in knowledge distillation.  The writing and organization of this paper are clear, and the motivation of comprehensive representation distillation is also meaningful and significant.  The experiments and analysis are sufficient. \n\nPost-rebuttal:\n\nMany thanks to the authors for addressing my concerns. After reading the rebuttal, I have updated my score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- the motivation of the interventional distillation is unclear. What is the confounder or bias that affected the comprehensive distillation? How to guarantee the prior knowledge K is not the same as the How to ensure that the prior knowledge K in interventional distillation and knowledge in comprehensive representation distillation are not in conflict.\n- how to know the prior knowledge K is a confounder of X and Y? it is against our intuition. Which type of prior knowledge?\n- in the introduction, the authors claim that \"good class representations are beneficial to sample representation learning, since they can shape the sample representation distribution.\" It is hard to understand. How do the class representations shape the sample representation distribution?\n-the idea of considering class representation or category structure into knowledge distillation is not novel. Please compare with the work with Chen. et.al.\nChen et.al., Improving Knowledge Distillation via Category Structure. ECCV 2020.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "aBeaFxw7yvt", "reviewer": "Reviewer_6Byr", "summary": "The paper addresses an interesting concern : how can we transfer knowledge from a large model (master) to a tiny one (student) taking into account possible bias in the learning process of the teacher\nThe framework considered is that of  image classification, where the context contained is the image can yield some bias in the classification task (for e.g. garden when distinguishing a dog from a cat)\nThe paper considered two kinds of knowledge transfer\n* one may want to transfer representation of each class\n* one may also want to transfer the classifier in itself\n\nThe approach proposed in this paper to circumvent the bias problem is based on causal  intervention.  The claim is that there is an underlying graph where the output of the classifier may have as parent in the graph, both the true representation of the object to identify and the context\nThe idea is to perform causal intervention to remove the influence of the context in the classification\nNumerical experiments are performed in the image context. The task is object clasiffication and several classical datasets are considered", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* I am quite convinced by the relevance of the problem which is addressed\n* The approach in itself : first separation of the two kinds of knowledge that we want to transfer and second use of do calculus to remove influence of the context in classification in the teacher model is smart. I like it\n* I wonder how difficult the objective function defined through do calculus is difficult to solve. I have no idea of possible properties of this function. It is probably non convex, I don't know if it is smooth. Can you give some details about that?\n* In numerical experiments, the compression rate is significant and the results seems better than other state-of-art approaches. Results are mainly given hen the teacher is a ResNet or a WRN. Other architectures are mentionned but the resulats are not given. Can you give some details?\n* What about robustness to noise of the approach? Are the performance of the student similar if the images are noisy or if there is noisy labels? One can think context may help when the images are mislabeled. Is there any drawback of this approach? What do we miss when we remove the impact of context?\n ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "OK from this point of view", "review_text": "", "extra_scores": {"ethical_concerns": "OK from this point of view", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3h"}, "extra_text": {}}, {"review_id": "_g3HXOOEayq", "reviewer": "Reviewer_YwMj", "summary": "Authors have proposed a novel framework (CID) for knowledge distillation that is able to capture sample and class representations and reduce knowledge bias induced by the context prior through causal intervention. There are two major differences between the proposed approach and previous work: first, CID is able to transfer class representations. Second, it uses soften logits as sample context information. Authors have provided extensive experimental results to show the effectiveness of their approach.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written and easy to follow. The idea is novel although there has been some usage of causal inference for knowledge distillation for the question answering application. The major concern that I have is about the experimental results. Some of the cited papers that are recently published in top venues (e.g.  the SSKD approach: Knowledge distillation meets self-supervision from ECCV 2020) are not included in the tables. In addition, there are discrepancies  between numbers reported here vs. numbers reported in previous works. As an example, the performance of AT is reported 71.78 in the Table 3 of SSKD paper for (ResNet-56, ResNet-20); while it is reported as 71.08±0.34 in the submitted work. Also, there are some (teacher, student) configurations that are missing in this paper compared to the previous works. \n\nI am more than happy to increase my score if authors can provide a fair explanation about issues in the experimental results.\n\n----------------\nPost-rebuttal:\n\nThanks to authors for addressing my concerns and questions regarding experimental results. I have updated my score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors have adequately discussed limitations and societal impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "Pejr-mg6LhR", "reviewer": "Reviewer_tHsZ", "summary": "This paper proposes a new knowledge distillation method called CIS that captures both sample and class representations from the teacher while removing the bias with causal intervention. Experiments on several benchmark datasets validate the effectiveness of the proposed method.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. This paper is well written and easy to follow.\n2. The motivation is clear. I think it is the first attempt to investigate the bias problem in teacher's prediction.\n3. Experiments are done on different benchmarks, and comparison covers SOTA methods.\n\nWeakness:\nThe class representation part is unclear to me. As is described in (2). The class representation dimension is related to the number of images in the current category. So it will inevitably increase the training time. However, this paper did not provide training time in the current version. Besides, the class representation idea is similar to [33]. Although [33] only considers within batch comparison, but it considers the inter-class comparison while this paper only considers the intra-class comparison. The advantage over [33] should be discussed and analysed. \n\nPost-rebuttal:\n\nThanks to the authors for addressing my concerns and questions regarding the class representation part. I keep my previous score.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper has addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1hour"}, "extra_text": {}}, {"review_id": "PXgd7sJdOYd", "reviewer": "Reviewer_3nmW", "summary": "This paper proposes an improved knowledge distillation process named Comprehensive, Interventional Distillation (CID). It adds a class representation term to the loss function of a knowledge distillation model to enforce the student and the teacher to learn not only similar sample representations but also class representations, where a class is represented by the similarity matrix among all of its samples. It further modifies the loss function term of the student to remove the bias learned by the teacher from the context information in training data, based on casual intervention. Experimental results on four real datasets confirm the effectiveness of the proposed model in improving the classification accuracy of the student model. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper studies an interesting problem, i.e.,  to transfer class representations and to remove biases with causal intervention in knowledge distillation. The paper is motivated with a figure (Figure 1) to highlight the importance of removing biases. However, it then makes an assumption that \" the training data used by the teacher and those used by the student are from the same distribution\". The importance of removing biases under such an assumption needs further justification. \n\nRemoving biases with causal intervention in knowledge distillation is new and interesting. The derivation of Equations (4) - (7) is  solid. The idea of learning and transferring class representations is also interesting but this is mainly a heuristic without performance guarantees. \n\nThe experiments are run on multiple real datasets of different sizes which is also good. \n\nThe paper is well written overall and is an interesting exploration. There are a few issues that need to addressed to further strengthen the paper as detailed in the \"Limitations And Societal Impact\" section. \n\nPost-rebuttal:\n\nI appreciate the authors' efforts to address my comments. I am not fully convinced by the response to Q2, as I am still unclear why learning the bias by the teacher is an issue given the assumption that the training data used by the teacher and those used by the student are from the same distribution (i.e., they may share the same bias). Since this is a basic assumption of the paper, I cannot boost my score. \n\nAnd update after further author response:\n\nI appreciate the further discussion, which clears the confusion. No further question or objection to acceptance from my end.  \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The experimental results are not very strong. The claim \"the performances drop significantly without anyone of these terms\" for Fig. 6 seems to be an overstatement. The performance drop for w/o CR and w/o INV are just about 0.5%. Similar claims appear in the discussions on the rest of the experimental results such as \"outperforms all the baselines significantly\", \"beats all the SOTA approaches substantially\", \"outperforms these competitors significantly\", while the performance difference between the proposed model CID and the best baseline is less than 1% in many cases. What is the definition of \"outperforms significantly/substantially\"?\n\nGiven the claim \"Since the bias knowledge in the teacher is caused by the training data, we assume that the training data used by the teacher and those used by the student are from the same distribution\", is the bias induced by the context learned by the teacher still a problem?\n\nAre there concrete examples in the test datasets that cause classification errors because of the context bias learned by the teacher model in the baseline models, which are correctly classified by the proposed CID model? \n\nWhat is the implication in terms the extra training introduced by the CID model?\n\nTypo: \"but also server as a tool\"", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is concerned with the distillation / compression of a large teacher model. Besides the transfer of the sample representation, the originality of the approach lies in the transfer of the class representation, and the use of a causal intervention (CID) aimed to prevent the transfer of the teacher's biases to the student (that can indeed be harmful even when the training and test distributions are the same). 

Formally, CID proceeds by aligning $P(outcome | do (input))$ for the teacher and student, using the backdoor adjustment formula to account for the confounders (context of the different classes). 

The merits of the approach are supported by extensive empirical comparisons with the state of the art, considering various architectures and datasets, and the relative importance of the three aspects (transferring the sample representation, transferring the class representation and using interventions to prevent the transfer of the biases) is shown using lesion studies. It is a bit disappointing that the interventional term, that is the most innovative and theoretically grounded one, is found to be the less important one after the lesion studies.
 
The authors did a very good job of answering the reviews, justifying the choice of the considered architectures, examining the impact of noise, clarifying the difference w.r.t. related works (e.g., Improving Knowledge Distillation via Category Structure) and  explaining the discrepancies with the results reported for the baselines in previous papers, depending on the use of pre-trained models. 

In summary, this paper tackles a relevant issue in an innovative way, and it is viewed as above the acceptance threshold by all reviewers.
Decision: Accept (Poster)