Contrastive Consistent Representation Distillation
Authors: Shipeng Fu
Haoran Yang
Xiaomin Yang
Venue: Submitted to ICLR 2023 (2023)
Keywords: contrastive learning
knowledge distillation
model compression

Abstract: The combination of knowledge distillation with contrastive learning has great potential to distill structural knowledge. Most of the contrastive-learning-based distillation methods treat the entire training dataset as the memory bank and maintain two memory banks, one for the student and one for the teacher. Besides, the representations in the two memory banks are updated in a momentum manner, leading to representation inconsistency. In this work, we propose Contrastive Consistent Representation Distillation (CoCoRD) to provide consistent representations for efficient contrastive-learning-based distillation. Instead of momentum-updating the cached representations, CoCoRD updates the encoders in a momentum manner. Specifically, the teacher is equipped with a momentum-updated projection head to generate consistent representations. These teacher representations are cached in a fixed-size queue which serves as the only memory bank in CoCoRD and is significantly smaller than the entire training dataset. Additionally, a slow-moving student, implemented as a momentum-based moving average of the student, is built to facilitate contrastive learning. CoCoRD, which utilizes only one memory bank and much fewer negative keys, provides highly competitive results under typical teacher-student settings. On ImageNet, CoCoRD-distilled ResNet50 outperforms the teacher ResNet101 by 0.2% top-1 accuracy. Furthermore, in PASCAL VOC and COCO detection, the detectors whose backbones are initialized by CoCoRD-distilled models exhibit considerable performance improvements.

Under review as a conference paper at ICLR 2023
CONTRASTIVE CONSISTENT REPRESENTATION DISTIL- LATION
Anonymous authors Paper under double-blind review
ABSTRACT
The combination of knowledge distillation with contrastive learning has great potential to distill structural knowledge. Most of the contrasive-leaminngbbeee dis- tillation methods treat the entire training dataset as the memory bank and maintain two memory banks, one for the student and one for the teacher. Besides, the repre- sentations in the two memory banks are updated in momentum manner, leading to representation inconsistency. In this work, we propose Contrastive Consistent Representation Distillation (CoCoRD) to provide consistent representations for distillation. Instead of momentum-updating
efficient contrastive-eaaring-baal the cached representations, CoCoRD updates the encoders in momentum manner. Specifically, the teacher is equipped with a momentum -udrenttd--udteed projection head generate consistent representations. The teacher representations cached in a fixed-size queue which serves as the only memory bank in CoCoRD and is significantly smaller than the entire training dataset. Additionally, slow-moving student, implemented as a momentum-basse moving average of the student, is built facilitate contrastive learning. CoCoRD, which utilizes only one memory bank and much fewer negative keys, provides highly competitive results under typical sttudet-ttudet settings. On ImageNet, CoCoRD- ResNet50 outperforms the teacher ResNet101 by 0.2% top-1 accuracy. Furthermore, in PASCAL VOC and COCO detection, the detectors whose backbones are initialized by CoCoRD- distilled models exhibit considerable performance improvements.
INTRODUCTION
The remarkable performance of convolutional neural networks (CNNs) in various computer vision tasks, such as image recognition (He et al., 2016; Huang al., 2017) and object detection (Girshick, 2015; Ren et al.. 2015; Redmon & Farhadi, 2017), has triggered interest in employing these powerful models beyond benchmark datasets. However, the cutting-edge performance of CNNs is always accompanied by substantial computational costs and storage consumption. Farly study has suggested that shallow feedforward networks can approximate arbitrary functions (Hornik et al., 1989). Numer- endeavors have been made to reduce computational overheads and storage burdens. Among those endeavors, Knowledge Distillation, a widely discussed topic, presents a potential solution by training compact student model with knowledge provided by cumbersome but well-trained teacher model.
The majority of distillation methods induce the student to imitate the teacher representa- tions (Zagoruyko & Komodakis, 2017; Park et al., 2019; Tian et al., 2020; Hinton et al., 2015; Chen et al., 2;;;b;c; Yim et al., 2017; Tung & Mori, 2019; Ahn et al., 2019). Although representa- tions provide more learning information, the difficulty of defining appropriate metrics to align the student representations the teacher ones challenges the distillation performance. Besides, failing to capture the dependencies between representation dimensions results in lame performance. To enhance performance, researchers attempt to distill structural knowledge by establishing connections between knowledge distillation and contrastive learning (Tian et al., 2020; Chen et al., 202
To efficiently retrieve representations of negative samples for contrastive learning, memory banks cache representations which are updated in a momentum manner, as shown in Fig. 1. However, the student is optimized sharply by the training optimizer. The student representations in the memory bank are inconsistent because the updated representations differ from those not updated in that iteration. Therefore, the student can easily contrast the positive and negative samples, keeping the student from learning good features. The storage size of memory bank is another factor of concern when applying hanrrattveeleaaming-aaae distillation methods. As in (Tian et al., 2020; Chen et al.,

---

Under review as a conference paper at ICLR 2023
cotwenite controsive fremiiy based diclletti
Teacher Memory Bank
mamemmu
Samsling
incongruent pairs
Student Memory Bank
foct reeeary beck cantalte
Contrastive
represettction rraling dtrout
Teacher
congruent pairs
Student
foom ponceding
Appiessstation
Teacher Dictionary
(% Ar,
Teacher
kp
Contrastive
Student
Prediction Loss
slow moving
9₂-
Student
Figure 1: The general pipelines of contrastive learning based knowledge distillation methods and CoCoRD. Instead momentum updating the representations, CoCoRD updates the encoder in momentum manner. The teacher dictionary which contains representations from preceding batches is implemented as a queue.
2021b), there are two memory banks and each of them contains representations of all training images, leading to massive GPU memory usage on large-scale datasets.
Motivated by the discussion above, we propose Contrastive Consistent Representation Distillation
(CoCoRD) novel way of distilling consistent representations with one fixed-size memory bank. Specifically, CoCoRD composed of four major components, as shown in Fig. (1) a fixed-size queue which is referred to the teacher dictionary, (2) teacher, (3) a student, and (4) slow-moving student. From a perspective of considering contrastive learning as a dictionary look-up task, the teacher dictionary is regarded as the memory bank, where all the representations serve as the negative keys. The encoded representations of the current batch from the teacher are enqueued. Once the queue is full of representations, the oldest ones dequeued. By introducing queue, the size of the memory bank is decoupled from dataset size and batch size, allowing it to be considerably smaller than dataset size and larger than the commonly-used batch size. The student is followed by projection head, which maps the student features to a representation space. The teacher projection head initialized the same the student one and is a momentum moving average of the student projection head if the teacher and the student have the same feature dimension; otherwise, the teacher projection head is randomly initialized and not updated. Since the contrast through the teacher dictionary is to draw distinctions on an instance level, the cached teacher representations which share the same class label as the student ones are mistakenly treated as negative keys, resulting in noise in the dictionary. To alleviate the impact of the noise, a slow-moving student, implemented momentum moving average of the student, is proposed to pull together anchor representations and class-positiog ones. As shown in Fig. 2, with momentum upddtte projection head, the slow-moving student projects a data augmentation version of the anchor image to the representation space, which serves the instance-negative but class-positive key. The main contributions listed follows:
We utilize only one lightweight memory bank (teacher dictionary), where all the representations are treated as negative keys. We experimentally demonstrate that a miniature teacher dictionary with much fewer negative keys can be sufficient for contrastive learning in knowledge distillation.
We equip the well-trained teacher with a momentum-updated projection head to provide consistent representations for the teacher dictionary. Besides, slow-moving student provides class-positivg representations to alleviate the impact of the potential noise in the teacher dictionary.
We verify the effectiveness of CoCoRD by achieving the state-of-the-as art performance in 11 out of 13 -eadehe-teachr combinations in terms of model compression. On ImageNet, the CoCoRD. distilled ResNet50 can outperform the teacher ResNet101 by 0.2% top-1 accuracy. Moreover, we initialize the backbones in object detection with CoCoRD- weights and observe considerable performance improvements over the counterparts that the vanilla students initialize.
2

---

Under review as a conference paper at ICLR 2023
fi Teacher
The teacher dictionary
momentum projection head
(RE)
( Rig
F
Contrastive Loss
9,
f projection head
Student
4t
predictor
swap
0's slow moving Student
Predict
Pr
x; to the student compute E E
x; to slow wwwwmmovvngg..oo student and
momentum projection head
4 prodt
Figure 2: Illustration of the proposed CoCoRD. Note that q detached from the computational graph during the distillation process. To which obtained by feeding z, to the slow-moving student, also detached. The teacher is frozen and the teacher dictionary does not receive gradient.
2 RELATED WORK
2.1 KNOWLEDGE DISTILLATION
Hinton et al. (Hinton et al., 2015) first propose distilling the softened logits from the teacher to the student. After the representative work (Hinton et al., 2015), various knowledge distillation methods (Wang & Li, 2021; Song al., 2021; Passban et al., 2020; Chen al., 202 la;)) aim to distill more informative knowledge via intermediate features. Among them, Passban et al. (Passban al., 2020) fuse all teacher information to avoid the loss of significant knowledge. Chen et al. (Chen et al., 2021a) propose semantic calibration based on the attention mechanism for adaptively assigning cross-layer knowledge. Chen et al. (Chen al., 2021c) introduce novel framework via knowledge review in which the knowledge of multiple layers in the teacher can be distilled for supervising one layer of the student. However, the methods mentioned above have difficulty in defining appropriate metrics to measure the distance between the student representations and the counterparts from the teacher. There are a few recent works exploiting the dependencies between representation dimensions based contrastive learning (Tian al., 2020; Chen al., 2021b) for boosting the distillation performance. In particular, Tian et al. (Tian et al., 2020) formulate capturing structural knowledge as contrastive learning and maximize the lower bound of mutual information between the teacher and the student. Chen et al. (Chen et al., 2021b) leverage primal and dual forms of Wasserstein distance, where the dual form yields a contrastive learning objective. In summary, the core of knowledge distillation lies in the definition of knowledge and the way the knowledge is distilled.
2.2 CONTRASTIVE LEANING
The main goal of contrastive learning is to learn a representation space where anchor representations stay close to the representations of the positive samples and distant from those of the negative samples. Contrastive learning is a powerful approach in self-supervised learning. To learn powerful feature representations in unsupervised fashion, Wu et al. (Wu et al., 2018) consider each instance as a distinct class of its own and use noise contrastive estimation (NCE) to tackle the computational challenges. Contrastive learning is first combined with knowledge distillation by CRD (Tian et al., 2020), which aims at exploring structural knowledge. In addition to CRD (Tian et al., 2020), WCoRD (Chen et al., 2021b) combines LCKT (Chen et al., 2021b) and GCKT (Chen et al., 2021b) based on Wasserstein dependency measure in contrastive learning (Ozair et al., 2019). However, the memory banks CRD and WCoRD contain representations of all the training images, which bring about storage challenges on large-scale datasets. Besides, momentum updates to representations can also lead to inconsistent representations that negatively affect the distillation performance. From a perspective of considering contrastive learning as a dictionary lookup task, we implement the memory bank as a frst-in-first-ou queue where all included representations serve as negative keys.
3 METHOD
The key idea of combining knowledge distillation with contrastive learning is straightforward, With knowledge distillation, proficient teacher can provide consistent representations that are beneficial for contrastive learning. With contrastive learning. the student can obtain powerful features whose representations are close to the positive teacher representations and distant from the negative ones in representation space. Contrastive learning can be generally formulated as dictionary look-up task. Given a query g and a dictionary K with N keys: K = (ki,, ... kNN,, contrastive learning matches the query g to the positive key k4 and pushes away from the negative keys cached in K.
3

---

Under review as a conference paper at ICLR 2023
3.1 CONTRAST AS LOOKING UP IN THE TEACHER DICTIONARY
In CoCoRD, the negative keys are encoded by the teacher and cached in fixed-size queue which is referred to as the teacher dictionary. Given an input image two views of x under random data augmentations form a positive pair (a query and a positive sample), which is encoded in each iteration.
We define the input to the student S as the query I, and the input to the teacher T as the positive sample 2,. The outputs at the penultimate layer (before the last fully-connected layer) are projected to representation space by a projection head. For simplicity of notation, the student nested functions up to the penultimate layer are denoted as (-(-) and the student projection head is denoted as f2(-). Therefore, the query representations 9s and the positive keys ki are given by:
q. = (z)() k = [f
(I)
where ge(-) denotes the teacher nested functions up the penultimate layer and ff() is the teacher projection head. J" and JL are two-layer perceptrons. Besides, the cached i-th negative key in the queue is denoted as k which is produced the same way as k; but from the preceding batches. The fixed-size teacher dictionary [- k } contains N negative keys. The representations of the current batch are added to the queue, while the oldest representations are removed from the queue.
The contrastive loss. The value of the contrastive loss should be small when qu is close to ki and distant from K, in the representation space. To meet this condition, we consider the wildly- and effective contrastive loss function: InfoNCE (Van den Oord et al., 2018):
exp(u= kj kT/T)
Lar loog exp(y= kj/r) + Σ exply, kī,/T)*
where r is a hyper-paramete parameter that controls the concentration level. N is the size of the teacher dictio- nary. fetr can be intuitively interpreted as the log loss of a softmax-based (N+1)-way classification task. In our case, we attempt to classify qs as kj in the scope of (A:] } U (ki A...... " J-
The consistency in the teacher dictionary. The introduction of the fixed-size teacher dictionary decouples the size of the memory bank from batch size and dataset size. The teacher dictionary can be larger than the commonly-used batch and smaller than the dataset. Therefore, we bypass huge batch, which aims at providing in-batch negative samples. Besides, we can avoid sampling inconsistent negative keys from the memory bank. The core to learning good features by contrastive learning lies in the rich and challenging negative representations. In CRD (Tian et al., 2020) and WCoRD (Chen et al., 2021b), the negative keys are momentum updated. The momentum update to the negative keys brings about two main issues: (1) the negative keys were updated only when they were last processed, and (2) the update interval for each negative key can be highly different. The two issues cause inconsistent negative keys. To provide consistent negative keys, we update the teacher projection head in a momentum manner. Since Ir are frozen in the distillation framework, momentum updating the teacher projection head results in consistent negative keys. Specifically, denoting the parameters of fl as wi and those of f as Wa, we update Wi as:
We + melle + (1 mm))w.
m. E [0, 1] is momentum coefficient which adjusts the update smoothness. Since w. are optimized by the training optimizer, the momentum update of wt makes the teacher projection head f progress more smoothly than the student projection head f". Therefore, the difference between the teacher projection heads at different iterations can be made small. As a result, the negative keys encoded at different iterations can be consistent. Besides, the teacher dictionary itself is gradually updated. The representations of the current batch are enqueued, while the representations of the oldest batch are dequeued. This gradual replacement is beneficial for maintaining the consistency of the queue since the oldest representations are the least consistent with the current ones.
3.2 REPRESENTATIONS OF ONE CLASS FLOCK TOGETHER
As shown in Eq. 2, classifying qa as kj in the scope of (ki, kū, kT,,.... } is discrimination on an instance level. However, ki which shares the same class label with ga should be close to q, in the representation space. Simply rejecting those kj is not beneficial for the student learning good features. To bring 9s closer to its aistacc-nnegatiee but class-positivy positive keys, whose nested functions up to the penultimate layer are denoted as g. (.). Specifically, the slow-moving student is implemented as momentum moving average of the student. The slow-moving student is
introduce slow-moving student
4

---

Under review as a conference paper at ICLR 2023
also accompanied with a projection head fa. which is also updated in a momentum manner. Denoting the parameters of Is as 8,, the parameters of 9¹, as 6¹, and those of f as w'z, we update 8 and wy by:
0 m, 8- + (1 m,)0, w/g t- m, w'g + (1 - my)w's:
where m, [0. 1] is another momentum coefficient and w, denotes the parameters of the student projection head J Therefore, the instance-negative but class-positive keys T.- can be obtain by:
C (''())))),,
the sestance-negative but class-positive keys
where x' is another view of i under the random data augmentations. Instead of directly narrowing
down the distance between and 4 we use Ys to predict ig which softens the constraint. Formally, a predictor hg implemented a two-layer perceptron, is proposed produce the prediction p. & (g())). The loss simply defined as the mean squared error between normalized ps and qz-:
2-- 9
Ps
Cpred
2 - 2
(6)
|pell
1pol12
11
Furthermore, symmetrize the loss by feeding x', to the student and x, to the slow-moving student compute Epred. Formally, denoting the representations output from x', by the student q and the corresponding instavce-negative but class-positive keys as To we compute Epred by:
Epred =
Po PPll2
T-
= 22 2(
P.
19:-12
PP
10--112"
Here Pa - h.( T:-
(r.)) and To A [g (x<)). Note that o- and Th- are detached from the current computational graph during the distillation process.
3.3 TRAINING THE STUDENT
With the slow- lovinoving student and the teacher, Eq. 2, Eq. and Eq. 7 aim at assisting the student to effectively learn powerful features through contrastive learning. The student still needs to learn features from the training data. For image classification, the task-specific loss is defined as the cross-entropy loss. Overall, the total loss Ltotal can be formulated as:
Ciota! = Xetr + >pred + Epred) + ActeColer
(8)
where Antry Apred and Act are three balancing factors. Lets A H(yyyy), where H(-)) refers to the standard cross-entropy, y denotes the one-hotel label and ys is the student output.
4 EXPERIMENTS
We validate the effectiveness of CoCoRD in improving the student performance. The student-teacher combinations are divided into two main categories: (1) students share the same architecture style with teachers, and (2) the architectures of the students are different from those of the teachers.
Datasets. To investigate the performance improvements of students, we employ two benchmarks: (1) CIFAR 100 (Krizhevsky et al., 2009) and (2) ImageNet-IK 1K (Russakovsky et al., 2015). CIFAR 100 has 100 classes and there are 500 training images and 100 validation images per class. ImageNet-IK, large-scale dataset, contains 1000 classes and provides 1.28 million training images and 50K validation images. To test the transferability of features that students learn by CoCoRD, we utilize two more datasets: (1) STL-10 (Coates al., 201 and (2) TinyImageNet (Chrabaszez et al., 2017). We only use the 5K labeled training images and 8K validation images from 10 classes in STL-10. Tiny TinylmageNet consists of 200 classes and each has 500 training images and 50 validation images.
4.1 EXPERIMENTS ON CIFARION 100
We experiment on CIFAR100 with 13 student-teacher combinations in total of which are student- teacher combinations with the same architecture style, and the remaining 6 are student-teacher combinations with different architectures. Table focuses on student-teacher combinations with the same architecture style, while Table 2 provides experimental results of student-teacher combinations with different architectures. As can be observed in both tables, KD (Hinton et al., 2015), a simple yet effective method, provides strong baseline. CoCoRD can consistently outperform KD and
'On CIFAR100 100, A Acte -1, A4rea=4. More training details and data augmentations provided in the supplementary materials
5

---

Under review as a conference paper at ICLR 2023
Table CIFARI00 aceuracy (%) students trained different distillation methods (ours CoCoRD), when having the same architecture style as the teacher. 1 denotes outperforming KD, and denotes underper- forming. For all the compared methods, we use author-provided or author author-verified code from the CRD repository. Our reported results are the averages over runs. The best result among the methods which are nor combined with another method is shown in bold. The hest result among the combined methods is underlined.
Teacher
WRN-40-2
WRN-40-2
resmet56 resnet20
resmet110 resnet20
resnet110 resnet32
resmet32x4 resmet8x4
vg
Student
WRN-16-2
WRN-40-1
vgg8
Teacher
75.61
75.61
72.34
74.31
74.31
79.42
74.64
Student
73.26
71.98
69.06
69.06
71.14
72.50
70.36
KD
74.92
73.54
70.66
70.67
73.08
73.33
72.98
FitNet
73.58 ())
72.24 (J)
69.21 (L)
68.99 (L)
71.06 (L)
73.50 (t)
71.02 (L)
AT
74,08 (1)
72.77
70.55 (L)
70.22 (L)
72.31 (L)
73.44 (1)
71.43 (L)
SP
73,83 (U)
72.43 (
69.67 ())
70.04
72.69 (U)
72.94 (L)
72.68 (L)
CC
73.56 (L)
72.21 (L)
69.63())
69.48 (L)
71.48 (L)
72.97 (L)
70.71 (L)
VID
74.11 (L)
73.30 (3)
70.38
70.16
72.61 (L)
73.09 (L)
71.23 (L)
RKD
73.35
72.22
69.61 (1)
(9)25
71.82
71.90
71.48 (L)
PKT
74.54 (L)
(3.45()
70.34 (
70)
72.61 (L)
73.64
72.88 (L)
AB
72.50 (L)
72.38 (
69.47 (1)
69,53
70.98 (L)
73.17
70.94 (L)
FT
73.25 (L)
71.59
69.84 (L)
70,22 (U)
72.37 (L)
72.86 (L)
70.58 (L)
FSP
72,91 (1)
n/a
69.95
70.11
71.89
72.62 (1)
70.23 (L)
NST
73.68 (1)
(2)24
69.60
69.53 (L)
71.96
73.30 (L)
71.53 (L)
CRD
75.48 (1)
71.14())
71.16
71.46
73,48
75.51 (1)
73.94 (1)
LCKT
75.22 (1)
71.11())
71)14
71.23 (1)
72.32
74.65 (1)
73.50 (1)
GCKT
75.47 (1)
74.23
71.21 (T)
71.43 (1)
73.41 (1)
75.45 (1)
74.10 (1)
CoCoRD (ours)
75.48 (1)
().17
71.74 (1)
72.11 (D)
74.20 (1)
75.29 (1)
73.99 (1)
CRD+KD
75.64 (1)
74.38
71.63 (1) 71,56 (1)
73.75 (1)
75.46 (1)
74.29 (1)
WCoRD
75.88 (1)
(1)
71.56 (1)
71.57 (1)
73.81 (1)
75.95 (1)
74.55 (1)
CoCoRD+KD
75.90
75.25 (1)
72.09 (1)
72.18 (1)
74.37 (1)
75.28 (1)
74.26 (1)
Table 2: CIFAR 100 test accuracy (%) of students trained with different distillation methods, when the teachers'
architectures significantly different from those of the students. denotes outperforming KD, and J. denotes underperforming. Our results are the averages over runs. The best result among the methods which are not combined with another method is shown in bold. The best result among the combined methods underlined.
Teacher Student
vg
ResNet50 MobileNetV2
ResNet50 veg8
resmet32x4
resmet32x4
WRN-40-2
MobileNetV2
ShuffleNetVi
ShuffleNetV2
ShuffleNetV1
Teacher
74.64
79.34
79.34
79.42
79.42
75.61
Student
64.60
64.60
70.36
70,50
71.82
70.50
KD
67.37
67.35
73.81
74.07
74.45
74.83
FilNet
64. (()
63.16 (U)
70.69 (L)
73.59 ())
73.54 (L)
73.73 (U)
AT
59.40 ())
58.58 (U)
71.84 (L)
71.73 (
72.73 (L)
73.32 (U)
SP
66.30 (L)
68.08 (1)
73.34
73.48
74.56
74.52 (L)
CC
64.86 (L)
65.43 (L)
70.25 (L)
71.14
71.29 (L)
71.38
VID
65.56 ())
67.57 (1)
70.30
73.38 ())
73,40 (L)
73.61 (U)
RKD
64.52 (L)
64.43 (L)
71.50
72.28 (1)
73.21 (L)
72.21 (L)
PKT
67.13 (L)
66.52 (L)
73.01 (1)
74.10
74.69 (1)
73,89 (U)
66.06 ())
67.20 (L)
70.65
73.55
74.31 (L)
73.34 (U)
FT
61.78 (
60.99 (L)
70.29
71.75
72,50
72.03 (L)
NST
58.16
(4,96
71.28 (L)
74.12
74.68
74.89 (1)
CRD
69.73 (1)
69.11(0)
74.30
75.11
75.65 (1)
76.05 (1)
LCKT
68.21 (1)
68.81 (1)
73.21
74.62
74.70
75.08 (1)
GCKT
68.78 (1)
69.20 (1)
74.29 (1)
75.18 (1)
75.78 (1)
76.13
CoCoRD (ours)
69.86 (1)
70.22 (1)
74.52 (1)
75,99 (1)
77.28 (1)
76.42 (1)
CRD+KD
69.94 (1)
69.54 (1)
74.58 (1)
75.12
76.05 (1)
76.27 (1)
WCoRD
69.47 (1)
70.45 (1)
74.86 (1)
75.40 (1)
75.96 (1)
76.32 (1)
CoCoRD+KD
69.26 (1)
69.89 (1)
74.62 (1)
76.48
77.39
76.56
achieve highly competitive performance compared with other state-of-the-art methods. Note that m.
in Formula 3 is set to 1 for the WRN-40-2WRRN-001 combination. Although the teacher projection head attached to WRN-40-2 is only randomly initialized and not updated during the distillation process, CoCoRD still achieves the state-of-the-ard result. This implies the features provided by the well-trained teacher from the penultimate layer are already distinguishing, which are then projected into the representation space by the frozen teacher projection head f..
Based on the discussion above, the teacher projection heads in Table 2 are randomly initialized since the difference in architecture style is very likely to bring about the difference in the input shape. Note

---

Under review as a conference paper at ICLR 2023
Table 3:
To evaluate transferability features learned by different distillation methods, we employ linear probing to perform 10-way classification on STL10 and 200-way classification on TinyImageNet. For this experiment, we use the combination of teacher WRN-40-2 and student WRN-16-2. Top-1 accuracy (%) is reported. The student baseline and teacher are trained from scratch. Details are in the supplemental materials.
Student
KD
AT
CRD
CRD+KD
CoCoRD
Teacher
69.93
70.82
70.39
71.36
71.59
73.63
68.31
CIFARI00 CIFARI00-TTiylmman -Tnylmmggmeee 34.53
33.83
33.80
35.88
36.07
38.39
32.38
Table 4: Top-1 and Top-S ervor rates (%) of the students ResNet-18 trained with different distillation methods on ImageNet-Ie IK validation set. The best performance is shown in bold.
Teacher
Student
AT
KD
SP
CC
CRD
CRD+KD
ReviewKD
SSKD
WCoRD
CoCoRD
Top-1 Top-5
26.70
30,24
29.30
29.34
29.38
30.04
28.83
28.62
28.39
28.48
28.51
28.26
8.58
10.92
10.00
10.12 10.20
10.83
9.87
9.51
9.49
9.33
9.84
9.30
that it is because of the projection heads that CoCoRD can achieve distillation under arohttec setting. The projection heads can project features at the penultimate layer of different shapes into one representation space, where we can easily define the contrastive loss based on Eq. 2.
As shown in Table 2, CoCoRD is highly effective for combinations of different architectures. Even the teacher projection head is not updated, CoCoRD can consistently achieve the best performance compared to those not combined with another method. Especially, for the resnet-32x4/ShufleeNee pair, CoCoRD presents 77.28% Top-1 accuracy, which is 1.5% higher than the second best GCKT (75.78%). On the other hand, methods based on intermediate features perform poorly with different- architecture combinations. The observation suggests that CoCoRD can largely blur the requirement for significant similarities between students and teachers. We conjecture that knowledge distillation based on features at the penultimate layer can avoid the conflicts of different inductive biases that different models exploit. This indicates that the proposed CoCoRD is more generally applicable for student-teacher combinations with different architectures.
Limitations. In Table 1, CoCoRD+KD does not bring further performance improvements over CoCoRD. The same phenomenon can be observed Table 2. MobileNetV2 (Sandler et al., 2018) does not obtain more performance improvements with CoCoRD+KD. These phenomena indicate that further investigations are needed to combine CoCoRD with other knowledge distillation methods and extremely lightweight student models are still challenging for knowledge distillation.
Linear probing. Following CRD (Tian et al., 2020), we employ linear probing to evaluate the transferability of the student features. We freeze the student and train a linear classifier the global average pooling features of the student to perform 10-way classification on STL10 and 200-way classification on TinylmageNet. As shown in Table. 3, CoCoRD exhibits strong transferability and outperform the second best (CRD+KD) by large margin on the two datasets (2.04% improvement on STL.0 and 2.32% on TinylmageNet). The proposed CoCoRD, which has a negligible performance drop on CIFAR 100 compared with the teacher, (Please see Table. 1), shows better transferability than the teacher (5.32% improvement on STL10 and 6.01% 1 TinyImageNet). The linear probing experiment indicates that CoCoRD-distilled models have better generalization ability.
4.2 EXPERIMENTS ON IMAGENET
To investigate the scalability of CoCoRD to large-scale datasets, we employ ResNet-18 and ResNet-34 as the teuchertteacher combination to perform experiments on ImageNet- IK. For fair comparison, we follow the standard PyTorch ImageNet training practice except that we have 100 training epochs like CRD and WCoRD. We also use the Py PyTorch-released ResNet-34/18 as our teacher/student. On ImageNet, we set =1,,=1, -dts==1, >pred=4 and only calculate Lyred- The 1 and Top-5 error rates of different distillation methods are provided Table 4 (the lower, the better). The results in Table show that the proposed CoCoRD achieves the best performance relative improvement of CoCoRD over WCoRD (Chen et al., 2021b) on Top-1 error is 14.45%, and the relative improvement of CoCoRD over CRD (Tian et al., 2020) Top-1 error is 40.43%. Both improvements validate the scalability of the proposed CoCoRD to large-scale datasets.
the large-scale ImageNet. The
4.3 ABLATION STUDY
4.3.1 STUDY OF ENCODER COMBINATIONS
By default, we use the teacher to generate representations for contrastive learning and the slow- moving student is employed produce representations of another view of the input of the student.
7

---

Under review as a conference paper at ICLR 2023
Table 5: CIFARI00 test accuracy (%) of resnet32 trained with different encoder combinations. The best performance and the corresponding encoder combinations are shown in bold. The teacher is resnetl mean denotes the average over 5 runs and std stands for the corresponding standard deviation. Note that the resnet 10 is pre-trained and the resnet32 is initialized the same the student and updated in a momentum manner.
Option
Eacoder
Cuitaatie
romut32
Couradio
Cognatc
Continaivs res
Cugnone
1ome1110
Cagusi
scmalli
Cognai:
Cagnnic
moc (fwod)
74,8 (10.14)
68 (107))
72.18 (10.37)
2377 (10.54
(:022))
Eails
Tea acturcy
73
73.6
T. resnett10, S: resnet32
reanet110, reanet32
3.07
0,08
3.05
01
0.1
71.0 736
512
1024 2031
2296 3197
16330
lempenaturre in InfeNNce (9) liffects of
The size of the teacher dichionary
(1) tiffects of the soacher distionary sine
Figure 3: Effects of the temperature (7) in InfoNCE with N=2048, as shown in (a), and the effects of the teacher dictionary size (N) with r=0.1, as shown in (b).
To investigate how the representation quality affects the distillation performance, we utilize different models to provide those representations. For clarity, the model that generates dictionary-caching representations is referred to as contrastive encoder. The model that produces the instance-negative but class-positive representations referred to as cognate encoder. Results are reported in Table 5. Comparing options A (the default option) and B, we can find that leveraging the pre-trained teacher to provide quality representations for contrastive learning is more beneficial for the distillation. Besides, removing the cognate encoder and setting Apred to zero (option E) lead to poor performance, suggesting the cognate encoder can alleviate the adverse impact of the potential noise. If we remove the contrastive encoder and still use the dictionary with cognate encoder (option F), the distillation process fails. The results in Table 5 can support the effectiveness of each encoder in CoCoRD.
4.3.2 STUDY OF MOMENTUM
Table 6: CIFARIOO test accuracy (%) ) of the student resnet32 trained with different combinations of m. (in Eq. equation 3) and m, (in Eq. equation 4). The best performance and the corresponding m. and m, are shown in bold. The teacher is resnetl 10 and the accuracy of vanilla student and the teacher can be found in Table 1. mean denotes the average over runs and std stands for the corresponding standard deviation.
1/4 0
0.999
0..999
me
0.99
0.999
0.9999
0.999
mean
73.00
72.87
73.34
73.89
74.07
73.70
73.80
73.80
73.65
73.54
72.93
72.86
73.12
std
0.33
0.32
0.23
0.24
0.14
0.32
0.12
0.25
0.22
0.52
0.40
0.50
0.20
As shown in Formulas 3 and 4, m. controls the progressing speed of the teacher projection head ff. while m, manages the speed of the slow-moving student and its projection head. To investigate the impact of momentum, my. The results are reported in Table 6. When =0 and m.=m,=1, CoCoRD can improve the student performance. The effectiveness in both cases implies CoCoRD is robust. Besides, with my fixed, large value of my (e.g. 0.99 or 0.999) works much better than m, suggesting that consistent representations in the teacher dictionary are beneficial for the distillation.
employ resnetl 10 the teacher to train resnet32 with different m. and
4.3.3 STUDY OF HYPER-PARANN PARAMETESS
The temperature т. The value of 7 in Eq. 2 varies from 0.07 0.1 As shown in Figure 3(a), CoCoRD is sensitive to T. Both extremely high and low temperature lead to sub-optimal performance. As suggested in CRD (Tian et al.. 2020), we set 7 to 0.1 for experiments on CIFAR100. while 7 is set to 0.07 on ImageNet. We suggest tuning the value of T based on the classification difficulty.
The size of the teacher dictionary. The number of negative keys is determined by the teacher dictionary size N. To investigate the effects of the teacher dictionary size, we validate various values of N. As shown in Figure ((b). extremely small teacher dictionary provides insufficient negative keys, leading to sub-optimal performance. However, the extremely large teacher dictionary can introduce
8

---

Under review as a conference paper at ICLR 2023
noise, which adversely affects the distillation performance. Based on our experiments, -N should suffice on CIFAR while N= 65536 on ImageNet. Note that the teacher dictionary in Co- CoRD is significantly smaller than the memory banks in CRD (Tian al., 2020) and WCoRD (Chen et al., 2021b), which is more economic for large-scale datasets.
The balancing factors. We conduct experiments on CIFARI00 to investigate the effects of the three bal- ancing factors Aetr, Acts and Apred- We use resnet32- resnetl as the student-teacher combination. For ex- periments on balancing factors, we set 7-0.1, N - m.=0.999 and m,=0.9. " "X" denotes we set the bal- ance factor to 0 and "v" means we set the balance fae- tor to the corresponding value provided in the second row. Details simple grid search for each balancing factor can be found in the supplementary material. As can from Table. 7, all components in CoCoRD are essential for achieving high distillation performance. When Aety
Table 7: The effects of the three balancing factors. CIFARI00 test accuracy (%) is reported. The best performance shown bold. Average over 5 runs. More details can be found in the supplementary material.
Aots
Aete
Apred
mean (std)
74.20 (+0.14)
x
/
fails
x
/
71.11 (#0.42)
/
x
72.92 (#0.23)
O, there is serious performance drop, which indicates contrasting student representa- tions with negative keys teacher dictionary necessary in improving the student performance. Moreover, by comparing the result when xpre==) with the result when Apred=4, we can see that the slow-moving student can reduce the negative effects of the noise the teacher dictionary.
4.4 TRANSFER LEARNING
Table 8: Transfer learning. For PASCAL VOC, Faster R-CNN is fine-tuned on VOC trainva 107+12 and evaluated on 2007test, For COCO, Mask R-CNN is fine-tuned on COCO train 2rain2017 7 and evaluated on va12017. The Faster/Mask R-CNN models with the R50-C4 backbones (He al., 2017). Numbers in green indicate the performance improvement over detectors initialized by vanilla student, Please see the supplementary material for details. ResNet101 is the teacher with 77.37% top-1 accuracy on ImageNet.
Classification
Object Detection
ImageNet
PASCAL. VOC Detection
CoCo Derection
Top-1 accuracy (%t
AP30
AP15
AP3g
AP15 27.8
scratcc
60.2
33,8
33.1
44.0
26.4
81.3
CRD
76.86 (+0.71)
81.7 (10.4)
(4.2))
(+1.2)
60.5 (+0.6)
40.7 (+0.7)
43.9
(+0.8)
CoCoRD
77.57 (+142)
82.0 (+0.7)
(+1.5)
61.1 (42.3)
60.9 (+1.0)
(+100
44.5 (=1.4)
We further validate the feature quality of CoCoRD disttileed models by transferring the model weights object detection task, including PASCAL VOC (Everingham al., 2010) and COCO detection (Lin et al., 2014). We fine-tune the pre-trained models in an end-to-end manner on the target datasets. The detector for PASCAL VOC is Faster R-CNN (Ren et al., 2015) with a backbone R50-C4. For COCO object detection, the model is Mask-RCNN (He et al., 2017) with the R50-C4 backbone. Note that the accuracy classification. As shown in Table 8, the CoCoRD-initializes detectors exhibit better performance than the student-initialiin and CRD-initialized counterparts. The valid reuse of model weights further demonstrates the transferability of
ResNet50 can outperform the teacher ResNet101 by 0.2% top-1
features.
5 CONCLUSION
In this paper, we propose a contrastive-leaming leaneni----n- based knowledge distillation method named Con- trastive Consistent Representation Distillation. From a perspective of regarding contrastive learning as dictionary looking-up task, we build a fixed-size dictionary to cache consistent teacher represen- tations. Besides, to alleviate the adverse impact of the potential noise in the teacher dictionary, we employ slow-moving student, implemented as a momentum- based moving average of the student, to provide instance-negative but class-positive targets. CoCoRD does not employ the entire dataset as the memory bank, which is economic for large-scale datasets. Extensive experiments demonstrate that CoCoRD, which utilizes fewer negative keys, boost the performance of the students on diverse image classification datasets. Additionally, the models distilled by CoCoRD on ImageNet classification efficiently improve object detection performance PASCAL VOC and COCO.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. CVPR, pp. 9163-9171, 2019.
Defang Chen, Jian-Ping Mei, Yuan Zhang, Can Wang, Zhe Wang, Yan Feng, and Chun Chen. Cross-layer distillation with semantic calibration. AAAI, volume 35, pp. 7028-7036, 2021a.
Liqun Chen, Dong Wang, Zhe Gan, Jingjing Liu, Ricardo Henao, and Lawrence Carin. Wasserstein contrastive representation distillation. CVPR, pp. 16395, 2021b.
Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge review. In CVPR, pp- 5008-5017, 2021c
Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as an alternative the cifar datasets. arXiv preprint arXiv:1707.08819. 2017.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 215-22221 223.. JMLR Workshop and Conference Proceedings, 2011.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2): 303-338, 2010.
Ross Girshick. Fast r-cnn. In ICCV, pp. 1440-1448, 2015.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. ICCV, pp. 2961-2969, 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. In NeurIPS Deep Learning and Representation Learning Workshop, 2015.
Kurt Homiik Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks universal approximators. Neural networks, 2(5):359-366, 1989.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinherger. Densely connected convolutional networks. In CVPR, pp. 4700-4708, 2017.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pp. 740-755. Springer, 2014.
Sherjil Ozair, Corey Lynch, Yoshua Bengio, Aaron Van den Oord, Sergey Levine, and Pierre Sermanet. Wasserstein dependency measure for representation learning. Advances in Neural Information Processing Systems, 32, 2019.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In CVPR, pp. 3967-3976, 2019.
Peyman Passban, Yimeng Wu, Mehdi Rezagholizadeh, and Qun Liu. Alp-kd: Attention-based layer projection for knowledge distillation. arXiv preprint 2rXiv:20/2.14000 2020.
Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In CVPR, pp. 7263-7271, 2017.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems, 28, 2015.
10

---

Under review as a conference paper at ICLR 2023
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 15(3): 211-252, 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In CVPR. pp. 4510-4520, 2018.
Liangchen Song, Jialian Wu, Ming Yang, Qian Zhang, Yuan Li, and Junsong Yuan. Robust knowledge transfer via hybrid forward on the teacher-student model. In AAAI, pp. 2558-2566, 2021.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR, 2020.
Frederick Tung and Greg Mori. Simisarity-preseriing knowledge distillation. In In/CCV, pp. 1374, 2019.
Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv e-prints, pp. arXiv-1807, 2018.
Xinglu Wang and Yingming Li. Harmonized dense knowledge distillation training for multi-exit architectures. In AAAI, volume 35, pp. 10-218, 2021.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, pp. 3733-3742. 2018.
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In CVPR, pp. 4133-4141, 2017.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. ICLR, 2017.

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.I QUANTITATIVE RESULTS ON THE ACHIEVED SPEED-UP, MEMORY REDUCE AND OTHERS
In the following three tables, we provide quantitative results on the achieved speed-up, memory cost reduce, and other quantitative information about the teacher/student (T/S) combinations used CIFARION 100 (in Tabs. and 2) and those T7S combinations used on ImageNet (Tabs. and 8). The results are measured with Intel Core 17-8700 CPU on Ubuntu 20.04 operating system and memory cost is measured by Pytorch Profiler in a forward pass.
Table 9: Quantitative results the achieved speed-up, up. parameter compression and memory reduce. The combinations are from Tab. The inference latency is measured on image of size 32x32.
Combination(T/S)
Inference Latency (ms)
Speed-up
Memory Cost (MB)
Mult-Add
Parameters (K)
WRN-40-2/WRN-16
21 21.28/7.98
62.50%
11.73/4.39
327.62M/101, 12M
2255/703
WRN-40-2/WRN-40-1
21.28/10.51
50.61%
11.73/5.87
327. (1.662//83.999
2255/570
resnet56/resnet20
21.75/1 1.48
47.22%
8.72/3.21
125. 125.76M/40.22M,
862/278
resnetl 10/resnet20
56.09/1 1848
79.53%
16.97/3.21
263.16M/40.. 82M
1737/278
resnetl 10/resnet32
56.09/17.01
69.67%
16.97/5.05
163.16 13M
1737/473
resmet32x44resnet8k4
24.93/7.52
69.84%
20.71/6.03
1.08G/177.07X
7434/1234
vggl3/vgg8
8.72/3.89
55.39%
4.20/2.10
285.2M/96.33M
9462/3965
Table 10: Quantitative results on achieved speed-up, parameter compression and memory cost reduce, The combinations are from Tah. 2. inference latency is measured on image of size 32x32.
Comhinationn(1SS)
Inference Latency (ms)
Speed-up
Memory (MB)
Muli-Add
Parameters (K)
Vgg13/MobileNal/
8.72/14.97
4,20/3.39
285. 285.2M/6.54M
9462/813
VesNet00MohileNav2
17.01/14.97
11.99%
3.63/3.39
83.67M/6.5
23713/813
ResNes50lvgg8
17.01/3.89
77.13%
3.63/2.10
83.67M/96.33M
2371 3/395/
vemneti2x4/Shutleeeaai
20.71/13.90
1.08G/38.72M,
7434/949
resaet3244/hhffeeNeee22
24.93/19.01
23.75%
20.71/9.01
1.08G/44.52M
7434/1356
WRN-40-2ShhffeNNai1
21.28/31.84
11.73/13.90
327.62M/38.720 72M
2255/949
Table 11: Quantitative results on the achieved speed-up. parameter compression and memory cost reduce. The combinations are from Tabs, 4 and 8. The inference latency is measured on image of size 224x224.
Comhination(T/S) Inference Latency (ms)
Speed-up
Memory (MB)
(M)
ResNet34/ResNe1118 43.97/28.52
35.14%
59.82/39.75
3.66G/188
21.80/1 1.69
ResNet100/ResNet550
104.37/50.84
51.29%
259.72/177.85
7.80G/4.09G
44.55/25.56
Additionally, we compare the size of the teacher dictionary in the proposed CoCoRD with the size of the memory banks in CRD. Note that the keys in CRD memory banks only 128-d while the keys in the proposed CoCoRD teacher dictionary are 2048-d. Even with higher dimensions of the stored keys, CoCoRD are still more storage efficient.
Table 12: Comparison on the size of memory bank(s). Note that there is one teacher dictiionry in the proposed CoCoRD while there are two memory banks in CRD.
CRD
CoCoRD
Relative Size
CIFARIC0
32.77% 1311 1311,92MB
51.20MB
16.78MB
ImageNet
536.87MB
40.92%
12

---

Under review as a conference paper at ICLR 2023
A.2 THEORETICAL STUDY
Given two deep neural networks, a teacher f and a student f and let x be the network input. We denote representations the penultimate layer fa(xx) and f (xr), respectively. We would like bring (xx) and f (1;) close while pushing apart (x4g and I (x,, (x; and 2y represent different training samples).
For clear notation, we define variables S and T for the student representations and the teacher ones of the data, respectively: x ~ p(z);; S (z)) (x).
Let us define a distribution with variable C. The latent variable C' decides whether the tuple (f (zz), fr (zz)) drawn form the joint distribution p(T, S) (when C=1) drawn from the product of marginal distributions (T)) (when C=0).
(T, = 1) = pTT, (T, S|C = 0) = p(T)p(S)
Suppose we are given congruent pair drawn from the joint distribution (i.e. the same input provided to T' and S) for every N incongruent pairs drawn from the product of marginals (independent randomly inputs provided and S). Then the priors on the latent are:
N
qCC 1) 1) N
W (C -
N
By Bayes' rule and simple manipulations, the posterior for C is given by:
q(T, - 1)
PTT,))
q(C 1T (TT, () 1) P(T, 5) + ND(TT)(())
We can observe a connection with mutual information:
log qC 1)T log(1 + + N ((T)p(()
logg(N)) + log P(T,S)
PTT,S)
((T))
Taking expectation both sides w.r.t pTT, S) and rearranging gives
I(T: S) 2 log(N)+ + E,CT,s(C=1) log q(C 1),
where T(T; is the mutual information between the distributions of the teacher and student repre- sentations. Though we do not know the true distribution q(C' = 1|T', S), neural network can be used to estimate whether pair comes from the joint distribution the marginals.
By maximizing KL divergence between the joint distribution p(T, S) and the product marginal distributions p(T))((S), we maximize the mutual information between the student representations and the teacher representations.
13


Reviews:
[{"review_id": "rBJPngyCNR", "reviewer": "Reviewer_gYcu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces and mitigates some challenges encountered in contrastive representation distillation. In particular, the authors point out that the existing methods (CRD and wCoRD) keep track of a memory bank for the negatives that often contain inconsistent representations, which can adversely affect the student model performance. To that end, the paper proposes using separate heads for the teacher and student models where the teacher head is updated using the student head using momentum, allowing the teacher representations to be consistent across different iterations. As a side benefit, the proposed model requires a smaller teacher dictionary than previous works. The paper also proposes reducing the euclidean distance between samples referred to as \"instance-negative class positive\" samples using an additional slowly moving student model. Several experiments show highly competitive results in teacher-student settings.\n", "summary_of_the_review": "I think the motivation of the paper is sound and the proposed method combines several existing techniques to address the challenges in contrastive representation distillation. While the experimental analysis looks promising, I think the novelty is limited.\n", "strength_and_weaknesses": "**Strengths**\n\n1. The proposed method is evaluated on several sets of vision tasks including classification, transferability, and object detection. The performance of the method looks promising with highly competitive results.\n2. The effectiveness of the proposed method is shown in the experiments and ablations. The paper also contains comparisons with relevant baselines from recent years.\n3. The paper generally reads well and is clear for the most part.\n\n**Weaknesses**\n\n1. While the results seem good, I find the novelty in this work limited. Most of the ideas (like momentum update of params) already exist and are known in the self-supervised learning and contrastive learning literature (though these ideas have not been applied in knowledge distillation).\n\n2. The motivation for section 3.2 is unclear. Mainly, the choice of using a slow-moving student model is not motivated or discussed. Furthermore, it is unclear how using a new view of the same input image ensures $q_r$ to become closer to \"instance-negative but class-positive keys\". \n\nOther Minor comments:\n\n1. I find the notation to be a little unclear. In equation 1, what does the index $r$ in $q_r$ refer to? It has not been introduced before. Is it supposed to be $s$ instead, since the input to the student model was $x_s$?\n2. It was mentioned in the introduction that when the representations are inconsistent, \"the student can easily contrast positive and negative samples\", why is that?\n3. In the introduction, Page 1 (last para, 3rd line): \"The student representations in the memory\" shouldn't it be \"teacher representations\" instead? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gEOywND-kQO", "reviewer": "Reviewer_iNks", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper propose a new way to combine knowledge distillation with contastive learning, which updates the encoders in a momentum manner to obtain efficient and consistent encoding. Experiments on both classification and detection have been conducted.", "summary_of_the_review": "Please refer to the strength and the weakness.", "strength_and_weaknesses": "Strength:\n1. Sufficient experiments have been conducted on both classification and detection.\n2. Good paper writing.\n\nWeakness:\n1. The performance improvements are not very significant. As claimed by the authors in the limitation section, a combination of their method with the previous KD does not lead to good performance. \n2. Lack of in-depth study on how the inconsistency of representations in the memory bank leads to negative influence. It will be better if quantitative results can be provided to study the relation between inconsistency and the performance of knowledge distillation.\n3. Lack of theoretical or experimental study on why does their method works.\n4. Some factors in their framework do not have enough ablation studies. For example, what if the branch of the slow-moving student is not used?\n5. SOTA knowledge distillation method usually distill the knowledge in not only the final layer but also the intermediate layers. However, in the proposed method, it seems that a projection head is required, which is usually only utilized at the final layer in previous contrastive learning methods. So I wonder whether this method can be used in knowledge distillation on the intermediate layers.\n6. In my opinion, the novelty of this paper is not that large, since the moving average trick has been widely used in contrastive learning, And KD+Contrastive Learning is also not novel.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "c52-gJz2Yf", "reviewer": "Reviewer_FUQY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a new contrastive distillation method that distills consistent representations with a light cost. Specifically, the proposed method uses 1) a fixed-size memory bank as the teacher dictionary to achieve the storage efficiency; 2) the moving average of the student's projection head as the teacher's projection head to produce consistent negative keys; 3) the moving average of the student model to bring $q_r$ closer to its instance-negative but class-positive keys. \n\nThe authors verify the effectiveness of the method on extensive tasks and models.", "summary_of_the_review": "The paper proposed a new contrastive distillation method that distills consistent representations with a light cost. The topic is important and the method is supported by extensive experiments. ", "strength_and_weaknesses": "Strength: \n\n1. The proposed method is well-motivated and novel. Improving the memory efficiency in contrastive learning is a valid practical concern. \n2. The experiment results on various models and tasks show clear gains, which demonstrate the effectiveness of the method.\n3. The paper provides ablation studies on all proposed components.\n\nWeakness:\n\n1. Can the authors provide a quantitative result on the achieved speed-up and reduce memory cost?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Q6tIZNvhaei", "reviewer": "Reviewer_mvUm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a way of performing knowledge distillation using contrastive learning. The authors point out that existing distillation methods that use contrastive learning have two main issues: (i) memory intensive - as it needs to store the representations of the whole dataset to construct negatives, (ii) the negatives which exist in the memory bank might not have been updated in a while, thereby potentially making the task of differentiating the positives (which are computed fresh in each iteration) from negatives (in the bank) easier than it should be. To this end, the authors propose using a queue data structure instead for storing the negatives. This queue is updated with fresh representations from the teacher and can also be much smaller than the size of the whole training dataset. Empirically, this alternative way of performing contrastive distillation helps in improving the performance of the distilled student a bit, and also helps in certain transfer learning setups.\n", "summary_of_the_review": "I like that the issues pertaining to contrastive distillation methods were raised, issues which I had not thought of before, but as explained in the “Weaknesses” section, I do not feel those issues are that important, particularly when resolutions to those issues could themselves introduce other issues. I do not think that the idea produced is novel/general enough for it to be applicable to all the problem settings, as is depicted in the results as well (Table1/2). Because of these reasons, I do not recommend acceptance. ", "strength_and_weaknesses": "Strengths\n\n- The issue about memory usage is a valid point to raise. CRD [1] does indeed need a big memory bank to store the features of the whole training dataset, an issue which will get worse with a bigger dataset (e.g., ImageNet). This becomes particularly important when someone has to save the model and resume training; one does not only need to store the teacher/student’s weights, but also need to store the whole training data’s feature set.\n\n- The other main issue, which is about the negatives potentially being out of touch with the fresh features that the teacher is computing, could be important as well. I use the phrase “potentially” because it is not clear how big of an issue that really is in practice. \n\n- The paper is well written, the issues clearly explained, and the mathematical formulation is easy to follow.\n\n\nWeaknesses\n\n- Even though the issues that the authors have raised pertaining to CRD are valid, in the sense that it would be nice if those did not exist, those are not the issues which I feel stand out too much. As I said in the “Strengths” section, it is not clear how much effect it will have that the negatives sampled for a query were updated a while ago.\n\n- The reason the concern raised above (the importance of recency in feature bank) is important is because the results, especially on the most comprehensive evaluation setup CIFAR-100, don’t show that big of an improvement over the existing contrastive distillation setups like CRD. Sometimes they are even worse.\n\n- But, I think it is not just that the issue of “recency in feature banks” (in CRD) was not important enough, and hence there wasn’t that much improvement brought about by the proposed method. I think that the new proposed way of sampling negatives could potentially be not so ideal in certain situations. Here is how: the batch size for CIFAR-100 setup is set to 64 (as per supplementary) while the queue size is set to 2048. This means that a particular batch of features coming out of the teacher will persist in the queue for (2048/64) 32 iterations. This means that for many different query images, there will be the same set of negatives being used (32 times) until they finally get out of the queue. Compare this to CRD, where for each query image, the negatives will be randomly coming from the whole training set, thereby reducing the possibility of that issue. I do not know how important this difference is in practice, but the fact that there is not a consistent increase in performance for CIFAR-100 might hint at a problem like this. In summary, the method that has been proposed is not a superset of CRD in terms of (theoretical) effectiveness. \n\n- Overall, there is not a straightforward way to fix the issues that the authors have raised about CRD. They need to introduce an additional student network to counter some issues because of the queue data structure, which leads to additional loss components which need to be tuned.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject