Distilling Text-Image Foundation Models
Authors: Qifeng Wu
Huan Wang
Xu Ma
Yun Fu
Venue: Submitted to ICLR 2023 (2023)
Keywords: foundation models
CLIP
knowledge distillation
capacity gap

Abstract: Large pretrained foundation models (such as CLIP, DALL-E) are among the most recent significant advances in the AI community. Their implication is profound. This paper examines the value of these foundation models as a model knowledge base -- we aim to distill the knowledge in these foundation models for training lightweight models designed for specific tasks in practical application scenarios with improved performance. Despite abundant progress in knowledge distillation (KD) in traditional models trained under the supervision of class labels in datasets encoded as integers, distilling such text-image contrastive learning model has not been explored extensively. Meanwhile, KD is well-known for being bothered by the capacity gap problem (i.e., distilling knowledge from a teacher significantly larger than a student often degrades the performance of the student). The teacher-student capacity gap in distilling foundation models is even larger. Therefore, how to overcome this potential issue is also elusive now. This paper presents detailed analyses of these questions aiming to successfully tap into a pretrained foundation model (CLIP) to boost the student's performance. Besides the practical performance benefits, several interesting discoveries are unveiled: (1) CLIP is not bothered by the capacity gap, which may let us re-evaluate if the "capacity-gap" issue is really due to the capacity gap (2) We find the reason is largely due to that CLIP is not over-confident on the wrong labels when misclassifies input image samples.

Under review as a conference paper at ICLR 2023
DISTILLING TEXT-IMAGE FOUNDATION MODELS
Anonymous authors
Paper under double-blind review
ABSTRACT
Large pretrained foundation models (such as CLIP) are among the most recent sig- nificant advances in the AI community. Their implication is profound. This paper examines the value of these foundation models as a model knowledge base we aim to distill the knowledge in these foundation models for training lightweight models designed for specific tasks in practical application scenarios with improved performance. Despite abundant progress in knowledge distillation (KD) in tradi- tional models trained under the supervision of class labels in datasets encoded as integers, distilling such text-image contrastive learning model has not been ex- plored extensively. Meanwhile, KD is well-known for being bothered by the ca- pacity gap problem (i.c., distilling knowledge from teacher significantly larger than a student often degrades the performance of the student). The teacher-studen capacity gap in distilling foundation models is even larger. Therefore, how to over- come this potential issue is also elusive now. This paper presents detailed analyses of these questions aiming to successfully tap into a pretrained foundation model (CLIP) boost the student's performance. Besides the practical performance benefits, several interesting discoveries are unveiled: (1) CLIP is not bothered by the capacity gap, which may let us re-evaluate if the "capacity-gap" issue really due to the capacity gap (2) We find the reason is largely due to that CLIP is not over-confident on the wrong labels when misclassifies input image samples.
INTRODUCTION
Large, pretrained, foundation models (c.g.. CLIP (Radford et al., 2021). DALL-E 2 (Ramesh al., 2022) and GPT-3 (Brown et al., 2020)) are capable of many complex tasks such as zero-shot predic- tion the ability of models to predict the classes to which the input samples belong during testing without previous exposure to samples from that classes during training, generating images according to text prompts, generating images inspired by their originals, translating, reading comprehension, etc. However, the scales, the numbers of parameters that these models contain so large that would be difficult to deploy such models to devices with limited computing power such as mo- bile phones, tablets, and laptops. In addition, even though these foundation models are versatile, demonstrating great competence in abundant tasks that are considered to be challenging for regular neural networks, in some situations, however, instead of using all the functions that these models are capable of, we may only need to use parts of or even a derivative of their functions. These facts indicate that deploying a full foundation model in all use cases could be waste of computational resource and memory, and such intentions could be even impractical in some situations. Therefore, the study of techniques that could be applied compress or enable the utilization of portion of the functions of such foundation models would be valuable and necessary.
To use a portion or derivatives of the functions of these huge, pretrained foundation models, one promising mechanism is to transfer the knowledge from the foundation models to lightweight, task- specific models. In (Hinton et al., 2014), a knowledge distillation (KD) algorithm proposed, which is able to improve the task-specifer performance of a model with a smaller scale (the student network) by transferring knowledge from another model with a larger scale and better performance specific to the task to it. The KD algorithm proposed by Hinton et al. (2014) (HKD) aims at minimizing both the Kullback-Leible divergence (KL divergence) loss between the outputs of the teacher network and the student network along with the cross entropy loss between the student network and class labels. However, given the differences in network architectures along with pretraining methods between foundation models and conventional models, applying HKD directly on foundation models

---

Under review as a conference paper at ICLR 2023
may not be effective approach to exploit the knowledge within the foundation models benefit the performance of lightweight models designed for definite tasks. Firstly, the teacher network, in this case, foundation model, is not pretrained to optimize its performance on the particular task that the student network is designed for. Moreover, the intrinsic properties of the dataset utilized for pretraining foundation models could be different from that of the dataset we adopt for a specific task. In addition, in (Cho & Hariharan, 2019; Mirzadeh et al., 2020), the existence of "capacity gap" between a teacher and a student is believed be the major factor that prevents the performance of a student network from further improving when the teacher network contains more parameters and have better task-specific performance. When foundation model, which contains a considerably larger quantity of parameters compared to conventional models, is adopted as teacher network for knowledge distillation, this problem could become even more severe.
In this paper, we focus on the image classification task, exploring and investigating knowledge distillation-relatee properties of pretrained foundation model CLIP (Radford et al., 2021) under various experimental settings.
Our contributions are:
We notice that naively distilling knowledge from CLIP (Radford al., 2021) to student networks does not lead to satisfactory results, meaning such student networks do not out- perform those distilled from more commonly adopted teacher networks (e.g.. ResNet 34, 50 (He et al. (2). We hence propose a process to improve the accuracy of the teacher network on image classification before knowledge distillation, which is the fine tuning of CLIP. This accuracy is the upper bound of that of the student network.
We find that distilling from CLIP is not vulnerable to the "capacity gap" issue even when the difference in the number of parameters between the teacher network and the student net- work reaches more than thousand times. Moreover, when there are only limited training samples available, the superiority of CLIP in knowledge distillation increases. Our exper- imental results suggest the reason may well be related to the training recipe of CLIP in- stead of the network architecture. Our further quantitative analysis of the output of teacher networks reveals that it is more probable for image classifying models trained with cross- entropy criterion to give a high score to a wrong label on misclassification, which can later mislead the student network in knowledge distillation. On the contrary, giving a relatively high score to wrong labels is less likely for models trained under CLIP paradigm. This can have profound impact on the understanding of the capacity gap issue
Based on these findings, we assign our finetuned CLIP supervise the training of the lightweight model MobileNetV3 (Howard et al., 2019), network designed for CPU de- ployments. The achieved performance turned out to be notably higher than that of those trained from scratch or under the supervision of regular networks.
2 RELATED WORK
Knowledge distillation. Buciluă et al. (2006); Hinton et al. (2014) proposed improve the perfor- mance of lightweight models on particular tasks (e.g.. image classification, speech recognition) by forcing such models (the students) to mimic cumbersome, over-parameterized models (the teach- ers) on the output level. Romero et al. (2015) followed this notion and proposed to maximize the similarity between the student and the teacher with respect feature maps of hidden layers. Tian et al. (2020) proposed a contrastive learning objective, which allows student network to learn much more important information from the data representation produced by teacher network. In other works related to knowledge distillation, the knowledge to be transferred from a teacher to stu- dent is defined as the association among input samples (Park et al., 2019; Tung & Mori, 2019), the probabilistic distributions of features (Passalis & Tefas, 2018), etc.
Capacity gap issues. Intuitively, with the supervision of a more complex teacher network compris- ing more parameters, the student network should be trained to perform better. In reality, however, the performance of a student network could not be enhanced indefinitely or become arbitrarily close to that of its corresponding teacher network. Cho & Hariharan (2019) pointed out the phenomenon that larger models may not correspond to better performing student networks, which was explained by their "mismatched capacity"". They proposed to adopt early-stopped knowledge distillation (Cho
2

---

Under review as a conference paper at ICLR 2023
Text Input with class labels embedded in(e.g.. photo of (labef)
T1
TM
Text Encoder
Ttle
Taly
Linear Block
Linear Block
Tal
Tale
T2lu
Linear
CLIP
Output
Image Encoder
Tiol
Batched Images
MLP
Namm fd
Hatche Namm
Linear
ReLU
Diepou
Linear Block
Figure 1: Finetuned CLIP system architecture. The standard prepossessing procedures, with data augmentation for ImageNet (Deng et al., 2009b) (e.g.. random resized crop, horizontal flip, normal- ization, etc.) are applied to the batched images as the input of CLIP image encoder. Class labels in the dataset we adopt (imagenet100) are embedded phrases as the input of CLIP text encoder (c.g., photo of (husky)). The labels are given extra context information to climinate ambiguities and to explain proper nouns (c.g., kuvatz + kuvatz, a rype of dog: - fig. a type of fruit). The pretrained CLIP gives positive text-image pairs high cosine similarity while suppressing that of the negative pairs, such that an image matched to its corresponding label in the form of text. A multi-layer perceptron appended
CLIP (Radford al., 2021) takes in its output. Parameters in the MLP are optimized in the finetuning process, during which the parameters in CLIP are frozen. The finetuning process is supervised by the integer-encoden labels in our imagenet100 dataset.
& Hariharan, 2019) to improve the performance of the student networks. Mirzadeh et al. (2020) proposed improve knowledge distillation performance by adopting intermediate-sn models to compensate for the capacity gap between the teacher networks and the ultimate student networks.
Foundation models. Foundation models are those with vast scale and trained on large amount of data, such that they are competent in various downstream tasks (Bommasani et al., 2021). I). In (Brown et al., 2020), GPT-3, a model with 175 billion parameters, demonstrated its prominent abil- ity in reading comprehension, commonsense reasoning, translating, etc. The "Bidirectional Encoder Representations from Transformers" or BERT (Kenton Toutanova, 2019), a language model pre- trained unlabeled text can be adapted for variety of natural language-redd rellaue related tasks (c.g., language inference, question answering) without major architectural modifications for specific tasks. Instead, finetuning the pretrained model with extra output layer would be sufficient. CL.PP (Radford al., 2021) models trained on a dataset consisting of 400 million text-image pairs, are competent in the zero-shot task on multiple datasets. DALL-E (Ramesh et al., 2022), model that adopts the frame- work of CLIP (Radford et al., 2021), is able to generate images based on text input and variations of images inspired by the originals.
Knowledge distillation on foundation models. Knowledge from foundation models can be trans- ferred variety of lightweight, task-specific models through knowledge distillation and hence improve their performance. In (Chen et al., 2019; Tang et al., 2019), knowledge within pretrained BERT (Kenton & Toutanova, 2019) was distilled leverage small-scaled models designed for spe- cific tasks such as natural language understanding, text generation, sentiment classification, etc. Notably, before knowledge distillation, the teacher network -BERT (Kenton & Toutanova, 2019) was finetuned. Jiao al. (2019) managed to shrink BERT (Kenton & Toutanova, 2019) as a whole through knowledge distillation without damaging its versatility and performance on different tasks. Wang et al. (2022) proposed multimodal adaptive distillation to improve the performance of uni- modal encoders in vision- languaee tasks (e.g., visual commonsense reasoning, visual question an- swering, visual entailment, etc.).
3

---

Under review as a conference paper at ICLR 2023
3 METHOD
3.1 PREREQUISITES: KNOWLEDGE DISTILLATION
In this paper, we adopt the knowledge distillation proposed by Hinton al. (2014) as the technique to transfer knowledge from CLIP to lightweight models, and we refer to this algorithm as Hinton knowledge distillation (HKD). The complete objective of HKD is a linear combination of two sub- objectives:
LKK aLKLDD + BLCCE
where and 3 are adjustable loss L Live and the cross-entropy loss LeE respectively. The cross-entropy loss asks a student network to learn from the hard labels of datasets:
weighting the Kullback-Leibler (KL) divergence
LCe Hyy,y
(2)
where y denotes the class labels encoded as integers and y denotes the output of student network. The KL divergence loss asks the student to mimic the teacher on the output level. In the calculation of KL divergence loss, a hyperparameter named distillation temperature T is introduced to soften the output of both the teacher and the student, allowing the probability distribution of teacher output to be more informative:
LKLDIE 7 K2 (T)la) /T)).
where y denotes the teacher output and a denotes softmax function.
3.2 FINETUNING CLIP
CLIP (Radford et al., 2021) stands for contrastive language-image pretraining. The two major com- ponents of the model are an image encoder and a text encoder. The inputs of CLIP are text-image pairs and the pretraining of the model enables the image encoder and the text encoder to generate adequate representations of the input images and text respectively. In addition, the representation of an image is trained to match the corresponding text representation by maximizing the cosine simi- larity between positive pairs while minimizing that between negative pairs. That is, let I,.... be the normalized image features and Ti.... be the normalized text features. The objective of pretraining to maximize I, T.T. for i = and minimize T.T for +
The finetuning of CLIP aims at optimizing the system performance on specific tasks and datasets, which contains two processes: the improvement of text prompts and the refinement model output. Figure 1 gives an illustration of the system architecture, in which CLIP (Radford et al., 2021) is implanted.
3.2.1 TEXT PROMPTS IMPROVEMENT
For image classification task, the text input of CLIP is usually class names embedded in sentences or phrases (e.g., this is a photo of {class}) and for CIFAR-10 dataset (Krizhevsky et al., 2009), the class names could be automobile, airplane, horse, etc. However, for larger datasets with more classes like ImageNet (Deng et al., 2009a), some class names become ambiguous due to polysemy while others could be proper nouns. Our approach providing extra descriptions to or specifying the parent class of certain labels. For instance, type of motor vehicle manufactured by FORD, with Model T. automobile, car, and substitute Sains Bernard with Saint Bernard, type of dog respectively according to what the labels in the dataset actually refer to.
replace the labels Model T, which refers to
3.2.2 OUTPUT REFINEMENT
To improve the performance of CLIP on a specific task and dataset, the output of CLIP refined. The procedures for refining the output of CLIP are: (1) appending extra multilayer perceptrons (MLP) fitting a particular dataset for image classification to the CLIP model; (2) freezing all the parameters in CLIP: (3) optimizing the parameters in the MLP on a dataset under the supervision of integer-encoded labels using conventional cross entropy criterion on the image classification task. This process can be mathematically formulated below.
4

---

Under review as a conference paper at ICLR 2023
Assume æ. t to be the image and text input CLIP respectively, and WCLIP to be the parameters in CLIP. Then we denote the output of CLIP to be:
y [(WLLIP x,
(4)
The contrastive output of CLIP y, in which image embedding is matched to its corresponding text embedding, will be passed to the input layer of the adjoining MLP. Let WMLP be the parameters in the MLP, and hence its output can be written as:
h WWWMP, y).
(5)
Let z' represents the ground-truth vector of labels encoded as integers, the objective function of output refinement can then be expressed as:
Lrefine H (2', (z',z)) x),,
which is to be minimized with respect WMLP through training.
3.3 EXPLORING THE CAPACITY GAP
Capacity gap or mismatched capacity (Cho & Hariharan, 2019) is considered to be a significant factor leading the phenomenon that teacher with higher capability may not necessarily further enhance the performance of a given student network in knowledge distillation. In this part, we pro- pose approaches examine the capacity gap resistance property of CLIP and metric revealing the reason explaining why distilling CLIP is not bothered by the capacity gap.
3.3.1 EXAMINE CAPACITY GAP RESISTANCE PROPERTY
Baseline comparison. Under this setting, a common, relatively small-scaled convolutional neural network (CNN) is selected to be the student, while regular CNNs and the finetuned CLIP model are selected to the candidate teacher networks. Comparisons are conducted among the accuracy of student networks with identical structures but distilled from different teacher networks. The involvement regular CNN with different scales is intended to demonstrate the negative impact of the capacity gap.
Reduced student network width. Reducing the widths student networks (CNN) means decreas- ing the number of filters in each of the convolution layers, resulting in reduction in the number of parameters in student networks. With the candidate teachers unchanged, the difference in parameter number the gap in capacity within teacher-student pairs will be enlarged, and hence the capacity gap resistance of the teacher networks can be further justified.
Low-shot classification. In this case, the students and teachers are both exposed to limited quantity of training images. Specifically, given a dataset D, the train set of D is denoted as Dirain. For each class in D. & pictures in Dtrein selected to form the train set for low-shot classification, while the test set of D denoted as Drest is adopted directly without any modification. This setting investigates the influence of the capacity gap under the situation of low training samples.
3.3.2 CAPACITY GAP RESISTANCE RELATED METRIC
In knowledge distillation, the media allowing the knowledge to be transferred from the teacher and the student is their output. The resistance to the capacity gap should be related to one or more quantifiable features within the output of the teacher networks. In the image classification task, the model output corresponding to an input image is an n-dimensional vector y", where n is the total number of classes in the given dataset. A well-trained model would assign the highest score to the element in y matching the class to which the input image belongs, otherwise, the input image is deemed (vG...,, ,.., ..., considered be correctly classified if and only yf is the maximum element y We believe that when the teacher misclassifies an input image, meaning the highest score is assigned to the element in yd not matching the label of the input image, if the score is relatively high (over confidence), the student could hence be misguided. Therefore, we propose a probabilistic metric to evaluate the ahooe--eentooned phenomenon that occurs in teacher network output:
be mistakenly classified. That is for label (O.........., n 1} and y² - where i E [0 - 1]. Assume an input image is with label i, and the image is
Nerr&oc
p
Nerr
5

---

Under review as a conference paper at ICLR 2023
Table 1: Student Accuracy (%) on test of imagenet100 among different teacher-student pairs. ResNet 18 (He et al., 2016) is adopted as the student network. "Params" stands for the number of parameters in models and it is calculated number between the teacher and student, which is measured in the number of times. "Scratch/Nonee stands for training from scratch without the supervision of teacher networks.
millions. "Param Gap" denotes the gap in parameter
ResNet 18 + ImageNet100
Teacher network
Teacher Accuracy(%) Student Accuracy(%)
Params
Param. Gap
Scratch /None
84.10
11.22
1.00
ResNet 34 (He et al., 2016)
85.98
85,88
21.32 M
1.90
ResNet 101 (He al., 2016)
87.36
85,80
42.60 M
3.80
Raw CLIP (Radford et al., 2021)
90.77
85.78
291.00 M
26.05 x
Finetuned CLIP (ours)
95.88
86.02
291.27 M
26.07 x
where Nerr represents the number of misclassified samples and Nerr&oe represents the number of samples that are both misclassified and the corresponding output vectors experience over confidence. In this paper, we define an output vector y" of model is over confidence if::
ma (0, 179; >= 7
(8)
where y = =oftmax(y")) y" = 1366..... .,. ..... yn 1]. and y is an adjustable parameter.
EXPERIMENTS
4.1 BASIC SETTINGS
Experiments in this work are conducted on machine with 4 NVIDIA GTX TITAN Xp GPUs. Data parallel technique is utilized. Dataset selection, networks involved in experiments, and configuration of hyperparameters are introduced as follows.
Dataset In this paper, we use a subset of ImageNet (Deng et al., 2009a) containing 100 classes randomly sampled from the original ImageNet dataset, and we call this dataset imagenet100. This dataset contains 1.2 million training samples and 50k testing samples, suggesting there are 1200 training images and 50 testing images per class. The sampling work only reduces the total number of classes and images compared to the original dataset, while the scales, aspects, and contents the images remain unchanged.
Networks The networks involved in our experiments are ResNet 18, 34, 50, 101 (He et al., 2016), and CLIP (Radford et al., 2021). We choose pretrained VIT-L/14, version of Vision Transformer (Dosovitskiy et al., 2021) to be the vision encoder in CLIP. The MLP appended to CLIP consists of trivial layers (see Figure 1) in neural networks: fully connected layers, batch normalization layers, dropout layers and we use ReLU as the activation function.
Hyperparameter settings We adopted part of the settings in (Matsubara, 2021), in which slightly higher student accuracy was reported (71 (71337% compared to that reported in (Hinton et al., 2014) (70.66%). Modifications have been made the hyperparameter configuration to enable it to be suit- able for our hardware. The number of training epochs in pretraining or finetuning teacher networks and knowledge distillation is set to 100, the batch size 128. The initial learning rate is set to 0, with multi-step learning rate decay schedule at the epoch 60 and 90 by factor of 0 1. Stochastic gradient descent optimizer is chosen in our experiments, with momentum of 0 9 and weight decay of 1e-4. In knowledge distillation experiments, we assign the distillation tempera- ture F to be 1, indicating that no label softening is applied. The cross-entropy loss (weighted by 3) between student output and class labels with integer encoding and the KL divergence loss (weighted by a) between the output of teacher and student contribute equally to the total loss in knowledge distillation. That is,
0.5.
4.2 BASELINE PERFORMANCE COMPARISON
We adopt ResNet 18 (He et al., 2016) to be the student network, ResNet 34, 101, CLIP (Radford al., 2021) without being finetuned (Raw CLIP) and finetuned CLIP be the candidate teacher

---

Under review as a conference paper at ICLR 2023
Table Accuracy (%) comparison test set of imagenet100 among different teacher-student pairs, with reduced student network width. ResNet 18 (He et al., 2016) is adopted as the student network. "(1/8)" suggests that the number of filters each convolution layer in residual blocks of the student network has reduced to 1/8 compared to that in the original structure. "Params" stands for the number of parameters models and it is calculated in millions. "Param. Gap" denotes the gap in parameter number between the teacher and student, which is measured in the number of times. "Scratch/None" stands for training from scratch without the supervision of teacher networks.
ResNet 18 (1/8) 1m0
Teacher network
Teacher Accuracy(S% (%)
Student Accuracy
Params
Param. Gap
Scratch /Noee
66.32
0.19 M
1.00 x
ResNet (He al., 2016)
85.98
66.48
21.32 M
109.45 x
ResNet 101 (He et al., 2016)
87.36
65.80
42.60 M
218.70 x
Finetuned CLIP (ours)
95.88
66.74
291.27
1495.54
networks. Except for raw CLIP, all models are pretrained or finetuned on our imagenet100 dataset. observe that even when the difference in parameter number
From the results shown in Table between the finetuned CLIP and ResNet 18 reached 26 times, the student still achieves the highest accuracy. In comparison, the parameter number difference between ResNet 101 and ResNet 18 is only 3.8 times but the student accuracy is slightly lower than that distilled from ResNet 34, which is a sign that the ResNet 101 ResNet 18 pair negatively influeneed by the capacity gap in knowledge distillation while the finetuned CLIP ResNet 18 pair is not. Further experiments are conducted to justify this observation.
4.3 REDUCED STUDENT NETWORK WIDTH
We further examine the observation we have in Section 4.2 that despite finetuned CLIP having the largest amount of parameters among all can- didate teacher networks, knowledge distillation from it shows no sign of being influenced by the capacity gap. We enlarge the parameter number difference in a teacher-student pair by fixing candidate teacher networks while shrinking the number of fil-
RN18 Scratch
90
RN34-RN18
80
RN101-RN18
Student Accuracy (%)
Finetuned CLIP-RN18
70
50
40
ters in convolution layers in residual
blocks in ResNet 18 (He al., 2016),
30
which is the student network in our work. In our experiment, the filter number in the student network is duced to 1/8 when compared to that in the original structure. Results in table 2 show that even the param- eter number gap between finetuned CLIP and the student network be- comes around 1.5 thousand times, the student network trained under the su- pervision of maintains the highest accuracy among all students in dif- ferent teacher-student pairs. In con- trast, in the ResNet 101-ResNe 18 pair, even there smaller gap in parameter number than that in finetuned CLIP-ResNet 18 pair, the student accuracy falls even lower than that trained from scratch. This could be viewed signal indicating that the capacity gap has a detrimental impact on knowledge distillation in ResNet 101 Ret-Re 18 pair.
10
0
50
100
200
500
1000
Num @uum_sapless Saaplleeeeee_eesesee
Figure 2: Low-shot distillation. Accuracy of student net- works trained with the supervision of different teachers and exposed different numbers of training samples in ima- gener100. We choose candidate sample numbers in each class to {50, 100, 200, 500, 1000].

---

Under review as a conference paper at ICLR 2023
Table 3: Low-shot distillation, where both students and teachers are exposed limited number of training samples. Accuracy (%) comparison test set of imagenet/00 among different teacher- student pairs. ResNet 18 (He al., 2016) is adopted as the student network. "Scratch/None" stands for training from scratch without the supervision of teacher networks. "Param Gap" denotes the gap parameter number between the teacher and student, which measured in the number of times. "k" denotes the number of training samples per class.
ResNet 18 ImageNet100
Teacher network
Student Accuracy (%)
Param. Gap
k
Scratch /None
36.82
1.00 x
50
ResNet 34 (He et al., 2016)
37.80
1.90 x
50
ResNet 101 (He et al., 2016)
34.46
3.80 x
50
Finetuned CLIP (ours)
39.86
26.07 X
50
Scratch /None
48.78
1.00 X
100
ResNet 34 (He et al., 2016)
50.68
1.90 X 100
ResNet 101 (He et al., 2016)
48.64
3.80 X 100
Finetuned CLIP (ours)
51.32
26.07 X 100
Scratch None
62.24
1.00 X 200
ResNet 34 (He et al., 2016)
63.52
1.90 X 200
ResNet 101 (He et al., 2016)
63.24
3.80 X 200
Finetuned CLIP (ours)
64.70
26.07 X 200
Scratch /None
77.26
1.00 X 500
ResNet 34 (He et al., 2016)
78.86
1.90 500
ResNet 101 (He et al., 2016)
77.92
3.80 X 500
Finetuned CLIP (ours)
78.90
26.07 X 500
Scratch /None
83.12
1.00 X 1000
ResNet 34 (He et al., 2016)
84.40
1.90 X 1000
ResNet 101 (He et al., 2016)
84.28
3.80 X 1000
Finetuned CLIP (ours)
84.86
26.07 X 1000
4.4 LOW-SHOT CLASSIFICATION
We explore the capacity gap resistance under low-shot settings, meaning only limited number of training samples are available. Similar to Section 4.2, candidate teachers are ResNet 34, 101, and Finetuned CLIP, with ResNet 18 to be the student network. Training data is portion of our imagenet /00 dataset, that is, for each of the class in imagenet100, images in the original training set sampled to form the training for low-shot classification, where k (50, 100, 200, 500, 1000)- Each teacher is trained or finetuned on the low-shot training sets and later utilized to supervise the training of the student network. In other words, regular candidate teacher networks (ResNets), the MLP appended to CLIP and the student network is exposed to the same training set for low-shot classification one low-shot setting. The results of low-shot classification are shown in Table and Figure 2. We observe that with lower average quantity of training samples with respect to the number of classes in the dataset, regular teacher networks become more vulnerable to the capacity gap, meaning the student accuracy degrades rapidly (ResNet 101 ResNet 18). In contrast, for finetuned CLIP, the student network distilled from consistently outperforms the rest especially when the number of available training samples is limited.
4.5 CAPACITY GAP RESISTANCE RELATED METRIC
In the work of CLIP (Radford et al., 2021), several vision models (image encoder) are trained includ- ing modified ResNet 50 and ResNet 101 (He et al., 2016) under the supervision text with class labels embedded in, where the modifications are imposed convolution layers and pooling layers outside residual blocks. We adapt these two pretrained image encoders to knowledge distillation on the image classification task. In addition, two networks having the same structure the modified ResNet 50, 101 are trained respectively but under the regular cross-entropy paradigm supervised by integer-encoded labels of the dataset. To further investigate the capacity gap resistance property
8

---

Under review as a conference paper at ICLR 2023
Table Comparing models with different pretraining methods, CLIP (Radford et al., 2021) versus regular cross-entropy paradigm. "*-img-enc" denotes pretrained image encoder in CLIP(Radford al., 2021), " "-mod-CE" denotes models having the same structures as that of the corresponding image encoders mentioned above but trained with cross-entropy loss. we again utilize ResNet 18 (He et al., 2016) with reduced filter number in residual blocks as student. p is the metric proposed in Section 3.3.2, which measures the probability that the model gives relatively high score to the wrong label in the model output on the condition that the highest score is assigned to a wrong label (misclassification)) where the scores are elements of a model output vector passed through the softmax function. In our experiment, the threshold is set to 0.5.
is the threshold for determining whether score is relatively high,
ResNet 18 (1/8) + ImageNet100
Teacher network
p(> = 0.5)
Student Accuracy($e (%)
RN50-img--an (Radford al., 2021)
0.29
65.70
RN101-img-ene (Radford et al., 2021)
0.31
66.66
RN50-mod-CE
0.41
65.86
RN101-mod-CE
0.39
65.44
of CLIP, the above-mentioned four models are assigned to be the candidate teacher networks and the ResNet 18 with reduced width is adopted as the student network the following knowledge distillation experiment (see Table 4). For model trained under the regular cross entropy paradigm, when compared to a model adopting the pretraining method of CLIP, more likely that would give the wrong label high score when misclassifying a sample, and we deem this overconfidence will misguide student network in knowledge distillation.
4.6 EXTRA KNOWLEDGE DISTILLATION EXPERIMENTS
We extend the superior performance of CLIP in knowledge distillation to supervise the training of MobileNetV3 (Howard al., 2019) and perform series of knowledge distillation experiments on our imagenet100 dataset. See Table 5.
Table 5: Accuracy (%) comparison test set of imagenet100 among different teacher-student pairs. MobileNetV3-L. (Howard et al., 2019) is adopted as the student network. "Params" stands for the number of parameters in models and it is calculated in millions. "Param Gap" denotes the gap in parameter number between the teacher and student, which is measured in the number of times. "Scratch/None" stands for training from scratch without the supervision of teacher networks.
MobileNetV3--L4 ImageNet100
Teacher network
Student Accuracy(%)
Params
Param. Gap
Scratch /None
80.94
2.77 M
1.00 x
ResNet 34 (He et al., 2016)
82.54
21.32 M
7.68 x
ResNet 101 (He et al., 2016)
82.00
42.60 M
31.77 x
Finetuned CLIP (ours)
84.76
291.27 M
104.99
5 CONCLUSION
In this paper, we have excessively examined that CLIP is robust to the impact of capacity gap issues in knowledge distillation under different experimental settings (extra small student network, low available training samples). We have demonstrated that the pretraining method of CLIP allows the model to overcome capacity gap issues because it is less likely for the model to overconfident on the wrong class label when it misclassifies input sample, which could mislead student network during knowledge distillation. This encouraging result suggests that the knowledge within CLIP could be further exploited through knowledge distillation to benefit networks with smaller scales designed to be deployed on devices with budget computational resources like mobile phones or those designed for tasks other than image classification.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
To the best of knowledge, the methods we proposed in this work and the experiments we con- ducted do not pose potential negative impacts on society and they comply with the ethical research standards of ICLR.
REPRODUCIBILITY STATEMENT
Code related to this work will be released upon publication for reproducing purposes.
REFERENCES
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv:2 072558 2021. 3
Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Wess,rt-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020. 1,3
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In SIGKDD, 2006. 2
Yen-Chun Chen, Zhe Gan, Yu Cheng, Jingzhou Liu, and Jingjing Liu. Distilling knowledge learned in bert for text generation. arXiv preprint arXiv:1911.03829, 2019. 3
Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In ICCV, 2019. 2.5
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In CVPR, 2009a. 4, 6
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009b. 3
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. 6
K. He, X. Zhang, Ren, and J. Sun. Deep residual learning for image recognition. CVPR, 2016.
2, 6, 8,9
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. In NeurlPS Workshop, 2014. 1, 2, 4, 6
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. arXiv preprint 1e0:::2250022444 2019. 2, 9
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv: 10351, 2019. 3
Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 4
10

---

Under review as a conference paper at ICLR 2023
Yoshitomo Matsubara. torchdistill: A modular, configuration-driver framework for knowledge dis- tillation. In International Workshop on Reproducible Research in Pattern Recognition, 2021. 6
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In AAAI, 2020. 2,
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In CVPR, 2019. 2
Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. ECCV, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICML, 2021. 1, 2, 3, 4, 6, 8,
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06 2022. 1,3
Adriana Romero, Samira Ebrahimi Kahou, Polytechnique Montréal, Bengio, Université Montréal, Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitmets: Hints for thin deep nets. In in International Conference on Learning Representations (ICLR, 2015. 2
Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. Distilling task- specific knowledge from bert into simple neural networks. arXiv preprint arXiv:1903 12136, 2019. 3
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR, 2020. 2
Frederick Tung and Greg Mori. Sinilarity-preserving knowledge distillation. In CVPR, 2019. 2
Zhecan Wang, Noel Codella, Yen-Chun Chen, Luowei Zhou, Xiyang Dai, Bin Xiao, Jianwei Yang. Haoxuan You, Kai-Wei Chang, Shih-fu Chang, et al. Multimodal adaptive distillation for lever- aging unimodal encoders for vision- languaee tasks, arXiv preprint arXiv:2204.104: 2022.


Reviews:
[{"review_id": "tFjgoUo5kW4", "reviewer": "Reviewer_JKPp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the knowledge distillation performance on image classification when using a finetuned CLIP model as a teacher.\nThe authors find that distilling from a finetuned CLIP does not suffer from the \"capacity gap\" issue that commonly happens when teacher and student have a large difference. The CLIP-based distillation also have good performance when downstream training samples are limited. They also present an analysis and find that the strength of CLIP-base distillation is largely due to that CLIP is not over-confidence on the wrong labels when misclassifies input samples.", "summary_of_the_review": "The study of this paper reveals that a finetuned CLIP can be served as a good teacher that does not suffer from \"capacity gap\". \nThis is an intersting direction that worth deeper investigation, for example, how could CLIP achieve such capability? Do other pre-trained models Does this conclusion holds for other datasets or tasks other than image classification? \n\nHowever, this paper fails to conduct an investigation thorough enough on any valuable direction. The experiments are also too limited to support the claims of this paper w.r.t benchmarks, teacher models and hyper-parameters.\n", "strength_and_weaknesses": "Strengths:\n+ The finding that a finetuned CLIP does not suffer from \"capacity gap\" is interesting.\n+ The finetuning method of CLIP seem novel in a limited degree.\n+ Introducing a metric that evaluates the over-confidence of teacher models.\n\nWeakness:\n- The study exhibited shows limited novelty, which I consider insufficient for an ICLR submission.\n  * Finetuning & distillation is a common practice for using large pre-trained models, for example the Big SimCLR [2].\n  * Finetuning a CLIP using both its image encoder and text encoder has been used by previous literatures, for example VLKD [1].\n  * Addressing the gap between teacher and student during distillation is indeed an import issue, as also mentioned by existing literatures such as [3], however, the scope of this paper is too limited (using CLIP as a teacher in image classification) to illustrate more values.\n- Methods are evaluated only on one customized dataset, resulting in suspicious effectiveness under more experimental settings.\n  * It is weird why the authors choose a customized subset of ImageNet (ImageNet-100) for evaluation instead of using the original ImageNet;\n  * More public datasets such as CIFAR or others should be considered to prove that the effectiveness of the proposed method is solid.\n  * Comparison of pre-trained teachers involves too few teacher types in Table 4, more should be included covering more network structures (e.g. ViT) and more pre-training strategies (e.g. MAE, MoCo, ...). Note that most these pre-trained backbones are publicly avaiable. \n  * The relation between the threshold p and student accuracy is not extensively studied. \n\n[1] Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation. ACL 2022.\n[2] Big Self-Supervised Models are Strong Semi-Supervised Learners. NIPS 2020.\n[3] Analyzing the Confidentiality of Undistillable Teachers in Knowledge Distillation. NIPS 2021.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "nRc-T4FIngq", "reviewer": "Reviewer_Af7J", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper applies knowledge distillation to CLIP for image classification and finds that student performance increases when using CLIP as a teacher despite the teacher being a larger size. Some initial explanatory answers are proposed.", "summary_of_the_review": "In summary, this work evaluates a popular technique (finetuning CLIP) in a non-standard setting, and only on one dataset. An explanation is proposed for the phenomena, but this analysis is severely underdeveloped. In order to make this work ready for publication, proper baselines need to be implemented, the experiments need to be run on more classification datasets, and the analysis needs more experiments to control for confounders as well. These are all significant and major changes, hence my recommendation.", "strength_and_weaknesses": "Strengths:\n- The experiments that are presented seem to be well executed.\n- The proposed method for knowledge distillation from CLIP, using both image and text features, seems to be novel (though the standard baseline wasn't compared here).\n\nMajor Weaknesses:\n- Why is the standard CLIP finetuning procedure not compared against? It is unclear as a reviewer whether the phenomena observed in the work is a result of the new proposed CLIP finetuning procedure or the standard CLIP finetuning procedure (which is widely used). In the standard procedure, the text encoder is thrown away, and a linear classification layer is appended to the end of the image encoder, which is then finetuned via the CE loss.\n- The gap resistance analysis is severely underdeveloped and has a major flaw. In Table 4, is it not the case that the img-enc versions have higher accuracies than the mod-CE versions? Thus, the higher student accuracy could simply be explained by higher teacher accuracy. In order for the overconfidence ratio to hold weight, the accuracy confounder needs to be accounted for. For example, one way to do this would be to recalibrate the non-CLIP teacher model after training, which would lower the overconfidence ratio while keeping accuracy the same. If the proposed explanation is true, then this means the student network accuracy should increase.\n- Limited experimental settings. It is unclear how many of the claims made in the paper are specific to ImageNet100 and the ResNet model family since the experiments are only conducted in this setting. To show that the phenomena is broader, more classification datasets are needed (Cifar10, cifar100, full ImageNet at the very least) and more model families are needed (ResNet, EfficientNet, ViTs, etc).\n\nMinor weaknesses:\n- All tables and figures should have error bars displaying stddev over random seeds.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7ILmaL4R5s", "reviewer": "Reviewer_wF3p", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents analysis and empirical findings of distilling knowledge from large image-text foundational models (e.g. CLIP). The key observations  are 1) resistance to the capacity gap of the teacher/student models, 2) the image-text models are not overconfident on the wrong labels.", "summary_of_the_review": "Although the problem is important and the initial results are interesting, the gains over the baselines are not significant enough. I’m also not convinced by the claims about overcoming the capacity gap (see weakness). Comparison with existing methods in experiments would also position this work better in the literature.", "strength_and_weaknesses": "Strength:\n* The problem of distilling knowledge from large foundation models is well-motivated.\n* The experimental findings of resistance to capacity gap are interesting. \n\nWeakness:\n* Resistance to the capacity gap is not very convincing. The gains of student models are very small over the baselines in Table 1 and 2. In my opinion, the proof of overcoming the capacity gap is to narrow or at least maintain the performance gap with the largest teacher. Table 1 shows that the gap widens with the strongest teacher model.\n* The hypothesis that the resistance to capacity gap comes from less over-confident errors can use more study, e.g. what happens if we make the predictions less confident through adjusting the temperature parameter.\n* Results in Table 5 show a bigger gap between CLIP model and other baselines. Any explanation for that?\n* Tables 1-5 are all ablations and analysis. It’d be good to see some comparison with existing methods e.g. ESKD + AT (Cho et al. 2019).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5TOQAuGw4-", "reviewer": "Reviewer_zsBj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on improving knowledge distillation from CLIP to a much smaller network. The authors propose to enhance the CLIP teacher model by (1) text prompts improvement, and (2) output refinement. They also propose a metric $\\rho$ to estimate how likely the teacher generates over-confident wrong supervisions.", "summary_of_the_review": "Overall, the reviewer thinks the experiments are not thorough enough. The comparison between CLIP teacher and other ResNet type of teacher is not fair. $\\rho$ does not seem to be a reliable metric.", "strength_and_weaknesses": "## Strength\n1. this paper is well-written and is easy to follow.\n1. The proposed method indeed improves knowledge distillation results, not by a large margin though.\n1. the experiments clearly demonstrate that CLIP indeed suffers less from the capacity gap issue in knowledge distillation.\n\n## Weakness\n1. $\\rho$ doesn't seem to be a reliable metric to determine whether a model performs better in knowledge distillation. Specifically, in Table 4, larger models trained with CE have smaller $\\rho$. Since larger teacher models also achieve higher accuracy, then why larger teacher models achieve worse performance? Also, why does the authors choose ResNet type of CLIP model in Table 4 rather than ViT type of models as in other experiments? \n1. It is hard to draw the conclusion that knowledge distillation from CLIP is superior to knowledge distillation from other pre-trained teachers as CLIP's accuracy is much higher than other teachers. It would be good to find a setting where the teachers perform roughly the same so that each model can be fairly compared. For example, the authors can use weaker/smaller CLIP models or evaluate on datasets where each model performs roughly the same.\n1. Given that CLIP teachers achieve much higher acc, the performance improvement of the student model using CLIP is only marginal.\n1. The experiments are not thorough enough. For example, the authors only demonstrate results on ImageNet-100. The authors only demonstrate results on ResNet type of student model. It would be good to also show ViT type of student model. It would be good to also show the results using other ViT type of teachers, not just CLIP.\n1. the ablation of the improved text prompt is missing", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject