Transferring Pre-trained Multimodal Representations with Cross-modal Similarity Matching
Authors: Byoungjip Kim
Sungik Choi
Dasol Hwang
Moontae Lee
Honglak Lee
Venue: NeurIPS 2022 Accept (2022)
Keywords: Multimodal Representation Learning
Transfer Learning
CLIP

Abstract: Despite surprising performance on zero-shot transfer, pre-training a large-scale multimodal model is often prohibitive as it requires a huge amount of data and computing resources. In this paper, we propose a method (BeamCLIP) that can effectively transfer the representations of a large pre-trained multimodal model (CLIP-ViT) into a small target model (e.g., ResNet-18). For unsupervised transfer, we introduce cross-modal similarity matching (CSM) that enables a student model to learn the representations of a teacher model by matching the relative similarity distribution across text prompt embeddings. To better encode the text prompts, we design context-based prompt augmentation (CPA) that can alleviate the lexical ambiguity of input text prompts. Our experiments show that unsupervised representation transfer of a pre-trained vision-language model enables a small ResNet-18 to achieve a better ImageNet-1K top-1 linear probe accuracy (66.2%) than vision-only self-supervised learning (SSL) methods (e.g., SimCLR: 51.8%, SwAV: 63.7%), while closing the gap with supervised learning (69.8%).

Transferring Pre-trained Multimodal Representations with Cross-modal Similarity Matching
Byoungjip Kim Sungik Choi', Dasol Hwang', Moontae Lee 1,2 Honglak Lee
LG Al Research University of Illinois Chicago"
(bjkim, sungik, choi, dasol hwang, moontae.les konglak)@lgreseen ai
Abstract
Despite surprising performance on zero-shot transfer, pre-training a large-scale multimodal model is often prohibitive as it requires huge amount of data and computing resources. effectively transfer the representations of a large pre-trained multimodal model (CLIP-Vit ViT) into a small target model (c.g.. ResNet-18) For unsupervised transfer, introduce cross-modal similarity matching (CSM) that enables student model to learn the representations of a teacher model by matching the relative similarity distribution across text prompt embeddings. To better encode the text prompts, we design context-based prompt augmentation (CPA) that can alleviate the lexical ambiguity of input prompts. Our experiments show that unsupervised represen- tation transfer of pre-trained vision-language model enables a small ResNet- to achieve better ImageNet-I K top-1 linear probe accuracy (66.2%) than vision-only self-supervised learning (SSL) methods (c.g., SimCLR: 51.8%, SwAV: 63.7%), while closing the gap with supervised learning (69.8%).
this paper, we propose method (BeamCLIP) that can
1 Introduction
ImageNet-Ie -K top-1 linear probe on ResNet-18
Learning transferable representations crucial for successful downstream tasks. Contrastive learning such as SimCLR 4 and MoCo-v2 [
have shown notable success by forcing features of individual classes be clustered and suffi- ciently scattered [4]] But their linear probe performances are still far behind the supervised learning as shown in Figure [ Recently, large- scale vision and language pre-trained (VLP) models provide highly transferable visual repre- sentations via language supervision. However, learning VLP models from scratch is prohibitive as it requires large amounts training data and computing resources. For example, training CLIP 32 requires 400M paired image-text data and several hundreds of GPUs. ALIGN [2 fur- ther scales up to leverage alternative texts speci- fied for descriptions of web images. While these models are often based on large Transformers 40, small Nets such ResNet-50 [17 and MobileNet 20 are still widely used in practice and even more crucial for low-resource
Top-1 accuracy (%)
50
Supervised
BeamCLiP (ours)
80
SwAV
BYOL-GA
McCo-V2
SimCLR
BYOL
100
150
250
300
350
400
epochs
Figure I: ImageNet-IK top-1 linear probe ac- curacy ResNet-18 representations. By trans- ferring CLIP-VIT 22 vision-language represen- tations ResNet-18. the BeamCLIP can learn better visual representations than vision-only self- supervised learning (SSL) methods in terms the linear probe accuracy.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

environments. We reformulate representation learning in terms of knowledge transfer from large pre-trained model small practical model.
Large-scale vision-language pre-trained models exhibit strong alignments between different modali- ties. CLIP 32 learns visual concepts from natural language supervision, mapping image and text into the same vector space. As their training data is not only huge but inaccessible, however, conventional knowledge distillation
IV based on the source training data is no longer a viable option. Instead, we propose cross-modal similarity matching (CSM). Imagine your goal is to learn a high quality representation for an input dog image in CIFAR-I0 23 as in Figure 2 Since CLIP was trained on numerous image-caption pairs, angular distances from the dog image embedding to the caption embeddings of other anchor prompt texts such as "A photo of cat" or "A photo of horse" must comprehensively preserve their visual differences. By training the student to preserve angular relations witnessed from the teacher, our model achieves near benchmark performance without accessing the original data for training CLIP.
To better encode the text prompts, we design context-based prompt augmentation (CPA) that can alleviate the lexical ambiguity of the input text prompts. We find that lexical ambiguity in prompt texts can lead semantically incorrect text embeddings. This may result in unexpected discrepancies of image-text alignment in the teacher's embedding space. Also, it is known that the zero-shot perfor- mance of CLIP can be improved by designing task-specific prompt texts. Inspired by this, we design CPA that extends the basic prompt of CLIP to better encode prototypical anchor representations.
Our experimental results show that the BeamCLIP ("beam" means to transmit) achieves the strongest and near benchmark performance on ImageNet- K [TQ top-1 linear probe accuracy when using most popular ResNet-18 and ResNet-50 as the student network. We also compare the effectiveness of the BeamCLIP against zero-shot transfer learning. Further, we provide ablation study results to show how much each component contributes to the performance.
Contributions of this paper can be summarized as follows:
We propose method (BeamCLIP' that can effectively transfer the representations of large pre-trained multimodal model (e.g., CLIP-ViT into a small target model (e.g.. ResNet- ResNet-50). To achieve this, introduce cross-modal similarity matching (CSM) and context-based prompi augmentation (CPA). (Figure 2
We empirically show that BeamCLIP enables small target model (e.g., ResNet- 18) to achieve a better ImageNet- IK linear probe accuracy than vision-only self-supervised learning (SSL) methods, by effectively transferring CLIP-ViT ViT representations. (Figure and Table R]
! Table 2
We also explore the zero-shot capability of the BeanCI.I (Table 5 and analyze the effectiveness of the BeamCLIP on various target datasets (Table 4 and Table 7
2
Related Work
Vision and language pre-trainig. Vision and language pre-training (VIPP) aims jointly learn vision and language representations that can be transferred to the downstream tasks such as visual question answering (VQA), image captioning, and vision and language navigation (VLN). There BERT-based vision and language models such as VLBERT 35), ViLBERT 26 and UNITER 7.
Also, there are contrastive learning-based models such as CLIP
3 and ALIGN 122 These models
use contrastive loss 30 to learn aligned vision and language representations by performing a task of matching a large-scale image and text pairs. The BeamCLIP aims to transfer the rich representations of large-scale vision and language pre-trained models such as CLIP and ALIGN to small target model.
Self-supervised learning. Self-supervised learning (SSL) aims to learn highly transferable repre- sentations by using unlabeled data. In computer vision, the early stage, task-specific self- eeefsuuee methods were introduced. These include Context Prediction [[I. Rotation Prediction [14]. and Col- orization 4% More recently, contrastive learning-based methods were introduced as a task-agnostic approach. These include SimCLR 4 and MoCo-v2 [6] However, since contrastive self-supervised methods require a large batch size, non-contrastive methods have been introduced. These include SwAV B. BYOL [6], and SimSiam [5]. In this paper, empirically show that the BeamCLIP can

---

provide better visual representations than the state-of-the-at SSL methods by leveraging large-scale pre-trained multimodal model.
Knowledge distillation. Knowledge distillation (KD) 19 aims to transfer rich knowledge from a strong teacher model to target student model. In a conventional setting, it encourages the student model to mimic the task-specific prediction of the teacher model. As the student model is trained to predict the same probability distribution over pre-defined classes as the teacher model's, using Kullback-Leible (KL) divergence is natural metric to measure the error between the two models. For classification task, the loss function can be formulated as follows:
LKD Σ H(pp + Σ KL(P llp.).
The first term indicates the supervised loss, where p; denotes the one-hot labels and Hp, q) denotes cross-entropy. The second term is the distillation loss, where pf" and p are the softmax predictions of the teacher and student models, respectively.
Similarity based knowledge distillation. Recently, similarity-basee knowledge distillation such as SEED 133 OSS [ and ISD [7] was introduced in the context of self-supervised learning (SSL). SEED showed that the linear probe accuracy of a small student (ResNet-18) 18) can be improved by transferring the representations of a larger teacher (ResNet-50)) pre-trained by SSL methods such MoCo-v2 ( Unlike this, OSS 3 aims to transfer representations of an evolving teacher (ResNet-50) into a smaller student (ResNet- 18) on the fly. Unlike SEED and OSS, ISD 37 considered the same size student and teacher network (ResNet-18) and showed a student can learn visual representations by iterativly distilling the similarity of teacher's representations. These works closely related to work. Unlike these works, the BeamCLIP aims transfer rich vision and language representations of large-scale pre-trained models such as CLIP- VIT/16 32 into smaller network such as ResNet-18.
Prompt engineering. Recently, researchers showed that prompt engineering effective at improving the performance of large-scale language models (LLMs) downstream tasks without fine-tuning. Prompts are input texts of language models that usually consist of a task description several examples. To further simplify prompt engineering, prompt tuning 24 proposed to add k learnable tokens to the input texts, while having language models frozen. Similar to GPT-3, is known that the zero-shot performance of CLIP B21 can be improved by designing the prompt texts to each task. For example, on satellite image classification datasets, "A satellite photo of [label}" provides better performance than the default "A photo of {label}"". Inspired by this, we propose context- based prompt augmentation that extends the basic prompt of CLIP to better encode prototypical text anchor representations by alleviating the lexical ambiguity of class label texts.
2 is surprisingly
3 Method
Problem formulation. Formally, our problem transfer aligned cross-modal representations of strong teacher model (-) into target student model fā(() with unlabeled data D. {a;;)_")- Given each unlabeled image I,, we formulate representation transfer as a regression task that matches teacher representations 5 (Ira) a student's. fā (wx). As the student network is parameterized by the learning objective is
Σ - (x
arg IIL O
Normalizing the representations via l2 -n-ormalizaaion (ie., qi f (zx))/11( and k fg (a///)//((<<)))>> leads to the following simplification:
arg min
Σ
1g: --
arg min > (2 24. kz)-
The problem now involves maximizing the cosine similarity between Iz-normalized representations from teacher and student models.
3

---

piot Heof
Seih-uprrvited Pre-training
Unloboled images
Studant
Softmax
piols bood"
encodar Af
Piel/A)
Iratance Similarity Matching
Cross-modil Similirity Matching
Casine Simikrity
R
Teacher (CLUP)
Skee-snd
Step-grad
Gree-Friiopyy
Entroyy Minimiztion
Product
Scfhax
Image encoder +
P(N(A)
Unlabeled
My sapee "eo!"
L24som
Tust unsosder
A (o)101
Context-based Prompt Augmentation
Prototypical Resresentattins
plece
pour
Pru-tréree
Figure 2: Overview of the BeamCLIP. Representation transfer can be viewed as a task in which, given query input, student model learns to regress a vector representation of teacher model. The BeamCLIP first measures the normalized cross-modal similarity of the query image compared to anchor text representations in the teacher's embedding space. Then, it encourages the student to mimic the same cross-modal similarity in the student's embedding space. To better align image representations, our method uses sele-supervised pre-training of the student model. Finally, to avoid text ambiguity, we uses context-basee prompt augmentation.
Method overview. The overview of Beam CLIP is shown in Figure 2 The teacher model of the BeamCLIP consists of an image encoder f and a text encoder I (.). These encoders are pre-trained under a simple task of matching images tations fo (zz) and text representations o (tx) are thus well-aligned within cross-modal embedding space. We provide the details of the BeamCLIP in the following sections. More specifically, we describe how extend the basic problem setting by leveraging the unique features of CLIP where vision and language representations are precisely aligned. Throughout the paper, we use the notation CLIP-ViT/16 denote the CLIP B2 model that uses Vision Transformer (ViT) [[2] with the patch size of 16x 16 as the image encoder. Similar to this, CLIP-RN50 denotes the CLIP model with ResNet-50 1 the image encoder.
with large-scale corpora. Image represen-
3.1 Similarity-based cross-modal representation transfer
To effectively distill cross-modal representations, we use similarity- -based matching as described above. Our similarity-basee representation transfer utilizes two carefully designed loss functions: (1) instance similarity matching (ISM) loss and (2) cross-modal similarity matching (CSM) loss.
Instance similarity matching. This objective is directly derived from Eq. Given a query image F; it encourages the student image encoder f (-) to regress the representation of the teacher image encoder (.). We apply conventional image augmentations (see Appendix B.1] on a query image n; and the same augmented image àt is fed to both the teacher and student image encoders. Given {xx J.N it is formulated as follows:
unlabeled query images D.,
N LISM Σg
f( {T() (G.)l (ST - Σ (ge ky).
(G.)
N
However, the similarity signal from a single instance is not enough to constraint the student represen- tations. For example, the topological ambiguity may occur in image encoding, since two symmetric representations have the same cosine similarity compared to single teacher representation (see Appendix B.2). We conjecture that this can be mitigated by incorporating multiple anchor points to the query points. Based on this idea, we introduce cross-modal similarity matching loss.
Cross-modal similarity matching. better align a student representation q with the teacher representation k; we introduce cross-modal similarity matching (CSM) loss. We use multiple anchor points to cope with the ambiguity problem mentioned above. Further, we use text representations as anchor points, since we can easily generate prototypical anchor points by using text prompts and class

---

Table Examples of context-based prompt augmentation for ambiguous class labels on Flowers 102.
Label Index
Label Name
Text Prompt
7
bird of paradise
A photo of (bird of paradise). (Strelitzia is genus of five species of perennial plants, native to South Africa. It belonge to the plant family Strelitziaceae).
10
snapdragon
photo of (snapdragon). (Antirrhinum is genus of plants commonly known as dragon flowers or snapdragons because of the flowers' fancied resemblance to the face of dragon that opens and closes its mouth when laterally squeezed)."
labels. Since image and text representations are precisely aligned in CLIP, we can effectively apply this approach. More specifically, the BeamCLIP first measures the normalized image-text similarity of the query image compared to prototypical text points in the teacher's 's embedding space. Then, it encourages the student to mimic the same image-text similarity in the student's embedding space.
More formally, we generate multiple anchor representations A = { by encoding class texts C' = { with the teacher text encoder o% (in other words, aj g (e))) To measure the similarity regarding to multiple anchor representations A, we define the normalized cross-modal similarity as follows:
exp ((ki a;)/T)
83 (Ki, A)
Em exp ((k: (mm/T)
where 7 is temperature hyperparameter that is set to 0.01 in our experiments.
Then, evaluate the cross-modal similarity distribution by using set of normalized cross-modal similarities:
(k (A) = [s1 (k;, A)..., SM (ki, A)]-
Then, the student model is optimized to mimic the normalized pross-modal similarity the teacher's embedding space by minimizing the cross entropy, i.e., H(P(k,]A) P(x:AA).
We further minimize the entropy of normalized cross-modal similarities the student embedding space i.e., H (P(: A)). This minimization helps the student provide query representations qi that are more attracted to anchor representations A } This entropy minimization also known to be effective in other domains such as remi-supervised learning [5] 29
j=1
Altogether, the CSM loss is formulated as follows:
LOSM Σ ((P((NA)) + Σ H(P(q)A)).
Final Loss. The final loss of the BeamCLIP is formulated follows:
LBeacCLIP LUSM + AiSm CIsM
(8)
where AeM is the scale hyperparameter that is set to 10 in our experiments.
3.2 Context-based prompt augmentation
found that lexical ambiguity in prompt texts can lead semantically incorrect text embeddings. This may result in unexpected discrepancy of image-text alignment in the teacher's embedding space. For example, Flowers 102 [28] dataset has some classes with unusual and ambiguous flower names, such as "snapdragon", "bird of paradise". and "colt's foot" Therefore, incorrect prototypical anchor points might be compared with a query image. To address this issue of semantic ambiguitics in the text, introduce context-based prompt augmentation (CPA), a data-driven approach that augments basic prompts with contextual text such as Wikipedia descriptions or hierarchical labels.
"Examples can be found at: 102 Category Flower Dataset

---

For prompt tuning with Wikipedia descriptions, we use the template "A photo of (label). [Wikipedia description)", We use this template for Flowers 102 and Pets37 in our experiments. We provide some examples from the Flowers 102 dataset in Table I For prompt tuning with hierar- chical labels, we use the template "A photo of a {fine label categorized as {coarse label}". We use this template for CIFAR100 and ImageNet in our experiments. Analogous examples from CIFAR100 can be found in Table in Appendix B.3
3.3 Other details
Self-supervised pre-training of student. To help the student mimic the teacher's cross-modal embedding space better, we pre-train the student image encoder with self-supervised method. Since self-supervised pre-training such as SimCLR !.. MoCo-v2 [6. and SwAV [ provides weakly clustered embedding space based on similarities, it can used as better initial state for the student to mimic the teacher's embedding space. The details can be found in B.4 We show the effect of SSL
pre-training of the student in the experiment section (see Table
Optimization. For optimization we use SGD with cosine annealing schedule (SGDR) 2 To
stabilize training, we use momentum encoder that updates its weights via exponential moving average (EMA) 18 [6]. The momentum encoder of student 0g is updated using the following rule:
gg + mdg + (1 m)0s
where f the image encoder of a student model and m a momentum hyperparameter that is set 0.99 in our experiments. The model hyperparameters are summarized in Table 12 in Appendix B.5
4 Experiments
Downstream datasets. We evaluate the BeamCLIP on six standard benchmark datasets: CIFARIO 23 CIFAR100 233. STLIO 9. Flowers102 [28 Pets37 [31) and ImageNet- 1K [1O]. Following
convention, we split the datasets into train, validation, and test sets. Then, we use train set for transfer, and test set for evaluation. For ImageNet, we use the validation set as a test set, since its test set does not provide labels. More details on the datasets are summarized Table in Appendix
4.1 Representation transfer with unlabeled target data
Setting. We compare the BeamCLIP with various self-supervised methods in terms of linear probe accuracy on ImageNet- I. Following the conventional protocol, we use ResNet-18 and ResNet-50 17 as the base encoder and evaluate the learned representations by using logistic regression. We use LBFGS algorithm 4< for logistic regression. Its hyperparameter C' is determined through coarse-graile -rssee-r hyperparameter search on the validation split. And, the accuracy is evaluated in the test split. We found that it provides the best linear probe accuracy when C is set to 30. We perform our experiments on 8 NVIDA A100 GPUs and it takes about 30 hours for 200 epoch training.
Table 2: ImageNet-11 1K top-1 linear probe accuracy ResNet-50. We compare the BeamCLIP with vision-only self-supervised methods in terms of linear probe accuracy on ImageNet- IK. The BeamCLIP representations provide higher linear probe accuracy than self-supervised methods. This means better transfcrabilityy The values are quoted from the original paper, and n/a means "not available" from the paper.
Epochs
Method
Teacher
Student
Batch
200
400
800
Supervised
X
RN50
256
76.2
SimCLR 4
X
RN50
512
65.6
66.7
67.4
MoCo-v2 6
X
RN50
256
67.5
70.1
71.1
BYOL-GA [6
X
RN50
4096
70.6
n/a
n/a
SWAV
RN50
256
72.0
74.3
n/a
Beam IeImCIIP (ours)
CLIP VIT-B/16
RN50
512
74.8
75.1
75.0

---

Table 3: ImageNet-1K top-1 linear probe accuracy ResNet-18.
Epochs
Method
Teacher
Student
Batch
100
200
400
Supervised
X
RN18
256
69.8
SimCLR
X
RNI8
256
47.1
49.9
51.8
MoCo- V2
X
RN18
256
48.6
49.9
51.9
BYOL 16
X
RN18
256
44.2
47.5
46.8
BYOL-GA
RN18
256
54.2
56.9
61.4
SwAV
RN18
57.7
61.2
63.7
OSS
SSL RNS0
RNI8
256
60.0
64.1
65.8
Beam CLIP (ours) CLIP ViT-B/16
RN18
256
63.8
64.8
66.2
Table Effect of self-supervised pre-training of student.
Method
Teacher
Student
Pre-training of Student
Batch
Epoch
Linear Probe
Supervised
X
RN50
X
256
76.2
BeamCLIP (ours)
CLIP VIT-B/16
RN50
SimCLR 4
512
200
74.8
BeamCLIP
(ours)
CLIP VIT-B/16
RN50
SWAV 3
512
200
75.8
Transfer to ResNet-50. Table shows the comparison of the BeamCLIP with vision-only self- supervised methods such as SimCLR [ MoCo-v2 [0, SwAV B. BYOL [T6], and SimSiam [5]. The BeamCLIP provides better visual representation by achieving 74.8% top-1 linear probe accuracy on ImageNet-IK [T]. While self-supervised methods take long training epochs to achieve comparable accuracy, BeamCLIP-RN50 achieves better accuracy with less training epochs. Also, note that BeamCLIP-RRN RN55''s representations provide better accuracy than CLIP- RR500s representations.
Transfer to ResNet-18. To check if the BeamCLIP transfer CLIP representations into smaller models than ResNet-50 (24M), we also measure ImageNet- top-1 linear probe accuracy on ResNet-18 (11M). ResNet- is trained from seratch (not self-supervised pre-trained with SimCLR), while transferring CLIP ViT-B/16 representations. As shown in Table 3 BeamCL.IP learns better representations than SSL methods such as SimCLR 4 MoCo-v2 6 BYOL 16 and SwAV 3
More importantly, the BeamCLIP provides better performance than OSS S that simultancously
learns and transfers representations from ResNet-50. The learning curve is presented in Figure in Appendix C.I
Effect of self-supervised pre-training. Table 4 shows ImageNett-1e 1K top-1 linear probe accuracy BeamCLIP RN50 representations by using different SSL pre-training. With the better SSL method (SwAV 3 SimCLR [4]), the BeamCLIP learn better representations with increased linear probe accuracy.
Representation transfer with unlabeled non-target data
To check if the BeamCLIP also inherits the powerful zero-shot capability of CLIP, we compare zero- shot accuracy CLIP variants on ImageNet-1e K. For zero-shot measure, we use CC-3M + and ImageNet-21K (12M samples) [X] that do not have overlap with ImageNet-1K. K. Table 5 shows the comparison of zero-shot accuracy. The BeamCLIP-
achieves about 57.5% zero-shot accuracy that is highly comparable with CLIP RN-50 (59.6%). The learning curve is presented in Figure 9 of Appendix C.T
4.3 Transfer learning accuracy on various target datasets
Setting. We evaluate how effectively the BeamCLIP transfers CLIP-VIT representations into student model by evaluating classification accuracy on various datasets (see Table (). We choose ResNet-50 [17 as the student model to compare the distilled target model with CLIP-RN50. Also, ResNet-50 is conventionally used in evaluating linear probe accuracy on ImageNet- KK. However, the BeamCLIP can adopt any architecture, not just ResNet-50.
7

---

Table 5: Comparison of zero-shot accuracy ImageNet-IK. On ImageNet-IK, the BeamCLIP RN50 achieves about 57.5% zero-shot accuracy that is higly comparable with CLIP RN-50 (59.6%). To achieve such high zero-shot accuracy, CLIP uses very large image-text pair data (WTT-400M). Instead, the BeamCLIP can achieve the comparable zero-shot accuracy by effectively transferring the teacher's representations, while using only 3% data (ImageNet-211 (12M)). Note that OpenCLIP provides only about 36.5% zero-shot accuracy with the similar amount of data (CC-12M).
Method
Image
Training
Teacher Model
Text
Batch Size
Epochs
ImageNet Zero Zero-shot
Enzoder
Data
Prompts
CLIP
CM
VIT-B/16
WIT-400M
X
32,768
32
68.6
CLIP
RNS0
WIT 400M
X
X
32,768
32
59.6
OpenCLIP
MI
RN50
YFCC-ISM
X
X
256 8
32
32.7
OpenCLIP
M1
RN50
CC-12M
X
256 x 8
32
36.5
BeamCLIP (ouns)
RNS0
CO-3M
CLIP VI-B/16
IN-IK
64 x
100
49.5
BeamCLin (ouns)
IN (12M)
VI-//16 CLIP VIT-R/16
64
50
53.6
BeamC1.IP (ouns)
RN50
IN-21K (12M)
IN-IK
64
200
57.5
As baselines, we choose two representative distillation methods among many methods: (1) conven- tional knowledge distillation (KD) 19 and (2) contrastive representation distillation (CRD) [38] Since conventional KD aims mimic the task-specific predictions the teacher model unlike the BeamCLIP we apply the KL divergence on minimizing the cross-modal similarity distribution (i.e., P(4lA) A) and P(k,) A)), instead of the Cross-entrops (CE). CRD proposes variant of InfoNCE loss for representation distillation, which we apply on normalized representations (i.e., qi and k;). The details of each method can be found in the related work section
Results. Table 6 shows comparison of teacher and student accuracy on various datasets. We empiri- cally demonstrate that the BeamCLIP can effectively transfer vision and language representations of a large teacher model (CLIP ViT-B/16) into a small student model (ResNer-50). We find that the KL divergence used in conventional knowledge distillation (KD) is not effective in transferring CLIP-ViT representations. Also, the contrastive learning-based approach is not effective. Unlike this, the BeamCLIP can effectively transfer CLIP ViT-B/16 representations into ResNet-50, achieving very high accuracy that is comparable or better than the teacher accuracy. KD simply minimizes the error between single instances. We conjecture that the moos--mdal similarity multiple anchor points introduced in the BeanCLIP helps the student preserve the topology of the teacher's 's embedding space.
Also, note that context-based prompt augmentation helps achieve better accuracy after representation transfer. Since Flowers 102 has many ambiguous labels, our experiment shows that text prompt augmentation significantly increases the student's accuracy compared to the teacher's accuracy.
Ablation study. Table 7 shows the ablation study results of the BeamCLIP Our empirical findings are as follows: (1) Instance similarity matching (ISM) is not enough by itself to preserve the topology
Table 6: Comparison of teacher and student accuracy on various datasets. Conventional knowl- edge distillation with the KL divergence is not effective in transferring CLIP-ViT representations. In contrast, the BeamCLIP effectively transfers CLIP ViT-B/16 representations into ResNet-50 by using unlabeled query data. We denoted with in cases student model surpasses the accuracy of the teacher model.
ImageNNr- K
Model Type
Param. Size
CIFARI00
Flowers102 102
Img. Enc.
CIFARIO
STLIO
Pets37
Method
CLIP 32
(zero-shot)
T
VIT-B/16
76M
91.6
68.7
98.2
70.4
88.9
68.6
CRD
IN
S
RN50
24M
76.78
38.83
81.41
26.15
62.22
30.90
KD
19
S
RN50
24M
90.89
58.02
93.28
49.01
77.51
56.38
BeamCLIP (ours)
S
RN50
24M
92.10'
67.35
97.45
75.86*
86.94
66.17
8

---

Table 7: Ablation study results. The acronym denotes the sub-methods introduces in the BeamCLIP (1-1) ISM means instance similarity matching loss, (1-2) CSM means cross- mross-modal similarity matching loss, (2) CPA means context- based prompt augmentation, and (3) SSL PT means self-supervised pre-training of student, All the technical components contribute to the improvements of transfer learning accuracy at the student. Note that we only perform CPA datasets with more than 100 of class labels.
ImageNet-11 KK
Img. Enc.
IS-
(12) 39CCN
(3)SL PT
CIFARIO
CIFAR1O0
Flowers102
(P)
Type
Method
CLIP
(zcro-shor)
T
VIT-B/16
91.6
68.7
70.4
68.6
S
RN50
Cosine
X
39.07
6.04
1.43
21.83
S
RN50
Cosine
x
X
SimCLR
87.00
48.26
3.59
51.83
Unsupervised
S
RN50
CE
X
91.28
58.90
12.94
63.30
Representation
S
RN50
CE
X
SimCLR
91.53
65.64
62.18
65.45
Transfer
S
RN50
X
CE+EntMin
X
SimCLR
91.71
66.14
63.96
66.23
(BeamCLIP)
S
RN50
Cosine
CE+EntMin
x
SimCLR SimCLR
92.10"
66.18
64. 14
65.76
RN50 S
Cosine
CE+EntMin
/
67.35
75.86"
66.17
CLIP-RNST
BesmCLIP-RNSB
photo of (anepdroganj
12
IPI
MES OE
4Ph Med T3
APL
MNT 13
photo of froos
photo of lpink roscy
bebt T3
IPy C239 Miat
LIC
BIVE
Inwe 47
JPC MRC 2223 Mett Meet
Wher
BP:022147
photo tycllow rosel'
Comparison of CLIP-RN50 and BeamCLIP-RN50, This figure shows the top-5 text- image retrieval results. A red rectangle denotes an incorrect result. CLIP-RN50 provides many incorrect results, since its zero-shot accuracy is relatively low. In contrast, BeamCLIP-RNy provides much improved results, since it transferred from V with higher zero-shot accuracy.
Figure 3:
of the teacher's embedding space. (2) Cross-modal similarity matching (SCM) compared to multiple anchor points helps the student mimic the teacher's embedding space. (3) Self-supervised pre- training of the student (SSL PT) helps the student mimic the teacher's embedding space. (4) Entropy minimization (EntMin) helps to improve the accuracy. (5) Context-based prompt augmentation (CPA) helps measure the similarity more precisely. As shown the table, Flowers 102 dataset is sensitive to self-supervised pre-training of student. We conjecture that since Flowers 102 dataset has only 1020 training samples for the 102 classes, it not enough to probe the teacher's 's representation space.
Qualitative result. To see the quality of the transferred representations, we analysed text-image retrieval results on the Flowers 102 dataset. Figure compares the top-5 text-image retrieval results A red rectangle denotes an incorrect result. Compared
between CLIP-RN50 and DeamCLIP-RN55, provides much improved results, since its representations are transferred from CLIP- V with higher zero-shot accuracy. More interestingly. BeanCLIP- RP500 provides surprisingly good text-image retrieval results, even though unseen text prompts such as "a photo of {pink rose]" or "a photo of {yellow rose}" are given.
to CLIP-RN50, BeamCLIP-

---

Transfer learning on CIFAR100
Transfer learing on CIFAR100
(with subset prompts from CIFAR10D classes)
(with random prompts from ImageNet classes)
40-
Accuracy
40
20
CUP Witdr16 teero shot) BeamCLiP-aN5O
20
CLP VIT-0/16 (nero shat)
BeamCLI-SNER (CE+EntMin)
(RL.
CLI--RNSO
(zero-hho
20
100
200
600
800
1000
of toxt prompts
text prompts
(a) Subset text prompts from CIFARI00 classes
(b) Random text prompts from ImageNet classes
Figure 4: The effect of random text prompts on CIFAR100. (a) The text prompts are randomly sampled from the set of 100 class names of CIFARI00 The red dotted line denotes the teacher's accuracy as an upper bound. It is more efficient as it closer to this line. As shown in the blue line, the BeamCLIP (CE+EntMin) can effectively transfer the CLIP representations, even when the class names of the target dataset are partially given. (b) The text prompts are randomly sampled from the 1000 class names of ImageNet. The BeamCLIP (CE+EntMin) is still effective, even though the class names are randomly sampled from a non-target dataset (ImageNet-IK3,
4.4 Effect of random text prompts
We measured how effective the BeamCLIP is in cases where the class names of the target dataset are not perfectly given. Figure shows the effect of the randomly sampled text prompts CIFAR We can see that the BeamCLIP is still effective, even when (a) the subset of the 100 class names of CIFARI00 are given the text prompts, or (b) the text prompts randomly sampled from non-target dataset (ImageNet-1Ke K). The exact values in Figure f are presented in Table [5 and Table Appendix Also, the additional results on CIFARIO are provided in Appendix
16
5 Limitations and Conclusion
Limitations. With the help of rich representations of pre-trained CLIP, the BeamCLIP can learn better representations than SSL methods. However, since SSL methods can increase the performance at longer training epochs, the performance margin may be decreased in such setting. Another shortcoming that context-based prompt augmentations may require additional engineering efforts.
Conclusion. In this paper, we provide the BeamCLIP that can effectively transfer large pre-trained vision-language model (e.g.. CLIP-VVT) into small target model (e.g.. ResNet-18) with cross-modal similarity matching (CSM) and context- based prompt augmentation (CPA). We empirically show that the BeamCLIP can learn better visual representations than vision-only self-supervised learning (SSL) methods, by leveraging a pre-trained vision-languag" model (CLIP). The BeamCLIP is not intended be another CLIP, but an effective CLIP student.
Broader impact
This research aims to provide a simple and effective way to leverage CLIP for representation learning. With the help of CLIP, the BeamCLIP can learn better representations than self-supervised learning (SSL) methods. Since training CLIP requires very large data and hundreds of GPUs, it is important to provide a way to effectively reuse the pre-trained CLIP rather than training from scratch on target model. We believe that the BeamCLIP can help to save cost and time.
Acknowledgements
We thank amonymous reviewers for their valuable comments. This work was fully supported by LG AI Research.
10

---

References
[1] L. Beyer, X. Zhai, A. Royer, L. Markeeva, R. Anil, and A. Kolesnikov. Knowledge distillation:
A good teacher is patient and consistent. arXiv preprint arXiv:2106.05233) 2021.
[2] Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, Shyam,
G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS), 2020.
[3] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Neural Information Processing Systems (NeurIPS), 2020.
[4] T. Chen, S. Korbblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), 2020.
[5] X. Chen and K. He. Exploring simple siamese representation learning. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15750-15758, 2021.
[6] X. Chen, H. Fan, R. Girshick, and K. He. Improved baselines with momentum contrastive learning, arXiv preprint arXiv:2003.0429) 2020.
[7] Y.-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and Liu. Uniter: Universal image-text representation learning. In European Conference on Computer Vision (ECCV), 2020.
[8] H. M. Choi, H. Kang, and D. Oh. Unsupervised representation transfer for small networks: T believe can distill on-the-fly. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[9] A. Coates, A. Ng, and H. Lee. An analysis of single-layer networks in unsupervised feature learning. In International Conference on Artificial Intelligence and Statistics, 2011.
[10] J. Deng, W. Dong, R. Socher, L.-J. K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. IEEE/CV Conference Computer Vision and Pattern Recognition (CVPR), 2009.
[11] C. Doersch, A. Gupta, and A. A. Efros. Unsupervised visual representation learning by context prediction. In IEEE/CVF International Conference on Computer Vision (ICCV), 2015.
[12] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
[13] Z. Fang, Wang, Wang, L. Zhang, Y. Yang, and Z. Liu. Seed: Self-supervised distillation for visual representation. In International Conference on Learning Representations (ICLR), 2021.
[14] S. Gidaris, P. Singh, and N. Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations (ICLR), 2018.
[15] Y. Grandvalet and Y. Bengio. Semi-supervised learning by entropy minimization. In Advances in Neural Information Processing Systems (NeurIPS), 2005.
[16] J.-B. Grill, Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[17] K. He, X. Zhang, S. Ren, J. Sun. Deep residual learning for image recognition. In IEFF/CVF conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[18] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
11

---

[19] G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.02531, 2015.
[20] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 1exXiv:7004..m 707.0444466.222 2017.
[21] G. Ilharco, M. Wortsman, R. Wightman, Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt. Openclip, Jul 2021. URL
https://gitae thub. com flwffmlfool wwwp////oonnaaaoo/o//oo www.attiios/oppencllal.a
[22] C. Jia, Yang. Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. V. Le, Y. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning (ICML), 2021.
[23] A. Krizhevsky, G. Hinton, et Learning multiple layers of features from tiny images. 2009.
[24] B. Lester, R. AI-Rfou, and N. Constant. The power of scale for parameter-efficiet prompt tuning. In Empirical Methods in Natural Language Processing (EMNLP), 2021.
[25] I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts. In Interna- tional Conference on Learning Representations (ICLR), 2017.
[26] Lu, D. Batra, D. Parikh, and S. Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and andion--nd-a tasks. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[27] R. Müller, S. Kornblith, and G. Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[28] M.-E. Nilsback and A. Zisserman. Automated flower classification over large number of classes. In Indian Conference on Computer Vision, Graphics & Image Processing, 2008
[29] A. Oliver, A. Odena, C. A. Raffel, E. D. Cubuk, and I. Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
[30] A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprini arXiv:1807.0331 1807.03748, 2018.
[31] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar. Cats and dogs. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
[32] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, A. Askell,
P. Mishkin, J. Clark, al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2 103.00020, 2021.
[33] T. Ridnik, E. Ben- Ban-Barcch, A. Noy, and L. Zelnik-Manor. Imagenet-21k pretraining for the masses. 2021.
[34] P. Sharma, N. Ding, S. Goodman, and R. Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), 2018.
[35] W. Su, X. Zhu, Y. Cao, B. Li, L. Lu, F. Wei, and J. Dai. VI-bert: Pre-training of generic linual-ei representations. In International Conference on Learning Representations (ICLR), 2019.
[36] C. Szegedy, V. Vanhoucke, S. loffe, J. Shlens, and Z. Wojna. Rethinking the inception ar- chitecture for computer vision. In IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[37] A. Tejankar, S. A. Koohpayegani, V. Pillai, P. Favaro, and H. Pirsiavash. Isd: Self Self-supervised learning by iterative similarity distillation. In IEEF/CVF International Conference on Computer Vision (ICCV), 2021.
12

---

[38] Y. Tian, D. Krishnan, and P. Isola. Contrastive representation distillation. In International Conference on Learning Representations (ICLR). 2020.
[39] M. Tschannen, Djolonga, Rubenstein, Gelly, and M. Lucic. On mutual infor- mation maximization for representation learning. In International Conference Learning Representations (ICLR), 2020.
[40] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS, 2017.
[41] T. Wang and P. Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. International Conference on Machine Learning, pages 9929-9939. PMLR, 2020.
[42] L. Yuan, F. E. Tay, G. Li, T. Wang, and Feng. Revisiting knowledge distillation via la- bel smoothing regularization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[43] R. Zhang, Isola, and A. A. Efros. Colorful image colorization. In European Conference on Computer Vision (ECCV), 2016.
[44] C. Zhu, R. H. Byrd. P. Lu, and J. Nocedal. Algorithm 778: L-bfgs-b: Fortran subroutines for large-scale (TOMS), ((4))5550-60) 1997.
optimization. ACM Transactions on mathematical software
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section |
(b) Did you describe the limitations of your work? [Yes] See Section 4 and Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section 5
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [No] Our work does not include thcoretical results.
(b) Did you include complete proofs of all theoretical results? [No] Our work does not include theoretical results.
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] We provide the code, data, and instructions in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section g Appendix A and Appendix B
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Section
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section 4
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 2
Section g and Section 4
(b) Did you mention license of the assets? [No] We only used public benchmark datasets and open-sourced software. See Section
13

---

(c) Did you include any new assets either in the supplemental material or as a URL? [No] We did not use new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes] We only used public benchmark datasets. See Section M (e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [No] We did not use any data containing personally identifiable information or offensive content.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [No] We did not use crowdsourcing or conduct research with human subjects.
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [No] We did not use crowdsourcing or conduct research with human subjects.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [No] We did not use crowdsourcing or conduct research with human subjects.
14


Reviews:
[{"review_id": "zYSud-O3msI", "reviewer": "Reviewer_nz7y", "summary": "The authors propose a method for transferring the representation power of CILP ViT-B/16 to a small ResNet-50 by utilizing the cross-modality of the embedding space.\nSpecifically, in the training stage, a student model not just tries to mimic the output image features of a teacher model, but also tries to follow the similarity distribution of the teacher model that is computed with the given set of text descriptions.\nThese text descriptions are similar to the ones that were used in the zero-shot classification task in the original CLIP paper, but it is amplified with the additional information of each image that could be found on Wikipedia.\nUsing this additional cross-modal guidance, the proposed method was able to train student networks that presents the SOTA performance on multiple classification tasks.", "questions": "Questions\n- From Table 4, I could see that SSL PT plays an important factor for some cases, though it is not described in the method part. Could you explain why this factor has such huge effect in the results?\n\nSuggestions\n- From Table 4, I found that CPA is most helpful for dataset Flowers102, which implies that this method is especially effective for fine-grained datasets. It was interesting for me, and I think tackling this kind of dataset-wise differences in effectiveness of each element would be great for following research.\n- Guess Figure 3 is plotting the same figure for two models?", "limitations": "Authors did not address the limitations and potential negative societal impact of their work.\n\nSuggestions\n- I want to know the authors’ opinion on how much the transfer learning is helpful for handling issues concerned with sensitive web data?", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The authors propose the novel technique to guide a student model using the multi-modality of CLIP embedding space.\n- The proposed method proved to be effective in boosting classification performance on multiple datasets.\n\nWeaknesses\n- The analysis on the experimental results is somewhat scant. It could be amplified by providing the dataset-wise differences and more results from the larger number of prior transferring methods.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "puMPM5w3HEF", "reviewer": "Reviewer_eofu", "summary": "This paper aims at transferring knowledge from a pre-trained language-and-vision model to a small CNN-based vision network. The authors argue that a smaller network can learn better from multimodal representations in comparison with previous vision-only self-supervised methods. For transferring knowledge they propose to use two similarity matching objectives: one that matches visual representations from the teacher and student networks, and one (cross-modal) that matches the visual representations of the student network with the textual representations from CLIP. In order to increase robustness on the cross-modal similarity matching, the authors further propose to use multiple anchor textual representations, and they better craft textual prompts depending on the task and dataset. Finally, they first pre-train the student network based on previous self-supervised objectives, before distilling knowledge from CLIP. Experimentally, they show that the representations transfer well to image classification tasks, they are competitive against supervised methods and improve over self-supervised vision-only approaches.", "questions": "1. Have you tried using other self-supervised methods for pre-training except for SimCLR?\n2. Why specific datasets (e.g., CIFAR100, Flowers102; Table 4) are so sensitive depending on pre-training in comparison with others (e.g., ImageNet)?\n", "limitations": "As mentioned in the weaknesses of the paper, there are two limitations: 1. The fact that the authors did not use multiple self-supervised pre-training methods and/or one of the most competitive (e.g., BYOL) for drawing safe conclusions given the large impact that pre-training has, and 2. There is no further analysis on the pre-training factor, which is so important for good performance.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper presents an interesting approach with novel components (e.g., cross-modal similarity matching, augmented textual representations) for transferring knowledge from multimodal representations to smaller vision-only encoders. \n2. The authors experimentally demonstrate that their approach offers good visual representations that improve accuracy on image classification over prior self-supervised vision-only methods. Moreover, the performance is now closer to supervised methods, closing the gap further.\n\n\n**Weaknesses**\n\n1. From the experimental results, it is demonstrated that pre-training with prior self-supervised objectives before distilling knowledge from CLIP is crucial for performance, especially in certain datasets. However, the authors have chosen the weakest self-supervised model (SimCLR; Table 2) for pre-training. Results and conclusions would be more complete if they have chosen more than one methods for pre-training and/or one the strongest ones. Now, there is a question of whether the same patterns can be found independently of the pre-training methods and whether specific of these methods have advantage as a pre-training step for knowledge distillation from CLIP.\n2. As mentioned in the above point, self-supervised pre-training is crucial, and for specific datasets (e.g., CIFAR100, Flowers102; Table 4) not pre-training has an overwhelming performance degradation (e.g., 12.94% accuracy vs. 62.18%). Given these huge gaps and the necessity of pre-training for the method to work, further analysis is needed and some discussion about why specific datasets are more sensitive to this than others.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "meuaV5puLua", "reviewer": "Reviewer_zQJm", "summary": "The authors propose a method for transferring CLIP's visual\nrepresentations into smaller models; this setup might be useful for\nresearchers working on, e.g., low resource devices, who can't run the\nmore powerful, full CLIP models. Their method not only minimizes the\ncosine similarity between the image representations of a teacher\n(ViT-B/16) and student (RN50) model, but also, incorporates a set of\n\"anchor\" vectors, computed by the text side of CLIP: the cross-modal\nsimilarity objective encourages the cross-modal similarities between\nthe teacher and the student to match, too.", "questions": "- Did the authors try transferring to datasets where the label set and the images are not available at training time?\n\n- Did the authors try with different teachers/students?", "limitations": "The authors mainly focus on making higher performance, lower-resource versions of existing CLIP models. Given that CLIP's negative impacts have been discussed in the original work and follow ups, I don't think significant additional discussion is needed here.\n\nI did think that the \"Broader impacts\" section was a bit short: can more information about FLOPs/energy use/on-device computation be discussed?", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe work is easy to follow, and save for a few typos (see below), easy\nto follow. The empirical results are promising: the authors\nsuccessfully distill ViT-B/16-CLIP into a set of RN50 weights. The\ncross-modal matching objective is clever, and thorough ablations\ndemonstrate that this innovation is indeed complementary (and, in fact,\nin isolation, is actually *more* important than the cosine loss!). I\nappreciated that the authors evaluated on a number of datasets beyond\nImageNet, albeit only in a zero-shot capacity.\n\nWeaknesses:\n\n- The authors distill a ViT-B/16 into a RN50. While the former takes\n  more FLOPs than the later, ViT-B/32 is cheaper than RN50, and the\n  CLIP version of that architecture outperforms both the author's RN50\n  and the CLIP RN50 model.  While this doesn't totally undercut the\n  presented results, it's a bit strange to me why the authors chose\n  the particular distillation pair that they did --- it seems like\n  ViT-B/32 (or even EfficientNet or MobileNet) would have been a good\n  choice for student, and perhaps L/14-336px CLIP as the teacher would\n  have made a more compelling setup.\n\n- The ablations suggest that the cosine similarity loss isn't really\n  required: the best zero-shot imagenet performance is actually\n  achieved only with the cross-modal loss, and the other datasets seem\n  to be within a close margin. This isn't really a /negative/,\n  per-say, but a bit contrary to the story told.\n\n- It would have been nice to see linear probe results for the other\n  datasets like Pets37 as well.\n\n- Even assuming that RN50 is the best choice of student, the empirical\n  results are somewhat unimpressive. Specifically, I believe that the\n  authors choose anchor points to be task-specific prompts and the author's\n  model is domain adapted to (unlabelled) dataset-specific images, i.e.,\n  there's reason to believe that the author's method is specifically\n  tuned to the tasks described. RN50-CLIP, which is not tuned to the\n  specific tasks described, achieves only slightly worse imagenet linear\n  probe accuracy: 73.3 vs 74.8.\n\n- The zero shot results in table 3 are interesting, but a bit\n  incomplete. The ViT-B/32-CLIP model smaller than the author's\n  distilled RN50 model achieves only slightly worse performance on\n  these tasks, but it isn't presented. Again, this observation doesn't\n  invalidate the author's results, it's just that one really needs to\n  buy that RN50 specifically is an interesting student model.\n\n- Overall, the authors clearly \"win\" on this particular architecture\n  by a few accuracy points on linear probe imagenet, but it's not\n  clear that this is really the best architecture to distill to\n  (particularly because ViT-B/32 is smaller and there's already a CLIP\n  model for that). To the author's credit, they recognize this on\n  L288, but I would have preferred to see that the author's method\n  generalizes to other teacher/student models, instead of just this\n  one combination.\n\nOverall, the authors have a promising core result: it's possible to\ndistill the knowledge of a CLIP model into a smaller set of\nweights. And, the innovation that enables this distillation (that\nimage-text distances should be preserved in addition to image-image\ndistances) is novel. However, the empirical results are slightly\nunderwhelming: the authors model, by virtue of the selection of the\nanchor points and distillation over unlabelled images from the corpora\nof interest, is tuned to this particular set of datasets (vs. CLIP\nthat isn't). Their performance improvement over CLIP is somewhat small\nin magnitude (73.3 vs 74.8). I would have liked to have seen: 1)\ndistillation to significantly more efficient architectures vs. RN50\n(ViT-B/32 CLIP is already smaller) and with better teachers than\nViT-B/16; and 2) linear probe evaluations on unseen tasks, where the\nimages haven't been seen by the distillation model at training time.\n\n\n\n\nPresentation fixes:\n\n- It should be specified that Table 3 results are *all* zero shot.\n- L71: week --> weak\n- The broader impact statement is a bit hollow -- I would have appreciated a fuller discussion of embedded systems and/or low resource computation\n- Table 4's caption has an incomplete sentence.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VeyCii4SPH5", "reviewer": "Reviewer_izRQ", "summary": "The authors propose a method (BeamCLIP) that can effectively transfer the representations of a large pre-trained multimodal model (CLIP ViTB/16) into a small target model (ResNet-50). The authors introduce two main components for the representation transfer: cross-modal similarity matching (CSM) and context-based prompt augmentation (CPA). CSM enables a student model to learn the representations of a teacher model by matching the relative similarity distribution across text prompt embeddings. CPA can alleviate the lexical ambiguity of input text prompts. ", "questions": "Since the proposed method uses both the images and the labels, is it fair to compare the proposed method with the self-supervised learning methods (e.g., SimCLR)? \n", "limitations": "As mentioned above, my main concern is fairness. The concern is mainly from the experiment settings. \nIn experiments, the authors compare the proposed method with the self-supervised methods that do not use any training labels (e.g., SimCLR, MoCo-V2). The proposed method uses all images and labels of training sets for transferring the representations from a multimodal model. It is not surprising that the proposed method can obtain better accuracy. On the other hand, the proposed method uses the same data as the fully supervised models and distills the knowledge from a multimodal model pre-trained on a large-scale dataset. It is expected that the proposed method outperforms the fully supervised methods. \n\nTo further showcase the proposed method, it'd be nice if the authors could demonstrate some possible capabilities: \n- Using data not overlapped with the target datasets for representation transfer and showing that the proposed methods obtain superior performance on target datasets. \n- Similar to the previous point. If the data mentioned above are hard to get, an easier way is to perform representation transfer on one dataset and testing on multiple target datasets \nAnother interesting direction might be to use less data (e.g., 1% of training data) for representation transfer and achieve similar results in the current manuscript.  \n\n", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths:\nThe experiments show that the representation transfer of a large pre-trained multimodal model can provide good ImageNet linear probe accuracy while outperforming the existing self-supervised learning method (SimCLR) and closing the gap with supervised learning. \n\n- Weaknesses:\nThe experiment settings seem not fair and practical. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Rvks2i7qf6W", "reviewer": "Reviewer_f8ft", "summary": "This paper presents a method to transfer multimodal representations of CLIP to smaller student networks for downstream tasks. In addition to the standard similarity loss of representations, the method also applies cross-modal similarity matching loss where the topology with respect to anchor prompts is preserved among teacher and student. It also incorporates an interesting prompt augmentation technique to deal with ambiguous or minor visual concepts. The method is tested on six benchmark datasets and showed promising performance.   \n\nPros:  \n(+) The proposed method is simple yet reasonable, also seems effective.  \n(+) It includes an interesting prompt engineering method.  \n(+) The paper is generally easy to follow.\n\nCons:  \n(-) The paper seems overclaiming the improvement compared to self-supervised learning.   \n(-) Discussion on the limitations and negative societal impacts of the work is largely missing.\n", "questions": "Main points are described at \"Strengths And Weaknesses\".\n\nMinor comments:   \n- According to Figure 2, in the teacher side, the gradient of CSM loss is stopped, but not for ISM loss. Is this really true? If so, what is the reason there?\n- In Table 4 (ablation study), three results are missing without explanation. What do they mean? The training wasn't successful?\n- The light gray font used in some tables is very hard to see.", "limitations": "The authors' checklist says as follows.\n- (b) Did you describe the limitations of your work? [Yes] See Section 4 and Section 5.\n- (c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 5.\n\nHowever, I cannot find a clear discussion in the mentioned parts. I think this kind of requirements should be addressed more seriously. \n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, I find no obvious weaknesses in the paper's technical side. While each component is not surprisingly novel, the overall integration is reasonable and can be a good baseline in the research of distillating CLIP.  In particular, I find context-based prompt augmentation is an inspiring idea to successfully transfer the knowledge in CLIP to various downstream applications. \n\nMy main concern is that the paper seems to exaggerate the contribution by unfair comparison. The paper repeatedly says \"outperforming the existing self-supervised learning methods (SimCLR: 65.6%)\", but as shown in Table 2, with longer epochs, SwAV and BYOL-GA achieves 74.3% which is very close to the proposed method (why they are missing in Figure 1?), and even SimCLR achieves 69.1%. Why not train your method  just longer? Is it expected to improve after 200 epochs or not?   \nYes, I know it is not easy to train longer with limited computational resources, but at least this point should be honestly discussed. There's no clear reason why we should stop at 200 epochs for all methods. Without such discussion, picking up a particular value of 65.6% and just saying \"outperform the existing self-supervised learning methods\" (L12, L67, L286) is really misguiding.   \n\nAnother thing is that the limitation of the method is hardly discussed. In my opinion, context-based prompt augmentation will require some domain knowledge and engineering efforts, and yet it is not always possible to find appropriate external knowledge source for augmentation. I think this is one of clear limitations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept