Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a method to transfer multimodal representations of CLIP to smaller student networks for downstream tasks. In addition to the standard similarity loss of representations, the method also applies cross-modal similarity matching loss where the topology with respect to anchor prompts is preserved among teacher and student. It also incorporates an interesting prompt augmentation technique to deal with ambiguous or minor visual concepts. The method is tested on six benchmark datasets and showed promising performance.   

Pros:  
(+) The proposed method is simple yet reasonable, also seems effective.  
(+) It includes an interesting prompt engineering method.  
(+) The paper is generally easy to follow.

Cons:  
(-) The paper seems overclaiming the improvement compared to self-supervised learning.   
(-) Discussion on the limitations and negative societal impacts of the work is largely missing.

strengths_and_weaknesses: Overall, I find no obvious weaknesses in the paper's technical side. While each component is not surprisingly novel, the overall integration is reasonable and can be a good baseline in the research of distillating CLIP.  In particular, I find context-based prompt augmentation is an inspiring idea to successfully transfer the knowledge in CLIP to various downstream applications. 

My main concern is that the paper seems to exaggerate the contribution by unfair comparison. The paper repeatedly says "outperforming the existing self-supervised learning methods (SimCLR: 65.6%)", but as shown in Table 2, with longer epochs, SwAV and BYOL-GA achieves 74.3% which is very close to the proposed method (why they are missing in Figure 1?), and even SimCLR achieves 69.1%. Why not train your method  just longer? Is it expected to improve after 200 epochs or not?   
Yes, I know it is not easy to train longer with limited computational resources, but at least this point should be honestly discussed. There's no clear reason why we should stop at 200 epochs for all methods. Without such discussion, picking up a particular value of 65.6% and just saying "outperform the existing self-supervised learning methods" (L12, L67, L286) is really misguiding.   

Another thing is that the limitation of the method is hardly discussed. In my opinion, context-based prompt augmentation will require some domain knowledge and engineering efforts, and yet it is not always possible to find appropriate external knowledge source for augmentation. I think this is one of clear limitations.
questions: Main points are described at "Strengths And Weaknesses".

Minor comments:   
- According to Figure 2, in the teacher side, the gradient of CSM loss is stopped, but not for ISM loss. Is this really true? If so, what is the reason there?
- In Table 4 (ablation study), three results are missing without explanation. What do they mean? The training wasn't successful?
- The light gray font used in some tables is very hard to see.
limitations: The authors' checklist says as follows.
- (b) Did you describe the limitations of your work? [Yes] See Section 4 and Section 5.
- (c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 5.

However, I cannot find a clear discussion in the mentioned parts. I think this kind of requirements should be addressed more seriously.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The authors propose a method (BeamCLIP) that can effectively transfer the representations of a large pre-trained multimodal model (CLIP ViTB/16) into a small target model (ResNet-50). The authors introduce two main components for the representation transfer: cross-modal similarity matching (CSM) and context-based prompt augmentation (CPA). CSM enables a student model to learn the representations of a teacher model by matching the relative similarity distribution across text prompt embeddings. CPA can alleviate the lexical ambiguity of input text prompts. 
strengths_and_weaknesses: - Strengths:
The experiments show that the representation transfer of a large pre-trained multimodal model can provide good ImageNet linear probe accuracy while outperforming the existing self-supervised learning method (SimCLR) and closing the gap with supervised learning. 

- Weaknesses:
The experiment settings seem not fair and practical. 
questions: Since the proposed method uses both the images and the labels, is it fair to compare the proposed method with the self-supervised learning methods (e.g., SimCLR)? 

limitations: As mentioned above, my main concern is fairness. The concern is mainly from the experiment settings. 
In experiments, the authors compare the proposed method with the self-supervised methods that do not use any training labels (e.g., SimCLR, MoCo-V2). The proposed method uses all images and labels of training sets for transferring the representations from a multimodal model. It is not surprising that the proposed method can obtain better accuracy. On the other hand, the proposed method uses the same data as the fully supervised models and distills the knowledge from a multimodal model pre-trained on a large-scale dataset. It is expected that the proposed method outperforms the fully supervised methods. 

To further showcase the proposed method, it'd be nice if the authors could demonstrate some possible capabilities: 
- Using data not overlapped with the target datasets for representation transfer and showing that the proposed methods obtain superior performance on target datasets. 
- Similar to the previous point. If the data mentioned above are hard to get, an easier way is to perform representation transfer on one dataset and testing on multiple target datasets 
Another interesting direction might be to use less data (e.g., 1% of training data) for representation transfer and achieve similar results in the current manuscript.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a method for transferring the representation power of CILP ViT-B/16 to a small ResNet-50 by utilizing the cross-modality of the embedding space.
Specifically, in the training stage, a student model not just tries to mimic the output image features of a teacher model, but also tries to follow the similarity distribution of the teacher model that is computed with the given set of text descriptions.
These text descriptions are similar to the ones that were used in the zero-shot classification task in the original CLIP paper, but it is amplified with the additional information of each image that could be found on Wikipedia.
Using this additional cross-modal guidance, the proposed method was able to train student networks that presents the SOTA performance on multiple classification tasks.
strengths_and_weaknesses: Strengths
- The authors propose the novel technique to guide a student model using the multi-modality of CLIP embedding space.
- The proposed method proved to be effective in boosting classification performance on multiple datasets.

Weaknesses
- The analysis on the experimental results is somewhat scant. It could be amplified by providing the dataset-wise differences and more results from the larger number of prior transferring methods.

questions: Questions
- From Table 4, I could see that SSL PT plays an important factor for some cases, though it is not described in the method part. Could you explain why this factor has such huge effect in the results?

Suggestions
- From Table 4, I found that CPA is most helpful for dataset Flowers102, which implies that this method is especially effective for fine-grained datasets. It was interesting for me, and I think tackling this kind of dataset-wise differences in effectiveness of each element would be great for following research.
- Guess Figure 3 is plotting the same figure for two models?
limitations: Authors did not address the limitations and potential negative societal impact of their work.

Suggestions
- I want to know the authors’ opinion on how much the transfer learning is helpful for handling issues concerned with sensitive web data?
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper aims at transferring knowledge from a pre-trained language-and-vision model to a small CNN-based vision network. The authors argue that a smaller network can learn better from multimodal representations in comparison with previous vision-only self-supervised methods. For transferring knowledge they propose to use two similarity matching objectives: one that matches visual representations from the teacher and student networks, and one (cross-modal) that matches the visual representations of the student network with the textual representations from CLIP. In order to increase robustness on the cross-modal similarity matching, the authors further propose to use multiple anchor textual representations, and they better craft textual prompts depending on the task and dataset. Finally, they first pre-train the student network based on previous self-supervised objectives, before distilling knowledge from CLIP. Experimentally, they show that the representations transfer well to image classification tasks, they are competitive against supervised methods and improve over self-supervised vision-only approaches.
strengths_and_weaknesses: **Strengths**

1. The paper presents an interesting approach with novel components (e.g., cross-modal similarity matching, augmented textual representations) for transferring knowledge from multimodal representations to smaller vision-only encoders. 
2. The authors experimentally demonstrate that their approach offers good visual representations that improve accuracy on image classification over prior self-supervised vision-only methods. Moreover, the performance is now closer to supervised methods, closing the gap further.


**Weaknesses**

1. From the experimental results, it is demonstrated that pre-training with prior self-supervised objectives before distilling knowledge from CLIP is crucial for performance, especially in certain datasets. However, the authors have chosen the weakest self-supervised model (SimCLR; Table 2) for pre-training. Results and conclusions would be more complete if they have chosen more than one methods for pre-training and/or one the strongest ones. Now, there is a question of whether the same patterns can be found independently of the pre-training methods and whether specific of these methods have advantage as a pre-training step for knowledge distillation from CLIP.
2. As mentioned in the above point, self-supervised pre-training is crucial, and for specific datasets (e.g., CIFAR100, Flowers102; Table 4) not pre-training has an overwhelming performance degradation (e.g., 12.94% accuracy vs. 62.18%). Given these huge gaps and the necessity of pre-training for the method to work, further analysis is needed and some discussion about why specific datasets are more sensitive to this than others.
questions: 1. Have you tried using other self-supervised methods for pre-training except for SimCLR?
2. Why specific datasets (e.g., CIFAR100, Flowers102; Table 4) are so sensitive depending on pre-training in comparison with others (e.g., ImageNet)?

limitations: As mentioned in the weaknesses of the paper, there are two limitations: 1. The fact that the authors did not use multiple self-supervised pre-training methods and/or one of the most competitive (e.g., BYOL) for drawing safe conclusions given the large impact that pre-training has, and 2. There is no further analysis on the pre-training factor, which is so important for good performance.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors propose a method for transferring CLIP's visual
representations into smaller models; this setup might be useful for
researchers working on, e.g., low resource devices, who can't run the
more powerful, full CLIP models. Their method not only minimizes the
cosine similarity between the image representations of a teacher
(ViT-B/16) and student (RN50) model, but also, incorporates a set of
"anchor" vectors, computed by the text side of CLIP: the cross-modal
similarity objective encourages the cross-modal similarities between
the teacher and the student to match, too.
strengths_and_weaknesses: Strengths:

The work is easy to follow, and save for a few typos (see below), easy
to follow. The empirical results are promising: the authors
successfully distill ViT-B/16-CLIP into a set of RN50 weights. The
cross-modal matching objective is clever, and thorough ablations
demonstrate that this innovation is indeed complementary (and, in fact,
in isolation, is actually *more* important than the cosine loss!). I
appreciated that the authors evaluated on a number of datasets beyond
ImageNet, albeit only in a zero-shot capacity.

Weaknesses:

- The authors distill a ViT-B/16 into a RN50. While the former takes
  more FLOPs than the later, ViT-B/32 is cheaper than RN50, and the
  CLIP version of that architecture outperforms both the author's RN50
  and the CLIP RN50 model.  While this doesn't totally undercut the
  presented results, it's a bit strange to me why the authors chose
  the particular distillation pair that they did --- it seems like
  ViT-B/32 (or even EfficientNet or MobileNet) would have been a good
  choice for student, and perhaps L/14-336px CLIP as the teacher would
  have made a more compelling setup.

- The ablations suggest that the cosine similarity loss isn't really
  required: the best zero-shot imagenet performance is actually
  achieved only with the cross-modal loss, and the other datasets seem
  to be within a close margin. This isn't really a /negative/,
  per-say, but a bit contrary to the story told.

- It would have been nice to see linear probe results for the other
  datasets like Pets37 as well.

- Even assuming that RN50 is the best choice of student, the empirical
  results are somewhat unimpressive. Specifically, I believe that the
  authors choose anchor points to be task-specific prompts and the author's
  model is domain adapted to (unlabelled) dataset-specific images, i.e.,
  there's reason to believe that the author's method is specifically
  tuned to the tasks described. RN50-CLIP, which is not tuned to the
  specific tasks described, achieves only slightly worse imagenet linear
  probe accuracy: 73.3 vs 74.8.

- The zero shot results in table 3 are interesting, but a bit
  incomplete. The ViT-B/32-CLIP model smaller than the author's
  distilled RN50 model achieves only slightly worse performance on
  these tasks, but it isn't presented. Again, this observation doesn't
  invalidate the author's results, it's just that one really needs to
  buy that RN50 specifically is an interesting student model.

- Overall, the authors clearly "win" on this particular architecture
  by a few accuracy points on linear probe imagenet, but it's not
  clear that this is really the best architecture to distill to
  (particularly because ViT-B/32 is smaller and there's already a CLIP
  model for that). To the author's credit, they recognize this on
  L288, but I would have preferred to see that the author's method
  generalizes to other teacher/student models, instead of just this
  one combination.

Overall, the authors have a promising core result: it's possible to
distill the knowledge of a CLIP model into a smaller set of
weights. And, the innovation that enables this distillation (that
image-text distances should be preserved in addition to image-image
distances) is novel. However, the empirical results are slightly
underwhelming: the authors model, by virtue of the selection of the
anchor points and distillation over unlabelled images from the corpora
of interest, is tuned to this particular set of datasets (vs. CLIP
that isn't). Their performance improvement over CLIP is somewhat small
in magnitude (73.3 vs 74.8). I would have liked to have seen: 1)
distillation to significantly more efficient architectures vs. RN50
(ViT-B/32 CLIP is already smaller) and with better teachers than
ViT-B/16; and 2) linear probe evaluations on unseen tasks, where the
images haven't been seen by the distillation model at training time.




Presentation fixes:

- It should be specified that Table 3 results are *all* zero shot.
- L71: week --> weak
- The broader impact statement is a bit hollow -- I would have appreciated a fuller discussion of embedded systems and/or low resource computation
- Table 4's caption has an incomplete sentence.
questions: - Did the authors try transferring to datasets where the label set and the images are not available at training time?

- Did the authors try with different teachers/students?
limitations: The authors mainly focus on making higher performance, lower-resource versions of existing CLIP models. Given that CLIP's negative impacts have been discussed in the original work and follow ups, I don't think significant additional discussion is needed here.

I did think that the "Broader impacts" section was a bit short: can more information about FLOPs/energy use/on-device computation be discussed?
review_title: null

Meta-review: metareview: This paper presents a method, BeamCLIP to transfer multimodal representations of CLIP to smaller student networks for downstream tasks. It combines both similarity loss of representations and cross-modal similarity matching loss among teacher and student. 
The method is tested on six benchmark datasets and showed convincing performance improvement. Reviewers' concerns on fairness and experimental comparison are properly handled in author feedback. 
confidence: Less certain
Decision: Accept