Data-scarce distillation for large-scale vision language models

Authors: Anna Bair, Madan Ravi Ganesh, Devin Willmott, J Zico Kolter
Primary area: foundation or frontier models, including LLMs
Keywords: distillation, few-shot, compression, CLIP, foundation models

Abstract:
Vision-language models (VLMs) have emerged as extremely strong zero-shot and few-shot image classifiers, performing on par with task-specific models. However, they can be unnecessarily heavy-weight for task-specific downstream applications. While existing lines of work have successfully compressed VLMs and other foundation models to varying degrees, most focus on preserving the generality of these models, rather than leveraging their power for a particular task.
In this work, we focus on the setting in which we have a limited amount of data on a downstream image classification task and a limited inference budget.
To satisfy these constraints, we focus on distilling the strong few-shot performance of CLIP on image classification tasks into a more efficient model. 
We introduce the SIDCLIP (Synthesize-Initialize-Distill CLIP) method and highlight its three components that are critical to obtaining strong performance: 1) augmenting the classifier with \textit{synthetic data} generated by leveraging CLIP itself; 2) \textit{initializing} the modeling process using a smaller CLIP model pretrained on the target architecture; and 3) incorporating \textit{knowledge distillation} to maximally mimic the performance of the larger model.  
Our set of proposed strategies produces a compact model that performs within 16\% and 10\% of CLIP's linear probe performance on 1 and 8 shot datasets respectively, while using a model with less than 2\% of the parameters of CLIP's image encoder. 
We hope our work can be useful as a practical guide for leveraging the power of foundation models in downstream data-scarce and budget constrained settings.

Decision: reject
Reviewer scores: 5, 6, 3, 6

Reviews and discussion:
<thread>
creation date:
1730127931678

thread_type:
official_review

<thread_object>
creator: reviewer_qxk8

<content>
summary:
This paper proposes a method to enhance model performance in data-scarce scenarios by distilling a smaller model using generated data derived from the CLIP model’s own image information. The paper also examines the impact of this generated data on model performance. While the approach shows some innovation in synthetic data generation, the overall methodology remains relatively simple and is validated only on three fine-grained datasets without experiments on larger-scale data.

strengths:
1. The paper introduces a novel approach for synthetic data generation.
2. It improves edge deployment of smaller models by distilling knowledge from the CLIP model.

weaknesses:
1. The experiments are limited in scale, with no evaluations on large datasets, such as ImageNet.
2. The range of compared models is narrow, including only EfficientNet-B0 (EB0).

questions:
1. What would the performance be if the CLIP teacher model provided a logits-based distillation?
2. Does the proposed method perform similarly across all backbones? Specifically, how does it perform with larger models, such as ResNet-18 mentioned in the paper?
3. How is the quality of generated data ensured?
I would be glad to raise my score if the above questions could be addressed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730580040276

thread_type:
official_review

<thread_object>
creator: reviewer_w72x

<content>
summary:
This paper proposes a framework to distill a vision-language model (CLIP) in a data-scarcity setting. Starting from a CLIP-ViT-L/14 pretrained model and a few-shot downstream data of interest, the authors propose to train a small network (EfficientNet B0) in order to tailor it for the specific dataset at hand. The framework, called SIDCLIP, consists of three components: 1- Synthesizing data using a pretrained latent diffusion model that takes as input a CLIP embedding, namely a linear combination of the text embedding describing the class and embeddings of images from the same class coming from the few-shot dataset, 2- Initializing the small student network by CLIP-like pretraining on DataComp, and 3- knowledge distillation from the large teacher model (i.e. CLIP-ViT-L/14) using classical logit matching. Experiments on 3 datasets demonstrate the effectiveness of each of the 3 components, as well as advantages compared to other "small" general-purpose baselines in the few-shot setting.

strengths:
* The paper is written in a simple language and is easy to understand and follow.
* The combination of synthetic data generated using the few-shot samples, initialization with CLIP-like pretraining, and knowledge distillation shows a good performance on the downstream dataset, and ablation shows the effectiveness of each component.

weaknesses:
* Besides combining synthetic data, initialization and distillation for the data-scarcity specific setting, the proposed approach is limited in novelty. The main elements (distillation [1] and data generation [2]) are straightforwardly utilized. While their combination is interesting, the results are not very surprising due to the expected complementary knowledge coming from different sources of information: the generation capacity of latent diffusion informed with the specific dataset at hand, the pretrained ViT-L/14 knowledge and the DataComp data for pretraining as initialization. 
* While the classical logit-based knowledge distillation approach seems to be effective, more advanced techniques were introduced in the literature ([A],[B],[C]). Since the approach is straightforward, it would be interesting to explore some other distillation methods as a positioning among other design choices. For example, some distillation methods were shown to improve classical distillation by decoupling target class and non-target class terms [B]. Other methods  show that distillation is also possible for "smaller" and "weaker" teacher [A]. Investigating such choices might help answering the question whether ViT-L/14 is needed as a teacher, or if the same or close results can be achieved with a smaller version of CLIP as teacher.
* The method is tested on only 3 datasets. Corroboration on more datasets helps assessing the value of the proposed framework and its practical use, especially that the target is a data-specialized model not a general purpose one. For example, the authors could add more experiments on common datasets used for few-shot learning with CLIP: ImageNet, DTD, EuroSAT, FGVCAircraft, SUN397, Caltech101... Please refer to [D] for more examples of such datasets. Some of these dataset (like DTD, EuroSAT, FGVCAircraft) differ from the used ones in being more specialized and fine-grained. How would each of the components help the total performance for such datasets?
* Compared to the baselines (TinyCLIP,TinyViT-5M), SIDCLIP uses a latent diffusion model, which makes the comparison unfair, and even the data-scarcity setting no more holding. Moreover, using DataComp pretraining data for the initialization stage makes the comparison more difficult, as TinyCLIP and TinyViT do not use the same dataset.

[1] Hinton et al., Distilling the Knowledge in a Neural Network. NeurIPS Workshops 2014                                                               
[2] Razzhigaev et al., Kandinsky: an Improved Text-to-Image Synthesis with Image Prior and Latent Diffusion. arxiv 2023                                        
[A] Yuan et al., Revisiting knowledge distillation via label smoothing regularization. CVPR 2020                                                                         
[B] Zhao et al., Decoupled Knowledge Distillation. CVPR 2022                                                                                                               
[C] Beyer et al., Knowledge distillation: A good teacher is patient and consistent. CVPR 2022                                                                                          
[D] Zhou et al., Learning to Prompt for Vision-Language Models. IJCV 2022

questions:
* For the ablation in Figure 1, are there any specific reasons for not showing other options like ENB0+S or ENB0+S+I? For example training ENB0+S (using synthetic data, no specific initialization, no distillation), might help better assessing the role of synthetic data.
* It is not clear how the linear probe (LP) was trained. Properly training LP is detrimental for getting the best of it, and classification results might largely fluctuate depending on the fine-tuning strategy [E],[F]. Could the authors give more details on this please?

[E] Yu et al., Task Residual for Tuning Vision-Language Models. CVPR 2023                                    
[F] Huang et al., LP++: A Surprisingly Strong Linear Probe for Few-Shot CLIP. CVPR 2024
</content>
</thread_object>

</thread>
<thread>
creation date:
1730657233031

thread_type:
official_review

<thread_object>
creator: reviewer_mcue

<content>
summary:
The paper focuses on the issues of limited data for a fine-grained downstream task and limited inference budget, and introduces a method called (Synthesize-Initialize-Distill CLIP) (SIDCLIP), involving three components: 
- Synthesize (S): a novel approach to generate synthetic data by leveraging CLIP;
- Initialize (I): model initialization of a smaller CLIP model pretrained on the target architecture; 
- Distill (D): knowledge distillation to a larger model. 
Experimental results are reported on three task-specific image classification datasets: StanfordCars, OxfordFlowers, and Food101.

strengths:
The author(s) show a lot of analysis for experiments to support their claim. While the proposal involves three components, there are specific ablation studies to show the advantage of the new model design and the comparison of eval metrics and visualization of synthetic image to exhibit the data quality.

weaknesses:
The new proposal is extending several existing methods and putting things together to reach some better performance. While this is innovative, it will be more convincing to analyze the role of each component for the overall performance improvement.

Line 339 has a typo "intialize."

questions:
Related to the weakness, for example, will the synthetic data help other models also?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730797659829

thread_type:
official_review

<thread_object>
creator: reviewer_nfde

<content>
summary:
This paper proposed SIDCLIP: Synthesize-Initialize-Distill CLIP, for few-shot distillation in data-scarce setting. SIDCLIP first augments limited labeled data using text-to-image generation model. Then, it trains a small model initialized with small CLIP. Finally it transfers knowledge from a large CLIP to the smaller model. Experiments on Stanford Cars, Oxford Flowers, Food-101 demonstrates the effectiveness of the proposed approach.

strengths:
* Unlike general-purpose distillation methods, the proposed approach focuses on specific downstream tasks, being highly effective in specialized applications.

* SIDCLIP has a fraction of the parameters of the original CLIP, making it computationally and parameter efficient.

weaknesses:
* The use of text-to-image generation to create synthetic data can introduce computational overhead, potentially making the framework slower or more resource-intensive.

* The paper primarily experiments with EfficientNet B0 as the small model architecture. How does SIDCLIP perform with different or larger backbones? Testing with a broader range of architectures would help demonstrating the generalizability of SIDCLIP.

* While SIDCLIP focuses on task-specific performance, its robustness in handling out-of-domain data (datasets significantly different from those in CLIP’s pretraining) is questionable. Since most of CLIP’s parameters remain frozen, there may be limitations in SIDCLIP’s adaptability to new domains, which could restrict its effectiveness in real-world, diverse data settings.

* The combination of synthetic data generation, initialization, and distillation steps may limit SIDCLIP’s practicality in scenarios requiring minimal preprocessing. The entire pipeline involves multiple stages that could increase setup and training time. It would be beneficial to provide the computation requirements compared to existing methods.

questions:
* Could the authors provide more quantitative details on SIDCLIP’s training efficiency compared to other distillation methods?

* How sensitive is SIDCLIP’s performance to the quality of synthetic data generated? How would different generation models (simpler generative model instead of Kandinsky) affect the few-shot results?
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a novel method for data-scarce distillation in large-scale vision-language models, focusing on leveraging CLIP's few-shot performance for efficient model deployment in resource-constrained settings. The key contribution is the introduction of the Synthesize-Initialize-Distill CLIP method, which combines synthetic data generation, model initialization, and knowledge distillation to produce a compact model while maintaining high performance.

However, there are concerns regarding the computational overhead introduced by synthetic data generation and its potential impact on the efficiency of the framework. Additionally, the novelty of combining existing methods, such as distillation and synthetic data generation, and the generalizability of the approach across different architectures and datasets were questioned. The reviewers' feedback suggests that a deeper analysis of the contribution of each component to the overall performance improvement would strengthen the paper.

Despite receiving final scores of 3, 5, 6, 6, and an average score of 5.0, the AC notes that there is significant potential in the proposed method but suggests that the authors revise the paper by addressing the reviewers' concerns, particularly regarding the novelty and generalization of the proposed solution. The AC encourages the authors to consider resubmitting the paper to the next conference with these revisions.