Accessing Vision Foundation Models via ImageNet-1K

Authors: Yitian Zhang, Xu Ma, Yue Bai, Huan Wang, Yun Fu
Primary area: foundation or frontier models, including LLMs
Keywords: Foundation Model, Model Compressing, DINOv2, CLIP, SynCLR

Abstract:
Vision foundation models are renowned for the generalization ability due to massive training data. Nevertheless, they demand tremendous training resources, and the training data is often inaccessible, e.g., CLIP, DINOv2, posing great challenges to developing derivatives that could facilitate the research. In this work, we offer a very simple and general solution, named Proteus, to distill foundation models into smaller equivalents on ImageNet-1K without access to the original training data. Specifically, we remove the designs from conventional knowledge distillation settings that result in dataset bias and present three levels of training objectives, i.e., token, patch, and feature, to maximize the efficacy of knowledge transfer. In this manner, Proteus is trained at ImageNet-level costs with surprising ability, facilitating the accessibility of training foundation models for the broader research community. When leveraging DINOv2-g/14 as the teacher, Proteus-L/14 matches the performance of the Oracle method DINOv2-L/14 (142M training data) across 19 benchmarks and outperforms other vision foundation models including CLIP-L/14 (400M), OpenCLIP-L/14 (400M/2B) and SynCLR-L/14 (600M) with a significantly smaller training set of 1.2M images.

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1729339130174

thread_type:
official_review

<thread_object>
creator: reviewer_ktgp

<content>
summary:
The paper proposes Proteus, a method for distilling large vision foundation models into smaller equivalents on ImageNet-1K without access to the original training data. By focusing on token, patch, and feature-level training objectives, Proteus achieves impressive results at ImageNet-level costs, enabling broader access to training foundation models. When compared to other models including DINOv2-L/14, CLIP-L/14, OpenCLIP-L/14, and SynCLR-L/14, Proteus demonstrates competitive performance across 15 benchmarks, matching the Oracle method in some cases.

strengths:
1. Proteus offers a practical and accessible method for distilling foundation models, addressing the resource-intensive nature of training such models.
2. Proteus demonstrates appealing performance and efficacy in knowledge transfer, matching or outperforming models trained on significantly larger datasets.
3. The paper is well-structured and effectively communicates the proposed solution and its results.

weaknesses:
1. What is the performance of Proteus-L/14 distilled from the DINOv2-L/14 teacher? This informs the extent of performance degradation when distillation using ImageNet-1K.
2. Validation on varying data scales: The paper lacks validation on different data scales, such as subsets of ImageNet-1K or larger datasets like ImageNet-21K, which could provide more comprehensive insights.
3. Validation on other models and domains: The authors are suggested  to validate the effectiveness of  Proteus on diverse models and domains, such as large-scale pre-trained LLMs or diffusion models, to validate generalizability.

questions:
Please refer to weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730430080154

thread_type:
official_review

<thread_object>
creator: reviewer_oegl

<content>
summary:
This paper proposes a simple yet effective method for distilling the generalization ability of visual foundation models using only the ImageNet dataset. Specifically, compared to traditional knowledge distillation, it omits hard logits distillation, cross-entropy loss, and KL-divergence loss, while introducing MSE loss to align the intermediate results of the student network with those of the teacher network. The alignment of intermediate results involves three levels: token level, feature level, and patch level. Experiments show that the proposed method effectively distills the generalization ability of the teacher network.

strengths:
1. This paper presents a simple yet effective method that utilizes the hidden layer features of the teacher network to successfully transfer its generalization ability to the student network.

2. The article provides extensive experimental validation of its claims on the ViT architecture, with datasets that include classification, segmentation, and depth estimation. The student network architecture encompasses various sizes, including ViT-small, ViT-base and ViT-large.

3. This paper is well-written.

weaknesses:
1. This paper is less novelty. Previous works have also utilized intermediate layers for distillation on ImageNet-1K and achieved good performance on other datasets, such as MiniViT[1], and [2]. What are the differences between this work and those approaches. What are the comparative results of these methods with your work?

2. Figure 3 is not referenced in the main text, and it is unclear what issue Figure 3 is intended to illustrate.

3. When validating the generalization ability of the student network, how many epochs were fine-tuned for fine-grained classification, segmentation, and depth estimation? Compared to baseline methods like DeiT and IBOT, which used the same amount of data, was the computational load higher?

4. Why does Figure 7 show that more data leads to lower results on the ImageNet linear evaluation? For instance, sampling 50% of the data seems to yield better results than sampling 100% of the data.

5. How do the results of distillation when integrating hard logits distillation, the Cross-Entropy loss, the KL-divergence loss, and the MSE loss compare to the results obtained using only the proposed method? Would there be a decline in performance?

[1] Zhang J, Peng H, Wu K, et al. Minivit: Compressing vision transformers with weight multiplexing[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 12145-12154.

[2] Zhang H, Zhou Y, Wang G H. Dense Vision Transformer Compression with Few Samples[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 15825-15834.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730443208816

thread_type:
official_review

<thread_object>
creator: reviewer_tbls

<content>
summary:
This paper aims to compress pre-trained vision foundation models, such as CLIP and DINO-V2, using only publicly available, small-scale datasets like ImageNet-1k, as the original training data is inaccessible. 
To achieve this, this paper proposed _Proteus_, which perform knowledge distillation (KD) from the pre-trained model at token-level, patch-level, and feature-level. With extensive experiments, this paper demonstrate that the distilled model achieves comparable performance with the foundation model across multiple tasks.

strengths:
1. This paper is well-motivated. With the development of foundation model trained with private data, it's difficult for the community to reproduce or compress the foundation model due to the inaccessible training data. Thus, how to use existing academic public datasets as a proxy to achieve comparable performance is an interesting question. 

2. The performance of the model is excellent, as it achieve comparable performance with the foundation model with much less training data. Besides, this paper conducts extensive experiments and ablation studies to demonstrate the generalization ability of Proteus. 

3. The empirically finding that conventional knowledge distillation with CE loss introduce dataset bias is interesting.

weaknesses:
1. The novelty of this paper is limited. Although the results are promising, the loss at token, feature, and patch level has been widely explored in vision transformer distillation [1,2,3,4]. Besides, all of them can be considered as KD with intermediate features, which has been a common setting in KD. However, this paper does not cite and discuss any recent related works that perform KD with intermediate feature maps. 

2. Although this paper presents experimental results to support that empirically removing CE loss can mitigate dataset bias, it would be better to provide some theoretical analysis to understand this question better.

[1] Yang, Zhendong, et al. "ViTKD: Feature-based Knowledge Distillation for Vision Transformers." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.  
[2] Chen, Xianing, et al. "Dearkd: data-efficient early knowledge distillation for vision transformers." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.   
[3] Wang, Wenhui, et al. "Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers." Advances in Neural Information Processing Systems 33 (2020): 5776-5788.   
[4] Wang, Wenhui, et al. "Minilmv2: Multi-head self-attention relation distillation for compressing pretrained transformers." arXiv preprint arXiv:2012.15828 (2020).

questions:
The definition of "feature" is ambiguous. Currently, this paper just defines it as "intermediate feature _v_" (Line 171) without any additional information. This paper should specify which specific feature maps of vision transformers is used, like the output of the last transformer encoder layer of ViTs, the outputs of all 12 transformer encoder layers of ViTs, the attention maps, the Q/K/V within self-attention modules, etc.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730498681076

thread_type:
official_review

<thread_object>
creator: reviewer_cepe

<content>
summary:
While many Vision Foundation Model (VFM) checkpoints are available, their training datasets are inaccessible. This paper shows that distilling a vision foundation model using only the ImageNet dataset gives us a model that matches the performance of a similar size model trained with massive data, on several vision benchmark datasets from different domains. The paper uses the popular intermediate feature distillation strategy for KD. Experiments were conducted using several teacher-student combinations.

strengths:
Presentation: 
The paper was written well and was easy to follow.

Experiments: 
Authors evaluated their models on a wide range of CV tasks (classification, segmentation, depth estimation) using several downstream datasets. Experimental results show that models distilled only on ImageNet using the proposed approach match/outperform same size models trained on significantly larger datasets without distillation. Experiments also show that the feature-based distillation performs better than logit-based distillation approach DeiT.

weaknesses:
``The problem setting of the paper is not convincing`` 
While the original datasets used to train models such as DINOV2 are unavailable, there are numerous large scale image datasets (DataComp, LAION, etc.) that are available to the community these days. If the goal is to get a small model that performs well on a broad array of tasks, I do not understand why one should restrict themselves to distilling on ImageNet. They can use more diverse datasets, for example, diverse subsets of data taken from one of the large-scale public datasets. If the paper had compared different datasets in terms of their KD effectiveness or proposed methods to create better general purpose KD datasets, that would have been an interesting and useful contribution to the community.

The authors performed an experiment by using target task datasets for distillation and showed improved performance. This is an obvious/expected behavior, which also suggests that "we should go beyond ImageNet". If one has access to some data from target tasks, they can curate better distillation datasets as shown in a recent paper: "Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models", ICML 2024.



``Limited technical contribution``
Feature distillation is a standard approach in ML community. There are numerous papers that proposed various distillation objectives based on intermediate features. See "Knowledge distillation: A survey", International Journal of Computer Vision, 2021.
The presented approach uses standard l2 distillation loss on classification and patch tokens. It leverages masking strategy which is also a commonly-used data augmentation approach when training vision models.


``Presentation - Redundant/Confusing terminology``
The terminology of token-level, feature-level and patch-level is a bit confusing/redundant. In a vision transformer setting each image patch becomes a token, so it is confusing to refer to "classification token-based loss" as "token-level" and "patch token-based loss" as "feature-level". Both losses are token-based losses (either CLS token or patch token). Alternatively, both losses can also be called feature-based losses (CLS token features or patch token features). Both feature-level and patch-level losses described here are based on patch-tokens. So, the terminology of "feature-level vs patch-level" is confusing since both are actually patch-level losses, one with masking and one without masking. 

Table 1 - What does "Hint" in Table.1 refer to?

questions:
Suggestions:

Training small models with distillation for improved performance is a fairly standard practice, and there are numerous distillation loss functions proposed over the last decade. An important question worth looking into in the current "data-centric ML world" is ``What is the right dataset to use for distillation?``. While the paper is looking into this, focusing solely on ImageNet dataset makes the paper's contribution fairly limited, especially given that there are several large-scale public image datasets available now. 

You could avoid the redundant/confusing terminology used for the three loss terms in the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732479172293

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
reply to all reviewers

comment:
Dear Reviewers:

Thanks for your valuable comments in the review process. As raised by most reviewers, we address the concern of 'limited novelty' from three perspectives


1. To the best of our knowledge, **we are the first to replicate the success of vision foundation models with ImageNet-1K as the training set** and experiments across various scales, tasks, datasets demonstrate the generalization ability of our delivered models. **This would benefit the research community as it eliminates the dependence on undisclosed datasets and significantly reduces the training costs associated with large-scale datasets, facilitating the accessibility of training foundation models for the broader research community.** In other words, we now have the ability to compress/train vision foundation models like DINOv2 to any scale that will satisfy the need, rather than being restricted to the limited configurations provided by the original authors.

2. Technically, we focus on the problem of data-free model compression in a task-agnostic context which is intrinsically different with conventional distillation methods from two perspectives. **(1) Data-free model compression**: our goal is to compress the foundation models, which are trained with tons of private data, on a much smaller dataset while maintaining its generalization ability. This problem is non-trivial as conventional knowledge distillation on ImageNet-1K (including the hint-based distillation methods) [1,2,3,4,5] leverage the projection head and Cross-Entropy loss which will result in dataset bias and hurt the generalization ability of student network based on our results in Table 1. **(2) Task-agnostic model compression**: most foundation models are trained with self-supervised learning objectives which promise their application on dense prediction tasks as well. This also distinguishes our method from prior arts as we aim to develop a uniform learning objective but they mainly consider the classification-level learning objective, which will limit their performance on dense prediction tasks which is shown in Table 2. **We have also add comparisons with a few distillation methods mentioned by the reviewers and the results in the table below suggests that directly adapting these methods to our problem does not work well.**

| Method  | Aircraft | Cal101 | Cars | C10 | C100 | DTD | Flowers | Food | Pets | SUN | VOC | CUB | Average | 
| :-----: | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Proteus-S | 65.5% | 95.1% | 77.0% | 97.8% | 87.3% | 78.4% | 99.0% | 87.9% | 95.7% | 71.7% | 87.1% | 86.7% | 85.8% |
| MiniViT-B | 61.6% | 92.4% | 66.1% | 96.0% | 82.5% | 74.4% | 92.3% | 79.6% | 93.1% | 64.8% | 85.8% | 76.7% | 80.4% |
| MiniLMv2-S | 53.1% | 87.8% | 45.2% | 92.5% | 76.7% | 72.1% | 93.4% | 76.5% | 83.4% | 59.7% | 82.3% | 63.4% | 73.8% |

3. We admit that our method is simple, but that is the reason why **our method can compress different foundation models, trained with different self-supervised learning objectives and data sources, with a uniform design (shown in Section 3.4).** Moreover, **our analysis on the proxy dataset (shown in Section 3.5) suggests that it is possible to access foundation models at even smaller datasets compared to ImageNet-1K which further proves the robustness and effectiveness of our design.**


We thank the Reviewers for the constructive suggestions which help us to improve the work. Considering the upcoming deadline of the discussion phase, we would appreciate it if the reviewer could provide any feedback based on our reply so that we could further improve our work. Thank you so much for being with us so far.

Sincerely,  
Authors


***

[1] Hinton G. Distilling the Knowledge in a Neural Network[J]. arXiv preprint arXiv:1503.02531, 2015.  
[2] Romero A, Ballas N, Kahou S E, et al. Fitnets: Hints for thin deep nets[J]. arXiv preprint arXiv:1412.6550, 2014.  
[3] Touvron H, Cord M, Douze M, et al. Training data-efficient image transformers & distillation through attention[C]//International conference on machine learning. PMLR, 2021: 10347-10357.  
[4] Yang Z, Li Z, Zeng A, et al. ViTKD: Feature-based Knowledge Distillation for Vision Transformers[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 1379-1388.  
[5] Chen X, Cao Q, Zhong Y, et al. Dearkd: data-efficient early knowledge distillation for vision transformers[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 12052-12062.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposed a new training recipe that enables distilling pretrained vision foundation models, e.g., CLIP and DINO-v2, into smaller-size models using only publicly available data such as ImageNet-1K. The recipe includes knowledge distillation at multiple levels, e.g., token, patch, and feature. Experiments show that the distilled model achieves comparable performance with the teacher model across multiple tasks.

The rebuttal has addressed the concerns of the reviewers about the novelty, technical contributions, and ablation experiments.

Although the technical contribution of knowledge distillation approach is not that significant, this work provides a valuable solution and baselines for training state-of-the-art vision foundation models at a relatively low cost. It allows and encourages more researchers in the community to explore and contribute.
Thus, I recommend to accept this paper.