Title: Frozen-DETR: Enhancing DETR with Image Understanding from Frozen Foundation Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Object Detection

DETR

Foundation Model
Abstract: Recent vision foundation models can extract universal representations and show impressive abilities in various tasks. However, their application on object detection is largely overlooked, especially without fine-tuning them. In this work, we show that frozen foundation models can be a versatile feature enhancer, even though they are not pre-trained for object detection. Specifically, we explore directly transferring the high-level image understanding of foundation models to detectors in the following two ways. First, the class token in foundation models provides an in-depth understanding of the complex scene, which facilitates decoding object queries in the detector's decoder by providing a compact context. Additionally, the patch tokens in foundation models can enrich the features in the detector's encoder by providing semantic details. Utilizing frozen foundation models as plug-and-play modules rather than the commonly used backbone can significantly enhance the detector's performance while preventing the problems caused by the architecture discrepancy between the detector's backbone and the foundation model. With such a novel paradigm, we boost the SOTA query-based detector DINO from 49.0% AP to 51.9% AP (+2.9% AP) and further to 53.8% AP (+4.8% AP)  by integrating one or two foundation models respectively, on the COCO validation set after training for 12 epochs with R50 as the detector's backbone. Code will be available.

Frozen-DETR: Enhancing DETR with Image Understanding from Frozen Foundation Models

Abstract

Recent vision foundation models can extract universal representations and show impressive abilities in various tasks. However, their application on object detection is largely overlooked, especially without fine-tuning them. In this work, we show that frozen foundation models can be a versatile feature enhancer, even though they are not pre-trained for object detection. Specifically, we explore directly transferring the high-level image understanding of foundation models to detectors in the following two ways. First, the class token in foundation models provides an in-depth understanding of the complex scene, which facilitates decoding object queries in the detector's decoder by providing a compact context. Additionally, the patch tokens in foundation models can enrich the features in the detector's encoder by providing semantic details. Utilizing frozen foundation models as plug-and-play modules rather than the commonly used backbone can significantly enhance the detector's performance while preventing the problems caused by the architecture discrepancy between the detector's backbone and the foundation model. With such a novel paradigm, we boost the SOTA query-based detector DINO from 49.0% AP to 51.9% AP (+2.9% AP) and further to 53.8% AP (+4.8% AP) by integrating one or two foundation models respectively, on the COCO validation set after training for 12 epochs with R50 as the detector's backbone.

1 Introduction

Understanding an image at both global and local levels is a key factor for a wide range of vision perception tasks. Typically, in object detection, an in-depth understanding of the image can assist model reasoning under many challenging situations. First, the conflict between detecting the whole object and parts of it always exists in object detection since parts of the object are also annotated in many scenarios, e.g ., objects under occlusion. With a global understanding of the image, detectors can recognize different parts of the same object and detect the objects as completely as possible, as shown in Figure 1 (a). Second, the co-occurrence of objects can facilitate finding some missing objects. In Figure 1 (b), a man is sitting on something, from which we can infer that it is a bench with a strange appearance. And the bottle aside the man may help us find the bottle in his hand. Moreover,

train

train

train

bench

bottle

bird

broccoli

bird bg

bg

bg

(a) clarify the relation between parts and the whole

(b) find missing objects

(c) correct wrong classification

some salient features can also help us distinguish similar objects, i.e ., distinguishing the fish and broccoli from birds as shown in Figure 1 (c). Prior efforts to enhance detectors' image understanding ability include using a strong backbone with a large receptive field [30, 23, 18] and explicitly injecting scene information to detectors [48, 14, 12]. Recent query-based detectors [3, 73, 22, 66] modeling object queries via the global self-attention mechanism also enjoy global reasoning. In this work, we explore the image understanding ability from outside knowledge rather than the detector itself.

In light of this, we focus on the recently attractive foundation models, which have shown an impressive understanding of the image via large-scale pre-training, even without task-specific data. For example, DNF CLIP [21] with ViT-H can achieve 84.4% zero-shot classification accuracy on ImageNet [17], achieving similar results to the same ViT-H [19] (85.1%) trained in a supervised way. Benefiting from the advanced architecture, extensively collected data, and well-designed pre-training tasks, the off-the-shelf foundation models are already equipped with strong image understanding abilities.

In this work, we propose Frozen-DETR, which uses a frozen foundation model as a plug-and-play module to boost the performance of query-based detectors. Instead of using the foundation model as a backbone, we regard it as a feature enhancer from two perspectives: First, to utilize the global image understanding ability of foundation models, we take the class token from them as the full image representation, termed image query. The image query is concatenated with object queries and facilitates decoding object queries in the decoder by providing a complex scene context. Second, the fine-grained patch tokens with high-level semantic cues from foundation models are considered as another level of the feature pyramid, which is then fused with the detector's feature pyramid via the encoder layers. The foundation model is parallel with the backbone and frozen during training.

Compared with methods that use the foundation model as a learnable backbone [40, 13] or a frozen backbone [58, 44], our method enjoys the advantages in the following three aspects: a) No architecture constraint . Since we do not require the foundation model to extract multi-scale features, any architecture, CNNs, ViTs, or hybrid ones, can be used as the foundation model's architecture. Besides, the detector and the foundation model can use different structures. b) Plug-and-play . Our method can be plugged into various query-based detectors without modifying the detector's structure, the foundation model's structure, and the training recipe. c) Asymmetric input size . We use the foundation model as a feature enhancer rather than a backbone. The input image size of the foundation model can be much smaller than the one for the backbone ( e.g ., 336 vs . 1333). Considering the asymmetric input size, we can use a large foundation model with an acceptable computation burden.

We find that CLIP [54] is one of the best candidates for Frozen-DETR, and its high-level semantic understanding significantly enhances the classification ability of DETRs in many challenging scenarios. Taking the well-known DINO [66] detector as the baseline, we boost its performance to 53.2% AP (+2.8%) on the COCO dataset. On the large vocabulary LVIS dataset, Frozen-DETR increases an impressive 6.6% AP. Considering the long-tail data distribution, Frozen-DETR increases 8.7% APr and 7.7% APc, showing the potential to alleviate the class imbalance problem. On the open-vocabulary scenario, we increase by 8.8% novel AP, showing strong open-vocabulary ability. Further, Frozen-DETR achieves almost the same performance on the COCO-O [50] dataset compared with the one on the in-domain COCO dataset, showing great domain generalization ability.

Object Queries

Backbone

Foundation

Model

Class Token (Image Query)

Transformer

Encoder

Self-Attention

Cross-Attention

FFN

Image Queries

discard

cat

cls / reg

Transformer

Decoder

resize to a

small size

discard

cat

2 Related Works

Foundation models and their application in object detection. Recent vision foundation models can be divided into supervised ones [57, 54, 39, 35] and self-supervised ones [11, 4, 60, 27, 1, 61, 10, 70, 52]. Training foundation models in a supervised manner need to collect web-scale high-quality datasets, such as image classification dataset ImageNet-22k [17] (DEiT-III [57]), image-text pair dataset WIT [54] (CLIP), grounding dataset GoldG [39] (GLIP), and segmentation dataset SA1B [35] (SAM). Training with gold annotations from humans, these models can be directly applied to many downstream tasks without fine-tuning. However, the human-annotated datasets are hard to scale up due to intensive labor. Self-supervised learning is an alternative. With well-designed pre-tasks, e.g ., contrastive learning [11, 4, 60, 65], mask image modeling [27, 1, 61, 10], or their combination [70, 52], models can learn distinctive representations without human annotations. But to unleash the power of self-supervised learning on downstream tasks, models should be fine-tuned with task-specific data.

Pre-training-then-fine-tuning is a common paradigm to use the pre-trained foundation models in object detection. By transferring the pre-trained knowledge, the detector can converge much faster than training from scratch [28], shows better advantages in robustness and uncertainty [31], and even gains higher performance. However, in this paradigm, the detector should use the same backbone as the pre-trained foundation model to transfer the pre-trained weights. Unfortunately, not all backbones are suitable for dense prediction tasks. Thus, many task-oriented designs are introduced to compensate for the structure inadaptability, e.g ., ViT-Adapter [13]. Besides, fine-tuning a large foundation model is not always acceptable due to resource constraints. To this end, a few works [58, 44] explore using frozen foundation models as the backbone. To make the frozen backbone work in object detection, modifying the structure (heavy neck and head) and the training recipe (long training schedule) is necessary. In this work, we explore using the foundation model as a feature enhancer rather than the backbone, which can avoid the problems mentioned above.

Query-based object detectors. Different from traditional detectors [55, 29, 42, 51], DETR [3] formulates object detection as a set prediction task, which is an end-to-end model without using nonmaximum suppression (NMS). The following improvements mainly include advanced formulations of object queries [73, 24, 62, 45, 47], stabilizing bipartite matching [36, 66, 46], providing more supervision signals [8, 6, 34, 74, 68], alleviating conflict or competition between object queries [67, 32, 53] and knowledge distillation [5, 33, 9]. In this work, we explore enhancing DETR from another aspect by utilizing the image understanding ability of off-the-shelf foundation models.

Foundation

Model

mean

attn_mask

crop

resize

image query

class token

patch token

(a) Crop Image (5 passes)

(c) Multiple Class Tokens (1 pass)

3 Foundation Models as Feature Enhancers

3.1 Preliminaries

Vision Transformer (ViT). Recently, transformer [59] has been shown to be a scalable architecture [16, 2, 37] and many foundation models [54, 52, 61, 27, 57] are based on it. Different from CNNs, ViT [19] first splits images into patches and projects each patch to a patch token. These tokens are flattened in spatial dimensions and modeled in a sequence manner. In addition to patch tokens, a learnable class token is prepended to the patch sequence. The patch tokens preserve the local details for each patch, while the class token represents the global information for the whole image. We take both the class token and patch tokens into account to enhance the detector's ability.

DETR. A common DETR-like [3] detector includes 3 parts: backbone, encoder, and decoder. The backbone can be any architecture that produces feature pyramids. The encoder uses deformable attention [73] to enhance the feature maps while avoiding unaffordable computation costs from global self-attention. Taking the refined feature maps and some object queries as inputs, the decoder aims to refine the object queries layer by layer and predicts the class label and box coordinates for each object query. In the subsequent improvement, each object query can be divided into two parts: content vector representing the feature for each object, and position vector indicating the location for each object. During the training, each object query will be optimized towards a single object or background.

In this work, we employ patch tokens from foundation models to enhance the DETR's encoder via feature fusion and facilitate the decoding process of DETR's decoder with the class token from foundation models.

3.2 Enhancing Decoder by Treating Class Tokens as Image Queries

We introduce a new kind of query into the decoder, termed image query. Different from conventional object queries whose content vector represents a single object and the position vector is the bounding box of the object, the image query represents the whole image, and its box is the full image boundary. Since pre-trained foundation models have a strong ability to understand complex images at a global level, we take advantage of their scene-understanding ability and treat the class token as the image query. With the image query as context, object queries can be better classified.

Specifically, as shown in Figure 2, for each image, we extract the image query by passing the raw image to the frozen foundation model. The image is resized to the pre-training image size of the foundation model. Then, in each decoder layer, we project the image query to the same dimensions as object queries and concatenate these two kinds of queries before feeding them into the self-attention module. In the following self-attention computation, object queries can interact adaptively with the image query to absorb high-level image understanding from the foundation model. Finally, the image query is discarded after passing the self-attention module.

Since a single global image query may overlook several inconspicuous small objects, to compensate for this and provide fine-grained local context, we propose to split the image into multiple sub-images evenly and equip each sub-image with an image query. For example, if we split the image into

2 × 2 = 4 sub-images, we will obtain 5 image queries, including an original global image query and four local image queries. A straightforward method to obtain multiple image queries is to crop the sub-images from the original one and pass them to the foundation model individually, as shown in Figure 3 (a). However, forwarding the foundation model multiple times is always time-consuming, especially when the foundation model is large enough.

We provide two fast implementations to tackle the above limitation. In the first implementation (Figure 3 (b)), we spatially split the entire patch tokens into multiple groups of tokens corresponding to the sub-images, where the mean feature of each token group is regarded as its local image query. The second one employs extra replicated class tokens to extract local image queries. We apply a masked attention operation to ensure each local class token focuses on its corresponding sub-image by restricting the interaction to a local area, as shown in Figure 3 (c). These two fast implementations can obtain global and all local image queries in a single forwarding process, thus significantly reducing the computation costs. Finally, all image queries are concatenated with object queries.

3.3 Enhancing Encoder by Feature Fusion

In addition to utilizing the global scene understanding of foundation models in the decoder, we further reuse the patch tokens with fine-grained semantic details. Specifically, as shown in Figure 2, we take patch tokens from the last layer of foundation models and reshape them to a 2D feature map. This 2D feature map is then concatenated with the feature maps from the detector's backbone and passed to the encoder, allowing adaptive fusion. After fusion, the backbone features are expected to assimilate the high-level semantic understanding in foundation models. We empirically find that we can simply discard the patch tokens after feature fusion, and using the enhanced feature maps from the detector's backbone is sufficient. Note that the feature map of patch tokens ( 24 × 24 for an image with input size 336 and patch size 14) is much smaller than feature maps from the backbone ( 167 × 167 for an image with input size 1333 and stride 8), the additional computation burden is acceptable.

4 Experiments

In the following subsections, we first explore the best practices of using foundation models as plugand-play modules and then show the highly engaging performance under various scenarios. Since the detector DINO [66] and the self-supervised foundation model DINOv2 [52] share the same name and may cause confusion. We rename the detector DINO as DINO-det in the following.

4.1 Image Queries Provide Complex Contexts

Setting. In this subsection, we take AdaMixer [24] as the baseline to explore the usage of image queries since it is a decoder-only detector and converges fast with a small computation cost. All experiments are conducted on the COCO [43] dataset with 300 queries, 12 training epochs, and 4 V100 GPUs. Unless otherwise specified, we employ the ImageNet-1k [17] supervised pre-training ResNet-50 (R50) [30] as the backbone of the detector.

Are image queries helpful? In this ablation study, we use the OpenAI CLIP ViT-L-14-336 [54] to extract the global image query. As shown in the upper part of Table 1, with only one global image query appended to object queries, the AP significantly increases by 0.9%. Although the global image query represents the information from a global view, the AP for both small (+0.8% AP s ) and large (+1.8% AP l ) objects are increased.

Are image queries equal to a better detector backbone? We change the detector's backbone to the CLIP R50 and train it along with the detector. As shown in the lower part of Table 1, using a stronger backbone improves the AP to 45.5%, which is in line with common experience. Nevertheless, using

an additional global image query further increases 0.6% AP, showing that using the foundation model as an image query is orthogonal to using the foundation model as a backbone.

Which foundation model is more suitable to extract image queries? In this experiment, we choose 5 representative foundation models with different pre-training methods: DEiT-III [57] (ImageNet-22k supervised pre-training), CLIP [54] (image-text pair alignment), MAE [27], BEiT-3 [61] (masked data modeling) and DINOv2 [52] (masked data modeling and online self-distillation). All the models use the ViT-L and the input image sizes are adjusted to 336 by interpolating the positional embedding. For models that do not use the class token during the pre-training (MAE and BEiT-3), we use the mean patch tokens as the image query. Besides, we also use the mean feature from the detector's backbone as the image query for a clear comparison, denoted as 'Self'. As shown in Table 2, using the mean backbone feature of the detector as the image query does not help much since the DETRs already model object queries with global self-attentions. Besides, we find that the foundation models which are pre-trained using human labels (DEiT-III and CLIP) perform better than the self-supervised counterparts, perhaps the self-supervised ones lack the high semantic from human supervision thus the features can not be utilized in the down-stream tasks without fine-tuning. Another possible reason is that models pre-trained with masked data modeling could focus more on local texture details. Since image-text pairs are more scalable than image classification datasets, we choose the OpenAI CLIP (ViT-L-14-336) as the foundation model in the following experiments.

Extracting image queries with fast implementations. As demonstrated in Section 3.2, using some additional local image queries to represent sub-images can preserve more details for the local context. Table 3 illustrates the results of different implementations. The training time t train is the total time for training 12 epochs and the inference time is tested on a single V100 GPU with batch size 1. As shown in Table 3, cropping sub-images from the original one can increase the AP to 45.7% but with more training time (+18h) and inference time (+150%) since images should pass through the whole foundation model for multiple times. In contrast, using the mean patch token or multiple class tokens can save much time as only one pass is needed. However, using the mean patch token is less effective since the patch token is less representative than the class token in ViT CLIP [64]. Thus, in the following experiments, we use multiple class tokens to extract multiple image queries by default.

How many image queries do we need? We conduct experiments with 0, 1, 5 (1+4), and 14 (1+4+9) image queries and achieve 44.1% AP, 45.0% AP, 45.8% AP, and 45.6% AP, respectively. We empirically find that using 5 image queries is enough and more image queries do not help. Thus we use 5 image queries by default.

4.2 Patch Tokens Enhance Feature Fusion in Encoder

Setting. In this subsection, we change the baseline to Co-DINO [74], which has both transformer encoder and decoder. For fast implementation, we use four-scale feature maps (1/8, 1/16, 1/32, 1/64) and do not use co-heads. All experiments are conducted on the COCO [43] dataset with R50, 900 queries, 12 training epochs, and 4 V100 GPUs.

Ablation studies on each component of feature fusion. As shown in Table 4, adding image queries to the new detector also gets 1.8% AP gains (Exp. 2), showing its generalizability. If we regard the patch tokens as an additional 24 × 24 feature map and append it to the encoder (5-scale feature maps now, Exp. 3), the AP further increases by 1.8% AP. Since the added feature map is small and the foundation model is frozen, the additional computation cost and training time GPU memory (batch size 2 per GPU) are acceptable. We further find sending the patch tokens to the cross-attention in the decoder (Exp. 4) is unnecessary and we simply discard the patch tokens after the encoder.

Ablation studies on the input image size of the foundation model. In Table 5, we change the input size to 224, 336, 448, and 560, and achieve 51.5% AP (333 GFLOPs), 52.6% AP (400 GFLOPs), 53.1% AP (494 GFLOPs), and 52.8% AP (615 GFLOPs). We find that the input image size is not necessarily the larger, the better. On the one hand, the image size should not be too large since the foundation model is pre-trained under a small resolution. On the other hand, the foundation model provides a high-level semantic image understanding rather than location texture details. Thus, the large input size is not necessary. Further, the large input size brings a huge computation burden since the self-attention module in the foundation models has a quadratic complexity in the length of patch tokens. By default, we use 336 as the input image size.

Ablation studies on the model size of the foundation model. In Table 6, we change the foundation model with various model sizes (R101, ViT-B-16, ViT-L-14) but keep the number of patch tokens to a similar size. There is a clear trend that a stronger foundation model can achieve higher performance, showing the great scalability of our method.

Fine-tuning or freezing the foundation model. To make the fine-tuning affordable, we use CLIP R101 as the foundation model in this experiment. As shown in Table 7, tuning the foundation model with the detector underperforms the frozen one. We assume that training with much fewer downstream task data breaks the pre-trained representations in foundation models.

4.3 Main Results on COCO dataset

In this subsection, we apply our Frozen-DETR (CLIP ViT-L-14-336) to various well-known detectors, including DAB-DETR [45], DN-DETR [36], MS-DETR [68], HPR [69], DINO-det [66] and CoDINO [74]. We strictly follow their experiment settings without changing any hyper-parameters.

Table 9: Results on LVIS v1 training with full annotations. *: Our implementation.

As shown in Table 8, Frozen-DETR outperforms baselines ranging from 2.7% AP to 4.0% AP. The results on different detectors show the generalization ability. Although the additional patch tokens from foundation models make our Frozen-DETR also have 5 scale feature maps in the encoder, the added feature map ( 24 × 24 ) is much smaller than the C 2 feature map in DINO-det-5scale ( 333 × 333 for input size 1333 and stride 4). Thus the additional computation costs of Frozen-DETR (279 → 400 GFLOPs) are also much smaller than DINO-det-5scale (279 → 860 GFLOPs). Further, Frozen-DETR (DINO-det-4scale) and Frozen-DETR (Co-DINO-4scale) also outperform DINO-det5scale and Co-DINO-5scale by 1.9% AP. Moreover, we can also increase 0.8% AP when using a strong backbone Swin-B [49] pre-trained on ImageNet-22k, demonstrating our great scalability. We find that large objects enjoy the most significant improvement from the image understanding of the foundation model. For example, there is an improvement of 6.3% AP l on the 12 epochs setting over DINO-det-4scale. We hypothesize that it is because large objects may easily be confused by the relation between the parts and the whole objects, as illustrated in Figure 1 (a). We also find that performance gains from more epochs for Frozen-DETR on Co-DINO (+0.3% AP) are less than Frozen-DETR on DINO-det (+1.3% AP). This is because most SOTA query-based detectors can converge extremely fast within 12 epochs. Some training strategies are excepted to further improve the performance with longer training schedules, e.g., large-scale jitter and other data augmentation.

4.4 Dose Frozen-DETR Work under Large Vocabulary Settings?

Closed-set Setting. Since Frozen-DETR has a strong advantage in classification compared with baselines benefiting from the image understanding of foundation models, we further validate the effectiveness of Frozen-DETR on the challenging LVIS v1 [26] dataset. LVIS dataset is a large vocabulary dataset (1203 classes) with long tail distribution. The classes are divided into rare, common, and frequent classes based on the number of annotations. We choose DINO-det-4scale [66] as the baseline and train the model for 24 epochs without using mask annotations. Following common practices, we use repeat factor sampling and Federated Loss [72]. As shown in Table 9, Frozen-DETR

has an impressive improvement of 6.6% AP over DINO-det, demonstrating that the benefit of image understanding from foundation models is even more significant in the more challenging scenario. Further, the improvement on rare (+8.7% AP) and common classes (+7.7% AP) is larger than frequent classes, showing that Frozen-DETR has the potential to alleviate the class imbalance problem.

Open-Vocabulary Setting. As CLIP demonstrates an impressive zero-shot ability, many works [25, 20, 71] aim to inherit its generalization ability to achieve open-vocabulary recognition. In this subsection, we also validate the open-vocabulary ability inherited by Frozen-DETR. In this setting, annotations for rare classes are removed and only common and frequent class annotations are used for training. The AP on rare classes is the main evaluation metric. We follow common practices by replacing the classifier with class prompts, which are encoded by CLIP text encoder with 80 handcrafted prompts, e.g ., 'a photo of {category} in the scene'. No other distillation methods [25, 63, 38] or additional datasets [71, 41] are used. As shown in Table 10, since we use CLIP as a frozen feature enhancer, the open-vocabulary ability is largely inherited. We outperform the baseline DINO-det by 8.8% AP r . We also outperform many detectors tailored for open-vocabulary detection, even though it is not a fair comparison as we use CLIP ViT-L-14-336 and others use CLIP ViT-B-32.

4.5 Combining Multiple Foundation Models

In this subsection, we explore whether combining multiple foundation models can further improve the performance since image understanding abilities from different aspects may be learned by different foundation models trained with different pre-tasks. Here we try to combine DINO-det-4scale [66] with CLIP [54] and DINOv2 [52]. Both foundation models use ViT-L-14-336. We only use the patch tokens from DINOv2 as another feature map ( 4+2 = 6 feature maps now). As shown in Table 11, we can further increase DINO-det to 53.8% AP (+4.8% AP), showing that different foundation models may be complementary in the aspect of image understanding.

4.6 Transfering Frozen-DETR to Other Domains

In the real world, input images always suffer from natural distribution shifts. We also find that FrozenDETR inherits great domain generalization ability from frozen foundation models. We directly transfer the model trained on the COCO dataset to the COCO-O dataset [50] without fine-tuning, which is a dataset having the same classes as COCO but different domains, such as sketch, weather, cartoon, painting, tattoo, and handmake. As shown in the Table 12, Frozen-DETR achieves almost the same performance on both datasets, while other detectors degrade a lot on the COCO-O. The performance of Frozen-DETR on COCO-O is two times higher than the baselines and even higher than detectors with strong backbones, showing its strong robustness.

4.7 How does Frozen-DETR work?

To understand how Frozen-DETR works, we conduct error analysis [7] in Table 13. The location error (Loc) denotes the predictions with correct labels but low IoUs. The classification error (Cls) denotes the predictions with the correct locations but incorrect labels. The background error (BG)

DINO

more continuous

more highlighted

indicates the detector erroneously marks a background region as an object. The false negative error (FN) means the detector overlooks some annotated objects.

The results in Table 13 validate that the benefit of CLIP and DINOv2 comes more from high-level semantic understanding rather than texture details, which is good for classification more than localization . With strong image understanding, the detector can find missing objects (lower false negative error) and correct wrong classification (lower classification and background error). We also visualize the feature maps ( l 2 norm) after the encoder in Figure 4 to better illustrate the benefit. The enhanced high-level semantic understanding leads to a more complete activation of objects (the first row), ensuring that objects are detected as complete entities. Further, it allows certain objects to stand out distinctly against the background (the second row). More visualizations are provided in the Appendix.

5 Conclusions

In this work, we show that frozen foundation models can be versatile feature enhancers, even though they are not pre-trained for object detection. We explore a new way to utilize pre-trained foundation models as a feature enhancer rather than a backbone. Class tokens from them provide a compact context for decoding object queries in the decoder. Patch tokens further inject semantic details into feature maps via feature fusion in the encoder. Experiments show that CLIP is one of the best candidates for Frozen-DETR and the image understanding ability in CLIP can greatly enhance the classification ability of DETRs, especially in large vocabulary settings. We hope our work can shed new light on reusing the existing strong foundation models on various downstream tasks.

Acknowledgments and Disclosure of Funding

This work was supported partially by NSFC(92470202, U21A20471), Guangdong NSF Project (No. 2023B1515040025) and the Major Key Project of PCL under Grant PCL2024A06. This work was also supported by Alibaba Innovative Research Program.

References

Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254 , 2021.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS , 2020.

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV , 2020.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV , 2021.

Jiahao Chang, Shuo Wang, Hai-Ming Xu, Zehui Chen, Chenhongyi Yang, and Feng Zhao. Detrdistill: A universal knowledge distillation framework for detr-families. In ICCV , 2023.

Fangyi Chen, Han Zhang, Kai Hu, Yu-kai Huang, Chenchen Zhu, and Marios Savvides. Enhanced training of query-based object detection via selective query recollection. In CVPR , 2023.

Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155 , 2019.

Qiang Chen, Xiaokang Chen, Jian Wang, Shan Zhang, Kun Yao, Haocheng Feng, Junyu Han, Errui Ding, Gang Zeng, and Jingdong Wang. Group detr: Fast detr training with group-wise one-to-many assignment. In ICCV , 2023.

Xiaokang Chen, Jiahui Chen, Yan Liu, and Gang Zeng. D3etr: Decoder distillation for detection transformer. arXiv preprint arXiv:2211.09768 , 2022.

Xinlei Chen, Zhuang Liu, Saining Xie, and Kaiming He. Deconstructing denoising diffusion models for self-supervised learning. arXiv preprint arXiv:2401.14404 , 2024.

Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In ICCV , 2021.

Zhao-Min Chen, Xin Jin, Borui Zhao, Xiu-Shen Wei, and Yanwen Guo. Hierarchical context embedding for region-based object detection. In ECCV , 2020.

Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. In ICLR , 2023.

Zhe Chen, Shaoli Huang, and Dacheng Tao. Context refinement for object detection. In ECCV , 2018.

Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In ICLR , 2023.

Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In ICML , 2023.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , 2009.

Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Scaling up your kernels to 31x31: Revisiting large kernel design in cnns. In CVPR , 2022.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2021.

Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open-vocabulary object detection with vision-language model. In CVRP , 2022.

Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. arXiv preprint arXiv:2309.17425 , 2023.

Shenghao Fu, Junkai Yan, Yipeng Gao, Xiaohua Xie, and Wei-Shi Zheng. Asag: Building strong one-decoder-layer sparse detectors via adaptive sparse anchor generation. In ICCV , 2023.

Shang-Hua Gao, Ming-Ming Cheng, Kai Zhao, Xin-Yu Zhang, Ming-Hsuan Yang, and Philip Torr. Res2net: A new multi-scale backbone architecture. IEEE TPAMI , 2019.

Ziteng Gao, Limin Wang, Bing Han, and Sheng Guo. Adamixer: A fast-converging query-based object detector. In CVPR , 2022.

Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In ICLR , 2022.

Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In CVPR , 2019.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR , 2022.

Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. In CVPR , 2019.

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV , 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016.

Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In ICML , 2019.

Zhengdong Hu, Yifan Sun, Jingdong Wang, and Yi Yang. Dac-detr: Divide the attention layers and conquer. In NeurIPS , 2024.

Linjiang Huang, Kaixin Lu, Guanglu Song, Liang Wang, Si Liu, Yu Liu, and Hongsheng Li. Teach-detr: Better training detr with teachers. IEEE TPAMI , 2023.

Ding Jia, Yuhui Yuan, Haodi He, Xiaopei Wu, Haojun Yu, Weihong Lin, Lei Sun, Chao Zhang, and Han Hu. Detrs with hybrid matching. In CVPR , 2023.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In ICCV , 2023.

Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang. Dn-detr: Accelerate detr training by introducing query denoising. In CVPR , 2022.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML , 2023.

Liangqi Li, Jiaxu Miao, Dahu Shi, Wenming Tan, Ye Ren, Yi Yang, and Shiliang Pu. Distilling detr with visual-linguistic knowledge for open-vocabulary object detection. In ICCV , 2023.

Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In CVPR , 2022.

Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In ECCV , 2022.

Chuang Lin, Peize Sun, Yi Jiang, Ping Luo, Lizhen Qu, Gholamreza Haffari, Zehuan Yuan, and Jianfei Cai. Learning object-language alignments for open-vocabulary object detection. In ICLR , 2023.

Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR , 2017.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV , 2014.

Yutong Lin, Ze Liu, Zheng Zhang, Han Hu, Nanning Zheng, Stephen Lin, and Yue Cao. Could giant pre-trained image models extract universal representations? In NeurIPS , 2022.

Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. Dab-detr: Dynamic anchor boxes are better queries for detr. In ICLR , 2022.

Shilong Liu, Tianhe Ren, Jiayu Chen, Zhaoyang Zeng, Hao Zhang, Feng Li, Hongyang Li, Jun Huang, Hang Su, Jun Zhu, et al. Detection transformer with stable matching. In ICCV , 2023.

Yang Liu, Yao Zhang, Yixin Wang, Yang Zhang, Jiang Tian, Zhongchao Shi, Jianping Fan, and Zhiqiang He. Sap-detr: Bridging the gap between salient points and queries-based transformer detector for fast model convergency. arXiv preprint arXiv:2211.02006 , 2022.

Yong Liu, Ruiping Wang, Shiguang Shan, and Xilin Chen. Structure inference net: Object detection using scene-level context and instance-level relationships. In CVPR , 2018.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV , 2021.

Xiaofeng Mao, Yuefeng Chen, Yao Zhu, Da Chen, Hang Su, Rong Zhang, and Hui Xue. Coco-o: A benchmark for object detectors under natural distribution shifts. In ICCV , 2023.

Qijie Mo, Yipeng Gao, Shenghao Fu, Junkai Yan, Ancong Wu, and Wei-Shi Zheng. Bridge past and future: Overcoming information asymmetry in incremental object detection. In ECCV , 2024.

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 , 2023.

Yifan Pu, Weicong Liang, Yiduo Hao, Yuhui Yuan, Yukang Yang, Chao Zhang, Han Hu, and Gao Huang. Rank-detr for high quality object detection. In NeurIPS , 2024.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021.

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurIPS , 2015.

Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, et al. Sparse r-cnn: End-to-end object detection with learnable proposals. In CVPR , 2021.

Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit iii: Revenge of the vit. In ECCV , 2022.

Cristina Vasconcelos, Vighnesh Birodkar, and Vincent Dumoulin. Proper reuse of image classification features improves object detection. In CVPR , 2022.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , 2017.

Shashanka Venkataramanan, Mamshad Nayeem Rizve, João Carreira, Yuki M Asano, and Yannis Avrithis. Is imagenet worth 1 video? learning strong image encoders from 1 long unlabelled video. In ICLR , 2024.

Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for vision and vision-language tasks. In CVPR , 2023.

Yingming Wang, Xiangyu Zhang, Tong Yang, and Jian Sun. Anchor detr: Query design for transformer-based detector. In AAAI , 2022.

Size Wu, Wenwei Zhang, Sheng Jin, Wentao Liu, and Chen Change Loy. Aligning bag of regions for open-vocabulary object detection. In CVPR , 2023.

Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Xiangtai Li, Wentao Liu, and Chen Change Loy. Clipself: Vision transformer distills itself for open-vocabulary dense prediction. In ICLR , 2024.

Junkai Yan, Lingxiao Yang, Yipeng Gao, and Wei-Shi Zheng. Self-supervised cross-stage regional contrastive learning for object detection. In ICME , 2023.

Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and HeungYeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. In ICLR , 2023.

Shilong Zhang, Xinjiang Wang, Jiaqi Wang, Jiangmiao Pang, Chengqi Lyu, Wenwei Zhang, Ping Luo, and Kai Chen. Dense distinct query for end-to-end object detection. In CVPR , 2023.

Chuyang Zhao, Yifan Sun, Wenhao Wang, Qiang Chen, Errui Ding, Yi Yang, and Jingdong Wang. Ms-detr: Efficient detr training with mixed supervision. In CVPR , 2024.

Jinjing Zhao, Fangyun Wei, and Chang Xu. Hybrid proposal refiner: Revisiting detr series from the faster r-cnn perspective. In CVPR , 2024.

Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:2111.07832 , 2021.

Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. In ECCV , 2022.

Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Probabilistic two-stage detection. arXiv preprint arXiv:2103.07461 , 2021.

Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR , 2021.

Zhuofan Zong, Guanglu Song, and Yu Liu. Detrs with collaborative hybrid assignments training. In ICCV , 2023.

A Comparisons between Using Foundation Models as a Backbone and as a Plug-and-Play Module

Foundation Model

image

1 / 32

1 / 16

1 / 8

1 / 4

🔥

Foundation Model

image

1 / 32

1 / 16

1 / 8

1 / 4

Adapter

🔥

🔥

Foundation

Model

image

Heavy Neck

Heavy Head

🔥

🔥

Foundation

Model

image

Backbone

🔥

resize

(336)

Encoder

🔥

Decoder

🔥

class token

patch   token

(a) ViTDet

(b) ViT-Adapter

In this work, we propose a novel paradigm (comparisons are shown in Figure 5) to integrate frozen vision foundation models with query-based detectors, firstly showing that frozen foundation models can be a versatile feature enhancer to boost the performance of detectors, even though they are not pre-trained for object detection.

In previous practices, large vision foundation models are always used as a pre-trained backbone and fine-tuned with detectors in an end-to-end manner. Although such a paradigm achieves high performance, the computation cost of fine-tuning such a large vision foundation model is unaffordable. We use ViT-L as an example to illustrate this problem, as ViT-L is a common architecture for most vision foundation models. In the above table, we choose DINO-det-4scale with R50 backbone as the baseline and compare it with three methods: our Frozen-DETR (CLIP ViT-L-336), DINO-det-5scale, and DINO-det-4scale with a foundation model (ViT-L) as the backbone. We use the ViT-L as the backbone following ViTDet. For the training, we use 4 A100 GPUs with 2 images per GPU except for the ViT-L backbone due to out-of-memory (OOM). For inference, we use a V100 GPU with batch size 1 in line with the main text. As shown in the Table 14, the computation cost in both training and inference for Frozen-DETR is the lowest among the three variants.

Compared with DINO-det-4scale with a foundation model as a backbone, training a ViT-L backbone needs 4.2 hours per epoch and 44 GB memory per GPU, which is significantly higher than our Frozen-DETR (1.4 hours and 15 GB with 2 images per GPU). For inference, using ViT-L as a backbone needs 10 GB GPU memory and runs at 2.1 FPS on a V100 GPU. While inference with Frozen-DETR only needs 3 GB GPU memory (3x fewer) and runs at 6.5 FPS (3x faster).

Compared with DINO-det-5scale, our Frozen-DETR not only runs faster but also significantly outperforms DINO-det-5scale by 1.8% AP (53.1% AP vs 51.3% AP), as shown in Table 8.

Thus, Frozen-DETR achieves a good performance-speed trade-off.

B More Visualizations

DINO

(a)

(a)(c)

(a)(b)(c)

(c)

(a)(b)(c)

dy bear: 0.64

botle 0.23

ottle: 0.21

cup: 0.29

In Figure 6, we show more results with or without the foundation model. With the high-level image understanding ability from foundation models, the detector can detect objects as completely as possible, such as the elephant in the first image and the person in the second image. Additionally, the detector can find some missing objects, e.g . the strange and incomplete dining table in the third image. Further, with the foundation model, the detector can correctly classify the cell phone rather than a handbag in the second image and the toy (the toy is not the class in COCO) rather than bottles in the fourth image.

C Do Foundation Models with Registers Further Improve Frozen-DETR?

Registers [15] are used in modern ViTs for mitigating artifacts, which are also helpful for our FrozenDETR. Since this work only releases the checkpoint for DINOv2, the following experiments are

conducted on DINOv2 and DINOv2 with registers (DINOv2-reg). In the Table 15, we find that using DINOv2 as the foundation model can even get better results than using CLIP, which is different from Table 2. We hypothesize there are two reasons: First, DINOv2 has both global-wise and token-wise pre-training pre-tasks. Thus the patch tokens are more informative. Further, DINOv2 ViT-L is distilled from ViT-giant, which equals a larger foundation model. Thus equiping the detector with DINOv2 gets higher performance. Further, we find that DINOv2-reg can mitigate artifacts in DINOv2 and further improve the performance.

D Limitations and Broader Impacts

Limitations. This work utilizes the image understanding ability in frozen foundation models. However, these models are trained on nature images and may not perform well in many challenging scenarios, e.g ., medical images. Although Frozen-DETR enjoys an asymmetric input size, which greatly reduces computation costs, it still slows down the detector. Distilling Frozen-DETR to a standard detector may solve the problem and preserve its high performance.

Broader Impacts. This work improves the existing SOTA DETR-like detectors, which can be applied to automatic driving systems and many other downstream tasks.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The motivation in the paper is clear and the findings can match the motivation completely.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are discussed in Section 5 of the main text.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper is an application in object detection and no theoretical result is included.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide experiment details at the beginning of every subsection in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The method proposed in this paper is simple and straightforward. We have provided detailed configurations to conduct the experiments. And we will release all the code and models upon acceptance.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See the beginning of each subsection in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Error bars are not reported because it would be too computationally expensive.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See the beginning of each subsection in Section 4 and the experimental result tables.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We follow the NeurIPS Code of Ethics strictly.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The broader impacts are discussed in Section 5 of the main text.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?

Answer: [No]

Justification: There is no obvious risk.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite their papers correctly.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We will clean the code and provide detailed instructions.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Frozen DETR, which leverages frozen foundation models as feature enhancers to improve the performance of the DETR object detection framework. By integrating feature maps from models like CLIP into the pyramid feature maps and feeding them into the encoder, Frozen DETR enriches the contextual information of objects within the original pyramid features, thereby enhancing DETR's performance.\n\nSTRENGTHS:\n1. The paper use fundation models as plug-and-play modules, which is easy to apply on different detectors and enhance the performance significantly.\n2. The paper conducts extensive experiments and comparisons with the SOTA methods on different datasets to show the effectiveness and robustness.\n\nWEAKNESSES:\n1. The use of foundation models increases inference costs. It would be beneficial to provide a detailed analysis of the computational cost introduced by these models.\n2. The performance improvement appears to diminish when using more advanced models like Co-DETR, especially during extended training periods such as 24 or 36 epochs. An explanation for this phenomenon would be helpful.\n3. The method seems applicable to other SOTA models, such as [1,2,3]. A comparison with these models would strengthen the evaluation of the proposed approach.\n\n[1] Zhao, J., Wei, F., & Xu, C. (2024). Hybrid Proposal Refiner: Revisiting DETR Series from the Faster R-CNN Perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 17416-17426).\n\n[2] Zhao, C., Sun, Y., Wang, W., Chen, Q., Ding, E., Yang, Y., & Wang, J. (2024). MS-DETR: Efficient DETR Training with Mixed Supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 17027-17036).\n\n[3] Wang, Y., Li, X., Weng, S., Zhang, G., Yue, H., Feng, H., ... & Ding, E. (2024). KD-DETR: Knowledge Distillation for Detection Transformer with Consistent Distillation Points Sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 16016-16025).\n\nQUESTIONS:\n1. Why is the performance (47.0 AP) of the Co DINO baseline in Tables 12 and 13 inconsistent with the main paper, which reports 52.1 AP?\n2. Previous work [1] noted that features from CLIP's vision encoder can have artifact problems, potentially resulting in noisy feature maps. Does this issue influence the performance of detector? If so, would applying the methods from [1] help mitigate these artifacts and improve performance?\n\n[1] Darcet, T., Oquab, M., Mairal, J., & Bojanowski, P. (2023). Vision transformers need registers. arXiv preprint arXiv:2309.16588.\n\nLIMITATIONS:\n1. As mentioned in the paper, the performance gains heavily depend on the quality and characteristics of the pre-trained foundation models. Variations in these models could lead to inconsistencies and affect the reliability of the results.\n2. Using foundation models increases inference inefficiency. Solutions such as knowledge distillation might help mitigate this issue and improve efficiency.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper incorporates frozen foundation model backbones into DETR pipelines. Specifically, the paper concatenates the output of the class token of the foundation models with the query vector in the decoder. Also, it concatenates the patch tokens with the feature pyramid from the Image backbone to improve accuracy.\n\nSTRENGTHS:\n1. The paper is well written.\n2. The major novelty is fusing features of the frozen foundation models with backbone output and the query vectors in the decoder.\n3. The idea of using reduced resolution for the foundation model is engaging in a sense to reduce the overall compute.\n\nWEAKNESSES:\nDespite the paper's merits, it also has weaknesses:\n\n1. The idea of feature fusion seems incremental from the perspective of technical contributions. By looking at the design, the feature fusion of the frozen foundation model is the only principle claim of the paper. Apart from that, I could not see any other piece of substantial advancement.\n\n2. Frozen-DETR is only tested with DINO, which has huge computational costs due to dense multiscale feature utilization in the encoder. To demonstrate the utility of the presented idea, there should be results with affordable DETRs, e.g. DN-DETR, Conditional-DETR, DAB-DETR, Anchor-DETR, IMFA -DETR, Deformable-DETR etc. These are the pivotal works in this area, and hence, evaluation of the proposed Frozen-DETR pipeline is a must for this paper, given its limited technical contributions.\n\n3. I am more concerned about the applicability of this method in the real world, considering its weaker accuracy improvement tradeoffs vs runtime. By looking at Tables 12 and 13, changing the image resolution does not aggressively improve the accuracy. For example, in Table 12, doubling the resolution of the foundation model halves the speed where the original speed is already very low while the accuracy improvement is 1.8AP. Similar effects are seen in Table 13. Due to this reason, the weakness in point 2 (see above) must be addressed.\n\nQUESTIONS:\n1. Include the results of pivotal DETR methods instead of only one DINO, which is computationally heavy.\n2. Provide detailed runtime of each model to assess better the work given limited technical contributions.\n\nIf this is resolved, I am open to adjusting my score.\n\nLIMITATIONS:\nPlease refer to the weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on enhancing the performance of query-based object detection models. By inserting a foundation model into the DETR framework and treating it as a plug-and-play module instead of a backbone, the performance of query-based detectors can be significantly improved. The detection performance of DETR (DINO) on COCO is substantially enhanced by inserting patch tokens into the DETR encoder, and class tokens into the DETR decoder. Moreover, since the inserted CLIP is frozen, smaller detectors can now be equipped with larger foundation models to boost efficiency. The authors have conducted extensive ablation studies to demonstrate the effectiveness and justifiability of the proposed method.\n\nSTRENGTHS:\n1. Through a decoupled design, the method can accept asymmetric input sizes, which greatly reduces computational load and allows smaller detectors to be paired with larger foundation models while maintaining an acceptable computation burden.\n\n2. The experiments are very comprehensive, essentially validating every design choice and modification proposed, and thoroughly ablating multiple feasible schemes for inserting CLIP into the model.\n\n3. The results include large vocabulary and open vocabulary tests, demonstrating the advantages of utilizing the frozen CLIP.\n\nWEAKNESSES:\nIn the experiments presented in Tables 1 and 2, the detector utilizes an R50 backbone, but most of the inserted foundation models are ViT-Ls. This design introduces a larger image encoder to provide extra information, thereby effectively creating a model ensemble to enhance the knowledge of the R50 backbone. However, as the size of the detector backbone increases— for instance, when switching from R50 to Swin-Large or even ViT-Large—does this approach of inserting a frozen CLIP still lead to noticeable improvement? The paper only provides results for the largest backbone (Swin-Base) in Table 6. I believe that more results on larger backbones is needed for further discussion to verify the potential of this method.\n\nQUESTIONS:\nIn the Supplementary Material's Table 13, the authors examine the impact of varying model sizes of the foundation model on Co-DINO. I am still curious whether the R50-CLIP would boost the performance of an R50 backbone detector. Does this increase necessitate that the model size of CLIP backbone surpasses that of the detector backbone?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores using frozen vision foundation models to enhance object detection performance without fine-tuning. The authors demonstrate that foundation models, although not pre-trained for object detection, can significantly improve detection accuracy by leveraging their high-level image understanding capabilities. This is achieved by using the class token from the foundation model to provide a global context and the patch tokens to enrich semantic details in the detection process. The proposed Frozen-DETR method boosts the state-of-the-art DINO detector's performance on the COCO dataset, achieving notable improvements in average precision (AP). The method also shows strong performance in large vocabulary settings and open-vocabulary detection, highlighting its robustness and versatility.\n\nSTRENGTHS:\n+ This paper proposes a relatively novel approach for leveraging visual foundation models for object detection.  \n+ The proposed method achieves promising results and outperforms other query-based object detection models.  \n+ The whole paper is easy to follow.\n\nWEAKNESSES:\nOverall I think this paper is interesting, their proposed method of using vision foundation models as additional knowledge is fair and relatively novel. One minor concern may be that, it is hard to say whether it is desirable to apply this relatively complicated framework instead of directly leveraging the pre-trained foundation models in real practice, but the improvement is solid based on the current evaluations. \n\n**A few suggestions on writing:**\n\n- I personally suggested splitting/reorganizing (e.g. one question + corresponding insight or one paragraph) the last paragraph of the introduction. The combination of questions and key insights is distractive.  \n- Figure 3 itself is hard to follow. Adding more details in the caption may help. (So as a few other figures/tables)\n- It's better to bold the best results in tables. \n- One minor comment: maybe use different notations for DINO (the detection baseline) and DINOv2 (the self-supervised encoder).\n\n\n\n**Minor questions/comments:** \n\n- l37-38, EVA-CLIP-18B is equipped with a ViT rather than R50. The version of CLIP using R50 as the backbone lagged behind. This evidence cannot support the claim that the improvement is from large-scale training.\n- l55: Is DINO the SOTA query-based detector? (refer to https://paperswithcode.com/sota/object-detection-on-coco for example)\n- The setup of Table 1 is a little unclear, is the backbone tunable? I assume they are not. Then, the comparison is slightly unfair -- the backbones are frozen for the top lines, and there are additional trainable parameters in the encoder backbone. In other words, if we finetune the encoder backbone of Table 1, will the gap between the two variants be even smaller, which questions the current design? \n- In Table 2, DEiT-III has comparable results with CLIP, how will this backbone perform for the main experiments?\n- One limitation of this approach is the running time. in Table 4, the authors reported the FPS, running time time has been 1.5x compared with the baseline model. However, if using the foundation model in a conventional way (i.e. directly using them as the backbone), the running time will not be affected in the inference time. Moreover, if using a ViT-based foundational model, how will the inference time and GPU memory change, since most of the recent powerful foundation models are equipped with ViT?\n\nQUESTIONS:\nSee the previous section.\n\nLIMITATIONS:\nThe authors discussed the limitations and the social impact at the end of the main paper. The limitation of this work may be that the visual foundation models are trained with natural images and they may not work well with images from other domains (e.g. medical images). There is no potential negative social impact from this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}