FineCLIP: Self-distilled Region-based CLIP for Better Fine-grained Understanding

Authors: dong  jing
xiaolong  he
yutian  luo
nanyi  fei
guoxing  yang
wei  wei
huiwen  zhao
zhiwu  lu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 05:05:32

Abstract: Contrastive Language-Image Pre-training (CLIP) achieves impressive performance on tasks like image classification and image-text retrieval by learning on large-scale image-text datasets. However, CLIP struggles with dense prediction tasks due to the poor grasp of the fine-grained details. Although existing works pay attention to this issue, they achieve limited improvements and usually sacrifice the important visual-semantic consistency. To overcome these limitations, we propose FineCLIP, which keeps the global contrastive learning to preserve the visual-semantic consistency and further enhances the fine-grained understanding through two innovations: 1) A real-time self-distillation scheme that facilitates the transfer of representation capability from global to local features. 2) A semantically-rich regional contrastive learning paradigm with generated region-text pairs, boosting the local representation capabilities with abundant fine-grained knowledge. 
Both cooperate to fully leverage diverse semantics and multi-grained complementary information.
To validate the superiority of our FineCLIP and the rationality of each design, we conduct extensive experiments on challenging dense prediction and image-level tasks. 
All the observations demonstrate the effectiveness of FineCLIP.

Main paper: 
 introduction: 
 Contrastive Language-Image Pre-training (CLIP) [18, 40] emerges as the foundational work in vision-language representation learning. By training on large-scale, noisy image-text pairs, CLIP aligns global image and text embeddings within a unified latent space, demonstrating remarkable successes across image-level tasks [1, 25, 61] , e.g. image classification and cross-modal retrieval.However, CLIP has shown notable limitations in understanding fine-grained details, such as identifying object attributes and their relationships [37, 41, 62, 70] . Especially, when applied to downstream tasks, CLIP struggles to extract valuable region representations from visual dense features, limiting its effectiveness in complex recognition scenarios [33, 72] . Recent works [60, 70] attribute this issue to the task domain shift: CLIP matches an image as a whole to text description but fails to capture fine-grained alignment between image regions and corresponding textual attributes.To address this problem, researchers have attempted to enhance fine-grained alignment using two primary strategies. The first strategy [27, 59, 66, 70] directly leverages CLIP to match image regions with template labels using large quantities of grounding annotations in a classification setting. However, the pre-defined template labels lack sufficient semantic diversity, restricting its generalization to open-world scenarios. The second strategy [55] proposes a uni-modal distillation scheme by aligning the region dense features of the trainable student model with the image-level representation of the corresponding image crops generated by the frozen teacher model. Although efficient, the frozen teacher model restricts the performance ceiling of the student model. Notably, both of the strategies disrupt the important semantic consistency of visual representations.In this work, we unify cross-modal regional alignment and uni-modal global-to-region guidance into a coherent framework. We present FineCLIP, an end-to-end universal vision-language framework that gains better fine-grained understanding by reasonably incorporating a multi-grained contrastive learning paradigm with a real-time self-distillation scheme. FineCLIP involves the following appealing designs: 1) In order to enrich the model with abundant and diverse fine-grained semantics, instead of using limited template labels in the classification setting, we build the regional contrastive learning paradigm using regions and corresponding text descriptions generated by advanced Large Vision-Language Model (LVLM). 2) To facilitate interactions between global embeddings of image region crops and corresponding region dense features for mutual guidance, we introduce a real-time self-distillation scheme that relies on and complements global contrastive learning. Instead of using the frozen teacher model for guidance, we use the trainable FineCLIP to independently teach itself, leading to improved performance as global representations are progressively refined during training.3) FineCLIP universally aligns visual embedding, visual dense features, and textual embedding into a unified latent space. With these design considerations, FineCLIP fully leverages available diverse semantics from fine-grained region descriptions and real-time optimal global representations, and boosts the interactions of multi-grained complementary information. When applied to downstream tasks, FineCLIP is capable of effectively handling both dense prediction using region dense features and image-level tasks using semantic-aligned global embeddings.Through extensive experimental evaluations, we show that FineCLIP surpasses previous arts on most dense prediction tasks and image-level tasks under fair comparison settings, demonstrating its effectiveness in both fine-grained understanding and semantic-aligned global representation. Furthermore, FineCLIP presents promising scaling ability, consistently showing faster performance improvements than other competing methods as we scale up the trainset.Our contribution is threefold: 1) We present FineCLIP, which clearly incorporates the multi-grained contrastive learning paradigm and the real-time self-distillation scheme to achieve better fine-grained understanding. 2) We develop an automated region-text data generation pipeline utilizing advanced LVLMs, demonstrating its effectiveness in providing valuable fine-grained semantics. 3) Extensive experiments on dense prediction and image-level benchmarks show that our FineCLIP consistently outperforms previous arts and exhibits promising scalability.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: We clearly claims the intuition and advantages of FineCLIP in our abstract and introduction. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
methodology: 
 2.1 Preliminary CLIP Architecture. CLIP is a dual-tower architecture composed of a vision encoder V (ViT [9] ) and a language encoder L (BERT [8] ). Given an image-text pair, CLIP outputs the visual [CLS] token v, visual dense features X corresponding to image patches, textual [CLS] token t and textual dense features of text tokens. The [CLS] tokens v and t serve as global image and text embedding, respectively. During the pre-training, the global contrastive loss is computed with v and t for instancelevel alignment. In downstream applications, the global embeddings v and t are crucial in image-level tasks such as image classification and image-text retrieval, whereas visual dense features X are vital for dense prediction tasks like object detection and semantic segmentation.Problem Definition. Our aim is to develop a comprehensive representation space where visual and semantic features are both globally and locally aligned, which contributes to create a robust vision-language model that can effectively address both image-level and dense prediction tasks.To achieve the goal, the model must satisfy two key requirements at both global and regional levels: 1) Given a text T that describes the content of an image I, the image embedding v should be matched to text embedding t. 2) Given a regional text T r describing the content of a specific region r within the image I, both the local visual feature p r , pooled from visual dense features X according to r, and region visual embedding v r of region crop I r should align with region text embedding t r of T r .A hat, a backpack and a book are on the table.ViT BERT Figure 1 : Overall architecture of FineCLIP. For simplicity, the diagram omits unused visual dense features of regions extracted by ViT and textual dense features generated by BERT. By integrating multi-grained contrastive learning as well as a real-time self-distillation scheme, FineCLIP aligns visual global embedding, regional dense features, and textual global embedding into a unified space, acquiring rich coarse and fine-grained knowledge from image-text and generated region-text pairs.Overview. As depicted in Figure 1 , FineCLIP adopts the same architecture with CLIP, which consists of the vision encoder V and the language encoder L, but employs more complex inputs and objectives.The input batch of FineCLIP contains image-text pairs {I i , T i } i≤B , region-text pairs {I r j , T r j } j≤M and a set of corresponding region coordinates {r j } j≤M , where B and M refers to the batch size and region count. Noted that the regions {I r j } j≤M are cropped from images {I i } i≤B according to region coordinates{r j } j≤M , and region texts {T r j } j≤M are generated by LVLMs. Recall the image and text processing of CLIP introduced in Section 2.1. For image-text pairs {I i , T i } i≤B and region-text pairs {I r j , T r j } j≤M , FineCLIP outputs the corresponding global imagetext embeddings {v i , t i } i≤B and {v r j , t r j } j≤M , respectively. Moreover, FineCLIP extracts regional visual representations {p r j } j≤M by pooling visual dense features {X i } i≤B according to region coordinates {r j } j≤M with RoIAlign [15] .Global Contrastive Learning. The global contrastive learning realizes the instance-level alignment, which enables FineCLIP to bolster multi-modal global embeddings and acquire rich coarse-grained knowledge. Initially, the cosine similarity between image embedding v and text embedding t is calculated asEQUATIONThe global contrastive loss forces FineCLIP to learn global image and text embeddings by maximizing the cosine similarity to the corresponding text and image embeddings, while minimizing the cosine similarity to other non-corresponding ones in the batch, i.e.EQUATIONwhere τ is the learnable temperature and initialized with 1e -2.Real-time Self-distillation Scheme. Since the global contrastive loss L GC supervises only global embeddings, it is poorly effective in improving local dense features.The distillation scheme [55, 60] , which transfers robust global representational capabilities to region features, emerges as a promising solution to this problem. Specifically, as shown in Figure 1 , our real-time self-distillation loss directly maximizes the cosine similarity between region features {p r j } j≤M pooled from image dense features and the visual representations of region crops {v r j } j≤M using the formulaEQUATIONSemantically-rich Regional Contrastive learning. Despite the remarkable enhancements through aforementioned designs, the model still lacks abundant fine-grained knowledge due to the utilization of coarse-grained training data.Intuitively, to enrich the model with fine-grained knowledge, it is crucial for the model to focus more on the specific content of image regions and learn more precise and detailed semantics. Therefore, we are motivated to implement regional contrastive learning that operates on the level of regions and related descriptions. As shown in Figure 1 , we leverage the advanced LVLM to generate high-quality region descriptions with diverse fine-grained semantics.The regional contrastive loss compels FineCLIP to learn pooled visual region features and text embeddings by maximizing the cosine similarity between matching text and region pairs, while minimizing the similarity with non-matching pairs in the batch, which is defined as follows ).L RC = - 1 2M M i=1 (The reason for utilizing pooled visual region features instead of image embeddings of region crops is two-fold. 1) Supervising pooled region features directly improves model's capability to extract valuable local dense features. 2) Enhanced by attention mechanisms in ViT, pooled region features encompass a border perception of entire image content compared to image embeddings of regions, which facilitates a deeper semantic understanding.Learning Objective. As depicted in Figure 1 , the learning objective of FineCLIP incorporates the above three components. Global contrastive loss L GC works for enhancing representation capabilities and semantic consistency of global visual embeddings, while the self-distillation loss L SD is designed to transfer the real-time strong representation capability of global embeddings to local visual features. Additionally, regional contrastive loss L RC is applied to enrich the FineCLIP with fine-grained knowledge and further improve its local features. The learning objective isEQUATIONwhere the λ and γ are hyper-parameters. As a result, FineCLIP fully leverages available diverse semantics and boosts the interactions within multi-grained complementary information.3 ExperimentQuestion: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We have discussed the limitation of FineCLIP in Section A.7.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
ablation study of fineclip: 
 Experiment Settings. In our ablation experiments, we train FineCLIP using 8×A800 GPUs on train2017 split of COCO dataset [30] , which includes approximately 118K human-annotated image-text pairs along with 970K region-label pairs. To provide abundant fine-grained knowledge, we replace labels provided by COCO with region descriptions generated by BLIP-2 [26] . FineCLIP is initialized by ViT-B/16 with default input image size of 224 × 224 and corresponding BERT from EVA-CLIP [49] . We train FineCLIP for 10 epochs using AdamW [32] optimizer with the batch size of 32 per GPU, the learning rate of 1e -5, and the weight decay of 0.1. The coefficients λ and γ in learning objective are both set to 1. In all experiments, we freeze the language encoder L to reduce computational overheads and improve training stability.Using the COCO val2017 split, we test FineCLIP designs on the box classification task with pooled region features and image-level retrieval tasks using global embeddings. We report the Top1 and Top5 mean accuracy of all annotated boxes in the box classification task, and the R@1 accuracy of image-to-text and text-to-image retrieval tasks as evaluation indicators. As shown in Table 1 (rows) #2 and #5, L RC significantly improves the model's region feature extraction (+5.5 of Top1 and +7.5 of Top5 mean accuracy on box classification) and slightly enhances retrieval performance. This improvement is attributed to the introduction of region-text pairs, which provide abundant fine-grained knowledge, and the positive effects of L RC on boosting local features. This result also indicates that, despite the possible noise in the region-text pairs generated by BLIP-2, they still offer valuable information for learning region representations. Ultimately, combining all three components enables FineCLIP to achieve optimal classification performance and competitive retrieval results, as demonstrated in Table 1 (row) #6. Ablation of Region Proposal Methods. Regional distillation and contrastive learning are highly sensitive to the quality of region proposals. To assess their impact on the performance of FineCLIP, we evaluate four different region proposal methods: manual annotations from COCO, FastSAM [69] , RPN [43] , and YOLOv9 [51] . The corresponding results are presented in Table 2 . These findings highlight three key insights: 1) Automated region proposals perform comparably to manually annotated high-quality regions. 2) More region proposals do not necessarily improve performance.Although FastSAM generates the most proposals, but they appear overly cluttered upon manual inspection, resulting in suboptimal model performance.3) RPN provides a moderate number of region proposals with satisfactory and balanced performance. YOLOv9, which focuses on specific object categories, produces fewer but more precise proposals, leading to the best box classification performance of FineCLIP. Ablation of Region Annotation Methods. We also explore the impact of region annotation methods on FineCLIP's performance. In addition to manual annotation, two main approaches are used for generating region textual descriptions. The first is a rule-based method [70] , which selects the region concept from a predefined concept pool based on similarity scores and integrates it into a description template. The second strategy leverages LVLM for region annotation.Specifically, we annotate the boxes in COCO train2017 split using the rule-based method, BLIP-2 [26] , and InternLM-XComposer [67] , and evaluate their impact on FineCLIP's performance. For LVLMs, we use the prompt: "Describe this image in one sentence." As shown in Table 3 , LVLMs outperforms rule-based method, highlighting the effectiveness of LVLMs in generating valuable fine-grained knowledge. Notably, BLIP-2 provides the greatest improvements to FineCLIP. Following the evaluation setting in Subsection 3.1, we compare FineCLIP with three closely related approaches: CLIP [40] , RegionCLIP [70] , and CLIPSelf [55] . To ensure a fair comparison, all methods adopt the ViT-B/16 as backbone and input images of 224 × 224 resolution. As presented in Table 4 , FineCLIP demonstrates the most improvement over Pre-trained CLIP in both dense feature extraction and global representation. In contrast, while RegionCLIP and CLIPSelf achieve moderate gains in box classification tasks, they struggle to maintain the important visual-semantic consistency.Additionally, we report the time overhead and GPU memory usage of the competing methods during training on COCO train2017 split. Due to FineCLIP's incorporating of the multi-grained contrastive learning paradigm and the self-distillation scheme, it requires comparatively higher GPU memory usage. Nevertheless, it's worth highlighting that the per-epoch training time for FineCLIP (11 minutes) is only marginally longer than that of CLIPSelf (10 minutes) and RegionCLIP (9 minutes). Therefore, the training time for FineCLIP remains well within acceptable limits.Data Preparation. In this subsection, we evaluate the performance of FineCLIP trained on a scaled dataset. We begin by constructing the trainset based on the Conceptual Caption dataset (CC3M) [47] , which comprises 3 million image-text pairs sourced from the internet. To meet training data requirements of FineCLIP, we follow a three-step process to create region-text pairs.Image Filtering: This step retains images with rich contents to facilitate the acquisition of clear and valuable regional proposals. Specifically, we filter out low-resolution images and those that fail to generate regions via the region proposal model. After this process, we retain 2.5 million high-resolution images from CC3M, referred to "CC2.5M".Region Proposal: Based on ablation results in Table 2 , we select YOLOv9 [51] to detect objects in images. This process yields 10.4 million high-quality regions (approximately four regions per image) and takes around 4.5 hours to complete.Region Annotation: According to results shown in Table 3 , we utilize the BLIP2-COCO-6.7B model to annotate region proposals, which takes approximately 12.5 hours. Zero-shot Comparisons. To investigate the impact of data scale on model performance, we sample three trainsets of varying sizes from CC2.5M: 100K, 500K, and 2.5M samples. We train FineCLIP and competing methods, including CLIP [40] , RegionCLIP [70] , and CLIPSelf [55] , using the official open-source code on these three trainsets. To ensure fairness, all methods involved in comparisons adopt ViT-B/16 as backbone and input images of 224 × 224 resolution. For each dataset, we train these models for 6 epochs and then evaluate their zero-shot performance on COCO benchmark, using the same metrics of ablation study in Section 3.1.In Figure 2 , we present the accuracy curves of four methods on three tasks as the dataset scales up, with detailed quantitative results shown in Appendix Table 11 . Overall, our FineCLIP outperforms the other three competing methods in all cases. Specifically, in terms of fine-grained understanding, as shown in Figure 2 (a), FineCLIP significantly surpasses other methods in the box classification task, with a remarkable +10.4 mAP over CLIP and +8.0 mAP over CLIPSelf. Notably, as the dataset size increases, FineCLIP's performance continues to grow rapidly, showing promising scalability, whereas the growth rates of RegionCLIP and CLIPSelf slow down, indicating that their training gradually converges to a relatively low performance level.As for the evaluation of image-text alignment, according to Figure 2 (b) and (c), FineCLIP even surpasses CLIP in retrieval tasks, demonstrating that enhanced local representation and the acquisition of fine-grained knowledge contribute to more robust global embeddings. In contrast, RegionCLIP and CLIPSelf fail to maintain semantic consistency in visual embeddings, with their performance deteriorating as the trainset size increases.Visualization Results. Figure 3 presents the visualized attention maps of our FineCLIP on images responding to complete sentences or individual words. We can see from Figure 3 (a)-(c) that FineCLIP comprehends sentence semantics and identifies related elements, even tiny objects like a "man" or irregularly shaped items such as a "kite." Figure 3(d)-(e) shows that FineCLIP can well locate different objects within the same image. Interestingly, FineCLIP can also capture abstract concepts such as "looking into the distance" in Figure 3 (c)) and actions. For instance, when recognizing "riding", FineCLIP focuses on both the rider and the horse, while for "watching soccer", it highlights human faces and the soccer ball on the ground. These results collectively indicate that FineCLIP effectively learns to understand fine-grained semantics.Open-Vocabulary Object Detection. To evaluate whether the improved fine-grained understanding learned with FineCLIP translates to tasks requiring fine-grained localization, we serve FineCLIP as the backbone for open-vocabulary object detection. Following the previous work [55] , we build open-vocabulary object detectors based on F-ViT architecture, which is a two-stage detector baseline built on frozen CLIP ViTs. Considering that the input resolution has a significant influence on detection performance, to ensure the comparison fairness, we utilize the checkpoints of FineCLIP and competing methods trained on CC2.5M with input image size of 224 × 224 for ViT-B/16 and 336 × 336 for ViT-L/14 to initialize the F-ViT. In training hyper-parameters, we employ AdamW "A man with a red helmet was riding a horse.""riding" "red helmet" (d) "A guy is surfing." "A man is flying kite." "Two men sat on a bench in the grass looking into the distance.""A black man in a white shirt is watching soccer."(e) "black man" "watching soccer" [20] ViT-B/16 30.2 -41.5 RO-ViT [20] ViT-L/16 33.0 -47.7 CFM-ViT [19] ViT-L/ 16 optimizer with batch size of 8, learning rate of 1e -4, and weight decay of 0.1. We train the models for 3 epochs on the OV-COCO [6] benchmark and 48 epochs on the OV-LVIS [13] benchmark.(a) (b) (c)For evaluation, we follow previous works [55, 70] Compared with cutting-edge CLIPSelf, our FineCLIP brings better improvements to baseline F-ViT on both benchmarks. Additionally, we also evaluate the detector trained on OV-LVIS on the validation split of COCO and object365 [46] v1 datasets, with results shown in Table 5c . FineCLIP surpasses CLIPSelf on the COCO benchmark and achieves similar performance on the Object365 benchmark. All these results demonstrate the effectiveness of our FineCLIP in fine-grained understanding.Open-Vocabulary Semantic Segmentation. Next, we explore the performance of FineCLIP when applied to the open-vocabulary semantic segmentation task. Following the previous work [55] , we build the segmentation model based on CatSeg [7] , which utilizes the visual dense features of CLIP ViTs (ViT-B/16 and ViT-L/14 from OpenAI) with a cost-aggregation module. To ensure comparison fairness, we train FineCLIP and CLIPSelf models, which are initialized with pre-trained OpenAI CLIP [17] , on CC2.5M with the same input image resolution of 384×384 for ViT-B/16 and 336×336 for ViT-L/14, and then replace the backbone of CatSeg with FineCLIP or CLIPSelf ViT for the following segmentation fine-tuning. We fine-tune segmentation models on COCO Stuff [4] and evaluate them on ADE20K [71] and PASCAL Context [34] dataset using mean IoU (mIoU) and mean pixel accuracy (mAcc).As shown in Table 6 , FineCLIP brings non-trivial improvements to CatSeg across most evaluation indicators, surpassing the enhancements provided by CLIPSelf. We observe that both FineCLIP and CLIPSelf cause a decrease in PC-59 mIoU, which may be attributed to the data distribution gap between CC2.5M and PASCAL Context. Furthermore, FineCLIP comprehensively improve the mAcc performance of CatSeg, indicating the enhanced per-pixel classification performance. Flickr30k MSCOCO image-to-text text-to-image image-to-text text-to-image Methods R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 Zero-shot Image-Text Retrieval. We further evaluate FineCLIP on zero-shot cross-modal retrieval tasks using Flicker30K [39] and MSCOCO [30] , with the results presented in Table 7 . Previous works, such as SPARC [3] , PACL [35] , GLoRIA [16] , MGCA [52] , and FILIP [58] , introduce fine-grained losses to extract token-level cross-modal alignments. The results of these methods, re-implemented using the same pre-training datasets (approximately 3.2 billion data points), architecture, and training steps, are taken from SPARC [3] . For fair comparisons, we train competing methods, initialized with pre-trained CLIP parameters, on CC2.5M for 4 epochs.Given that CC2.5M has in a limited data distribution, further training of pre-trained models on this dataset inevitably leads to more or less performance decay. Compared to the baseline CLIP, FineCLIP better maintains retrieval performance, demonstrating its effectiveness in enhancing global embeddings and capturing valuable semantics. In contrast, RegionCLIP and CLIPSelf struggle to maintain global embeddings for addressing retrieval tasks. Surprisingly, FineCLIP even outperforms most pre-trained models with fine-grained losses, strongly supporting the effectiveness of FineCLIP.4 Related WorkQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We will release the code and generated textual descriptions of regions soon. Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
fine-grained understanding in vision-language models: 
 Early works focused on learning from intensive human labels by training image classifiers [9, 14, 22, 48, 50] . These classifiers focus only on a limited range of objects, making it difficult to cover a wide range of semantics. CLIP [40] and its diverse variants [12, 18, 68] popularized learning general visual-language representations by pre-training on noisy large-scale datasets like LAION-400M [44] and LAION-5B [45] , which exhibit potent representation capabilities and exceptional generalizability.Despite the great achievements, CLIP model has shown weak alignment between regions and corresponding texts [37, 41, 62] . This problem can be roughly attributes to limitations of 1) CLIP loss which ignores the supervision of visual and textual dense features, and 2) brief coarse captions that are insufficient to allow the model in comprehending image details [10] . Recent works attempted to enhance CLIP's fine-grained understanding by building strong region-text alignment. One line of work leverages region-level annotations for vision-language pre-training [59, 64, 66] . For instance, GLIP [27] utilized large-scale human-labeled grounding data to align semantics at phase and region level, which achieved stronger performance on fully-supervised detection benchmarks. Region-CLIP [70] proposed to generate region descriptions by filling plausible concepts into pre-defined templates. Since the semantics of synthesized descriptions are limited by the pre-defined concept pool, RegionCLIP essentially models the finite category classification task. Although this approach brings benefits to downstream detection tasks, it still struggles to cover broad semantic diversity of open-world scenarios. Another remarkable work is CLIPSelf [55] , which proposed to distill the global representation capability of the frozen teacher model to dense feature extraction of the student model. While CLIPSelf successfully enhancing the local representations, the frozen teacher model limits the performance ceiling of the student model, which is not consistent with the intention of pre-training. A separate line of work explored incorporating losses between image patch and text token embeddings to learn representations encoding more fine-grained details [3, 16, 35, 52, 60] .The process of generating region-text pairs from images involves two steps: proposing regions and annotating the regions with texts. Common strategies for region proposal include random cropping, using Region Proposal Network (RPN) [43] or detectors [42, 65] to generate bounding boxes around objects, or leveraging segmentation models [24] such as SAM [21] . Region annotations can be obtained by expert manual labeling, synthesizing captions using traditional NLP techniques, or generating them with models. A notable work of the second way is Kosmos-2 [38] , which introduced a pseudo-labeling pipeline that utilizes the pre-trained GLIP [27] to automatically generate fine-grained pseudo-labels of region boxes. In the era of large models, recent LVLMs [2, 31, 36, 53] have demonstrated impressive capabilities of visual understanding, instruction-following and generalization. By setting prompts, users can control the characteristics of outputs, such as length and writing style, to obtain high-quality responses, making it suitable for image fine-grained annotation. 
conclusion: 
 In this work, we present FineCLIP, a coherent framework that unifies cross-model multi-grained alignment and uni-modal global-to-region guidance. FineCLIP effectively leverages diverse semantics from automatically generated regional data and enhances the interactions of multi-grained complementary information through a real-time self-distillation scheme. Extensive experiments demonstrate the superior performance of FineCLIP in fine-grained understanding tasks, including box classification, open-vocabulary object detection and segmentation, as well as in global representation tasks like image-text retrieval. We believe this study provides valuable insights into related fields. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss the broader impacts in Sention A.6.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]Justification: As mentioned in Section A.6, our work leverages open-source datasets and models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We already cite the original papers that produced the code package or dataset.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 More Ablation Study Ablation of ViT Backbones. We also explores the influence of using different ViT backbones on FineCLIP. As shown in Table 9 , when FineCLIP employ ViT-L/14, which has large number of parameters, as the initialization, FineCLIP achieves significantly improvements on both types of tasks compared to using ViT-B/16. This result demonstrates the general applicability of our method to different ViT architectures. Effect Exploration of L SD on Zero-shot Setting. We conduct further validation under zero-shot setting by using trainset in different scales, with results shown in Table 10 . We observe an interesting trend: as the scale of trainset increases, the negative impact of L SD on retrieval tasks gradually diminishes. This means that L SD tends to have a positive impact on retrieval performance when FineCLIP is fine-tuned with larger-scale data. We provide the detailed zero-shot comparison results on COCO dataset in Table 11 . FineCLIP significantly outperforms the other three comparison algorithms in all scenarios and maintains the fastest performance growth as the dataset expands. Surprisingly, FineCLIP's performance in the retrieval tasks under the zero-shot setting even surpasses CLIP, indicating that FineCLIP can learn valuable regional semantics, and further enhance its global understanding capability. We also evaluate the performance of FineCLIP in open-vocabulary detection tasks when adopting settings similar to CLIPSelf. Following CLIPSelf, we train FineCLIP on COCO Train2017 split, using the region proposals from the trainset and region captions generated by BLIP2. After fine-tuning, FineCLIP's ViT parameters are used to initialize the F-ViT for downstream OV-COCO training. As FineCLIP is essentially a contrastive learning method, it needs a larger batch size than distillation methods for effective training. Due to the GPU memory limitations, we could only increase the training image resolution of FineCLIP to 640, which is much lower than 1024 used by CLIPSelf.The results in Table 12 show that despite FineCLIP's lower input image resolution, F-ViT+FineCLIP still outperforms F-ViT+CLIPSelf in AP base 50 (57.8 vs. 54.9) and AP 50 (51.4 vs. 50.4) metrics.K-Means Visualization of Dense Features. We present K-Means visualization of dense features generated by CLIP and FineCLIP ViT in Figure 4 . It can be seen that the dense features produced by CLIP are noisier and exhibit significant instability. In contrast, the results corresponding to FineCLIP are clearer and more consistent with the local semantic.In this subsection, we discuss the differences between FineCLIP and two closely related works. We depict the main architectures of CLIP, RegionCLIP, CLIPSelf and FineCLIP in Figure 5 .Relation to RegionCLIP. RegionCLIP proposes a regional contrastive learning paradigm based on synthesized region concepts for enhancing fine-grained understanding. However, these synthesized region concepts, generated by filling pseudo object label selected from a label pool into a pre-defined description template, lack sufficient semantic diversity. Considering that the negative textual samples for contrastive are object concepts that are not matched to the region but matched to other regions in the batch, it essentially models an object classification tasks with predefined finite categories. In contrast, our FineCLIP learns from diverse and detailed regional descriptions generated by LVLM, which are more representative of real-world scenarios.Relation to CLIPSelf. CLIPSelf builds a promising uni-modal global-to-local guidance scheme, relying on a pre-trained frozen teacher model. We think the main advantage of this scheme is the introduction of position inductive bias, enabling dense features to grasp local semantics around them, which is crucial for most downstream tasks. However, a notable shortcoming of CLIPSelf is that the frozen teacher model limits the performance of student model, and also discrpts the semantic consistency in visual features. By comparison, our FineCLIP fully leverages the model to teach itself with the help of global contrastive learning. While retaining this position inductive bias, FineCLIP enhances local features with real-time global embeddings enhanced by contrastive learning, thereby achieving more valuable local representations.Our FineCLIP contributes to develop a coherent learning framework that enhances CLIP's fine-grained understanding capabilities. With the growing adoption of transformers, FineCLIP is expected to better address various downstream tasks. To ensure a positive social impact, we conduct experiments using academic open-source datasets that do not involve personal privacy issues.FineCLIP has strict region-level data requirements. Additionally, due to the limitations in computational resources, we cannot afford to train FineCLIP from scratch on the billion-level dataset, making it difficult to fully realize its potential.In this work, we address fine-grained vision-language representation learning by introducing generated region-text pairs and developing a unified architecture. While FineCLIP has shown initial success, there is still significant room for improvement in region-text generation.Firstly, existing region proposal methods struggle to balance category richness and accurate segmentation. Our experiments with existing RPN, detectors, and segmentation models revealed that they either limit the number of categories or produce disorganized proposals. This results in suboptimal performance on tasks with numerous categories, such as the LVIS benchmark. One potential improvement is to train a more robust RPN on datasets with a greater variety of categories, enhancing region proposal quality. Secondly, due to computational constraints, we did not use the most powerful LVLM for annotations. A stronger LVLM would intuitively produce higher quality annotations.We hope these insights inspire future research. 


Decision: This paper presents a method called Fine VLIP for understanding fine-grained details. The authors proposed two novel losses to consider real-time self-distillation and regional contrastive learning. The validity of the proposed method is demonstrated through detailed experiments by showing superior performance to the previous works in most cases. Ablation study also demonstrates the importance of all the key components. The reviewers’ scores are consistently positive.

Here are just a few comments to improve this paper:
- The authors are advised to make the texts in the figures and tables much larger, e.g., as large as those in the main texts.
- “log” in (4) should be “\log”. Similarly, “exp” should be “\exp”.
- “We” in line 228 should be “we”.

Reviews: review: paper_summary: paper_summary: To address CLIP's limitations in understanding fine-grained details, the authors propose FineCLIP, a method for training CLIP-based architectures that proposes two novel losses, a real-time self-distillation loss and a regional contrastive loss. The regional contrastive loss is designed to encourage learning of fine-grained semantic features by querying a Large Vision-Language Model (LVLM) with region crops for a detailed description, then enforcing a contrastive loss between the corresponding text embedding and region's pooled visual embedding. The real-time self-distillation loss encourages pooled regional vision encoder features to be consistent with the vision encoder's embedding of region crops and is designed to prevent loss of global features during training with the regional contrastive loss. To assess the performance of FineCLIP, the authors conduct experiments on Open-Vocabulary Object Detection, Open-Vocabulary Semantic Segmentation and Zero-Shot Image-Text Retrieval and find FineCLIP performs favorably compared to the baselines trained and evaluated in the same setting. They perform ablation studies to assess the contributions of FineCLIP's losses, the region annotation method, and impact of the train dataset size.
strength_weakness: strength_weakness: 1. Trainability: It is difficult to tell how usable FineCLIP is without details on the time it takes to train a single epoch and the GPU memory the method uses. How big is the LVLM? Is it queried on the fly or ahead of time? How long does inference take? The authors should include this information for FineCLIP and CLIPSelf to demonstrate the method's ease of use during training and inference.
2. Analysis of results: The authors provide a short explanation that the mixed results of FineCLIP in Table 4 on PC-59 and ADE-847 are due to the data distribution gap between CC2.5M and Pascal Context, but it would be better to provide more of a semantic analysis in which classes are underperforming and why this is the case. This would help strengthen the analysis of FineCLIP's limitations.
questions: questions: 1. Under what conditions does adding $\mathcal{L}_{SD}$ help? It's confusing why all three losses in row 5 of Table 1 don't give the best retrieval performance.
2. In l. 26 what are the domains specifically in the "domain shift"?
3. In l. 48 what losses are happening during the pretraining stage?
4. In l. 108-109 it says "instance-level" alignment but does this instance refer to a single image not an instance of an object?
7. In Table 2, what is the time required for each region annotation method?
8. In section 4.2, the smallest dataset is 100k samples. However, often in the case of dense prediction tasks, there is a very limited amount of training data. Why did the authors choose to scale the dataset down rather than up and see how it affects performance?
9. Can the authors explain the performance discrepancy between CLIPSelf in Table 3 in the original paper [1] versus this work?

Minor Typos/Suggestions:
- l. 22 remove "been"
- l. 134 "learn" <- is this right?
- l. 344 "realize"
- l. 285 "detention" <- "detection"
- l. 228 "We" is capitalized
- l. 262 "quantitative"
- l. 272 "trainset set"
- l. 285 "detention"

[1] Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Xiangtai Li, Wentao Liu, and Chen Change Loy. CLIPSelf: Vision transformer distills itself for open-vocabulary dense prediction. In CVPR, 2024.
limitations: limitations: The authors have included a limitations section on the scalability of FineCLIP to larger datasets, but might improve this section with an analysis of how FineCLIP performs with smaller datasets, and a more detailed semantic analysis of when FineCLIP does not outperform previous methods.
paper_summary: paper_summary: The paper introduces FineCLIP, a novel vision-language model designed to enhance fine-grained understanding in image-text tasks. It addresses limitations in existing models like CLIP, which struggle with dense prediction tasks due to a lack of fine-grained detail comprehension. The authors propose two main innovations: A real-time self-distillation scheme that transfers representation capabilities from global to local image features, facilitating a deeper understanding of local image details. A semantically-rich regional contrastive learning approach using generated region-text pairs to boost local representation capabilities with fine-grained knowledge.
strength_weakness: strength_weakness: I understand that the author discusses the differences and experimental comparisons with existing methods such as RegionCLIP and CLIPSelf. However, in my opinion, the self-distillation scheme is a very common practice, and its contribution is not particularly novel. The semantically-rich regional contrastive learning paradigm is a significantly effective method for aligning image local regions and text. However, sending all segmented local images into the vision-language model (VLLM) to generate text descriptions does not seem very efficient, especially when scaled to larger datasets.

The paper could provide more details on the computational efficiency of FineCLIP, especially the training cost, and a comparison with an existing model (CLIPSelf).

Regarding the ablation study in Table 1, the author should further explain why using only $L_{SD}$ for supervision leads to model collapse. Additionally, the effect of using $L_RC$ alone should be supplemented.

In the open-vocabulary semantic segmentation task shown in Table 4, CatSeg+FineCLIP shows only a minor improvement compared to CatSeg+CLIPSelf. This does not sufficiently support the author's claim of fine-grained semantic alignment.

Importantly, if my understanding is incorrect, please correct me, and I will improve my score accordingly.
questions: questions: Minor  question

The authors highlight the 'real-time' capability of the self-distillation scheme. Could you please clarify the significance of this feature? Is real-time operation essential for the effectiveness of your method or does it offer specific advantages in the given context?
limitations: limitations: Yes, the authors addressed limitations, potential negative societal impact, and mitigation.
paper_summary: paper_summary: This paper attempts to overcome the problem of CLIP lacking fine-grained details when adapting to dense prediction tasks. It proposes a unified framework with three training losses: contrastive loss for global image-text pair, region alignment for region-region annotation, and self-distillation for image region embedding and region feature. By combining the three aspects, the FineCLIP shows its effectiveness in fine-grained understanding, at the same time maintains the global representation alignment.
strength_weakness: strength_weakness: The innovation somewhat combines the previous works regionCLIP and CLIPSelf on finetuning of CLIP, although this work shows evolved implementation details, such as adopting LVLMs to generate region annotations and self-distillation instead of frozen teacher.
questions: questions: 1. I noticed that in Sec.4.1 ablation study, the results in Tab. 1 are obtained by training FineCLIP on the train2017 split of the COCO dataset, while in Sec. 4.2 the training set is CC3M. With the same COCO validation set, training with CC3M shows worse performance on retrieval task (54.5 vs. 62.2, 40.2 vs. 47.6) but better top1 box classification result (50.7 vs. 48.4). Any idea about this result?
2. About region proposal and annotation, I’m curious about some details: 
- Are the region information prepared before the CLIP finetuning or real-time processing during the training?
- Will different region proposal methods have a big influence on the final results since it is mentioned that “after careful manual evaluation, we choose YOLOv9” in P6 L237? Maybe you can provide the details about the procedure in the appendix. 
- The details about "We use pixel count as the criterion and discard images that fail to propose regions in the following process." to choose CC2.5M.
3. CLIPSelf is an ICLR2024 accepted paper, not CVPR 2024. The reference of [57] is not correct.
4. The table head ("image-to-text" and "text-to-image") of Tab. 5 doesn't align well with the below contents.
limitations: limitations: N/A
paper_summary: paper_summary: This paper integrates three existing techniques in vision-language pre-training into a single end-to-end fine-tuning framework, referred to as FineCLIP:
- global contrastive: This aligns the global representation of an image with the text embedding.
- reginal contrastive: This aligns the pooled region-level representations, defined by the generated proposals, with the regional caption generated by a LVLM.
- regional self-distillation: This aligns the pooled region-level representations with the global representations of the corresponding local crops, following the approach of CLIPSelf [1]. Uniquely, it employs the updated vision encoder itself as the teacher.

In addition, the authors construct a new subset, “CC2.5M”, based on CC3M for further fine-tuning.

[1] CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction. ICLR 2024
strength_weakness: strength_weakness: __W1: limited practical impact.__ 

- As a direct successor to CLIPSelf, the performance on both OV-COCO and OV-LVIS benchmarks is significantly lower than the original baseline of CLIPSelf. For instance, FineCLIP achieves 29.8 AP$_{50}^{novel}$ on OV-COCO with ViT-B/16, while CLIPSelf reaches 37.6 with an even smaller dataset. I understand that they’re under different input resolutions, but why not directly increase the input resolution of FineCLIP instead of increasing the scale of the dataset?

- Moreover, a significant advantage of FineCLIP is its ability to preserve the power of image-level inference. However, according to Table 5, all the fine-tuning strategies cause a performance drop to some extent compared to the original pre-trained CLIP on all the image-level benchmarks.

__W2: The experiments are insufficient and incomplete.__
- Input resolution: Input resolution: Since CLIPSelf can consistently benefit from higher image resolution, a comparison with CLIPSelf under a relatively low resolution (224$\times$224) is unfair. I would suggest including the performance comparison with higher resolution under the original settings of CLIPSelf.

- The mean mask AP on rare categories (mAP$_r$) is missing for OV-LVIS.

__W3:The limitation of CLIPSelf stated in this paper remains unsolved.__
> line150: performance is capped by the teacher’s capabilities.

The supervision of self-distillation heavily relies on the quality of the image-level representation of the teacher (or the updated visual encoder of FineCLIP), but according to Table 5, the image-level performance suffers from a significant drop when compared to the frozen encoder. It’s somewhat unconvincing to me to draw the conclusion that the updated visual encoder can benefit the self-distillation process.

Perhaps the proposed strategy could potentially be beneficial to large-scale vision-language pre-training from scratch given the observations of fine-tuning on CC2.5M, but the current results of this paper are not very convincing to me. I am open to reconsidering my score if my concerns here are addressed.
questions: questions: - Most of my questions are outlined in the weaknesses section.

- Minor: Some detailed settings are missing. For instance, for Cat-Seg+CLIPSelf and Cat-Seg+FineCLIP, is the region defined by the generated proposals or patches (grids)?
limitations: limitations: Limitations are properly discussed.

Comments: title: Response to Reviewer Ask3
A new concern about region proposal and region annotation
Response to Reviewer 5tL8
Response to rebuttal
Response to Reviewer 5tL8
Response to Rebuttal
Thanks for the response of Reviewer oDkS
Thanks for the response of Reviewer Ask3
Thanks again for the additional response
Replying to Reviewer oDkS
Reseponse to rebuttal by authors
Thanks for the rebuttal
Replying to Reviewer rzQ8
Replying to Rebuttal by Authors
Additional experiments with small-scale trainset
Typo in my question 6
Further Clarification on Q2 and Q3
comment: Thank you for your thoughtful question. We'd like to address it from the following perspectives:

i) As shown in Table 6 of the attached PDF, we also evaluated FineCLIP under conditions similar to those used for CLIPSelf$^{[1]}$.
Specifically, we trained FineCLIP on COCO Train2017, using the region proposals provided by the dataset, with region annotations generated by BLIP2.
As stated in the BLIP2 paper$^{[2]}$, BLIP2 was not trained on the OV-COCO and OV-LVIS datasets, so there is no issue of label leakage.
In this fair comparison, FineCLIP outperforms CLIPSelf on both $AP_{50}^{base}$ and $AP_{50}$, even with lower resolution inputs.

ii) FineCLIP's strength lies in its ability to learn from training data more effectively.
As you noted, CLIPSelf relies on a pre-trained frozen teacher model, while RegionCLIP depends on pseudo-labels generated by CLIP.
We understand your concerns, but the inability of these two methods to fully leverage region-text data is their limitation, not our fault.
In contrast, FineCLIP not only distills region embeddings to local dense features but also acquires knowledge from training data through the multi-grained contrastive learning paradigm.

We hope the above responses could address your concerns. Please reach out if there is any confusion. Thank you very much.

Reference:

[1] CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction

[2] BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models.
With the help of well-trained Yolov9 and LVLM, FineCLIP obtains bounding boxes for regions and the corresponding labels (text descriptions) and uses them in the fine-tuning of the pre-trained model. These annotations most likely include the region-category pairs in the subsequent OVD task, regardless of the novel or base categories. Therefore, the problem is that using detectors and an extra strong LVLM to gather training data is unfair since the detector might be trained on the novel classes in the final OVD test set, and the extra LVLM, like BLIP2, will give the class information to the region proposals generated by the detector. Then, the CLIP is being trained with those classes using more direct supervision, which is against the idea of OVD detection. We are not sure how much the performance increase is from that supervision and how much from the improved regional awareness ability of CLIP. 

RegionCLIP and CLIPSelf may not have the same problem. RegionCLIP uses RPN to get regions, while it relies on CLIP itself to match region and text descriptions. CLIPSelf also generates region proposals, while it distillates the region embedding from CLIP itself to its dense feature, and there are no new text annotations involved. Therefore, for these two methods, there are no direct supervision annotations that are related to the OVD test set and added to the original training data (CC3M) in the pre-trained model fine-tuning stage.
Thanks for your feedback. We understand your concerns regarding the GPU memory usage. 

First,  we believe that using frameworks such as DeepSpeed or FlashAttention can significantly reduce the GPU memory consumption, which is a widely adopted strategy in the training of large models.

Also, as you have mentioned, contrastive learning is usually conducted in multi-GPU environments, which can easily meet FineCLIP's GPU memory requirements.

We hope this response could address your concerns. Thank you very much again.
My concern on semantic analysis has been addressed but my concern on the usability of the method has not been assuaged because of FineCLIP's large memory requirements in row 5 Table 1 of the rebuttal pdf. Such a large memory requirement may be prohibitory. One question I have is whether the authors believe it can be reduced.
Thank you for your response. We are pleased to see that your concerns have been addressed.

Regarding the multi-gpu training code, we have included it in the supplementary material and will open-source the code as soon as our paper is accepted.

We notice that you lowered the Rating. If there are any remaining questions or concerns, we'd be happy to address them. We look forward to your reply. Thank you.
Thanks to the authors for their comprehensive response. While my concerns about the semantic analysis of FineCLIP's performance and the performance at smaller dataset scales has been addressed, the computational efficiency of the method is still concerning: the memory requirement of the method may require a multi-gpu implementation and my hope is that the authors release code that supports this.
We sincerely thank you for raising your score. We greatly appreciate your constructive comments. We will carefully incorporate the key points of our discussion into the revision. Thank you very much again!
Thank you for your valuable feedback. We are pleased to have addressed the concerns you previously raised and would like to further discuss the new points you've mentioned.

Concern 1: The Influence of Region Proposals.

As shown in Table 5 of the attached PDF document, the region proposal method indeed has a direct impact on FineCLIP's performance. However, as demonstrated in Table 8 in the Appendix of our paper, even with the least effective region proposal method, FineCLIP still significantly outperforms other competing methods.

Concern 2: Resource Consumption.

As shown in Table 1 of the attached PDF file, while FineCLIP does require high GPU memory usage, its training time is very similar to that of CLIPSelf and RegionCLIP.
Therefore, we believe the training time required for FineCLIP is acceptable.

Concern 3: FineCLIP's ability to pre-training from scratch.

The primary focus of our paper is to validate the feasibility of FineCLIP in common fine-tuning settings. The extensive experiments presented in the paper and during the rebuttal phase are sufficient to demonstrate FineCLIP's superiority. While we did not conduct experiments to pre-train FineCLIP from scratch, the scalability experiments in Section 4.2 provide insight into FineCLIP's potential in this regard. 

We understand and respect your objective evaluation of our paper. However, we believe that the existing experiments and discussions adequately demonstrate the contributions of our work. Notably, after further discussion, Reviewer oDkS and Reviewer rzQ8 have both raised their scores. Given that your previous concerns have been addressed, we kindly request you to consider restoring your original score. We would greatly appreciate it.

We sincerely appreciate your constructive comments and look forward to your reply. Thank you once again for your understanding and support of our work.
I notice that the GPU memory usage for FineCLIP is indeed higher compared to the previous method, while the training time is acceptable. I'll raise my rating to a BA.
Thank you for your valuable feedback. Below are our responses to your additional questions.

Q1: Compared to using ViT-B/16 as the backbone, CatSeg+FineCLIP does not demonstrate the same significant advantage over CatSeg+CLIPSelf when using ViT-L/14 as the backbone.

A1: This is an insightful observation. We believe that one important reason for this is that CatSeg with ViT-L/14 already exhibits much better performance than with ViT-B/16, thereby making further improvements more difficult to achieve. Nonetheless, FineCLIP still provides more remarkable enhancements to CatSeg than CLIPSelf when using ViT-L/14.


Q2: Could the authors provide a clear overview of the resource consumption?

A2: We detail the training time cost and GPU memory usage in Table 1 of the attached PDF document. We fully understand your concerns. Due to FineCLIP incorporating the multi-grained contrastive learning paradigm and the self-distillation scheme, it requires relatively higher GPU memory usage. However, it is important to note that the per-epoch training time cost for FineCLIP (11 minutes) is very close to that of CLIPSelf (10 minutes) and RegionCLIP (9 minutes). Thus, the training time required for FineCLIP is entirely acceptable.

Furthermore, we also present the details of the data preparation in A2 of the common responses. The construction of CC2.5M trainset can be completed within a single day.


Q3: Could the authors briefly explain why FineCLIP shows more substantial improvement in open-vocabulary object detection tasks?

A3: Good question. We believe there are three main reasons.

i) The semantic information contained in visual dense features is crucial for performance in open-vocabulary object detection tasks. FineCLIP successfully enhances the accuracy and richness of the semantic representation of visual dense features through regional contrastive learning and self-distillation scheme.

ii) The CC2.5M trainset we constructed provides abundant and valuable fine-grained semantic knowledge.

iii) Compared to CLIPSelf, FineCLIP learns the knowledge from training data more effectively. While that CLIPSelf learns only from the region embeddings derived from a frozen teacher model, FineCLIP not only learns from region embeddings that can be improved during the training but also achieves fine-grained region-text alignment through regional contrastive learning.

Q4: Will the data and code be made publicly available?

A4: Yes, we will open-source the data, code and model weights immediately if our paper is accepted.

We hope the above clarifications could address your concerns. We are looking forward to your feedback if there is still any confusion. Thank you very much again for your insightful comments.
Thanks for the response. It addressed my questions. However, considering that the final results will be primarily influenced by the quality of region preparation, together with the concerns from Reviewer oDkS about resource consumption, Reviewer rzQ8 about the doubt of FineCLIP's ability to pre-training from scratch, I re-evaluated the limitation of the proposed method. I decided to lower the rating to 5, which shows that my evaluation of this work is reduced, but I have no problem with it being accepted.
The rebuttal has addressed most of my concerns. However, I still have concerns regarding the resource consumption of the proposed approach relative to its performance improvement. Specifically, I notice that CatSeg+FineCLIP does not demonstrate a significant advantage over CatSeg+CLIPSelf when using ViT-L/14 as the backbone. Could the authors provide a clear overview of the resource consumption? Additionally, could the authors briefly explain why FineCLIP shows more substantial improvement in open-vocabulary object detection? Will the data and code be made publicly available?
We sincerely appreciate your positive feedback and insightful comments. We will carefully incorporate the key points of our discussion into the revision. Thank you very much again.
Thanks for the response. Most of my concerns have been adequately addressed, but I don't fully agree with A4. For FineCLIP, the self-distillation mechanism relies on high-quality supervision from $[CLS]$, which may not be compatible with an initialized visual encoder for pre-training from scratch. Anyway, I will raise my score to 5, as the method appears to be effective for fine-tuning.
Thank you for your suggestion. Following your advice, we carefully conduct additional experiments to investigate the performance of FineCLIP with a small-scale trainset. Specifically, we randomly select 1,000 data samples from CC2.5M as our trainset and evaluate the performance of methods on the COCO val2017 split. The results are summarized in the Rebuttal Table 1 below.

Rebuttal Table 1: Zero-shot performance comparisons with 1K training samples.

| #   | Method           | Box |        Classification           | Retrieval |           |
|-----|------------------|--------------------|------------------|-----------|-----------|
|     |                  | Top1               | Top5             | I2T       | T2I       |
| 1   | CLIP             | 30.2               | 52.3             | 51.5      | 35.8      |
| 2   | RegionCLIP       | 35.0               | 62.1             | 19.4      | 13.1      |
| 3   | CLIPSelf         | 37.9               | 66.2             | 35.2      | 28.4      |
| 4   | FineCLIP (Ours)  | 42.0               | 66.8             | 49.2      | 36.0      |

As shown in the table, FineCLIP effectively enhances CLIP's fine-grained understanding capability, even with a very small amount of training data (1K samples). Moreover, FineCLIP still outperforms other competing methods by providing the most significant improvement in box classification task while maintaining strong retrieval ability.

We hope these experiments could address your concerns. We greatly appreciate your insightful feedback and are happy to provide further clarification if needed. Thank you very much again.
Sorry, my question was not clear. 100k samples is very large for a dense prediction dataset. E.G. Cityscapes has <3k finely labeled samples. Do you have any insight into how FineCLIP performs when there are only several thousand samples in the dataset?
We greatly appreciate your constructive comments and suggestions. We'd like to address your concerns regrading the image-level performance of FineCLIP with further clarification.

It should be noted that Table 5 presents the performance of models on **zero-shot** retrieval benchmarks.
These zero-shot benchmarks (i.e., Flickr30k and MSCOCO) involve the evaluation data that is somewhat out-of-distribution w.r.t. the trainset.
As you have pointed out, we had also noticed that the domain gap between the trainset and evaluation data might degrade the model's zero-shot performance when we were preparing this paper submission.

Importantly, as shown in Figure 2, we find that data scale is crucial for zero-shot capabilities, and both CLIP and FineCLIP demonstrate excellent scalability.
In this paper, we thus chose to use the largest dataset within our capacity, CC2.5M, as the trainset to preserve the model's zero-shot ability as much as possible.
According to Table 5, the performance drop for CLIP or FineCLIP is not that significant.
Given the scalability of CLIP and FineCLIP demonstrated in Figure 2, we can reasonably infer that their zero-shot image-level performance could be further improved and even surpasses that of the pre-trained CLIP by increasing the trainset scale.

Moreover, it should also be noted that Table 5 does not show the model's performance on in-domain data.
According to Table 8 of the Appendix, when models are trained and evaluated on COCO, FineCLIP's performance on retrieval tasks significantly surpasses that of pre-trained CLIP. 
This result indicates that FineCLIP greatly enhances image-level representation on in-domain data, enabling it to better benefit the self-distillation process than pre-trained CLIP.

We hope that the above clarification could address your concerns. We are looking forward to your feedback if there is still any confusion. Thank you very much again.