FineCLIP: Self-distilled Region-based CLIP for Better Fine-grained Understanding
Authors: Dong Jing
Xiaolong He
Yutian Luo
Nanyi Fei
Guoxing Yang
Wei Wei
Huiwen Zhao
Zhiwu Lu
Venue: NeurIPS 2024 poster (2024)
Keywords: CLIP
Fine-grained Understanding
Primary area: machine_vision

Abstract: Contrastive Language-Image Pre-training (CLIP) achieves impressive performance on tasks like image classification and image-text retrieval by learning on large-scale image-text datasets. However, CLIP struggles with dense prediction tasks due to the poor grasp of the fine-grained details. Although existing works pay attention to this issue, they achieve limited improvements and usually sacrifice the important visual-semantic consistency. To overcome these limitations, we propose FineCLIP, which keeps the global contrastive learning to preserve the visual-semantic consistency and further enhances the fine-grained understanding through two innovations: 1) A real-time self-distillation scheme that facilitates the transfer of representation capability from global to local features. 2) A semantically-rich regional contrastive learning paradigm with generated region-text pairs, boosting the local representation capabilities with abundant fine-grained knowledge. 
Both cooperate to fully leverage diverse semantics and multi-grained complementary information.
To validate the superiority of our FineCLIP and the rationality of each design, we conduct extensive experiments on challenging dense prediction and image-level tasks. 
All the observations demonstrate the effectiveness of FineCLIP.

FineCLIP: Self-distilled Region-based CLIP for Better Fine-grained Understanding
Dong Jing" Xiaolong He* Yutian Luo', Nanyi Fei?
Guoxing Yang', Wei Wei Huiwen Zhao?, Zhiwu Lu t1 GGaoling School of Artificial Intelligence, Renmin University of China MetaaBrain AGI Lab, Shanghai, China R&& Management Department, Honor Device Co., Ltd (jingdong98, xiaolonghe, Luzhiwu) @ruc edu. cn
Abstract
Contrastive Language-Iman Pre-training (CLIP) achieves impressive performance tasks like image classification and image-text retrieval by learning on large- scale image-text datasets. However, CLIP struggles with dense prediction tasks due to the poor grasp of the fine-grained details. Although existing works pay attention to this issue, they achieve limited improvements and usually sacrifice the important visual-semantic consistency. To overcome these limitations, we propose FineCLIP, which keeps the global contrastive learning to preserve the visual- semantic consistency and further enhances the fine-grained understanding through two innovations: 1) A real-time self-distillation scheme that facilitates the transfer representation capability from global to local features. 2) A semantically regional contrastive learning paradigm with generated region-text pairs, boosting the local representation capabilities with abundant fine- tine-grained knowledge. Both cooperate to fully leverage diverse semantics and multi-g tiitingg complementary information. To validate the superiority of our FineCLIP and the rationality of each design, we conduct extensive experiments challenging dense prediction and image-level tasks. All the observations demonstrate the effectiveness of FineCLIP.
Introduction
Contrastive Language-Image Pre-training (CLIP) 18 40 emerges as the foundational work in lision-language representation learning. By training large-scale, noisy image-text pairs, CLIP aligns global image and text embeddings within a unified latent space, demonstrating remarkable successes across image-level tasks
25 61. e.g. image classification and cross-modal retrieval.
However, CLIP has shown notable limitations in understanding fine-grained details, such as identify- ing object attributes and their relationships [37
41 62 770 Especially, when applied to downstream tasks, CL.IP struggles to extract valuable region representations from visual dense features, limiting its effectiveness in complex recognition scenarios [33 72 Recent works (60, 70) attribute this issue to the task domain shift: CLIP matches an image as whole to text description but fails to capture fine-gra fine-grained alignment between image regions and corresponding textual attributes.
To address this problem, researchers have attempted to enhance fine- grainea alignment using two primary strategies. The first strategy [27 [5 66, 70) directly leverages CLIP to match image regions with template labels using large quantities of grounding annotations in classification setting. However, the pre-defined template labels lack sufficient semantic diversity, restricting its
"Equal Contribution
t Corresponding Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

generalization to open-world scenarios. The second strategy [55] proposes a uni-modat distillation scheme by aligning the region dense features of the trainable student model with the image-level representation of the corresponding image crops generated by the frozen teacher model. Although efficient, the frozen teacher model restricts the performance ceiling of the student model. Notably, both of the strategies disrupt the important semantic consistency of visual representations.
In this work, we unify cross-modal regional alignment and uni-modal global-to-region guidance into a coherent framework. We present FineCLIP, an end-to-end universal vision-language framework that gains better fine- aneegrained understanding by reasonably incorporating a multi- multi-grained contrastive learning paradigm with a real-time self-distillation scheme. FineCLIP involves the following ap- pealing designs: 1) In order to enrich the model with abundant and diverse fine-grained semanties, instead of using limited template labels in the classification setting, we build the regional contrastive learning paradigm using regions and corresponding text descriptions generated by advanced Large Vision-Language Model (LVL.M). 2) To facilitate interactions between global embeddings of image region crops and corresponding region dense features for mutual guidance, we introduce real-time self-distillation scheme that relies on and complements global contrastive learning. Instead of using the frozen teacher model for guidance, we use the trainable FineCLIP to independently teach itself, leading to improved performance global representations are progressively refined during training. 3) FineCLIP universally aligns visual embedding, visual dense features, and textual embedding into a unified latent space. With these design considerations, FineCLIP fully leverages available diverse semantics from fine- ain-ggrined region descriptions and real tal-timee optimal global representations, and boosts the interactions of multi multi-grained complementary information. When applied to downstream tasks, FineCIIP is capable of effectively handling both dense prediction using region dense features and image-level tasks using semantic-aligned global embeddings.
Through extensive experimental evaluations, we show that FineCLIP surpasses previous arts on most dense prediction tasks and image-level tasks under fair comparison settings, demonstrating its effectiveness in both fine-grained understanding and semantic-aligned global representation. Furthermore, FineCLIP presents promising scaling ability, consistently showing faster performance improvements than other competing methods as we scale up the trainset.
We present FineCLIP, which clearly incorporates the multi qulti-grained contrastive learning paradigm and the real-time distiillaation scheme to achieve better fine-grained understanding. 2) We develop automated region-text data generation pipeline utilizing advanced LVLMs, demonstrating its effectiveness in providing valuable fine ane-grained semantics. 3) Extensive experiments on dense prediction and image-level benchmarks show that our FineCLIP consistently outperforms previous arts and exhibits promising scalability.
Our contribution is threefold: 1)
2 Methodology
2.1 Preliminary
CLIP Architecture. CLIP is a dual-tower architecture composed of a vision encoder V (ViT 9 and language encoder L (BERT B)). Given image-text pair, CLIP outputs the visual [CLS] token v, visual dense features X corresponding to image patches, textual [CLS] token t and textual dense features of text tokens. The [CLS] tokens and serve global image and text embedding, respectively. During the pre-training, the global contrastive loss is computed with v and I. for instance- level alignment. In downstream applications, the global embeddings v and crucial in image-level tasks such as image classification and image-text retrieval, whereas visual dense features X' are vital for dense prediction tasks like object detection and semantic segmentation.
Problem Definition. Our aim is to develop a comprehensive representation space where visual and semantic features are both globally and locally aligned, which contributes to create robust lasion-language model that can effectively address both image-level and dense prediction tasks.
To achieve the goal, the model must satisfy two key requirements at both global and regional levels: 1) Given a text T that describes the content of an image 7, the image embedding u should matched to text embedding l. 2) Given regional text 7 describing the content of specific region within the image T, both the local visual feature p", pooled from visual dense features X' according to r, and region visual embedding v" of region crop I" should align with region text embedding f" of T".

---

1. Global Contrastive
2. Regional Contrastive
Visual [CZS]
Embedding
3. Real-time Self-
Region Pooling
Textual [CLS]
Embedding
VIT
BERT
Visual Dense
Tokenizer
Feature
crop regions
Ala bock Acap onll spoenced
LVLM
khii of discalate a loyaa eiler whllte plaio acveppatidd
Region
A
Coordinates
image
regions
region texts
image text
Figure 1: Overall architecture of FineCLIP. For simplicity, the diagram omits unused visual dense features of regions extracted by ViT and textual dense features generated by BERT. By integrating multi-grained contrastive learning as well as real-time self-distillatio scheme, FineCLIP aligns visual global embedding, regional dense features, and textual global embedding into unified space, acquiring rich coarse and fine-grained knowledge from image-text and generated region-text pairs.
FineCLIP
Overview. As depicted in Figure FineCLIP adopts the same architecture with CLIP, which consists of the vision encoder V and the language encoder C, but employs more complex inputs objectives.
The input batch of FineCLIP contains image-text pairs {4, Tisss, region-text pairs { TJ his and set of corresponding region coordinates {ry trybism, where B and M refers to the batch size and region count. Noted that the regions {Iī j.s are cropped from images {[g according to region coordinates JsM, and region texts {T] Jis. are generated by LVLMs.
Recall the image and text processing of CLIP introduced Section 2.1 For image-text pairs (T, Tigs and region-text pairs TTj,T, FineCLIP outputs the corresponding global image- text embeddings {u,, t. Jess and fof,, respectiell. Moreover, FineCLIP extracts regional visual representations {p?) J.sM by pooling visual dense features {Ai)iss according region coordinates {r,)ism with RolAlign [15]
Global Contrastive Learning. The global contrastive learning realizes the instance-level alignment, which enables FineCLIP to bolster multi-modal global embeddings and acquire rich -aarse-grained knowledge. Initially, the cosine similarity between image embedding u and text embedding is calculated
S(v. t)
(1)
||a|| ||+||
The global contrastive loss forces FineCLIP to learn global image and text embeddings by maximizing the cosine similarity to the corresponding text and image embeddings, while minimizing the cosine similarity to other non-corresponding ones in the batch, i.c.
1
1
+ log exp(S(t,: vz)/T) Σ exp (Sep(((t.t
exp(S(v;,
Lec
(log
(2)
2B
Σ exp(S(tz, vz)/T)
where 7 is the learnable temperature and initialized with le 2.
Real-time Self-distillation Scheme. Since the global contrastive loss Lee supervises only global embeddings, is poorly effective improving local dense features.
The distillation scheme 55 60 which transfers robust global representational capabilities to region
features, emerges as a promising solution to this problem. The previous work of Wu al. (55) implemented the distillation scheme by guiding trainable student model's local feature extraction

---

with the global visual embeddings of corresponding image crops from frozen teacher model. While efficient, this method depends on the pre-trained teacher model and does not support training from scratch. Additionally, because the student model aligns fully with the frozen teacher model, its performance is capped by the teacher's 's capabilities. This performance ceiling quickly reached during pre-training, limiting further improvement and scalability, as illustrated in Figure 2
We propose real-time self-distillation scheme that relies on and complements global contrastive learning, eliminating the need for frozen teacher model. Instead of relying on a frozen teacher model for providing high-quality global image embeddings, global contrastive learning consistently enhances the student's global representation capability during the training, allowing the student model to teach itself independently. Importantly, our implementation leverages global representations during the training for guidance, resulting in better scalability and fine-grained understanding ability of the student model.
optimal
Specifically, as shown in Figure 1 our real-time self-distillation loss directly maximizes the cosine similarity between region features {pf)isM pooled from image dense features and the visual representations of region crops {uf)isM using the formula
LSD M E (1-58
(1-S(p))))))) S(p),
Semantically-rich Regional Contrastive learning. Despite the remarkable enhancements through aforementioned designs, the model still lacks abundant fine- fineggrined knowledge due to the utilization of coarse grannee training data.
Intuitively, to enrich the model with fine-grained knowledge, it is crucial for the model to focus more on the specific content of image regions and learn more precise and detailed semantics. Therefore, we are motivated to implement regional contrastive learning that operates on the level of regions and related descriptions. As shown in Figure I we leverage the advanced LVLM generate high-quality region descriptions with diverse fine-grained semantics.
The regional contrastive loss compels FineCLIP learn pooled visual region features and text embeddings by maximizing the cosine similarity between matching text and region pairs, while minimizing the similarity with non-matching pairs in the batch, which is defined as follows
1 M
exp(S(p). ΣM exp exp(S(r5,15)/7
LJc 2M
(log
+ log
EM (5)(5(()))5/7))
The reason for utilizing pooled visual region features instead of image embeddings of region crops is two-fold. 1) Supervising pooled region features directly improves model's 's capability to extract valuable local dense features. 2) Enhaneed by attention mechanisms in ViT, pooled region features encompass border perception of entire image content compared to image embeddings of regions, which facilitates a deeper semantic understanding.
Learning Objective. As depicted in Figure | the learning objective of FineCLIP incorporates the above three components. Global contrastive loss Lae works for enhancing representation capabilities and semantic consistency of global visual embeddings, while the self seff-ditiillation loss LSD is designed transfer the real-time strong representation capability of global embeddings to local visual features. Additionally, regional contrastive loss LRC is applied to enrich the FineCLIP with fine-grained knowledge and further improve its local features. The learning objective
LGe + À LSD + 7 LRC,
where the À and y hyper-parameters. As result, FineCLIP fully leverages available diverse semantics and boosts the interactions within multi- multi-grained complementary information.
Experiment
3.1 Ablation Study of FineCLIP
Experiment Settings. In our ablation experiments, we train FineCLIP using A800 GPUs on train2017 split of COCO dataset (30), which includes approximately 118K human-annotated

---

image-text pairs along with 970K region-label pairs. To provide abundant fine- fine-grained knowledge, we replace labels provided by COCO with region descriptions generated by BLIP-2 [26]. FineCLIP is initialized by ViT-B/16 with default input image size of 224 x 224 and corresponding BERT from EVA-CLIP [49]. We train FineCLIP for 10 epochs using Adam [2 optimizer with the batch size of 32 per GPU, the learning rate of l 5, and the weight decay of 0.1. The coefficients À and y in learning objective are both set to 1. In all experiments, we freeze the language encoder to reduce computational overheads and improve training stability.
Using the COCO va12017 split, we test FineCLIP designs on the box classification task with pooled region features and image-level retrieval tasks using global embeddings. We report the Topl and Top5 mean accuracy of all annotated boxes in the box classification task, and the @@ accuracy of image-to-tee and text-to-image retrieval tasks as evaluation indicators.
Ablation of Objective Components. The training objective of FineCLIP in- cludes three components: LGc, LSD and LRC. We first examine the com- bination of LGe and LSD. In Ta- ble (row) #1, using only LeD for supervision causes the model's accu- racy on both box classification and retrieval tasks to drop to near zero. This training collapse is predictable, as the global representation ability of the model is entirely compromised without the supervision of LGc during the training. In Table [ (row) #3, incorporating Loc to support global embeddings allows the training to proceed stably. (rows) #2 and #3 reveals that while the self-distillation loss improves the box
Table I: Ablation study on the objective components.
Box Classification Top1
Retrieval
Loe
LsD
LRC
Top5
12T
T21
1
0.0
0.0
0.0
0.1
2
V
42.3
66.6
62.4
48.8
3
V
V
43.7
72.9
60.0
47.1
4
45.5
72.0
39.5
30.4
5
47.8
74.1
62.5
48.9
6
V
48.4
75.6
62.2
47.6
Comparing Table [ classification performance, it slightly reduces retrieval performance, leading to a trade-off between local feature extraction and global representations.
As shown in Table (rows) #2 and #5, LRc significantly improves the model's 's region feature extraction (+5.5 of Topl and +7.5 of Top5 mean accuracy on box classification) and slightly enhances retrieval performance. This improvement is attributed to the introduction of region-text pairs, which provide abundant fine-grained knowledge, and the positive effects of Lnce on boosting local features. This result also indicates that, despite the possible noise in the region-text pairs generated by BLIP-2, they still offer valuable information for learning region representations. Ultimately, combining all three components enables FineCLIP to achieve optimal classification performance and competitive retrieval results, as demonstrated in Table (row) #6.
Table 2: Performance comparisons of FineCLIP using different region proposal methods.
Box Classification Top1
Retrieval 12T
Num of Regions (per image)
Region Proposal Method
Top5
T21
Time Overhead
Manual 30
48.4
75.6
62.2
47.6
2
AastS 69
47.1
73.7
60.7
46.5
23
15 min
3
RPN 4
48.8
76.0
61.5
46.9
16
10 min
4 YOLOv9
51
49.6
76.5
60.9
47.4
10 min
Ablation of Region Proposal Methods. Regional distillation and contrastive learning are highly sensitive to the quality of region proposals. To assess their impact on the performance FineCLIP, evaluate four different region proposal methods: manual annotations from COCO, FastSAM (69). RPN [43, and YOLOv9 [II. The corresponding results are presented in Table 2 These findings highlight three key insights: 1) Automated region proposals perform comparably to manually annotated high-quality regions. 2) More region proposals do not necessarily improve performance. Although FastSAM generates the most proposals, but they appear overly cluttered upon manual inspection, resulting in suboptimal model performance. 3) RPN provides a moderate number of region proposals with satisfactory and balanced performance. YOLOv9, which focuses on specific object categories, produces fewer but more precise proposals, leading to the best box classification performance of FineCLIP.
5

---

Ablation of Region Annotation Methods. We also explore the im- pact of region annotation methods on addition
Table Performance comparisons of FineCLIP using differ- ent region annotation methods.
FineCLIP's performance. to manual annotation, two main ap- proaches are used for generating re- gion textual descriptions. The first is rule-based method [70), which selects
Box Classification Top5
Retrieval
Region Annotation Method
Topl
I2T
T21
1
Rule-base 70
43.1
71.3
58.6
46.9
2
Intern Aitem-XCoomoser 67
47.0
75.9
60.1
45.1
3
BLIPP 226
48.4
75.6
62.2
47.6
the region concept from predefined
concept pool based on similarity scores and integrates it into a description template. The second
strategy leverages LVLM for region annotation.
Specifically, we annotate the boxes in COCO train2017 split using the rule-based method, BLIP- 2 126.. and Miteml.M-XCooposee [67] and evaluate their impact on FineCI.IP's performance. For LVLMs, we use the prompt: "Describe this image in one sentence." " As shown in Table LVLMM outperforms rule-based method, highlighting the effectiveness of LVLMs in generating valuable fine-grained knowledge. Notably, BLIP-2 provides the greatest improvements to FineCLIP.
Table Performance comparisons of FineCLIP and competing methods on COCO.
Box Classification Top1
Retrieval 12T
Time Overhead (per epoch)
GPU Memory Usage (per card)
Methods
Top5
T21
Pre-trained CLIP
31.1
53.7
59.3
42.4
CLIP 40
42.3
66.6
62.4
48.8
6 min
8G
3
RegionCLIP 70
40.0
65.3
25.1
31.2
9 min
5G
4
CLIPSelf 55
43.7
72.3
33.3
21.2
10 min
6G
5 FineCIIP((Our))
48.4
75.6
62.2
47.6
11 min
36G
3.2 Comparisons with Competing Methods
Following the evaluation setting in Subsection 3.1 we compare FineCLIP with three closely related approaches: CLIP 40) RegionCLIP 170), and CLIPSelf [55]. To ensure fair comparison, all methods adopt the ViT-B/16 as backbone and input images of 224 224 resolution. As presented in Table FineCLIP demonstrates the most improvement over Pre-trained CL.IP in both dense feature extraction and global representation. In contrast, while Region and CLIPSelf achieve moderate gains in box classification tasks, they struggle to maintain the important visual-semantic consistency.
Additionally, we report the time overhead and GPU memory usage the competing methods during training on COCO train22 2017 split. Due to FineCLIP's incorporating of the multi-grained contrastive learning paradigm and the self dif-distillation scheme, it requires comparatively higher GPU memory usage. Nevertheless, it's worth highlighting that the per-epoch training time for FineCLIP (11 minutes) only marginally longer than that of CLIPSelf (10 minutes) and RegionCLIP (9 minutes). Therefore, the training time for FineCLIP remains well within acceptable limits.
3.3 Comparisons on Scaled Trainset
Data Preparation. In this subsection, we evaluate the performance of FineCLIP trained on scaled dataset. We begin by constructing the trainset based on the Conceptual Caption dataset (CC3M) [47) which comprises million image-text pairs sourced from the internet. To meet training data requirements of FineCLIP, we follow a three-step process to create region-text pairs.
This step retains images with rich contents to facilitate the acquisition of clear and valuable regional proposals. Specifically, we filter out low-resolution images and those that fail to generate regions via the region proposal model. After this process, we retain 2.5 million high-resolution images from CC3M, referred to "CC2.5M".
Image Filtering:
Region Proposal: Based ablation results in Table 2 select YOLOV9 [51 to detect objects in images. This process yields 10.4 million high-quality regions (approximately four regions per image) and takes around 4.5 hours complete.
Region Annotation: According to results shown in Table 3 we utilize the BLIP2-COCO-6.2 model to annotate region proposals, which takes approximately 12.5 hours.

---

Zaro-shot of dessiffccaonn
50.7
Zero shrll R assuracy of 12T retrieval
Zoro shot R&D anturacy of T2) retrioval
-0
60
51.0
37.7
48
box
46
40
44
42.7
42
42.3
26.1
40
15
10
Regionc1 CLil
RepentLI
CLiSaal
3.4
CLIPSel
2.0
FinCCLPP
FinCLIPP
FAeCUPP
SOOK Number training samples
580K
2.5M
Number training samples
Number of training samples
(8) Topl mean accuracy of models on COCO box classification task.
(b) Accuraey models on COCO
(c) Accureyy of models on COCO R& text-to-image reteieval task.
R@I image-to-text retricval task.
Figure 2: seoo comparisons of models pre-trained on datasets in three different scales.
Zero-shot Comparisons. To investigate the impact of data scale on model performance, we sample three trainsets of varying sizes from CC2.5M: 100K, 500K, and 2.5M samples. We train FineCLIP and competing methods, including CLIP [40]. RegionCLIPP 770. and CLIPPSlf [55]. using the official open-source code these three trainsets. To ensure fairness, all methods involved in comparisons adopt ViT-B/16 as backhone and input images of 224 x 224 resolution. For each dataset, we train these models for 6 epochs and then evaluate their zero-shot performance on COCO benchmark, using the same metrics of ablation study in Section 3.1
In Figure 2 present the accuracy curves of four methods three tasks the dataset scales up, with detailed quantitative results shown in Appendix Table I1 Overall, our FineCLIP outperforms the other three competing methods in all cases. Specifically, in terms of fine-graing understanding, as shown in Figure 2 a), FineCLIP significantly surpasses other methods in the box classification task, with remarkable +10.4 mAP over CL.IP and +8.0 mAP over CLIPSelf. Notably, the dataset size increases, FineCLIPP' performance continues to grow rapidly, showing promising scalability, whereas the growth rates of RegionCLIIP and CLIPSelf slow down, indicating that their training gradually converges to a relatively low performance level.
As for the evaluation of image-text alignment, according Figure 2( and (c), FineCLIP even surpasses CLIP in retrieval tasks, demonstrating that enhanced local representation and the acquisition of tineggrained knowledge contribute to robust global embeddings. In contrast, RegionCLIP and CLIPSelf fail to maintain semantic consistency in visual embeddings, with their performance deteriorating as the trainset size increases.
Visualization Results. Figure 3 presents the visualized attention maps of our FineCLIP on images responding to complete sentences or individual words. We can see from (igurBaa)(C that FineCLIP comprehends sentence semantics and identifies related elements, even tiny objects like a "man" or irregularly shaped items such as "kite." Figure (()-(()) shows that FineCLIP can well locate different objects within the same image. Interestingly, FineCLIP can also capture abstract concepts such as "looking into the distance" in Figure 3)) and actions. For instance, when recognizing "riding", FineCLIP focuses on both the rider and the horse, while for "watching soccer", it highlights human faces and the soccer ball on the ground. These results collectively indicate that FineCLIP effectively learns to understand fine- tie-ggrined semantics.
3.4 Application to Fine-grained Localization
Open-Voue Vocabulary Object Detection. To evaluate whether the improved fine- aree-graind understanding learned with FineCLIP translates to tasks requiring fine- mme-grained localization, we serve FineCLIP as the backbone for open-vocabulary object detection. Following the previous work [53), we build open-vocabulary object detectors based on V-VIT architecture, which is two-stage detector baseline built on frozen CLIP VITs. Considering that the input resolution has a significant influence on detection performance, to ensure the comparison fairness, we utilize the checkpoints of FineCLIP and competing methods trained on CC2.5M with input image size of 224 224 for ViT-B/16 and 336 x 336 for ViT-L/14 to initialize the F-VIT. In training paree-parameters, employ AdamW
7

---

"Two men sat on bench in the grass looking into the distance.
"A guy is surfing""
"A man is flying kite."
(b)
(c)
"A with red helmer was riding horse."
"A black white shirt is watching soccer:
(d)
"red helmet"
"riding"
"black man" "watching soccer"
Figure 3: Visualizations of attention maps of our FineCLIP using GAE [5] images responding to
complete sentences or individual words. (a)-(c) Image attention maps w.r.t. different sentences. (d)(e) Image attention maps w.r.t. different words.
Table 5: Results open-vocabulary object detection. t means the CLIP VIT backbone is initialized with the checkpoint of the corresponding method trained on CC2.5M.
OV-COCO benchmark
(b) OV-LVIS benchmark
Method
Backbone
Aprueel
AP
APsi
Method
Backbone
mAP:
mAP,
mAP,
mAP
OV.RCNN
53
RN50
17.5
41.0
34,9
FATTTCLIPSeet
VIT-B/16
7.6
10.5
10.6
9.3
RegionCLIP
RN50
26.8
54,8
47.5
PB-OVD
RN50
30.8
46.1
42.1
FAVTVFFneCLLP VIT-/116
8.0
10.9
10.4
9.5
Detic 73
RNS0
27.8
51.1
45.0
FATTHCLIPSeIf
VIF-1/14
19.2
22.0
20.6
20.5
VLDei 29
RN50
32.0
50.6
45.8
ENTT+FineCLII"
VIT-L/14
19.6
23.1
20.2
20.9
F-VLM 23
RNS0
39.6
BARON-Cap
RNS0
33.1
54.8
49.1
CORA 56
RN50
35.1
35.5
35.4
RO-VIT
VIT-B/16
(c) Transfer evaluation of the LVIS-trained detector on COCO and Object365 benchmarks.
RO-VT 20
VIT:L/16
33.0
47.7
CTM VTT
19
VTT:/11
34.1
46.0
VIT-B/16
41.0
34.9
Method
Benchmark
APsa
APts
AP,
APn.
APt
FETTTCCLIPSIII
WTT
25.4
40.9
36.8
E-VI++FnnCLLP!
VIT-B/16
29.8,18.3
45.9
41.7
F-VT+CLIPSeII
32.3
51.7
34,5
13.1
35.9
54.1
F
3366
52.7
36.1
143
38.5
55.1
EVT
V/T:/114
24.7
53.6
46.0
ENTICCLIPSeN
F-TTCCLPSaa
Object365
12.1
19.9
12.5
22
134
30,2
EXIT+FIneCLIP
VIT-L/14
40.0;
57.2
52.7
F-VIT4FinSCLPP
12.1
19.8
12.6
23
13.3
20.3
optimizer with batch size of 8, learning rate of 1e 4, and weight decay of 0.1. We train the models for 3 epochs on the OV-COCO (6) benchmark and 48 epochs on the OV-LVIS [33 benchmark.
For evaluation, we follow previous works 155 70 to report box AP at IoU 0.5 of base, novel and all categories (A Pnovel, A Aplase plase and APso) on OV-COCO, and the for base, novel and all categories (mAP,, mAP, f. mAP, mAP) on OV-LVIS as comparison indicators. The results are shown in Table 5 By replacing the frozen CLIP ViTs with FineCLIP checkpoints, F-ViT gains significant performance improvements (24.7 vs 40.0 AProvel on OV-COCO). Notably, FineCLIP outperforms the existing open-vocabulary object detection methods on the OV-COCO benchmark under fair training settings. Compared with cutting-edge CLIPSelf, our FineCLIP brings better improvements to baseline F-VIT on both benchmarks. Additionally, we also evaluate the detector trained on OV-LVIS on the validation split of COCO and object365 [46] v1 datasets, with results shown in Table 5c FineCLIP surpasses CLIPSelf on the COCO benchmark and achieves similar performance on the Object365 benchmark. All these results demonstrate the effectiveness of our FineCLIP in fine-grained understanding.
Open- Vocabulary Semantic Segmentation. Next, we explore the performance of FineCIIPP when applied to the open-vocabulay semantic segmentation task. Following the previous work [55),

---

Table 6: Results open- vocabulary semantic segmentation. f means the CLIP VIT backbone is initialized with the checkpoint of the corresponding method trained on CC2.5M.
ADE-150
ADE-847
PC-59
Method
Backhone
mloU
mAce
mloU
mAce
mloU
mAce
OVSeg 28
VIT-B/16
24.8
7.1
53.3
ViT-B/16
27.5
10.1
21.1
73.0
SAN 157
ViT-L/14
32.1
50.7
12.4
25.2
57.7
77.6
CatSeg
ViT-B/16
27.2
41.2
8.4
16.6
57.5
74.0
CatSeg
VIT-L/14
31.5
46.2
10.8
20.5
62.0
78.3
CLISet
VIT-B/16
29.7
45.1
10.1
17.2
55.3
73.4
CatSeg+CLIPSef
VIT-L/14
34,9
52.9
13.6
23.0
59.1
77.1
CauSee+FineCLIPP
VIT-B/16
32.4-5.2
50.5
12.2
22.2
56.0
74.4
CatSog+FineCLIPt
VIT-L/14
36.1
53.5
14.1
23.8ra.3
59.9
78.3
build the segmentation model based on CatSeg 77. which utilizes the visual dense features of CLIP VITs (ViT-B/16 and ViT-L/14 from OpenAI) with cost-aggregation module. To ensure comparison fairness, we train FineCLIP and CLIPSelf models, which are initialized with pre-trained OpenAl CLIP [17], CC2.5M with the same input image resolution of 384 384 for ViT-B/16 and 336 x 336 for ViT-L/14, and then replace the backbone of CatSeg with FineCLIP or CLIPSelf ViT for the following segmentation fine-tuning. We fine-tune segmentation models COCO Stuff [4] and evaluate them on ADE20K 71 and PASCAL Context 34 dataset using mean IoU (mloU) and mean pixel accuracy (mAcc).
As shown in Table 6 FineCLIP brings non-trivial improvements to CatSeg across most evaluation indicators, surpassing the enhancements provided by CLIPSelf. We observe that both FineCLIP and CLIPSelf cause a decrease in PC-59 mloU, which may be attributed to the data distribution gap between CC2.5M and PASCAL Context. Furthermore, FineCLIP comprehensively improve the mAce performance of CatSeg, indicating the enhanced per-pixel classification performance.
Table 7: Comparative results on zero-shot image-text retrieval on the Flickr30k and MSCOCO datasets. R@i denotes Recall i. All approaches adopt ViT-B/16 architecture with input image size of 224 x 224. i indicates that the method is initialized with pre-trained CLIP and further trained on CC2.5M. The methods with gray background are pre-trained on large-scale dataset.
Flickr30k
MSCOCO
image-to-text R
lext-to-image R@5
image-to-text R
text-to-jmage R@5
Methods
R@
R@
R@1
R@ 10
R@1
R@ 10
R@
R 10
CLIP
84.0
96.1
98.2
71.6
90.3
56.2
80.6
88.2
42.4
68.6
78.3
SPARC 3
84.4
97.6
98.7
72.0
91.2
94.9
57.6
81.2
88.5
43.0
68.6
78.5
PACL
69.6
89.7
94.2
54.9
80.7
87.3
41.8
67.8
77.6
29.1
54.3
65.5
GLoRIA
78.0
95.5
98.0
68.4
88.9
93.2
49.7
75.4
84.6
38.9
65.1
75.2
MGCA
82.2
98.1
67.7
80.5
87.8
65.7
75.3
FILIP 58
69.0
89.8
94.0
55.8
81.5
87.9
40.2
66.0
76.3
29.5
55.3
66.3
CLIP
81.6
96.2
98.0
64.9
51.1
76.4
37.6
63.9
74.3
RegionCLIPP
3.9
18.4
7.9
11.5
11.8
19.0
CLIPSelf
33.8
61.7
73.0
35.0
32.7
50.4
34.5
45.1
FineCLIpt
82.5
66.5
3.5 Application to Image-level Task
Zero-shot Image-Texs Text Retrieval. We further evaluate FineCLIP on zero-shot cross-modal retrieval tasks using Flicker30K (39 and MSCOCO (30], with the results presented in Table Previous works, such as SPARC [3] PACL 35 GLoRIA [6), MGCA [52], and FILIP [58], introduce fine-grained losses to extract token-level cross-modal alignments. The results of these methods, re-implemented using the same pre-training datasets (approximately 3.2 billion data points), architecture, and training steps, taken from SPARC []. For fair comparisons, train competing methods, initialized with pre-trained CLIP parameters, on CC2.5M for epochs.
Given that CC2.5M has in limited data distribution, further training of pre-trained models on this dataset inevitably leads to more or less performance decay. Compared to the baseline CLIPP, FineCLIP better maintains retrieval performance, demonstrating its effectiveness in enhancing global

---

embeddings and capturing valuable semantics. In contrast, RegionCLIP and CLIPSelf struggle to maintain global embeddings for addressing retrieval tasks. Surprisingly, FineCLIP even outperforms most pre-trained models with fine-grained losses, strongly supporting the effectiveness of FineCLIP.
Related Work
4.1
Fine-grained Understanding in Vision-language Models
Early works focused on learning from intensive human labels by training image classifiers 2 22 48 [5]. These classifiers focus only on a limited range of objects, making it difficult to cover wide range of semantics. CLIP 40] and its diverse variants 12 18 68 popularized learning general visual-language representations by pre- pre-training on noisy large-scale datasets like LAION-400M 44 and LAION 5B [45), which exhibit potent representation capabilities and exceptional generalizability
14
Despite the great achievements, CLIP model has shown weak alignment between regions and corre- sponding texts [37 41 62]. This problem
be roughly attributes to limitations of 1) CLIP loss which ignores the supervision of visual and textual dense features, and 2) brief coarse captions that insufficient allow the model in comprehending image details [10]. Recent works attempted to enhance CLIP's fine-grained understanding by building strong region-text alignment. One line of work leverages region-level annotations for vision-language pre-training (59 64 66). For instance, GLIP (27] utilized large-scale human-labelee grounding data to align semantics at phase and region level, which achieved stronger performance on fully-supervised detection benchmarks. Region- CLIP [70 proposed to generate region descriptions by filling plausible concepts into pre-defined templates. Since the semantics of synthesized descriptions are limited by the pre-defined concept pool, RegionCLIP essentially models the finite category classification task. Although this approach brings benefits to downstream detection tasks, it still struggles to cover broad semantic diversity of open-world scenarios. Another remarkable work is CLIPSeef (55]. which proposed to distill the global representation capability of the frozen teacher model to dense feature extraction of the student model. While CLIPSelf successfully enhancing the local representations, the frozen teacher model limits the performance ceiling of the student model, which is not consistent with the intention of pre-training. A separate line of work explored incorporating losses between image patch and text
token embeddings to learn representations encoding more -ra-grai details 3 16 35 52 60)
4.2 Fine-grained Image Annotation
The process of generating region-text pairs from images involves two steps: proposing regions and annotating the regions with texts. Common strategies for region proposal include random cropping, using Region Proposal Network (RPN) 4 or detectors 442 65 to generate bounding
boxes around objects, or leveraging segmentation models 24 such as SAM (2 Region annotations
can be obtained by expert manual labeling, synthesizing captions using traditional NLP techniques, or generating them with models. A notable work of the second way is Kosmos-2 (3X] which introduced pseudo-labeling pipeline that utilizes the pre-trained GLIP 27 to automatically generate fine-grained pseudo-labels of region boxes. In the era of large models, recent LVLMs 2 31 36
53] have demonstrated impressive capabilities of visual understanding, instruction-following and generalization. By setting prompts, users can control the characteristics of outputs, such as length and writing style, to obtain high-quality responses, making it suitable for image mne-ggained annotation.
5
Conclusion
In this work, we present FineCLIP, a coherent framework that unifies cross-model multi-grained alignment and uni-modal global-to-region guidance. FineCLIP effectively leverages diverse se- mantics from automatically generated regional data and enhances the interactions of multi guati-g complementary information through real-time self-distillation scheme. Extensive experiments demonstrate the superior performance of FineCLIP in fine-grained understanding tasks, including box classification, open-vocabulat object detection and segmentation, well as in global representation tasks like image-text retrieval. We believe this study provides valuable insights into related fields.
Acknowledge
This work partially supported by National Natural Science Foundation of China (62376274, 62437002) and Beijing Natural Science Foundation (L233008).
10

---

References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, et al. Flamingo: a visual language model for few-shot learning. ArXiv, 2022.
[2] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. ArXiv, 2023.
[3] Ioana Bica, Anastasija Ili',, Matthias Bauer, Goker Erdogan, Matko Bovsnjak, Christos Kapla- nis, Alexey A. Gritsenko, Matthias Minderer, Charles Blundell, Razvan Pascanu, and Jovana Mitrovi''. c. Improving fine-grained understanding in image-text pre-training. ArXiv, 2024.
[4] Holger Caesar, Jasper R. Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. CVPR, 2016.
[5] Hila Chefer, Shir Gur, and Lior Wolf. Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. ICCV, 2021.
[6] Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco captions: Data collection and evaluation server. ArXiv, 2015.
[7] Seokju Cho, Heeseong Shin, Sunghwan Hong. Seungjun An, Seungjun Lee, Anurag Arnab, Paul Hongsuck Sco, and Seungryong Kim. Cat-seg: Cost aggregation for open-vocabulary semantic segmentation. ArXiv, 2023.
[8] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
[10] Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites. ArXiv, 2023.
[11] Mingfei Gao, Chen Xing, Juan Carlos Niebles, Junnan Li, Ran Xu, Wenhao Liu, and Caiming Xiong. Open vocabulary object detection with pseudo bounding-box labels. ECCV, 2022.
[12] Shashank Goel, Hritik Bansal, Sumit Kaur Bhatia, Ryan A. Rossi, Vishwa Vinay, and Aditya Grover. Cyelip: Cyclic contrastive language-image pretraining. ArXiv, 2022.
[13] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In CVPR, 2019.
[14] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2015.
[15] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask r-cnn. In ICCV, 2017.
[16] Shih-Cheng Huang, Liyue Shen, Matthew P. Lungren, and Serena Yeung. Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition. ICCV, 2021.
[17] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, 2021.
[18] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021.

---

[19] Dahun Kim, Anelia Angelova, and Weicheng Kuo. Contrastive feature masking open- vocabuuayy vision transformer. In ICCV, 2023.
[20] Dahun Kim, Anelia Angelova, and Weicheng Kuo. Region-aware pretraining for open- vocabulary object detection with vision transformers. In CVPR, 2023.
[21] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Yen Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. ICCV, 2023.
[22] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NeurIPS, 2012.
[23] Weicheng Kuo, Yin Cui, Xiuye Gu, A. Piergiovanni, and Anelia Angelova. F-VLM: Open- vocabulary object detection upon frozen vision and language models. ICLR, 2023.
[24] Feng Li, Hao Zhang, Hu-Sheng Xu, Siyi Liu, Lei Zhang, Lionel Ming shuan Ni, and Heung yeung Shum. Mask dino: Towards unified transformer-ban framework for object detection and segmentation. CVPR, 2022.
[25] Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq R. Joty, Caiming Xiong, and Steven H. Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In NeurlPS, 2021.
[26]
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. 2 Bootstrapping language- image pre-training with frozen image encoders and large language models. In ICMI., 2023.
[27] Liunian Harold Li, Pengehuan Zhang, Haotian Zhang. Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai- Chang, and Jianfeng Gao. Grounded language-image pre-training. CVPR, 2021.
[28] Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. epen-vocabulary semantic segmentation with mask-adapted clip. CVPR, 2023.
[29] Chuang Lin, Peize Sun, Yi Jiang, Ping Luo, Lizhen Qu, Gholamreza Haffari, Zehuan Yuan, and Jianfei Cai. Learning object-language alignments for open-vocabulayy object detection. In ICLR, 2023.
[30] Tsung-Yo Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[31] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023.
[32] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2017.
[33] Matthias Minderer, Alexey A. Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arna,, Mostafa Dehghani, Zhuoran Shen, Xiao Wang, Xiaohua Zhai, Thomas Kipf, and Neil Houlsby. Simple open-vocabulary object detection. ECCV, 2022.
[34] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whar Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In CVPR, 2014.
[35] Jishnu Mukhoti, Tsung- Yu Lin, Omid Poursaeed, Rui Wang, Ashish Shah, Philip S. Torr, and Ser Nam Lim. Open vocabulary semantic segmentation with patch aligned contrastive learning. CVPR, 2022.
[36] OpenAl. Gpt-4v(ision) system card. 2023.
[37] Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. Teaching clip to count to ten. ICCV, 2023.
12

---

[38]
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. ArXiv, 2023.
[39] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In ICCV, 2015.
[40] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[41] Kanchana Ranasinghe, Brandon McKinzie, Sachin Ravi, Yinfei Yang, Alexander Toshev, and Jonathon Shlens. Perceptual grouping in contrastive vision-language models. ICCV, 2022.
[42] Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. CVPR, 2015.
[43] Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. NeurlPS, 2015.
[44] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. ArXiv, 2021.
[45] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaezmarezyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models. ArXiv, 2022.
[46] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. ICCV, 2019.
[47] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, 2018.
[48] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.
[49] Quan Sun, Yuxin Fang, Ledell Yu Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. ArXiv, 2023.
[50] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott F. Reed, Dragomir Anguelov, D. Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. CVPR, 2014.
[51] Chien-Ye Yao Wang. I-Hau Ych, and Hongpeng Liao. Yolov9: Learning what you want to learn using programmable gradient information. ArXiv, 2024.
[52] Fuying Wang, Yuyin Zhou, Shujun Wang, Varut Vardhanabhuti. and Lequan Yu. Multi- granularity cross-modal alignment for generalized medical visual representation learning. ArXiv, 2022.
[53] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu. Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvlm: Visual expert for pretrained language models. ArXiv, 2023.
[54] Size Wu, Wenwei Zhang, Sheng Jin, Wentao Liu, and Chen Change Loy. Aligning bag of regions for open-vocabulary object detection. In CVPR, 2023.
[55] Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Xiangtai Li, Wentao Liu, and Chen Change Loy. CLIPSelf: Vision transformer distills itself for open-vocabulary dense prediction. In ICLR, 2024.
13

---

[56] Xiaoshi Wu, Feng Zhu, Rui Zhao, and Hongsheng Li. Cora: Adapting clip for voca-uucyb detection with region prompting and anchor pre-matching. In CVPR, 2023.
[57] Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open-vocabulary semantic segmentation. CVPR, 2023.
[58] Lewei Yao, Runhu Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. ArXiv, 2021.
[59] Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, W. Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. Detclip: Dictionary-enricher visual-concept paralleled pre-training for open-world detection. ArXiv, 2022.
[60] Lewei Yao, Jianhua Han, Xiaodan Liang. Danqian Xu, W. Zhang. Zhenguo Li, and Hang Xu. Detclipv2: Scalable open-vocabulary object detection pre-training via word-region alignment. CVPR, 2023.
[61] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners image-text foundation models. ArXiv, 2022.
[62] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Y. Zou. When and why vision-language models behave like bags-of-words, and what to do about it? ArXiv, 2022.
[63] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In CVPR, 2021.
[64] Yan Zeng, Xinsong Zhang, and Hang Li. Multi-graine vision language pre-training: texts with visual concepts. ArXiv, 2021.
Aligning
[65] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun-Juan Zhu, Lionel Ming shuan Ni, and Heung yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. ArXiv, 2022.
[66] Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Harold Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision-language understanding. In NeurIPS, 2022.
[67] Pan Zhang, Xiaoyi Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuangrui Ding, Songyang Zhang, Haodong Duan, Hang Yan, Xinyu Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Y. Qiao, Da Lin, and Jiaqi Wang. Internln-xcopposer vision-language large model for advanced text-image comprehension and composition. ArXiv, 2023.
[68] Yuhao Zhang, Hang Jiang, Yasuhide Miura. Christopher D. Manning, and C. Langlotz. Con- trastive learning of medical visual representations from paired images and text. In ArXiv, 2020.
[69] Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang. Fast segment anything. ArXiv, 2023.
[70] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionelip: Region- based language-image pretraining. In CVPR, 2022.
[71] Bolci Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. IJCV, 2016.
[72] Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In ECCV, 2021.
[73] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Phillip Krähenbühl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. In ECCV, 2022.
14

---

A Appendix
A.1 More Ablation Study
Ablation of Input Image Sizes. To explore the impact of input image size on FineCLIP, we set up four different image resolutions [224, 320, 448, 6400 used for training and infer- ence. fit different image resolutions, the posi- tion encoding of VIT is up-sampled to the target shape through bicubic interpolation. According to the results shown in Table increase the image resolution from 224 to 640, FineCLIP's performance on box classification task improves due to the increasingly detailed information provided by images. On the other hand, when the image size becomes too large, VIT receives an excessive number of image patches, which greatly increases the complexity for the global embedding to summarize global information, leading to rise and then decline in FineCLIP's performance retrieval tasks as the image size increases.
Table 8: Ablation study on input image sizes.
Box Classification Top5
Retrieval
#
Image Size
Top1
12T
T21
224
48.4
75.6
62.2
47.6
2
320
51.2
78.0
63.3
48.0
gradually
3
448
52.6
79.7
62.9
48.2
4
640
54.3
80.6
60.7
46.9
Ablation of VIT Backhones. We also ex- plores the influence of using different ViT back- bones on FineCLIP. As shown in Table when FineCLIP employ ViT-L/14, which has large number of parameters, as the initialization, FineCLIP achieves significantly improvements on both types of tasks compared to using ViT- B/16. This result demonstrates the general applicability of our method to different ViT architectures.
Table 9; Ablation study on ViT backbones.
Box Classification
Retrieval I2T
Backbone
Params
Top1
Top5
T21
VIT-B/16
149M
48.4
75.6
62.2
47.6
VIT-L/14
428M
52.6
79.2
66.0
53.4
Table 10:
The study on the impact of LSD to FineCLIP performance in zero-shot setting across different amount of training samples.
Num of Training Samples
100K
500K
2.5M
Box Classification
Retrieval
Box Classilication
Retrieval
Box Classification
Retrieval
Objective Function
Topl
Top5
Top1
Top5
12T
T21
Topt
Top5
T21
Loo Luc
46.6
71.2
50.6
38.4
48.2
73.1
51.6
39.0
49.5
74.7
39.8
Loo Lsc + LaD
47.8
74.1
49.5
37.2
49.4
89.7
51.1
39.0
50.7
91.4
54.4
40.2
Effect Exploration of LSD on Zero-shot Setting, We conduct further validation under zero-shot setting by using trainset different scales, with results shown in Table We observe an interesting trend: as the scale of trainset increases, the negative impact of LsD on retrieval tasks gradually diminishes. This means that Lsn tends to have a positive impact on retrieval performance when FineCLIP is fine-tuned with larger-scale data.
Table II: Results of zero-shot comparisons with datasets in different scales.
Num of Training Samples
100K
500K
2.5M
Box Classification
Retrieval
Box Clastification
Retrieval
Box Classification
Rerrieval
Method
Topl
Tops
12T
T21
Top1
Top5
12T
T21
Topl
Top5
I2T
T21
CLIP
34.7
69.7
47.9
36.1
38.9
73.2
48.9
36.7
40.3
75.6
$1.0
37.7
RegionCLIP
40.7
65.6
18.9
12.4
41.7
63.7
15.6
11.0
42.3
63.4
3.4
2.0
CLIPSelf
41.1
69.7
38.0
25.8
42.5
70.7
34.2
21.4
42.7
70.9
18.8
16.1
FineCLIP (Ours)
47.8
74.1
49.5
37.2
49.4
89.7
51.1
39.0
50.7
91.4
54.4
40.2
A.2
Detailed Zero-shot Comparison Results
We provide the detailed zero-shot comparison results on COCO dataset in Table 11 FineCLIP significantly outperforms the other three comparison algorithms in all scenarios and maintains the fastest performance growth as the dataset expands. Surprisingly. FineCLIP'ss performance in the retrieval tasks under zero-shot setting even surpasses CLIP, indicating that FineCIII can learn valuable regional semantics, and further enhance its global understanding capability.
15

---

Images
CLIP
FineCLIP
Images
CLIP
FineCLIP
Figure 4: K-Means visualization of the dense features of CLIP VIT. We show the raw images, the K-means results of the pre-trained CLIP ViT, and those of our FineCLIP ViT.
Table 12: Comparisons on OV-COCO benchmark with CLIPSelf training settings.
Method
Backbone
Region Type
Input Image Size
A Pnovel
AP
APso
F-VIT
VIT-B/16
17.5
41.0
34.9
C-VISeCLIPSaI
ViT-B/16
Region Proposal
1024 640
37.6
54.9
50.4
FineLIPI
VIT-B/16
Region Proposal
33.5
57.8
51.4
A.3 Comparisons with CLIPSelf Training Setting
We also evaluate the performance of FineCLIP in open-vocabulary detection tasks when adopting settings similar to CLIPSelf. Following CLIPSelf, train FineCLIP COCO Train2017 split, using the region proposals from the trainset and region captions generated by BLIP2. After fine-tuning, FineCLIP's ViT parameters used to initialize the F-VIT for downstream OV-COCO training. As FineCLIP is essentially a contrastive learning method, it needs a larger batch size than distillation could only increase the methods for effective training. Due to the GPU memory limitations, training image resolution of FineCLIP to 640, which is much lower than 1024 used by CLIPSelf.
The results in Table 12 show that despite FineCLIP's lower input image resolution, F-ViT+FineCLIP still outperforms F- CILICee AP phose (57.8 vs. 54.9) and A (51.4 vs. 50.4) metrics.
A.4 More Visualization Results
K-Means Visualization of Dense Features. We present K-Means visualization of dense features generated by CLIP and FineCLIP VIT in Figure It can be seen that the dense features produced by CLIP are noisier and exhibit significant instability. contrast, the results corresponding to FineCLIP are clearer and more consistent with the local semantic.
A.5 Discussion
In this subsection, we discuss the differences between FineCLIP and two closely related works. We depict the main architectures of CLIP, RegionCLIP, CLIPSelf and FineCLIP in Figure
Relation to RegionCLIP. RegionCLIP proposes a regional contrastive learning paradigm based on synthesized region concepts for enhancing fine-grain -reiggr understanding, However, these synthesized region concepts, generated by filling pseudo object label selected from a label pool into pre-defined description template, lack sufficient semantic diversity. Considering that the negative textual samples for contrastive are object concepts that are not matched to the region but matched to other regions in the batch, it essentially models an object classification tasks with predefined finite categories. In contrast, our FineCLIP learns from diverse and detailed regional descriptions generated by LVLM, which are more representative of real-world scenarios.
Relation CLIPSelf. CLIPSelf builds a promising uni-modal global-to-local guidance scheme, relying pre-trained frozen teacher model. We think the main advantage of this scheme is the
16

---

rogional contrestive
distllanion
diciilation
Frannica Fross
regional
Eegler
glotol contaadive
Region Poeling
Region Pooling
Regin Pooling
global conmestivv
Jaii
V
Images
Iingee
Images
Region
Images
Region Images
Images
Realon
Region
Imags
Texts
Conceyps
Brages
Texts
Texa
(4) CLIP
(b) RegionCLIP
(c) CLIPSelf
(d) FineCLIP (ours)
Figure 5: Illustration CLIP variants.
introduction of position inductive bias, enabling dense features grasp local semantics around them, which is crucial for most downstream tasks. However, a notable shortcoming of CLIPSelf that the frozen teacher model limits the performance of student model, and also discrpts the semantic consistency in visual features. By comparison, our FineCLIP fully leverages the model to teach itself with the help of global contrastive learning. While retaining this position inductive bias, FineCLIP enhances local features with real-time global embeddings enhanced by contrastive learning, thereby achieving more valuable local representations.
A.6 Broader Impact
Our FineCL.IP contributes to develop a coherent learning framework that enhances CLIP's fine-grained understanding capabilities. With the growing adoption of transformers, FineCLIP is expected to better address various downstream tasks. To ensure a positive social impact, we conduct experiments using academic open-source datasets that do involve personal privacy issues.
A.7 Limitation
FineCLIP has strict region-level data requirements. Additionally, due to the limitations computa- tional resources, cannot afford train FineCLIP from scratch the billion-level dataset, making difficult to fully realize its potential.
A.8 Future Direction
In this work, address fine-gras fine-grained vision- Laniuaee representation learning by introducing generated region-text pairs and developing a unified architecture. While FineCLIP has shown initial success, there is still significant room for improvement in region-text generation.
Firstly, existing region proposal methods struggle balance category richness and accurate segmen- tation. Our experiments with existing RPN, detectors, and segmentation models revealed that they either limit the number categories or produce disorganized proposals. This results in suboptimal performance tasks with numerous categories, such the LVIS benchmark. One potential im- provement is to train a more robust RPN on datasets with a greater variety categories, enhancing region proposal quality. Secondly, due to computational constraints, we did not use the most powerful LVLM for annotations. stronger LVLM would intuitively produce higher quality annotations.
We hope these insights inspire future research.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly claims the intuition and advantages of FineCLIP in our abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitation of FineCLIP in Section A.7
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
Our work does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes].
Justification: We present the detailed architecture and loss function of the FineCLIP in Section and describe the setup of the hyper-parametent and implementation details each experiment. We will release the code soon.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
19

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We will release the code and generated textual descriptions of regions soon.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nipp) cc/
pub / gui C ssi Pooooliii cyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (ht
/Code Submi s ossionn iisPoliccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We introduce the data preparation and implementation details in Section 3
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
20

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We introduce the GPUs we used in Section 3.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Our research conform with the NeurlPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
We discuss broader impacts in Sention A.6
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: As mentioned in Section A.6 our work leverages open-source datasets and models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We already cite the original papers that produced the code package dataset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperssww www.osocittcceeeoeo com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "hhPSfYvlDm", "reviewer": "Reviewer_5tL8", "summary": "To address CLIP's limitations in understanding fine-grained details, the authors propose FineCLIP, a method for training CLIP-based architectures that proposes two novel losses, a real-time self-distillation loss and a regional contrastive loss. The regional contrastive loss is designed to encourage learning of fine-grained semantic features by querying a Large Vision-Language Model (LVLM) with region crops for a detailed description, then enforcing a contrastive loss between the corresponding text embedding and region's pooled visual embedding. The real-time self-distillation loss encourages pooled regional vision encoder features to be consistent with the vision encoder's embedding of region crops and is designed to prevent loss of global features during training with the regional contrastive loss. To assess the performance of FineCLIP, the authors conduct experiments on Open-Vocabulary Object Detection, Open-Vocabulary Semantic Segmentation and Zero-Shot Image-Text Retrieval and find FineCLIP performs favorably compared to the baselines trained and evaluated in the same setting. They perform ablation studies to assess the contributions of FineCLIP's losses, the region annotation method, and impact of the train dataset size.", "questions": "1. Under what conditions does adding $\\mathcal{L}_{SD}$ help? It's confusing why all three losses in row 5 of Table 1 don't give the best retrieval performance.\n2. In l. 26 what are the domains specifically in the \"domain shift\"?\n3. In l. 48 what losses are happening during the pretraining stage?\n4. In l. 108-109 it says \"instance-level\" alignment but does this instance refer to a single image not an instance of an object?\n7. In Table 2, what is the time required for each region annotation method?\n8. In section 4.2, the smallest dataset is 100k samples. However, often in the case of dense prediction tasks, there is a very limited amount of training data. Why did the authors choose to scale the dataset down rather than up and see how it affects performance?\n9. Can the authors explain the performance discrepancy between CLIPSelf in Table 3 in the original paper [1] versus this work?\n\nMinor Typos/Suggestions:\n- l. 22 remove \"been\"\n- l. 134 \"learn\" <- is this right?\n- l. 344 \"realize\"\n- l. 285 \"detention\" <- \"detection\"\n- l. 228 \"We\" is capitalized\n- l. 262 \"quantitative\"\n- l. 272 \"trainset set\"\n- l. 285 \"detention\"\n\n[1] Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Xiangtai Li, Wentao Liu, and Chen Change Loy. CLIPSelf: Vision transformer distills itself for open-vocabulary dense prediction. In CVPR, 2024.", "limitations": "The authors have included a limitations section on the scalability of FineCLIP to larger datasets, but might improve this section with an analysis of how FineCLIP performs with smaller datasets, and a more detailed semantic analysis of when FineCLIP does not outperform previous methods.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The writing is easy to follow.\n2. The FineCLIP method consists of losses that are straightforward to implement and can be trained end-to-end.\n3. The ablation studies clearly show the contribution of each component of FineCLIP to its performance and justify the choice of region annotation method. The accompanying analyses, including the study of dataset size, are detailed and insightful.", "weaknesses": "1. Trainability: It is difficult to tell how usable FineCLIP is without details on the time it takes to train a single epoch and the GPU memory the method uses. How big is the LVLM? Is it queried on the fly or ahead of time? How long does inference take? The authors should include this information for FineCLIP and CLIPSelf to demonstrate the method's ease of use during training and inference.\n2. Analysis of results: The authors provide a short explanation that the mixed results of FineCLIP in Table 4 on PC-59 and ADE-847 are due to the data distribution gap between CC2.5M and Pascal Context, but it would be better to provide more of a semantic analysis in which classes are underperforming and why this is the case. This would help strengthen the analysis of FineCLIP's limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: The authors could provide more semantic analysis to better explain the performance of FineCLIP on downstream tasks.\n\nA1: Following your suggestion, we add semantic distribution statistics in Table 1 of the attached PDF document. \nThe results show that the training data in CC2.5M related to the labels in PC-59 has the lowest proportion, which explains why training on CC2.5M does not significantly improve model performance on PC-59. \nFurthermore, ADE-847 has more than half of all 847 categories being irrelevant or only weakly relevant to CC2.5M, making it the most challenging benchmark.\n\nQ2: Under what conditions does add $L_{SD}$ help? It's confusing why all three losses in row 5 of Table 1 don't give the best retrieval performance.\n\nA2: According to the results shown in Table 1 (line \\#2 vs \\#3, line \\#4 vs \\#5) of our paper, $L_{SD}$ improves local feature extraction but slightly reduces retrieval performance. \n\nImportantly, we conduct further validation by using trainset in different scales, with results shown in Table 3 of the attached PDF document. \nWe observe an interesting trend: as the scale of trainset increases, the negative impact of $L_{SD}$ on retrieval tasks gradually diminishes. \nThis means that $L_{SD}$ tends to have a positive impact on retrieval performance when FineCLIP is fine-tuned with larger-scale data.\n\nQ3: Why did the authors choose to scale the dataset down rather than up and see how it affects performance?\n\nA3: Due to the limitations in computational resources, we could only afford to experiment with a total of 2.5M training samples. With our current experiment setup, using a single server with 8xA800 GPUs, the experiments in Section 4.2 alone would take at least two weeks. If we increase the training samples from \\{100K, 500K, 2.5M\\} to \\{1M, 5M, 25M\\}, the experimental duration will extend to at least two months, which is far beyond our capacity.\n\nQ4: Can the authors explain the performance discrepancy between CLIPSelf in Table 3 in the original paper versus this work?\n\nA4: The training image resolution is the key factor of this performance discrepancy. Specifically, in our paper, the training image resolution was standardized to 224 for ViT/B and 336 for ViT/L, whereas the original CLIPSelf used a resolution of 1024. CLIPSelf, an efficient distillation method based on pre-trained CLIP, allows the input image resolution to be increased to 1024 due to its low computational costs. According to Table 1(c) in CLIPSelf paper $^{[1]}$, increasing resolution from 320 to 1024 boosts Top 1 Mean Accuracy by 25.6\\%. However, contrastive per-training methods have much higher GPU memory costs, requiring a trade-off between batch size and input image size, making it challenging to achieve such high resolutions. To ensure a fair comparison, we used the default image size employed by CLIP, resulting in a performance decline for CLIPSelf in this work compared to the original paper.\n\nQ5: What are the domains specifically in the \"domain shift\" in Line 26?\n\nA5: The term \"domain shift\" in Line 26 follows the usage in RegionCLIP paper $^{[2]}$. It means that CLIP aligns global image contents with corresponding global text descriptions, but cannot realize well fine-grained alignment between local image regions and corresponding local textual concepts. This lack of fine-grained alignment results in suboptimal performance on downstream tasks requiring fine-grained abilities. We understand your concern that \"domain shift\" might be too abstract in this context. \"Task shift\" might be a more accurate expression.\n\nQ6: What losses are happening during the pre-training stage in Line 48?\n\nA6: The \"pre-training\" mentioned in Line 48 refers to the training process of FineCLIP, not the pre-training of CLIP. Therefore, the loss function at this stage is the complete loss function of FineCLIP as defined in Equation 5. We describe it in this way because FineCLIP is essentially a pre-training method based on contrastive learning combined with self-distillation scheme.\n\nQ7: Does the \"instance\" in Line 108-109 refer to a single image or an instance of object?\n\nA7: The \"instance\" in Line 108-109 refers to the image and corresponding text.\n\nReferences:\n\n[1] CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction.\n\n[2] RegionCLIP: Region-based Language-Image Pretraining.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cDm8qLBCKP", "reviewer": "Reviewer_oDkS", "summary": "The paper introduces FineCLIP, a novel vision-language model designed to enhance fine-grained understanding in image-text tasks. It addresses limitations in existing models like CLIP, which struggle with dense prediction tasks due to a lack of fine-grained detail comprehension. The authors propose two main innovations: A real-time self-distillation scheme that transfers representation capabilities from global to local image features, facilitating a deeper understanding of local image details. A semantically-rich regional contrastive learning approach using generated region-text pairs to boost local representation capabilities with fine-grained knowledge.", "questions": "Minor  question\n\nThe authors highlight the 'real-time' capability of the self-distillation scheme. Could you please clarify the significance of this feature? Is real-time operation essential for the effectiveness of your method or does it offer specific advantages in the given context?", "limitations": "Yes, the authors addressed limitations, potential negative societal impact, and mitigation.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper introduces FineCLIP, a novel vision-language model designed to enhance fine-grained understanding in image-text tasks. It addresses limitations in existing models like CLIP, which struggle with dense prediction tasks due to a lack of fine-grained detail comprehension. The authors propose two main innovations: A real-time self-distillation scheme that transfers representation capabilities from global to local image features, facilitating a deeper understanding of local image details. A semantically-rich regional contrastive learning approach using generated region-text pairs to boost local representation capabilities with fine-grained knowledge.", "weaknesses": "I understand that the author discusses the differences and experimental comparisons with existing methods such as RegionCLIP and CLIPSelf. However, in my opinion, the self-distillation scheme is a very common practice, and its contribution is not particularly novel. The semantically-rich regional contrastive learning paradigm is a significantly effective method for aligning image local regions and text. However, sending all segmented local images into the vision-language model (VLLM) to generate text descriptions does not seem very efficient, especially when scaled to larger datasets.\n\nThe paper could provide more details on the computational efficiency of FineCLIP, especially the training cost, and a comparison with an existing model (CLIPSelf).\n\nRegarding the ablation study in Table 1, the author should further explain why using only $L_{SD}$ for supervision leads to model collapse. Additionally, the effect of using $L_RC$ alone should be supplemented.\n\nIn the open-vocabulary semantic segmentation task shown in Table 4, CatSeg+FineCLIP shows only a minor improvement compared to CatSeg+CLIPSelf. This does not sufficiently support the author's claim of fine-grained semantic alignment.\n\nImportantly, if my understanding is incorrect, please correct me, and I will improve my score accordingly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: As a common practice, the contribution of self-distillation scheme is not novel. What are significance and advantages of \"real-time\" capability of the self-distillation scheme? Why does using only $L_{SD}$ for supervision lead to model collapse? \n\nA1: We fully understand your concerns. To better illustrate the novelty of FineCLIP, we will answer the above questions together.\n\nSelf-distillation is indeed a common practice, typically involving the distillation of some capabilities from a frozen teacher model to a trainable student model, as done in CLIPSelf. However, this implementation is only suitable for further fine-tuning of CLIP, and fundamentally conflicts with multi-modal pre-training scenarios. This conflict mainly arises from two aspects: i) This implementation requires a pre-trained teacher model as a prerequisite. ii) The frozen teacher model limits the performance ceiling of the student model, whereas the goal of pre-training is to continuously improve the model. Given that self-distillation has shown great potential in enhancing local feature extraction abilities, the value of our real-time self-distillation scheme lies in effectively integrating the self-distillation strategy into the pre-training process, fully leveraging its advantages to build a more powerful pre-trained model.\n\nIn terms of implementation, our real-time self-distillation scheme enables a trainable model to act both as the teacher and the student, achieving true \"self-distillation\". The novelty of this scheme is that it resolves the conflicts mentioned earlier: i) This scheme does not require a frozen teacher model, allowing the model to realize self-guidance. ii) During the pre-training process, the model continuously improves its global representation capabilities, thus more effectively guiding local feature extraction and breaking through performance ceilings, embodying the value of \"real-time\". We have demonstrated the effectiveness, stability, and scalability of FineCLIP through extensive experiments.\n\nIt is important to note that this scheme is tailored for multi-modal pre-training and relies on global contrastive learning (also denoted as \"real-time\" in our paper) to work effectively. When only $L_{SD}$ is used for supervision, the lack of global contrastive learning to maintain the model's global representation capabilities leads to the degradation of semantic meaning in model's global embeddings and local features. Consequently, the uni-modal alignment enforced by $L_{SD}$ becomes meaningless, ultimately causing the model collapse.\n\n\nQ2: Sending all segmented local images into VLLM to generate text descriptions does not seem very efficient, especially when scaled to larger datasets.\n\nA2: Good question. We answer it from the following three aspects:\n\ni) Demand for data scale and quality. In the era of large models, the scale and quality of data have been proven to be critical factors affecting the performance of pre-trained models$^{[1]}$. Therefore, expanding the data scale and improving the data quality is an inevitable trend. Compared to manual annotation, using LLM or VLLM for automated data generation is much more efficient.\n\nii) Successful cases of large-scale data annotation with large models. In the field of LLM, using LLMs for large-scale pre-training data synthesis and cleaning has become a standard practice$^{[2,3]}$. In the contrastive learning domain, LaCLIP$^{[4]}$ utilizes LLM to augment the textual content of LAION-400M dataset$^{[5]}$, processing data on a scale larger than that involved in our work.\n\niii) Efficiency of data annotation with large models. In our response to common questions, we show the time cost of data preparation. Given the relatively small amount of data used in our experiment, further optimizations on the inference speed are not performed. However, deploying models using the vLLM$^{[6]}$ framework could significantly increase the throughput of data generation, potentially improving processing speed by at least 20 times.\n\nIn summary, using LVLM for large-scale data annotation is not only feasible but also advantageous in efficiency.\n\nQ3: As shown in Table 4, CatSeg+FineCLIP shows only a minor improvement compared to CatSeg+CLIPSelf, which is not sufficiently support the author's claim of fine-grained semantic alignment.\n\nA3: We appreciate the opportunity to clarify the performance of FineCLIP. \nWe believe that the improvements brought by FineCLIP to CatSeg are comprehensive and significant, adequately demonstrating FineCLIP's superiority in fine-grained semantic understanding. Please see our detailed explanations blow.\n\nOn one hand, according to Table 4 in our paper, CatSeg+FineCLIP surpasses CatSeg+CLIPSelf on all metrics. On the other hand, we calculate the average improvements brought by CLIPSelf and FineCLIP to CatSeg on mIoU and mAcc across three benchmarks for straightforward comparisons, and show the results in Table 7 of the attached PDF document. We can find that the improvements provided by FineCLIP significantly exceed those brought by CLIPSelf, particularly when using the ViT-B/16 backbone.\n\nQ4: The effect of using $L_{RC}$ alone should be supplemented.\n\nA4: Thanks for your suggestion. We present the results of using $L_{RC}$ alone in line \\#4 of Table 4 in the attached PDF document. \nWe can observe that regional contrastive learning benefits the model's local feature extraction abilities and can effectively combine with global contrastive learning and self-distillation scheme to obtain the optimal fine-grained understanding performance.\n\nReferences:\n\n[1] Scaling Laws for Neural Language Models.\n\n[2] The Llama 3 Herd of Models.\n\n[3] Qwen2 Technical Report.\n\n[4] Improving CLIP Training with Language Rewrites.\n\n[5] Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs.\n\n[6] Efficient Memory Management for Large Language Model Serving with Paged Attention.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rsgIhR0kep", "reviewer": "Reviewer_Ask3", "summary": "This paper attempts to overcome the problem of CLIP lacking fine-grained details when adapting to dense prediction tasks. It proposes a unified framework with three training losses: contrastive loss for global image-text pair, region alignment for region-region annotation, and self-distillation for image region embedding and region feature. By combining the three aspects, the FineCLIP shows its effectiveness in fine-grained understanding, at the same time maintains the global representation alignment.", "questions": "1. I noticed that in Sec.4.1 ablation study, the results in Tab. 1 are obtained by training FineCLIP on the train2017 split of the COCO dataset, while in Sec. 4.2 the training set is CC3M. With the same COCO validation set, training with CC3M shows worse performance on retrieval task (54.5 vs. 62.2, 40.2 vs. 47.6) but better top1 box classification result (50.7 vs. 48.4). Any idea about this result?\n2. About region proposal and annotation, I’m curious about some details: \n- Are the region information prepared before the CLIP finetuning or real-time processing during the training?\n- Will different region proposal methods have a big influence on the final results since it is mentioned that “after careful manual evaluation, we choose YOLOv9” in P6 L237? Maybe you can provide the details about the procedure in the appendix. \n- The details about \"We use pixel count as the criterion and discard images that fail to propose regions in the following process.\" to choose CC2.5M.\n3. CLIPSelf is an ICLR2024 accepted paper, not CVPR 2024. The reference of [57] is not correct.\n4. The table head (\"image-to-text\" and \"text-to-image\") of Tab. 5 doesn't align well with the below contents.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1.\tThe proposed FineCLIP incorporates multi-grained contrastive learnings and a self-distillation scheme, which achieves better fine-grained understanding and also keeps the image-level performance.\n2.\tThis paper is well-written and clearly describes the technical details.\n3.\tThe experiments are extensive (dense prediction tasks and image-level tasks on multiple datasets) and impressive.", "weaknesses": "The innovation somewhat combines the previous works regionCLIP and CLIPSelf on finetuning of CLIP, although this work shows evolved implementation details, such as adopting LVLMs to generate region annotations and self-distillation instead of frozen teacher.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: With the same COCO validation set, training with CC2.5M shows worse performance on retrieval task but better top1 box classification accuracy. Any idea about this result?\n\nA1: Good question. We believe this result can be explained by the differences between retrieval and box classification tasks, as well as the effectiveness of FineCLIP in utilizing region-text pairs to enhance local feature extraction capabilities.\n\ni) Task Complexity Differences. From the perspective of text descriptions alone, there is a significant difference in class complexity between retrieval and box classification tasks. The retrieval task involves diverse global text descriptions that are flexible combinations of many objects, making it highly sensitive to the overall image content and corresponding textual content and style. This data distribution gap for retrieval task between trainset and testset is hard to eliminate. In contrast, the box classification task focuses on identifying a relatively smaller number of local objects with clearer concepts. Therefore, it is less affected by the zero-shot setting, which means that if the trainset includes objects from testset and the model has excellent local content learning capabilities, the model can still perform well on the testset.\n\nii) Effectiveness of FineCLIP. According to Figure 2(a) in our paper, FineCLIP greatly outperforms RegionCLIP and CLIPSelf, both of which also utilize region information. This result indicates that the region-text pairs generated using YOLO and BLIP2 effectively cover objects in testset, and FineCLIP has superior local feature learning capabilities compared to competing methods.\n\niii) Impact of Trainset Size. Additionally, we observe that when using the same scale of 100K training samples, FineCLIP under zero-shot setting does not surpass that of using in-domain COCO train2017 split in either task. However, as the data scale increases, FineCLIP achieves superior performance in box classification task, demonstrating FineCLIP's outstanding scalability.\n\n\nQ2: Will different region proposal methods have a big influence on the final results?\n\nA2: Thanks. Based on COCO Train2017 split, we evaluate four different region proposal methods: manual annotation $^{[1]}$, FastSAM $^{[2]}$, RPN $^{[3]}$ and YOLOv9 $^{[4]}$, to show their impact on the performance of FineCLIP. The results are detailed in Table 5 of the attached PDF document. From the obtained results, we find three key insights below.\n\ni) Automated vs. Manual Proposals. \nAutomated region proposals yield results comparable to manually annotated high-quality regions. They slightly perform better in box classification and worse in retrieval tasks, proving the feasibility of automated methods.\n\nii) Box Quantity: More boxes do not necessarily mean better performance. FastSAM generates the most boxes, but they appear too cluttered upon manual inspection, leading to poor model performance. \n\niii) RPN offers a moderate number of region proposals, resulting in balanced performance. YOLOv9, focusing more on specific object categories, produces fewer but more precise boxes, achieving the best box classification performance.\n\nOverall, different region proposal methods indeed have a big influence on the final results. In this paper, we select YOLOv9 because of its efficiency and its significant enhancement of local feature extraction.\n\n\nReferences:\n\n[1] Microsoft COCO: Common Objects in Context.\n\n[2] Fast Segment Anything.\n\n[3] Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.\n\n[4] YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dpWNZs4xPw", "reviewer": "Reviewer_rzQ8", "summary": "This paper integrates three existing techniques in vision-language pre-training into a single end-to-end fine-tuning framework, referred to as FineCLIP:\n- global contrastive: This aligns the global representation of an image with the text embedding.\n- reginal contrastive: This aligns the pooled region-level representations, defined by the generated proposals, with the regional caption generated by a LVLM.\n- regional self-distillation: This aligns the pooled region-level representations with the global representations of the corresponding local crops, following the approach of CLIPSelf [1]. Uniquely, it employs the updated vision encoder itself as the teacher.\n\nIn addition, the authors construct a new subset, “CC2.5M”, based on CC3M for further fine-tuning.\n\n[1] CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction. ICLR 2024", "questions": "- Most of my questions are outlined in the weaknesses section.\n\n- Minor: Some detailed settings are missing. For instance, for Cat-Seg+CLIPSelf and Cat-Seg+FineCLIP, is the region defined by the generated proposals or patches (grids)?", "limitations": "Limitations are properly discussed.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Compared to its close competitors, CLIPSelf and RegionCLIP, the proposed method demonstrates a performance gain on various dense-level tasks while preserving the image-level inference capability.\n- The method integrates the pre-training strategies of both CLIPSelf and RegionCLIP into a single unified framework. This could potentially benefit the pre-training of vision-language foundational models, facilitating both coarse-grained and fine-grained understanding.", "weaknesses": "__W1: limited practical impact.__ \n\n- As a direct successor to CLIPSelf, the performance on both OV-COCO and OV-LVIS benchmarks is significantly lower than the original baseline of CLIPSelf. For instance, FineCLIP achieves 29.8 AP$_{50}^{novel}$ on OV-COCO with ViT-B/16, while CLIPSelf reaches 37.6 with an even smaller dataset. I understand that they’re under different input resolutions, but why not directly increase the input resolution of FineCLIP instead of increasing the scale of the dataset?\n\n- Moreover, a significant advantage of FineCLIP is its ability to preserve the power of image-level inference. However, according to Table 5, all the fine-tuning strategies cause a performance drop to some extent compared to the original pre-trained CLIP on all the image-level benchmarks.\n\n__W2: The experiments are insufficient and incomplete.__\n- Input resolution: Input resolution: Since CLIPSelf can consistently benefit from higher image resolution, a comparison with CLIPSelf under a relatively low resolution (224$\\times$224) is unfair. I would suggest including the performance comparison with higher resolution under the original settings of CLIPSelf.\n\n- The mean mask AP on rare categories (mAP$_r$) is missing for OV-LVIS.\n\n__W3:The limitation of CLIPSelf stated in this paper remains unsolved.__\n> line150: performance is capped by the teacher’s capabilities.\n\nThe supervision of self-distillation heavily relies on the quality of the image-level representation of the teacher (or the updated visual encoder of FineCLIP), but according to Table 5, the image-level performance suffers from a significant drop when compared to the frozen encoder. It’s somewhat unconvincing to me to draw the conclusion that the updated visual encoder can benefit the self-distillation process.\n\nPerhaps the proposed strategy could potentially be beneficial to large-scale vision-language pre-training from scratch given the observations of fine-tuning on CC2.5M, but the current results of this paper are not very convincing to me. I am open to reconsidering my score if my concerns here are addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: The performance of FineCLIP on both OV-COCO and OV-LVIS benchmarks is significantly lower than the original baseline of CLIPSelf. Why not directly increase the input resolution of FineCLIP instead of increasing the scale of the dataset?\n\n\nA1: Thanks for your suggestion. We appreciate the opportunity to clarify the performance of FineCLIP in open-vocabulary detection tasks when adopting settings similar to CLIPSelf. \n\n\nFollowing CLIPSelf, we train FineCLIP on COCO Train2017 split, using the region proposals from the trainset and region captions generated by BLIP2.\nAfter fine-tuning, FineCLIP's ViT parameters are used to initialize the F-ViT for downstream OV-COCO training.\nAs FineCLIP is essentially a contrastive learning method, it needs a larger batch size than distillation methods for effective training.\nDue to the GPU memory limitations, we could only increase the training image resolution of FineCLIP to 640, which is much lower than 1024 used by CLIPSelf.\n\n\nThe results in Table 6 of the attached PDF show that despite FineCLIP's lower training image resolution, F-ViT+FineCLIP still outperforms F-ViT+CLIPSelf in $AP_{50}^{base}$ (57.8 vs. 54.9) and $AP_{50}$ (51.4 vs. 50.4) metrics.\nDue to the time limitation, we could only make this validation on OV-COCO benchmark.\n\nMoreover, we believe that the settings for downstream tasks in our paper are adequately impartial. Even if the scores of different methods might be lower than those reported in original papers, these scores remain fair and reliable for comparison.\nWe understand your concern that the current settings may not fully exploit the potential of distillation methods. \nHowever, we also have to avoid the risk that  high training resolution adversely affects the training stability and effectiveness of contrastive learning methods. In this paper, we thus choose to increase the scale of the dataset instead of increasing the input resolution of FineCLIP. \n\n\nQ2: According to Table 5, all the fine-tuning strategies cause a performance drop to some extent compared to the original pre-trained CLIP on all the image-level benchmarks, which cannot demonstrate FineCLIP's ability to preserve the power of image-level inference.\n\nA2: Good question. We believe the performance decline on image-level benchmarks is caused by both training data and method factors. Therefore, we cannot ignore the impact of the training data, nor can we claim that the method is ineffective based sorely on the performance decline. This is crucial for us to determine whether FineCLIP has a positive or negative effect on image-level task. Considering that different methods used the same training data during fine-tuning, comparing the performance of methods after fine-tuning is sufficient to determine which method is better method. According to Table 5, FineCLIP comprehensively outperforms CLIP after fine-tuning, indicating that FineCLIP maintains stronger global representation capabilities than CLIP. Therefore, we can also reasonably infer that FineCLIP has the potential to exhibit better image-level representation capabilities after pre-training compared to pre-trained CLIP.\n\n\nQ3: Considering that the supervision of self-distillation heavily relies on the quality of the image-level representation of the updated visual encoder of FineCLIP, the image-level performance suffers from a significant drop when compared to the frozen encoder. It's hard to conclude that the updated visual encoder can benefit the self-distillation process.\n\nA3: Good question. \nTo begin with, it is important to note that the evaluation on zero-shot retrieval tasks has biases, as it focus more on measuring generalization ability, rather than the representation capability in a specific data domain. \nFor instance, as shown in Table 8 in the appendix of our paper, FineCLIP fine-tuned on COCO Train2017 greatly outperforms the pre-trained CLIP. \nConsequently, we believe the image-level performance of FineCLIP's updated encoder is significantly improved in the relevant data domain during the training.\n\n\nA significant issue of the frozen pre-trained encoder is that it cannot learn from training data. In small-scale training scenarios, such as distillation, the frozen pre-trained encoder works well. However, as the training data scale expands, these methods would quickly hit bottlenecks, as indicated by the green line corresponding to CLIPSelf in Figure 2.\nIn contrast, the updated visual encoder in our FineCLIP could be consistently improved during the training, thereby better surpporting the self-distillation process. \n\nFurthermore, based on extensive experiments presented in our paper, FineCLIP with an updated encoder surpasses CLIPSelf with a frozen encoder in most evaluation metrics, clearly demonstrating the effectiveness of FineCLIP.\n\n\nQ4: Perhaps the proposed strategy could potentially be beneficial to large-scale vision-language pre-training from scratch given the observations of fine-tuning on CC2.5M, but the current results of this paper are not very convincing to the reviewer.\n\nA4: We believe the distinction between per-training and fine-tuning primarily lies in the scale of training data. Even when FineCLIP is initialized with pre-trained CLIP parameters, its training stage can also be considered as pre-training, if the data scale is sufficiently large. Therefore, there is no fundamental difference between pre-training and fine-tuning. In Section 4.2, by continuously expanding the data scale, Figure 2 shows that FineCLIP demonstrates excellent scalability in both global and local representation capabilities, indicating that our method has great potential to build large-scale vision-language pre-training models.\n\n\nQ5: For Cat-Seg+CLIPSelf and Cat-Seg+FineCLIP, is the region defined by the generated proposals or patches (grids)?\n\nA5: To ensure fairness, all models trained on CC2.5M utilize generated region proposals.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We'd like to thank all the reviewers for the valuable comments and suggestions. We will respond to common questions in this general rebuttal.

Q1: The authors should provide more information of the training time cost and GPU memory usage to demonstrate the ease of FineCLIP.

A1: Good suggestion. We present these details in Table 1 of the attached PDF document. As a multi-grained contrastive learning method, FineCLIP requires relatively high GPU memory usage, but its training time cost is very close to that of CLIPSelf and RegionCLIP under the same setting.

Q2: Questions about CC2.5M construction. i) What is the criteria for selecting CC2.5M from CC3M? ii) Are the region information prepared before the CLIP fine-tuning or real-time processing during the training? iii) What about the time cost of CC2.5M construction?

A2: i) The image selection process consists of two steps. To begin with, we use the YOLOv9 to propose regions for all images in CC3M, discarding images that do not yield any region proposals. Then, we filter out the images with low resolution, which results in 2.5 million selected images.
ii) The region proposals and annotations are prepared before training, rather than being processed in real-time. This is due to two drawbacks of real-time processing: a reduction in training speed, and the randomness of region annotations (which compromises the comparison fairness). 
iii) We utilize 8xA800 GPUs for data preparation. It takes 4.5 hours to generate 10.4 million region proposals using YOLOv9 and 12.5 hours to caption these regions with BLIP2-COCO-6.7B.

Meta-review: This paper presents a method called Fine VLIP for understanding fine-grained details. The authors proposed two novel losses to consider real-time self-distillation and regional contrastive learning. The validity of the proposed method is demonstrated through detailed experiments by showing superior performance to the previous works in most cases. Ablation study also demonstrates the importance of all the key components. The reviewers’ scores are consistently positive.

Here are just a few comments to improve this paper:
- The authors are advised to make the texts in the figures and tables much larger, e.g., as large as those in the main texts.
- “log” in (4) should be “\log”. Similarly, “exp” should be “\exp”.
- “We” in line 228 should be “we”.
Decision: Accept (poster)