FG-CLIP: Fine-Grained Visual and Textual Alignment
Authors: Chunyu Xie
Bin Wang
Fanjing Kong
Jincheng Li
Dawei Liang
Gengshen Zhang
Dawei Leng
Yuhui Yin
Venue: ICML 2025 poster (2025)
Keywords: Vision-Language Model
Contrastive Learning
Image-Text Dataset
Primary area: deep_learning->foundation_models

Abstract: Contrastive Language-Image Pre-training (CLIP) excels in multimodal tasks such as image-text retrieval and zero-shot classification but struggles with fine-grained understanding due to its focus on coarse-grained short captions. To address this, we propose Fine-Grained CLIP (FG-CLIP), which enhances fine-grained understanding through three key innovations. First, we leverage large multimodal models to generate 1.6 billion long caption-image pairs for capturing global-level semantic details. Second, a high-quality dataset is constructed with 12 million images and 40 million region-specific bounding boxes aligned with detailed captions to ensure precise, context-rich representations. Third, 10 million hard fine-grained negative samples are incorporated to improve the model's ability to distinguish subtle semantic differences. We construct a comprehensive dataset, termed FineHARD, by integrating high-quality region-specific annotations with challenging fine-grained negative samples. Corresponding training methods are meticulously designed for these data. Extensive experiments demonstrate that FG-CLIP outperforms the original CLIP and other state-of-the-art methods across various downstream tasks, including fine-grained understanding, open-vocabulary object detection, image-text retrieval, and general multimodal benchmarks. These results highlight FG-CLIP's effectiveness in capturing fine-grained image details and improving overall model performance. The data, code, and models are available at https://github.com/360CVGroup/FG-CLIP.

FG-CLIP: Fine- Grained Visual and Textual Alignment
Chunyu Xie Xie
Bin Wang?"
Fanjing Kong² 2 Jincheng Li Dawei Liang? Gengshen Zhang?
Dawei Leng² Yuhui Yin
Code:
Model: http:///uugiingggeec.cooooooooop
Dataset: http:/hhgginggeee..oodudueee//s/
Abstract
1. Introduction
Contrastive Language- Image Pre-training (CLIP) excels in multimodal tasks such as image-text retrieval and zero-shot classification but strug- gles with fine-grained understanding due to its focus on coarse-graines short captions. To ad- dress this, we propose Fine-Grained CLIP (FG- CLIP), which enhances fine-gre fine-grained understand- ing through three key innovations. First, we lever- age large multimodal models to generate 1.6 bil- lion long caption-image pairs for capturing global- level semantic details. Second, a high-quality dataset constructed with million images and 40 million region-specific bounding boxes aligned with detailed captions ensure precise, context-rich representations. Third, 10 million hard fine-gra tne-grained negative samples are incorpo- rated to improve the model's ability to distin- guish subtle semantic diffcrences. We construct a comprehensive dataset, termed FineHARD, by integrating high-quality region-specifie an- notations with hard fine-grained negative sam- ples. Corresponding training methods are meticu- lously designed for these data. Extensive exper- iments demonstrate that FG-CLIP outperforms the original CLIP and other state-of-the-ar meth- ods across various downstream tasks, including fine-graines understanding, open- apen-vocabulary ob- ject detection, image-text retrieval, and general multimodal benchmarks. These results highlight FG-CLIP's effectiveness in capturing fine-gre graingai image details and improving overall model perfor- mance. The data, code, and models are available at https://github.comm Www.o/////0OGGG///G--GCCCCCC
The integration of vision and language (Alayrac et al., 2022; Ramesh et al., 2022; Lin et al., 2023; Gabeff et al., 2024) has been a long-standing goal in artificial intelligence, aiming to develop models that can understand and reason about the world visually and linguistically rich manner. Recent advances in multimodal pre-training, such CLIP (Radford et al., 2021), have made significant strides in this direction by learning joint representations of images and text through contrastive learning. These models have achieved state- of-the-art performance in a variety of downstream tasks, including image-text retrieval (Pan et al., 2023; Sun et al., 2024; Zhang et al., 2024), image captioning (Mokady et al., 2021; Li et al., 2024; Yao et al., 2024), and visual question answering (Li et al., 2023a; Parelli et al., 2023; Team et al., 2024; Wang et al., 2025). However, despite their impressive capabilities, these models often struggle with fine-grained details, particularly in recognizing object attributes and their relationships.
Recent works (Liu al., 2023a; Wu et al., 2024b; Zhang et al., 2024; Zheng et al., 2024; Jing al., 2024) point out two primary reasons for the limitations in CLIP's fine- grained learning capability. First, the original CL.I model's text encoder supports only up to 77 tokens, restricting its ca- pacity to process detailed descriptions and hindering its abil- ity to capture nuanced textual information. Second, CLIPP aligns entire images with corresponding text descriptions, making it challenging to extract valuable region-specific rep- resentations from visual features. Consequently, the model struggles to achieve fine- fne-grained alignment between image regions and their corresponding textual attributes, limiting its effectiveness in complex recognition scenarios.
To address these issues, researchers have proposed extend- ing the positional encoding to support longer token se- quences (Wu al., 2024b; Zhang et al., 2024; Zheng et al., 2024) and integrating object detection datasets into CLIP training (Zhong et al., 2022; Jing et al., 2024). By align- ing bounding boxes with category labels, these methods
'qqual contribution 'Beihang University 2360 Research. Correspondence to: Dawei Leng <lengdawei@@300.cn
Proceedings of the 4 International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

FG-CLIP Fine-Grained Visual and Textual Alignment
aim enhance regional feature extraction. Although these approaches have shown some improvements, they still fall short in fine-grained visual recognition and open-vocabulary object detection. Existing methods (Jing et al., 2024; Zhang et al., 2024) typically introduce relatively few long captions, usually on a million scale, which is inadequate for effective learning of fine-grained details. Additionally, aligning im- age regions with category labels limits semantic diversity, restricting the model's generalization to open-world scenar- ios. Furthermore, the lack of hard fine-grained negative samples limits the model's ability to distinguish between objects of the same category but with different attributes. In this work, we introduce Fine-Grained CLIP (FG-CLIP), a novel approach designed to enhance CLIP's fine-grained understanding capabilities through three key innovations.
tion (Kazemzadeh et al., 2014), and reducing output halluci- nation (Li al., 2023c). We provide visualization results in Appendix C to demonstrate the improvement in fine-grained understanding. These results highlight FG-CLIP's effective- ness in capturing fine-grained image details and improving overall model performance. To facilitate future research and application, we make the models, datasets, and code pub- licly available at
2. Related Work
2.1. Contrastive Language-Image Pre-training
Contrastive learning has emerged as a powerful paradigm in multimodal pre-training, significantly advancing the field of image-texi alignment. Models like CLIP have revolu- tionized this area by leveraging large-scale image-text pairs to learn rich representations without explicit supervision. CLIP achieves this through a dual-encoder architecture that maps images and their corresponding text descriptions into shared embedding space, where semantically similar pairs are pulled closer together while dissimilar pairs are pushed apart. This approach not only simplifies data labeling but also enables zero-shot transfer to downstream tasks, demon- strating impressive performance on various benchmarks such as image classification (Deng et al., 2009; Recht et al., 2019) and image-tex retrieval (Young et al., 2014; Lin et al., 2014; Urbanek et al., 2024; Chen et al., 2024a).
First, we significantly enhance global-level semantic align- ment by generating long captions using sfte-offthe-art large multimodal models (LMMs) (Hong et al., 2024). This pro- cess introduces 1.6 billion long caption-image pairs, provid- ing an unprecedented scale of data that allows FG-CLIP to capture nuanced details at the global-level semantic layer, thereby enhancing its ability to perceive complex and de- tailed information.
Second, to improve fine-grained alignment between images and text, we develop high-quality visual grounding dataset. This dataset includes detailed descriptions for 40 million bounding boxes across 12 million images, ensuring that each region precisely annotated with context-rich captions. By creating such an extensive and richly annotated dataset, we enable the model to learn precise and contextually rich representations, significantly enhancing its performance tasks that require fine-graines understanding.
2.2. Fine-Grained Understanding
Despite its success, CLIP faces limitations in handling fine- grained visual details. Its text encoder is constrained to 77 tokens, limiting its capacity to process detailed and com- plex descriptions. Additionally, CLIP aligns entire images with corresponding text, making it challenging to extract valuable region-specific representations. To address these limitations, models like LongCLIP (Zhang et al., 2024) ex- tend the maximum token length of the text encoder, enabling it to handle longer and more detailed textual information. GLIP (Li et al., 2022) and RegionCLIP (Zhong et al., 2022) introduce grounding data, enhancing the model's ability to align specific regions within images with corresponding text, thereby improving performance on downstream detec- tion tasks (Xie et al., 2018: Gupta et al., 2019; Zhou et al., 2022b; Minderer et al., 2024). However, even with these improvements, existing models still struggle to fully capture and align fine- ine-grained features across diverse datasets.
Third, to further enhance model robustness and discrimina- tion abilities, we introduce a large-scale corpus of 10 million hard fine-grai ane-ggrined negative samples. By incorporating these challenging negative samples into the training process, FG- CLIP learns to distinguish subtle differences in semantically similar but distinct pairs, thereby significantly improving its performance across various downstream tasks. We integrate the high-quality visual grounding data and hard fine-graine grained negative samples as a whole dataset called FineHARD.
Compared to previous methods, FG-CLIP demonstrates sig- nificant improvements across a wide range of benchmark tasks. Our comprehensive enhancements enable the model to achieve superior performance in capturing nuanced visual details, as evidenced by our state-of-the-art results on tasks such as fine- -rainne understanding, bounding box classifica- tion, long caption image-text retrieval, and open-vocabulary object detection. Moreover, when utilized as the back- bone for LMMs (Liu et al., 2023b), FG-CLIP also demon- strates performance improvements in tasks involving at- tribute analysis (Hudson & Manning, 2019), object localiza-
2.3. Image-Tex: Datasets
Image-text datasets (Gu et al., 2022; Xie et al., 2023; Fu et al., 2024) play a pivotal role in the performance of multimodal models. While existing datasets such as LAION (Schuhmann et al., 2021; 2022), COCO (Lin et al.,
2

---

FG-CLIP Fine-Grained Visual and Textual Alignment
Glvval-Level Captions
Region-Level Captions
Long Caption
Short Caption
Region Captions
a sleck wireless keyboard and mousse selup
Positive Region Description
The focal large. flat-screen monitor positioned centrally on dark, curved displaying an abstract wallpaper with bright of orange blue. Relow the monitor, we see sleek, wireless keyboard and mouse setup positioned neatly the desk's surfice... montoor
large flat-screon monit positioned contrally dark curved desk
weoden wircless keyboard and sctup
This image depicls modem computer workslation.
Negative Region Descriptions
cylindrical tan mug with metallic rim which might contain heverage
moust keyboard
Text Encoder
CLS Kong
CLS CLScbon shoet fest
CLS
CoSaai Pagiinn
CLS
por leal
CLS Jeer
2013
fest
global
CLS Joe
region
Image
ROI
Feature
Encoder
Align
L Lhadd
Feature
Figure 1. Overview FG-CLIP CLSSing denotes the image class features output by the Vision Transformer (ViT), while CLStext represents the class features summarized by the text model for multiple inputs, including long captions, short captions, region captions, and positive&negatie descriptions of specific regions within images. FG-CLIP's training proceeds in two stages: the first stage leverages global-level caption-image pairs achieve initial fine-grained alignment, while the second stage supplements these with additional region-level captions, including detailed region captions and positive/negative region descriptions to further refine the alignment.
2014), Flickr30K (Young et al., 2014), and Conceptual Cap- tions (Sharma et al., 2018; Changpinyo et al., 2021) offer valuable resources, they often emphasize general scene de- scriptions, neglecting fine-grained details critical for ad- vanced applications. Researchers have adopted several strategies to mitigate these limitations. One approach in- volves leveraging advanced large multimodal models (Lau- rençon et al., 2024; Wang et al., 2024; Wu et al., 2024c; Chen et al., 2024b; Team et al., 2024) to refine and enrich text descriptions through recaptioning. For instance, Long- CLIP (Zhang et al., 2024) utilizes 1 million long caption- image pairs from ShareGPT4V (Chen et al., 2024a), and FineCLIP (Jing al., 2024) constructs dataset of 2.5 million long caption-image pairs. Although these efforts hance data richness, they remain limited in scale compared to the vast amount of data in the image-text field. Another strategy is to implement pseudo-labeling pipelines using pre-trained object detection models (Li et al., 2023b; Ma al., 2024; Hou al., 2024) to automatically generate fine- grained pseudo-labels for region boxes, similar to the GRIT dataset utilized in Kosmos-2 (Peng et al., 2024). These methods help improve region-speeific alignment but may introduce noise due to automated labeling.
guish, limiting the model's ability to learn subtle variations. The absence of hard negative samples impedes true fine- grained understanding, as models struggle to discern small but meaningful differences in visual and textual features. Addressing this gap is essential for developing models ca- pable of reliably performing fine-grained recognition and alignment tasks, thereby enabling them handle the anced distinctions necessary for advanced applications.
3. Approach
3.1. Fine-Grained CL.P
Figure 1 provides an overview Fine-Grrined CLIP (FG- CLIP). Our proposed FG-CLIP extends the traditional dual- encoder architecture of CLIP better capture fine-grained details in images and text. We leverage a two-stage training paradigm to achieve this enhancement, In the first stage, FG- CLIP focuses on aligning global representations of images and text using only global contrastive learning. The sec- ond stage builds on this foundation by introducing regional contrastive learning and hard fine-grained negative samples learning, leveraging region-text data further refine the model's understanding of fine-grained details.
Another significant challenge the scarcity of hard fine- grained negative samples. Existing datasets predominantly consist of positive examples that are relatively easy distin-
Global Contrastive Learning. Global contrastive learn- ing aims to enhance the model's fine- gininee understanding
3

---

FG-CLIP: Fine-Grained Visual and Textual Alignment
by introducing a method of augmenting long caption align- ment utilizing Large Multimodal Models (LMMss). This approach generates additional long captions that provide richer context and finer- sher-gga descriptions. The inclu- sion of long captions enables the model to perceive and align with global-Ievel semantic details, thereby enhancing fine-grained understanding and context awareness. In addi- tion, we retain the alignment of short caption- -imge pairs. The long captions complement these short captions, ensur- ing that the model learns from both detailed, nuanced long captions for complex semantic information and concise, di- rect short captions for basic concepts. This dual approach improves the model's overall performance in capturing a broader spectrum of visual information.
contrastive learning significantly improving its detail per- ception capabilities in both granular and holistic contexts.
Regional Contrastive Learning. Regional contrastive learning focuses on aligning specific regions within im- ages with corresponding text segments. To achieve this, we employ RoLAlign (He et al., 2017) to extract region- specific features from the image. These extracted features are then processed by applying average pooling over the tokens within each detected region, resulting in a set of region embeddings {rx {r>} where K denotes the total number of valid bounding boxes across all images within a batch. This approach differs from global contrastive learn- ing, which typically relies on the [CLS] token for deriving image-level features. For text, we segment the full-image caption into phrases or sentences that correspond indi- vidual bounding boxes, obtaining text embeddings 4g. The regional contrastive loss is defined as:
In our framework, both short and long captions are aligned with images by utilizing the [CLS] token features extracted from the text encoder for the captions and the [CLS] token features from the image encoder for the images. To accom- modate longer and more detailed captions while preserv- ing the alignment of short captions, position embeddings of FG-CLIP's text encoder are extended. Specifically, for sequences shorter than or equal to 20 tokens, we use the original position embedding directly. For longer sequences, apply linear interpolation with a factor of 4 for posi- tions beyond 20, extending the maximum length from 77 to 248 tokens. This modification ensures that the model can effectively handle longer, more descriptive text while maintaining computational efficiency.
K Σ (log exp(s(r,,()//T)
Leegional
2K
E exp(s(rr.()/))
1=1
exp(s(l;,ii/+t)
+ +log
(3)
This encourages the model to learn fine-grained alignments between specific regions and textual descriptions.
During each training step. the model employs both a short caption and long caption for every image ensure com- prehensive and fine-grained understanding. Given image- text pair, the outputs of both encoders are embeddings V E TRd for images and í. E Rd for text, where d is the dimensionality of the embedding space. We compute the similarity between each pair using the cosine similarity met- ric:
Hard Fine-Grained Negative Samples Learning. To address the scareity of challenging fine-grained negative samples, we introduce a hard negative mining strategy. We define hard negative samples as those that are semantically close but not identical to the positive sample. These hard negatives are constructed by rewriting the descriptions of bounding boxes, modifying certain attributes to create subtle differences. The specific process of obtaining hard fine- grained negative samples can be found in Section 3.2.
V.
sv, t)
||
To incorporate hard negative samples into the learning pro- cess, we extend the loss function to include a term for hard negatives. For each region-text pair, we compute the sim- ilarity between the regional feature and both the positive description and the corresponding negative sample descrip- tions. The hard negative loss I-hard is defined as:
The objective function for global contrastive learning is based on the InfoNCE loss (He et al., 2020), which maxi- mizes the similarity between matching pairs while minimiz- ing the similarity between mismatched pairs. Specifically, the loss for a batch of N image-text pairs is given by:
K
exp(s(r,,l..//7) 5xeexp(((rrhh.)/777
Liard = K
Σ log
(4)
N
exp(s(1i,4ii/7)
L Lglobal
(log
where M denotes the total number of captions for each region, with = corresponding to the positive sample, and > 1 corresponding to the negative samples.
2N
Σn exp(s(u;: (/)T)
exp(s(t,,4)//
log Σ exp(s(tt, U,)T))
(2)
In the second stage, we integrate all three components: Global Contrastive Learning, Regional Contrastive Learn- ing, and Hard Grained Negative Samples Learning, to
where 7 is learnable temperature parameter. This global
4

---

FG-CLIP Fine-Grained Visual and Textual Alignment
FineHARD, featuring precise region-specific captions and ensure comprehensive and nuanced alignment tasks. The learning objective in the second stage combines these ele- ments:
hard negative samples. We curate the overall dataset based GRIT (Peng et al., 2024) images. The process begins with generating detailed image captions using C02 19B (Hong et al., 2024), ensuring comprehensive and nu- anced descriptions that capture the full context of each im- age, Following (Peng et al., 2024), we then use SpaCy (Hon- nibal et al., 2020) to parse the captions and extract the refer- ring expressions. Subsequently, the images and referring ex- pressions are fed into the pretrained object detection model, i.e., Yolo-Workd (Cheng et al., 2024) to obtain the associated bounding boxes. Non-maximum suppression is applied to eliminate overlapping bounding boxes, retaining only those with predicted confidence scores higher than 0.4, This pro- cess results in 12 million images and 40 million bounding boxes with fine-grained region captions. We provide ex- amples of the images and their corresponding captions in Appendix A.
L = Lylobal + a * Lregionnl + B Lhard.
(5)
Here, the hyperparameters a and B are set to 0.1 and 0.5, respectively, to balance the regional contrastive loss and
the hard negative loss, ensuring that each loss operates on similar scales.
This integrated approach ensures that FG-CLIP not only captures global-level semantic details but also distinguishes subtle differences in semantically similar pairs, enhancing its overall performance across various downstream tasks.
3.2. Curated Dataset
In this section, we describe the meticulous process of curat- ing datasets for our FG-CLIP model, emphasizing both scale and quality to address the limitations of existing models in fine-grained understanding.
Next, to create challenging fine-grained negative samples, we modify attributes of bounding box descriptions while keeping the object names unchanged. For this task, we employ an open-source large language model, Llama-3.1- 70B (Dubey et al., 2024), to generate 10 negative samples for each positive sample. To ensure clarity, we remove spe- cial symbols such as semicolons, commas, and line breaks from the generated descriptions. A quality check of 3,000 negative samples reveals that 98.9% are qualified, with only 1. % considered noise-a level within the expected toler- ance for unsupervised methods. This process generates sub- tle variations that better reflect real-world scenarios where objects may appear similar but differ in specific details. We illustrate examples of the fine-grained negative samples in Appendix B.
Enhancing LAION-2B Data with Detailed Recaptioning. In the first stage of training, utilize enhanced version of the LAION-2B dataset (Schuhmann et al., 2022), where images are recaptioned with detailed descriptions generated by large multimodal models, i.e., CLgVLLM2-11 19B (Hong et al., 2024). This approach generates more detailed and contextually rich captions, crucial for capturing subtle dif- ferences in visual content. The original LAION-20 dataset often suffers from overly generic imprecise captions, lead- ing to suboptimal performance in fine-grained tasks. For instance, an image of a bird might be described as "a bird", without specifying the species or environment. Such generic captions limit the model's 's ability to recognize fine details.
The resulting dataset includes 12 million images with fine- grained captions, 40 million bounding boxes with detailed region descriptions, and 10 million hard negative samples. The data pipeline utilizes a cluster of 160x910B NPUs and takes 7 days to complete. This comprehensive dataset enhances the model's ability to capture fine-grained details and provides a robust foundation for training FG-CLIP to distinguish subtle differences in visual and textual features.
By leveraging advanced large multimodal models, we gen- erate detailed deseriptions that not only identify objects but also provide rich contextual information about their at- tributes, actions, and relationships within the scene. For instance, rather than a generic description like "a bird", our refined captions read "a red-winged blackbird perched on a tree branch in a park." Utilizing a cluster of 160x910B NPUs, the data processing is completed 30 days. An ablation study detailed in Section 4.5 evaluates the im- pact of using these high-quality, detailed captions. The results demonstrate significant improvements in model per- formance across various tasks, underscoring the critical role of large-scale, high-quality text annotations in enhancing both model accuracy and context understanding.
4. Experiments
4.1. Implementation Details
In the first stage, we train dataset of 1.6 billion im- ages, each paired with short and long texts. The model is initialized with weights from the original CLIP (Radford et al., 2021). For both ViT-B and ViT-L (Dosovitskiy, 2021) configurations, the batch size per NPU is set to 384. The learnable temperature parameter initialized to 0.07. We utilize the AdamW optimizer with a learning rate of Ic-4,
Fine-Grained Visual Grounding-RReaptionn--- ative Dataset (FineHARD). For the second stage, we develop a high-quality visual grounding dataset named
Neg-
5

---

FG-CLIP: Fine-Grained Visual and Textual Alignment
Table Results on FG-OVD benchmark. Accuracy is reported.
Table 3. Performance open- open-vocabulary object detection task.
Cra-Graineed Understanding medium
OV-COCO
Method
Backbone
Method
Backhone
hard
easy
trivial
Phonee
APSi
AP55
CLIP
ViT-B/16
12.0
23.1
22.2
58.5
OV-RCNN
RN50
17.5
41.0
34,9
EVA-CLIP
VIT-B/16
14.0
30.1
29.4
58.3
RegionCLIP
RN50
26.8
54.8
47.5
Long-CLIP
ViT-B/16
9.2
18.4
16.2
51.8
Detic
RN50
27.8
51.1
45.0
FineCLIP
ViT-B/16
26.8
49.8
50.4
71.9
VLDet
RN50
32.0
50.6
45.8
FG-CLIP
VIT-B/16
46.1
66.6
68.7
83.4
RO-VIT
ViT-B/16
30.2
41.5
RO-VIT
ViT-L/I6
33.0
47.7
CLIP
ViT-L/14
15.4
25.3
25.7
38.8
CFM-VIT
VIT-L/I6
34.1
46.0
EVA-CLIP
VIT-L/14
18.3
38.4
35.2
62.7
Long- CLIP
VIT-L/14
9.6
19.7
16.0
39.8
F-VIT
ViT-B/16
17.5
41.0
34,9
FineCLIP
VIT-L/14
22.8
46.0
46.0
73.6
R-VIT+CIIPSEI
ViT-B/16
33.6
54.2
48.8
FG-CLIP
VIT-L/14
48.4
69.5
71.2
89.7
F-VIT+FineCLIP
ViT-B/16
29.8
45.9
41.7
F-VII+F--CLIP
VIT-B/16
35.1
51.7
47.4
F-VIT
ViT-L/14
24.7
53.6
46.0
Table 2. Bounding box classification results.
CLIPSeL
VIT-L/14
38.4
60.6
54.8
F-VIT+FineCLP
VIT-L/I4
40.0
57.2
52,7
Boox Classification
F-VIT+G-CCIP
VIT-L/I4
41.2
58.0
53.6
Method
Backbone
COCO
LVIS
Open Images
CLIP
ViT-B/16
44.2
20.9
15.3
2021), RegionCLIP (Zhong et al., 2022), Detic (Zhou et al.,
EVA-CLIP
VIT-B/16
30.6
14.4
8.8
2022b), VLDet (Lin et al., 2022), RO-ViT (Kim et al.,
RegionCLIP
VIT-B/16
40.0
22.2
19.1
2023b), CFM-ViT (Kim et al., 2023a), F-ViT(Wu et al.,
CLIPSelf
VIT-B/16
43.7
7.8
11.4
2024a), and CLIPSelf (Wu et al., 2024a).
Long-CLIP
VIT-B/16
36.7
18.2
14.9
FineCLIP
VIT-B/16
48.4
23.3
18.1
FG-CLIP
VIT-B/16
52.3
28.6
20.6
Fine-Grained Understanding. Based on the fine-grain ine-grained benchmark FG-OVD constructed by (Bianchi et al., 2024), we evaluate open-source image-text alignment models. Un- like previous benchmarks such as MSCOCO (Lin al., 2014) and Flickr (Young et al., 2014), which rely on global information for matching, this benchmark focuses on iden- tifying specific local regions within images. Each region has one corresponding positive description and ten nega- tive descriptions, with the negative samples derived from the positive text. This benchmark primarily comprises four subsets of varying difficulty levels: hard, medium, easy, and trivial. The increasing difficulty across these subsets is reflected in the degree of distinction between the texts to be matched. In the hard, medium, and easy subsets, one, two, and three attribute words are replaced, respectively. In the trivial subset, the texts are entirely unrelated. For source collection of specific attribute words, please refer to (Bianchi et al., 2024).
CLIP
VIT-L/14
33.8
9.3
8.3
EVA-CLIP
VIT-L/14
32.1
18.3
9.3
Long-CLIP
VIT-L/14
35.6
8.9
FineCLIP
VIT-L/14
54.5
22.5
19.1
FG-CLIP
VIT-L/14
63.2
38.3
23,8
weight decay of 0.05, 31 of 0.9, 32 of 0.98, and warmup steps for the first 200 iterations. The entire training process employs DeepSpeed's s Zero-2 optimization technique and Bíloat16 precision to accelerate training, and the model is trained for one epoch.
In the second stage, train dataset of 12 million images. Apart from long and short captions, this dataset includes high-quality visual grounding annotations and hard fine-graiie -re-grained negative samples. The model is initialized with weights obtained from the first stage. The batch size per GPU is set to 512. We employ the AdamW optimizer with learning rate of le-6, weight decay of 0.001, 31 of 0.9, 32 of 0.98, and warmup steps for the first 50 iterations. Training acceleration techniques include DeepSpeed's Zero- 2 optimization, CUDA's TF32 technology, and Bfloat1 precision, and the model is trained for one epoch.
During testing, following FineCLIP, we first extract dense features from the model by removing the last self self-attention layer as suggested by (Zhou et al., 2022a). Subsequently, we combine the bounding box information provided by the benchmark with ROIAlign to obtain representative features. These features are used calculate similarity scores with both positive and negative sample descriptions. Top-1 accu- racy is adopted the evaluation metric.
4.2. Comparisons on Fine-grained Region-level Task
In this section, the primary methods included for com- parison are CLIP (Radford et al., 2021), EVA-CLIP (Sun et al., 2023), Long-CLLn CLIP (Zhang et al., 2024), and FineCLIP (Jing et al., 2024). Additional methods involved in open- vocabulary detection include OV-RCNN (Zareian et al.,
As shown in Table 1, FG-CLIP achieves significant improve- ments over existing models, particularly on the challenging hard and medium subsets, thanks to its hard fine- grainnd negative samples learning strategy. Examples of different
6

---

FG-CLIP Fine-Grained Visual and Textual Alignment
Table Comparisons on image-level tasks, including long/short caption image-text retrieval, and zero-shot image classification.
ShareGPT4V
DCI
MSCOCO
Flickr30k 12T
ImageNet-1K Top-1
ImageNet-v2
Method
Backbone
12T
T21
12T
T21
12T
T21
T2I
Top-1
CLIP
ViT-B/16
78.2
79.6
45.5
43.0
51.8
32.7
82.2
62.1
68.4
61.9
EVA-CLIP
ViT-B/16
90.5
85.5
41.9
41.2
58.7
41.6
85.7
71.2
74.7
67.0
Long-CLIP
ViT-B/16
94.7
93.4
51.7
57.3
57.6
40.4
85.9
70.7
66.8
61.2
FineCLIP
VIT-B/16
70.6
73.3
35.5
34.4
54.5
40.2
82.5
67.9
55.7
48.8
FG-CLIP
VIT-B/16
96.7
94.9
61.8
60.6
64.1
45,4
90.7
76.4
69.0
61.8
CLIP
ViT-L/14
86.5
83.6
37.2
36.4
58.0
37.1
87.4
67.3
76.6
70.9
EVA-CLIP
VIT-L/14
91.5
89.4
47.2
47.8
64.2
47.9
89.2
77.9
80.4
73.8
Long-CLIP
ViT-L/14
95.8
95.6
44.2
52.5
62.8
46.3
90.0
76.2
73.5
67.9
FineCLIP
VIT-L/14
73.4
82.7
40.1
46.2
60.8
53.4
FG-CLIP
VIT-L/14
97.4
96.8
66.7
66.1
68.9
50.9
93.7
81.5
76.1
69.0
models' performance can be found in Appendix D.1.
Table 5. Comparisons on General Multimodal Benchmarks.
Bounding Box Classification. To assess the model's local information recognition capabilities, we conduct zero-shot testing on COCO-va12011 (Lin et al., 2014), LVIS (Gupta al., 2019), and Open Images (Kuznetsova et al., 2020), following the protocol of (Jing et al., 2024). This evaluation focuses on how well the model can classify objects within bounding boxes using only textual descriptions. Similar to the fine-gre ine-grained understanding, we integrate known bound- ing box information from the benchmark with ROIAlign to obtain localized dense representations. Using all categories textual inputs, we perform matching and recognition for each bounding box, evaluating Top-1 accuracy.
RefCOCO testA
Method
GQA
POPE
val
testB
LLaVA-v1.5+CLIP
61.9
85,9
76.2
83.4
67.9
+1.0
+0.9
+5.2
+3.1
+7.0
LLAAA-
62.9
86.8
81.4
86.5
74.9
measures the model's ability to recognize novel objects. Our findings indicate that FG-CLIP achieves leading perfor- mance in apen-vocabulary detection tasks, highlighting its effectiveness recognizing and localizing novel objects.
4.3. Comparisons on Image-level Task
As shown in Table 2, FG-CLIP achieves leading perfor- mance in bounding box classification with the help of the regional contrastive learning strategy. Notably, Long- CLIP (Zhang et al., 2024), fine-tuned from CLIP using long texts, shows a significant decline in performance, indicat- ing that long texts affect regional information granularity. Furthermore, FineCIIIP uses region alignment data and in- corporates a real-time self-distillation scheme, leading to meaningful improvements. While FineCLIP makes signif- icant progress, FG-CLIP excels it by integrating regional and global information. This approach enhances FG-CLIPP's ability to accurately recognize and classify regions within images, highlighting the effectiveness of FG-CLIP's training strategy.
Long/short Caption Image- Retrieval. To evaluate retrieval performance comprehensively, we conduct exper- iments on both long caption and short caption image-text retrieval tasks. For long-text retrieval, we follow the proto- col of Long-CLID CLIP and use the 1K subset of ShareGPT4V (Chen et al., 2024a) provided by it as the testset. Addi- tionally, we incorporate a more challenging long caption image-text pair dataset from DCI (Urbanek et al., 2024), consisting of 7,805 pairs, into the evaluation. For short-text retrieval, employ the classic MSCOCO 5K (Lin et al., 2014) and Flickr 1K (Young et al., 2014) evaluation sets, which are widely used benchmarks for assessing image-text alignment models. As shown in Table 4, FG-CLIP achieves significant performance improvements in both long/short caption image-text retrieval tasks. The model's s ability to handle diverse caption lengths highlights its versatility and robustness in multimodal alignment.
Open-Vocabutt Vocabulary Object Detection. To further evaluate the fine-grained localization capability of our method, we employ FG-CLIP as the backbone for downstream open- vocabulary detection tasks. Following prior work (Wu al., 2024a), employ two-stage detection architecture, F- VTT, with frozen visual encoder. The comparative results are summarized in Table 3. Consistent with previous stud- ies, we report the box AP at IoU 0.5 for base, novel, and all categories (APg A.t and AP on OV-COCo. Notably, Apnoeel is the primary focus of interest, as it
Zero-shot Image Classification. We evaluate the zero- shot classification performance of our model on ImageNet- 1K (Deng et al., 2009) and ImageNet-v2 (Recht et al., 2019). As illustrated in Table 4, despite being marginally behind EVA-CLIP, which trained on a larger dataset, FG- CLIP demonstrates stable classification performance with enhanced regional and textual understanding capabilities

---

FG-CLIP: Fine-Grained Visual and Textual Alignment
Table 6. Ablation study results for FG-CLIP. This table compares the performance of different configurations of our FG-CLIP model across multiple evaluation metrics, including long caption image-text retrieval (DCT), short caption image-text retrieval (MSCOCO), bounding box classification CCOCO-val2013 and fine-grat fine-grained understanding (FG-OVD). The results highlight the incremental improvements achieved by incorporating global contrastive learning (Ligohhl)), regional contrastive learning (Lregional) ) and hard fine-grained negative samples learning (Lard)).
Long Retrieval 12T
Short Retrieval I2T
BBox Classification Top-5
Fine- GraieGG Understanding hard
Method
T21
T21
Top-1
medium
casy
CLIP
45,5
43.0
51.8
32.7
44.2
72.3
12.0
23.1
22.2
FG-CLIP Stagel
58.3
57.5
64.6
44.9
47.2
74.2
21.8
41.6
36.2
+Stage2 (Legtohat)
62.7
61.2
64.4
46.4
46.8
73.6
25.4
46.8
42.9
+Stage2 (Lgtohat
62.4
61.1
64.7
45.7
53.7
81.2
24.5
47.1
49.5
+Stage2 Lregooaa
61.8
60.6
64.1
45.4
52.3
79.7
46.1
66.6
68.7
compared to the original baseline, CLIP. Additionally, when compared to Long- and FineCLIP, both of which aim to enhance fineggraiee recognition capabilitiis, our model exhibits a notable advantage in classification accuracy.
outperforms CLIP in all metries, highlighting the impor- tance of fine-grained training data. Further improvements are observed when adding Lglobal in Stage 2, particularly in long caption image-text retrieval (DCI (Urbanek et al., 2024)) and fine-graing understanding (FG-OVD (Bianchi et al., 2024)). This underscores the effectiveness of detailed caption data combined with global contrastive learning in improving model performance.
4.4. Comparisons on General Multimodal Benchmarks
We compare FG-CLIP visual feature extractor for mul- timodal large language models with our baseline, CLIP. Specifically, we conduct experiments using LLa -LaVA-v1.5-7B (Liu et al., 2023b), which itself is trained using CLIP. To en- sure a fair comparison, all parameter configurations are kept consistent with those in the original LLLVV, and the model trained using the data provided by LLaVA. Our evaluation focuses benchmark sets related to attribute analysis, ob- ject localization, and output hallucination, which are GQA (Hudson & Manning, 2019), RefCOCO (Kazemzadeh et al., 2014), and POPE (Li et al., 2023c), respectively.
Regional Contrastive Learning. We introduce regional contrastive learning Tregional to evaluate its impact on cap- turing detailed image regions. Compared to configurations using only Lytohat, adding Lregional leads to substantial improvements in bounding box classification accuracy from 46.8% to 53.7%, and FG-OVD easy dataset accuracy from 42.9% to 49.5%. These gains highlight the effectiveness of L regional in refining the model's 's ability to understand fine- grained details within specific image regions. Moreover, this component maintains strong performance in both retrieval and classification tasks, demonstrating its versatility.
As shown in Table 5, FG-CLIP achieves certain improve- ments on GQA, which involves attribute-based question answering, and on POPE, which evaluates output halluci- nation. Additionally, it demonstrates significant gains ReiCOCO, benchmark set that involves both attribute analysis and object localization. These results indicate the effectiveness of FG-CLIP's training strategy and the data construction, which are specifically designed to enhance fine-grained recognition and regional alignment. We pro- vide more results in Section D.3.
Hard Fine-Grained Negative Samples Learning. We in- corporate hard fine-grained negative samples learning Lhard to distinguish subtle differences in semantically similar but distinct region-text pairs. By comparing configurations with and without Lhard, we observe significant improvements in FG-OVD performance. Accuracy on the hard dataset in- creases from 24.5% to 46. %, while on the medium dataset it rises from 47.1 % to 66.6% and on the easy dataset it jumps from 49.5% to 68.7%. These results underscore the importance of Lhard in distinguishing subtle semantic differences. Hard fine-grained negative samples learning effectively addresses challenge cases, thereby enhancing the model's stability and discriminative power.
4.5. A Ablation Study
To systematically evaluate the contributions of different components in our FG-CLIP model, we conduct an ablation study with results summarized in Table 6.
Global Contrastive Learning and Detailed Recaptioning Data. We start by comparing the original CLIP model with FG-CLIP Stage 1 and Stage 2 incorporating global contrastive learning Lytobat- The results demonstrate that generating detailed captions significantly enhances perfor- mance across various tasks. Specifically, FG-CLIP Stage
5. Conclusion
In this work, we introduce Fine-Grained CLIP (FG-CLIP), a novel approach that significantly advances fine- mne-grained un- derstanding. By integrating advanced alignment techniques
8

---

FG-CLIP Fine-Grained Visual and Textual Alignment
with large-scale, high-quality datasets and hard negative samples, FG-CLIP captures global-level and region-level semantic details and distinguishes subtle differences more effectively. Extensive experiments across diverse down- stream tasks validate the model's 's superior performance. In addition, we propose FineHARD as a unified dataset that combines high-quality region-specifie annotations with chal- lenging ane-grained negative samples, offering a valuable resource for advancing multimodal research. Looking ahead, exploring the integration of more sophisticated multimodal models and expanding dataset diversity will be crucial for pushing the boundaries of fine-grained understanding,
touch, vision, and language datasct for multimodal alignment. In ICML, pp. 14080-14101, 2024.
Gabeff, V., Rußwurm, M., Tuia, D., and Mathis, A. Wildclip:
Scene and animal attribute retrieval from camera trap data with aunain- vision-language models. IJC,, pp. 1-17, 2024.
Gu, J., Meng, X., Lu, G., Hou, L.,, Minzhe, N., Liang, X., Yao, I.,, Huang. R., Zhang, W., Jiang, X., et al. Wukong: A 100 million large-scale chinese cross-modal pre-training benchmark. In NewrlPS, volume 35, pp. 26418-222 (6418-26431, 2022.
Gupta, A., Dollar, P., and Girshick, R. Lvis: A dataset for large vocabulary instance segmentation. In CVPR, pp. 2019.
He. K., Gkioxari. G., Dollár, P., and Girshick, R. Mask r-cnn. In ICCV, pp. 2961-2969, 2017.
Impact Statement
This paper aims to advance the field of Machine Learning, which has broad implications for society. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Momentum contrast for unsupervised visual representation learnin.. In CVPR, pp. 9729-9738, 2020.
Hong. W., Wang. W., Ding, M. Yu, W., Lv, Q. Wang. Y., Cheng, Y., Huang, S.. Ji, J.. Xue, Z. et al. Cogvlm2: Visual language models for image and video understanding. arXiv preprint arXiv:2408. 16500, 2024.
References
Alayrac, J.,,., Donahue, J., Lue, Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: a visual language model for few-shot lcarning. In NewrlPS, volume 35, pp. 23716-23736, 2022.
Honnibal, M., Montani, L., Van Landeghem, S., Boyd, A., et al. spacy: Industrial-strest strength natural language processing in python. 2020.
Hou, X., Liu, M.. Zhang, S., Wei, P., and Chen, B. Salience detr: Enhancing detection transformer with hierarchical salience filtering refinement. In CVPR, pp. 17574-17583, 2024.
Bianchi, L.. Carrara, F., Messina, N., Gennaro, C., and Falchi, F. The devil is in the fine-grained details: Evaluating open- vocabulary object detectors for fine-grained understanding. In CVPR, pp. 22520-22529, 2024.
Hudson, D. A. and Manning, C. D. Gqa: A new dataset for real- world visual reasoning and compositional question answering. In CVPR, pp. 6700-6709, 2019.
Changpinyo, S., Sharma, P., Ding, N., and Soricut, R. Conceptual 12m: Pushing web-scale image-text pre-training recognize long-tail visual concepts. In CVPR, pp. 3558-3568, 2021.
Jing, D., He, X., Luo. Y.. Fei, N., Yang. G., Wei, W., Zhao, H., and Lu, Z. Fineclip: Self-distilled region-based clip for better ine-grained understanding. In NeurlPS, 2024.
Chen, L.. Li, J., Dong. X., Zhang, P., He, C., Wang. J., Zhao, F,, and Lin, D. Sharegpt4y: Improving large multi- modll models with better captions. In ECCV, pp. 370-387, 2024a.
Kazemzadeh, S., Ordonez, V., Matten, M., and Berg, T. Refer- itgame: Referring to objects in photographs of natural scenes. In EMNLP, pp. 787-798, 2014.
Chen, Z., WU, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, I.,, al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguisti tasks. In CVPR, pp. 24185-24199 2024b.
Kim, D., Angelova, A., and Kuo, W. Contrastive feature masking open-vocabulary vision transformer. In ICCV. pp. 15602-15612, 2023a.
Cheng. T., Song, L., Ge, Y., Liu, W., Wang. X., and Shan, Y. Yolo- world: Real-time open-vocabllary object detection. In CVPR, pp. 16901-16911, 2024.
Kim, D., Angelova, A., and Kuo. W. Region-aware pretraining for opcn-vocabulary object detection with vision transformers. In CVPR, pp. 11144-11154, 2023b.
Deng, J., Dong, W., Socher, R., Li, L.-J,, J,, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-255, 2009.
Kuznetsova, A., Rom, H., Alldrin, N., Uijlings. J., Krasin, I., Pont- Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection scale. IJC,, 12867):195-11981 2020.
Dosovitskiy, A. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2021.
Dubey, A.. Jauhri, A.. Pandey, A., Kadian, A., Al-Dahle. A., Letman, A.. Mathur, A., Schelten, A., Yang, A.. Fan, A., et al. The llama 3 herd of models. arXiv preprint 20Xv:2240..21788. 2024.
Laurençon, H., Marafioti, A., Sanh, V., and Tronchon, L. Building and better understanding rision-languag models: insights and future directions. arXiv preprint arXiv:2408. 12637, 2024.
Li, J., Li, D.,, Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML. pp. 19730-19742, 2023a.
Fu, L.,, Datta, G., Huang, H., Panitch, W. C.-H., Drake, J., Ortiz, J., Mukadam, M., Lambeta, M., Calandra, R., and Goldberg, K. A

---

FG-CLIP Fine-Grained Visual and Textual Alignment
Li, J., Xic, C., Wu, X., Wang, B., and Leng, D. What makes good open- open-vocabulary detector: A disassembling perspective. arXiv preprint arXiv:2309.0022
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C.,, and Chen, M. Hicrarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06122. 1(2)3, 2022.
2023b.
Li, L. H., Zhang, P. Zhang. H., Yang, J., Li, C., Zhong, Y. Wang, L., Yuan, L.. Zhang, L., Hwang. J.-N., et al. Grounded language- image pre-training. In CVPR. pp. 10965-1099777
Recht, B., Rnelofs, R., Schmidt, I.,, and Shankar, V. Do imagenet classifiers generalize to imagenet? In ICMI, pp. 5389-5400, 2019.
2022.
Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W. X., and Wen, J.- R. Evaluating object hallucination in large vision-language models. In EMNLP, 2023c. URL https:://ppeteee net/
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarezyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A. Laion-400m: Open dataset of clip-filtere 400 million image- text pairs, arXiv preprint arX21:211.002114, 2021.
Li, Z., Yang, B., Liu, Q. Ma. Z., Zhang, S., Yang, J., Sun, Y., Liu, Y., and Bai, X. Monkey: Image resolution and text label are important things for large multi-modal models. In CVPR, 2024.
Schuhmann, C.. Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. Laion-5b: An open large-scale datasct for training next generation image-tex text models. In NeurlPS, volume 35, pp. 25278-25294, 2022.
Lin. C., Sun, P., Jiang, Y., Luo. P.. Qu, L., Haffari, G. Yuan, Z. and Cai, J. Learning object-languae alignments for open- vocabulary object detection, arXiv preprint wxXi:2221..488. 2022.
Sharma, P., Ding, N., Goodman, S., and Soricut, R. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, pp. 2556-2565, 2018.
Lin. T.-Y.. Maire, M.. Belongie, S., Hays, J., Perona, P., Ramanan, D.. Dollár, P. and Zitnick. C. L. Microsoft coco: Common objects in context. In ECCV, pp. 740-755, 2014.
Sun, Q. Fang, Y., Wu, I.., Wang. X., and Cao, Y. Eva-clip: Improved training techniques for clip at scale, arXiv preprint 2iiv:2303.13389 2023.
Lin, W., Zhao, 7-, Zhang, X., Wu, C., Zhang, Y., Wang, Y., and Xie, W. Pme-clip: Contrastive language-image pre-training using biomedical documents. In MICCAL, pp. 525-536, 2023.
Sun, Z., Fang, Y., Wu, T., Zhang, P., Zang. Y., Kong, S., Xiong, Y., Lin, D., and Wang, J. Alpha-clip: A clip model focusing on wherever you want. In CVPR, pp. 13019-13029, 2024.
Liu, C., Zhang, Y., Wang. H.. Chen, W., Wang, F., Huang, Y., Shen, Y.-D., and Wang. I., Efficient token-guided image-text retrieval with consistent multimodal contrastive training. IEEE Transactions on Image Processing, 32:3622-3633 2023a.
Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint 24Xhv:403.055534 2024.
Liu, H., Li, C., Wu, Q. and Lee, Y. J. Visual instruction tuning. In NewrIPS, volume 36, pp. 34892-3499 2023b.
Urbanek, J., Bordes, E, Astolfi, P., Williamson, M., Sharma, V., and Romero-Soriano, A. A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions. In CVPR, pp. 26700-26709, 2024.
Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al. Mmbench: Is your multi-modal model an all-around player? In ECCV, pp. 216-233, 2024.
Wang. B., Xie, C., Leng, D., and Yin, Y. laa: Inner-adaptor archi- tecture empowers frozen large language model with multimodal capabilities. In AAAI, volume 39, pp. 11035-21043, 2025.
Ma, C., Jiang, Y., Wu, J., Yuan, Z., and Qi, X. Groma: Localized visual tokenization for grounding multimodal large language models. In ECCV, pp. 417-435, 2024.
Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al. Qwen2-vl: Enhancing vision- language model's perception of the world at any resolution. arXiv preprint arXiv: 24..2229991111 12191, 2024.
Minderer, M., Gritsenko. A., and Houlsby, N. Scaling open- vocabulary object detection. In NeurlPS, volume 36, 2024.
Mokady, R., Hertz, A., and Bermano, A, H. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.0973: 2021.
Wu, S., Zhang, W., Xu, L., Jin, S., Li, X., Liu, W., and Loy, C. C. CLIPSelf: Vision transformer distills itself for open- vocabulary dense prediction. In ICLR, 2024a. URL https: / //openzeview net forum? Did--jzzjccsss.
Pan, J., Ma, Q. and Bai, C. A prior instruction representation framework for remote sensing image-text retrieval. In ACM MM, pp. 611 611-620, 2023.
Parelli, M. Delitzas. A., Hars, N., Vlassis, G., Anagnostidis, S., Bachmann, G., and Hofmann, T. Clip-guided vision-language pre-training for question answering in 3d scenes. In CVPR. pp. 5607-5612. 2023.
Wu, W., Zheng, K., Ma. S., Lu, F., Guo, Y., Zhang, Y.. Chen, W., Guo, Q. Shen, Y., and Zha, Z.-J. Lollip: Improving language- image pre-training for long text understanding. arXiv preprint arXiv:2110.0544 2024b.
Peng, Z., Wang, W., Dong, I.,, Hao, Y., Huang, S., Ma, S., Ye, Q., and Wei, F. Kosmos-2: Grounding multimodal large language models to the world. In ICLR, 2024.
Wu, Z.. Chen, X., Pan, Z., Liu, Liu, W., Dai, D., Gao, H., Ma. Y., Wu, C.. Wang, B., et al. Deepseek-v12: Mixture-of vision- languge models for advanced multimodal understanding. arXiv preprint arXiv:2412. 24..22112..0002, 2024c.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agar- wal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language su- pervision. ICMI., pp. 8748-8763, 2021.
Xie, C., Li, C., Zhang, B., Han, J., Zhen, X., and Chen, J. Memory attention networks for skeleton-based action recognition. In IJCAL, pp. 1639-1645, 2018
10

---

FG-CLIP: Fine-Grained Visual and Textual Alignment
Xie, C., Cai, H., Li, J., Kong, F., Wu, X., Song, J., Morimitsu, H., Yao, L., Wang, D., Zhang, X., et al. Cemb: large-scale chinese cross-modal benchmark. In ACM MM, pp. 4219-4227, 2023.
Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Li, H., Zhao, W., He, 7., et Miniepm-v: A gpt-4v level mllm your phone. arXiv preprint AXiv:2408.0/800 2024.
Young, P. Lai, A., Hodosh, M., and Hockenmaier, J. From image descriptions visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
Zareian, A., Rosa, K. D., Hu, H., and Chang, S.-F. Open- vocabulary object detection using captions. In CVPR, pp. 14393- 14402, 2021.
Zhang. B., Zhang, P. Dong, X., Zang. Y., and Wang. J. Long- clip: Unlocking the long-text capability of clip. In ECCV, pp. 310-325, 2024.
Zheng, K., Zhang, Y.. Wu, W., Lu, F., Ma, S., Jin, X., Chen, W., and Shen, Y. Dreamlip: Language-image pre-training with long captions. ECCV, pp. 73-90, 90, 2024.
Zhong, Y., Yang. J., Zhang, P., Li, C., Codella, N., Li, H., Zhou, L., Dai, X., Yuan, L., Li, et Regionclip: Region-based based 2022.
language-iman image pretraining. In CVPR, pp.
Zhou, C., Loy, C., and Dai. B. Extract free dense labels from clip. In ECCV, pp. 696-712, 2222a.
Zhou, X., Girdhar, R., Joulin. A., Krähenbühl, P. and Misra, tecting twenty-thouse thousand classes using image-level supervision. In ECCV, pp. 350-368, 2022b.

---

FG-CLIP Fine-Grained Visual and Textual Alignment
A. Examples of Curated Visual Grounding Data
Figure 2 shows visual grounding examples utilized in our experiments. Each example comprises an image, accompanied by its corresponding long and short captions, as well as multiple region-speeific annotations, each with a detailed description.
long caption
The image shows sewing pruject in progress. There sewing machine with needle and presser foot engaged the process. A piece of fabric with a pattern of pink shapes on an orange background is being sown; the odges of the fabric are neatly folded over one another, and appears to be pianed in place to hold its shape during the sewing. The sowing machine positioned cutting when working with fabric... In the background, there is white ceramic pot partially visible with green plant that adds a touch of color to the setting. The lighting seoms to be indoor, likely coming from overhead, as there are no shadows visible on abjects,
pink cutting mat that bas grid pattern, which is typically used ensure accurate
the ovenal lighting uppears even comsistent... The style up photogmaph, capturing the details of the sewing process with high clarity. There people characters in the image, so there are no emotions to cunvey. The image is factual and does not contain any subjective interpretation of emotions or style.
image is a close-
short caption
sewing machine with semi-finished pink baby hat --------ggion caption-........
pink cutting mat that a grid pattern which is typically used ensure accurate cutting when working with fabric region caption.........
white ceramic pot partially visible with green plant that adds touch of color to the setting
regionn caption
baby hat with white spots to be finished
-eo...---ong caption-
The image presents a bedroom scene centered on a large bed with a tall, upholstered headboard. The beadboard is light brown and has smooth texture without any visible patterss. Two reading lights are mounted on the wall on each side of the headhoard, extending outward with flexible beight that seems suitable for reading. The bed frame is also light brown and has contemporary design... On the bed, there is a white duvet and couple of pillows, with one of them being a smaller pillow that matches the color of the reading lights, and the other being larger pillow that appears have different color, possibly a neutral tone. foot of the bed, there small wooden side lable with a black metal frame, which has minimalistic design with flat surface and single visible drawer. The table's color paletic includes brown and black, complementing the overall room lones.. The lighting within the image is not discernible due to the lack shadows or highlights that could indicate specific light source. Instead, the room appears be lit by ambient light that creates a soft glow, giving the space cozy atmosphere... The style of the image is photograph. The edges and shadows suggest natural light source and the clarity of the objects points towards a high-resolution image captured by camern. sh..r. caption
and small light the end. These lights positioned at
Modemn bedside table simple style the hotel room metal furniture
eesen-rrgion caption
small weoden side table with a black motal frame which has minimalistic design with a flat surface and a single visible drawer
region caption--....... white duvet and a couple of pillows with one of them
....----ggionn region caption-.......
smaller pillow that matches the color of the reading lights and a larger that appears to have di differennt color
-....---oong caption-........
This image depicts aerial view of an oil rig's ss platform. In the foreground, there is worker engaged in activity the odge of the platform, wearing safety gear such as jeans, yellow hard bat, knee pads, and brown work boots. The worker appears to be suspended or working at height, secured by a safety harness and other equipment The rig's infrustructure is comprised metal heams, pipes, and various work platforms... The background reveals large body of water, which could be the occan. On the water's surface, there is green helipad with circular symbol, suggesting that this is a designated area for landing helicopters. The helipad is connected to the rig by a walkway. The weather appears to be clear, with bright sunlight easting shadows on the structure of the rig. The image is taken from a high vantage point, looking downwards towards the worker and the rig's platform, highlighting the scale and complexity of the offshore oil operation... The style of the image is a high-resolation photograph, capturing the intricate details of the industrial setting. There are no people or characters other than the worker, so there are no emotions to convey. The description is factual and does not include any subjective interpretations or personal opinions about the image.
shor..--hort caption..........
An overhead view of the legs and harness of industrial painter suspended from the underside of an offshore rig derrick, high above the deck below.
-e-in---rgion caption worker engaged in activity the edge of the platform wearing safety gear such as jeans yellow hard hat knee
pads and brown work boots regiion caplion a circular symbol suggesting that this designated area for landing helicopters
rtgionn caption
a high vantage point looking downwards towards the worker and the rig's platforn highlightng the scale and complexity of the offshore oil operation
Figure Examples of curated visual grounding data.
12

---

FG-CLIP: Fine-Grained Visual and Textual Alignment
B. Positive and Negative Descriptions Related to Image Regions
To generate hard fine-grained negative samples, we modify the attributes of bounding box descriptions while keeping object names unchanged. Figure illustrates examples of positive and corresponding negative descriptions for image regions.
very long waoden jetty. jutting out into calm blue wooden lake with mountains sunsetting in background
10: Vvery long iron jony. jutting out into calm pink wooden lake with mountains sunsetting in background
T: vayy long silver jetly, jutting inlo calm lurqueise wooden lake with mountains sunsctiing background
22: Very long brass jetty, jutting into calm lavender wonden with mountains sunserting background
13: 'vryy long golden jetty, julling out inlo calm peach weoden like with mosntains sunsetting background'
Very long bronze jerty, jutting out into calm mint wooden with mountains sunsetting background
15:: vveyy long copper jerty, jutting out into calm coml wooden lake with mountains sunsetting in hackground
very long steel jerry, jutting out a calm magents wooden lake with mountains sunsetting in background" "Ph very long chrome jetty, jurting out into calm charcoal wooden lake with mountains sunsetting in background lake with mountains sunsetting background" 1t: vvry long aluminum jetty, julting out into calm amber wooden lake with mountains sunsctiing in background'
very long tiaaiuu jetty, jutting out calm plum
positivp-.....
british shorthair with red collar
Twrtish shorthair with green collar
'I: british shorthair with copper collar
british shorthair with grass green collar
13: british sharthair with deep blue collar'
4:: beiiish shorthair with white collar
15: british sharthair with black collar
1s: beiiish shorthair with light blue collar'
brilish shorthair with a silver collar' W: british shorthair with dark blue collar'
9:: brilish shorthair with a yellow collar"
posilive-.......
A two-box fasthack configuraiion combined with cleverly concenled air intakes and lightweight roof aseembly
10: 'A brass riveted configuration combined with eleverfy concealed air intakes and copper moo assemblyy
1: AA copper bolted configuration combined with cleveriy concealed intakes and copper roof assembly
2: ' metallic braided configuration combined with cleverly conccaled air intakes and motallic roof assembly'
35 AA woven mesh configuration combined with cleverly concealed air intakes and woven roof assembly
41: ' translucent grid configuration combined with cleverly concesled air intakes and translucent roof assembly.
15: 'A diamondd-latt configuration combined with cleverly concealed intakes and diamoed-plate rool assemblyy
66: ' chrome- chrome-plated configuration combined with cleverly conceuled air intakes and chrome plated roof assembly
77 'A striped metal configuration combined with cloverly concealed air intakes and stripal roof assembbly.
W: 'A glossy ceramic configuation combined eleverly concenled intakes and glassy assemblyy
9.: 'A beaded mesh configuration combined with cloverly concealed air intakes and beaded roof usemblyy
.......n positive in alps
....... negutives
light blue
Swiss alps'
11 la grass green train in Swiss alps'
cupper train
alps'
15: à deep blue train in Swin Swiss alps!
train Swiss
'5" à black train in Swiss alps'
tunquoise train Swiss alps'
77 la brown train in Swiss alpe
la silver train in Swiss alpss
19: la grey train in Swiss alps'
positive
beige couch with brick background
ncgatives--........
W: 'A turquoise couch with silver background
A copper couch with silver background"
2: ' turquoise couch with cupper background
19: 'A copper couch with turquoise hackgrounct
14: A bronze couch with silver background
19: ' bronze couch with copper background'
16: 'A copper couch with bronze background'
7: 'A turquoise couch with bronze background
8:: ' silver couch with turquoise background
9: 'A silver couch with copper background"
Figure 3. Examples of positive and negative deseriptions related image regions.
13

---

FG-CLIP: Fine-Grained Visual and Textual Alignment
C. Visualization Comparison
As illustrated in Figure 4, we present a comparison of similarity matrix visualizations for different methods using challenging sample images. We utilize the dense image feature extraction strategy introduced by (Zhou et al., 2022a). In the figure, warmer colors (e.g.. yellow) denote higher similarity, whereas cooler colors (e.g.. blue) indicate lower relevance. Our goal for the model to precisely comprehend and interpret the tineggrineed details within the images.
In the first scenario, the image contains three dogs of different colors, and we compute the similarity matrix using only the phrase "Black dog" with each image token. It can be observed that CLIP and EVA-CLIP fail to accurately identify the target dog, FineCLIP captures some relevant tokens but not all, whereas FG-CLIP identifies a larger number of relevant tokens, demonstrating superior performance.
In the second scenario, the image contains multiple black entities, and we compute the similarity matrix using only the phrase "Black nose", which occupies very small area within the image. CLIP fails to identify the target, while EVA-CLIP and FineCLIP locate the target but also respond to many other black regions. In contrast, FG-CLIP accurately identifies the target, showcasing its precision in fine- fineggrained localization.
In the third scenario, the image features gemstones of three different colors, and we compute the similarity matrix using only the phrase "Red gemstone". Both CLIP and EVA-CLIP fail to locate the target at the bottom and exhibit high responses gemstones of other colors. FineCLIP shows slightly lower localization accuracy compared to FG-CLIP, which precisely distinguishes between the colors of different gemstones and achieves accurate localization.
dog
FineCLIPP
Black nose
CLIP
EVA-CLIP
FineCIIP
FG-CLIP
Red
CLIP
EVA-CLIP
FincCLIP
FG-CLIP
Figure 4. Feature visual comparisons of different methods.
Additionally, we utilize FG-CLIP to conduct correlation analysis between different input texts and the same image. The results in Figure 5 indicate that FG-CLIP provides precise positional understanding of different targets within the image. This demonstrates the model's s stable visual localization capabilities and its fine-grained understanding of image content.
To evaluate the impact of hard fine-grained negative samples learning, we further provide the qualitative results in Figure After performing hard negative sampling, our FG-CLIP can capture the regions more accurately. For example, the highlighted region of "Man in red clothes" with hard negative loss 1st row shows significantly better than that without hard negative loss.
14

---

FG-CLIP: Fine-Grained Visual and Textual Alignment
Bookshelf
Black Chair
Black Monitor
Lamp with Black Shade
Green potied plants
Laptop
Cell
Apple
Banana
Dread
Tat
Person
Figure 5. Feature visual comparisons of different input texts.
wil loge Man in clothes
fawe Man blue clothes
clothes
*us Man blue clothes
wio lowa legged stool
fsue legged
+/uy legged
+ legged
Paper cup
lan. cup
-bo Paper cup
Aev cup
Figure 6. Performance of hard fine-grained negative samples learning.
15

---

FG-CLIP: Fine-Grained Visual and Textual Alignment
D. Further Experiments
Table 7. Comparisons of different methods on fine-grained benchmark.
Image with region
Positive and Negative Region Descriptions
CLIP
EVA-CLIP
FineCLIP
FG-CLIP
Origin: A table made dark brown wood.
0.73
0.79
0.62
1.0
I: A tahle made of pink wood.
0.0
0.58
0.0
2: table made of dark brown paper.
0.48
0.34
0.48
3: A lable made of light grey wood.
0.27
0.0
0.39
0.55
4: A tahle made of dark brown wool.
0.27
0.11
0.03
0.21
S: A table made of yellow wood.
0.38
0.44
0.38
0.14
6: A table made of dark brown velvel.
1.0
0.63
0.0
0.01
7: A table made dark brown text.
0.94
0.72
0.38
0.55
8: tahle made of grey wood.
0.53
0.47
9: A lable made of dark brown plastic,
0.45
1.0
0.86
0.26
10: A table made of green wood.
0.47
0.16
0.64
0.46
Origin: A brown leather bandbug.
0.62
0.79
0.78
1.0
I: A brown metal handbug.
0.45
0.88
0.56
0.37
2: A brown text hundbbgg.
0,53
1.0
0.90
0,50
3: A brown wool handbag.
0.0
0.0
0.63
0.13
4: A orange leather handhag.
0.32
0.74
0.34
0.53
S: brown paper handbag.
0.60
6: light orange leather handbag.
0.08
0.72
0.08
0.56
7: A brown glass handhag.
1.0
0.59
0.75
0.0
8: A durk red leather handbag.
0.43
0.59
0.15
0.93
9: A dark yellow leather handbag.
0.31
0.82
0.11
0.73
10: A purple leather handbag.
0.21
0.91
0.0
0.27
Origin: A brown dog with black nose.
0.75
0.0
0.76
1.0
1: A brown dog with light red nose.
0.37
0.45
0.64
0.80
2: brown dog with dark yellow nose.
0.10
0.93
0.90
3: light blue dog
black nose.
4: brown dog with yellow nose.
0.38
0.40
10
0.86
5: A red dog with black nose.
1.0
1.0
0.59
0.32
6: A brown dog with light orange
0.16
0.67
0.71
0.85
7: A brown dog with light yellow nose.
0.0
0.58
0.83
0.83
8: brown dog with light nose.
0.26
0.44
0.48
0.52
9: green dog
black nose.
0.88
10: A brown dog with dark purple
0.41
0.12
0.48
0.73
Origin: A light blue plastic trash can.
0.89
0.95
0.52
1.0
I: light blue
trash
0.90
1.0
1.0
0.77
2: A dark purple plastic trash can.
0.60
0.70
0.03
0.64
3: A light blue wool trush can.
0.68
0.57
0.37
0.29
4: A dark green plastic trash can.
0.92
0.74
0.99
0.65
5: A dark orange plastic trash can.
0.58
0.87
6: black plastic trash can.
0.66
0.65
0.77
7: purple plastic trash
0.68
0.90
0.36
0.75
8: A light blue crochct trash can.
0.0
0.0
0.10
0.0
9: A light blue glass trash can.
1.0
0.72
0.52
0.64
10: A light orange plastic trash can.
0.68
0.77
0.02
0.88
Origin: A red plastic bucket.
0.97
0.53
0.70
1.0
I: A green plastic bucket.
0.94
0.77
0.46
0.68
2: A red metal bucket.
0.83
0.29
0.54
0.61
3: A rod crochet bucket.
0.0
0.10
0.10
0.39
4: red ceramic bucket.
0.58
0.06
0.15
0.58
5: red fahric bucket.
0.55
0.05
0.0
0.43
6: A red stone bucket.
1.0
0.10
0.84
0.48
7: A red rattan hucket.
0.47
0.51
0.58
0.0
8: A red wool bucket.
0.39
0.14
0.31
9: A yellow plastic bucket.
0.77
1.0
0.67
10: A light green plastic bucket.
0.70
0.87
0.45
0.28
D.I. Comparison of Different Methods on Fine-Grained Benchmark
As shown in Table 7, we select several samples from the test set of FG-OVD (Bianchi et al., 2024) and visualize the comparison results different methods. We employ the testing strategy detailed in Section 4.2 and match the text with localized dense feature. The similarity scores computed between regions and texts are normalized, where the sentence with the lowest similarity is assigned score of 0.0, and the sentence with the highest similarity is assigned score of 1.0.
16

---

FG-CLIP! Fine-Grained Visual and Textual Alignment
FG-CLIP demonstrates strong capability in identifying these extremely difficult samples, whereas other methods struggle achieve comparable performance.
D.2. Performance Comparison Identical Datasets
Table 8. Comparisons of different methods on the dataset.
Method
Data Source
SCCO-Box-Top-1
COCO-Retrioval-
10CO-Retrieeva--11
FineCLIP FineCLIP FG-CLIP (Ours)
FineCLIP (CC2.5M)
50.7
54.4
40.2
FG-CLIP (12M)
53.5
59.6
46.2
FG-CLIP (12M)
56.1
65.9
47.1
To evaluate the effectiveness of our proposed FG-CLIP method, we conduct experiments on the same dataset to ensure fair comparison. Specifically, we compare FineCIIP and FG-CLIP using the 12M dataset due to time constraints, instead the larger 1.6B+12M setup. From Table 8, the substantial improvements (Row -> Row 2 and Row 2 -> Row 3) highlight that both our proposed dataset and model architecture are significant for FG-CLIP.
D.3. Performance on General Multimodal Benchmarks
Table 9. Comparisons on General Multimodal Benchmarks.
RefCOCO testA
MMBench EE
MMBench- CN
Method
GQA
POPE
val
testB
dev
test
dev
test
LLaVA-v1.+CLIIP
61.9
85.9
76.2
83.4
67.9
65.1
66.5
58.2
58.4
+1.0
+0,9
+5.2
+3.1
+7.0
+1.5
+0.2
+0.6
+0.9
62.9
86.8
81.4
86.5
74.9
66.6
66.7
58.8
59.3
In addition to GQA, POPE, and RefCOCO, we conduct experiments on other general multimodal benchmarks (Liu et al., 2024). The experimental results Table show that LLaVA with FG-CLIP achieves better performance.
17


Reviews:
[{"review_id": "fPEJIwwuLH", "reviewer": "Reviewer_gRNB", "summary": "This paper proposes to fine-grained CLIP by introducing additional high-quality data and designing specific loss functions for training. As for the data, original CLIP only use short global caption data, while this work introduces long global caption, region-level caption, and region-level negative caption for training. For model training, this work introduces 2 stage training framework, with the first stage focusing on global-caption training and the second stage using all data. Experiments on several fine-grained downstream tasks demonstrate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your positive feedback and recognizing the non-trivial contributions of our work. In response to your specific questions, we provide detailed explanations below, aiming to clarify any concerns. \n\n__1. Response to questions in Experimental Designs Or Analyses__\n\nThanks for your insightful comments and suggestions. Your points about considering long and short captions at both global image level and region-level boxes, as well as the introduction of hard negatives during global image level training, are indeed critical aspects that deserve further elaboration.\n\nAs mentioned in our manuscript (line 264), the captions used for region-level training are derived from the global long caption using the SpaCy tool. Specifically, we extract region-specific descriptions from the global long caption to generate a single type of text description for each region-level box. This method is efficient and already contains detailed information about the objects within those regions. Given that this approach adequately describes the objects, it provides sufficient context for regional contrastive learning without the need for additional caption types. Therefore, we only utilized this single type of caption for Regional Contrastive Learning. This streamlined approach not only enhances computational efficiency but also ensures that the model focuses on high-quality, relevant textual descriptions, leading to improved performance.\n\nRegarding the consideration of hard negatives during global image level training, we acknowledge that this could potentially enhance the robustness of the model. However, there are practical challenges associated with implementing this approach:\n- Limitations with Long Captions: We introduce the process of creating challenging fine-grained negative samples in Section 3.2. Specifically, we modify attributes of bounding box descriptions while keeping the object names unchanged. For global long captions, which describe multiple regions within an image, generating meaningful hard negatives by modifying attribute words for each region becomes complex. The resulting negative samples might deviate too far from the original caption, losing their effectiveness as \"hard\" negatives. Consequently, they may no longer serve as hard negative examples for training.\n- Limitations with Short Captions: As illustrated in Figure 1 of our manuscript, global short captions often lack detailed descriptions of individual objects or regions. This makes it difficult to create meaningful hard negatives through attribute modification, as the short captions may not contain enough fine-grained information to make such modifications impactful.\n\n__2. Response to questions in Supplementary Material__\n\nThis is indeed an interesting phenomenon and can be attributed to several factors.\n\nThe fine-grained hard negatives we created involve modifying only a few attribute words in the captions. In some cases, these modifications may still result in captions that are quite similar to the original descriptions of the corresponding image regions. This similarity can cause FG-CLIP to give relatively high scores to these hard negatives because the model perceives them as still being relevant to the image regions. Additionally, in certain scenarios, the modified attributes might correspond to minor changes within the image region, which do not significantly alter the overall visual content. Consequently, the model continues to assign high scores due to the minimal perceptual difference between the original and modified captions.\n\n__3. Response to Question 1 in Questions For Authors__\n\nThank you for your question regarding how we sample K regions in a batch (Line 216). To clarify:\n\nK is not a fixed number but rather represents the total number of valid bounding boxes (bbox) across all images within a batch. This means that K dynamically adjusts based on the actual number of available regions in the batch. This approach ensures flexibility and adaptability in handling batches with varying numbers of regions without introducing artificial constraints.\n\nWe acknowledge that this explanation may have been unclear in the initial submission. To avoid any potential misunderstandings, we will provide a more detailed clarification in the final version of our manuscript.\n\n__4. Response to Question 2 in Questions For Authors__\n\nWe introduce this process in lines 263 to 272 of our manuscript. Specifically, we utilize SpaCy to parse the captions and extract referring expressions. These extracted expressions are then fed into a detection model to generate corresponding bounding boxes. Non-maximum suppression is applied to eliminate overlapping bounding boxes, retaining only those with predicted confidence scores higher than 0.4. This method allows us to effectively link textual referring expressions with their corresponding visual elements, facilitating more accurate and contextually relevant training for our model.", "claims_and_evidence": "The proposed idea are verified by sufficient ablation studies with the newly introduced data. Table 6 demonstrates the effectiveness of global contrastive learning, regional contrastive learning, and hard fine-grained negative samples learning.\n\n=======================\n\nAfter carefully reading the rebuttal and other reviews, I think more in-depth discussions of hard-negative should be included in the main text. The challenges mentioned in the rebuttal can possibly be addressed by some carefully designed methods. I also agree with Reviewer 1aL6 that the novelty is somehow limited as both fine-grained CLIP and hard negative ideas have been explored before. But I acknowledge that the manuscript is solid work as other reviewers. So my final decision is borderline.", "methods_and_evaluation_criteria": "Yes, the proposed method is evaluated on fine-grained tasks, e.g., fine-grained understanding, bounding box classification, long caption image-text retrieval, and open-vocabulary object detection. It also maintains strong performance on coarse-grained tasks, e.g., short caption Image-Text Retrieval, and Zero-shot Image Classification. The evaluation metrics follow the standard practices for each task.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Overall, this work has made non-trivial contributions by introducing better training data with fine-grained details, which improve the performance for fine-grained tasks. The training is conducted in two levels: global image level and local region level. Besides, hard negatives are also introduced during training. \n\nHowever, this work did not sufficiently discuss all possible combinations of training loss. For example, both long and short captions are introduced for global image level training. Do we also need to consider long and short captions for each region-level boxes? Hard-negatives are only introduced at the local region level training. Do we also need to consider hard negatives during global image level training? Should we consider hard-negatives for both long and short caption? \n\nThere could be many loss function terms if we consider all the situations above. Can the author provide more in-depth analysis why they use the loss function proposed in the paper?", "relation_to_broader_scientific_literature": "The work is built upon CLIP, and enrich the fine-grained understanding ability with newly-introduced captions with fine-grained details.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. In Line 216, how did you sample K regions in a batch. Is this from the same image? If so, how did you sample K regions if the number of regions in the current image is less than K?\n2. In Line 218, how did you segment the full-captions into small phases and establish the connections with visual boxes?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "QOnJOLP9Hc", "reviewer": "Reviewer_6zz8", "summary": "The proposed method introduces Fine-Grained CLIP (FG-CLIP) for enhancing CLIP's fine-grained understanding capabilities. The authors propose three components to address this challenge: First, they generate 1.6 billion long caption-image pairs for capturing global-level semantic details. Second, they construct a high-quality dataset with 12 million images and 40 million region-specific bounding boxes along with detailed captions. Third, they introduce 10 million hard fine-grained negative samples to help the model learn to distinguish between subtle semantic differences. For the proposed method, the training occurs in 2 stages. First stage focuses on global contrastive learning using the long caption-image pairs. Second stage incorporates regional contrastive learning and hard negative samples. The proposed model extends CLIP with position embeddings that can handle longer text (up to 248 tokens vs. the original 77) and uses ROIAlign to extract region-specific features from images. The paper further showcases results on various downstream tasks, including fine-grained understanding, open-vocabulary object detection, image-text retrieval, and general multimodal benchmarks.\n\n## Update after rebuttal\nThe authors have answered all the concerns mentioned in the weaknesses and experimental design sections. Furthermore, additional visualizations and results on datasets like OpenImages and LVIS strengthen their claims. They also propose a lower-resource training setup and plan to release a distilled model, improving accessibility. I increase my score to accept.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for your constructive comments. We address your concerns below.\n\n__1. Response to Questions 1-3 in Experimental Designs Or Analyses and Weakness 2__\n\nThank you for your comments regarding the improvement factor and fair comparison. The improvement of FG-CLIP stems from both the model architecture and the proposed new dataset. To support this, we provide the differences between FineCLIP and FG-CLIP. Moreover, we train our proposed dataset on FineCLIP to conduct a fair comparison.\n\n1.1 Differences between FineCLIP and our FG-CLIP. \n\n- A large-scale dataset with diverse captions can enhance FG-CLIP's fine-grained understanding capabilities. Specifically, our FG-CLIP integrates short captions with long captions, while FineCLIP only utilizes short captions during global contrastive learning. Moreover, we use a large-scale dataset (i.e., 1.6B+12M) rather than the small dataset of FineCLIP (i.e., 2.5M).\n- FG-CLIP discards the self-distillation strategy used in FineCLIP, which introduces significant additional computational overhead. For example, when using a single GPU with a batch size of 32 samples, incorporating the self-distillation strategy increases memory usage from 25GB to 75GB during training and reduces FPS from 25 to 9.8. This is mainly due to additional feature extraction for each bounding box in the input images.\n- Hard fine-grained negative sample learning helps FG-CLIP distinguish subtle differences in semantically similar pairs.\n\n1.2 We then train our proposed dataset on FineCLIP. Due to time constraints, we perform this experiment on the 12M dataset, instead of the larger 1.6B+12M setup. From the table, the substantial improvements (Row 1 -> Row 2 & Row 2 -> Row 3) highlight that both our proposed dataset and model architecture are significant for FG-CLIP.\n\nMethod|Data Source|COCO-Box-Top-1|COCO-Retrieval-I2T|COCO-Retrieval-T2I\n-|-|-|-|-\nFineCLIP|FineCLIP (CC2.5M)|50.7|54.4|40.2\nFineCLIP|FG-CLIP (12M)|53.5|59.6|46.2\nFG-CLIP (Ours)|FG-CLIP (12M)|56.1|65.9|47.1\n\n__2. Response to Question 4 in Experimental Designs Or Analyses__\n\nThank you for this insightful question. The qualitative results in https://anonymous.4open.science/r/ICML_RE-3CF6/partshow_pic.png show that FG-CLIP is able to capture different parts of images that are strongly related to the input text, demonstrating its fine-grained understanding capabilities. We follow the same experimental settings as Appendix C.1.\n\n__3. Response to Question 5 in Experimental Designs Or Analyses__\n\nThank you for pointing this out. In the \"hard\" fine-grained task, only one attribute within the caption is replaced. Similarly, there are two replaced attributes within the caption in the \"medium\" task. However, regional contrastive learning primarily enhances region-text alignment in FG-CLIP, which may struggle to distinguish these fewer replaced attributes. To this end, we propose hard negative sample learning to further improve the overall performance of FG-CLIP. We will make this clearer in the manuscript.\n\n__4. Response to Weakness 1__\n\nThank you for raising this point. We follow the same experimental settings as Appendix C.1 and further provide the qualitative results in https://anonymous.4open.science/r/ICML_RE-3CF6/fgshow.png . After performing hard negative sampling, our FG-CLIP can capture the regions more accurately. For example, the highlighted region of  \"Man in red clothes\"  with hard negative loss in 1st row shows significantly better than that without hard negative loss.\n\n__5. Response to Weakness 3__\n\nThank you for this suggestion. We compare several baselines and FG-CLIP on the validation set of OpenImages. We conduct bounding box classification and follow the settings of COCO. The results further demonstrate the effectiveness of FG-CLIP’s fine-grained capabilities.\n\nMethod|Top-1|Top-5\n-|-|-\nCLIP|18.02|40.90\nEVA|8.83|20.90\nFineCLIP|18.10|42.16\nFG-CLIP|20.60|47.43\n\n__6. Response to Questions For Authors__\n\nThank you for these insightful questions. For the first four questions, we refer you to our earlier responses. Here, we focus on addressing the 5th question regarding computational resources.\n\nFor researchers with fewer computing resources, we suggest discarding the first stage and directly training FG-CLIP on the dataset of 12 million images in the second stage. We conduct this experiment using 4×NVIDIA A100 GPUs, and the training process takes approximately 14 hours. Our experimental results in https://anonymous.4open.science/r/ICML_RE-3CF6/result_different_data.md indicate that this approach achieves slightly lower performance compared to the original settings but remains effective.\n\nAnother possible method is to distill a model based on our pre-trained weights, which significantly reduces the computational burden. We plan to release detailed guidelines and tools for implementing this distillation process in future work, making our approach more accessible to researchers with limited resources.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence across multiple experiments. The paper's claims regarding improved fine-grained understanding are substantiated in Table 1 and Appendix C.2, while claims about enhanced bounding box classification capabilities are validated in Table 2. The assertions about superior long caption image-text retrieval and open-vocabulary object detection are demonstrated through comprehensive results in Table 3. Furthermore, the claims about FG-CLIP's effectiveness when used as a backbone for Large Multimodal Models for attribute analysis, object localization, and reducing output hallucination are thoroughly supported by Tables 4 and 5. The paper also provides visualization evidence in Appendix C.1, further reinforcing these claims. Overall, most claims presented in the introduction are backed by quantitative and qualitative results, providing a strong empirical foundation for the paper's contributions.\n\nThere is one concern however, further discussed in the “Experimental Designs Or Analyses”, namely whether individual contribution of the new data vs. the new model was substantiated.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are appropriate for the problem at hand. The authors properly identify limitations in existing CLIP models regarding fine-grained understanding and propose targeted solutions.", "theoretical_claims": "No theoretical claims in the paper; the method proposes empirical and dataset curation. Equations 1-5 are standard loss functions.", "experimental_designs_or_analyses": "- There is, however, a major question of fairness in comparison since the current method is trained on data different from that of previous methods. It would be more convincing if prior methods, such as FineCLIP, should have been trained on this new dataset to determine whether the improvements stem primarily from the data quality or the model architecture itself.\n- The paper doesn't clearly show if using more training data is the main reason for better results, especially since FG-CLIP's local and global feature approach is similar to FineCLIP.\n- Testing other recent methods on the proposed dataset would make the comparisons more complete.\n- More examples showing how the model describes different parts of images would better prove its fine-grained understanding.\n- The results in Table 6 raise questions because adding regional contrastive learning doesn't improve performance much on “hard” and “medium” fine-grained tasks, which needs more explanation.", "relation_to_broader_scientific_literature": "The key contribution follows previous research on vision-language models like FineCLIP and RegionCLIP, focusing specifically on region-level representation techniques. The key contribution of the proposed novel approach is related to advanced region-based reasoning and cross-modal alignment methods previously explored in multimodal learning research.", "essential_references_not_discussed": "I think all the relevant literature has been discussed and referenced.", "other_strengths_and_weaknesses": "Strengths:\n- Achieves state-of-the-art performance on multiple downstream tasks, including fine-grained understanding, object detection, and image-text retrieval.\n-Extensive dataset curation with 1.6 billion long caption-image pairs, 40 million region-specific annotations, and 10 million hard negative samples provides a valuable contribution (beyond the model).\n-Comprehensive ablation studies demonstrate the individual contributions of each proposed component (global contrastive learning, regional alignment, and hard negative samples).\n\nWeakness:\n- While Table 6 shows quantitative improvements from hard negative sampling, there is limited qualitative analysis (such as t-SNE visualizations) demonstrating how subtle differences are actually differentiated.\n-The paper shows impressive performance improvements, but it is not entirely clear if these come from the larger dataset or the new model design. While FG-CLIP offers interesting ways of handling local and global features, more detailed studies would help us understand precisely what is driving these performance gains.\n- OpenImages or specialized fine-grained datasets such as NUS-WIDE that contain naturally fine-grained categories would strengthen the evaluation.", "other_comments_or_suggestions": "No typos found.", "questions_for_authors": "- Is the proposed method better because of the design or just because of more data? Could you test this by training other models like FineCLIP on the new proposed dataset?\n- Why doesn't regional contrastive learning improve results much on hard and medium fine-grained tasks in Table 6? This seems unexpected.\n- Can you show visual examples of how your hard negative sampling helps tell apart similar items? Some visualizations would make your point clearer.\n- Have you tested your approach on other datasets like OpenImages or NUS-WIDE? This would show your method works broadly.\n- Your training needs lots of computing power (160×910B NPUs and 8×NVIDIA H800 GPUs). Could researchers with less resources still use your approach? Are there more efficient ways to get similar results?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CECZKiUfAL", "reviewer": "Reviewer_1aL6", "summary": "This paper proposed FG-CLIP, A region level contrastive learning model for fine-grained image representation. Through training on large scale synthetic data. This model achieve strong performance compared to previous methods on fine-grained region-level tasks like fine-grained understanding, OVD, image-text retrieval and image classification, serving as a promising vision fundation model compared to CLIP.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__1. Response to Question 1 in Essential References Not Discussed__\n\nThanks for your comments. We discuss the difference between FG-CLIP and FineCLIP and provide experimental results in the first response to reviewer 6zz8. We refer you to that response for more details. \n\n__2. Response to Question 2 in Essential References Not Discussed and Question 1 in Questions For Authors__\n\nThanks for your suggestion to include a comparison with Alpha-CLIP on region level classification. We test Alpha-CLIP using its \"clip_b16_grit1m+mim_fultune_4xe\" weight across three testing configurations: Alpha Map, RoIAlign, and Crop&Resize. The Alpha Map method involves creating a mask based on the label box information and then combining this mask with the entire image, which aligns with Alpha-CLIP 's training objective. The RoIAlign method is a testing approach in our FG-CLIP. Meanwhile, the Crop&Resize method refers to cropping the image region corresponding to the box and resizing it to the resolution required by ViT. The table in https://anonymous.4open.science/r/ICML_RE-3CF6/compare_with_alphaclip.md shows FG-CLIP consistently outperforms Alpha-CLIP across different configurations. We will add these results to our manuscript.\n\n__3. Response to Major weakness__\n\nThank you for your insightful comment regarding the methods of regional contrastive learning and hard negative sampling, as well as the novelty and contribution of our dataset.\n\n__3.1 Regional Contrastive Learning and Hard Negative Sampling__\n\nWe have discussed the difference between FG-CLIP and FineCLIP in an earlier response. Here, we highlight distinctions between hard negative sampling of FG-CLIP and ALBEF.\n\nALBEF proposes a strategy to sample hard negatives for its ITM task. Specifically, ALBEF samples hard negatives using softmax-normalized image-to-text and text-to-image similarity to find in-batch negatives, selecting one negative text/image per mini-batch. In contrast, FG-CLIP conducts a novel pipeline to create challenging fine-grained negative samples. As shown in lines 236 to 251 of the right column of the manuscript, we modify the attributes of bounding box descriptions while keeping the object names unchanged. We generate 10 negative samples for each positive sample. This process generates subtle variations where objects may appear similar but differ in specific details. \n\n__3.2 Contribution of the Dataset__\n\nOur work leverages two high-quality datasets. In the first stage, we utilize an extensive dataset of 1.6 billion long caption-image pairs to capture global-level semantic details. In the second stage, we employ a carefully curated dataset of 12 million images with 40 million corresponding bounding boxes and captions, which are specifically designed to provide fine-grained annotations. We also generate 10 million challenging fine-grained negative samples, improving the model's ability to distinguish subtle differences. These datasets enable the model to achieve superior performance on various benchmarks. In summary, we have validated the entire synthesis pipeline used to create these datasets, including innovative techniques such as generating challenging fine-grained negative samples, which provide reference for others. We plan to make this dataset public to support further research in visual grounding and fine-grained understanding.\n\n__4. Response to Question 2 in Questions For Authors__\n\nTo quantitatively discuss the diversity of our dataset, we compare it with other fine-grained datasets such as LVIS and V3Det. We extract and aggregate category labels from captions generated through steps involving CogVLM and YOLO-world. The following table compares the number of images and unique category labels across different datasets. Notably, even when sampling an equivalent number of images (243k), our dataset yields more unique category labels than V3Det, indicating higher diversity. We visualize the category labels of sampled data (equivalent to 243k images) using t-SNE plots in https://anonymous.4open.science/r/ICML_RE-3CF6/data_tsne_pic.png The visualization also shows that our dataset has a more diverse set of category labels at the same image scale. As the dataset scales up to 12M images, the diversity in category labels and captions increases significantly.\n\nDataset|Image|Caption|Category Label\n-|-|-|-\nLVIS|164k|1.27M|1.2k\nV3Det|243k|1.75M|13k\nOurs (sampling 243k)|243k|815k|21k\nOurs|12M|40M|128k\n\nTo defend the claim of our paper, we evaluate our model on LVIS. The results in the following table show FG-CLIP achieves SOTA performance.\n\nMethod|Top-1|Top-5\n-|-|-\nCLIP|24.79|46.63\nEVA|14.36|29.11\nFineCLIP|23.29|44.17\nFG-CLIP|28.55|52.60\n\n__5. Response to Question 3 in Questions For Authors__\n\nWe conduct experiments on other LVLM benchmarks at https://anonymous.4open.science/r/ICML_RE-3CF6/add_result_mm_compare.md . The experimental results show that LLaVA with FG-CLIP achieves better prefomance.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The evaluation criteria generally make sense. But more FG dataset like OV-LVIS, V3Det should be considered.", "theoretical_claims": "The theroretical claims are correct as they mostly from previous method (FineCLIP).", "experimental_designs_or_analyses": "The experimental designs are valid.", "relation_to_broader_scientific_literature": "I believe the technical novelty is kind of limited as the region level contrastive learning and hard nagtive samples are direct adoption of existing methods. the main contribution is a large scale synthetic region-text paired data.", "essential_references_not_discussed": "The key contribution of model and loss is from FineCLIP published in NeurIPS 2024. The authors should directly discuss this in the methods section. \n\nExperiment results on region level classification should consider Alpha-CLIP in CVPR 2024 for comparison.", "other_strengths_and_weaknesses": "Major strength: Strong performance is achieved compaired to previous methods.\n\nMajor weakness: The methods of regional contrastive learning and hard nagative sampling are direct adoption from FineCLIP and ALBEF, which make the model contribution marginal. The major contribution is the dataset. As the dataset is also collected and synthesised without new techinical contribution, the novelity of this paper is kind of limited.", "other_comments_or_suggestions": "No", "questions_for_authors": "1. Alpha-CLIP should be considered for comparison.\n\n2. I personally believe rewritted captions from LVLM like CogVLM and detection through YOLO-world will restrict the diversity of the data. As the model also benchmarked on data that are either synthesized (Share4V) or not that finegrained enough (COCO, ImageNet, RefCOCO...).\nThus I believe:\n* Data diverity analysis is important to be quantitatively discussed.\n* Consider more fine-grained dataset like LVIS, V3Det to defend the claim of the paper.\n\n3. If this paper want to claim that FG-CLIP serves as better visual encoder for LVLM, merely tested on GQA and POPE is not enough. More numbers on other LVLM benchmarks should be included.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RB8tCpuokT", "reviewer": "Reviewer_pLwP", "summary": "Based on the observation that CLIP struggles to handle with fing-grained understanding tasks, this paper propose \n1. a larger dataset including abundant images, bounding boxes and captions;\n2. to incorporate long captions, short captions and hard negative strategies to enhance CLIP's ability during training. \n\nThe extensive experiments demonstrate the effectiveness of this paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__1. Response to Weakness 1__\n\nThanks for pointing out this problem. We perform ablation studies on the number of hard negative samples. Specifically, we test configurations with 1, 5, and 10 hard negative samples per positive sample. Our experiments show that 10 hard negative samples per image-text pair yield the best performance. We agree that this ablation study adds significant value to our manuscript and will include it in the final version. Thank you for pointing out this important aspect.\n\n| Fine-Grained Understanding| hard | medium | easy |\n|----------|--------------|--------------|---------|\n| FG-CLIP (1 hard negative sample)  | 44.49 | 65.87 | 67.82 | \n| FG-CLIP (5 hard negative samples)| 45.86 | 66.78 | 67.21 | \n| FG-CLIP (10 hard negative samples)  | 46.40 | 67.15 | 68.59 | \n\n__2. Response to Weakness 2__\n\nThank you for your comment regarding the need for more visualizations of hard negative samples. We appreciate the opportunity to provide additional insights into how our method, FG-CLIP, benefits from hard negative sampling. We provide more visualization in https://anonymous.4open.science/r/ICML_RE-3CF6/fgshow.png . Specifically, we extract the dense image feature and visualize the similarity matrix to qualitatively analyze the impact of hard negative sampling. As illustrated in the figures, our FG-CLIP can capture the regions more accurately after performing hard negative sampling. For example, in the first row, the phrase \"Man in red clothes\" is accurately identified with hard negative loss, whereas without it, the model struggles to capture the correct region.\n\n__3. Response to questions in Other Comments Or Suggestions__\n\nThank you for your valuable suggestion regarding the comparison of our dataset with previous ones such as LAION, COCO, and others. We agree that a detailed discussion in the form of a table would better highlight the unique strengths and contributions of our dataset.\n\nIn addition to our visual grounding dataset of 12 million images with corresponding bounding boxes and captions, we have also incorporated an additional 1.6 billion image-text pairs in the first stage of training. The dataset in the first stage is generated using a large multimodal model to produce higher-quality fine-grained long captions compared for capturing global-level semantic details.\n\nWe have compared our dataset with several related datasets, as shown in the table below. Overall, our dataset stands out in terms of scale and quality, particularly in its fine-grained annotations and challenging negative samples. Here are the key points of comparison:\n- Scale: Our dataset contains the largest number of images, bounding boxes, and captions among all the datasets except LAION. While LAION has the largest number of captions (2B), the quality of these captions is often noisy and inconsistent. Our supplementary dataset adds 1.6B high-quality, fine-grained long captions, enhancing the overall utility of our data.\n- Bounding Boxes: Among the widely used datasets, only COCO provides bounding box annotations. However, our dataset surpasses COCO by an order of magnitude, with 40M bounding boxes compared to COCO's 1.5M.\n- Hard Fine-Grained Negative Samples: A distinctive feature of our dataset is the inclusion of 10M hard fine-grained negative samples. These samples help the model differentiate subtle differences in semantically similar pairs, thereby improving its performance across various downstream tasks.\n\n|Dataset|Image |Image caption| Bounding box| Region caption| Hard fine-grained negative sample | \n| --- | --- | --- | --- | --- |--- |\n|LAION-2B| 2B| 2B|0|0|0|\n|Flickr30k|30K|150K|0|0|0|\n|CC3M|3M|3M|0|0|0|\n|COCO|330K|330K|1.5M|1.5M|0|\n|Ours in stage1|1.6B|1.6B+1.6B|0|0|0|\n|Ours in stage2|12M|12M+12M|40M|40M|10M|", "claims_and_evidence": "Yes, the paper claims that original CLIP model cannot well cope with fine-grained understanding tasks. \nAnd this claim is supported by experimental results in Table 1 to Table 6.", "methods_and_evaluation_criteria": "Yes, the proposed method and dataset makes sense for the problem.", "theoretical_claims": "This paper doesn't include any theoretical claims.", "experimental_designs_or_analyses": "Yes, I have checked the experiment section. The experimental designs are sound to evaluate the fine-grained understanding ability of CLIPs (including original CLIP, EVA-CLIP, etc. and this paper proposed FG-CLIP).", "relation_to_broader_scientific_literature": "1. This paper enhances fine-grained understanding ability of CLIPs.\n2. A larger dataset is proposed for the community.", "essential_references_not_discussed": "No, all related works are discussed in the paper.", "other_strengths_and_weaknesses": "Strengths:\n\n1. A well-written paper to follow. \n2. Effective module design. \n\nWeaknesses:\n\n1. Limited ablation and disscussion with negative sampling. How many hard negative samples per image-text pair?\n2. More visualization on hard negative samples.", "other_comments_or_suggestions": "This paper construct a high-quality visual grounding dataset, with 12 million images and corresponding bounding box and captions. It would be better to discuss this dataset with previous ones like LAION, COCO, and so on in a Table.", "questions_for_authors": "Please see above weaknesses and comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The submission received relatively positive final ratings (1 Accept, 2 Weak Accept, and 1 Weak Reject). All three positive have mentioned that almost all their concerns have been addressed. The sole negative reviewer one didn't respond during the discussion period. Thus, we think that there is no more concerns about this submission, and recommend **Weak Accept** if there is room in the program.
Decision: Accept (poster)