Contrastive Localized Language-Image Pre-Training

Authors: Hong-You Chen, Zhengfeng Lai, Haotian Zhang, Xinze Wang, Marcin Eichner, Keen You, Meng Cao, Bowen Zhang, Yinfei Yang, Zhe Gan
Primary area: foundation or frontier models, including LLMs
Keywords: pre-training, CLIP, multimodal

Abstract:
Contrastive Language-Image Pre-training (CLIP) has been a celebrated method for training vision encoders to generate image/text representations facilitating various applications. Recently, CLIP has been widely adopted as the vision backbone of multimodal large language models (MLLMs) to connect image inputs for language interactions. The success of CLIP as a vision-language foundation model relies on aligning web-crawled noisy text annotations at image levels. Nevertheless, such criteria may become insufficient for downstream tasks in need of fine-grained vision representations, especially when region-level understanding is demanding for MLLMs. In this paper, we improve the localization capability of CLIP with several advances. We propose a pre-training method called Contrastive Localized Language-Image Pre-training (CLOC) by complementing CLIP with region-text contrastive loss and modules. We formulate a new concept, promptable embeddings, of which the encoder produces image embeddings easy to transform into region representations given spatial hints. To support large-scale pre-training, we design a visually-enriched and spatially-localized captioning framework to effectively generate region-text pseudo-labels at scale. By scaling up to billions of annotated images, CLOC enables high-quality regional embeddings for image region recognition and retrieval tasks, and can be a drop-in replacement of CLIP to enhance MLLMs, especially on referring and grounding tasks.

Decision: reject
Reviewer scores: 5, 6, 6, 1

Reviews and discussion:
<thread>
creation date:
1729669894908

thread_type:
official_review

<thread_object>
creator: reviewer_mads

<content>
summary:
The paper proposes to combine ITC with region text prediction. The authors show that a data labelling pipeline at scale for region proposal combined with a image-text dataset leads to model outperforming CLIP baseline for MMLLM among other tasks.

strengths:
The author's model is well engineered to take latency into context while adding additional blocks.
The data engine uses synthetic captions rather than alt-text to generate data for the box predictor which is lesser explored.
The authors show improvement in multiple tasks over CLIP.

weaknesses:
This paper would have been strong if other works did not explore better pretraining over CLIP for open vocabulary computer vision tasks. However, that is not true anymore. This work ignores all the advancement in the field of pretraining image-text representation models.
Several works in combining SSL with ITC exist and show better localization performance e.g. MaskCLIP and SILC. Moreover, BRAVE at ECCV2024 showed that these works also improve MMLLM performance. 
Moreover, the primary idea of this work has also been explored in "LocCa: Visual Pretraining with Location-aware Captioners" recently.

While I commend the authors for doing an extensive study of their method vs CLIP, we have a different landscape today. It is not fair to claim large improvements without putting other improvements from the field in context. Moreover, while the authors improve upon CLIP, their performance metrics on imagenet or LVIS fall short compared to other works from the field. Therefore I unfortunately have to recommend a rejection for this paper.

questions:
No questions, the work is very easy to read. I commend the authors for that.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730564279383

thread_type:
official_review

<thread_object>
creator: reviewer_fruc

<content>
summary:
This paper enhances the region-level representation in CLIP and introduces a new concept termed "promptable embeddings." The region-level representation is a critical task in this domain. The experimental results appear promising.

strengths:
1. This paper addresses a significant challenge: enhancing the region-level representation of CLIP, which is crucial for various applications, including multi-modal large language models (MLLMs).
2. The experimental results presented in this paper demonstrate promising performance across both image-level and region-level tasks.

weaknesses:
Several related works are not adequately addressed in the manuscript, such as 'UMG-CLIP: A Unified Multi-Granularity Vision Generalist for Open-World Understanding' (ECCV 2024).

questions:
Will the collected fine-grained region-text annotations be made publicly available? This aspect is highlighted as one of the main contributions of the study.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730645509335

thread_type:
official_review

<thread_object>
creator: reviewer_nxpc

<content>
summary:
Contrastive image-text pretraining has achieved significant success in recent years; however, low-quality annotations and the limitations of contrastive learning alone remain challenges. This work aims to enhance CLIP by incorporating fine-grained annotations, enabling the model to scale to billions of data points and demonstrating substantial improvements over the CLIP baseline.

CLOC can be viewed as a practical approach to enhancing dataset quality, with a particular focus on fine-grained grounding details.

strengths:
1. The data refinement process is scalable to billions of samples, demonstrating its practical utility in vision-language pretraining.

2. The approach is intuitive and straightforward, leading to improved grounding capabilities.

3. The architecture is easily adaptable for downstream grounding tasks, enhancing versatility.

4. This method effectively handles large-scale datasets (billions-level in this work), with comprehensive experiments conducted despite significant computational demands.

5. In addition to widely used recaption, this work also include region phrase candidates extraction and open-vocabulary detection with extracted phrases.

weaknesses:
1. Visually-enriched and spatially-localized models, such as QWEN, rely on robust training to achieve high-quality outputs. These models contribute essential knowledge to the process, with the quality of grounding pseudo-labels also largely dependent on these off-the-shelf models.

2. Significant work has already been done to improve dataset captions, as seen in studies like *Improving Multimodal Datasets with Image Captioning* by Thao et al. (NeurIPS 2023). Integrating object grounding information into vision-language pretraining is also an established concept, exemplified by X-VLM (Zeng et al., ICML 2021). Although CLOC’s approach to grounding differs somewhat from X-VLM, these differences are minor and do not detract from the overall contribution.

questions:
1. What explains the significant improvement that CLOC achieves over LLaVAW and MME-C? In contrast, GQA, which heavily relies on precise object grounding, shows only limited improvement.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730710745109

thread_type:
official_review

<thread_object>
creator: reviewer_ljyf

<content>
summary:
This paper addresses a critical limitation of CLIP by introducing CLOC (Contrastive Localized Language-Image Pre-training), which enhances CLIP's capability to understand region-level semantics while maintaining its strong image-level performance. The key contributions include: (1) a novel concept of "promptable embeddings" that allows easy transformation of image embeddings into region representations, (2) a lightweight Prompter module that effectively extracts region features without compromising CLIP's original capabilities, and (3) a scalable pseudo-labeling pipeline (VESL) that generates high-quality region-text annotations. The approach is validated through extensive experiments across 31 evaluation tasks, showing particular effectiveness in MLLM applications.

strengths:
1. Strong practical value:
    - Addresses a real limitation in CLIP that affects its use in MLLMs
    - Provides a drop-in replacement solution without compromising original capabilities
    - Demonstrates consistent improvements across various downstream tasks
2. Technical novelty:
    - Introduces an elegant concept of "promptable embeddings"
    - Develops a lightweight and effective Prompter module (though it reminds me of prompt encoder of SAM[^1])
    - Proposes a scalable approach to generate region-text annotations
3. Comprehensive evaluation:
    - Extensive experiments across 31 tasks
    - Thorough ablation studies
    - Strong performance improvements in MLLM applications

[^1]: Kirillov, Alexander, et al. "Segment anything." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

weaknesses:
1. Insufficient comparison with related works:
    - Missing comparison with DenseCLIP[^1] and similar region-aware CLIP variants [^2, ^3, ^4, ^5]
    - While RegionCLIP[^6] is mentioned, the differentiation from other similar approaches is not clearly articulated
    - Limited discussion of trade-offs compared to existing solutions
2. Missing implementation details:
    - The region sampling strategy (how 4 regions per image are selected) is not specified
    - The rationale for setting λ as the ratio of images with region labels in the mini-batch is not explained
    - The impact of batch composition (ratio of images with/without region labels) on training is not discussed
    - The threshold (0.9) for similar text filtering lacks justification
    - The definition and extraction method of "leaf entities" in NER is unclear
    - Computational overhead analysis of the Prompter compared to RoI pooling is missing
3. Limited analysis of scalability:
    - The saturation of performance at 300M images for region tasks needs more investigation
    - The impact of the threshold for similar text filtering on training time is not analyzed (in addition to (10) in table 2)

[^1]: Rao, Yongming, et al. "Denseclip: Language-guided dense prediction with context-aware prompting." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.  
[^2]: Gu, Xiuye, et al. "Open-vocabulary object detection via vision and language knowledge distillation." arXiv preprint arXiv:2104.13921 (2021).  
[^3]: Zhou, Xingyi, et al. "Detecting twenty-thousand classes using image-level supervision." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.  
[^4]: Feng, Chengjian, et al. "Promptdet: Towards open-vocabulary detection using uncurated images." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.  
[^5]: Lin, Jiayi, and Shaogang Gong. "Gridclip: One-stage object detection by grid-level clip representation learning." arXiv preprint arXiv:2303.09252 (2023).  
[^6]: Zhong, Yiwu, et al. "Regionclip: Region-based language-image pretraining." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

questions:
1. Region Sampling Strategy
    - How are the 4 regions per image selected during training?
    - Is there any consideration for overlap between regions?
    - How does the sampling strategy affect the model's performance?
2. Training Implementation
    - What is the rationale behind setting λ as the ratio of images with region labels?
    - How do you handle the batch composition in terms of images with/without region labels?
    - Have you experimented with different similar text filtering thresholds besides 0.9?
3. Scalability and Efficiency
    - Why does the performance saturate at 300M images for region tasks but continue to improve for MLLM tasks?
    - How does the computational cost of Prompter compare to traditional RoI pooling approaches?
    - What is the memory overhead of storing region embeddings compared to traditional CLIP?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a pretraining method to enhance the localization ability of contrastive image-text pretraining (i.e., CLIP), which is considered an important topic in the research field as CLIP models are widely used across various applications. The paper presents a new methodology with experimental results across different model sizes (ViT-B/L/H) and tasks. The paper's strengths are its strong motivation, clear writing, and comprehensive experiments.

However, as Reviewer MadS and Reviewer LJyF pointed out, the paper lacks conceptual and experimental comparison with relevant previous works (e.g., MaskCLIP, SILC, BRAVE, etc.). While the reviewers and the AC understand that these previous works may differ in method direction and experimental settings, the AC believes that it would be valuable to compare them as baselines to demonstrate the advantages of the proposed method within the broader context of CLIP localization ability. 

Given the paper's notable strengths, the AC recommends submitting this work by addressing these comparison aspects to future conferences/journals rather than this ICLR.