Contrastive Localized Language-Image Pre-Training
Authors: Hong-You Chen
Zhengfeng Lai
Haotian Zhang
Xinze Wang
Marcin Eichner
Keen You
Meng Cao
Bowen Zhang
Yinfei Yang
Zhe Gan
Venue: ICML 2025 poster (2025)
Keywords: CLIP
MLLM
Foundation Models
Primary area: deep_learning->foundation_models

Abstract: CLIP has been a celebrated method for training vision encoders to generate image/text representations facilitating various applications. Recently, it has been widely adopted as the vision backbone of multimodal large language models (MLLMs). The success of CLIP relies on aligning web-crawled noisy text annotations at image levels. However, such criteria may be insufficient for downstream tasks in need of fine-grained vision representations, especially when understanding region-level is demanding for MLLMs. We improve the localization capability of CLIP with several advances. Our proposed pre-training method, Contrastive Localized Language-Image Pre-training (CLOC), complements CLIP with region-text contrastive loss and modules. We formulate a new concept, promptable embeddings, of which the encoder produces image embeddings easy to transform into region representations given spatial hints. To support large-scale pre-training, we design a visually-enriched and spatially-localized captioning framework to effectively generate region-text labels. By scaling up to billions of annotated images, CLOC enables high-quality regional embeddings for recognition and retrieval tasks, and can be a drop-in replacement of CLIP to enhance MLLMs, especially on referring and grounding tasks.

Contrastive Localized Language-Image Pre- Pre-Training
Hong-You Chen'
Zhengfeng Lai"
Haotian Zhang'
Xinze Wang'
Marcin Eichner' Keen You'
Meng Cao
Bowen Zhang'
Yinfei Yang'
Zhe Gan
Abstract
ages and text by contrastive training on large-scale data collected from the Web. Strong transferability and gen- eralizability have been proven in extensive downstream tasks such as zero-shot image classification and image- text retrieval. Even beyond, language-supervised meth- ods like CLIP have become arguably the default choice of vision backbone for multimodal large language models (MLLMs) (Liu al. 2023 McKinzie et al., 2024) due to their superior prior knowledge in aligning vision and lan- guage (Tong et al. 2024).
CLIP has been a celebrated method for train- ing vision encoders to generate image/text repre- sentations facilitating various applications. Re- cently, it has been widely adopted as the vision backbone of multimodal large language models (MLLMs). The success of CLIP relies on align- ing web-crawled noisy text annotations at image levels. However, such criteria may be insufficient for downstream tasks in need of fine-gr -naiggained vision representations, especially when under- standing region-level is demanding for MILMSs We improve the localization capability of CLIP with several advances. Our proposed pre-training method, Contrastive Localized Language-Image Pre-training (CLOC), complements CLIP with region-text contrastive loss and modules. We for- mulate new concept. promptable embeddings, of which the encoder produces image embed- dings easy to transform into region representa- tions given spatial hints. To support large-scale pre-training, we design visually-enriched and spatially-localizer captioning framework to ef- fectively generate region-text labels. By scal- ing up to billions of annotated images, CLOC enables high-quality regional embeddings for recognition and retrieval tasks, and can be drop- in replacement of CLIIP to enhance MI.LMs, es- pecially on referring and grounding tasks.
As VL research gets increasing attention, various advanced multimodal tasks are demanding stronger vision capabili-
ties. For instance, recent MILMMs (Rasheed et al. 2024
Ren et al. 2024 Lai et al.
2023 Chen al., 2023 Peng
et al. 2023 Zhang al. 2025 have been focusing on more fine-grained understanding tasks that require com- prehension of the semantics at region levels such as visual question answering (VQA) with referring and grounding instructions. These MLLMS are fine-tuned on referring and grounding data with CLIP as the vision backhone, as seen in works like Kosmos-2 (Peng al. (023) and Ferret (You) et al. 2023 Zhang al. 2024). Due to the need for such region-level understanding, CLIP, which aligns entire im- ages with text captions, seems insufficient, as its image-text contrastive loss primarily emphasizes global semantics.
To remedy such core localization capability for CLIP, we ask a challenging and fundamental question: can we pre- train a stronger image encoder (1) with enhanced local- ization capability that can be inherently integrated into MLLMs, (2) and refine CLIP's original image embeddings to the region for zero-shot recognition/vetrivaal?
1. Introduction
Here, we explore a data-driven approach that complements the original CLIP image-text pre-training objective with ex- plicit region- text supervision. Though conceptually simple, several challenges exist. First, it lacks public datasets with region-text annotations at scales large enough for CLIP training, which typically requires hundreds of millions even billions of images. Existing region-text corpus like Visual Genome (Krishna et al. (017)) contains about 108K images, and the largest noisy-labeled grounded dataset GRIT (Peng et al., 2023) features only around 20M images. Indeed, such a deficiency of labeled datasets has probably
Vision-language (VL) pre-training has been an important foundation for the recent tremendous growth of multimodal applications. Contrastive Language-Imag
Pre-training (CLIP) RRadford et al., 2021 Jia et al. (222)) is a success- ful VL representation learning method that connects im-
*Work done while at Apple. 'Apple AI/ML. Correspondence to: Zhe Gan <zhe.gant @ app...com>.
Proceedings of the 1nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Contrastive Localized Language-Image Pre- Training
1. Pseudo-Labeling
2. Encoder Pre-Training
3a. Region-Text Tasks
3b. MLLM Fine- Tuning
"a stunning view"
"coffee mug"
Visually- Enriched Spatially- Captioner
Image Encoder
Text Encoder
Image Encoder
Text Encoder
Image Encoder
"What on the table [ragion]
Prompter
stunning view
Ma large TV
Prompter
Prompter
"blcycle": 0.0
LLM Decoder
"magazine": 0.01
Localized language-iman -iigg contrastive loss
"table": 0.05
"[region] four glass goblets and a bottle of wine."
glass table with top"
"coffee mug": 0.94
Figure I. Our CLOC pre-training framework. (1) visually-enriched and spatially-localized captioning pipeline pscudo-labels bounding boxes with detailed descriptions for key regions. (2) A lightweight Prompter attached on the CLIP image encoder can be prompted transform the image embedding into focuse- features. All parameters are trained end-to-end from scratch with our contrastive localized aunguage-image loss the annotated region-text datasets. After pre-training, (3a) region features can be generated via the Prompter for region-text tasks like object classification a training-free fashion. (3b) The image encoder, along with the optional Prompter, can also strengthen MLLMs fine-tuning by enhancing their fine-grained image understanding capabilities.
limited the literature mainly consider semi-supervised or weakly-supervised approaches somewhat compro-
inal CLIP loss with region-text contrastive loss, where the region embeddings extracted from the image em- bedding by lightweight extractor module conditioned on the spatial hints (i.e., prompts).
mise (Naeem et al. 2023 Yao et al. 2022 2023a).
Second, a plausible solution is to scale up training data in
We design a large-scale pseudo-labeling engine to sup- port CLOC training. We combine visual-enriched image captioners and open-vocabulary detectors for an effective recipe that improves previous practice of region anno- tations (Minderer al., 2024 Peng et al., 2023). This approach yields a two-billion image-text dataset with fine-graing grained region-text annotations, which serves as the foundation for training our CLOCC model.
pursuit of image regions pseudo-labeled with text annota- tions via some open-vocabulary detectors (Minderer et al.) 2024 Zhang al., 2022). Though it seems feasible, we found it non-trivial to design such pipeline the annota- tions are noisy and will greatly affect the final model per- formance. Third, even if the region-text datasets are given, it is under-explored how to effectively train on them in terms of co-designs of training objectives, model architec- ture, and more design details.
Through extensive experiments across evaluation tasks, including standard image-text tasks, newly con- structed region-text tasks, and downstream evaluations with MILMs, we demonstrate that CLOC significantly and consistently outperforms the CLIP counterpart.
To this end, we propose a new pre-training framework illus- trated in In/iiuure 1 named Contrastive Localized Language- Image Pre- Pre-Training (CLOC), to improve CLIP with bet- ter localization capability, especially for MLLMs, by over- coming the above difficulties. Our main contributions are:
We are working on releasing our pre- rre-triined checkpoints and the constructed region-text annotations along with the final version to accelerate future research.
We propose new learning goal, Promptable Embed- dings, that a strong vision encoder should produce image embeddings that can be easily transformed into region representations, given some spatial hints (e.g., box refer- ring or texi prompts). This formulation not only facili- tates the encoder as prior of fine-gras fine-grained VL. alignment, but also enables new possibilities for the interactions be- tween the image encoder and the language decoder.
2. Related Work
Vision encoder pre-training. A popular approach to MILMS such as LLaVA (Liu et al. 2023), typically con- nects a vision encoder (e.g., ViT (Dosovitskiy et al. (021)) to digest visual input and maps them to the input space of the LLM as tokens. Among various types of vision encoders (Qqua et al. 2023 He et al. 2022) language-
To optimize towards the goal, we design simple and min- imal modifications top of CLIP. We augment the orig-
2

---

Contrastive Localized Language-Image Pre- Pre-Training
supervised methods like CLIP (Radford et al. 2021 Jia
to enhance CLIP's localization capabiliites. Our approach is promising, since the advances of these labeling methods can further improve ours in the future.
et al. 2 becomes the popular choice, due to its superior performance on MLLMs reported (Tong et al. 2024
Other training approaches like captioning loss Tschannen et al. 2024) are also popular: Wann et al. (2024) further in- corporates bounding box coordinates. However, they need to train a full encoder-decoder transformer with smaller batch sizes, which is less efficient than CLIP. Also, the vi- sion embeddings are not directly aligned with languages, so more limited for search or retrieval tasks. Therefore, our scope focuses on improving the CLIP approach.
3. CLOC
3.1. From Image- Text to Region- Text Alignment
CLIP (Radford et al. 2021 contrastively aligns the embed- ding from pair of image and text encoders (fr and fr). Let a mini-batch of N image-text pairs {(x; yi) } be sampled from the large-scale training set during each train- ing iteration. The contrastive loss is defined as follows:
Improving localization of CLIP. Since CLIP was intro- duced, many follow-up works have been proposed to im- prove it from various aspects, for different target tasks, and with different approaches. From the aspect relevant to our work, improving the localization capability, most works specifically focus on the downstream dense vision tasks such open- vocabulcyyy detection (Minderer et al.] 2024 Yao et al. 2022 Wu et al., (023). Another, and ar- guably more challenging, direction is to maintain the gen- eralizability of CLIP on image-level tasks while improving localization. Recent works (Nacem et al. 2023 Bica et al.)
Lau := (CI-T 1/2
(I)
1 N
log (ep(eim(f((t))f(()))))
CI-T= ==
Σ
(xsmmsttt(()) fr(,)))
where sim(;,) is the similarity function and 7 is (learn- able) temperature. The CLIP loss LCup averages the sym- metrical contrastive loss in which cross-entropy normalized along image-to-texx and text-to-image axes, respectively.
2024 Dong et al., (023) combine localization eecalization-enhanningg unsupervised objectives with the CLIP loss, but do not at- tempt supervision on large-scale explicit pseudo-labeled data like ours and have more computational overhead. Alpha-CLIP (Sun et al. 2024) shows that the SAM (Kir- illov et al., 2023) can provide useful conditions for CLIP Another relevant works, UMG-CLIP (Shi et al.] 2025). fine-tunes a pre-trained CLIP model for dense vision tasks on 41M image/region-tert dataset. Although there is a technical similarity to ours, the goals and positioning of our work and theirs are quite different. UMG-CLIP is designed primarily for vision-centrie tasks such as detection and seg- mentation, whereas our focus is on MLLMs. MLLM VQA tasks require more extensive language understanding and thus demand large-scale pre-training data. Our approach pre-trains from scratch on up to 2B images thanks our scalable labeling pipeline. Last but not least, we believe that fine- tanintuiigg approach like UMG-CLIP is complemen- tary to CLOOC pretraining rather than conflicting with it.
Conceptually, CCLPP aligns images with their associated text, but it overlooks subimage semantics. We propose aug- menting this with region-text alignment on top of LCUP.. Specifically, assume an image-text pair (at y) can be de- composed into image regions x(1), ...... _o("), and there ex- ist regional captions g'm) that describe the correspond- ing regions ((m)) Thus, the original input (x, y) be- y )) and (x, y) is a special case when the "region" " itself is the whole image. We identify several research questions and will answer them in the following sections:
comes region-text pairs {(x:(1, , y(""),.... (x.(m),
1. Considering the goal is to train an image encoder fr with enhanced localization capability, how should we formulate region-text text alignment goal that improves
fr? We propose a novel learning task called promptable embeddings in Section 3.2]
2. How to properly extract region embedding from fr as an effective joint design? We propose a lightweight promptable region extractor in Section 3.3
Synthetic annotations for pre-training. The literature has been exploring scalable ways to generate high-quality syn- thetic annotations for CLIP. For instance, several works demonstrate that visually- risuall--erichedd image captions improve CLIP (Lai et al., 2024). MOFI (Wu et al. 2024) augments CLIP with an extreme multi-classification task. However, these works only consider image-level annotations but not explicit region-level labels. In the context of dense vision tasks like vocabuocay detection and segmentation, pseudo-labeling in a self-training paradigm has proven an effective approach (Kirillov et al.) 2023 Minderer et al., 2024). We are inspired by these efforts and combine them
3. How to generate meaningful image regions with high- quality captions? Furthermore, in many cases, the ideal region caption 3 (((m)) may not exist in the image-level caption, i.e., y ((m might not be a substring of the orig- inal y. We design an effective and scalable data engine as visually-enriched and spatially-localized labeler to generate high-quality region-text pairs in Section 4 4. With the above considerations, we discuss how to train the model with minimal conflicts towards a drop-in re- placement of the CLIP model in Section 3.4
3

---

Contrastive Localized Language-Image Pre- Pra-irain
CLIP on image-text pairs
CLOC on region-text pairs
"a bedroom with a stunning ocean view large
Text Encoder
"a stunning ocean view"
Text Encoder
LgghtWeight Encodar
LcLoc
CCliP
Xumii
Image
Positional Encodings
Light-Wiiht Enooder
grounding
Encoder
y y.max
Paoling & Projection
Figure 2. CLOC promptable embedding architecture. CLOC builds upon the image embedding from CLIP (before pooling and pro- jection) and transforms it into region-awaa eegren-wware vision embedding given encoded prompt; e.g. positional encodings of box coordinates or regional caption encoded by the CLIP text encoder.
3.2. Promptable Embeddings
(e.g., box), while CLOC predicts region embedding conditioned on instead.
To optimize CLIP with better feature localization and even- tually learn enhanced CLIP vision encoder fr for var- ious VL. downstream tasks, we argue that it will require at least two capabilities. (i) First, the encoder should rec- ognize fine-grained small objects (e.g., this image crop is an "airplane wheel"). (ii) Second, the image embedding produced by the encoder provides a holistic understand- ing such that MILMM can reason hierarchy relationships within the scene (e.g.. "The plane is lowering its front landing gear."). As discussed in Sec- tion 2 many previous works improve CLIP toward object detection tasks thus mainly focusing on (1) only; egg., Re- gionCLIP (Zhong et al. 2022) that crops out image regions and uses them as additional input images to re-train the CLIP encoders for recognizing objects. However, to sup- port comprehensive VL tasks, (i) is necessary but insuffi- cient without (6i).
To this end, decompose the Docation-image-eex triplets as localized region-text pairs. Let x! (m) P Prompter(4"" fr(z)) and y, y.'h) is the caption of the
gion specified by 1!T)) 1! E R is the mn-th box of image represented as two coordinates (i.e.. top-left and bottom- right corners). We then formulate symmetric region-text contrastive loss similar EEquation I
advanced spatial
CR
log P
MN
(m)) (1h))
exp
simmi
(z (m)
frr (y) (mm)) 1)/
== Σ Eifm) exp (sim(s (m) fr (m())/"))))/
where M the number of regions 1(Th) sampled randomly per image. We set M = 4 by default. We will discuss im- plementing the Prompter in Section 3.3 and generating ((m)) with (m)) Section 4 Cr the symmetric con- trastive loss normalized along text-to-region axis, just like in Equation 1] Intuitively, for each region (m) of image i, it contrasts with all the other region captions y of other regions (m') from the same (when (when + j). We define LCLOC = (CR-T + R/) We re-formulate the CLIP loss based on image-text pairs (x, y) into localized language-imasd contrastive loss for region-text alignment based on triplets of ((T), 2r, y). where is location representation such a bounding box, and possibly there are several boxes set (4) per image. To make it compatible with CLIP training, we construct promptable embedding transform module, or in short, re- gion prompter z = P Prompter(1, (x))), that extracts the region embedding specified by from the image embedding fz()). This formulation is inspired by the success of the segmentation model SAM (Kirillov et al. (023) which pre- diets segmentation masks conditioned location prompt
To achieve this, we introduce a new concept, promptable embedding. We consider a scenario similar to MLLM use cases, where answers are generated using CLIP image to- kens alongside a question. We hypothesize that a strong encoder for MILMs should produce image embedding that can easily be transformed into region representations, given location cues.
j) or other images
As the P pomptee is a simple transformer encoder, it al- lows flexible types of prompts besides bounding boxes have used, such as points, free-form referring, text, and etc. We further consider the case where the prompt is free-form text, and leave others for future study. We add ground- ing loss that extracts a region feature from the image (e.g.. picture of the bedroom) given its regional caption (e.g.. "a large TV"), and predicts the bounding box with an MLP regression head, ie..
(m)) Head (=() (")) la,
Cgrounding
4MN
(t.)
4

---

Contrastive Localized Language-Image Pre- Pre-Train
where z(y) P Prompter(Sr(y)) (x)) is the grounded embedding conditioned on the text (encoded by the CLIP text encoder). All the learnable parameters are trained end-to-end. With a scalar A, the overall loss is
Table I. Region-text dataset statistics. We summarize the text token length for both images and regions. Partial statistics of the proprietary datasets revealed by their papers. "The 20M subset of GRIT is released; we removed the invalid URLs.
# of regions img region
L: = CCUP + XCCoOC + Egrounding).
Dataset
per img
text len. len.
text
imgs
FlickrEntities (Plummer al.
2015)
32K
8.7
3.3. CLOC Model Architecture
RefCOCO (Yu et al. 2016
20K
2.5
3.6
RefCOCO+ Yu et al.
2016
20K
2.5
3.5
We implement the promptable embedding introduced in Section 3.2 with minimal extra modules on top of the original CLIP. As illustrated in Figure 2] the CLIP model remains the same for computing LCuP- For LeLoe/Lgromdinge the image embedding is re-used from the CLIP ViT but before the pooled projection and nor- malization fj. To extract the region embedding * = P Prompter(1,) f) (x)) from the image, we consider the lo- cation representation as two coordinates (top-left and bottom-right corners of box), each vectorized by posi- tional encoding. The Prompter is a lightweight one- layer transformer encoder. It takes the positional encod- ings prepended with the image tokens from ViT together as the input, and outputs the region embedding with pooled projection layer. For the grounding loss, we re-use the same CLIP text encoder for encoding the region captions
RefCOCOg (Mao et al. 2016
27K
2.1
8.4
Visual Genome (Krishna et al.
2017
108K
38.0
GRIT (prop.) (Peg et al. 2023)
91M
1.5
4.7
GRIT (released) (Peng al. 2022
17M
1.8
17.2
4.6
Florence-2 (prop.) (Xtao et al. 2024)
126M
5.4
70.5
2.6
OWLv2 (prop.) (Minderer ct al. 20244)
2B
Minderer et al. (2024) on WiT VESL WiT'(Our) VESL WiT+DFN (Ours)
300M
5.1
17.1
3.9
300M
11.6
44.9
2.1
2B
11.5
35.9
2.1
Table 2. Object hallucination in captions. We evaluate different captioners and the captions generated by our pipeline using the CHAIR score (Rohrbach et al. (018);; the lower the better
Model
CHAIR; CHAIR.
InstructBLIP Dai et al. 2023
14.5
30.0
MiniGPT-4 Zhu et al. 2023
8.2
24.2
P Prompter(fr(t)) f' (z)) to predict the bounding boxes with two-layer MLP head. Overall, CLOC only adds lightweight additional parameters of the heads. Note that, the main overheads during forward are from ViT im- age encoding - CLOC reuses it for multiple prompts.
Shikra (Chen et al. 2023
7.0
22.0
LLaVA-T.5 Liu 2023)
6.2
20.6
VESL (Ours)
5.9
19.6
Our Prompter mimics such inductive bias in pre-training via single-attention-laye encoder that may leverage bet- ter global context reasoning compared to Rols.
3.4. Discussions on Design Choices and Extensions
We provide discussions here on the rationale behind our design choices and some minor extensions.
Avoiding region-text confficts. While region annota- tions introduce location information, concern of con- trastive learning may be similar objects within an image (e.g., "boats" in the harbor) or a mini-batch. To miti- gate it, we apply two tricks. First, fortunately, we found it sufficient to sample a few regions per image for each update, egg., we set M = in Equation 2) in exper- iments. Second, we filter similar texts when comput- ing the negatives in the contrastive loss. More specif- ically, these pairs of fr(y, and ignored in the denominators of both CR-T/T-RR if s sim sim (fr(w) (mm) ). Fr(y, (m))) > 0.9 (no gradients on fT).
Extracting region embedding with visual prompts. Training with LcLoc in Eqquaion 4 requires extracting re- gion embeddings from the image features given the bound- ing boxes. Another alternative could be Region-of-Interes (Rol) pooling/alignment (He et al. 2017) from the spatial image feature of ViT before pooling. Rol operations are popular, especially in the object detection literature. How- ever, as will be evidenced by worse performance in Sec- tion we found spatial pooling an over-strong premise for CLOC pre-training here for several reasons.
(m))) are "masked"
First, unlike object detection datasets that typically con- tain golden labels, here the pseudo-labels are much noisier on the large-scale web-crawled images. The resulting Rol features may be inaccurate due to the imprecise bounding boxes, making model training less effective. Second, un- like dense vision tasks that directly rely on the spatial fea- tures, MLLM has a transformer decoder that consists of several attention layers such that the constraint of seman- ties in the spatial feature space becomes somewhat indirect.
4. VESL Captioning Pipeline
As discussed in Section and 3.1 a key bottleneck of CLOC is the region-text datasets in terms of both the data size and the label quality since there are no public datasets with region-text annotations at scales large enough for con- trastive pre-training. Inspired by recent works that en-
5

---

Contrastive Localized Language-Image Pre- Pre-Training
Image
Text queries
Visually-Enriched Captioner (e.g., VeCap)
Vpen-Vocabulary Detector (eg.. OWLv2, GUIV22)
WilH
picturesque, green-raofed cottage nostles alongside river, with two boats parked the adjacent grassy area, possibly for recrestional or transportation purposes. serene embellished with vibrant garden and colorful flowers, creating a tranquil ambiance."
Confidence filtering & cleaning
AltText: JULY 17,2016
"GIETHOORN, NETHERLANDS
N-grams or name entity recognition
Name entity recognifion
"GIETHOORN, NETERLANDS"
"green-roofed cottage"
"GIETHOORN"
"boats" "a river"
"NETERLANDS"
"colorful flowers"
Previous methods
Our candidate phrases extraction
Figure 3. Our Hisually-Enricher and Spattilly-Loocllizee (VESL) captioning pipeline. We leverage an existing open-vocabulary detector (e.g., OWLv2) that predicts bounding boxes the images and assigns the labels from the given text phrase candidates. Previous methods often use the alt-text attached to the images, which is prone to insufficient region descriptions. We found it crucial to re-caption images with the visually-cnriched captioner VeCap (Lai et al. 2024 for better visual concept exploitation for the detcctor.
rich image captions for better CLIP training, we make step further for Visually-Enricher and Spatially-Loealizedd (VESL) captioning that generates fine-grained captions at region level. The goal of VESL. is, given an image with the original web-crawled alt-text, to annotate it with the grounded bounding boxes, each associated with a cap- tion natural language for optimizing Equation in Sec- tion]
via self self-training. We are inspired by its success and extend it to CLOC contrastive learning with important modifica- tions. Different from (Minderer et al. 2024) that generates candidate phrases from the n-grams of the web-crawled alt- text of the images for OV detection, we found the alt-text might not have enough details describing the image region content, thus limiting the diversity and quality of the an- notations predicted by the OV detector. We thus caption each image augmented with more visual details. How- ever, the long captions make the n-grams candidates ver- bose and grow exponentially, thus we generate high-qualin candidates via name entity recognition instead. We found that such a pipeline produces training data more suitable for CLOC, as will be validated in Section [ We believe our approach is promising, as could enjoy future im- provements from better building blocks rely such as re-captioning (Li et al. 2024 Fan et al. 2024] and open- vocabulary detection (Minderer al. 2024) models.
Concretely, VESL is pseudo-labeling pipeline with the following steps, with pseudo codes in Appendiix C
1. Re-captioning with visual concept exploitation: We follow the VeCap2 (Lai et al., 2024) 2025) to generate long, diverse, and detailed image captions.
Region phrase candidates extraction: We apply name entity recognition (NER) extract phrases from the
visually-enriched captions candidates describing re- gion inside the image, inspired by ZZang et al. (2022).
Our pre-training datasets. Our pre- pre-training data consists of two parts: (i) image-text pairs, and (ii) region-text pairs. For image-text pairs, we reproduce the image re-captioning pipeline from VeCap (Lai et al., 2024), and generate syn- thetic captions for WiT-300M (Wu et al. 2024) and DFN- SB (Fang al.) 2023) images. For region-text pairs, we pseudo-label WTT-300M and 2B-image subset of DFN- 5B using our VESL pipeline. In VESL, adopted the official OWLv2 L/14 model (Minderer al. 2024) as the open-vocabulary detector. All images are pseudo-labeled with 448 x 448 resolution, where a maximum number of 20 phrase queries are sampled for moderate computation budget. Table muummaizse the statisties of existing region-
3. Open-vocabulary detection with extracted phrases: the final region-text annotations are generated by a pre- trained open- vocabulary detector. matches the phrases extracted from Step 2 to the bounding boxes proposed by the detector. We adopted OWLx2 detector (Minderer al.) 2024) which combines the CLIP image/text encoders with detection heads. The boxes with confidence larger than 0.1 are kept as the region location and the most confident phrases are considered as their captions.
Remarks. We highlight our insights behind the proposed recipe. The most relevant work was proposed in (Minderer al. 2024 that scales up open- vocabulcyy (OV) detection
6

---

Contrastive Localized Language-Image Pre- Pre-Training
text datasets and ours. Notably, we also ablate annotating WTT-300M following (Minderer al. 2024) and found it detects less objects with longer region text, likely due to the verbose n-grams of alt-text are in lower quality than our approach, as discussed in the remarks.
5.2. Image and Region Classification and Retrieval
CLOC encoder produces not only image embedding but also region embeddings. It can be used directly for region- level tasks without further training, in analogy to the zero- shot capability of CLIP on images. This emergent capabil- ity enables us to construct region-level zero-shot tasks for fast development and ablation studies.
Although synthetic captions are generated as detailed de- scriptions, one may wonder if hallucinations would be in- troduced into the captions. As VESL grounded the objects into the captions, gated. In Table 2] we evaluate the hallucination metric, the CHAIR score (Rohrbach et al., (2018), where lower is bet- ter, to assess the quality of synthetic captions. As shown in the table, our captioner demonstrates high quality com- pared to other models.
In addition to image-level evaluation like ImageNet clas- sification and COCO image-fext retrieval, we additionally construct region-level tasks, including region object recog- nition and text retrieval. More specifically, the region-level tasks leverage the labeled bounding boxes in the evaluation set for CLOC to extract region embedding. For region re- trieval, we use a validation set of the GRIT dataset (Peng et al. 2023) and encode both the image regions and the re- gion captions. For region classification, the class names are encoded as text embedding (80 / 1203 classes for COCO / LVIS, respectively), and the highest cosine similarity for each region embedding is predicted as its class. We high- light important variables for the performance in Table 3 with the following observations:
expect that hallucinations miti-
5. Experiments
5.1. Setup (more details in Appendix B)
Pre-training. We follow OpenAI-CLIP (Radford et al., 2021) to train both our CLIP baseline and CLOC model using a similar budget of around 14B images seen. Many previous models are trained on quite different data, labels, training cost, architectures, and etc. This makes it hard to draw a fair comparison directly, and some of them are not open-sourced. For a faithful comparison of our proposed methodology, we use the same hyper-parameters and im- age datasets for both the CLIP baseline and CLOC. We experimented with the ViT B/16 and L/14 architectures, pre-trained with 224 x 224 and 336 x 336 image resolu- tions, respectively. All parameters are trained end-to-end from scratch. We implement the above in JAX (Bradbury et al. 2018).
CLOCC performs decently on region-level tasks] with strong image-level performance (2 vs. (3).
The Prompter r is important ingredient for CLOC's success to go beyond CLIP (3) & 13 vs. + 2 14). We replace the Prompter er e with Rol alignment to extract re- gion features and train with LCLOe (similar to (Shi et al. 2025)). We found performs much worse on region- level tasks, possibly due to difficulties of strong Rol con- straints and the noisy labels as discussed in Section 3.4
VESL helps, as the visually-enriched captions improve image retrieval tasks (as expected (Lai al. (0244) and
Evaluation tasks. The image encoders are evaluated across a wide range of downstream tasks. First, we assess
the versatile visual concepts candidates facilitate the OV detector, supporting Section 4 ( 13 vs. 5 (15).
performance on ImageNet image classification (Deng al.)
OV detector OWLv2 > GLIPv2 in VESL (3 vs. 6).
2009 Shankar et al.) 2020) and COCO retrieval (Lin et al.
Tricks in Section 3.4 offer slight performance gains, but
2014 Second, we construct region-level tasks, including
LCooc is already highly effective on its own (10 1.)
COCO object recognition and region-text retrieval using
Practically, sampling 2/4 boxes works (12) well already.
the GRIT dataset (Peng et al. 2023). Furthermore, we
Scaling up images saturated on region tasks but further
show CLOC is particularly useful for MLLMs, validated
improved on MILM tasks (Q & Table 4
by the Ferret model (You et al. 2023) which requires fine-
Scaling up the ViT model sizes can further improve both image and region tasks (13 16
grained image understanding for referring and grounding tasks. We also evaluate on general multimodal benchmarks
using 1.. (Liu et al.] 2023) and Open-I.LaVA NeXT (Liu et al. 2024| Chen & Xing 2024), which both use the 7B Vicuna LLM. For all evaluation tasks, we use the same official preer-parameters fine-tuning datasets, and codebase for all the image encoders we experimented with, without specific tuning.
Overall, CLOC not only achieves strong performance on image-level tasks, but unlocks zero-shot region-level capa- bility. Below, with our design choices validated, (13 will be used by default if not speeified.
For reference, in different sctup, Wu ct al. 2023 reports 46.5% mAce on the COCO region classification task. trained with 320 320 COCO images directly. In contrast, our approach achieves over 70% mAce, pre-trained on 221 web-crawled dataset with our object labels (thus not a fair comparisons).
7

---

Contrastive Localized Language-Image Pre- Training
Table 3. Zero-shot evaluation on image-level tasks @recall of COCO retrieval and accuracy of ImageNet (IN) classification) and
region-level tasks (recall@ 10 of GRIT region retrieval and mAce of region object recognition on COCO and LVIS), using ViT-B/16 as the default encoder backbone. The indentation with different symbols denotes removing (-) or changing a component (o).
Models
Training Labels|
Image tasks
Region tasks
Avg.
Image
Region coco-i2i COCO-2i INvl INV2 GRIT-12r RecCOCo RecLVIS |Image Region
OPenAI-CLIP
proprietary
52.4
33.1
68.3 62.3
54.0
CLIP
WIT+DFN
66.3
45.1
76.2
69.6
64.3
CLOC
WiT
WiT
68.8
50.1
66.7 59.7
65.1
67.2
70.6
26.7
61.3
57.4
Prompter
WIT
WiT
67.0
49.7
65.6 58.6
44,8
4.4
55.3
13.2
60.2
29.4
VESL
WIT
WIT
53.9
36.3
66.6 59.5
71.5
63.8
62.2
22.2
54.1
54.9
w/ GLIPv2
WIT
WIT
68.8
50.0
65.8
59.2
67.9
71.1
64.9
23.1
61.0
56.8
CLOC
WIT+DEN
WIT
66.1
46.5
75.5
68.6
65.8
67.4
70.1
27.2
64.2
57.6
P Prompter WIT+DEN
WIT
65.8
46.5
75.7
68.0
55.5
18.4
67.1
24.6
64.0
41.4
10
text filtering
WIT+DEN
WIT
65.4
46.0
75.7
68.4
66.3
66.5
68.7
24.8
63.9
56.6
Lyranddngg
WIT+DEN
WiT
66.0
46.3
75.7
67.9
66.0
66.8
70.0
25.8
64.0
57.2
M 2
WIT+DFN
WIT
66.6
46.2
75.5
67.9
66,5
67.0
69.8
25.8
64.1
57.3
CLOC
WIT+DEN DIN+
69.2
49.3
74.9
67.0
63.9
65.9
71.1
28.5
65.1
57.3
1
Prompter WIN DIN+DFN
70.2
49.7
74.7
67.6
65.7
23.0
67.1
25.4
65.6
45.3
15
VESL
WIT+DEN WIT+DEN
65.3
46.6
75.5
67.7
55.7
22.3
66.3
25.3
63.8
42.4
16
VIT L/14
WiT+DEN WIT+DEN
74.8
54.4
80.1
73.2
66.9
68.3
72.9
32.6
70.6
60.2
7
VIT H/14
WIT+DEN WIT+DEN
75.7
55.1
81.3
74.7
67.4
69.4
73.0
35.6
71.7
61.3
Table 4. "eerett-ennce for referring and grounding VQA, based Ferret (You et al. (203) equipped with different image encoders. Models evaluated with OpenAl gpt API. "replace Ferret
with our CLOC ViT as a drop-in replacement. We use the official codebase for training the Ferret model. We further consider a variant based on Ferret: the Ferret model im- plements spatial-aware visual sampler that samples im- age features from the region specified in the question. We replace the sophisticated visual sampler with our simple P Prompter introduced in Section 3.3 to extract region em- bedding with z P Prompter(( (x)) instead, as illus- trated in Figure rright).
visual sampler with P Prompter; Details in Section 5.3
Method
VIT
Region Align
w/ region
#images
Ground Descript.Rennn Conv.
Ref-
Ref-
Avg. ACLIP
labels
CLIP B/16
None
None
47.5
50.3
45.3
47.7
CLOC B/16
Rol-Align
300M
48.0
48.4
40.0
45,5
CLOC B/16
Prompter
300M
50.2
55.5
41.5
49.1
CLOC
B/16 Prompter
2B
53.6
53.7
42.2
49.8
(+2.D)
CLOC B/16 Prompter
2B
54.8
54.9
44.7
51.5 (43.7)
In Table we evaluate different pre-trained image en- coders on the Ferret-Bench benchmark (You et al., 2023). Ferret-Bench includes challenging multimodal dialogue- style VQA of three tasks constructed with GPT-4. Results show that our P Prompter is essential to improve upon the CLIP baseline - Rol-Align may even slightly degrade per- formance. Scaling region labels from 300M to 2B further improves performance. Interestingly, our P Prompter (de- noted as *) can be a replacement of the FERRET visual sampler in fine-tuning which is simpler and performs even better up to 6% against both the OpenAl-CLIP and our in- house CLIP. We also evaluate CLOC (2B labeled) on other referring and grounding tasks ranging from referring ob- ject classification, referring expression comprehension, and phrase grounding across multiple datasets. As summarized in Table 5] CLOC is also superior evidenced by 1 3% improvements in average of 13 evaluation sets.
OAL-CLIP L/14
None
None
50.8
55.4
45.7
50.6
CLIP L/14
None
None
54.2
54.6
43.3
50.7
CLOC L/14
Prompter
300M
51.0
65.7
44.9
53,9
CLOC L/14 Prompter
2B
55.9
63.3
46.0
55.1 (14.4)
CLOC L/14 Prompter
2B
56.3
67.4
47.1
56.9 (46.2)
5.3. Referring and Grounding with Ferret
As discussed in Section 1 key motivation is to provide an enhanced image encoder for training MLLMs, particu- larly for tasks requiring fine-grai grained image understanding. A notable example is Ferret YYou et al. 2023), a recently proposed MILMM that builds on LLaVA and aims to handle more advanced spatial interactions, such as referring and grounding in VQA tasks. Ferret can take region prompts such box, point, or free-form location referring to the input image as input, and answer question specific to the region such as "Do you know when the object[region was invented?" Ferret thus requires fine-grained image fea- tures from the vision encoder for spatial reasoning.
5.4. General VQA with LLaVA-1.5 and LLaVA-NeXT
We further show that the CLOC encoder is also competi- tive against CLIP general VQA tasks without regression
We evaluate CLOC by replacing the CLIP ViT encoder
8

---

Contrastive Localized Language-Image Pre- Training
Table 5. Results on referred-I.IS object classification, referring expression comprehension (0.5 IoU RefCOCO, RefCOCO+, Ref- COCOg), and phrase grounding (0,5 IoU on Flickr30k Entities). FERRET ": replace visual sampler in FERRET with CLOC prompter.
Model
Encoder
RefLVIS
RefCOCO
RefCOCO+ RefCOCOg
Flickr
Avg.
box point free- form
val testA testB val testA testB
test
val test (A CLIP)
FERRET
CLIP B/16
72.5 56.9 57.2 80.7 84.2 77.1 71.9 76.1 63.7
75.9
76.2
76.2 78.3
72.8
FERRET
CLOC B/16
74.3 56.7 60.2
84.2 87.0 80.0 74.7 80.0 67.0
78.8
79.5
80.0
75.7 (+2.9)
FERRET
CLOC B/16
78.9 58.2
61.4
84.4 86.8
78.9 74.0 78.7 65.5
77.00
78.7
80.1 81.4
75.8 (+3.0)
Shikra (Chen et al.
2023)
OpenAI-CLIP L/14
57.8 67.7
n/a
88.0 90.6 80.2 81.6 87.4 72.1
82.3
82.2
75.8
FERRET
Zhang el al. 2024
OpenAL-CLIP L/14
79.4 67.9
69.8
87.5 91.4 82.5 80.8 87.4 73.1
83.9
84.8
82.2
80.8
FERRET
CLIP L/14
78.7 66.9
70.2
88.0 90.4 83.5 80.1 85.8 73.3
82.8
83,4
80.0
80.2
FERRET
CLOC
81.6 67.9
69.9
89.0
91.0 84.7 81.4 86.8 74.7
84.0
85.2
83.3
81.7 (+1.5)
FERRET"
CLOC L/14
77..8 67.9 69.1
88.2 91.1 84.5 80.6 86.7 73.9
84.8 85.1
82.4
81.4 (+1.2)
Table 6. Results on multimodal benchmarks using LL5 and Open-LLaVA-NeeT with ViT-L/14 and Vicuna-7B.
Table 7. Results on open detection.
COCO-Det (minival)
ODinW (test)
LVIS-Det (minival)
Method
ViT
LLLVA W TextVQA GQA MM POPE MMEp MMEc
1LaVA-155
GLIP-T
T/16
46.6
46.5
26.0
GLIP-L
L/14
49.8
52.1
37.3
CLIP
59.3
53.3
62.2
30.0
86.7
1451.4
254.3 1482.0
CLOCC-
B/16
47.3
48.4
29.6
CLOC
64.3
54.9
62.7
31.5
87.3
288.9
CLOC-L
L/14
50.8
53.6
38.1
Open LLLVA-NeXT
CLIP
67.3
61.4
63.5
38.5
87.9
1486.1
279.6
different VIT backbones, suggesting that CLOC offers ad- vantages in localization and object detection performance. Notably, GLIP employs DyHead (Dai et al. 2021), a strong detector head module, on top of the encoder, whereas our ablation study uses only two simple heads for classifica- tion and regression. This further supports that the encoder representation in CLOC is indeed superior.
CLOC
69.5
61.9
64.2
40.2
88.3
1451.1
312.5
and can even provide performance improvements. We use the Vicuna 7B LLM decoder for two experiments based on LLaVA-1.5 (frozen encoder) and Open-LLaVA-NeXT (unfrozen encoder with AnyRes (Liu et al., 2024) inputs). Since general VQA does not provide spatial referring in- puts, we simply replace the VIT in LLaVA. Table 6 sum- marizes the results. Encouragingly. with our CLOC de- signs, the improved region-level alignment is also benefi- cial to some general multimodal benchmarks, as they may also require fine-grained image understanding.
6. Conclusion
Please see Appendix D for more discussions where we comment on the limitations, future directions, computation cost, design rationales, etc.
We tackle a deficiency of CLIP, to make the seman- tics aligned in the vision space for both image and re- gion level. We propose pre-training framework that ables a strong encoder to adapt seamlessly for downstream use in MILMs, supporting feature interaction via input prompts. Our encoder creates new possibility for adapt- ing the features with input prompts of interaction together with MLLMs. To resolve the need for large-scale region- text training data, we carefully design a pseudo-laheling pipeline for visually-enriched and spatially-Iocalize cap- tions. Our pre- trained encoder is essentially drop-in replacement of CLIP with competitive image-text perfor- mance and extra capability demonstrated in region-text tasks and VQA tasks with MLLMs.
5.5. Vocabulcyy Detection
In Table 3 models demonstrate that the region classification task (predicting class names given a bound- ing box). our approach achieves over 70% mAce in COCO, significantly outperforming 47% reported in pre- vious work (Wu et al. 2023).
As introduced in |subsection 3.2 CLOC consists of re- gion regression head and a classification head. Here we further benchmark CLOC as open- vocabulary detec- tion model. We provide zero-shot evaluation results in Ta- ble 7 on COCO Detection (minival), ODinW (test-dev), and LVIS-Det (minival). When comparing GLIP (Li et al. 2022) and CLOC, we observe that CLOC consistently achieves better results than GL.P across all datasets across

---

Contrastive Localized Language-Ingg Imagg Pre- Training
Impact Statement
al. Maskelip: Masked self-distillation advances con- trastive language-image pretraining- In Proceedings of the IEEE/CVF Conference Computer Vision and Pat- tem Recognition, pp. (0995-11005, 2023.
The goal of our work is to improve the performance of the well established CLIP pre-training. There are many poten- tial societal consequences of our work, none which we feel must be specifically highlighted here.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021.
Acknowledgment
We thank Yanghao Li, Felix Bai, and many others for their invaluable help and feedback.
Fan, L., Krishnan, D., Isola, P., Katabi, D., and Tian, Im- proving clip training with language rewrites. Advances in Neural Information Processing Systems, 36, 2024.
References
Bica, I., Ilié, A., Bauer, M., Erdogan, G., Bošnjak, M., Kaplanis, C., Gritsenko, A., Minderer, M., Blun- dell, C., Pascanu, R., al. Improving fine-grained un- derstanding in image-text pre-training. arXiv preprint arXiv:2401.09865, 2024.
Fang, A., Jose, A. M., Jain, A., Schmidt, I.., Toshev, A., and Shankar, V. Data filtering networks. arXiv preprint arXiv:2309. 2023.
He, K., Gkioxari, G., Dollár, P., and Girshick, R. Mask r-cnn. In Proceedings of the IEEE international confer- ence on computer vision, pp. 2961-2969, 2017.
Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-MMlne, S., and Zhang, Q. JAX: composable transformations of Python+NumPP URL http://githab.com
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
programs, 2018.
jax-ml/ja
Chen, K., Zhang, Z. Zeng, W., Zhang. R., Zhu, F., and Zhao, R. Shikra: Unleashing multimodal Ilm's referen- tial dialogue magic. arXiv preprint 3xiv:2306615195 2023.
Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus). arXiv preprint arXiv: 08415, 2016.
Jia, C.,, Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z.,, Pham, Le, Q. Sung, Y--.., Li, Z., and Duerig, Scaling up visual and vision-language representation learning with noisy text supervision. In International conference machine learning, pp. 4904-4916. PMLR, 2021.
Chen, L. and Xing, L. Open-Ilava-next: An open-source implementation of llava-next series for facilitating the large multi-modal model commu- nity. /etpss//gittin
com/ http//xxaccceee//////
Open- LLaVA NeXT 2024.
Kirillov, A., Mintun, E.,, Ravi, N., Mao, H., Rolland, C., Gustafson, I.., Xiao, T., Whitehead, S., Berg, A. C.,, Lo, W.-Y,, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pp. 4015-4026, 2023.
Dai, W., Li, J. Li, D., Tiong, A. M. H., Zhao, J., Wang. W., Li, B., Fung, P. and Hoi, S. Instructblip: To- wards general-purposs vision-language models with in- struction uning, 2023. URL httpttt////aii..rrrtt
abs/2305.055 06500
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., Bernstein, M. S., and Fei-Fei, Visual genome: Connecting language and vision using crowdsoureed dense image annotations. International Journal of Com- puter Vision 123:32-73. 2017.
Dai, X., Chen, Y., Xiao, B., Chen, D., Liu, M., Yuan, L., and Zhang, L. Dynamic head: Unifying object detection heads with attentions. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 7373-7382, 2021.
Lai, X., Tian, Z., Chen, Y., Li, Y., Yuan, Y., Liu, S., and Jia, J. Lisa: Reasoning segmentation via large language model. In CVPR, 2023.
Deng, J., Dong, W., Socher, R., Li, L.JJ., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pallern recognition, pp. 248-255. Ieee, 2009.
Lai, Z., Zhang, H., Wu, W., Bai, H., Timofeev, A., Du, X., Gan, Z., Shan, J., Chuah, C.-N., Yang, Y., et al. Veclip: Improving clip training via visual- enriched captions. In ECCV, 2024.
Dong, X., Bao, J., Zheng, Y., Zhang. T., Chen, D., Yang, H., Zeng, M., Zhang, W., Yuan, L., Chen, D.,
10

---

Contrastive Localized Language-Image Pre- Pre-Training
Lai, Z., Saveris, V., Chen, C., Chen, H.-Y., Zhang, H., Zhang, B., Tebar, J. L., Hu, w., Gan, Z., Grasch, P., et al. Revisit large-scale image-caption data in pre- training multimodal foundation models. In ICLR, 2025.
Peng, Z., Wang, W., Dong, L., Hao, Y., Huang, S., Ma, S., and Wei, F. Kosmos-2: Grounding multimodal large language models to the world. ArXiv, abs/2306.1480 2023.
Li, L. H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y., Wang, l.,, Yuan, I.., Zhang, I.,, Hwang, J.-N., et al. Grounded language-image pre-training, In Proceedings the IEEE/CVF conference on computer vision and pat- tern recognition, pp. 10965-10975, 2022.
Plummer, B. A., Wang, L.,, Cervantes, C. M., Caicedo, J. C.,, Hockenmaier, J., and Lazebnik, Flickr30k en- tities: Collecting region-to- to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pp. 2641 2015.
Li, X., Tu, H., Hui, M., Wang, 7., Zhao, B., Xiao, J., Ren, Mei, J., Liu, Q., Zheng, H., et al. What if we recaption billions of web images with llama-?? arXiv preprint arXiv:2406.0844 08478, 2024.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natu- ral language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13, pp. 740- 755. Springer, 2014.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21 (140):1667, 2020.
Liu, H., Li, C., Wu, Q. and Lee, Y. Visual instruction tuning. In NeurlPS, 2023.
Rasheed, H., Maaz, M., Shaji, S., Shaker, A., Khan, S., Cholakkal, H.. Anwer, R. M., Xing, E., Yang. M.-H., and Khan, F. S. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 13009-13018, June 2024.
Liu, H., Li, C., Li, Y. Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Llava-next: Improved reasoning. ocr, and world knowledge, January 2024. URL https://11 alaaalllvvvaiia vl. www................. io/blog/ 2024 30- 1lava next/
Ren, Z., Huang, Z., Wei, Y., Zhao, Y., Fu, D., Feng. and Jin, X. Pixellm: Pixel reasoning with large multimodal model. In CVPR, 2024.
Mao, J.. Huang. J.. Toshev, A., Camburu, O., Yuille, A. L., and Murphy, K. Generation and comprehension un- ambiguous object descriptions.
Proceedings of the IEEE conference on computer vision and pattern recog- nition, pp. 11-20, 2016.
Rohrbach, A., Hendricks, L. A., Burns, K., Darrell, T., and Saenko, K. Object hallucination in image captioning. arXiv preprint arXiv: 02.900226,,0. 2018
McKinzie, B., Gan, Z., Fauconnier, J.-P., Dodge, S., Zhang, B., Dufter, P., Shah, D., Du, X., Peng. F,, Weers, F., et al. Mml: Methods, analysis & insights from multimodal I1m pre-training. In ECCV, 2024.
Shankar, V., Roclofs, R., Mania, H., Fang, A., Recht, B., and Schmidt, L. Evaluating machine accuracy on Ima- geNet. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learn- ing. volume 119 of Proceedings of Machine Learning Research, pp. 8634 8644. PMLR, 13-18 Jul 2020.
Minderer, M., Gritsenko, A. and Houlsby, N. Scaling open-vocabulan object detection. Advances in Neural Information Processing Systems, 36, 2024.
Shi, B., Zhao, P., Wang, Z., Zhang, Y., Wang, Y., Li, J., Dai, W., Zou, J., Xiong, H., Tian, Q., et al. Umg-clip: A unified multi-granule uulti-granularity vision generalist for open- world understanding. In European Conference on Com- puter Vision, pp. 259-277. Springer, 2025.
Naeem, M. F, Xian, Y., Zhai, X., Hoyer, I.., Van Gool, L., and Tombari, F. Silc: Improving vision lan- guage pretraining with self-distillation. arXiv preprint arXiv:2310. 1..100.13555 13355, 2023.
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F, El-Nouby, A., et al. Dinov2: Learning robust visual features without supervision. arXiv preprint AXiv:2304.07119,, 2023.
Sun, Z., Fang, Y., Wu, T., Zhang, P., Zang, Y., Kong, S., Xiong, Y., Lin, D., and Wang, J. Alpha-clip: A clip model focusing on wherever you want. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pp. 13019-13029, June 2024.

---

Contrastive Localized Language-Image Pre- Training
Tong, S., Brown, E., Wu, P., Woo, S., Middepogu, M., Akula, S. C., Yang, J., Yang, S., Iyer, A., Pan, X., et al. Cambrian-l: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.1686, 2024.
Yu, L., Poirson, P., Yang, S., Berg, A. C., and Berg, T. L. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Am- sterdam, The Netherlands, October 11-14, 2016, Pro- ceedings, Part 11 14, pp. 69-85. Springer, 2016.
Tschannen, M., Kumar, M., Steiner, A., Zhai, X., Houlsby, N., and Beyer, L. Image captioners are scalable vision learners too. Advances Neural Information Processing Systems, 36, 2024.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L.. Sig- moid loss for language image pre-training. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pp. 11975-11986, 2023.
Wan, B., Tschannen, M., Xian, Y., Pavetic, F., Alabdul- mohsin, I., Wang. X., Pinto, A. S., Steiner, A., Beyer, L., and Zhai, X. Locca: Visual pretraining with location- aware captioners. arXiv preprint 1XXi:2243311996 2024.
Zhang, H., Zhang, P., Hu, X., Chen, Y.-.., Li, L.,, Dai, X., Wang. L.,, Yuan, L.,, Hwang. J.-N., and Gao, J. Glipv2: Unifying localization and vision-language understand- ing. Advances in Neural Information Processing Sys- tems, 35:36067-36080, 2022.
Zhang, H., You, H., Dufter, P., Zhang, B., Chen, C., Chen, H.-Y., Fu, T.-J,, Wang, W. Y., Chang, S.-F, Gan, Z et al. Ferret-v2: An improved baseline for referring and grounding with large language models. In Conference on Language Modeling, 2024.
Wu, s., Zhang, W., Xu, L., Jin, S., Li, X., Liu, W., and Loy, C. C. Clipself: Vision transformer distills itself for open-vocabulary dense prediction. arXiv preprint arXiv:2310.01403, 2023.
Wu, W., Timofeev, A., Chen, C., Zhang, B., Duan, K., Liu, S., Zheng. Y., Shlens, J., Du, X., Gan, Z., al. Mofi: Learning image representations from noisy entity anno- tated images. In ICLR, 2024.
Zhang, H., Gao, M., Gan, Z., Dufter, P., Wenzel, N., Huang, F., Shah, D., Du, X., Zhang, B., Li, Y., et al. Mm1. 5: Methods, analysis & insights from multimodal Ilm fine-tuning, In ICLR, 2025.
Xiao, B., Wu, H., Xu, W., Dai, X., Hu, H., Lu, Y., Zeng, M., Liu, C., and Yuan, L. Florence-2: Advancing unified representation for a variety of vision tasks. In Proceed- ings of the IEEF/CVF Conference on Computer Vision and Pattern Recogmition, pp. 4818-4829, 2024.
Zhong, Y., Yang, J., Zhang, P., Li, C., Codella, N., Li, L. H., Zhou, L., Dai, X., Yuan, L., Li, Y., et al. Re- gionclip: Region-based language-image pretraining. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16793-16803, 2022.
Yao, L., Han, J., Wen, Y., Liang, X., Xu, D., Zhang, W., Li, Z., Xu, C., and Xu, H. Detclip: Dictionary- enriched visual-concept paralleled pre-training for open- world detection. Advances in Neural Information Pro- cessing Systems, 35:9125-9133 2022.
Zhu, D. Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
Yao, L., Han, J., Liang, X., Xu, D., Zhang, W., Li, 7., and Xu, H. Detclipv2: Scalable open- ppn-vocabulaayy object de- tection pre-training via word-region alignment. In Pro- ceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pp. 23497-23506. June 2023a.
Yao, L., Han, J., Liang, X., Xu, D., Zhang, W., Li, Z., and Xu, H. Detclipy2: Scalable open- open-vocabulary object de- tection pre-training via word-region alignment. In Pro- ceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pp. 23497-23506 2023b.
You, H., Zhang, H., Gan, Z., Du, X., Zhang, B., Wang, Z., Cao, L., Chang. S.-F, and Yang, Y. Ferret: Refer and ground anything anywhere at any granularity. In ICLR, 2023.
12

---

Contrastive Localized Language-Image Pre- Training
Appendix
tion, the class names are encoded by the text encoder into class embedding. We do not add the text prompts (e.g.. "a photo of ...") to object classes used when CLIP (Radford) et al. (021) evaluated on image classification. The CLOC model takes all the labeled bounding boxes in the images to generate a region embedding x = Pr Promptex(1, fz))... The class embedding with the highest similarity is pre- dicted as the class of the region (i.e., out of 80 / 1203 classes for COCO LVIS).
A. Reproducibility Statement
We made our best efforts to exhaustively state the imple- mentation details. Training hyper-parameters and model architectures are discussed in Section 3.2 3.3 and with a summary in AApendix B and Table A For evaluation,
as mentioned in Section 5.1 we strictly follow the official setup with the codebase released by the original authors if applicable, with details provided in Section [B.2] For our datasets, we provide data processing details in Section + and example codes in Apppeddix C We are working hard on releasing the annotations with internal approvals.
For region retrieval, similarly, the CLOC model encodes both the image regions and the region captions from the public region-text GRIT dataset that the regions are anno- tated by the Kosmos-2 pipeline (Peng et al.] 2023). We randomly sampled 2K image validation set for fast eval- uation. We have verified it is statistically stable compared to the whole set that contains about 20M in total. Unlike image-text retrieval the image captions are likely unique, the objects in regions of many images might be duplicated. Therefore, we opt to report recall @ 10 rather than recall @ for GRIT region retrieval in Table 3
B. Experiment Details
We provide the omitted experiment details for pre-training and the downstream evaluation tasks.
B.1. Pre-training Hypeer-paraeetes
MILM tasks. To demonstrate our CLOC can benefit MLLM end tasks as a better image backbone, we consider two sets of MLLM experiments.
For pre-training both the in-house CLIP baseline and CLOOC we mainly follow the hyper-parameters in (Rad- ford et al. (02I) to train on our in-house datasets. The training images are identical for CLIP and CLOC, while CLOC is trained on the extra region-text annotations of the same images via the proposed VESL pipeline (details in Section C TTale A] summarizes the general training hyper- parameters used for all experiments and the setup for com- ponents specific to CLOC.
First, we experiment with the FERRET MILM that is pable of taking spatial referring inputs for grounding and referring VQA tasks" FERRET can consume a point, a bounding box, or a free-form referring. It designs a quite complicated visual sampler module that involves point sampling and kNN grouping. We suggest the readers re- fer to Figure 3 and Section 3.2 in (You et al.] (023) for more details. Here consider two variants of use cases of CLOC compatible with FERRET: (1) we only take the VIT encoder in CLOC to replace the CLIP ViT and still use the original FERRET visual sampler or (2) we further replace the visual sampler with our simple P cmmpter (essentially a lightweight transformer encoder with box positional en- codings) in Section 3.3 as illustrated Figure 3bb).. More specifically, we simply convert all types of spatial refer- ring as boxes. As evidenced by Table 4 and Table 5] our P Prompter can indeed be much simpler alternative and may perform even better as it is more consistent with CLOC pre-training.
In terms of the CLOC architecture, illustrated in Fig- ure 2] the image and text encoders including the attention pooling and projection layers follow the same as OpenAl- CLIP (Radford et al. 202I). Our Prompter consists of a positional encoding matrix for bounding boxes, and a single-layer single-head transformer encoder with another set of the global average pooler and a projection layer to map the region embeddings into the same dimension as the CLIP text/image embeddings.
B.2. Evaluation Tasks
We provide more details about the tasks constructed for evaluating the encoders in Section 5
Second, we evaluate on general VQA tasks that do not con- sider extra spatial referring inputs. The pre-trained VIT of CLOC is drop-in-reppaacement of CLIP ViT in two sets of experiments of LLaVA-1.5 (Liu al.) (023) and LLaVA-NeXT (Liu et al. 2024). The main difference in- cludes different supervised fine-tuning (SFT) sets. Also, LLaVA-NeXT uses the AnyRes technique that decomposes an image into several subimages that are encoded inde-
Zero-shot region tasks. Our CLOCC training augments new capability for CLIP to generate region-level embed- dings. This enables us to perform zero-shot region-text tasks, in analogy to the image-text zero-shot tasks like Im- ageNet classification and COCO text-image retrieval that CLIP has been evaluated on.
In similar rationale of image-level evaluation, we further construct region-level tasks including region object recog- nition and region- text retrieval. For region object recogni-
We use the official Ferret codebase: https: https://github. com/ apple/ml /ferrt
13

---

Contrastive Localized Language-Imare Pre- Training
Table A. Pre-training hyper-parammee and settings for the in-house CLIP baseline and CLOC.
General
Batch size
32768
Image size
224 x 224 (ViT B/16) or 336 336 (ViT L/14, H/14)
Image pre ve-processing
long-side resizing with padding (i.e.. tf. image. with pad)
Text tokenizer
TS (Rafell et al. 2020 lowercase
Text maximum length
77 tokens
Steps
439087 (i.c., ~ 14B examples seen)
Optimizer
AdamW W (81 = 0.9, 32 0.98)
Peak learning rate (LR)
0.0005
LR schedule
cosine decays with linear warm-up (first 2k steps)
Weight decay
0.2
Dropout rate
0.0
CLOC
# of sampled regions CLOC loss weight Encoding box prompts Encoding text prompts Prompter er architecture single-layer single-head transformer encoder (same feature dimension as the ViT) 2-layer MLP with GELUs activations (Hendrycks Gimpel 2016)
maximum M = 4 per image
X 1.0 dlof
region
(in Equation 4
latch
sinusoidal positional encoding of coordinates (top-left and bottom right of a box) encoded by re-using the text encoder (w/ pooling & projection)
BoxHead architecture
pendently with the ViT and concatenated together as the input for the decoder. LLaVA-1.5 by default freezes the VIT while LLaVA-NeXTT fine-tunes all parameters during SFT. Since the official LLaVA-NeXT trained on some the
The code listing below shows the Python example imple- mentation, where stop-words and common generic words are filtered, following (Minderer al. 2024).
Generating bounding boxes and assigning region captions can be done by querying an OV objection detector. We adopted the OWLv2 detector (Minderer et al. (024) with their pre-trained L/14 checkpoin to annotate inputs with 448 x 448 image resolutions.
proprietary datasets that are not reproducible, OpeN-LLaVA-NeXT repository Our experiments in Ta- ble 6 demonstrate CLOC not only slightly improves such general VQA besides FERRET tasks but also generalizes well for both LLaVA-1.5 and LLaVA-NeXT settings.
from typing import iterable, List import nitk
C. VESL Data Engine
STOPWORDS and CONMON CENERIC MORDS folloving: Section A.2 (Minderer et all,, 2024)
We provide more information about our pseudo-labelie data pipeline proposed in Section 4
Stepwords from nirk.corpus.com sat...capss.ttooordae wtw.eppoods..wrdd..... ("english")
C.1. Implementation Details
8
STOPKORDS_ET EN frozenset {(
"=", "about", "above", "after", "again", "against", *all", *am", *an", "and", *any", *are", *ast, "at", "be", "because", been"" "before", "being", "below", "between", "both", "but", "by", *can", did", "do", "does", "doing", tcon", "down", "during", "each", "rew", for", *éram", "further", "had", "has", thave", "having", "he", "her", "here" "Lers", "herself", "him", "himself", "his", "how", "i" "LE", *int, into*, "is", "it", "ics", "icself", "Just", "me", "more", "most", "ry", myaell", "no", "nor", "not", "now", "of", "aff", "an", "once", "only", "or", other", "our", "ours", "ourselves", "out", "over", "own", "s", "sane", she", "should", "so", "some", "such", "e", "than", "that", "the", "thoir" "theirs", "them", "thenselves", "then", "there", "these", "they", " this*, "those", "through", "Lo", "Loo", "undes", "until", *up", "vezy", "was",
As already mentioned in Section 4 there are three steps for
10
VESL: image re-captioning, region phrase candidates ex-
traction from the captions, and open-vocabulary (OV) de- tection given the region candidates as queries.
For the re-captioning, the goal is to replace AltText with long, diverse, and detailed captions that can be used gen- erate more visual concepts as the region candidate phrases for the OV detector. Technically, any strong image cap- tioner can be an option. In our paper, we adopt the VeCap pipeline (Lai et al. 2024) and leverage their images with enriched captions.
To extract region phrase candidates from the long captions, adopt name entity recognition (NER) to extract leaf en- tities from the captions, inspired by (Zhang al. 2022
*OWL.2 CLIP L/14 ST+FT in: htpp://giitb..com/goggle rreartthceeetrccmiiiieeeeengneor vit
14

---

Contrastive Localized Language-Image Pre- Pre-Train
"we", "were", "what",
C.2. More Visualizations
21
"when", "whore", "which", "while", "who", "whom", why", "wil1" "with", "you", "your", "yours", "yourself", "youraelves"
As mentioned in the remarks of Section + we found the AltText soureed from the original web-crawled images might not have enough details describing the subimage content, thus limiting the diversity and quality of the text candidate queries for the OV detector to detect more mean- ingful objects. In Figure A] we show some cherry-pickod examples (since the web-crawled images are quite noisy) just to demonstrate the reasons why high-quality captions can help our region-text annotation pipeline. In (Minderer ct al. 2024). the queries are generated by the n-grams of the AltText, while ours are by NER as described in Sec- tion C.1 on top of the visually-enriched re-captions. Note that, in both methods we use the same pre-trained OV de- tector but with different approaches to generate the queries.
22
23 ))
24
25
These words were found by mansally noting through the most common 1000 wozds
26
in aample of alt-texta and selacting genartc words
without specitic meaning:
27
COMMON SENNERIC_KORDS frozenset
28
"al bblbbbs", "sliexpress", "amazon", *available", baskground", "blog", "buy", "co", *com", "deacription", "dly", "doknload", facebock", "Iree", "gif", "ho", "ideas", aillusrration", "f1 "fllustrations", Image", "images", "img", "instagram", "ipg", "enline", *ozg", "eriginsi", " page", "péf", "phozo", "photography", "photos", "plecl iok", "ploture", piotires", " "poen", "premium", "resolution", "royalzy", "sale", "sex", "shutterstock", "ssook", "svg", "Lhumbnell"; "sumbir", "sungie", "sxittes", "uk", "uploaded", "vector", "vectors", "vidas", "videns", *willpapert, wall papers", "wholesale", "wus", "xxx", "youtibe"
29
30
31
32
33
34
As shown in Figure A for easier images like the first row, both methods are doing reasonably well to detect "mes- sage card". However, when the scene becomes compli- cated (e.g., the second row), our methods can detect more objects since more visual concepts can be extracted from our rich caption as queries for the detector. Similarly, it can be seen that our method captures more items that the AlrText missed, e.g., "banana", "'ggs", "butter", etc in the third row; "drawstring" in the fourth row; "apples" and "vases" in the last row. Also, it is more likely to extract more detailed description of the region rather than a class name, such as "green-roofed cottage nestles" in Figure 3 and "decorative metal tree sculptures" in the image in the last row of Figure A, We believe such high-quality re- gion labels essentially contribute to better supervision for CLOC pre-training.
35
36
37 })
38
39
40 attl___toooorraaaaoooooooo
iterable(ass) (ste))
hoo :
41
:eturn set lquery_worda) issubset (8TOPMORDS_EN
42
43
44 def _gst__nmo_enttite mame_entities (wards: List (str]) List [str]:
45
46
Returna name eatities of image caption as queries,
68mm lar to GLTP.
47
48
pos_sags mittk.ps_tte (words)
49
grammar = "NP: (<DT>>4Jd, (ad 194]"
50
qp nitk, RegexpParsarrrr (grammar)
51
result = ep.parse (pos_tags)
52
53
queries =
54
for subtree in sustrees ():
55
if Labele query_wards = [t[O) for t in subtree. leaves
"N²":
56
01
D. More Discussions
57
Don't use it if it only consists of stop
words.
Limitations. One limitation for CLOC is the labeling ef- forts in preparing the training data. As we discussed in Sec- tion T there are no public large-scale region-text datasets since it is expensive to infer such labels up to the scales we consider here. Unlike previous work (Zhong et al.) 2022] that cropping boxes from images for annotating, our VESL inference in image-level thus the cost does not scale with the number of detected regions. With that being said, such inference still requires hundreds of GPUs running in par- allel for days to scale up to billions of images. We are working on releasing the annotations to accelerate future research for the community.
58
1f _i____l_ stoopoorss :ords] continue
59
60
appendd" word( rsturn queries
61
62
63
64
def
65
caption: str, max_nun_gueries: int 20,
66 }
>>
List[str]]
67 caption loperan.lower()
68
tckens -
lcapt Remove common generic words.
69
70
worda = [w for w in tokens if not in
COWNON_GENERIC, NORDS]
71
queries _get__ame__s__eeeee_____________ www__aameeetttieseeeeeeeeeee (words) (words) max [nx_run_gueries]
72
return queries
73
74 candidate_qurier find_noua www.aeeaunassseeeeeeeeeeeeeeeeee icaptien]
For CLOC, we focus on the training objective and frame- work formulation, while making minimal efforts on hyper- parameter tuning, architecture search, dataset cleaning, and etc., thus better performance could be achieved. Besides, although we have included extensive standard evaluation tasks, the finengrained region knowledge could also be use-
Listing f. Python example codes for Step 2 VESL Section for extracting text candidate queries from a caption.
15

---

Contrastive Localized Language-Image Pre- Tre-Trainn
ful more other under-explored tasks.
paper focuses on.
Future directions. We suggest promising future direc- tions. In Section 3.2 our P prmppee formulation can take flexible prompts to guide the embeddings for specific tasks. In this work, we consider prompt as a single bounding box or a text caption, but it has the potential to expand to various types such as points, a mask, users' free-form re- ferring, or multiple prompts in multiple types together. We think a more versatile Prompter with co-designs for dif- ferent objectives can have a big potential. Similarly, our VESL labeling pipeline limits to detection box format. An- notators supported for more formats may further boost it. We believe our approach is promising, as more attention has been drawn recently for better re-captions (Li et al.) 2024 Fan et al., 2024) that VESL relies on. In addition, CLOCC model provides a new capability to extract region features without further training, and thus can be used foundation model for exploring new VL applications.
(2) Why not just train a CLIP with object detection? One may wonder why we do not just train an encoder with joint optimization of the CLIP contrastive loss with some object detection loss instead of the CLOC design of [Equation 4
Although it sounds like plausible approach, we would like to point out that contrastive pre-training and object de- tection are fundamentally quite different in their technical rationales. CLIP pre-training is often on large batches of low-resolution and noisy images, while object detection is trained on small batches of high-resolution images. CLIP is by default trained from scratch and object detection is typ- ically initialized from pre-trained encoders and focuses on the detection head. Furthermore, detection requires heavy computation on box proposals to detect all boxes appear- ing in image, while region-text contrastive design allows us to flexibly sample fewer regions per image as motivated in Equation 3| Overall, their data pipeline and distributed training setup are not on the same scale thus such joint training may not be very reasonable.
Training cost. We comment on the computation cost of our framework. Our large models (ViT L/14) were trained 1024 v5p TPUs for about 6 days. To optimize Equation 2] CLOC needs extra computation. The main overheads come from the contrastive matrix but not the lightweight P Prompter. Fortunately, we found it feasible since (I) only a few boxes in each image need to be sampled per update; (2) the loss computation becomes smaller pro- portion when the ViT scales up. Overall, we found the computation acceptable compared to CLIP. More memory- efficient optimization like SigLIP (Zhai et al. 2023) can be implemented with JAX shard Shard_mar ops.
With that being said, some previous works do have attempts that are the exceptions but only for some but not all of the mentioned aspects, and mainly for the purpose of detection. For instance, DetCLIP-v2 Yao et al. (023b) adds image- text contrastive loss into detection loss to improve open- vocabulary capability for detection. OWLv2 pre-trains the detector with rather small resolutions but still with a batch size of a maximum 256 since each image will need to pre- dict up 100 boxes during training. Both DetCLIP-v2 and OWLv2 fine-tune from a pre-trained encoder.
Discussions on design rationals. Besides the main discus- sions we have stressed in the main text, here we provide more thoughts behind our design rationales that a reader may be wondering.
On the contrary, we study pre-training the encoder from scratch, which may be complementary to the previous ef- forts. CLOC maximizes the similarity in co-design with CLIP, thus making it much easier to develop within the same codebase.
(I) Why not use local-enhanced encoder? We would like to note that many encoders with great localization like DINOv2 (Oquab et a 2023). OWLv2 (Minderer et al.) 2024). CLIP-Self (Wu et al. 2023), etc. are developed specifically for dense vision tasks that cannot perform im- age zero-shot tasks like CLIP and CLOCC We would like to emphasize that our goal is build a trop-iin-eellacmment of CLIP encoder with better localization, without sacrificing CLIP's original capabilities such image zero-shot tasks and its important backbone position for MLLMs. Further- more, perhaps well-known within the MLLMs community, these encoders have been shown in recent reports that they are not comparable enough to compete with CLIP as the vi- sion backbone for MLLM tasks (Tong et al. 2024) due to CLIP's superiority in vision-language alignment. We thus believe enhancing CLIP itself is more demanding as this
(3) Do we really need train CLOC from scratch? What if we fine-tune from CLIP? As CLIP pre-training is expen- sive, one may wonder if it is necessary to train from scratch on the proposed region-text datasets, or if we can initial- ize from a standard CLIP trained on image-text pairs only and fine-tunes with CLOC for a shorter stage. Our early investigation, even with extensive hyper-parameter tuning. suggests it is likely to be suboptimal compared to training from scratch directly. For instance, we initialize from the CLIP model 2 in Table 3|and fine-tunes it for another extra 100K steps with the CLOC training loss Equation 4 The model reaches 64.1% 99%%.11%%% mAce on COCO/LVIS region recognition, which is much worse than 10.1% 27. of the coineddfrom--craane model ② even with more overall training steps.
httg:://aa..aattthooosssooooeees map.html
16

---

Contrastive Localized Language-Ingg Image Pre- Training
Minderer et al., 2024 TAltTex ngrams)
Ours (VESL)
Avarety cstorul patlumed abdty notea amrrgged whas baokgrunn, offorig users a visollly appodling poloction
mossage card, massage cart, shop cont. womon
090-
asibrii aust Leman Zurchal& Chickpss
Lemon 2.8 7uoo? ae Chickpea Fils
Plcs, had with a warity ol regeadles indudin exotiit...
scde sdittle rprecierts. bananat, busai and sggs. The benanas ane postoree
ingredients needed
700
Piik Owars Drkandw Ovey Creatch Cargo Parta
degietedd standing in room, creading cocual and reloxod....
690
600
2 Picco Mooi Wall Dooor Set Throughout 4 Ploco Wall Dosor Belo By Chertio fiom 300
wall are eemplementedl by voons, one pooiiteeed roarthe
200
200
6/6
Figure A. Examples comparing our VESL and the labeling approach in (Minderer et al. 2024 that directly uses the n-grams of the crawled AltText. For VESL, each image is annotated with the visual-enriched caption replace the AltText, which is used to generate region text candidates that capture the image content better.
17


Reviews:
[{"review_id": "sBfSyLFUyo", "reviewer": "Reviewer_QmDj", "summary": "This paper proposes Contrastive Localized Language-Image Pre-training (CLOC), an approach extending CLIP-style image-text contrastive learning to also incorporate region-level alignment. The authors introduce a lightweight “Prompter” module that can transform global image embeddings into region-aware representations given bounding boxes. They further design a large-scale pseudo-labeling pipeline (VESL) to generate region-text annotations, resulting in a 2B dataset. Through extensive experiments across classification, retrieval, and multimodal reasoning tasks, the method demonstrates solid improvements over standard CLIP, particularly in fine-grained vision-language scenarios like referring expression comprehension and region-based VQA.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the positive review and constructive comments. \n\n> The quality and diversity of region-text pairs depend heavily on the open-vocabulary detector and captioning pipeline.\n\nThank you for pointing out this important aspect. We agree that the quality of the open-vocabulary detector and captioner is important. Our pipeline is built upon the recent advances in these models (L275, right). We think the proposed framework is promising since, with better and better detectors and captioners introduced, the pipeline can seemingly enjoy the improvements, such as less bias and errors. We briefly discussed it in L782 (left column) of the Appendix, and will further emphasize the pipeline’s dependency on these models for pseudo-labeling in the final version.\n\nTo further address the reviewer’s concern, we use the hallucination metric (CHAIR score, where lower is better) to assess the quality of the synthetic captions. As shown below, our captioner demonstrates high quality compared to other models.\n\n| Captioner | CHAIR_i | CHAIR_s |\n|--------|----------|----------|\n| InstructBLIP | 14.5 | 30.0 | \n| MiniGPT-4 | 8.2 | 24.2 |\n| Shikra | 7.0 | 22.0 |\n| LLaVA-1.5 | 6.2 | 20.6 |\n| Ours | 5.9 | 19.6 |\n\n> Text-based region prompts\n\nIn our experiments, we observed a reasonably low L1 distance of 0.02 between the predicted boxes and ground-truth boxes when the Prompter receives a text region description as input, indicating that text-based region prompts performed well out of the box. In the revision, we will include qualitative visualization examples to illustrate this. A more in-depth investigation is left for future work.\n\n> Region caption is short \n\nThe reviewer is correct that the region-level captions are much shorter than the image-level captions. However, we want to clarify that in Equation 4, we still retain the original image-level CLIP loss, ensuring that the image embedding quality remains on par with the original CLIP, as evidenced by the “Image tasks” results in Table 2.\n\n> Comparison to open-vocab detection models \n\nFirst, in our paper, we included a comparison in Footnote 1 (L379, left), demonstrating that on the region classification task (predicting class names given a bounding box), our approach achieves over 70% mAcc on COCO, significantly outperforming the 47% reported in previous work. \n\nFor comparison with open-vocabulary detection models, we also provide zero-shot evaluation results on COCO Detection (minival), ODinW (test-dev), and LVIS-Det (minival). When comparing GLIP [1] and CLOC, we observe that CLOC consistently achieves better results than GLIP across all backbone categories (T / B / L), suggesting that CLOC offers advantages in localization and object detection performance. Notably, GLIP employs DyHead—a strong decoder/head module—on top of the encoder, whereas our ablation study uses only two simple heads for classification and regression. This further supports that the encoder representation in CLOC is indeed superior. See the table below for detailed results.\n\n| Model | ViT | COCO-Det (minival) | ODinW (test) | LVIS-Det (minival) |\n|--------|----------|---------------------|--------------|--------------------|\n| GLIP-T | ViT-T/16 | 46.6 | 46.5 | 26.0 |\n| GLIP-L | ViT-L/14 | 49.8 | 52.1 | 37.3 |\n| CLOC-B | ViT-B/16 | 47.3 | 48.4 | 29.6 |\n| CLOC-L | ViT-L/14 | 50.8 | 53.6 | 38.1 |\n\n> How bounding boxes are sampled during training\n\n During training, we simply sample 4 boxes randomly (padded if fewer) per image.\n \n> Designs of the prompter\n\nIn our experiments, we mainly consider the prompter takes a bounding box or a single text embedding as the prompt. We compared with a baseline implementation of RoIAlign in Table 2 (row 4, 9, 14), and confirm that the proposed prompter is a better design (discussion in L360 right and also Section 3.4). For other designs, we consider the following as promising future work: (1) different types of prompts, such as points, masks, etc.; (2) multi-prompts or compositional prompts for higher-level prompting. We included more discussions in L770 “Future directions” paragraph.\n\n> LocCa reference \n\nWe thank the reviewer for this suggestion. LocCa is indeed a relevant work which we have cited and discussed in Section 2 (L100, right). However, LocCa differs significantly from our approach in two important ways: (1) it employs a full encoder-decoder transformer architecture, thus being substantially less efficient, especially for large-scale training; (2) LocCa embeddings do not directly facilitate zero-shot retrieval or classification tasks as our embeddings do (Table 2). The focus of our method remains specifically on improving CLIP-based localization capabilities, and we will further clarify this distinction in our final revision.\n\n[1] Grounded Language-Image Pre-training, CVPR 2022.", "claims_and_evidence": "- Claim: CLOC enhances fine-grained visual understanding in downstream tasks that require identifying or referring to specific image regions.\n\n  Evidence: Experiments on region-level classification and retrieval benchmarks (e.g., COCO, GRIT) and on MLLM tasks (Ferret, LLaVA) show that CLOC consistently outperforms the CLIP baseline in tasks needing spatial grounding.", "methods_and_evaluation_criteria": "Yes. The proposed methods make sense.", "theoretical_claims": "No theoretical claims are made.", "experimental_designs_or_analyses": "Yes. I have checked the experimental designs.", "relation_to_broader_scientific_literature": "This work is a CLIP Extension, which tries to incorporate region-level alignment.", "essential_references_not_discussed": "The author should discuss works such as [*1].\n\n[*1] Wan, Bo, et al. \"Locca: Visual pretraining with location-aware captioners.\" Advances in Neural Information Processing Systems 37 (2024): 116355-116387.", "other_strengths_and_weaknesses": "**Strengths**\n- The prompter module is lightweight, which only causes minimal overhead compared to baseline CLIP.\n- The performance looks promising. From the table and Figure, we can see consistent improvements in fine-grained tasks, region-level retrieval, and large multimodal model reasoning.\n- The data creation pipeline is scalable and might make a great contribution to the community.  \n\n**Weaknesses**\n-  The quality and diversity of region-text pairs depend heavily on the open-vocabulary detector and captioning pipeline—if these pipelines introduce bias or errors, the final model inherits them.", "other_comments_or_suggestions": "- It would be interesting to see if text-based region prompts (e.g., referencing “the person on the left”) work well out of the box at inference without bounding boxes.", "questions_for_authors": "- The length of the average of the text per caption is 2.1, which is much shorter than RefCOCOg or WiT. Will this potentially affect the expression performance of the image embedding?\n- Is it possible to include some direct comparison of CLOC to dedicated open-vocabulary detection models (e.g., in terms of box AP metrics)?\n- Can you provide more detail on how the bounding boxes are sampled in the training?\n- Did the authors consider other designs of the prompter?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fEQwqj5PyR", "reviewer": "Reviewer_anvz", "summary": "The submission introduces a new pre-training method called Contrastive Localized Language-Image Pre-training (CLOC). The pre-training method extends CLIP pre-training with additional losses based on the outputs of a new \"Prompter\" module. This new module consists of a light-weight transformer layer that enhances CLIP image embeddings for regional losses (similarity to bounding box, and grounding of region description).\n\nFor training CLOC, the paper also introduces a new captioning pipeline termed Visually-Enriched and Spatially-Localized (VESL). This pipeline first generates detailed image captions and then uses a text-conditioned zero-shot detection model to generate bounding boxes for sub-queries of the caption generated by named-entity recognition.\n\nThe paper then compares a CLOC model with a CLIP baseline that is trained on the same data using the same hyper parameters but without the CLOC losses. The performance is compared on Ferret bench, RefLVIS, RefCOCO, and various VQA benchmarks.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the positive review and constructive comments. \n \n> The main weakness of the work is the superficial ablation of the VESL captioning pipeline introduced. \n\nThank you for your constructive comments, and we will consider better examples in our figures. We agree with the reviewer that image captioning could suffer from hallucination, though it enriches the visual description. The reviewer is also correct that our pipeline filters such cases by relying on the object detector, which is quite effective as it was pre-trained on over thousands of common objects. A central design challenge for our pipeline is balancing the benefits of richer captions with the potential risk of hallucination. We believe our design is robust, considering the effectiveness of current state-of-the-art open-vocabulary object detectors and the inherent resilience of contrastive learning objectives to noisy text annotations in large-scale training.\n\nIn Table 2 (rows 3-5 vs. 13-15), we compare with the baseline that does not use the image captioner but the AltText (e.g., left of Figure 3). It could be hard to extract useful object phrases from AltText for the open-vocab detector. \n \nFor the hallucination issue, we have investigated our captioning pipeline and compared it with other recent works to show its superiority in terms of the hallucination. We compared it to public models: LLaVA-1.5, Shikra, MiniGPT-4, and InstructBLIP on CHAIR scores [1] (lower values refer to less hallucination). CHAIR_i measures the fraction of hallucinated object instances and CHAIR_s calculates the fraction of sentences containing at least one hallucinated object. The results are summarized below: \n\n| Captioner | CHAIR_i | CHAIR_s |\n|--------|----------|----------|\n| InstructBLIP | 14.5 | 30.0 | \n| MiniGPT-4 | 8.2 | 24.2 |\n| Shikra | 7.0 | 22.0 |\n| LLaVA-1.5 | 6.2 | 20.6 |\n| Ours | 5.9 | 19.6 |\n\nWhile the long captions from our captioning pipeline may still unavoidably introduce hallucinations even though our captioning pipeline has less hallucinations compared to other models, it can be further mitigated since we only consider confident objects agreed by the detector (L296 left). We also remove very generic words and stopwords, as noted in the code in Listing 1 in the appendix. We believe having more accurate object labels is the key to the improvements in Table 2 for our pipeline, as evidenced by the statistics 11.6 regions per image identified by the pipeline (only 5.1 for the baseline) we report in Table 1 (L275 right). \n\n> lines 260-262 (right column): are the pairs ignored or are the gradients on $f_T$ ignored?\n\nFor filtering region-text conflicts in Section 3.4, the region-text pairs are ignored in Equation 2. That is, these elements are “masked” and will not be considered in the contrastive loss matrix. We will make this clearer in the final version.\n \n> Remarks on CLIP\n\nWe apologize for any confusion regarding this statement. Our reference to “CLIP” in the context of MLLMs was intended to denote the broader family of language-supervised methods, including both CLIP (as a representative model) and SigLIP.  Specifically, we were citing the first row block in Table 12 of Section D (Tong, 2024), which demonstrates that these methods outperform others, such as self-supervised approaches, in MLLMs. We will make this clearer and revise both the introduction and related works as you suggested.\n\n> Typos\n\nThank you for pointing out the typos, and we have fixed them in the revised manuscript.\n\n[1] Rohrbach, A., Hendricks, L.A., Burns, K., Darrell, T. and Saenko, K., 2018. Object hallucination in image captioning. arXiv preprint arXiv:1809.02156.", "claims_and_evidence": "The paper claims that CLOC outperforms traditional CLIP on referring and grounding tasks. This claim is supported by Tables 3 and 4.\n\nThe paper also claims that CLOC unlocks zero-shot region-level capabilities. This claim is supported by Table 2.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are valid and make sense to evaluate the proposed method.", "theoretical_claims": "There are no theoretical claims in the paper.", "experimental_designs_or_analyses": "The experimental designs are appropriate to confirm the two claims mentioned above.\n\nThe authors first introduce their own reproduction of CLIP and compare it to the original OpenAI CLIP. Then they ablate various design decisions of their proposed method (CLOC / VESL) and report zero-shot performance on various image and region tasks.\n\nThe baseline CLIP and the improved CLOC models are then compared in different benchmarks to confirm the claims about improved capabilities with respect to referring and grounding tasks (Ferret-Bench, RefLVIS, RefCOCO, Flickr), and also improvements on some image-level multimodal benchmarks (Table 5).", "relation_to_broader_scientific_literature": "The pre-training recipe (CLOC) is mainly anchored on the original CLIP paper (Radford, 2021). This is done on the context of MLLMs that use CLIP as a vision encoder, and here the paper refers to (Tong, 2024). The idea of the light-weight Prompter module is introduced by referring to (Kirillov, 2023). The data annotation pipeline that is used to train the CLOC model is discussed in the context of previous works such as (Minderer, 2024) and (Kirillov, 2023).", "essential_references_not_discussed": "I think all essential references are mentioned.", "other_strengths_and_weaknesses": "The paper is well written and illustrated. The formulation of the added module and losses is very clear, and the paper does a great job at walking the reader through the process. I also enjoyed the way how the ablations are first presented, and discussed by comparing individual rows in the main text.\n\nThe main weakness of the work is the superficial ablation of the VESL captioning pipeline introduced. I do not find the selected examples in Figure A very convincing, and it's easy to imagine that captioning hallucinations might create problems (although they might be filtered by the object detection model), and that the strict named-entity recognition loses a lot of the interesting information (note that the other figures are a bit misleading in this respect, e.g. a description like \"a stunning ocean view\" would never be extracted by NER). From the numbers presented in Table 2, I'm a bit puzzled to see such different effects of rows 3-5 vs. 13-15.", "other_comments_or_suggestions": "Missing clarity:\n1. lines 260-262 (right column): are the pairs ignored or are the gradients on $f_T$ ignored?\n\nTypos:\n1. line 117: \"Another less and arguably more\"\n1. line 124: \"and are more computation overhead\"\n1. line 276: \"annotates it\"\n1. line 294 (right column): \"We implement the in JAX\"\n1. line 429 (right column): \"in the foresee of\"\n\nVarious remarks:\n\n1. \"CLIP has become arguably the default choice of vision backbone for multimodal large language models (MLLMs) (Liu et al., 2023; McKinzie et al., 2024) due to its superior prior knowledge in aligning vision and language (Tong et al., 2024).\" – Looking at (Tong, 2024) Section D, it's not clear to me how this reference would give evidence to the claim that CLIP is the default choice due to its superior prior knowledge (e.g. vs. SigLIP). Consider backing up this claim more clearly, or re-formulating (both in \"Introduction\" and \"Related Work\" sections).", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "No concerns."}}, {"review_id": "T9E6S3AgnN", "reviewer": "Reviewer_rnyd", "summary": "This work introduces a dynamic attention mechanism, inspired by SAM, to aggregate regional image features and perform contrastive learning at both the image-text and region-text levels. The approach is novel in the context of visual pretraining.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the positive review and constructive comments. \n\n> Unclear whether the performance gains stem from the use of more or cleaner data or from the pretraining schema itself.\n\nIn Table 2, we provide detailed ablations of our proposed ingredients on top of the CLIP we trained by ourselves, including the prompter design, the training labels generated by our pipeline. Note that, the CLIP model (row 2) we compared in the experiments was trained on the same image data as CLOC for a fair comparison.\n\n> Can this work be seamlessly extended to zero-shot object detection?\n\nYes, CLOC can be extended to zero-shot object detection. We provide additional evaluation results on COCO Detection (minival), ODinW (test-dev), and LVIS-Det (minival). When comparing GLIP [1] and CLOC, we observe that CLOC consistently achieves better results than GLIP across all backbone categories (T / B / L), suggesting that CLOC offers advantages in localization and object detection performance. Notably, GLIP employs DyHead—a strong decoder/head module—on top of the encoder, whereas our ablation study uses only two simple heads for classification and regression. This further supports that the encoder representation in CLOC is indeed superior. See the table below for detailed results.\n\n| Model | ViT | COCO-Det (minival) | ODinW (test) | LVIS-Det (minival) |\n|--------|----------|---------------------|--------------|--------------------|\n| GLIP-T | ViT-T/16 | 46.6 | 46.5 | 26.0 |\n| GLIP-L | ViT-L/14 | 49.8 | 52.1 | 37.3 |\n| CLOC-B | ViT-B/16 | 47.3 | 48.4 | 29.6 |\n| CLOC-L | ViT-L/14 | 50.8 | 53.6 | 38.1 |\n \n> A broader comparison with additional methods\n\nThank you for the suggestion. We agree that a broader comparison to more encoders would certainly be great. However, many previous models (LocCa, RegionCLIP) are trained on quite different data, labels, training cost, architectures etc. that might make it hard to draw a fair comparison directly, and some of them are not open-sourced. Therefore, we have limited the scope of our paper to the CLIP method and carefully ablated it in the same setting (e.g., training images, number of steps) as closely as possible. \n \n> More discussion and comparisons on training efficiency (e.g., with CLIP, LocCa)\n \nWe provide discussion in the “Training cost” paragraph (L789 left). Compared to CLIP, the extra cost is small for the object-level contrastive loss and the prompter — we observe about 10% more GPU time. Notably, the lightweight prompter operates on the image embedding that is shared across all the prompts within an image. The main overhead is to compute the image embedding through the ViT, which does not scale with the number of prompts. Compared to LocCa, our CLOC is much more lightweight since LocCa needs a full encoder-decoder transformer for autoregressive next-token prediction. \n\n> Attribute binding issue of CLIP\n\nIn this paper, indeed, we do not tailor to address the attribute binding shortcoming of the original CLIP directly. However, with our promptable embedding design (Figure 2 & Section 3.2), we think CLOC provides an alternative approach that allows the users to interact with our encoder to obtain a fine-grained embedding given a prompt of their interest by specifying a box location or an object description. \n\n> Minor fixes for Eq2 and Eq4\n\nThanks for pointing them out. We will revise them accordingly. \n\n> Will you release the pre-trained model (including the Prompter) to the public?\n\nYes, we aim to release the pre-trained model, and are actively working on that. \n\n[1] Grounded Language-Image Pre-training, CVPR 2022.", "claims_and_evidence": "The experiments (Table 2, 3) demonstrate strong results in region-aware visual pretraining compared to CLIP. However, it remains unclear whether the performance gains stem from the use of more or cleaner data or from the pretraining schema itself.", "methods_and_evaluation_criteria": "The methods seem reasonable and elegant to me. However, the evaluation should be more comprehensive, including comparisons with other location-aware pretraining methods (e.g., LocCa, RegionCLIP) on the same benchmarks, such as RefCOCOs.", "theoretical_claims": "I checked, please see \"Other Strengths And Weaknesses\"", "experimental_designs_or_analyses": "Can this work be seamlessly extended to zero-shot object detection?", "relation_to_broader_scientific_literature": "Visual encoder pretraining", "essential_references_not_discussed": "It covers well, need discuss more on training efficiency.", "other_strengths_and_weaknesses": "Weakness:\n\n1. A broader comparison with additional methods, as mentioned above.\n2. More discussion and comparisons on training efficiency (e.g., with CLIP, LocCa) would be valuable.\n3. The general issue of attribute binding, inherited from CLIP, is not well addressed.\n\nMinor:\n1. For Eq2, Please pay attention to the superscripts in the formula, especially since $m'$ is not defined.\n2. For Equ4, please well define $L_{CLOC}$", "other_comments_or_suggestions": "See above", "questions_for_authors": "Will you release the pretrianed model (including the Prompter)  to the public?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W9SgW6mJmG", "reviewer": "Reviewer_JoXU", "summary": "This paper explores a data-driven approach to enhance the regional representation capabilities of CLIP. The authors designed a data annotation pipeline to expand regional-level annotations and developed a training architecture featuring a Prompter. This architecture enables more effective utilization of the annotated data for fine-grained training. Experimental results, particularly those obtained under certain MLLM settings, demonstrate the advantages of the proposed method.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the positive review and constructive comments. \n\n \n> UMG-CLIP \n\nWe thank you for pointing out the reference that we briefly compared in L363 left. We agree that there is some technical similarity between our CLOC and UMG-CLIP, but the goals and positioning of our work and theirs are quite different from the following perspectives. \n\nUMG-CLIP is designed primarily for vision-centric tasks such as detection and segmentation, whereas our focus is on MLLMs (L33, right in the Introduction). Unlike open-vocabulary vision tasks, which involve a relatively small set of classes, MLLM VQA tasks require more extensive language understanding and thus demand large-scale pre-training data. This distinction also influences the design of our prompter architecture, which incorporates attention layers tailored to the downstream use cases of the LLM decoder (further discussed in L256, left).\n \nRegarding the annotation pipeline, the primary difference lies in scale. UMG-CLIP fine-tunes a pre-trained CLIP model for dense vision tasks on a 41M-image dataset, whereas our approach pre-trains from scratch on up to 2B images. Additionally, unlike UMG-CLIP and RegionCLIP, which first predict bounding boxes and then run a captioner on each box, our VESL pipeline (Section 4) does not scale with the number of boxes, making it significantly more efficient for data annotation. Notably, annotating billions of images for our experiments required over 500+ GPUs for more than a week (L757, right). \n\nLast but not the least, we see the UMG-CLIP fine-tuning approach as complementary to our CLOC pre-training rather than conflicting with it.\n \n> Open-world detection \n\nWe thank you for the advice on evaluating our encoder on open-vocab dense vision tasks. First, we want to emphasize that our original motivation for this work is for MLLM tasks with localization use cases such as conversational referring and grounding (e.g., Table 4), but not for dense vision tasks. And in Table 2, we have reported competitive results of zero-shot object recognition and retrieval given bounding boxes.\n\nTo further address your concern, for open-vocabulary detection, we provide additional zero-shot evaluation results on COCO Detection (minival), ODinW (test-dev), and LVIS-Det (minival). When comparing GLIP [1] and CLOC, we observe that CLOC consistently achieves better results than GLIP across all backbone categories (T / B / L), suggesting that CLOC offers advantages in localization and object detection performance. Notably, GLIP employs DyHead—a strong decoder/head module—on top of the encoder, whereas our ablation study uses only two simple heads for classification and regression. This further supports that the encoder representation in CLOC is indeed superior. See the table below for detailed results.\n\n| Model | ViT | COCO-Det (minival) | ODinW (test) | LVIS-Det (minival) |\n|--------|----------|---------------------|--------------|--------------------|\n| GLIP-T | ViT-T/16 | 46.6 | 46.5 | 26.0 |\n| GLIP-L | ViT-L/14 | 49.8 | 52.1 | 37.3 |\n| CLOC-B | ViT-B/16 | 47.3 | 48.4 | 29.6 |\n| CLOC-L | ViT-L/14 | 50.8 | 53.6 | 38.1 |\n\n[1] Grounded Language-Image Pre-training, CVPR 2022.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No proofs for theoretical claims.", "experimental_designs_or_analyses": "The experimental designs and analyses in the paper have several sound aspects, but also some areas that could be further explored. Please refer to the 'Weaknesses' part for more details.", "relation_to_broader_scientific_literature": "Compared with previous work, the main contributions of this paper are: 1) construction of a large-scale region-text dataset; 2) design of a fine-grained pre-training paradigm; 3) exploration of scalability in MLLM scenarios.", "essential_references_not_discussed": "Please refer to the 'Weaknesses' part for more details.", "other_strengths_and_weaknesses": "I believe this is a high-quality paper, and I greatly appreciate the authors' contributions in terms of the data pipeline, training architecture, and MLLM evaluation. However, I have the following concerns:\n1. UMG-CLIP [ECCV 2024] focuses on similar issues as this paper. Nevertheless, there is a lack of comparison with it in the authors' discussion. I expect to see more in-depth discussions, including:\na) Data annotation process. UMG-CLIP first uses an open-vocabulary detector to predict bounding boxes and then generates descriptions for each box. In contrast, this paper first identifies entities and then predicts bounding boxes for each entity (somewhat similar to RegionCLIP). The advantages and disadvantages of these two pipelines need to be explored. In particular, I noticed that UMG-CLIP claims to have good accuracy for bounding boxes (Table 14 in their paper), and its region-level descriptions seem to be more detailed (Figure 3 in their paper).\nb) Training architecture. The main difference between this paper and UMG-CLIP appears to be the replacement of ROI-ALIGN with a prompter. The authors claim that this is because the inaccurate pseudo-annotations limit the effectiveness of ROI-ALIGN. If this is the case, could ROI-ALIGN be considered for use when ground-truth annotations are available? Additionally, the paper lacks an evaluation of annotation accuracy. Does this imply that the bounding boxes generated by this paper are of poor quality, and the training architecture is a compromise due to the low-quality data?\n2. I look forward to seeing more validation results, such as experimental verifications in open-world detection (similar to RegionCLIP) or segmentation scenarios (similar to UMG-CLIP).", "other_comments_or_suggestions": "Please refer to the 'Weaknesses' part.", "questions_for_authors": "Please refer to the 'Weaknesses' part.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes Contrastive Localized Language-Image Pre-training (CLOC), a CLIP-like pretraining method that enhances region-level vision-language alignment using a new lightweight Prompter module and a large-scale region-text annotation pipeline. It improves fine-grained recognition and retrieval tasks, such as referring and grounding tasks, outperforming standard CLIP.

After the author-reviewer discussion, the reviewers reached a positive consensus (1 strong accept, 2 accept, 1 weak accept). Their positive comments are mostly based on the strong and solid experimental results.

I also agree with the reviewers. Overall, I recommend "Accept" for this paper.
Decision: Accept (poster)