Contrastive Localized Language-Image Pre-Training

## Abstract

Abstract Contrastive Language-Image Pre-training (CLIP) has been a celebrated method for training vision encoders to generate image/text representations facilitating various applications. Recently, CLIP has been widely adopted as the vision backbone of multimodal large language models (MLLMs) to connect image inputs for language interactions. The success of CLIP as a vision-language foundation model relies on aligning web-crawled noisy text annotations at image levels . Nevertheless, such criteria may become insufficient for downstream tasks in need of fine-grained vision representations, especially when region-level understanding is demanding for MLLMs. In this paper, we improve the localization capability of CLIP with several advances. We propose a pre-training method called C ontrastive Loc alized Language-Image Pre-training ( CLOC ) by complementing CLIP with region-text contrastive loss and modules. We formulate a new concept, promptable embeddings , of which the encoder produces image embeddings easy to transform into region representations given spatial hints. To support large-scale pre-training, we design a visually-enriched and spatially-localized captioning framework to effectively generate region-text pseudo-labels at scale. By scaling up to billions of annotated images, CLOC enables high-quality regional embeddings for image region recognition and retrieval tasks, and can be a drop-in replacement of CLIP to enhance MLLMs, especially on referring and grounding tasks.

## 1 Introduction

Large-scale vision-language (VL) pre-training has been an important foundation for the recent tremendous growth of multimodal applications. Contrastive Language-Image Pre-training (CLIP) (Radford et al., ; Jia et al., ) has become a great success of VL representation learning that connects images and text by contrastive training on web-crawled image-text pairs. It has been proven strong transferability and generalizability on extensive downstream tasks such as zero-shot image classification and image-text retrieval.
Even beyond, CLIP has become arguably the default choice of vision backbone for multimodal large language models (MLLMs) (Liu et al., ; McKinzie et al., ; Zhang et al., ) due to its superior prior knowledge in aligning vision and language (Tong et al., ), facilitating vision inputs to be injected into language models.

Figure: Figure 1: Overview of our CLOC pre-training framework. (1) A visually-enriched and spatially-localized captioning pipeline generates pseudo-labeled bounding boxes with detailed descriptions for key image regions. (2) A lightweight Prompter attached on top of the CLIP image encoder can be prompted to transform the image embedding into the region-focused feature. All parameters are trained end-to-end from scratch with our contrastive localized language-image loss on the annotated region-text datasets. After pre-training, (3a) region features can be generated via the Prompter for region-text tasks like object classification in a training-free fashion. (3b) The image encoder, along with the optional Prompter, can also strengthen MLLMs fine-tuning by enhancing their fine-grained image understanding capabilities.
Refer to caption: extracted/5897759/figures/cloc_overview.png

As VL research gets increasing attention and expedites progress, various more advanced multimodal tasks are demanding stronger vision capabilities. For instance, recent MLLMs (Rasheed et al., ; Ren et al., ; Lai et al., ; Chen et al., ; Peng et al., ) have been focusing on more fine-grained understanding tasks that require comprehension of the semantic at *region levels* such as visual question answering (VQA) with referring and grounding instructions. These MLLMs are fine-tuned on referring and grounding data with CLIP as the vision backbone, as seen in works like Kosmos-2 (Peng et al., ) and Ferret (You et al., ; Zhang et al., ). Due to the need for such region-level understanding, CLIP, which aligns entire images with text captions, seems insufficient, as its regular image-text contrastive loss primarily emphasizes global semantics.

To remedy such core localization capability for CLIP, we ask a challenging and fundamental question: *without sacrificing CLIP’s original strong image-level knowledge, can we pre-train a stronger image encoder with enhanced localization capability that can be inherently integrated into MLLMs?*

To this end, we explore a data-driven approach that complements the original CLIP image-text pre-training objective with explicit region-text supervision. Though conceptually simple, several challenges exist. *First*, it lacks public datasets with region-text annotations at scales large enough for CLIP training, which typically requires hundreds of millions even billions of images. Existing region-text corpus like Visual Genome (Krishna et al., ) contains about $108$K images, and the largest noisy-labeled grounded dataset GRIT (Peng et al., ) features only $20$M images. Indeed, such deficiency has probably limited the literature to mainly consider semi-supervised or weakly-supervised approaches as somewhat a compromise (Naeem et al., ; Yao et al., ; ).

*Second*, a plausible solution is to scale up training data in pursuit of image regions pseudo-labeled with text annotations via some open-vocabulary detectors (Minderer et al., ; Zhang et al., ). Though it seems feasible, we found it non-trivial to design such a pipeline as the annotations are noisy and will greatly affect the final model performance.
*Third*, even if the region-text datasets are given, it is under-explored how to effectively train on them in terms of co-designs of training objectives, model architecture, and more design details.

To this end, we propose a new pre-training framework illustrated in Figure 1, named Contrastive Localized Language-Image Pre-Training (CLOC), to improve CLIP with better localization capability, especially for MLLMs, by overcoming the above difficulties. Our main contributions are:

- •
We propose a new learning goal, Promptable Embeddings, that *a strong vision encoder should produce image embeddings that can be easily transformed into region representations, given some spatial hints (e.g., box referring or text prompts)*. This formulation not only facilitates the encoder as a prior of fine-grained VL alignment, but also enables new possibilities for the interactions between the image encoder and the language decoder.
- •
To optimize towards the goal, we design simple and minimal modifications on top of CLIP. We augment the original CLIP loss with a region-text contrastive loss, where the region embeddings are extracted from the image embedding by a lightweight extractor module conditioned on the spatial hints (*i.e.*, *prompts*).
- •
We design a large-scale pseudo-labeling data engine to support CLOC training. We properly combine visual-enriched image captioners and open-vocabulary detectors for an effective recipe that improves previous practice of region annotations (Minderer et al., ; Peng et al., ). This approach yields a two-billion image-text dataset with fine-grained region-text annotations, which serves as the foundation for training our CLOC model.
- •
Through extensive experiments across 31 evaluation tasks, including standard image-text tasks, newly constructed region-text tasks, and downstream evaluations with MLLMs, we demonstrate that CLOC significantly and consistently outperforms the CLIP counterpart.
- •
We are working on releasing our pre-trained checkpoints and the constructed region-text annotations along with the final version to accelerate future research within the community.

## 2 Related Work

#### Improving localization of CLIP.

Since CLIP was introduced, many follow-up works have been proposed to improve it from various aspects, for different target tasks, and with different approaches.
From the aspect relevant to our work, improving the localization capability, most works specifically focus on the downstream dense vision tasks such as open-vocabulary detection (Minderer et al., ; Yao et al., ; Wu et al., ). Another less and arguably more challenging thread is to maintain the generalizability of CLIP on image-level tasks while improving localization. Recent works like SILC (Naeem et al., ) and SPARC (Bica et al., ) combine localization-enhancing unsupervised objectives with the CLIP loss, but do not attempt with supervision on large-scale explicit pseudo-labeled data like ours. Alpha-CLIP (Sun et al., ) shows that the SAM segmentation model (Kirillov et al., ) can provide useful conditions for CLIP.

#### Vision encoder pre-training for MLLMs.

Building upon the success of large language models (LLMs), a popular approach to MLLMs like LLaVA (Liu et al., ), typically connects a vision encoder (*e.g.*, ViT (Dosovitskiy et al., )) to digest visual inputs and maps them to the LLM decoder input space as token embeddings. Among various types of vision encoders (Oquab et al., ; He et al., ), CLIP (Radford et al., ; Jia et al., ) becomes the most popular choice, due to its superior performance on MLLM benchmarks reported by recent studies (Tong et al., ).

#### Synthetic annotations for pre-training.

Large-scale training data are the fuel of pre-training, especially for CLIP. The literature has been exploring scalable ways to generate high-quality synthetic annotations. For instance, several works demonstrate that visually-enriched image captions improve CLIP (Lai et al., ). MOFI (Wu et al., ) constructs a large alt-text set and augments CLIP with a multi-classification task. However, these works only consider image-level annotations but not explicit region-level labels. In the context of dense vision tasks like open-vocabulary detection and segmentation, pseudo-labeling in a self-training paradigm has proven an effective approach (Kirillov et al., ; Minderer et al., ). We are inspired by these efforts and build on them to enhance CLIP’s localization capabilities.

## 3 CLOC : Contrastive Localized Language-Image Pre-Training

### 3.1 Preliminary: from Image-Text to Region-Text Alignment

Contrastive Language-Image Pre-training (CLIP) (Radford et al., ) trains a pair of image and text encoders (denoted as $f_{I}$ and $f_{T}$, respectively) by contrastively aligning the image and text embeddings. Let a mini-batch of $N$ image-text pairs $\{({\bm{x}}_{i},{\bm{y}}_{i})\}_{i=1}^{N}$ be sampled from the large-scale training set during each training iteration. The contrastive loss is defined as follows:

$$ ${\mathcal{L}}_{\text{CLIP}}:=({\mathcal{L}}_{I\rightarrow T}+{\mathcal{L}}_{T \rightarrow I})/2.\>{\mathcal{L}}_{I\rightarrow T}:=-\frac{1}{N}\sum_{i=1}^{N} \log\frac{\exp\big{(}\texttt{sim}(f_{I}({\bm{x}}_{i}),f_{T}({\bm{y}}_{i}))/ \tau\big{)}}{\sum_{j=1}^{N}\exp\big{(}\texttt{sim}(f_{I}({\bm{x}}_{i}),f_{T}({ \bm{y}}_{j}))/\tau\big{)}},$ (1) $$

where $\texttt{sim}(\cdot,\cdot)$ is the similarity measurement function and $\tau$ is a (learnable) logit temperature. The CLIP loss ${\mathcal{L}}_{\text{CLIP}}$ averages the symmetrical contrastive loss in which cross-entropy normalized along image-to-text and text-to-image axes, respectively.

Conceptually, the CLIP loss aligns images with their associated text, but it overlooks regional information and spatial semantics. We propose augmenting this with *region-text* alignment on top of ${\mathcal{L}}_{\text{CLIP}}$. Specifically, assume an image-text pair $({\bm{x}},{\bm{y}})$ can be decomposed into image regions ${\bm{x}}^{(1)},\dots,{\bm{x}}^{(m)}$, and there exist fine-grained captions ${\bm{y}}^{\prime(m)}$ that describe the corresponding image regions ${\bm{x}}^{(m)}$. Thus, the original input $({\bm{x}},{\bm{y}})$ becomes $\{({\bm{x}}^{(1)},{\bm{y}}^{\prime(1)}),\dots,({\bm{x}}^{(m)},{\bm{y}}^{\prime
(m)})\}$ for region-text considerations, and $({\bm{x}},{\bm{y}})$ is a special case when the “region” itself is the whole image. Based on this, we identify several research questions and will answer them in the following sections:

- 1.
Considering the goal is to train an image encoder $f_{I}$ with enhanced localization capability, how should we formulate a region-text alignment goal that improves $f_{I}$? We propose a novel learning task called *promptable embeddings* in Section 3.2.
- 2.
How to properly extract region embedding from $f_{I}({\bm{x}})$ as an effective joint design? We propose a lightweight promptable region extractor in Section 3.3.
- 3.
How to generate meaningful image regions with high-quality captions? Furthermore, in many cases, the ideal region caption ${\bm{y}}^{\prime(m)}$ may not exist in the image-level caption, *i.e.*, ${\bm{y}}^{\prime(m)}$ might not be a substring of the original ${\bm{y}}$. We design an effective and scalable data engine as a visually-enriched and spatially-localized labeler to generate high-quality region-text pairs in Section 4.
- 4.
With the above considerations, we discuss how to train the model with minimal conflicts towards a drop-in replacement of the CLIP model in Section 3.4.

Figure: Figure 2: CLOC promptable embedding architecture. CLOC builds upon the image embedding from CLIP (before pooling and projection) and transforms it into a region-aware vision embedding given an encoded prompt; *e.g.*, positional encodings of box coordinates or regional caption encoded by the CLIP text encoder.
Refer to caption: extracted/5897759/figures/cloc_arch.png

### 3.2 Promptable Embeddings

To optimize CLIP with better feature localization and eventually learn an enhanced CLIP vision encoder $f_{I}$ for various VL downstream tasks, we argue that it will require at least two capabilities. ($i$) First, the encoder should recognize fine-grained small objects (*e.g.*, this image crop is an “airplane wheel”). ($ii$) Second, the *image embedding* produced by the encoder provides a holistic understanding such that an MLLM can reason more advanced spatial hierarchy relationships within the scene (*e.g.*, “The plane is lowering its front landing gear.”). As discussed in Section 2, many previous works improve CLIP toward object detection tasks thus mainly focusing on ($i$) only; *e.g.*, RegionCLIP (Zhong et al., ) that crops out image regions and uses them as additional input images to re-train the CLIP encoders for recognizing objects. However, to support comprehensive VL tasks, ($i$) is necessary but insufficient without ($ii$).

To achieve this, we introduce a new concept, *promptable embedding*. We consider a scenario similar to MLLM use cases, where answers are generated using CLIP image tokens alongside a question. We hypothesize that a strong encoder for MLLMs should produce an *image* embedding that can *easily be transformed into region representations, given location cues*.

We re-formulate the CLIP loss based on image-text pairs $({\bm{x}},{\bm{y}})$ into a localized language-image contrastive loss for region-text alignment based on triplets of $(\{{\bm{l}}\},{\bm{x}},{\bm{y}})$, where ${\bm{l}}$ is a location representation such as a bounding box, and possibly there are several boxes as a set $\{{\bm{l}}\}$ per image. To make it compatible with CLIP training, we construct a promptable embedding transform module, or in short, *region prompter* ${\bm{z}}=\texttt{Prompter}({\bm{l}},f_{I}({\bm{x}}))$, that extracts the region embedding specified by ${\bm{l}}$ from the image embedding $f_{I}({\bm{x}})$. This formulation is inspired by the success of the segmentation model SAM (Kirillov et al., ) that predicts the *segmentation masks* conditioned on location prompt (*e.g.*, a box), while CLOC predicts a *region embedding* conditioned on ${\bm{l}}$ instead.

To this end, we decompose the location-image-text triplets as localized region-text pairs. Let ${\bm{z}}_{i}^{(m)}=\texttt{Prompter}({\bm{l}}_{i}^{(m)},f_{I}({\bm{x}}_{i}))$ and ${\bm{y}}_{i}^{(m)}$ is the caption of the region specified by ${\bm{l}}_{i}^{(m)}$. ${\bm{l}}_{i}^{(m)}\in\mathbb{R}^{4}$ is the $m$-th box of image $i$ represented as two coordinates (*i.e.*, top-left and bottom-right corners). We then formulate a symmetric region-text contrastive loss similar to Equation 1:

$$ ${\mathcal{L}}_{R\rightarrow T}:=-\frac{1}{MN}\sum_{i=1}^{N}\sum_{{\bm{l}}_{i}^ {(m)}\sim\{{\bm{l}}_{i}\}}\log\frac{\exp\Bigl{(}\texttt{sim}\bigl{(}{\bm{z}}_{ i}^{(m)},f_{T}({\bm{y}}_{i}^{(m)})\bigr{)}/\tau\Bigr{)}}{\sum_{j=1}^{N}\sum_{{ \bm{l}}_{j}^{(m)}\sim\{{\bm{l}}_{j}\}}\exp\Big{(}\texttt{sim}\bigl{(}{\bm{z}}_ {i}^{(m)},f_{T}({\bm{y}}_{j}^{(m^{\prime})})\bigr{)}/\tau\Big{)}},$ (2) $$

where $M$ is the number of regions ${\bm{l}}_{i}^{(m)}$ sampled per image. We set $M=4$ by default. We will discuss implementing the Prompter in Section 3.3, and generating ${\bm{l}}_{i}^{(m)}$ with ${\bm{y}}_{i}^{(m)}$ in Section 4. ${\mathcal{L}}_{T\rightarrow R}$ is the symmetric contrastive loss normalized along text-to-region axis, just like in Equation 1. We define ${\mathcal{L}}_{\text{{CLOC}}}=({\mathcal{L}}_{R\rightarrow T}+{\mathcal{L}}_{T
\rightarrow R})/2$.

As the Prompter is a simple transformer encoder, it allows flexible types of prompts besides bounding boxes we have used, such as points, free-form referring, text, and etc. We further consider the case where the prompt is free-form text, and leave others for future study. We add a grounding loss that extracts a region feature from the image (*e.g.*, a picture of the bedroom) given its regional caption (*e.g.*, “a large TV”), and predicts the bounding box with an MLP regression head, *i.e.*,

$$ ${\mathcal{L}}_{\text{grounding}}:=\frac{1}{4MN}\sum_{i=1}^{N}\sum_{{\bm{l}}_{i }^{(m)}\sim\{{\bm{l}}_{i}\}}\|{\bm{l}}_{i}^{(m)}-\texttt{BoxHead}\big{(}{\bm{z }}({\bm{y}}_{i}^{(m)})\big{)}\|_{2},$ (3) $$

where ${\bm{z}}({\bm{y}}):=\texttt{Prompter}(f_{T}({\bm{y}}),f_{I}({\bm{x}}))$ is the grounded embedding conditioned on the text (encoded by the CLIP text encoder). The overall loss is

$$ ${\mathcal{L}}:={\mathcal{L}}_{\text{CLIP}}+\lambda({\mathcal{L}}_{\text{{CLOC} }}+{\mathcal{L}}_{\text{grounding}}),$ (4) $$

where $\lambda$ is a weighting scalar. In experiments, we set $\lambda$ to be the ratio of images in the mini-batch that contain region labels without extra tuning. All the learnable parameters are trained end-to-end.

### 3.3 CLOC Model Architecture

We implement the promptable embedding introduced in Section 3.2 with minimal extra modules on top of the original CLIP image and text encoders. As illustrated in Figure 2, the original CLIP model remains the same for computing ${\mathcal{L}}_{\text{CLIP}}$. For computing ${\mathcal{L}}_{\text{{CLOC}}}$/${\mathcal{L}}_{\text{grounding}}$, the image embedding is re-used from the CLIP ViT but before the pooled projection and normalization $f_{I}^{\prime}$. To extract the region embedding ${\bm{z}}=\texttt{Prompter}({\bm{l}},f_{I}^{\prime}({\bm{x}}))$ from the image, we consider the location representation ${\bm{l}}$ as two coordinates (top-left and bottom-right corners of a box), each vectorized by positional encoding. The Prompter is a simple and lightweight one-layer transformer encoder. It takes the positional encodings prepended with the sequence of image tokens from ViT together as the input, and outputs the region embedding with a pooled projection layer. For the grounding loss, we re-use the same CLIP text encoder for encoding the region captions ${\bm{z}}=\texttt{Prompter}(f_{T}({\bm{y}}),f_{I}^{\prime}({\bm{x}}))$ to predict the bounding boxes with a two-layer MLP head. Overall, CLOC only adds additional learnable parameters of the lightweight Prompter. Note, that the main overheads in a single forward are from encoding the image via the ViT – CLOC reuses it for multiple prompts.

### 3.4 Discussions on Design Choices and Extensions

We provide discussions here on the rationale behind our design choices and some minor extensions.

#### Extracting region embedding with visual prompts.

To train our model with ${\mathcal{L}}_{\textsc{CLOC}}$ in Equation 4, it requires extracting region embeddings from the image features given the bounding boxes. A perhaps straightforward alternative could be Region-of-Interest (RoI) pooling/alignment (He et al., ) from the spatial image feature of ViT before pooling. RoI operations are popular, especially in the object detection literature.(^1^11We observe a withdrawn arXiv preprint (https://arxiv.org/abs/2401.06397) proposes to extract RoI features for CLIP regional contrastive learning but only focus on dense vision tasks, not MLLMs.) However, as will be evidenced by worse performance in Section 5, we found it suboptimal for CLOC pre-training here for several reasons. First, unlike object detection datasets that typically contain golden labels, here the pseudo-labels are much noisier on the large-scale web-crawled images. Therefore, the resulting RoI features may be inaccurate due to the imprecise bounding boxes, making model training less effective. Second, unlike dense vision tasks that directly rely on the spatial features, MLLM has a transformer decoder that consists of several attention layers such that the constraint of semantics in the spatial feature space becomes somewhat indirect. Our Prompter mimics such inductive bias in pre-training via a single-attention-layer encoder that may leverage better global context reasoning compared to RoIs.

#### Avoiding region-text conflicts.

While region annotations introduce location information, a concern of contrastive learning on the regional captions may be that there are many similar objects within an image (*e.g.*, “boats” in the harbor) or a mini-batch. To mitigate such concerns, we apply two tricks. First, fortunately, we found it sufficient to sample a few regions per image for each update, *e.g.*, we set $M=4$ in Equation 2 in experiments. Second, we can filter similar texts when computing the negatives in the contrastive loss. More specifically, we ignore the pairs of $\big{(}{\bm{z}}_{i}^{(m)},f_{T}({\bm{y}}_{j}^{(m^{\prime})})\big{)}$ in the denominators of both ${\mathcal{L}}_{R\rightarrow T/{T\rightarrow R}}$, if $\texttt{sim}\big{(}f_{T}({\bm{y}}_{i}^{(m)}),f_{T}({\bm{y}}_{j}^{(m^{\prime})}
)\big{)}>0.9$, without gradients on $f_{T}$.

## 4 Visually-Enriched and Spatially-Localized Captioning Pipeline

Figure: Figure 3: Overview of the Visually-Enriched and Spatially-Localized (VESL) captioning pipeline. We leverage an existing open-vocabulary detector (*e.g.*, OWLv2) that typically predicts bounding boxes on the images, and assigns the labels from the given text phrase candidates. Previous methods do not tailor how the text phrases are prepared and often use the alt-text attached to the images, which is prone to insufficient region descriptions. We found it crucial for CLOC to train on data from our VESL that re-captions images with the visually-enriched captioner VeCap (Lai et al., ) for better visual concept exploitation of the detector.
Refer to caption: extracted/5897759/figures/cloc_vesl.png

**Table 1: Region-text dataset statistics. We summarize the text token length for both images and regions. Partial statistics of the proprietary datasets revealed by their papers. ^∗The $20$M subset of GRIT is released at: [https://huggingface.co/datasets/zzliang/GRIT](https://huggingface.co/datasets/zzliang/GRIT); we removed the invalid images.**
| Dataset | # of images | regions per image | image caption length | region text length |
| --- | --- | --- | --- | --- |
| Flickr Entities (Plummer et al., ) | 32K | 8.7 | – | – |
| RefCOCO (Yu et al., ) | 20K | 2.5 | – | 3.6 |
| RefCOCO+ (Yu et al., ) | 20K | 2.5 | – | 3.5 |
| RefCOCOg (Mao et al., ) | 27K | 2.1 | – | 8.4 |
| Visual Genome (Krishna et al., ) | 108K | 38.0 | – | – |
| GRIT (proprietary) (Peng et al., ) | 91M | 1.5 | – | 4.7 |
| GRIT (released, clean) (Peng et al., )^∗ | 17M | 1.8 | 17.2 | 4.6 |
| Florence-2 (proprietary) (Xiao et al., ) | 126M | 5.4 | 70.5 | 2.6 |
| OWLv2 (proprietary) (Minderer et al., ) | 2B | – | – | – |
| WiT labeled w/ Minderer et al. () | 300M | 5.1 | 17.1 | 3.9 |
| VESL WiT (Ours) | 300M | 11.6 | 44.9 | 2.1 |
| VESL WiT+DFN (Ours) | 2B | 11.5 | 35.9 | 2.1 |

As discussed in Section 1 and 3.1, a key bottleneck of CLOC training is the region-text annotation datasets in terms of both the data size scales and the label quality, since there are no public datasets with region-text annotations at scales large enough for contrastive pre-training.

Inspired by recent works that re-caption images with visually-enriched captions for better CLIP training, we make a step further for Visually-Enriched and Spatially-Localized (VESL) labeler which generates more fine-grained captions at the region level. The goal of VESL is, given an image (possibly with the original web-crawled alt-text), annotate it with the grounded bounding boxes each associated with a caption in natural language for optimizing Equation 2 in Section 3.2.

Concretely, VESL is constructed as a pseudo-labeling pipeline with the following steps:

- 1.
Image re-captioning with visual concept exploitation: We follow the VeCap framework (Lai et al., ) to generate long, diverse, and detailed image captions.
- 2.
Region phrase candidates extraction: Inspired by Zhang et al. (), we apply name entity recognition (NER) to extract leaf entities from the visually-enriched captions as potential candidate phrases describing a region inside the image.
- 3.
Open-vocabulary detection with extracted phrases: we generate the final region-text annotations via a pre-trained open-vocabulary detector queried with the phrases extracted from Step 2 to match the bounding boxes proposed by the detector. We adopted the OWLv2 detector (Minderer et al., ) which contains the CLIP image/text encoder with the detection head. The boxes with detection confidence larger than $0.1$ are kept as the region location and the phrases matched with the highest score are considered as their captions.

#### Remarks.

We highlight our insights behind the proposed recipe. The most relevant work was proposed in (Minderer et al., ) that scales up open-vocabulary (OV) detection via self-training. We are inspired by its success and extend it to CLOC contrastive learning with important modifications. Different from (Minderer et al., ) that generates candidate phrases from the $n$-grams of the web-crawled alt-text of the images for OV detection, we found the alt-text might not have enough details describing the image region content, thus limiting the diversity and quality of the annotations predicted by the OV detector. We thus caption each image augmented with more visual details. However, the long captions make the $n$-grams candidates verbose and grow exponentially, thus we generate high-quality candidates via name entity recognition instead. We found such a pipeline produces training data more suitable for CLOC, as will be validated in Section 5.

#### Our pre-training datasets.

Our pre-training data consists of two parts: ($i$) image-text pairs, and ($ii$) region-text pairs. For image-text pairs, we reproduce the image re-captioning pipeline from VeCap (Lai et al., ), and generate synthetic captions for WiT-300M (Wu et al., ) and DFN-5B (Fang et al., ) images.
For region-text pairs, we pseudo-label WiT-300M and a 2B-image subset of DFN-5B using our VESL pipeline. In VESL, we adopted the official OWLv2 L/14 model (Minderer et al., ) as the open-vocabulary detector.(^2^22OWLv2 CLIP L/14 ST+FT in: https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit) All images are pseudo-labeled with $448\times 448$ resolution, where a maximum number of $20$ phrase queries are sampled for moderate computation budget. Table 1 summarizes the statistics of existing region-text datasets and ours. Notably, we also ablate annotating WiT-300M following Minderer et al. () and found it detects *less* objects with longer region text, likely due to verbose $n$-grams of alt-text are in lower quality than our approach, as discussed in the remarks. Examples and pseudo codes are in Appendix B.

## 5 Experiments

### 5.1 Setup

#### Pre-training.

We follow OpenAI-CLIP (Radford et al., ) to train both our CLIP baseline model and CLOC model using a similar budget of around 14B images seen(^3^33To avoid confusion, we will refer to CLIP as training with the standard CLIP loss on the same data as CLOC and refer to OpenAI-CLIP as the public pre-trained checkpoint released from Radford et al. ().). For a fair comparison, we use the same hyper-parameters and images for both the CLIP baseline and CLOC. We experimented with the ViT B/16 and L/14 architectures, pre-trained with $224\times 224$ and $336\times 336$ image resolutions, respectively. All parameters are trained end-to-end from scratch. We implement the codebase based on open-source AXLearn framework(^4^44[https://github.com/apple/axlearn](https://github.com/apple/axlearn)) in JAX (Bradbury et al., ). We provide hyper-parameters and more details in Appendix A.

#### Evaluation tasks.

We evaluate our image encoders across a wide range of downstream tasks. First, we assess performance on ImageNet image classification (Deng et al., ; Shankar et al., ) and COCO retrieval (Lin et al., ). Second, we construct region-level tasks, including COCO object recognition and region-text retrieval using the GRIT dataset (Peng et al., ). Furthermore, we show CLOC is particularly useful for MLLMs, validated by the Ferret model (You et al., ) which requires fine-grained image understanding for referring and grounding tasks. We also evaluate on general multimodal benchmarks using LLaVA-1.5 (Liu et al., ) and LLaVA-NeXT (Liu et al., )(^5^55We use the codebase: [https://github.com/xiaoachen98/Open-LLaVA-NeXT](https://github.com/xiaoachen98/Open-LLaVA-NeXT).), which both use the 7B Vicuna LLM. For all evaluation tasks, we use the same official hyper-parameters, fine-tuning datasets, and codebase for all the image encoders we experimented with, without specific tuning. More details are provided in each subsection and in Appendix A.

### 5.2 Image and Region Classification and Retrieval Tasks

The proposed CLOC training framework enables the encoder to produce not only image embedding but also region embeddings. It can directly be used for region-level tasks without further training, in analogy to the zero-shot capability of CLIP on images. To evaluate such capability and also for fast development and ablation study, we first construct several region-level zero-shot tasks.

Besides *image*-level evaluation like ImageNet classification and COCO image-text retrieval, we additionally construct *region*-level tasks, including region object recognition and region-text retrieval. More specifically, the region-level tasks leverage the labeled bounding boxes in the evaluation set for CLOC to extract region embedding. For region retrieval, we use a validation set of the GRIT dataset (Peng et al., ) and encode both the image regions and the region captions. For region classification, the class names are encoded as text embedding ($80$ / $1203$ classes for COCO / LVIS, respectively), and the highest cosine similarity for each region embedding is predicted as its class.

**Table 2: Zero-shot evaluation on image-level tasks (accuracy of ImageNet (IN) classification, recall@1 of COCO retrieval) and region-level tasks (mAcc of region object recognition on COCO and LVIS, recall@10 of GRIT region retrieval), using ViT-B/16 as the default encoder backbone. The indentation with different symbols denotes removing (–) or changing a component ($\circ$).**
|  | Models | Training data | Image tasks | Region tasks | Avg. |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | Image | Region | COCO (i2t) | COCO (t2i) | INv1 | INv2 | GRIT (r2t) | GRIT (t2r) | COCO Recog. | LVIS Recog. | Image | Region |
| 1 | OpenAI-CLIP | proprietary | - | 52.4 | 33.1 | 68.3 | 62.3 | - | - | - | - | 54.0 | - |
| 2 | CLIP | WiT+DFN | - | 66.3 | 45.1 | 76.2 | 69.6 | - | - | - | - | 64.3 | - |
| 3 | CLOC | WiT | WiT | 68.8 | 50.1 | 66.7 | 59.7 | 65.1 | 67.2 | 70.6 | 26.7 | 61.3 | 57.4 |
| 4 | – Prompter | WiT | WiT | 67.0 | 49.7 | 65.6 | 58.6 | 44.8 | 4.4 | 55.3 | 13.2 | 60.2 | 29.4 |
| 5 | – VESL | WiT | WiT | 53.9 | 36.3 | 66.6 | 59.5 | 71.5 | 63.8 | 62.2 | 22.2 | 54.1 | 54.9 |
| 6 | $\circ$ w/ GLIPv2 | WiT | WiT | 68.8 | 50.0 | 65.8 | 59.2 | 67.9 | 71.1 | 64.9 | 23.1 | 61.0 | 56.8 |
| 8 | CLOC | WiT+DFN | WiT | 66.1 | 46.5 | 75.5 | 68.6 | 65.8 | 67.4 | 70.1 | 27.2 | 64.2 | 57.6 |
| 9 | – Prompter | WiT+DFN | WiT | 65.8 | 46.5 | 75.7 | 68.0 | 55.5 | 18.4 | 67.1 | 24.6 | 64.0 | 41.4 |
| 10 | – text filtering | WiT+DFN | WiT | 65.4 | 46.0 | 75.7 | 68.4 | 66.3 | 66.5 | 68.7 | 24.8 | 63.9 | 56.6 |
| 11 | – ${\mathcal{L}}_{\text{grounding}}$ | WiT+DFN | WiT | 66.0 | 46.3 | 75.7 | 67.9 | 66.0 | 66.8 | 70.0 | 25.8 | 64.0 | 57.2 |
| 12 | $\circ$ $M=2$ | WiT+DFN | WiT | 66.6 | 46.2 | 75.5 | 67.9 | 66.5 | 67.0 | 69.8 | 25.8 | 64.1 | 57.3 |
| 13 | CLOC | WiT+DFN | WiT+DFN | 69.2 | 49.3 | 74.9 | 67.0 | 63.9 | 65.9 | 71.1 | 28.5 | 65.1 | 57.3 |
| 14 | – Prompter | WiT+DFN | WiT+DFN | 70.2 | 49.7 | 74.7 | 67.6 | 65.7 | 23.0 | 67.1 | 25.4 | 65.6 | 45.3 |
| 15 | – VESL | WiT+DFN | WiT+DFN | 65.3 | 46.6 | 75.5 | 67.7 | 55.7 | 22.3 | 66.3 | 25.3 | 63.8 | 42.4 |
| 16 | $\circ$ ViT L/14 | WiT+DFN | WiT+DFN | 74.8 | 54.4 | 80.1 | 73.2 | 66.9 | 68.3 | 72.9 | 32.6 | 70.6 | 60.2 |
| 17 | $\circ$ ViT H/14 | WiT+DFN | WiT+DFN | 75.7 | 55.1 | 81.3 | 74.7 | 67.4 | 69.4 | 73.0 | 35.6 | 71.7 | 61.3 |

We highlight important variables for the performance in Table 2 with the following observations:

- •
CLOC performs decently on region-level tasks(^6^66For reference, in a different data setup, Wu et al. () reports 46.5% mAcc on the same COCO region classification task, trained using $320\times 320$ COCO training images. In contrast, our approach achieves over 70% mAcc, pre-trained on a $224\times 224$ large-scale web-crawled dataset with object annotations (thus might not be fair comparisons).) while maintaining strong performance on image-level metrics ( 2 vs.  8 13).
- •
The Prompter is an important ingredient for CLOC’s success to go beyond CLIP without a compromise ( 3 vs.  4; 8 vs.  9; 13 vs.  14). We replace the Prompter with RoI alignment to extract region features and train with ${\mathcal{L}}_{\text{{CLOC}}}$. We found it performs much worse on region-level tasks than CLOC, possibly due to such strong constraints of RoI features being difficult to learn on the noisy labels with the CLIP loss as discussed in Section 3.4.
- •
VESL outperforms Minderer et al. () baseline approach, as the visually-enriched captions improve image retrieval tasks (as expected (Lai et al., )), while also offering versatile visual concepts as text candidates for the OV detector, supporting Section 4 ( 3 vs.  5; 13 vs.  15).
- •
Given the same captions in VESL, OWLv2 slightly outperforms the GLIPv2 detector ( 3 vs.  6).
- •
Tricks in Section 3.4 offer slight performance gains, but ${\mathcal{L}}_{\text{{CLOC}}}$ is already highly effective on its own ( 10 11).
- •
Region tasks work well when sampling $2$ or $4$ boxes per image, making CLOC practical ( 12).
- •
Scaling up region labels seems saturated at 300M images on region tasks ( 3 8 13), while we found it will further improve in MLLM tasks as will be shown in Table 3.
- •
Scaling up the ViT model sizes can further improve both image and region tasks ( 13 16 17).

Overall, CLOC not only achieves strong performance on image-level tasks, but unlocks a new capability for zero-shot region-level tasks. We have validated our design choices for architectures, training, and data. Below, the complete setup 13 will be used as default if not specified.

**Table 3: Results on Ferret-Bench for referring and grounding VQA, based on Ferret (You et al., ) equipped with different image encoders. Models are evaluated with OpenAI gpt-4o API instead of the deprecated gpt-4-0314 in the paper. ^∗replace Ferret visual sampler with Prompter; see Section 5.3 for details.**
| Method | ViT | Region Alignment | # of images w/ region labels | Referring Description | Referring Reasoning | Grounding in Conversation | Avg.<br>( $\Delta$ to CLIP) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| CLIP | B/16 | None | None | 47.5 | 50.3 | 45.3 | 47.7 |
| CLOC | B/16 | RoI-Align | 300M | 48.0 | 48.4 | 40.0 | 45.5 |
| CLOC | B/16 | Prompter | 300M | 50.2 | 55.5 | 41.5 | 49.1 |
| CLOC | B/16 | Prompter | 2B | 53.6 | 53.7 | 42.2 | 49.8 (+2.1) |
| CLOC ^∗ | B/16 | Prompter | 2B | 54.8 | 54.9 | 44.7 | 51.5 (+3.7) |
| OpenAI-CLIP | L/14 | None | None | 50.8 | 55.4 | 45.7 | 50.6 |
| CLIP | L/14 | None | None | 54.2 | 54.6 | 43.3 | 50.7 |
| CLOC | L/14 | Prompter | 300M | 51.0 | 65.7 | 44.9 | 53.9 |
| CLOC | L/14 | Prompter | 2B | 55.9 | 63.3 | 46.0 | 55.1 (+4.4) |
| CLOC ^∗ | L/14 | Prompter | 2B | 56.3 | 67.4 | 47.1 | 56.9 (+6.2) |

### 5.3 Referring and Grounding with Ferret

As discussed in Section 1, a key motivation is to provide an enhanced image encoder for training MLLMs, particularly for tasks requiring fine-grained image understanding. A notable example is Ferret (You et al., ), a recently proposed MLLM that builds on LLaVA and aims to handle more advanced spatial interactions, such as referring and grounding in VQA tasks. Ferret can take region prompts such as a box, a point, or a free-form location referring to the input image as input, and answer a question specific to the region such as “Do you know when the object[region] was invented?” Ferret thus requires fine-grained image features from the vision encoder for spatial reasoning.

We evaluate CLOC by replacing the CLIP ViT encoder with our CLOC ViT as a drop-in replacement. We follow the official codebase(^7^77We use the official Ferret codebase: [https://github.com/apple/ml-ferret](https://github.com/apple/ml-ferret).) for training the Ferret model. We further consider a variant based on Ferret: the Ferret model implements a spatial-aware visual sampler that samples image features from the region specified in the question. We replace the sophisticated visual sampler with our simple Prompter introduced in Section 3.3 to extract region embedding with ${\bm{z}}=\texttt{Prompter}({\bm{l}},f_{I}^{\prime}({\bm{x}}))$ instead, as illustrated in Figure 1(right).

In Table 3, we evaluate different pre-trained image encoders on the Ferret-Bench benchmark (You et al., ). Ferret-Bench includes challenging multimodal dialogue-style VQA of three tasks constructed with GPT-4. Results show that our Prompter is essential to improve upon the CLIP baseline – RoI-Align may even slightly degrade performance. Scaling region labels from $300$M to $2$B further improves performance. Interestingly, our Prompter (denoted as ^∗) can be a replacement of the Ferret visual sampler in fine-tuning, which is simpler and performs even better up to $6\%$ against both the OpenAI-CLIP and our in-house CLIP. We also evaluate CLOC (2B labeled) on other referring and grounding tasks ranging from referring object classification, referring expression comprehension, and phrase grounding across multiple datasets. As summarized in Table 4, CLOC is also superior evidenced by $1\sim 3\%$ improvements in average of $13$ evaluation sets.

**Table 4: Results on referred object classification (LVIS), referring expression comprehension ($0.5$ IoU on RefCOCO, RefCOCO+, RefCOCOg), and phrase grounding ($0.5$ IoU on Flickr30k Entities) with Ferret. Shikra: baseline in Chen et al. (). Ferret^∗: replace visual sampler with CLOC prompter.**
| Model | Encoder | LVIS | RefCOCO | RefCOCO+ | RefCOCOg | Flickr | Avg. |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | box | point | free-form | val | testA | testB | val | testA | testB | val | test | val | test | ( $\Delta$ to CLIP) |
| Ferret | CLIP B/16 | 72.5 | 56.9 | 57.2 | 80.7 | 84.2 | 77.1 | 71.9 | 76.1 | 63.7 | 75.9 | 76.2 | 76.2 | 78.3 | 72.8 |
| Ferret | CLOC B/16 | 74.3 | 56.7 | 60.2 | 84.2 | 87.0 | 80.0 | 74.7 | 80.0 | 67.0 | 78.8 | 79.5 | 80.0 | 81.5 | 75.7 (+2.9) |
| Ferret ^∗ | CLOC B/16 | 78.9 | 58.2 | 61.4 | 84.4 | 86.8 | 78.9 | 74.0 | 78.7 | 65.5 | 78.0 | 78.7 | 80.1 | 81.4 | 75.8 (+3.0) |
| Shikra | OpenAI-CLIP L/14 | 57.8 | 67.7 | n/a | 87.0 | 90.6 | 80.2 | 81.6 | 87.4 | 72.1 | 82.3 | 82.2 | 75.8 | 76.5 | - |
| Ferret | OpenAI-CLIP L/14 | 79.4 | 67.9 | 69.8 | 87.5 | 91.4 | 82.5 | 80.8 | 87.4 | 73.1 | 83.9 | 84.8 | 80.4 | 82.2 | 80.8 |
| Ferret | CLIP L/14 | 78.7 | 66.9 | 70.2 | 88.0 | 90.4 | 83.5 | 80.1 | 85.8 | 73.3 | 82.8 | 83.4 | 79.0 | 80.1 | 80.2 |
| Ferret | CLOC L/14 | 81.6 | 67.9 | 69.9 | 89.0 | 91.0 | 84.7 | 81.4 | 86.8 | 74.7 | 84.0 | 85.2 | 82.3 | 83.3 | 81.7 (+1.5) |
| Ferret ^∗ | CLOC L/14 | 79.8 | 67.9 | 69.1 | 88.2 | 91.1 | 84.5 | 80.6 | 86.7 | 73.9 | 84.8 | 85.1 | 82.4 | 83.5 | 81.4 (+1.2) |

**Table 5: Results on multimodal benchmarks using LLaVA-1.5/NeXT with ViT-L/14 and Vicuna-7B.**
| Method | MLLM | LLaVA${}^{\text{W}}$ | TextVQA | GQA | MM-Vet | POPE | MME-P | MME-C |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CLIP | LLaVA-1.5 | 59.3 | 53.3 | 62.2 | 30.0 | 86.7 | 1451.4 | 254.3 |
| CLOC | LLaVA-1.5 | 64.3 | 54.9 | 62.7 | 31.5 | 87.3 | 1482.0 | 288.9 |
| CLIP | Open-LLaVA-NeXT | 67.3 | 61.4 | 63.5 | 38.5 | 87.9 | 1486.1 | 279.6 |
| CLOC | Open-LLaVA-NeXT | 69.5 | 61.9 | 64.2 | 40.2 | 88.3 | 1451.1 | 312.5 |

### 5.4 General VQA with LLaVA-1.5 and LLaVA-NeXT

We further show that the CLOC encoder is also competitive against CLIP on general VQA tasks without regression and can even provide performance improvements. We use the Vicuna 7B LLM decoder for two experiments based on LLaVA-1.5 (frozen encoder) and Open-LLaVA-NeXT (unfrozen encoder with AnyRes (Liu et al., ) inputs). Since general VQA does not provide spatial referring inputs, we simply replace the ViT in LLaVA. Table 5 summarizes the results. Encouragingly, with our CLOC designs, the improved region-level alignment is also beneficial to some general multimodal benchmarks, as they may also require fine-grained image understanding.

## 6 Conclusion

Please see Appendix C for more discussions where we comment on the limitations, future directions, computation cost, design rationales, etc.

We tackle a deficiency of CLIP, to make the semantics aligned in the vision space not only at the image level but also at the region level. We propose a new pre-training framework that includes innovations in a new learning formulation that a strong encoder should be easily transformed in the foresee of downstream use of MLLMs. Our encoder creates a new possibility for adapting the features with input prompts of interaction together with MLLMs. To resolve the need for large-scale region-text training data, we carefully design a pseudo-labeling pipeline for visually-enriched and spatially-localized captions. Our pre-trained encoder is essentially a drop-in replacement of CLIP, with competitive image-text performance, and extra capability demonstrated in region-text tasks and VQA tasks with MLLMs.

## Acknowledgement

We thank Yanghao Li, Felix Bai, and many others for their invaluable help and feedback.

## Reproducibility Statement

We made our best efforts to exhaustively state the implementation details. Training hyper-parameters and model architectures are discussed in Section 3.2, 3.3, and 5.1, with a summary in Appendix A and Table A. For evaluation, as mentioned in Section 5.1, we strictly follow the official setup with the codebase released by the original authors if applicable, with details provided in Section A.2. For our datasets, we provide data processing details in Section 4 and example codes in Appendix B. We are working hard on releasing the annotations with internal approvals.

## Appendix

## Appendix A Experiment Details

Below, we provide more details on pre-training hyper-parameters and downstream evaluation tasks.

### A.1 Pre-training Hyper-parameters

For pre-training both the in-house CLIP baseline and CLOC, we mainly follow the hyper-parameters in Radford et al. () with our in-house pre-training datasets. The images used for model training are identical for CLIP and CLOC, while CLOC is trained on the extra region-text annotations of the same images via the proposed VESL pipeline (details in Section B). Table A summarizes the training hyper-parameters used for all experiments and the setup for components specific to CLOC.

In terms of the CLOC architecture, as illustrated in Figure 2, the image and text encoders including the attention pooling and projection layers follow the same as OpenAI-CLIP (Radford et al., ). Our Prompter consists of a positional encoding matrix for bounding boxes, and a single-layer single-head transformer encoder with another set of the global average pooler and a projection layer to map the region embeddings into the same dimension as the CLIP text/image embeddings.

**Table A: Pre-training hyper-parameters and settings for the in-house CLIP baseline and CLOC.**
| General |  |
| --- | --- |
| Batch size | $32768$ |
| Image size | $224\times 224$ (ViT B/16) or $336\times 336$ (ViT L/14, H/14) |
| Image pre-processing | long-side resizing with padding (*i.e.*, tf.image.resize_with_pad) |
| Text tokenizer | T5 (Raffel et al., ), lowercase |
| Text maximum length | $77$ tokens |
| Steps | $439087$ (*i.e.*, $\sim 14$B examples seen) |
| Optimizer | AdamW ($\beta_{1}=0.9,\beta_{2}=0.98$) |
| Peak learning rate (LR) | $0.0005$ |
| LR schedule | cosine decays with linear warm-up (first $2$k steps) |
| Weight decay | $0.2$ |
| Dropout rate | $0.0$ |
| CLOC |  |
| # of sampled regions | maximum $M=4$ per image |
| CLOC loss weight | $\lambda=1.0\times\frac{\#\text{of images contain region text in the mini-batch<br>}}{\text{batch size}}$ (in Equation 4) |
| Encoding box prompts | sinusoidal positional encoding of coordinates (top-left and bottom right of a box) |
| Encoding text prompts | encoded by re-using the text encoder (w/ pooling & projection) |
| Prompter architecture | a single-layer single-head transformer encoder (same feature dimension as the ViT) |
| BoxHead architecture | 2-layer MLP with GELUs activations (Hendrycks & Gimpel, ) |

### A.2 Evaluation Tasks

We provide more details on the tasks constructed for evaluating the encoders in Section 5.

#### Zero-shot region tasks.

Our CLOC training augments a new capability for CLIP to generate region-level embeddings. This enables us to perform zero-shot region-text tasks, in analogy to the image-text zero-shot tasks like ImageNet classification and COCO image-text retrieval that CLIP has been evaluated on.

In a similar rationale of image-level evaluation, we further construct region-level tasks including region object recognition and region-text retrieval. For region object recognition, the class names are encoded by the text encoder into class embedding. We do not add the text prompts (*e.g.*, “a photo of …”) to object classes used when CLIP (Radford et al., ) evaluated on image classification. The CLOC model takes all the labeled bounding boxes in the images to generate a region embedding ${\bm{z}}=\texttt{Prompter}({\bm{l}},f_{I}({\bm{x}}))$. The class embedding with the highest similarity is predicted as the class of the region (*i.e.*, out of $80$ / $1203$ classes for COCO / LVIS).

For region retrieval, similarly, the CLOC model encodes both the image regions and the region captions from the public region-text GRIT dataset that the regions are annotated by the Kosmos-2 pipeline (Peng et al., ). We randomly sampled a $2$K image validation set for fast evaluation. We have verified it is statistically stable compared to the whole set that contains about $20$M in total. Unlike image-text retrieval where the image captions are likely unique, the objects in regions of many images might be duplicated. Therefore, we opt to report recall@10 rather than recall@1 for GRIT region retrieval in Table 2.

#### MLLM tasks.

To demonstrate our CLOC can benefit MLLM downstream tasks as a better image backbone, we consider two sets of MLLM experiments.

First, we experiment with Ferret (You et al., ) that is capable of taking spatial referring inputs for grounding and referring VQA tasks. Ferret can consume a point, a bounding box, or a free-form referring. It designs a complicated visual sampler module that involves point sampling and $k$NN grouping. We suggest the readers refer to Figure 3 and Section 3.2 in You et al. () for more details. Here, we consider two methods to use CLOC for Ferret model training: ($i$) we only take the ViT encoder in CLOC to replace the CLIP ViT and still use the original Ferret visual sampler; ($ii$) we further replace the visual sampler with our simple Prompter (essentially a lightweight transformer encoder with box positional encodings) in Section 3.3 as illustrated in Figure 1(3b). More specifically, we simply convert all types of spatial referring as boxes. As evidenced by Table 3 and Table 4, our Prompter can indeed be a much simpler alternative and may perform even better as it is more consistent with CLOC pre-training.

Second, we evaluate on general VQA tasks that do not consider extra spatial referring inputs. The pre-trained ViT of CLOC is a drop-in-replacement of CLIP ViT in two sets of experiments of LLaVA-1.5 (Liu et al., ) and LLaVA-NeXT (Liu et al., ). The main difference includes different supervised fine-tuning (SFT) datasets. Also, LLaVA-NeXT uses the AnyRes technique that decomposes an image into several subimages that are encoded independently with the ViT and concatenated together as the input for the decoder. LLaVA-1.5 by default freezes the ViT, while LLaVA-NeXT fine-tunes all parameters during SFT. Since the official LLaVA-NeXT is trained on some proprietary datasets that are not reproducible, we use the Open-LLaVA-NeXT repository.(^8^88[https://github.com/xiaoachen98/Open-LLaVA-NeXT](https://github.com/xiaoachen98/Open-LLaVA-NeXT)) Our experiments in Table 5 demonstrate CLOC not only improves general VQA tasks besides referring and grounding tasks, but also generalizes well for both LLaVA-1.5 and LLaVA-NeXT settings.

## Appendix B VESL Data Engine

We provide more details about our pseudo-labeling data pipeline proposed in Section 4.

### B.1 Implementation Details

As already mentioned in Section 4, there are three steps for VESL: image re-captioning, region phrase candidates extraction from the captions, and open-vocabulary (OV) detection given the region candidates as queries.

For re-captioning, the goal is to replace alt-text with long, diverse, and detailed captions that can be used to generate more visual concepts as the region candidate phrases for the OV detector. Technically, any strong image captioner can be an option. In our paper, we adopt the VeCap pipeline (Lai et al., ) and leverage their images with enriched captions.

To extract region phrase candidates from the long captions, we adopt name entity recognition (NER) to extract leaf entities from the captions, inspired by Zhang et al. (). The code listing below shows the Python example implementation, where stop words and common generic words are filtered, following Minderer et al. ().

Generating bounding boxes and assigning region captions can be done by querying an OV objection detector. We adopted the OWLv2 detector (Minderer et al., ) with their pre-trained L/14 checkpoint to annotate inputs with $448\times 448$ image resolutions.

Figure: Listing 1: Python example code for Step 2 of VESL in Section 4 for extracting text candidate queries from a caption.

### B.2 More Visualizations

As mentioned in the remarks of Section 4, we found the alt-text sourced from the original web-crawled images might not contain enough details describing the regional content, thus limiting the diversity and quality of the text candidate queries for the OV detector to detect more meaningful objects. In Figure A, we show some cherry-picked examples (since the web-crawled images are quite noisy) to demonstrate the reasons why high-quality captions can help our region-text annotation pipeline. In Minderer et al. (), the queries are generated by the $n$-grams of the alt-text, while ours are by NER as described in Section B.1 on top of the visually-enriched captions. Note that, in both methods we use the same OV detector but with different approaches to generate the queries.

As shown in Figure A, for simple images like in the first row, both methods are doing reasonably well to detect “message card”. However, when the scene becomes complicated (*e.g.*, the second row), our methods can detect more objects since more visual concepts can be extracted from our rich caption as queries for the detector.
Similarly, it can be seen that our method captures more items that the alt-text missed, *e.g.*, “banana”, “eggs”, “butter”, etc. in the third row; “drawstring” in the fourth row; “apples” and “vases” in the last row. Also, it is more likely to extract a more detailed description of the region rather than a class name, such as “green-roofed cottage nestles” in Figure 3 and “decorative metal tree sculptures” for the image in the last row of Figure A. We believe such high-quality region labels essentially contribute to better supervision for CLOC pre-training.

Figure: Figure A: Examples comparing our VESL and the labeling approach in Minderer et al. () that directly uses the $n$-grams of the crawled alt-text. For VESL, each image is annotated with the visual-enriched caption to replace the alt-text, which is used to generate region text candidates that capture the image content better.
Refer to caption: extracted/5897759/figures/cloc_vis.png

## Appendix C More Discussions

#### Limitations.

One limitation for CLOC is the labeling efforts in preparing the training data. As we discussed in Section 1, there are no public large-scale region-text datasets since it is expensive to infer such labels up to the scales we consider here. Unlike previous work (Zhong et al., ) that cropping boxes from images for annotating, our VESL pipeline conducts inference in *image-level*, thus the cost does not scale with the number of detected regions. With that being said, such inference still requires hundreds of GPUs running in parallel for days to scale up to billions of images. We are working on releasing the annotations to accelerate future research for the community.

For CLOC, we focus on the training objective and framework formulation, while making minimal efforts on hyper-parameter tuning, architecture search, dataset cleaning, and etc., thus better performance could be achieved. Besides, although we have included extensive standard evaluation tasks, the fine-grained region knowledge could also be useful on more under-explored tasks.

#### Future directions.

We suggest promising future directions. In Section 3.2, our Prompter formulation can take flexible prompts to guide the embeddings for specific tasks. In this work, we consider a prompt as a single bounding box or a text caption, but it has the potential to expand to various types such as points, a mask, users’ free-form referring, or multiple prompts in multiple types together. We believe a more versatile Prompter with co-designs for different objectives can have a big potential. Similarly, our VESL labeling pipeline limits to the detection box format. Annotators supported for more formats may further boost it. We believe our approach is promising, as more attention has been drawn recently for better image re-captioning (Li et al., ; Fan et al., ) that VESL relies on. In addition, CLOC provides a new capability to extract region features without further training, thus can be used as a foundation model for exploring new VL applications.

#### Training cost.

We comment on the computation cost of our framework.
Our large models (ViT L/14) were trained on $1024$ v5p TPUs for about $6$ days. To optimize Equation 2, CLOC needs extra computation. The main overheads come from the contrastive matrix but not the lightweight Prompter. Fortunately, we found it feasible since ($i$) only a few boxes in each image need to be sampled per update; ($ii$) the loss computation becomes a smaller proportion when the ViT scales up. Overall, we found the computation acceptable compared to CLIP. More memory-efficient optimization like SigLIP (Zhai et al., ) can be implemented with JAX shard_map(^9^99https://jax.readthedocs.io/en/latest/jep/14273-shard-map.html) ops.

#### Discussions on design rationales.

Besides the main discussions we have stressed in the main text, here we provide more thoughts behind our design rationales that a reader may be wondering.

*(1) Why not use other locality-enhanced image encoders?* We would like to note that many image encoders with localization capabilities like DINOv2 (Oquab et al., ), OWLv2 (Minderer et al., ), CLIP-Self (Wu et al., ) etc. are developed specifically for dense vision tasks that cannot perform zero-shot image-level tasks like CLIP and CLOC. Our goal is to develop a drop-in replacement for the CLIP encoder that improves localization while maintaining all of CLIP’s original strengths, such as zero-shot image-level tasks and its critical role as the image backbone for MLLMs. Furthermore, perhaps well-known within the MLLM community, these encoders have been shown in recent reports that they are not comparable enough to compete with CLIP as the vision backbone for MLLM tasks (Tong et al., ) due to CLIP’s superiority in vision-language alignment. We thus believe enhancing CLIP itself is more demanding as this paper focuses on.

*(2) Why not just train a CLIP with object detection?* One may wonder why we do not just train an encoder with joint optimization of the CLIP contrastive loss with some object detection loss instead of the CLOC design of Equation 4.

Although it sounds like a plausible approach, we would like to point out that contrastive pre-training and object detection are fundamentally different in their technical rationales. CLIP pre-training is often conducted on large batches of low-resolution and noisy images, while object detection is trained on small batches of high-resolution images. CLIP is by default trained from scratch and object detection is typically initialized from pre-trained encoders and focuses on the detection head. Furthermore, detection requires heavy computation on box proposals to detect all boxes appearing in an image, while our region-text contrastive design allows us to flexibly sample fewer regions per image as motivated in Equation 3. Overall, their data pipeline and distributed training setup are not on the same scale thus such joint training may not be very reasonable.

With that being said, some previous works do have attempts that are the exceptions but only for some but not all of the mentioned aspects, and mainly for the purpose of detection. For instance, DetCLIP-v2 (Yao et al., ) adds image-text contrastive loss into detection loss to improve open-vocabulary capability for detection. OWLv2 pre-trains the detector with rather small resolutions but still with a batch size of a maximum $256$ since each image will need to predict up to $100$ boxes during training. Both DetCLIP-v2 and OWLv2 are fine-tuned from a pre-trained encoder.

On the contrary, we study pre-training the image encoder from scratch, which may be complementary to the previous efforts. CLOC maximizes the similarity in co-design with CLIP, thus making it much easier to develop within the same codebase.

*(3) Do we really need to train CLOC from scratch? What if we fine-tune from CLIP?* As CLIP pre-training is expensive, one may wonder if it is necessary to train from scratch on the proposed region-text datasets, or if we can initialize from a standard CLIP trained on image-text pairs only and fine-tunes with CLOC for a shorter stage. Our early investigation, even with extensive hyper-parameter tuning, suggests it is likely to be suboptimal compared to training from scratch directly. For instance, we initialize from the CLIP model 2 in Table 2 and fine-tunes it for another extra $100$K steps with the CLOC training loss Equation 4. The model reaches $64.1\%/19.1\%$ mAcc on COCO/LVIS region recognition, which is much worse than $70.1\%/27.2\%$ of the model 8 trained from scratch, even with more overall training steps.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Region Sampling Strategy
    - How are the 4 regions per image selected during training?
    - Is there any consideration for overlap between regions?
    - How does the sampling strategy affect the model's performance?
2. Training Implementation
    - What is the rationale behind setting λ as the ratio of images with region labels?
    - How do you handle the batch composition in terms of images with/without region labels?
    - Have you experimented with different similar text filtering thresholds besides 0.9?
3. Scalability and Efficiency
    - Why does the performance saturate at 300M images for region tasks but continue to improve for MLLM tasks?
    - How does the computational cost of Prompter compare to traditional RoI pooling approaches?
    - What is the memory overhead of storing region embeddings compared to traditional CLIP?
soundness: 3
strengths: 1. Strong practical value:
    - Addresses a real limitation in CLIP that affects its use in MLLMs
    - Provides a drop-in replacement solution without compromising original capabilities
    - Demonstrates consistent improvements across various downstream tasks
2. Technical novelty:
    - Introduces an elegant concept of "promptable embeddings"
    - Develops a lightweight and effective Prompter module (though it reminds me of prompt encoder of SAM[^1])
    - Proposes a scalable approach to generate region-text annotations
3. Comprehensive evaluation:
    - Extensive experiments across 31 tasks
    - Thorough ablation studies
    - Strong performance improvements in MLLM applications

[^1]: Kirillov, Alexander, et al. "Segment anything." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
summary: This paper addresses a critical limitation of CLIP by introducing CLOC (Contrastive Localized Language-Image Pre-training), which enhances CLIP's capability to understand region-level semantics while maintaining its strong image-level performance. The key contributions include: (1) a novel concept of "promptable embeddings" that allows easy transformation of image embeddings into region representations, (2) a lightweight Prompter module that effectively extracts region features without compromising CLIP's original capabilities, and (3) a scalable pseudo-labeling pipeline (VESL) that generates high-quality region-text annotations. The approach is validated through extensive experiments across 31 evaluation tasks, showing particular effectiveness in MLLM applications.
weaknesses: 1. Insufficient comparison with related works:
    - Missing comparison with DenseCLIP[^1] and similar region-aware CLIP variants [^2, ^3, ^4, ^5]
    - While RegionCLIP[^6] is mentioned, the differentiation from other similar approaches is not clearly articulated
    - Limited discussion of trade-offs compared to existing solutions
2. Missing implementation details:
    - The region sampling strategy (how 4 regions per image are selected) is not specified
    - The rationale for setting λ as the ratio of images with region labels in the mini-batch is not explained
    - The impact of batch composition (ratio of images with/without region labels) on training is not discussed
    - The threshold (0.9) for similar text filtering lacks justification
    - The definition and extraction method of "leaf entities" in NER is unclear
    - Computational overhead analysis of the Prompter compared to RoI pooling is missing
3. Limited analysis of scalability:
    - The saturation of performance at 300M images for region tasks needs more investigation
    - The impact of the threshold for similar text filtering on training time is not analyzed (in addition to (10) in table 2)

[^1]: Rao, Yongming, et al. "Denseclip: Language-guided dense prediction with context-aware prompting." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.  
[^2]: Gu, Xiuye, et al. "Open-vocabulary object detection via vision and language knowledge distillation." arXiv preprint arXiv:2104.13921 (2021).  
[^3]: Zhou, Xingyi, et al. "Detecting twenty-thousand classes using image-level supervision." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.  
[^4]: Feng, Chengjian, et al. "Promptdet: Towards open-vocabulary detection using uncurated images." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.  
[^5]: Lin, Jiayi, and Shaogang Gong. "Gridclip: One-stage object detection by grid-level clip representation learning." arXiv preprint arXiv:2303.09252 (2023).  
[^6]: Zhong, Yiwu, et al. "Regionclip: Region-based language-image pretraining." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. What explains the significant improvement that CLOC achieves over LLaVAW and MME-C? In contrast, GQA, which heavily relies on precise object grounding, shows only limited improvement.
soundness: 2
strengths: 1. The data refinement process is scalable to billions of samples, demonstrating its practical utility in vision-language pretraining.

2. The approach is intuitive and straightforward, leading to improved grounding capabilities.

3. The architecture is easily adaptable for downstream grounding tasks, enhancing versatility.

4. This method effectively handles large-scale datasets (billions-level in this work), with comprehensive experiments conducted despite significant computational demands.

5. In addition to widely used recaption, this work also include region phrase candidates extraction and open-vocabulary detection with extracted phrases.
summary: Contrastive image-text pretraining has achieved significant success in recent years; however, low-quality annotations and the limitations of contrastive learning alone remain challenges. This work aims to enhance CLIP by incorporating fine-grained annotations, enabling the model to scale to billions of data points and demonstrating substantial improvements over the CLIP baseline.

CLOC can be viewed as a practical approach to enhancing dataset quality, with a particular focus on fine-grained grounding details.
weaknesses: 1. Visually-enriched and spatially-localized models, such as QWEN, rely on robust training to achieve high-quality outputs. These models contribute essential knowledge to the process, with the quality of grounding pseudo-labels also largely dependent on these off-the-shelf models.

2. Significant work has already been done to improve dataset captions, as seen in studies like *Improving Multimodal Datasets with Image Captioning* by Thao et al. (NeurIPS 2023). Integrating object grounding information into vision-language pretraining is also an established concept, exemplified by X-VLM (Zeng et al., ICML 2021). Although CLOC’s approach to grounding differs somewhat from X-VLM, these differences are minor and do not detract from the overall contribution.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Will the collected fine-grained region-text annotations be made publicly available? This aspect is highlighted as one of the main contributions of the study.
soundness: 3
strengths: 1. This paper addresses a significant challenge: enhancing the region-level representation of CLIP, which is crucial for various applications, including multi-modal large language models (MLLMs).
2. The experimental results presented in this paper demonstrate promising performance across both image-level and region-level tasks.
summary: This paper enhances the region-level representation in CLIP and introduces a new concept termed "promptable embeddings." The region-level representation is a critical task in this domain. The experimental results appear promising.
weaknesses: Several related works are not adequately addressed in the manuscript, such as 'UMG-CLIP: A Unified Multi-Granularity Vision Generalist for Open-World Understanding' (ECCV 2024).

---

confidence: 5
contribution: 1
initial_rating: 1
presentation: 3
questions: No questions, the work is very easy to read. I commend the authors for that.
soundness: 3
strengths: The author's model is well engineered to take latency into context while adding additional blocks.
The data engine uses synthetic captions rather than alt-text to generate data for the box predictor which is lesser explored.
The authors show improvement in multiple tasks over CLIP.
summary: The paper proposes to combine ITC with region text prediction. The authors show that a data labelling pipeline at scale for region proposal combined with a image-text dataset leads to model outperforming CLIP baseline for MMLLM among other tasks.
weaknesses: This paper would have been strong if other works did not explore better pretraining over CLIP for open vocabulary computer vision tasks. However, that is not true anymore. This work ignores all the advancement in the field of pretraining image-text representation models.
Several works in combining SSL with ITC exist and show better localization performance e.g. MaskCLIP and SILC. Moreover, BRAVE at ECCV2024 showed that these works also improve MMLLM performance. 
Moreover, the primary idea of this work has also been explored in "LocCa: Visual Pretraining with Location-aware Captioners" recently.

While I commend the authors for doing an extensive study of their method vs CLIP, we have a different landscape today. It is not fair to claim large improvements without putting other improvements from the field in context. Moreover, while the authors improve upon CLIP, their performance metrics on imagenet or LVIS fall short compared to other works from the field. Therefore I unfortunately have to recommend a rejection for this paper.

Meta-review: This paper proposes a pretraining method to enhance the localization ability of contrastive image-text pretraining (i.e., CLIP), which is considered an important topic in the research field as CLIP models are widely used across various applications. The paper presents a new methodology with experimental results across different model sizes (ViT-B/L/H) and tasks. The paper's strengths are its strong motivation, clear writing, and comprehensive experiments.

However, as Reviewer MadS and Reviewer LJyF pointed out, the paper lacks conceptual and experimental comparison with relevant previous works (e.g., MaskCLIP, SILC, BRAVE, etc.). While the reviewers and the AC understand that these previous works may differ in method direction and experimental settings, the AC believes that it would be valuable to compare them as baselines to demonstrate the advantages of the proposed method within the broader context of CLIP localization ability. 

Given the paper's notable strengths, the AC recommends submitting this work by addressing these comparison aspects to future conferences/journals rather than this ICLR.

Decision: Reject