LocCa: Visual Pretraining with Location-aware Captioners
Authors: Bo Wan
Michael Tschannen
Yongqin Xian
Filip Pavetic
Ibrahim Alabdulmohsin
Xiao Wang
André Susano Pinto
Andreas Peter Steiner
Lucas Beyer
Xiaohua Zhai
Venue: NeurIPS 2024 poster (2024)
Keywords: Vision Language Models
Visual Pretraining
Location-aware Generation
Primary area: machine_vision
TL;DR: We explore location-aware tasks as proxies for generative  visual pretraining (as opposed to transfer/instruction tuning in prior works).

Abstract: Image captioning was recently found to be an effective pretraining method similar to contrastive pretraining. This opens up the largely-unexplored potential of using natural language as a flexible and powerful interface for handling diverse pretraining tasks. In this paper, we demonstrate this with a novel visual pretraining paradigm, LocCa, that incorporates location-aware tasks into captioners to teach models to extract rich information from images. Specifically, LocCa employs two tasks, bounding box prediction and location-dependent captioning, conditioned on the image pixel input. Thanks to the multitask capabilities of an encoder-decoder architecture, we show that an image captioner can effortlessly handle multiple tasks during pretraining. LocCa significantly outperforms standard captioners on downstream localization tasks, achieving state-of-the-art results on RefCOCO/+/g, while maintaining comparable performance on holistic tasks. Our work paves the way for further exploration of natural language interfaces in visual pretraining.

LocCa: Visual Pretraining with Location-aware Captioners
Bo Wan
Michael Tschannen;
Yonggin Xiang
Filip Pavetic,
Ibrahim Alabdulmohsin;
Xiao Wang,
André Susano Pinto,
Andreas Steiner;
Lucas Beyer;
Xiaohua Zhaij
GGoogle DeepMind, Zürich
2 Google, Zürich
KU Leuven
Abstract
Image captioning was recently found be an effective pretraining method similar to contrastive pretraining. This opens up the unexploede potential of using natural language as flexible and powerful interface for handling diverse pretraining tasks. In this paper, we demonstrate this with novel visual pretraining paradigm, LocCa, that incorporates location-awan) tasks into captioners to teach models extract rich information from images. Specifically, LocCa employs two tasks, bounding box prediction and location-dependent captioning, conditioned on the image pixel input. Thanks to the multitask capabilities of encoder-decoder architecture, we show that an image captioner can effortlessly handle multiple tasks during pretraining. LocCa significantly outperforms standard captioners downstream localization tasks, achieving saate-of-the-re art results on ReiCOCO/+/g, while maintaining comparable performance holistic tasks. Our work paves the way for further exploration of natural language interfaces in visual pretraining.
Introduction
where vision models pretrained on large-scale annotated datasets (5) 6 4 with supervised classification loss. Yet, the manual annotation required for such datasets is time-consuming and costly, posing a challenge to scalability.
Remarkable progress has been made in large-scale visual pretraining 22 3
extract learning signals from web-crawled image-text pairwise datasets (9] 10], circumventing the need for extensive manual annotations. The contrastively pretrained models demonstrate remarkable capability on zero-shot transfer tasks, especially downstream applications that require fine- tineggriied visual [IT] [2] or textual understanding [13]. More recently, image captioning has been shown as an alternative visual pretraining task learn capable vision encoders [14), where encoder-decoder architecture is pretrained to generate text captions from the image input. Some studies, such as [15] 16), pioneered the joint pretraining of contrastive and generative methods. Typically, encoded image features fed into two parallel branches: one employs a text encoder to produce sentence embeddings for contrastive learning, while the other utilizes text decoder generate image captions. Despite the effectiveness of these works, they typically focus on holistic understanding of images, often overlooking the region-specific details of the visual content.
In light of this, the modern contrastive pretraining methods
The recent success of image captioning [4] and the advancements in multitasking learning of opens up the unexplodde potential of using natural language
decoders 17
18 19
"Work done during Google DeepMind internship. Project lead. All authors made significant technical contributions.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

A picture with puffin standing on cliff edge and another puffin curled up back.
puffii standing on cliff edge [20, 480, 150, 200]
[400, 110, 460, 40] another puffin curled up in the back
Cross
Attention
Transformer Decoder
Location- aware Captioner
Vision Transformer
Cap:
ARef:
GCap:
Figure 1: Overview of LocCa. LocCa consists of a standard vision transformer and transformer decoder. The vision transformer takes image pixel as input, produces visual tokens as cross attention input to the transformer decoder. The transformer decoder trained to read out rich information from the visual tokens. We adopt the following three task for pretraining: Cap, AREF and GCAP.
flexible and powerful interface for handling diverse tasks. We demonstrate this with novel visual pretraining paradigm, LocCa, that enhances the visual representation with location-aware context. Works including 20 21 22 investigate the matching of image regions with corresponding text during pretraining. The central concept involves extracting Region of Interest (Rol) features from image embedding to facilitate contrastive learning with corresponding textual features. These approaches yield encouraging outcomes in location-sensitive tasks, such as object detection [23] 24 25 26 and referring expression
However, they require complex model architectures (c.g. RPN [24] and FPN (25 for Rol generation. Also, given the presence of multiple object candidates within an image, region- wise matching becomes computationally demanding.
29 30
By contrast, LocCa is a simple yet effective location-ane aware captioning pretraining method as shown in Figure which uses autoregressive decoder
interface to handle additional location-aware pretraining tasks. Concretely, other than the image-to-text captioning task, LoeCa also pretrains the model with two location-aware tasks: (i) automatic referring expressions, which amounts predict bounding box coordinates from automatically generated captions for specific image regions, and (ii) grounded captioning to jointly predict box coordinates and captions from the image. Specifically, LocCa leverages multi-task decoder 17 for pretraining, where the model outputs are conditioned on the task prefixes for each task. Thanks to the shared vision transformer for multiple tasks, the additional localization losses are relatively cheap to compute, while the model inference speed is identical the standard image caption pretrained models.
Our experimental results show that the LoeCa performs significantly better on downstream tasks that require localization capabilities, while maintaining the same level of capabilities holistic tasks. We summarize our contributions as follows: (i) For the first time we explore location-aware tasks as proxies for generative visual pretraining (as opposed transfer/instruction tuning prior works), enabling flexibly customized inference (detailed in Sec 3.3): (ii) Without bells and whistles, LocCa achieves state-of-the-an results on localization tasks, while preserving the competitive performance on holistic tasks; and (iii When integrated in vision-language models, i.e. PaLI-3 [1, the vision encoder outperforms strong SigLIP baselines B
2 Related Works
Contrastive visual pretraining is prominent direction in training vision and vision-language founda- explore image-only contrastive loss by matching different views of the same image in the self-supervised learning setting. In vasion-language model pretraining, CLIP (7] and ALIGN [8] show that a two-tower architecture trained with the contrastive objective on noisy image-text pairs can learn highly transferable image and text representations for various
tion models. Early works 32 33 34 35

---

downstream tasks. There have been many follow-up works [TO 36 13, 37 that further improve the
zero-shot image classification and retrieval performance. Notably, 20, 21 propose to incorporate location cues by contrastively aligning image regions and text phrases. In contrast, our work focuses on learning location-awan
laiiunl model with a generative loss.
A natural alternative to contrastive pretraining is image captioning: Rather than matching image and text embeddings, one tries to predict captions from an image embedding. Early works investigate this approach at small scale [38
39
40
41 Later works augment large-scale contrastive pretraining
with captioning loss 15
16
or scale-up captioning as a stand-alone task without investigating
transfer to broad set of vision tasks 43 44) [4] recently showed that image captioning alone
leads to visual representations competitive with contrastive pretraining.
Many recent large multimodal models are trained with a mixture tasks (45
18
9
46
49 Among the most popular types of tasks those which can be formulated by mapping image to a text string, including captioning, detection, and VQA (50] 45 [8] popular group of tasks are dense prediction tasks such as segmentation and depth prediction [9 46). While several studies have enhanced model pretraining by incorporating location information, their methodologies primarily leverage either pretrained language decoders 08 51 52 or pretrained cross-modis encoder-decodes [51] to integrate vision and language features for multitasking purposes, often neglecting the independent significance of visual pretraining from scratch. Furthermore, there a trend of towards co- co-training on images, video, and audio [53] [4 [55], highlighting the multifaceted nature of current multi-modal research. Crucially, essentially all of these works rely on pretrained vision and language backhones, and merely fine-tune those together on the described tasks. Here, by contrast, we use multi-task pretraining to train visual encoders from scratch.
19
10
47
31
48
Another
3 Location-aware Captioner
In this section, we introduce the location-aware image captioner LocCa for multitask pretraining. LocCa builds on an image captioner but provides a recipe for integrating rrcation-awars information during model pretraining.
3.1 Pretraining tasks
The pretraining phase of LocCa draws inspiration from pioneering works that have successfully
integrated unified decoder for multitasking based pretrained models [18 10 19 46 52], utilizing
a task-specific prefix for each distinct task. This enhances the model's ability to handle multiple tasks concurrently.
For conventional image captioning, the process involves taking an image æ as input and generating a sequence of text tokens y [y1, ....... 3/.. In the LocCa framework, a task-specific prefix, labeled as "Cap:", is added to the beginning of the caption sequence to designate the task at hand. Moreover, LocCa integrates two additional location-aware tasks during its pretraining phase: automatic referring expression (AREF) and grounded captioning (GCAP). These tasks are inspired by referring expression
comprehension 27 28 29 30 and dense captioning [56] 57 58 59 60 respectively. The key
difference is that LocCa predicts both regional captions and box coordinates sequentially with task prefixes, instead of relying on either caption box conditional inputs (see Fig.
The foundation of LocCa's pretraining is built upon dense, automatically generated region annotations. Each image is associated with a comprchensive set of annotations {(b, c)). where b E A denotes the bounding box coordinates, and represents the corresponding textual descriptions or labels. For every bounding box, two distinct prompts are generated cater to the aforementioned location-aware tasks: "'Ref:: {c} {b}" for automatic referring expression and "GCap: {b) {c}" for grounded captioning, each prefixed with "ARef:" and "GCap:" respectively. These prompts are then tokenized to produce the sequence y for each task, facilitating pretraining with a text interface.
For each image, LocCa utilizes the same visual features extracted by the image encoder and performs three tasks using the same decoder parallel. This pretraining scheme aims to make LocCa adept at linking fine- ine-grained regional visual elements with appropriate textual descriptions.
3

---

3.2 Model details
Architecture LocCa utilizes a conventional encoder-decoder framework, where the encoder comprises Vision Transformer 2 to transform the input image x into sequence of feature emheddings. The decoder, built on a Transformer architecture [61], processes these image features, employing cross-attention across each layer to integrate visual and textual information effectively.
Autogressive decoding In the decoding stage, LocCa uses causal attention masks [6]] to guide the prediction of each token in the sequence, ensuring that each token is generated based on the ones before it, in a step-by-step manner. This setup helps in creating coherent and context-aware captions, drawing from the visual cues encoded earlier and maintaining logical flow in the generated text.
Parallel prediction Inspired by [14, LocCa also adopts parallel prediction for fraction of the training examples (concretely 50%) in the standard image captioning task. This technique requires the model to predict the caption tokens independently in parallel, focusing exclusively on visual information to prevent the model from relying preceding text tokens to predict the following ones. This strategy was shown to improve the learned representation on a range of tasks 14.
Objective The optimization of LocCa's parameters 0 is achieved through the maximization of the log- logllikeihhood: Σ log Pe 2). It is important to emphasize that, in the learning process for the location-awas aware tasks, LocCa is structured to predict captions and bounding boxes sequentially, contrasting with traditional approaches that might predict caption based given bounding box or vice versa 18, 51). This is achieved by applying losses to the entire prompt excluding the task prefix. Taking the AREF task as example, the overall loss is computed for both textual predictions and box coordinates b. The loss on e guides the model to identify a foreground region and generate its caption, while the loss on b aims at refining the model's ability to accurately regress the box location relative to the caption.
3.3 Discussion
To the best of our knowledge. LocCa is the first end-to-end method that incorporates the location- aware tasks (i.e., AREF and GCAP) into generative VI.M pretraining. Our key novelty lies in the formulation of the location-aware proxy tasks which allows for scalable pretraining and enhances the visual localization capabilities. Compared to 22
that require either a pretrained visual encoder or language decoder, LocCa does not need any pretrained model for initialization. Compared to [22 62) that employ complex architectures with multiple losses, LocCa adopts simple encoder-decoder architecture with single generative loss. Compared to [T8 19] that are pretrained on a large number of tasks, LocCa introduces the novel use of simple AREF and GCAP proxy tasks for visual pretraining.
18
51
52
The dual-faceted loss structure of AREF and GCAP achieves comparable results to directly adopting the traditional referring expression and dense captioning tasks. However, it enhances inference flexibility of LocCa, allowing for varied input configurations. For instance, users can input a single task prefix (e.g., "'Reff") to prompt the model to identify and describe an area of interest along with its location. Alternatively, by inputting both the task and a conditional input (c.g.. "ARef: a black and white cat :"), LocCa can be directed to focus solely on predicting the location. This flexibility allows for customized responses to various inquiries, highlighting the model's adaptability to meet specific user needs.
4 Experiments
4.1 Experimental setup
Pretraining dataset We use subset of the WebLI dataset [10 corresponding to English websites and apply text-based filtering (8] obtain IB image/alt-texx pairs. The WebLI data was de-duplicated w.r.t. all the images in the evaluation data sets used in this paper. To obtain fine ine-grained object locations, a publicly available OWL-VIT CLIP L/14 model 63 is applied to generate detection pseudo annotations. Specifically, two groups of box categories are generated: the n-gram texts from alt-text and the object categories as used by PaI.I 10), more details can be found in (64] In LocCa pretraining, we first filter the candidate bounding boxes according to their OWL-ViT confidence score

---

with threshold of 0.3, and then randomly sample one box-caption or box-category pair for referring expression and grounded captioning separately.
Baselines Our main baselines are CLIP-style contrastively pretrained dual-encoder models 7 on our dataset (referred to as CLIP* to differentiate from the model released by [ well as the prettriinnd encoder-decoder models from [14]. For both types of models we follow the exact training recipe from [14] For the captioning-base autoregressive captioning (Cap) as well as the variant with parallel prediction (CapPa). This variant removes the decoder attention mask for a fraction of the training examples and replaces the decoder input, which corresponds to the nightes output sequence during autoregressive training, with a sequence of all mask tokens. Parallel prediction (14) improves the representation learning capabilities captioning -based pretraining at no extra computation cost.
pretraining we consider standard
Implementation details LocCa adopts an encoder-decoder structure where, unless otherwise noted, the encoder defaults to a standard ViT-L/14 design with 24 transformer blocks handling input image patches of size 14. The decoder, following [14], is Transformer-L model consisting of 12 transformer decoder blocks. In total, the model comprises approximately 600M parameters. Two more LocCa setups with ViT-B/16 and ViT-G/14 (see 4 for model specifications) are adopted for ablation tests and scaling experiments respectively.
Pretraining details LocCa is pretrained for about 9 billion image/alt-text seen examples, which corresponds to about epochs on our tailored subset of WebLI. For the optimizer, we employ the Scaling- ViT AdaFactor variant [4], combined with a cosine schedule that includes 10,000 warmup steps. The batch size is set at 8,192, while the learning rate and decay factor are adjusted to 10-3 and 10-1, respectively. During this process, images uniformly resized pixels. Alt-texts are tokenized into a vocabulary consisting of 32,000 tokens using a sentence piece model trained on the English segment of C4 [65), with a cap on the sequence length 64 tokens. We represent bounding box coordinates using up to 500 integral numbers, which are then directly converted into strings for straightforward representation of coordinate tokens. For parallel prediction in the vanilla image captioning task, the fraction of examples is set to 50% by default. The pretraining of LocCa,, takes 153 hours using 256 TPUv3 chips.
resolution of 224 224
Downstream tasks Our goal with LocCa is to preserve or even improve the capability on various image-level understanding tasks, and get a higher performance on fine-grained location-aware tasks. To this end, we assess the capabilities of LocCa in holistic and location-aware image understanding tasks across range of downstream tasks. In the realm of holistic image understanding, focus on the same LiT-Decoder tasks [17] in CapPa (14] including image classification (CLS) 66
67 68 69), image captioning (CAP) 70 71, optical character recognition (OCR-VQA) (72).
visual question answering (VQA) (73) and graph question answering (GQA) (74). For evaluating location aware image understanding, we choose two widely recognized tasks like referring expression comprehension (REC) (27). referring expression segmentation (RES) (75) and object detection (OD) 24 Additionally, paralleling the approaches of PaLI [TO, 31, we integrate LocCa a
pretrained large language model to assess performance on a variety of vision-language tasks, including image captioning and visual question answering. Notably, despite LocCa not being exposed to any video content during pretraining, we adapt it to various video-related tasks, such as video captioning, QA, and classification. This adaptation aims to assess its generalization capabilities to new modalities, demonstrating its versatility compared to other image-text pretraining approaches. We adopt different strategies for transferring LocCa to downstream tasks, as summarized in Appendix A.
4.2 Quantitative results
We conduct extensive experiments to evaluate LocCa. The integration of location aware cues enables LocCa to maintain its performance on holistic image understanding tasks while achieving substantially improved outcomes on location-aware tasks. Further enhanced by advanced pretrained large language model, LoeCa exhibits exceptional performance across a range of lision-language tasks, substantially surpassing baseline models.
Referring Expression Comprehension In this section, we present results on the referring expres- sion comprehension benchmarks, including RefCOCO, RefCOCO+ and RefCOCOg 28 As shown in Table [ LocCa establishes a new state-of-the-an across these benchmarks. This advancement is particularly noteworthy considering the inherent limitations of previous methods. For example,
5

---

Table 1: Result comparison with previous SOTA methods on RefCOCO benchmarks.
RefCOCO
RefCOCO+
ReiCOCOg
MODEL
testA
val
testA
testB
val-u
test-u
Models that trained valiest insuges, lext
PixelLLM
89.8
92.2
86.4
83.2
87.0
78.9
84.6
86.0
UniTAB 76
88.59
91.06
83.75
80.97
85,36
71.55
84.58
84,70
OFA 18
90.05
92.93
85.26
85.80
89.87
79.22
85,89
86,55
UNINEXT
91.43
93.73
88.93
83.09
87.90
76.15
86.91
87.48
LocCaz
91.94
94.56
89.13
86.47
91.67
80.43
87.46
87.95
OFA 18
92.04
94,03
88,44
87.86
91.70
80.71
88.07
88.78
UNINEXT 3/
92.64
94.33
91.46
85.24
89.63
79.79
88.73
89.37
ONE-
92.58
94.18
89.26
88.77
92.21
83.23
89.22
89.27
PEACE 55 02
Shikra1se 78
87.83
91.11
81.81
82,89
87.79
74.41
82,64
83.16
Ferretiaa 79
89.48
92.41
84.36
82.81
88.14
75.17
85.83
86.34
LocCag
92.99
95.02
90.48
89.12
92.87
83.55
89.24
89.90
Modets that have val/iest images duriing pre-training, for
UNITER, 80
81.41
87.04
74.17
75.90
81.45
66.70
74.86
75.77
VILLA 81
82.39
87.48 89.58
74.84
76.17
81.54
66.84
76.18
76.71
MDETR SP
86.75
81,41
79.52
84.09
70.62
81.64
80.89
Models that have never seen val/test images, see text for more.
RefTR[83]
85,65
88.73 92.75
81.16
77.55
82.26
68,
79.25
80,01
LocCaz
89.70
85.30
83.85
89.40
76.76
84.62
85,86
LocCag
91.20
93.34
87.56
86.89
90.71
80.73
87.34
87.90
tailored for irrection-based tasks, and OFA, which requires a pretrained BART 85 for initialization, both achieve good results but are constrained by their specialized setups. Besides, some location-aware VLMs, such as Shikra [78] and Ferret (79), require an LLM for knowledge- contrast, LocCa employs standard encoder-decoder architecture with auto-regressive pretraining from scratch, significantly outperforming these methods across all benchmarks without the need for complex task-specific adaptations.
UNINEXT 777.
which adopts a Deformable DETR architecture 84
reasoning, which increases inference costs. In
Achieving good performance on RefCOCO usually requires pretraining at high image resolutions. For instance, OFA, is pretrained at 480²px, while UNI-NEXT undergoes multi-scale pre-training. opt for a standard 224"px pretrain resolution model to RefCOCO by fine-tuning with 640ºpx resolution, using learning rate 10- and no weight decay. We report the standard metric A.5@0.5 on the validation and test sets.
simplicity. We transfer the 224 pretrained
Notably, we train on the combined training sets of RefCOCO, RefCOCO+ and RefCOCOg but removing all validation and test images from this combination. The splits of these three datasets are largely overlapping, meaning that methods trained on the combined training sets without de- duplication, a recently common phenomenon, have trained on about half of the test images, see Appendix for details. We provide the list of removed image IDs in the Appendix A.3 Furthermore, some models such as UNITER 80), VILLA [T], and MDETR 82 use COCO pre- detection components which have seen test images from the three RefCOCO versions. We have removed all training, validation, and test images from the COCO dataset from our pre-training data, as well as near-duplicates thereof. Hence, we group models reported in Table into three distinct categories. We transfer LocCa on both the full and the "clean" combined training set and provide both results. We hope that, going forward, the community evaluates models RefCOCO in this clean setup.
Moreover, as shown in Table 2 LocCa improved significantly over all the baselines of image-text pretrained models. To ensure a fair comparison with contrastive baselines (i.e. CLIP) which lack text decoder for box prediction, we employ the LiT-Decoder [77] setup for comparison on RefCOCO benchmarks. It involves freezing the pretrained image encoder while training text decoder from scratch with a small resolution 2242px. This setup necessary to properly compare with CLIP- style models without decoder. The performance improvements attributable to location-aware pretraining are evident, demonstrating the enhanced sensitivity of visual encoder object regions, which is crucial for excelling in referring expression tasks.

---

Table 2: Comparison with baselines RefCOCOs. A randomly initialized decoder is trained for REC and RES, with frozen image encoders. We report Acc@0.5 for REC and mloU for RES. Here CLIP uses model checkpoints released by 77]; all other baselines the same data as LocCa.
RefCOCO
RefCOCO+
RefCOCOg
MODEL
testA
testB
testA
testB
val-u
test-u
CLIP
7
65.21
71.28
58.17
57.53
66.44
47.77
59.32
60.24
REC
CLIP
58.28
63.59
53.73
49.01
55.94
41.96
55.70
55.88
Cap 14
60.64
65.47
56.17
52.56
58.32
45.99
56.75
57.99
CapPa
44
64.17
69,90
58.25
56.14
63.68
48.18
58,90
59.91
LocCa
88.34
91.20
85.10
79.39
85.13
72.61
81.69
82.64
CLIP t
36.15
38.25
35.82
31.40
36.00
28.69
29.21
29.44
CLIP*
32.78
34.89
33.03
27.49
30.14
25.07
26.99
26.83
RES
Cap [14
34.84
37.68
35,39
31.93
35.24
28.79
28.84
29.11
CapPa 14
36.62
39.23
36.67
32.54
37.49
29.59
30.40
30.43
LocCa
64.98
65.39
64.09
57.85
60.92
52.72
55.84
56.95
Table 3: Results on holistic image understanding tasks. Here CLIP uses model checkpoints released by [7]: all other baselines are trained on the same data as LocCa.
Classification
Captioning
OCR
VQA
MODEL
sun
food
res
pet
COCO
Flickr
VQA
VQAv2
GQA
CLIP 7
84.8
84.8
95.2
96.3
95.4
124.4
87.1
64.1
70.4
58.7
CLIP*
84.7
85.7
94.6
96.4
95.2
123.2
85.5
61.3
68.5
55.3
Cap
18
83.8
84.7
93.4
95.1
95.0
125,9
88.3
64.2
70.9
58.5
CapPa 14
84.4
84.9
93.8
96.0
95.6
125.8
89.3
65.6
70.9
58.3
LocCa
84.5
84.9
93.9
96.0
95.0
127.1
90.7
64.5
72.8
61.8
LocCag
85.8
95.4
96.1
95.8
130.9
92.6
67.6
73.9
61.5
Referring Expression Segmentation For referring expression segmentation, we extend the REC task by adding suffix "Mask:" the text, followed by the indexes of segmentation tokens. The segmentation tokens specify the precise shape of the segmentation mask within the bounding box that is identified during the REC task. This process leverages a pretrained VQ-VAE [66 to convert the semantic masks tokens, please refer Appendix A.3 for more details.
LocCa is adapted RES under the same settings in REC, using the "clean" combined training sets of RefCOCO, RefCOCO+, and RefCOCOg (c.f. Appendix(c. Specifically, we compare different frozen encoders and train decoder from scratch. As shown in Table [ LocCa's 's vision encoder outperforms other encoders substantially, thereby providing further validation of its location-wiss sensitivity. Moreover, we employ the full encoder-decoder LocCa model and fine-tune it for RES. Notably, LocCa achieves competitive results even compared to the state-of-the-aa PaLI-3 model, albeit with considerably fewer parameters (0.6B vs. 5B). See Appendix A.3 for details.
Holistic Image Understanding While being great on the referring expression comprehension tasks, also verified that LocCa performs equally well on the holistic image understanding tasks. Interestingly, we found that LocCa outperforms the image-text pretrained baselines on the object- centric tasks like VQAv2 and GQA.
Following the similar evaluation protocol in [4], we evaluate the capability of LocCa with the "LiT decoder" [17] setup, to investigate the adaptation capability of the learned representations to interface with a text decoder. Here we report the classification accuracy on 5 classification (CLS) datasets (ImageNet-Il 1k (5), Sun-397 [67], Food-101 101 [666), Resisc-45 (68). Oxford-Pet (69). and also answer accuracy on VQAv2 and GQA and OCR- VQA datasets. Besides, we report the CIDEr score on captioning (CAP) datasets (COCO 770 and Flickr (7l)..
As shown Table the performance of LocCa is better than image-text pretrained baselines on image captioning, VQA and GQA, comparable image classification, and slightly lags OCR-

---

mAP and on COCO detection
(a) Ablation on pretrained resolution
RatCOCO AVG
RefCOCC++ AVG
mAP
56.3
91.3
91.4
83.5
50
48.5
45.7
47.7
48.3
91.0
85.0
pretrain 224
45
$4.01
pretrain res 384
40.9
39.0
39.2
(b) String vs Special tokenization
33.2
RefOOCO AVG
RefCOCO+ AVG
91.6
30
28.1
91.2
25.3
25.1
string
90.4
special
C1.IP
Cap
CapPa
LocCa
234
224
540
Transfer resolusion
Transfer resolusion
Figure 2: Result COCO detection with a limit of 25 output boxes. For reward tuned models show both the results be- fore (dark blue and orange) and after (light blue and orange) reinforce tuning 88
Figure 3: Ablation studies on (a) impact of differ- ent pretrained image resolutions on string token; and (b) string special token of box coordinates with pretrained res 224. The results are the average Acc@0., of the val&test splits RefCOCO/+.
Table 4: Results on PaI.I33 31 transfers to diverse captioning and question answering tasks. LocCa consistently outperforms other image encoders, especially tasks requiring understanding of objects, including both natural and text (OCR) objects.
MODEL
COCO
VQAv2
OKVQA
TextVQA
ST-VQA
TallyQA
SigLIP
135.8
75.6
57.5
41.1
46.2
74.9/61.4
Capı 14
135.0
75.3
57.5
44.6
44.5
73.2/62.0
CapPaz
14
135.3
75.5
57.7
44.0
45.2
73.4/60.9
LocCar.
138.9
77.6
58.4
49.2
50.9
79.3/64.1
SigLIPa 140.3
77.5
58.6
50.6
50.5
76.3/61.8
LocCag 140.9
78.1
59.8
52.1
52.3
79.0/64.2
VQA. Notably, both VQA and GQA necessitate fine-grained instance-level comprehension, focusing on the spatial and semantic relationships between objects accurately provide the correct answer. For example, GQA involves complicated information about objects, attributes and relations provided by scene graphs (57]. Such observation further reveals the advantage of LoeCa on fine-grained object-level sensitivity.
It is also interesting to investigate the full potential of the LocCa model by fine-tuning it holistic tasks with a small 3 x 10-6 learning rate without weight decay. To this end, we choose the widely used COCO image captioning task
example. LocCa achieves competitive results of 138.0 CIDEr score with a standard 2242px px. When increasing the transfer resolution to 6402px. px, the LocCaz model achieves 140.3 CIDEr score without CIDEr metric optimization [87].
Vision-Language Models with LocCa In this section, we present results on vision-language tasks with LoeCa plugged into pretrained large language model. More specifically, we use PaI.I-3 (31) here to test the vision encoder quality. Importantly, we discovered that the generative LocCa vision backbone outperforms the SigLIP backbone, which is the default setup used in PaLI-3.
We select wide range of tasks to assess the models' proficiency in understanding various visual concepts, including: image scene (COCO Caption), objects (VQAv2 and TallyQA (89). visually- situated QA [90], ST-VQA (91) and general knowledge (OKVQA (22). Remarkably, as shown in Table 4 LocCa consistently surpasses the Cap and CapPa vision encoders by significant margin across these tasks. Note that SigLIP, Table 4 uses image patch size 16 while all the other models use patch size 14. SigL.P,, is marked grey because it's not directly comparable. The LocCag model consistently outperforms SigLIP across all the tasks. With the knowledge of object and textual regions, LocCa is better positioned to understand more challenging and subtle

---

concepts like counting, visual relations, and word characters. Further fine-tuning details and the hyperparameters we used are provided in Appendix AA.4 [A.4
Object Detection To evaluate for object detection, we use COCO detection dataset and follow [8 which models the task with an encoder-decoder model that outputs sequences of bounding boxes similar to pix2seq The model is trained in two phases, first it maximizes the log- log-likelihood of generating the ground truth sequences of boxes, secondly uses reinforce to tune the model for reward related to the mAP metric.
pretrain their encoder-decoder model in Objects365 [93 and we found this to be critical. In particular the task output format is different from the ones LocCa used during pretraining and the size of COCO is too small to learn it without overfitting. To measure the performance of the encoders, we opt to initialize the decoder from an Objects365 pretrained model. More details of adapting LocCa for object detection please refer Appendix A.5
Both 88 50
We present the comparative results in Figure 2 where LocCa significantly outperforms all image- text pretrained baselines in both mAP and AR, at both stages - before and after the application of reinforcement tuning. This performance aligns with our expectations regarding LocCa's object-centric comprehension, attributed to the integration of additional location-aware pretraining.
Zero-shot Transfer Following locked- locked-image tuning 33)] we freeze the pre-trained vision encoder and train text encoder contrastively to perform zero-shot classification and zero-shot retrieval on downstream tasks. With an l/14 architecture and 3B examples LiTe-tune duration, LocCas, achieves 77.1% O-shot accuracy on ImageNet, which is competitive to the same size CapPas, [14] model's s 76.4% accuracy. On COCO retrieval tasks, LocCa,, achieves 46.6% and 64.6% on text-to-image and maae--to-text retrieval tasks respectively. It outperforms CapPay is 43.9% and 60.3% COCO retrieval results by large margin. Please refer to Appendix [A.6 for more details.
Semantic Segmentation We investigate the dense feature learning capabilities of LocCa by evaluating it for semantic segmentation on ADE20k (95] along with Cap and CapPa. To this end, we use the Segmenter framework [94] which attaches a linear layer to every patch embedding to predict the semantic label of that patch, and obtains the high-resolution semantic map by upsampling the low-resolution map. The results in (Sh that LocCa outperforms Cap and CapPa by about 2 mloU points, and the Seg VIT-L baseline which is based on an transfer details are provided in Appendix A.7
Table 5: Fine-tuning vision back- bones with a linear head [94] for se- mantic seg. on ADE20K.
MODEL
mIoU
Seg ViT-L (44
50.71
Cap
+9.
CapPa
49.92
LocCa
51.81 +0.3
ViT-L by about point. The
Video Understanding We follow PaLI-3 31 to evaluate LocCa and CapPa on video understanding tasks. Specifically, we use the image encoder to process each frame separately and concatenate all resulting tokens. A LiT-Decoder [17 is then tuned on a mixture of six video understanding tasks. Despite no video has been seen during pretraining, LocCa showcases the video understanding capabilities and outperforms CapPa on most of the datasets. In particular, for the video captioning task on VATEX (96) dataset, LocCa achieves a CIDER score of 65.0 vs 64.0 of CapPa. For the video QA task on MSVD 997 dataset, LocCa obtains an accuracy of 50.9 vs 50.0 of CapPa. More details could be found from Appendix A.8
4.3 Qualitative Results
In this section, we discuss the raw LocCa model's zero-shot capability to detect multiple objects using its own decoder, despite only a single object per example being observed during pretraining. To achieve this, we employ beam search when decoding the output from LocCa. This involves using the prompt of a single task prefix "GCap:" to instruct LocCa to predict the Rols along with their labels. As shown in Appendix Fig. [6) we observe that depending on the setting we get lower number of bounding boxes with correct class names, or higher number of boxes with class names which start to contain noise. We provide more discussions in Appendix B Nevertheless, pre-trained LocCa model shows powerful capability after short finetuning clean downstream dataset as shown in Table Finding a decoding strategy which can output high number of boxes and quality labels at the same time is open problem.

---

Table 6:
Ablation study on applying loss on AREF and GCAP tasks during training.
RefCOCO
RefCOCO+
AREF GCAP
INet
COCO
VQAv2
GQA
val
testA
testB
testA
testB
V
80.4
117.7
68.4
59.0
88.0
90.8
84.0
80.2
84.7
73.8
79.7
115.9
67.5
57.9
86.8
89.6
83.0
77.7
83.5
71.2
X
V
79.7
114.8
67.4
57.7
83.7
87.2
80.7
72.3
77.9
66.6
x
X
78.3
111.0
66.2
54.3
75.2
78.8
69.5
64.7
71.0
55.9
4.4 Ablations
Pretraining and transfer resolutions We present results with different pretrain resolution and transfer resolution combinations in Figure B (a). To get LocCa model with higher 384 pretrain resolution, we finetune the 224 resolution model using the target 384 resolution for 900M examples seen on same dataset. Figure 3 (a), we transfer both the 224 resolution model and the 384 resolution model to RefCOCOs using different transfer resolutions (224, 384, 640) as marked on the x-axis. We find that the higher pretrain model achieves slightly higher or competitive results compared to the 224 resolution models. We opt for the 224 resolution pretrain models by default in this paper for simplicity. More results with high pretrain resolution could found in the Appendix (A.10)
Coordinate tokenization Previous studies [ 31] often tokenize object coordinates by adding location vocabulary. In contrast, LoeCa simplifies this process by directly converting integral coordinates into textual strings, which are then tokenized with the same text tokenizer. This section presents an ablation study on the RefCOCO benchmarks to examine the differences between these two options. As shown in Figure 3 (b), both tokenization strategies for box coordinates yield remarkably similar results. However, our approach is notably simpler and more straightforward.
Selection of pretraining tasks To evaluate the importance of the selected tasks for pretraining LocCa, an ablation study was conducted focusing on the effects of AREF and GCAP tasks. Specifi- cally, the study explored the impact on LocCa by removing these tasks individually and collectively. with the model defaulting to the CapPa baseline when both are excluded. All these models are pretrained on the WebLI split for 900M examples with the resolution of 224 px, and subsequently evaluated on the holistic tasks using LiT-Decoder setup and on RefCOCOs by fine-tuning the whole model. As shown in T incorporating any pecaioon-warg task into the pretraining of LocCa yields significant performance improvements, particularly evident in the RefCOCO results. Interest- ingly, incorporating solely the GCAP task leads to marked improvement in RefCOCO performance compared to the CapPa baseline, despite GCAP not being directly aligned with the AREF task. This highlights the importance of introducing object concepts for enhancing regional-level understanding. which is beneficial for fine-grained visual comprehension and applicable to other object-sensitive tasks. Furthermore, the combined inclusion of both AREF and GCAP tasks yields even better results, demonstrating their complementary nature in improving LocCa's performance.
Conclusion
LocCa introduces a novel visual pretraining paradigm, using natural language interface construct the proxy location-aware pretrain tasks. This model excels in understanding both the overall scene and specific spatial details, setting new performance standards on location-aware tasks while preserving the capability on holistic image understanding. LocCa simplifies the process of blending location information with visual data, offering significant improvements on tasks requiring detailed spatial awareness. Our contributions pave the way for advanced model capabilities in processing broad spectrum of vision-langueenn language tasks, demonstrating LocCa's versatility and effectiveness.
LocCa already demonstrates superior zero-shot detection capability in qualitative results. However, it lacks the capability for zero-shot segmentation due the absence of pretraining on pixel-level anno- tations, and we leave this for future work. Building on the robust foundation of LocCa, a promising direction for future work involves enhancing its pixel-level precision through the incorporation of segmentation tasks during the pretraining phase. This extension aims to equip LocCa with a more nuanced understanding of images, enabling it to discern and interpret intricate details and textures with unparalleled accuracy, further broadening its applicability across diverse vision- language tasks.
10

---

Acknowledgements
We thank Matthias Minderer and Jeremiah Harmsen for their valuable feedback on the manuscript, and Debidatta Dwibedi and Xingyi Zhou for their feedback on dense captioning evaluators. We also thank Paul Voigtlaender for discussions on the referring expression segmentation task. Finally, we thank the Google DeepMind team for providing supportive research environment. We used the big_ vision codebase [98] 99 for all experiments in this project.
References
[1] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (BiT): General visual representation learning. In ECCV, 2020.
[2] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbomn Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition scale. In ICLR, 2021.
[3] Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR, pages 11966-11776 2022.
[4] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision trans- formers. CVPR, 2022.
[5] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[6] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In ICCV, 2017.
[7] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
[8] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICMI., 2021.
[9] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION-400M: Open dataset of CLIP-filtered 400 million image-text pairs. CORR, arXiv:2111.00 1.021 14, 2021.
[10] Xi Chen, Xiao Wang, Soravit Changpinyo, A. Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thap- liyal, James Bradbury, Weicheng Kuo, Mojtaha Seyedhosseini, Chao Jia, Bureu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. Pali: A jointly-scaled multilingual language-image model. CoRR, arXiv:2209.066 06794, 2022.
[11] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. In ICLR, 2022.
[12] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. CoRR, arXiv:2303.05999 2023.
[13] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training In ICCV, pages 11941-11952, 2023.

---

[14] Michael Tschannen, Manoj Kumar, Andreas Steiner, Xiaohua Zhai, Neil Houlsby, and Lucas Beyer. Image captioners are scalable vision learners too. NeurIPS, 2023.
[15] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Trans. Machine Learning Research, 2022.
[16] Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language- image pre-training for unified vision-language understanding and generation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, ICML, volume 162, pages 12888 12900, 2022.
[17] Lucas Beyer, Bo Wan, Gagan Madan, Filip Pavetic, Andreas Steiner, Alexander Kolesnikov, André Susano Pinto, Emanuele Bugliarello, Xiao Wang. Qihang Yu, Liang- Chich Chen, and Xiaohua Zhai. A study of autoregressive decoders for multi-tasking in computer vision. CoRR, arXiv:2303. 17376, 2023.
[18] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. OFA: unifying architectures, tasks, and modalities through a simple sequence-to-sequene
learning framework. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, ICML, volume 162, pages 23318-23382 233400, 2022.
[19] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. UNIFIED-IO: A unified model for vision, language, and multi-moda tasks. In ICLR, 2023.
[20] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language-image pre-training. In CVPR, pages 10955-10965, 2022.
[21] Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Harold Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localiza- tion and vision-language understanding. In NeurlPS, 2022.
[22] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang. Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-based language-image pretraining. In CVPR, pages 16793-16803, 2022.
[23] Ross Girshick. Fast r-cnn. In ICCV, pages 1440-1443 1448, 2015.
[24] Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurIPS, 2015.
[25] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Be- longie. Feature pyramid networks for object detection. In CVPR, pages 936-944 1666944,22 2017.
[26] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask r-cnn. In ICCV, pages 2980-2988, 2017.
[27] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referl(Game: Referring to objects photographs of natural scenes. In EMNLP, pages 787-798, 2014.
[28) Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Modeling context in referring expressions. In ECCV, 2016.
[29] Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. Mattnet: Modular attention network for referring expression comprehension. In CVPR, 2018.
[30] Yanyuan Qiao, Chaorui Deng, and Qi Wu, Referring expression comprehension: A survey of methods and datasets. CoRR, arXiv:2007.00554. 2020.
[31] Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, Daniel Salz, Xi Xiong, Daniel Vlasic, Filip Pavetic, Keran Rong, Tianli Yu, Daniel Keysers, Xiaohua Zhai, and Radu Soricut. Pali-3 vision language models: Smaller, faster, stronger. CoRR, 3rXiv:2310.09199 2023.
12

---

[32] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
[33] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukeuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent A new approach to self-supervised learning. In NeurIPS, 2020.
[34] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pages 9726-9735 2020.
[35] Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CORR,
2018.
[36] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov. and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning, In CVPR. 2022.
[37] Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling sanguage-imag pre-training via masking. CoRR, arXiv:2212.00794, 2022.
[38] Armand Joulin, Laurens Van Der Maaten, Allan Jabri, and Nicolas Vasilache. Learning visual features from large weakly supervised data. ECCV, pages 67-84, 2016.
[39] Ang Li, Allan Jabri, Armand Joulin, and Laurens Van Der Maaten. Learning visual n-grams from web data. In ICCV, pages 4183-4192, 2017.
[40] Karan Desai and Justin Johnson. VirTex: Learning visual representations from textual annota- tions. In CVPR, 2021.
[41] Mert Bulent Sariyildiz, Julien Perez, and Diane Larlus. Learning visual representations with caption annotations. In ECCV, pages 153-170, 2020.
[42] Weicheng Kuo, AJ Piergiovanni, Dahun Kim, Xiyang Luo, Ben Caine, Wei Li, Abhijit Ogale, Luowei Zhou, Andrew Dai, Zhifeng Chen, et al. Mammut: A simple architecture for joint learning for multimodal tasks. CoRR, arXiv:2303.16839, 2023.
[43] Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang. Scaling up vision-language pre-training for image captioning. In CVPR, pages 17980-17989, 2022.
[44] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. In ICLR, 2022.
[45] Jianfeng Wang. Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. GIT: A generative image-to-text transformer for vision and language. CoRR, N.t::2255.14100 2022.
[46] Xueyan Zou, Zi-Yi Yi Dou, Jianwei Yang. Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang. Lu Yuan, Nanyun Peng, Lijuan Wang, Yong Jae Lee, and Jianfeng Gao. Generalized decoding for pixel, image, and language. In CVPR, pages 15116-15127, 2023.
[47] Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang. Yi Tay, Siamak Shakeri, Mostafa Dehghani, Daniel Salz, Mario Lucie, Michael Tschannen, Arsha Nagrani, Hexiang Hu, Mandar Joshi, Bo Pang, Ceslee Montgomery, Paulina Pietrzyk, Marvin Ritter, A. J. Piergiovanni, Matthias Minderer, Filip Pavetic, Austin Waters, Gang Li, Ibrahim Alabdulmohsin, Lucas Beyer, Julien Amelot, Kenton Lee. Andreas Peter Steiner, Yang Li. Daniel Keysers, Anurag Arnab, Yuanzhong Xu, Keran Rong, Alexander Kolesnikov, Mojtaba Seyedhosseini, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. Pali-x: On scaling up multilingual vision and language model. CORR, arXiv:2305. 18565, 2023.
[48] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. CoRR, arXiv:2304.08485, 2023.
13

---

[49] Zi-Yi Dou, Aishwarya Kamath, Zhe Gan, Pengchuan Zhang, Jianfeng Wang, Linjie Li, Zicheng Liu, Ce Liu, Yann LeCun, Nanyun Peng, et al. Coarse-to-fine lasounlanguage pre-training with fusion in the backbone. In NeurlPS, 2022.
[50] Ting Chen, Saurabh Saxena, Lala Li, David J. Fleet, and Geoffrey E. Hinton. Pix2seq: A language modeling framework for object detection. In ICLR, 2022.
[51] Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for a variety of vision tasks. CoRR, arXiv:2311 N..02331..0622,, 2023.
[52] Jiarui Xu, Xingyi Zhou, Shen Yan, Xiuye Gu, Anurag Amab, Chen Sun, Xiaolong Wang, and Cordelia Schmid. Pixel aligned language models. CoRR, arXiv:2312.092277, 2023.
[53] Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang. Yin Cui, and Boqing Gong. VATT: transformers for multimodal seff-supervised learning from raw video, audio and text. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, NeurlPS 2021, pages 24206-24221, 2021.
[54] Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salchi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. MERLOT RESERVE: neural script knowledge through vision and language and sound. In CVPR, pages 16354-16366, 2022.
[55] Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Kevin Qinghong Lin, Satoshi Tsutsui, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. All in one: Exploring unified video-language pre-training. In CVPR, pages 6598-6608 2023.
[56] Justin Johnson, Andrej Karpathy, and Li Fei-Fei. Densecap: Fully convolutional localization networks for dense captioning. In CVPR, 2016.
[57] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Fei-Fei Li. Visual genome: Connecting language and vision using crowdsourced dense image annotations. CoRR, arXiv:1602.07332 2016.
[58] Jialian Wu, Jianfeng Wang, Zhengyuan Yang. Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang. Grit: A generative region-to-text transformer for object understanding, CoRR, arXiv:2212.00280, 2022.
[59] Guojun Yin, Lu Sheng, Bin Liu, Nenghai Yu, Xiaogang Wang, and Jing Shao. Context and attribute grounded dense captioning. In CVPR, 2016.
[60] Debidatta Dwibedi, Vidhi Jain, Jonathan Tompson, Andrew Zisserman, and Yusuf Aytar. Flexcap: Generating rich, localized, and flexible captions in images. CoRR, 12206 2024.
[61] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, pages 5998 -6088, 2017.
[62] Peng Wang, Shijie Wang, Junyang Lin, Shuai Bai, Xiaohuan Zhou, Jingren Zhou, Xinggang Wang, and Chang Zhou. ONE-PÉACE. exploring one general representation model toward unlimited modalities. CoRR. 1IXiv:2305.11172, 2020.
[63] Matthias Minderer, Alexey A. Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arna,, Mostafa Dehghani, Zhuoran Shen, Xiao Wang, Xiaohua Zhai, Thomas Kipf, and Neil Houlsby. Simple open-vocabulary object detection. In ECCV, volume 13670, pages 728-755, 2022.
[64] Matthias Minderer, Alexey A. Gritsenko, and Neil Houlsby. Scaling open opcabuocabulary object detection. CORR, arXiv:2306.09683 2023.
[65] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, arXiv:1910.10682 2019.
14

---

[66] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- 101-110--0 mining discriminative components with random forests. In ECCV, 2014.
[67] Jianxiong Xiao, Krista A. Ehinger, James Hays, Antonio Torralba, and Aude Oliva. SUN database: Exploring large collection of scene categories. IJV,, 119(1):3-22 22, 2016.
[68] Gong Cheng, Junwei Han, and Xiaogiang Lu. Remote sensing image scene classification: Benchmark and state of the art. CoRR, arXiv:1703.001211 2017.
[69] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In CVPR, 2012.
[70] Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. CoRR, arXiv:1504.00325, 2015.
[71] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metries for semantic inference over event descriptions. TACL., 2:67-78, 2014.
[72] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. OCR-VQAL Visual Question Answering by Reading Text Images. In ICDAR, 2019.
[73] Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruy Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding visual question answering. In CVPR. 2017.
[74] Drew A. Hudson and Christopher D. Manning. GQA: new dataset for compositional question answering over real-world images. In CVPR, 2019.
[75] Ronghang Hu, Marcus Rohrbach, and Trevor Darrell. Segmentation from natural language expressions. In ECCV, 2016.
[76] Zhengyuan Yang, Zhe Gan, Jianfeng Wang. Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. Unitab: Unifying text and box outputs for grounded vision-language modeling. ECCV, 2022.
[77] Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang. Zehuan Yuan, Ping Luo, and Huchuan Lu. Universal instance perception object discovery and retrieval. In CVPR, 2023.
[78] Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint 230.:2900.11195, 2023.
[79] Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. In ICLR, 2024.
[80] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: universal image-text representation learning. In ECCV, 2020.
[81] Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu, Large- scale adversarial training for vision-and- languuge representation learning. CoRR, arXiv:2006.06195. 2020.
[82] Aishwarya Kamath, Mannat Singh, Yann LeCun, Ishan Misra, Gabriel Synnaeve, and Nicolas Carion. 2021.
detection for end-to-end multi-modal understanding. In ICCV,
[83] Li Muchen and Sigal Leonid. Referring transformer: A one-step approach to multi-task visual grounding. In NewrIPS, 2021.
[84] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2021.
15

---

[85] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising seuuencc-to-sqqueeee pre-training for natural language generation, translation, and comprehension. In ACI., pages 7871-7880, 2020.
[86] Jia Ning, Chen Li, Zheng Zhang. Chunyu Wang, Zigang Geng, Qi Dai, Kun He, and Han Hu. All in tokens: Unifying output space of visual tasks via soft token. In ICCV, 2023.
[87] Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy, Improved image captioning via policy gradient optimization of spider. In ICCV. IEEE, 2017.
[88] André Susano Pinto, Alexander Kolesnikov, Yuge Shi, Lucas Beyer, and Xiaohua Zhai. Tuning computer vision models with task rewards. In JCML, 2023.
[89] Manoj Acharya, Kushal Kafle, and Christopher Kanan. Tallyqa: Answering complex counting questions. In AAAI, 2019.
[90] Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, pages 8317-8326, 2019.
[91] Ali Furkan Biten, Rubèn Tito, Andrés Mafla, Lluis Gomez, Marçal Rusiñol,, C.V. Jawahar, Ernest Valveny, and Dimosthenis Karatzas. Scene text visual question answering. In ICCV, pages 4290-4300, 2019.
[92] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In CVPR, 2019.
[93] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In ICCV, October 2019.
[94] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In CVPR, pages 7262-7272 2021.
[95] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ADE20K dataset. IJCV, 127:302-321, 2019.
[96] Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. VaTeX: A large-scale, high-quality multilingual dataset for aileo-and-language research. In ICCV, October 2019.
[97] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In ACM Multimedia, 2017.
[98] Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov. Big vision. https: com/
reaaeerrreerrcchbiig.iiiioooo 2022.
[99] Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov. Better plain vit baselines for imagenet- 1k. CoRR, arXiv:2205.01550, 2022.
[100] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A massively multilingual pre-trained text-to-text transformer. In NAACL-HLI, 2021.
[101] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. IJCV, 2020.
[102] Jiang Liu, Hui Ding, Zhaowei Cai, Yuting Zhang, Ravi Kumar Satzoda, Vijay Mahadevan, and R. Manmatha. Polyformer: Referring image segmentation as sequential polygon generation. In CVPR, pages 18653-18663, 2023.
16

---

[103] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In CVPR, 2016.
[104] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, June 2016.
[105] Estehan Real, Jonathon Shlens, Stefano Mazzocchi, Xin Pan, and Vincent Vanhoucke. Youtube- boundingboxes: A large CoRR, arXiv:1702.00822. 2017.
human-annotated data set for object detection in video.
[106] David Fouhey, Wei-cheng Kuo, Alexei A. Efros, and Jitendra Malik. From lifestyle vlogs to everyday interactions. In CVPR, June 2018.
[107] Ramakrishna Vedantam, C Lawrence Zitmick, and Devi Parikh. Cider: Consensus- Consensus-based image description evaluation. CVPR, 2015.
[108] Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model region-of of-iteeresttttt arXiv preprint ar11v:2307.0360,, 2023.
[109] Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M. Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S. Khan. Glamm: Pixel grounding large multimodal model. In CVPR, 2024.
17

---

A Details on transfer results and ablations
A.1 Transfer downstream tasks
In this study, we introduce three methodologies to evaluate the performance of the pretrained LocCa across a variety of downstream tasks. The first methodology focuses on assessing the efficacy and generalization capability of the standalone pretrained visual encoder. In this setup, the visual encoder is kept frozen, and we employ two main strategies: (i) Drawing inspiration from Lit-Decoder [17], train multi-task decoder from scratch for all downstream tasks. This approach is particularly suited for holistic image understanding and video-related tasks due to the broad range of tasks, allowing for simplified evaluation process with single decoder; and (ii) Aligning with the recent advancements 10 31 (2 that showcase the exceptional results achieved by integrating a pretrained visual encoder with large language model (100) across multitude of downstream tasks, we adopt the training strategy outlined in P-LI-3 31,, substituting their visual encoder with our
LocCa pretrained encoder.
The second methodology entails assessing the performance of the full LocCa model by fine-tuning on downstream tasks using minimal learning rate and weight decay. Given the high cost associated with fine-tuning the entire model, we selectively evaluate its adaptability on just two tasks: image captioning for holistic image understanding and referring expression comprehension for location- aware understanding.
The third methodology involves combining the pretrained LocCa visual encoder with a task-specific pretrained decoder for selected tasks, such as object detection. This approach designed to leverage the strengths of LocCa's 's visual encoder in particular scenarios, notably in contexts requiring fine- grained object-centric understanding.
A.2 LiT-Decoder setup for Referring Expression Comprehension and holistic image understanding tasks
For the evaluation of different frozen vision backbones on classification, captioning, and VQA with the protocol from (17] (LiT-Decoder) rely the from [17 in the data mixing strategy).
from [1] (which only differ
We also apply this protocol to Referring Expression Comprehension to compare the encoders only (without pretrained decoders), with small modifications. Specifically, besides training on a data mix of the different RefCOCO variants, the learning rate from 10 33 to x 10 10-4. Also note that transfer is done at the pretraining resolution of 224 px, unlike the fine-tuning transfers which are done at higher resolution.
reduce the number of decoder layers from 12 to 6, and reduce
A.3 Referring Expression Segmentation
For referring expression segmentation, we extend the REC task by adding a suffix "Mask:" to the text, followed by 16 integer numbers corresponding to segmentation tokens. The segmentation tokens specify the precise shape of the segmentation task within the bounding box that is predicted as part of the REC task. We use the vector-quantized variational auto-encoder (VQ-VAE) from [86], which was trained as in 31 on OpenImages data [101]. The VQ-VAE converts a single channel 64 x 64 pixel mask into sequence of 16 integer values from discrete codebook of 128 tokens. LocCa trained to predict the 16 segmentation tokens, and the VQ-VAE decoder is then used to convert these tokens to 64 x 64 pixels segmentation mask. This mask is then resized to the predicted box coordinates (with bilinear interpolation), before computing the IoU with the ground truth mask.
In Table 7
we showcase the results of employing the full encoder-decoder LocCa model and fine- tuning it for RES task. It achieves competitive results even compared to the state-of-the-ar PaLI-3 model, with considerably smaller model size.
A.4 Vision-Language Models with LocCa
We follow the setup of PaL.-3 31 ablations for all PaI.I33 transfer evaluations. More specifically, we pretrain PaI.-3 stage 1 (frozen image encoder, 2242px resolution) with batch 16k for 13k steps (i.c. 208 million examples). Then we fine-tune PaLI-3 model on each downstream task separately,
18

---

Table 7: mloU on RefCOCO segmentation results when fine-tuning the full LocCa model, and comparison with models from the literature.
RefCOCO
RefCOCO+
RefCOCOg
MODEL
val
testA
testB
val
testA
testB
val-u
test-u
PaI.I33
77.33
72.53
72.72
PolyFormer, 102
76.94
78.49
74.83
72.15
75.71
66.73
71.15
71.17
LocCaz
76.98
78.25
72.90
71.25
76.52
63.67
69.51
70.44
Table 8: Details of PaLI-3 separately.
31 transfer evaluations, where the model is fine-tuned for each task
PARAM
COCO
VQAv2
OKVQA
TextVQa
ST-VQA
TallyQA
steps (k)
20
10
10
10
10
batch size
256
256
128
128
128
128
learning rate
lc-4
lc-4
3o-5
lc-4
lc-4
3c-5
weight decay
lc-4
5c-5
lc-4
lc-4
1c-4
Ic-4
keeping image encoder frozen and the same 224 resolution. The details of each task is listed in
Table 8
A.5
Object Detection
To validate the effectiveness LoeCa compared with image-text pretrained baselines on object detection, employ VIT-B/16 encoder and pretrain all these models 9B examples with 2x4 Subsequently, we integrate the pretrained visual encoder with a 6-layer 0bjects365 pretrained decoder, adapting the combined model for the COCO detection task. This adaptation processes inputs at a 640 px and is designed to predict up to 25 bounding boxes, using a batch size of 256. First we fine-tune the model for 10k steps with learning rate of 10-1, employing a standard auto-regressive loss. This is followed by an additional fine-tuning phase focused on maximizing the mAP reward for 20k steps with learning rate of 10-6
WebLI subset for
A.6 Zero-shot transfer
Following locked-image tuning 336 we freeze the pre-trained vision encoder and train text encoder contrastively to perform zero-shot classification and zero-shot retrieval on downstream tasks. We take the L/14 vision encoder from a LocCa, model, and then train text encoder from scratch to pair with the frozen L/14 model. Input image is resized to resolution 2242px. px. The model trained for 3B seen examples, with standard 10 learning rate and 10 weight decay. Beyond that, also attach randomly initiated MAP head to the L/14 vision encoder and finetune the MAP head following the CapPa baseline 14
A.7 Semantic segmentation ADE20k
We follow the setup of Segmenter 94 which fine-tunes the vision encoder jointly with a linear head predicting the semantic label for every patch, followed by upsampling. The input image resolution is set to 512 x 512, and we apply random resizing with aspect ratio in the range [0.5, 2.0], photometric jitter, and random horizontal flipping. We train for 160k steps with batch 16 using Adam with learning rate 3 x 10 and decoupled weight decay of 0,01. To accommodate variable inference resolution for evaluation, we apply our model sliding- window fashion at the training resolution.
bliding-window
A.8 Experiments on video understanding
Even though LocCa has never seen any videos during pretraining, its training recipe results in an improvement upon CapPa in many video-related tasks, including captioning, QA, and classification. Here, we follow the setup used in PaLI-3 (3I), in which we use the image encoder to process each
19

---

Table 9: Video evaluation results with LocCa-pretrained vision encoder. We use CIDEr 107 (C) and
BLEU-4 (B) for captioning. matching aceuracy (A) for QA and YTBB, and the Jackard index (J) for VLOGS. LocCa improves upon CapPa, particularly when using the larger ViT-L/16 encoder.
Captioning
QA
Classification
MODEL
MSRVTT
VATEX
MSRVTT-QA
MSVD-QA A
YTBB
VLOGGS
C
C
A
CapPa/B
47.4+.3
41.8+.3
49.7+.2
28.9
37.7
45.6+.9
94.6+.1
52.7
LocCa/B
47.6
41.82.0
49.7+.2
29.1
37.9
44.32.4
94.2
54.0
CapPa/L
55.61.3
47.11.1
64.01.2
36.21.1
39.91.0
50.01.3
94.8
58.01.3
LocCa/L
55.1..2
47.32.4
65.0
36.8
40.5
50.9
94.7...
58 58.6
frame separately and concatenate all resulting tokens. Then, we tune a LiT decoder [17] by freezing the pretrained image encoder while training two-layer text decoder from scratch with resolution 224 on a mixture of six video tasks, each of which is given an equal weight during training; i.c. the same number of examples 5K steps, using a maximum text length of 64 tokens and batch size 256. We use label smoothing of 0.1 [103], learning rate 10-3 with weight decay 10-4, and cosine learning rate schedule with 10% warm-up. Encoder ViT-B/16 is pretrained on 3B examples while ViT-L/16 is pretrained on 9B examples.
from each dataset. We use C4 tokenizer [65]. The model is trained for
The six datasets are: (1) MSRVTT T04 a collection of video clips annotated with captions, (2) VATEX (96), another collection with 10 captions each, (3) MSRVTT-QA 197), which contains QA pairs generated from video descriptions using an automated tool, (4) MSVD-QA [97], another collection of QA pairs, (5) YTBB [I05], video classification dataset containing 23 classes, and (6) VLOGS [166]. a multi-label classification containing 23 classes Of the six datasets, VLOGS is location-related, since the task involves tracking the movement of hand. Indeed, while LocCA leads to favorable improvements overall over CapPa, especially using ViT-L/16 encoder, the gain is particularly notable in VLOGS expected. Table summarizes all the results.
Grounded captioning
For the grounded captioning task, we follow the standard setup that generates regional captions based on given bounding box location [58] 108 [09]. We report the grounded captioning results on the Visual Genome dataset and compare them with state-of-the-ar methods.
As shown in Table 10 LocCa (0.6B, without LLM), outperforms GPT4Rol 108 (7B, with LLM)
on both METEOR and CIDEr scores. METEOR evaluates the precision, recall, and alignment of words between the generated and reference captions, while CIDEr assesses the similarity of n-grams between them. When compared with GLaMM [09] (7B, with LLM), LocCa performs better on METEOR but lags CIDEr. This difference can be attributed the relatively simple captions in Visual Genome, which typically consist of only a few words. LocCa uses a simpler decoder (0.3B params only) that closely matches the reference captions in terms of word choice and order, which aligns well with the dataset's simple nature. GLaMM, on the other hand, uses a more complex LLM as decoder, which likely generates more diverse captions that include n-grams that match the reference captions more effectively.
It is important note that the pretrained LocCa encoder complements multimodal LLMs (i.e. as a better alternative option to the CLIP encoder), and we anticipate further performance gains on downstream tasks when combining both.
Table 10: Grounded captioning results on the Visual Genome dataset.
Model
# Param
mAP
METEOR
CIDEr
GRIT
IB
15.5
17.1
142
GPT4RoI
7B
17.4
145.2
GLaMM
7B
19.7
180.5
LocCar
0.6B
34.5
20.7
157.0
20

---

A.9 Experiments on grounded captioning
A.10 Impact of high resolution pretraining and tokenization strategies
In this section, we provides the complete results of the ablation studies on different image resolution pretraining in Fig. and different tokenization strategies for box coordinates in Fig. focusing on the RefCOCO benchmarks.
Ablation on pretrained resolution
pretrain res pretrain 384
92.0
89.50
86.5
94.4
91.4
88.75
85.5
93.8
88.00
84.5
RefCOCO/val
RefCOCO/testB
RefCOCO+/val
87.25
91.5
88.5
80.25
87.5
79.50
85.5
86.5
Re|COCO+//esLA
Re/COCO+/testB
ReICOCOg/val
RefCOCOg/test
88.5
78.75
84.0
85.5
224
384
640
224
384
640
224
384
640
224
384
640
Transfer resolusion
Transfer resolusion
Transfer resolusion
Transfer resolusion
Figure Ablation studies on string vs special tokenization for box coordinates. The image resolution
is 224.
Ablation on String Special tokenization
string tokenization
special tokenization
92.0
89.25
86.5
94.25
91.4
88.50
85.5
90.8
93.50
87.75
84.5
ReICOCO/val
ReICOCO/tesLA
RefCOCO/tesB
ReICOCO+/val
87.0
80
87.5
89.0
79
85.5
86.5
RefCOCO+/tetA
ReCCOCO+/tstB
RefCOCOg/val
RefCOCOg/test
87.5
78
84.0
85.5
224
384
640
224
384
640
224
384
640
224
384
640
Transfer resolusion
Transfer resolusion
Transfer resolusion
Transfer resolusion
Figure 5: Ablation studies on impact of different pretrained image resolutions on string token, we use string tokenization for box coordinates.
21

---

Dowr search Esinglt boars)
L-0.,, tap-kwag
1+1, no tac-k
NAS) contions
Biat RAITER
bowe search bungle brans)
1-0.5, hop lowid
1-1, no tac
beooe (AN9))
boune
DCOMT scarch Isingle Boans
C-0.5, top K--6p
C- no
didi
pauased
loom binglht benare
I-D.5, lags ko-d
L-1, lagak
Cnly DINDS Intthouu
bount search Isinght bearms
L-0.5 tap km60
L+1 bac-
boxic capcono
miist witch
Brocking sus leve Shileny wes
bowit Isingle bears)
L-0.5, tap-kwop
L+1,
Ofly DOICS Withiar lire3
only Harthran HARR)
bower search Lungin brans!
1wO.S, hop lowaz
tw1, toc
RONDS woth noba copaoon
Brcurt borth BANS tepplen
Year bsngle bearn!
Lw0.h, Eap
Iupo
Figure 6: Visualizations of LocCa,,'s 's zero-shot predictions on Visual Genome 57 from left to right we increase noise when sampling. The top rows show all the boxes without con-maximum suppression and the bottom rows show captions for boxes passing the non-maximum suppression filter. We can see that adding more noise increases the variety and amount decoded boxes and at the same time degrades the quality of captions.
22

---

B More Visualization Results
We provide visualizations LocCa,'s zero-shot predictions on Visual Genome (57 have two notable observations: (i) When comparing the top rows to the bottom rows, LocCa excels at identifying foreground regions, yet the box regions exhibit significant overlap without non-me mmammmmmum suppression. This is because only one single object per example is observed during LocCa pretraining. leading to the selection of Rols that are highly random. (ii) From left to right we increase noise during sampling, which increases the variety and quantity of decoded boxes, while simultaneously degrades the quality of captions. explore to explore more diverse Rols, whereas text sampling requires lower noise levels to ensure the generated texts are highly semantic relevant.
Fig. We
hypothesize that box sampling demands higher noise levels to
C Duplicate image ids for RefCOCOs
typically train their models using the combined training sets of RefCOCO, RefCOCO+, and RefCOCOg. However, the splits of these three datasets largely overlap, implying that methods trained more than half of the test images. In Table []] we present the image ratio of validation and test splits of RefCOCOs that overlap with the combined training sets. Adhering the general principle, provide list of image IDs in the training set of RefCOCO benchmarks that are duplicated with validation and test images.
Recent studies in referring expression comprehension 52 76
77 18
Table 11: The image ratio of validation and test splits of RefCOCOs that overlap with the combined training sets.
RefCOCO
RefCOCO+
RefCOCOg
val
testA
testB
val
testA
testB
val-u
test-u
Ratio
61.2%
60.5%
65.1% 1%
61.2%
60.5%
65.1 %
48.8%
48.3%
23

---

154
305
1994
1962
2417
24
7534.
11282
17514,700 7001 00227622111
11324
11774.
7944
12224
7965
12377
7046
12
8065
8300.
12440
8320
124
8429
556
12614
8956
9017
901K
5860
90229
5000
76
9057,
13070
9218
1370
9725.
6407
13836
6747
1400
100994
100
1402
6000
10228
702007
7145
107 0000001188000
14408
10965
10001
14000
149675
16
16
160
166001
16
19273 21750, 24999
180094
18222
1529
19
192
19999
1996
25 23967
88
281770
21501 24842.
24762. 252214
21530
25237,2550
25355
22196
22224
26414
22287
22572
256
220000
22
29
36092
26421
22/0
23014
26438,
23141
25194
26997
27070
23420
25558
14
37474.
24096
37495,
27790
30815
24129 28012
2440X 28038
24084
24406
20009
31333
1720,1133 N1774 X5571,
71363 35796
1883
180014
335321
34233
351377
35330 35201 1537N
35473 35290
35558, 39159,
36-45
86488
16755, 00081 170000
97539
97582
37694
578477 577002 18118 55266
38552, 2222222225
59128
196299
39811
99 401
30, 20346
00755.
41713
41730, 41818 4198%
429041
43609 #5055 45064
44120
44200
40000
4
46322
400TT 50134,, 201000000
477001
473500 S1563, 01655,57000
473911 51700, 51885, 51906
4772
40297
45665
-2002
501152 SE904, 0063325666666
50001
51000 50000
51550 01330,15555 045434544433
5210
52168
32219 5244K
52894,
5222
52725,
$2525
55004
53252
55555
SE570
55558, 53601 55928
00739,55400 14000 578339,
54572,
54806
55236
55232.
55585
55733,
55871, 50005 50507
50172
56634
56616. 50013, 60155
56738 57551
STOM 6145%
57870L 615421
590777,
59133, 62452
500225 62477
S99 65235
6122%, 0122,,63722 00677.66517 64317, 09144T
61372,
01
61.4611
01843
62131
(2223
627577 67356 71
22000020660 67456 71271. 71399, 31714, 11970
63332 633-1 63300
69820 03220,66667 63867.
68430, 10011110000 72396 22565.7000
68459) 00096,687666 687566
64962
09452
65011
11 651
09488
16,666 65760 65541,
69971
09978,
65542,
700994 11501-150
66236,644
66118
665666
20415.
102100 75671
66593,
66637.
N0745.
7075S,
66757, T1099
71229.
67615 67748 681 1818 68397
72731
72995
73000 26744 795877
7O000 77 80521, 84166, 00960
74127 13332-77 80034 8025E
74155. 77374
744
74204 72882
1 24563
14922 74925
22225,7754400 74947. 74945
75500
75021
222211120 758 75881, 75924
76
76
76,001 79785.
77321
78009. 78221, 78277, 78-25,
78482
785
81377
56 78572, 79003,
81 5 0 20
81790, 1000
79315 79441,
79456
70611
83005
75701, 83353, 83448,
SONI, SOCUS, 53666 1
83815 5, 87488
809 18.
84558
1 44994, 84712, 84900 8480%,
85893
(5900 0 85075, 16216
66217.8800 864550
66549
86744 89788
66754
1 87757, 87792, 98200 58609,
88653 91055
38677
SE726 8880S 89005, 59151,
89651
89734 92646
198SS 07604, 97411
SSMOO 93171, 114111.9555 07766..95 9
90577 93786 93885
001200000000 905900
90444, 90869, D0573, 0008300000000
(9999,9611 94618,
(21188994333 94826 95066 95257.
00830
90985. 07785.99188
15 11511899555511
95562, 956766 95812 2,, 9617799777
91056, 911 300 91 288, 01784, 0200999973
(16177.99220 06244,
92165, 022165,22440
961388 11645599000 1
92430 92480
96008
968599 AO 101807, 104670
97685,
97531,
27331,9355 0000059000000 97818, 9
93581,
99040,991 991 19.119999999 99211. 111399223 99:51
99293,999
99727, 100)
62. (00200
100722-
100777,
171410 101-47,
101573 1 1
101627,
1018322 101587, 12.100000
101886
1021 402144,1000 4,24,002200 102208,100 102281, 10231,007000 107500
102667
102667
1 163223 10325) 106148 148 106315,
1051,,104111 103510. 1041011111300
104126,
20426.114444 104246,
10427.1144271 104272
304, 1043IA 107009,
104752 107767
100065
111842, 11 1992, 1 117000011 12222,111 1124990
107846, 10836. 108375, 1084994 108920
111842,
1 1 1054900
11666,10521111 105219, (0119,1155))
105470, 10500 105660, 105714. 115774,110
108920, 109055.
109577
105719, 177199111100
11577)
105999, 106
11227 11 (166IT 119263,
109776. 10
1131122, 116887) 12344,111544
1098123, 1131577
110222
110252
1157444
10GERR, 1
106557.
115676,
115844
106832,
110989
106978 LI 11005,
11 11
106994,
11328
111119,111 114492
1071001
11 1142200
107155, 1112111
1
131453,
115500 1 115554
121558
(
121619, 1230
118543, 1 118697. (
115564 115564
116640
118790, (
1 116949 116661 119129. 902099222999 122099, 1232559,
118827 01227,111000
121958, 1700011 1255000
[19093, 121994,
11607,11666244 116824, 119534, 123476.
1 1168822
19554, 1 000714,11000 122916, 122247,
11702 1177177 1441,11994 120155, 123154
122452,
1174455
120274, 122459
11757A 120335, 134843 120001
1
117721
1E40C1
1111,11000 134055, 122223
100001 121445
1 112700
134060
1 1241 124178
1247711
002566.126000 1256900
1257777
127316
1222161,223000 122388. 127543. 127560
127013,
12263%
122220
127945,
1 10.00. 1281122 151056
128.36, 131074,
1282874
1283837,
188385,188888
128773
132522
120771. 170606. 180116 18016),
132574,
12597411550 132586
1071,11000000
133291,
180231,
13336] 133053, 1763144
130724,
110330, 180518 190872,
1336100,
1339001,
1341006
181121112222 181277 111449, 111587
13459K
191595.
131733 1357700
131816 182188
135132,
13553V 13566
136331, 1994200
1367760
(27211 1 181911911
12772 SO00HH
137918
1381 ,1, 138436,
138007 1880671
179066,
179173, 141662. 1417.,447202 141702. 0777022141333
189322
130859,
1995666
10066881800000 199000,
189774
1
199811,
1999111 143123
1 10000,110 10000,,10200 140291. 140733,
140718,
100300
141101,
141603, 04003,,41662
141827. 00227,141955
141952. 1-2322,
023322.112233
182451.
112339, 1-12669
143000, 10008.111333 187766,
143258, 14770L.
1422177
143334, 143470,
147333 147755
147766
144220
148218
1488077,
14453
148977,
144817)
144872
189253,
144811
199999
14400
145TFX 14588%
(45192
11001 120010 1501011
150177
152954
00777.15055 556614 14, 150701,
153340, 153475, 15359)
10000.111166. 151163 13LTT1
155671 193749
111121,
100K,1112000 198265,
153814,
1538-5,
154244,
151699,
02244,154711
151755,
154713, 154895
077756,119900 151965
019000,122000
152000 111.1155000 155268
154911.1550
012237,112223
|55260
19223%
155379,
1-2222,,
1925011
155540,
132550, 1325500
155860,
155904.
158302
15612% 1
15968h
156600
158701-
15900%
150868
190968
150057.
1569291
1602914 160503,
1571943
1572/1
1 167991
1896181,
157345
157438
152253
157770 1O..RR
161719
1618835
162019, 1651
1621009
1582102
165558 168868
162283
162300,
122000,1123300
162551,
166003.
166210,
02230,166333
162700
166323
160054
166633
166762.
163766
167261.
163394,
07226611677555
163559,
167755,1150 167765.
077755,1177785
163599
077765,119000
168022.
164045
168179,
168366
100
168482
16-38)
16-885
1686-11,
164935
168865.
169179.
165077
169656
189725
1703991
170
170633
170855,
170712
170800
170976
1702200
171251
17 1 (715X)
171411 1724114
1 077533,174477 177472,
175102 175695,
12295 10 100022,12240000
(1672,1177510 177316,
11998,1752284 175281,
172466,
1734566
177817.
1254:00 175523
172555
[77821
173252
177915,
175881,
177917.
1781111111111
176008.
120.2216022222 176229
192. 175
174137
174554
175763,
1747100
176386,
1768744
1747480
126790
178987.
126777
100,11168110 176877 172011,
179011.
179164,
179390,
1
18 179816.179166,
22-112-1
1 172119,
1772284
177311
179969.
1
182297 18332, 1871477
1017 192220,
182285
1 1820000
1900081,
181111 183922
18370077
183094
1 181922 111117555
182
182338
183755
186336
190813, 19133,9196111
190606, 1900E2,
191008, 195422
1875111
191305. 1913277.
187999,
188057,
188184
191561,
188587
191154,
18862)
192301
191711 199283, 196277
188885
9922119,1000
192407, 192476,
189911
| 189230 192524,
18935)
19287S, 193042
110666,11998
189924
193368,
190036,
1931711
190216.11000 192682,
1010111111 190277
1913225
1 194050 19415 10 191169
195439
198551,
191000
1918577 19865
195051,
195524
1958611
196111 1996238
198155 1931199958
196156
196170, 196 196698, 198661666600
199771 197222, 197251,
197722
197925
1975811,
197663
198785
199234
199331
199-053
199888, 19888,1199000 199963, 200010, 200181, 101081,,00000
200377, 200604, 201184. 201322 201636, 201897,
202657,
202755
203036
2030AR 2O10
203175. 206731.
208175,2004555
20-455
1 103994, 20-94,2000000 204053, 20123,,204444 201294,
20074,224555 204159, 204199. 201500,
2008000,000000 205000, 2050000001111 205121, 205202,
205223,
205250, 01550,2055000 205
2057941
205953,
000311200000
2 2075811,
207622
20BEP9 210604, 214523, 217487,
21576,22675111 219722
205724,,2
208449
20
2091820
209EM6
209429
2195577,
209563 2122550000
20 209903 212757.
2101ST 210252 210252
210271210071 210271, 210279, 213426, 217429,
01154882211111 211133
211576,
21166 2116R3, 215 407, 215421, 2155563
2120N7
212/500,
21 11991111666 21GNTM, 299349, 210.219955 95355 220066222225 223977, 225-66 225539,
212641, 210670, 210.221000 210840L 21 2UDDES, SETEIT
212974, 213408, 213419, 2.11,,2..10 212276,274422
215026.
215191, 1151111111211111
21520.1550011111 21524 25
21
215700 21 215908
216391 219248,
217271 220083 223/65
217993
21 217.111...... 2 21 21SUSS, SRNIS,,21100 119099,,210 2181
218114 218579
218124 221799
215994,
221 2 224734, 224755 226R17, 22 111961..200
221 189, 22211 13, 2222000
222676,
2196801
21 220001
221078 225579,
2201 201116222222222 2252700 225641,
225601
2N150 223653
220SE 20050022233300 22379L
221050, 221 1891
223831,
1. 223900 22,056, 22,000, 226400 229422
221187, 22 2212522
221625
220600222111 101166 200072
224092 22001122000 226712 226122226644 226134,
224421 226961. 226960, 227198, 227205.
224838.
COEIS8R2222200 220069, 200.6922250000
2250044
225 225758 226102 2OL..22266 226176, TO, 226324, 220348,
2 220590 TO ZEGGTA
226651.
IENREE ZEENTI, 297273,
CA 2H0AE6, 224244,
200515
SI.EIS,,. 2005E 24400% 233834, 237922,
2300001
227520, 22755-1
251045,
228000, 228119 221122211..
2311057
231992 2A
2EN1333,2900 229041. 1.9961122991 12910229908 RESII,,,22111 229193, 229062, 229415, 233153
2325TT 2363HT,
2. 23 236856
EEZTNO,
2EHIUT 233022, 2100010
2 297191,
2 22 2ERNTS, 237355, 237367,
22124422222000 220033, 2.
230002
2388007,
1300N 296187
23612226144 181118..2228 241521, 241336 2A0.6
220712
237137,
21155552277775 CO00S0A2299000 239773, 2395E
233250,
238007.
238070,
238231, 238232 2322388,288888 238502, 2280222866.. 2411600
218620. 241187
238067.
ZAZIAS, 1 238713, C PA 2A2145,22
243959 249959 246386 2465 24664 246641, 24675,226666
242016 2 ZHITI, 2A1387
248016,210 24
(IIII 199461, 22995 239859, 239590 242455
24221A 2
2AN 241125 240777,
24622H 2O00E 2 247082
2006 242200 244525 247110. 257265.
220225 2201311
204827
242941 240444,228444
234844,
242854, 2411722241155 241875 247979,
24:846, 247600
2415TT 2 200500, 2A1SSOO,,2SSSS 240521,
2419,,22211,,, 24HUSZ SAUUUS,,,AAN
2431N2,
2ABLIS, 115118,,2 PANSST, 248572 2485799224660
243 2A2574,
2A5225
240662 2AA.6E,222200 1212----2000
243717--20006
243008 240289
2 25 SERINT, ZAYNIIA
246542,
25RIGA,
2ANTA 2ANTRA, 24 Z4RESS, 208932, 211982,28995
2SSERO, 1
253251, 1E1221,,2120 2577G7,
248957. SE09ST.24990 248577 250T76
246755, 24025
1ES385, 25T4R0,
200805 240293 250397 254821 258571, 261283,
248KE4, 2545770 258505
07722622277255 251308,200 200000221122 251823, 251868 252022 286659 IICOSS
255201, 25NTUS EENTOS 259120
247271, 211221,,2775 24750OL
25G 2001000210 20, 201521.
25622256211 200 2S9ETS, 201073,
256N6A,
147999924999 211111,,2230 2EEET3, 282122 282708, 232937.
2821 2821366,
246666
257451 25 25 25 25
257915 EE 2002999 200317, SOOEEI,,2000 200000, 1000002000000
25796
25T1E44, 2 258 1659 200008 260068, 750059 264165,
2ESTTT,,,955, 250482, 261096 200.22002200
2600E 200110 20 2001, 200181, 1. 26020000000 260200,
200290000000 2000 363420, 305510 1
200912 260957,
261 0005122245000 2645667
261503, 26472264741 1. IN 202T 26000A
201990
200001
262180L DANEEA 262522 26S9N0, 26620T, 20 COSSE1,,22000 206387 222310, (0002223771 27070002277777 215775. RESSSE,2700 378931
263UR9, 263176 11111166,000 206228, 2062000 2.
2OCIIA 254015 260816
200.0022220000 2625022
2 RESGES, DENTELE JESTIA, 268T60 MA8T76,,600 168664,2000 272235, 27503 SE77EE, 283000
267907 211641,22 2 SYMIEE 274986 114.99922255
Z6N2000 26N200, 268822,
26882S, 2686542666666
268726,
20022222288880 268881,
2000000220000 268199
200015
299 269244
200 269300 272773,
260899
2003S00266500 26ASAA 173197, COEE222276000 276009 1ES.577777940
2608A 269600 2741,221155 80, 20, 274375 224499 (224222 (104442274400
26
2611890
199, EE TH
20000-2-00000 2009t1
271444 2AA 22
2 176711 I, 2M02ST
272058,
2755EA, 27R7472 2ITTIS 285170
ZR5SSS ST7416, 2TIESS, CEEES5,222222 272299 275901, 2I5TE 277507 28251
272716
272729,
2ES7220027777 275917.
2759176, 279377, 28357288555 2 283666 283673 2800555 2801 14.
276354, 200417 20000-20000
210221 2200006
274667 2747770
2774 277479 282310,
117449,27777
28015 284FTS, CERTTTA,,.000 284934 24494,,244444 284964,
201051, 2000444850000 255000,
28
28123T, 283064, 357711,
281464, 2NSUSS, 288559, 288611 10 200660008888
42, 252142 285907, 389140,
2825ON, 285548, 28078322886888 280762,
SHREN DEEIE 20600 286000, 23070D COO0111180000 3999866, 200077, 200000 299311
283479, 286051, 1, 280132, 280190, (00019900000000
170, 285
285214 200221,,2888
285599
20S. 185829,
200111
2 200185,
286460
286460, CAALA 386745, 2O7140, 297249,
29I2ES DHI2G,, 2908354
290ERO, 2ORERO, 2905499 2
2STROE 287702, 587567, 2.7771 117718,201 250009,
2.8994 3989943, 189140,2800 2922 292222222222111
2918977
289267, 2R9AAN,
2I 292386, 201.862222999 292240,, 592235
24

---

10 2000
290002
7906 20 900034
200100
301413, 3072261 2000002
299041
0015770
29012
902353 000000
2
39
299931
3033744
300234
204125
3043116
301400
30
1000000
30000%
25% 241890 119991
1157% AAREEE T T10221
31233 SELEE
312247, 31
312262 S 113390, 31 1154445
1117784 S13386 913941
113077
SESEE B13236 31 BESRYY 10021110000
113360, 215002 1
TH 119644, DOTTEEEEEEE
3190021,
119896
330007,
1.07,--3122 520125, $20127, 0177782000 9202222
LINNEE
321 CEEEEEEEEEEE
BIRIEE 322411, EI200H 5220044
SIEREE TEEEEN 2ONRAS
331216 SEEEEE
S0150 381.504 SECEDA,
SIONEE
$33385 333008 ALNEEL JRREEE, EESDE
B3338E, B97TEE, SERRE 1ESAAE100
13222033000 33090
CO 2 10-2118--7660
FRORR 1 S2928,, 192258,18445 139455, 00045511000000 199539, 7300077 810816 108866669111 221122 1IP 29 OOIIPE TSSSEE BOISOO CO0SS,
342011 SERIIL TAEESA,
$423007 343063 1422996 IER0M S15155, 843301
14539L TOSSEN EASSEEESSS SSSEEES
544190
HARTAL i SRRSE ES070 35/815 350950
S47206
100006--7777 342927 144744 H4522 34882. 848500 84960%
351154
101555555730 S55755, 3513977 251336 00666,331000
159/991 344145 354523 354811 356690 196700 SORCEE
555440
355571
(58114
858224 386E3) 358338
3382299 309405 3891685 165118 1
3555593
1 S50RS 760017 FOOIIT,
340570
140588
000888567711 500769, 394414
360759
1608 100811180 161
162247
12244,666665 163657 016877766000
763699 2661 161190
363252 10000
3 16R14S RA 166711
366070, 11877-1900
163993, 363601, 38RRM,
10611112000 12 20011
300000
568514 S
167792-967983
1003411.11500 168900
165.00
260956
388855,
1 360500 TETTES T 160801 - 220124 320182 000022222200
3200000 10000 172347, 135331 375330
170074 222222
220532 872378
220723
170631
171986.7700 171020 121 161, 57 221.82 33 1--466 571 1866 573-444
371871, 1. 371923, SSGULS 176190 SIN203
1722112
172389
173530 378996, 1780011
172351, 177351,172555
,72558
172600 172744
172871, 333395, 17287113733955
5222000 S8L115
182784 T SOSIIE 1A 180/42 28038
1A24447700 188596, 1
397701 987116 187165
(6999911111 187717 587344
594744
184476
304900 (15500100040
365535, 1854011
188884 188961 18817A 2 181 3812240
188882
580157. 191161
1867344
1
1 392366 M2267, 182684
200000
392180 2218011220 90001144000000 396105 104380,
300055 300135
3930000 12207% 394151 107300 197431
396155
194P9 107560, 197760, 198056 FOR 17,
1 100300000 10000
40ZE 407E22 407264,
398872 198872 402264 4022, 402288.410
305701 17701,00 196014
198901. 1002000
402575
402037
30-7844
402800
1 400560
506035
1E0000. 107212
0172/22,97721
(0000077700000 405531 402200
197217.
400124,
10030 40385S
40H NOITR,, 1 40420E 4071 75,
AOLASS.. 207700004055 404-77
401963,
4044770 407888
40003 201857
0044455500000 401588
40 10866618888811 1186611119990 109166. 10666009-55-
405161 145163,10880 108661.
411104, 411 1111 41 411238, 41 111288
405013, 4051 105136,400 405530,
405587
A11 4465
109-133 409678
41 1778
0OT758.00444444 409754, 409824, 409825, 409918
41 11833 111861
401663, 445700 (05122 (0012211-00000 4861187
411864
0018874400333 406230
410024. 410107, 410165 41 1222 41800 221.000.11110 421595 424694,
00023004006000 406205
400005546635 J0677A 4L0707 41 41308S, 41556,411880
ANGOSS, 11070K 413615 413615,, 4155000 422004 423052.
40TUSS
4 119916, 41 11000022-100
4077 4077446 110669, 415235
10963. 4..3..44000
4074400
4148507
000669,410000 410922
4 115529,
418076 118076, 416117 177.411166666
416286
416355
416651, A 1
4166
4170%%
112231
417366 4208973 424376
4178322 42100 424406
4178650, 121295 424885
118063,, 42204422222
013000441771 4187411111 422255
419019 919066 419028, 119110 419324, 41 119395 419999, 116661 10666.622000
420022 12016 (2002)
42006A
2220294422000 018884442305 427301
422367
01267442258 422583, 422762.
422960.
423250 433341,
423711 421806,
424064
4241 424152
424190 191,
42-278 00--278844435
424844
42716 12716,, 427435 00225,,427555 42755,
425125, 425628 125721-
125723
425825
427628
427654
426383
7277, 427771,
126510.
42TH03
126551,
42785
420557 428783 437897 437547,
426517 129009
1269799000 4294500
4269887,
429520
427
427320,
433921 134201, (0000.11199911 (3:951, 435029,
480563 430731, 430880 430925
01561,440733
43555441111 471178, 431211 I, 2,
435272. 425153,
435477 A3S79
A.T.1445580 435869, 136025 1361 136168 136620, (36/1),
45 706 058.4533133
122088--333 43350A
437683 137080
1369411
437754 437224,
4337 437632,
433796 438015
433.443300 439011
0060055130000 410002
438098 458292, 436331, 138--38--30
100.4388-66 138462, A78478, 438663,
438663
(9900)
439174, 13ST 4438100
439733 443900
439683 44322
439900 0433404330000 446677
446726, 0 0177264448888 4488641 447179. 147179,449
1 04-44,4400000 46-003,, 00003,1100000 000066082284
400012 400012,
00003,,4400000 460089, 490511, 0411.4446144
447-24 H47-24,447
048285, 446364 003544444444
4474544755 7.T.57.4475577 447574
44UR2A, 441205,
447681
445122 245127,6458
441599,
447681, 44793
44 441600
447934 448 AA. -
645323
465 155-00 499469
41239
142000
445462
449780
0555/0
4821
450162. 121116.4445
1001110044445 4434
0001222..5000 450551,
COR.,M3300 ME303, 450707,
446383, A50878. 450914 451119,
446670
451283, 451336, 111337.45
A513A 451333
451-82 148224555
45
45 1800
1818
4S1892
AS2220
451504
452873 1000001555000
455002
453.55335 450633
453639
4 453701, 153906 153906, 153920,1510 1500,1111444 151144
15-246,
454258, A58057
14, 454570 54570-1555 455113
155677-
1 400228 00022884600370 460370, 460442. 461099,
457065 157065,455
457144477991 00019004555255 457225, 45725447766...
457555
457660
461
40200 462067,
457976, 7 A62IN3 4024444462442 462445,
458 24, 458172 459631 158751 163138
402957,
463224,
458761
458827
465417,
157.44493300 159345,
463505.
463953,
404166 0111666666617
584174
4 46510L 1000,455550
1 16524) 165457
466
466242
466523 45885
168177100 4709223
168401 470977 4741 23, 474892 13842244448
468518
465760,
469545, 48445,449555 46959,,4999588 469567.
3577.446655 169655
469825
470004
470072
0000-0007700 0 1000)
470500
470976
47 01966
47 11116, 471111
471065
471065, 4710,4771998 17888,471980
971966,
47254 472654
4722 492
47295 475751
4T200G 422072 4735-49
473352,
4733777
473500
170.447388 1E3399,4770 177266.
474424444 424401
17489, 47475
478144
475007
478164
475037_
475129, 475256 475533, 175651,
4 P
77596 09999,
476347,
176934,
177005
100000..777000 4770
4801
99, 4802400
47392, 47392,475500 0758800773555 477399.
4900012 482200
23079
1 00771,,42777 182775, 18575488000 483708, 485T5T,
000442,,00000 480A8, TH 1651
48 1912
481355,
AS1S 128. ASLE 105330043500
481724
122251115555 205.200.850000 485015,
482526
4823300
482455 485764
A 485002,
482731,
A05.04886600 485632,
000 13910 LUT, 199165, 1911999992220
48 4891
483383
48361 I, 483766. 280016,
483984, 480300,
484108 480713
481206
484208
199226
484307
187009
307 2814 487284,442
484385 487464, A82502
181620,
183306
085766.455
485167
487992, 188073,
187992,88880 1188073,88800 188199
2000
49015
190493
190507,
C 190NST.
491 2912 491072- 4586659 490000
191177 491120
491177
191666 191 FOT, 191739, 1920HO, 192138 4957 T7S, 4908S 8, 498261
15,
4922, 192219
P 19226
19225
00000022220000 02200
000008,922555 492635,
19289)
193001951011 497311 19.51119000 500594, 500603 1012111
193626, 498297,
192933
S9EES, AEL25S,,AAATE 495834, 49ATOC 40500449555 4952E1, 495522 49914 L11, 49915 199155,49999 1993 38, 19666 199682, 1E1138 199966 00066,5005 500057, 509214
0222611,4800 486674 495457 496731,
496754 396839, 497.99947796
498308. 500086, .01
499679 495706 195706,49000 498724 498720, (OROO.9988EE 198854, 1991 22.
000224,,0000 500
SOOESS000000 500561.
100051...50000 SOETTTESSETE 508777 S0G//N0
501710 842, 50201S 50 5021 34, 2AA766996000
502145 48,
NO2300, S024407 AI SALATI SUESS
00273 502726 20
50022,05573 3054T8, 303497, 206226,
SORSONSSSNNN SOE 501541,
3042111-X00555 10.55
SU461O, AONNIO,,,NN 504744,
504769, MOS 2 SOSSE SOSS8A, SOS 585,
5U592A, S00922,,UNAAA
SUGILO,
SOGINNSSGINNG S0GI55 S06162, 182,
CO 500892 50H004 ST3455
30N073, 507215, 107215,5000 307266, 507344 24,3342,5500 5076E2
50 S07761, SOT 16765 507776,
7OT8L, S01815, 5,795, SOSE22,55111 40, 10 508311 508 29, 508 156. 508-60558800
909205
SU9SSS, SOSSSS,SHHEE SUMIS2, S097466 S10342, 51 DARE,, ST3811, S14064, 2.
S10495, 51 314644, S1 312688 S1 1, L. SL
S10572, 5 S10G17, STOSGI, S107T STOVIG, SHORES, 514295, 517451, 517492 520456. 520479, 2204999520 520767,
51 1106666111800 S11146, 51 S11580, ST
$11930, SE5IE2
$11967 196966,,222 112222,51222 102400
313683, 36..3.8888 S16263,
314213, $14230,
10006011000 S14243, 514255
5 1 517685,
S 21 $17805,
31 A..22 11
51550 11S590,5110 5 15102,51 115815, 10........... 1147715119L 1101,,,5990 S200AA,22110 $22191, 200.,,2200 922240,2220 82222
515
516481, 519026, 520100,
11 516791, S 12000,0000 121112250000
517403, 5202772 SIZRAS, $224 $22416,
517869 11869, 517920
17920,51998 $1799, 119862222211
S 521338, $21336, 521. S214
S SINTKS, SI8P1S, IN SI S18966, S1920S. $1945
$21514, 221514,5211 SIIGIN,
$2170221100 52, S2200,2200 S22002, 524155 S24220,
S19616,
422074,
5201999 A22.53222244 $22542, 222342,522288
S2044,,,22244 $2.8166,2222 $2266 S24991
$2244. 222466.,2220
S208R5, S2100 $22771
S22/006
321194,
5229477 122847,5220 SZRRHA, SERANT, 123561, SE3577,,2EEE
SE3STT, SESTO5, 228795, SEMOR SEMASS,
SE2EE1..2222 524340,
12447 1244766
$24520 S2A002.246.
124520, 124662,
SZGIES SOOLESSS6000 5284MA, DO88MM,,2HEN S28851 128851,
S2GIT0, 526521,
526552, 226552,5288 526595 226897,,50 528992, 29999,2,20
SOR000S,00007 525713,
SOORRASRRA 520012 524024
5271 327139, 29, 927207 527267,
S273ES 330629, 2006626550000
S275HT, SE SBORDS, S3
S2779, S1779,, 2212011,5222 531277, 581284
SERWER SETRES
528071 I. 528198, 2HI1S,.S20
528941
528970,
529345, SEROOO RESSIS, 538861, S37807, 1. 537950,
S30132 S33897 188991953400 S340ST S34155 S3GT 46,
S31201 ST4ZZA, 2022A255411 554311
531444 SAL444,,1ISS SEISSO, SH1752, 831834, SEZRSS,
SEE376 S1 S32595 SEDEEE SE3EE2, SIEESS,
SESZTS, S3R2D0 S SESEET,SSS
534140 SSSAALLSER SEASEA, SAASEA,SAEE SHOTIL, SE5DAS, 335101,SSEE
SESZIN, AANEEEAAAAEE SASE29, MAS229,95 SAS234, S SESING S
EE BESSSN, S8S4IN 537770 54135S, 541558, 541452544777 541 SI,
11N666..5NN SESNTA 538
3358T4, SASNTA,,HAAA,, SEGIRA,
S3612T 236127, S38395T
584166, S3G2AA 300EE
MRSSS,,30355 536555 520576, 128316,,3000 316730, 166622056600 526822 316823,52000
520902 116902,52000 S36MED, S3TER L. 531401
587555 537720 117720,5777
196, 538263, S42222 C.2222,552111 542177.
0085537-788500 238244
SERSTA 5E85TA, SEIGRA SESTST, SEHRUS, RESHUS, S391SS 599158, 539475 530633, SOONSS SSO8SI, 599941 S405T0, SOIITT,,SAAE COEETO SAIAEG 541255515555 541255,
54195
$41949,
542442 122442,442 SAZTES, 542799 542936, 142996,52200 542988, S4IZER, SE32EE,,111O S43617, SE1612, SE3790, SEEKEE, SESSSE,,SSEEE SESKIN, 543882, 5 349957, 34415044100 $4416 BAIEE,,44EE 344294 S44831, SERE311,5445
545022
545145, 545213, 545214, 545260, S45325,
546408, CE53225555551 SASREI, S45411, SASTEI, 815799, SESE50, S4614 5460993, 146218, 246242, AAGADR,,5AA
546247,
547055, A471665,555
S47165,
5473155 S47TA
174411.5771 447U74,541 S4H1 R...SAA S48AIG, 448575,545 SA872,,SA 249127.5590
S49366. 189366,59900 A499999550 AOX00, 350129, 350134
34, 5501400 ESIEOO,,SNN 550811, 551554, 550532,
BE 550726 553177
SSIMTZ 351172, 55 5511977 35 222442551110 18133,,.580 SS1472, 251522 451524,5555 531000, 551793, 351514, 351809, 231809,5511 SSINTA, 3521S4 352184, $521999 S52222
$52550 283999955444 SS64ZA, 555700.
553126. 55654
553308,
SERKORSSREE 583498, 553586, S54051, SAAUR1,,SAAA 554168, 2144168,555E. SAAE988SSAAE S54699, 154699 S54TER, S44TER,55AA 554950, 585022, 555120, A55112,5550 555854, 3E5TTI, RESTTL, S55794, 1587944,56616 196162, 556176, 1141766,560 SS6999
S54598,
55600
1369NNSS 356830, 556888, 557602, 557628 157628, S57678, S51044 35700445577 S57746, 157746,5588 158226, 558372,
558804, 558K24, 219822,55500
558800 5591 559152, $59297 589271, 15941 559658
25

---

ST16E AAAA
57166
PENNIO 578525, A
278307
SHISTS, SR150A
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification: We discussed the main contribution and scope of LocCa in abstract and the last paragraph of Introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discussed this in Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
27

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We discussed implementation details and pretraining details in Sec. 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
28

---

Answer: [No]
Justification: LocCa pretrained on a subset of the WebLI dataset, which consists of image-ter texte-texttte pairs and is not publicly accessible. Nonetheless, we provide details on how to prepare a pretraining dataset, applicable to any public image-text datasets such as LAION[9]. The code will be released soon.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
Sellicc/gudes/// /tte///uummsssiiioooooo..ooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips https/cccpullcccuuuccdeoeoo Sttp:///eeeSSumimieeeeee wwweeeuubmssssoonnee seiiisssiiiiiiiioioo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We provided detailed description of the datasets, model size, optimization, and inference details in our Experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: For most experiments we ran 3 seeds and report the mean but the results show marginal variability. For video understanding, we present the mean and standard deviation in Table
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
29

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We discussed this in Sec
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpg////aaaorgg..c//o//all/lttt
Answer: [Yes]
Justification:
Our research conforms with the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: This work belongs to a large body of work lasion-lanugage pretraining from web image/text data. The same potential positive and negative impacts as for prior work apply, and these were discussed in-depth there, see, for example [59].
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
30

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: The main focus of this work is on visual representation learning, rather than vision/language generation. Furthermore, we do not plan release model checkpoints or data.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We rely on widely used academic data sets and referenced all of them according to established standards. Further, we carefully cited the source of our baseline CLIP and compared with it in Sec. 4.1 Baselines.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URI
The name the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
31

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
[NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not rely on crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
32

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "V8mDM9cfeO", "reviewer": "Reviewer_88rJ", "summary": "This paper presents LocCa, a novel visual pretraining paradigm that incorporates location-aware tasks into captioners. Specifically, LocCa employs two tasks, bounding box prediction and location-dependent captioning, conditioned on the image pixel input. This multi-task training helps LocCa significantly outperforms standard captioners on downstream localization tasks while maintains comparable performance on holistic tasks.", "questions": "Several works also investigate he matching of image regions with corresponding text during pretraining. The authors claim that compared to these methods, the proposed LocCa can git rid of complex model architectures and become more computationally efficient. To demonstrate this, more experimental results should be provided to validate this claim, for example, FLOPs, trainable paramters or training time.", "limitations": "Several works also investigate he matching of image regions with corresponding text during pretraining. The authors claim that compared to these methods, the proposed LocCa can git rid of complex model architectures and become more computationally efficient. To demonstrate this, more experimental results should be provided to validate this claim, for example, FLOPs, trainable paramters or training time.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written. The core contributions are clearly presented.\n\n2. The proposed method achieves state-of-the-art performance.", "weaknesses": "Several works also investigate he matching of image regions with corresponding text during pretraining. The authors claim that compared to these methods, the proposed LocCa can git rid of complex model architectures and become more computationally efficient. To demonstrate this, more experimental results should be provided to validate this claim, for example, FLOPs, trainable paramters or training time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Previous work, such as RegionCLIP, has also attempted to align image regions with corresponding textual descriptions using contrastive pre-training schemes. However, these approaches tend to be resource-intensive and less efficient:\n\n- In terms of trainable parameters, RegionCLIP [1] uses a pre-trained CLIP from OpenAI [2] as a teacher model to train a student visual encoder of equivalent size. Besides, it incorporates an ROI-Align module for extracting regional features, which increases the complexity. Conversely, LocCa aligns with the Cap model in simplicity, having a similar number of parameters, i.e., about half the parameters of the RegionCLIP model, taking the teacher model into account. \n\n- In terms of training time, RegionCLIP requires extracting 100 RoIs per image and conducting dual-level contrastive learning—both at the image and region levels. This significantly increases the computational load per example by a factor of 100x|C| (where |C| is the average number of concepts per caption) during the contrastive learning stage. According to [3], with a pre-trained CLIP, training RegionCLIP (with RN50 encoder) takes 6 days using 32 V100 GPUs on 57.6M examples, which is 80k V100-hours or approximately 20k TPUv4-hours per billion examples [4][5]. Conversely, LocCa (B/16 encoder) reuses encoded visual features across 3 tasks, leading to an overall training time per billion examples that is only 1.3 times longer (611 vs 454 TPUv4-hours per billion examples) than that of Cap, which is much faster than RegionCLIP.\n\n| Model      | Params | TPUv4-hrs. |\n|------------|:--------:| :----------: |\n| B/16 Cap     | 192 M  | 454     |\n| B/16 CLIP*  | 197 M  | 444     |\n| B/16 LocCa | 192 M  | 611      |\n| R50 RegionCLIP | -  | 20000  |\n\n\nReference:\n\n[1] Zhong et. al. RegionCLIP: Region-based Language-Image Pretraining.\n\n[2] https://github.com/openai/CLIP\n\n[3] https://github.com/microsoft/RegionCLIP/issues/50\n\n[4] https://lambdalabs.com/blog/nvidia-a100-vs-v100-benchmarks\n\n[5] https://cloud.google.com/blog/products/ai-machine-learning/google-wins-mlperf-benchmarks-with-tpu-v4", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2r11vLNf80", "reviewer": "Reviewer_gCZj", "summary": "This work proposed a location-aware pre-training for vision-language learning. The pre-training contains two tasks: one has location input and output caption/text; the other one has text input and output location of the corresponding object. The model is trained from scratch in ~1B large-scale image-text pairs with object location extracted by an off-the-shelf detection model. The experiments demonstrate the strong performance of the proposed model and the generalizability of the pre-trained vision encoder.", "questions": "Please see the weaknesses.\n\n=====================\n\nThank the author for the answers. Please make sure to add the experiments of Visual7W and LookTwice-QA as promised. I'd like to raise my rating.", "limitations": "Yes, It is discussed in the appendix.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The experiment section covers a wide range of tasks including location-aware tasks and holistic vision-language tasks. The ablation is also comprehensive. It's also impressive that the pre-trained vision backbone can be used as a general visual encoder to replace other CLIP-like models.\n\n2. The model is lightweight. In the era of scaling up models, the proposed model can achieve great performance with just 600+M parameters.", "weaknesses": "1. The key novelty is claimed to be pre-training a vision-language model via two location-aware tasks (Line139): `automatic referring expression` and `grounded captioning`. However, similar location-input-text-output and text-input-location-output tasks have been widely used in the training (mostly fine-tuning though) of location-aware MLLMs recently, eg, Shikra[1], Ferret[2], GLaMM[3]. Moreover, Ferret-v2 [4] also proposed dense referring and dense grounding as a pre-training stage. Those two tasks are quite similar to what this work proposes, and they even involve multiple regions in one round.\n\n2. Missing comparison with many location-aware Multimodal LLM methods.  Location-aware MLLMs load pre-trained LLMs and train on a moderate amount of data (100k-1M) in a few steps (mostly within 3 epochs). They can already show great performance in basic tasks and reasoning tasks. This work instead trains the model from scratch with large-scale data (1B) with longer training time. The authors should analyze the benefits and drawbacks of each line of work.\n\n3. Missing Evaluations on: (1). Location-aware Reasoning tasks, such as Visual-7W[5], LookTwice-QA[6] used in Shikra[1] and Ferret-Bench used in Ferret[2]. (2). Grounded captioning capability is not evaluated, for example, Flickr30k used in GLaMM and Ferret. (3). Comparison with Multimodal LLMs in RefCOCOs and above-mentioned tasks.\n\n4. Training data, WebLI dataset, is not available to the public. Considering the main novelty, location-aware pre-training, largely depends on the scale and quality of pre-training dataset, the work is hard to be reproduced.\n\n\nRefs: \\\n[1] Chen, Keqin, et al. \"Shikra: Unleashing multimodal llm's referential dialogue magic.\" arXiv preprint arXiv:2306.15195 (2023). \\\n[2] You, Haoxuan, et al. \"Ferret: Refer and ground anything anywhere at any granularity.\" ICLR 2024 \\\n[3] Rasheed, Hanoona, et al. \"Glamm: Pixel grounding large multimodal model.\" CVPR. 2024. \\\n[4] Zhang, Haotian, et al. \"Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Models.\" arXiv preprint arXiv:2404.07973 (2024). \\\n[5] Zhu, Yuke, et al. \"Visual7w: Grounded question answering in images.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.\n[6] Mani, Arjun, et al. \"Point and ask: Incorporating pointing into visual question answering.\" arXiv preprint arXiv:2011.13681 (2020).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. **Weaknesses 1&2: Comparison with location-aware MLLMs** \n\n    Thanks for highlighting the connections with existing location-aware MLLMs like Shikra, Ferret, and GLaMM which we will cite in our final manuscript. We discuss the differences between LocCa and the referenced work as follows:\n\n    - Firstly, we appreciate the opportunity to clarify the fundamental objectives and innovations of our work with LocCa, as discussed in Sec. 3.3 of our manuscript. Our primary goal with LocCa is not to enhance MLLMs with location awareness as described in the referenced works. These works typically utilize a pretrained CLIP image encoder and incorporate location-aware tasks during either the fine-tuning stages or as secondary enhancements to already trained models. In contrast, LocCa is a new approach to visual pretraining that inherently integrates location-aware tasks from the very beginning. \n    - Besides, LocCa modifies the standard REF and GCAP tasks to better accommodate this integration, as detailed in Lines 130-137 in the paper. As a complementary, the visual encoder developed through LocCa can be easily incorporated into any MLLM, as evidenced by its application in the PaLI-3 experiments shown in Table 4 in the paper.\n   - Ferret-v2 is a concurrent paper. We also investigated dense referring and dense grounding as proxy tasks during LocCa pretraining. The results were on par with AREF/GCAP, but required a significantly longer decoding length for the decoder, making the process more computationally intensive while providing little or no benefits.\n\n\n2. **Weaknesses 3: Missing Evaluations**\n\n   - (1) Thanks for highlighting the missing tasks on location-aware reasoning. Both Visual7W and LookTwice-QA are relevant. In Visual7W, six types of questions (what, where, when, who, why, and how) assess a model’s visual understanding capabilities, akin to the VQA/GQA tasks we selected in Table 3 in the paper. The seventh question category (which) is similar to the task of REC. Therefore, we actually almost covered all the task types presented in Visual7W. Furthermore, Visual7W is derived from MS-COCO, which also underlies VQAv2 and RefCOCOs for which we present results in the paper.  LookTwice-QA requires situating a local region in the broader context of the image, which is meaningful while not widely used in recent research. **We plan to include both tasks in our future evaluations**. \n\n        Ferret-Bench typically necessitates a LLM for knowledge-based reasoning, which is not the main focus of LocCa. While integrating the pretrained LocCa visual encoder with a LLM (e.g., Vicuna used in Ferret) for complex tasks is feasible, it falls outside the scope of this work.\n\n   - (2) For grounded captioning, we clarify that GRIT/GPT4RoI/GLaMM evaluate on RefCOCO and Visual Genome, whereas Ferret evaluates on Flickr30k. The evaluation tasks differ between these two studies: the former involves generating regional captions based on a given bounding box location, while Ferret requires the model to generate a caption and provide bounding boxes for all noun phrases within the caption. Our setup aligns more closely with GRIT/GPT4RoI/GLaMM, so we report our grounded captioning results on Visual Genome and make comparisons with these methods. \n\n        As shown in the **Table 1 in the rebuttal PDF file**, the LocCa$_L$ model with only 0.6B parameters (without LLM), outperforms GPT4RoI-13B (with LLM) on both METEOR and CIDEr scores for the Visual Genome grounded captioning task. When compared with GLaMM, it performs better on the METEOR metric but lags on CIDEr. METEOR is an evaluation metric that focuses on the precision, recall, and alignment of words between the generated caption and the reference caption, while CIDEr evaluates the similarity of n-grams between the generated captions and the reference captions. Visual Genome has relatively simple captions, typically consisting of a few words. LocCa adopts a simpler decoder (0.3B params only) that closely matches the reference captions in terms of word choice and order. The simplicity of the decoder might limit its ability to produce complex sentences but enables it to generate concise captions that align well with VG's simple nature. GLaMM, on the other hand, uses a more complex LLM as its decoder (Vicuna-7B, 20x params compared to LocCa's decoder), which likely generates more diverse captions that include n-grams that match the reference captions more effectively. This could explain why LocCa has a higher METEOR score but a lower CIDEr score compared to GLaMM. However, it is important to note that the **pretrained LocCa encoder is complementary to MLLMs (i.e. as a better alternative option to the CLIP encoder)**, and we expect further performance gains on downstream tasks when combining both.\n                \n   - (3) As shown in the **Table 2 in the rebuttal PDF file**, LocCa$_G$ achieves higher performance than MLLM (e.g. Shikra, Ferret) on RefCOCOs even with a small decoder rather than a LLM (1.3B vs 13B).\n\n3. **Weaknesses 4: Reproduction**\n\n   - First, in our paper we only adopted a publicly available OWL-ViT CLIP L/14 model [7] to generate detection pseudo annotations. We believe a similar dataset (e.g. based on publicly accessible data) could be reproduced with the technical details provided and the public OWL-ViT models.\n\n   - In the meantime, as in other areas of large-scale science, we believe that publishing as many details as possible on the findings of state-of-the-art systems is beneficial to the community. We will also try our best to provide detailed descriptions on construction of the dataset, and model training methods.\n\n   - Finally, in our ongoing work, we are actively exploring training our models using only publicly accessible data, so that we can make a fast and accessible model. We are making our best effort to release these artifacts.\n\nRefs:\n\n[7] Minderer et. al. Simple Open-Vocabulary Object Detection with Vision Transformers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kr1FhhsQaW", "reviewer": "Reviewer_ris8", "summary": "The paper introduces LocCa, a new visual pretraining approach that incorporates location-specific tasks into image captioning-based vision language models, improving their ability to extract detailed information from images. The authors propose two location-aware tasks, automatic referring expressions (AREF), which predicts bounding box coordinates from captions, and grounded captioning (GCAP), to jointly predict the box coordinates and captions from image, utilizing a multitask encoder-decoder architecture. As a result, LocCa significantly outperforms standard captioning models on localization challenges, achieving top results on datasets such as RefCOCO/+/g, while maintaining similar performance on broader tasks.", "questions": "- I have two questions about the proposed training tasks.\n    - During the pretraining process or after training, is there any trend related to the size of the object in the image? Such analysis can be done among downstream tasks, such as object detection or RefCOCO series datasets.\n    - Can you provide more details on the detection visualization analysis presented in Appendix C? Specifically, what is the connection between observing only one object per example and the substantial bounding box overlap observed without Non-Maximum Suppression (NMS)? Additionally, doesn't the LocCa pretraining process involve comprehension of multiple objects through its three tasks: captioning, AREF, and GCAP? How does this align with the single-object observation?", "limitations": "The paper addresses the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is clear and well-written.\n- The proposed approach demonstrates good performance on a broad number of downstream tasks.\n- The paper provides a wide range of analyses.", "weaknesses": "- The model's training process relies on regional captions and bounding boxes, necessitating the use of pre-existing tools like OWL-ViT to produce detailed object locations for its training data. This approach creates a significant dependency on these off-the-shelf models, potentially limiting the quality of the training data. Assembling a large-scale dataset becomes challenging under these constraints. While the authors have managed to curate an impressive billion-scale dataset, this quantity doesn't necessarily ensure data quality. The reliance on external models for data generation raises questions about the overall integrity and reliability of the training information.\n\n- Typo\n    - Line 134: AREFtask → AREF task\n    - Line 240: dentified → identified", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. **Weaknesses: Data quality and dependency on external models & Typos** \n\n    - To address the concerns about data quality and dependency on external models like OWL-ViT, we would like to emphasize our strategic use of simple filtering techniques to enhance data quality. Specifically, we employ a confidence score threshold of greater than 0.3 for selecting bounding boxes. This threshold is chosen based on its proven effectiveness in balancing accuracy and inclusivity, thus minimizing the inclusion of erroneous or irrelevant data. This filtering approach is similar to the use of the CLIP score in creating high-quality image-text datasets like LAION-400M [1] and YFCC15M [2], and its effectiveness has also been demonstrated in OWL-ViT-2 [3].\n\n    - Thanks for pointing out the typos; we will correct them in the final manuscript.\n\n2. **Q1: Analysis on object size**\n\n    We have analyzed the prediction results related to box size in the MSCOCO-DET dataset. The table below presents the mean Average Precision (mAP) for different box sizes, using the widely accepted classifications: large (>96$^2$ pixels), medium (32$^2$ - 96$^2$ pixels), and small (<32$^2$ pixels). We found that larger boxes consistently achieve higher mAP, a trend echoed in other object detection studies such as Pix2Seq [4] and SSD [5]. This pattern holds across various visual backbones, including CLIP, Cap, CapPa, and LocCa. The higher mAP for larger boxes can be attributed to the more detailed visual features they provide, making them easier for models to detect.\n\n\n    | Model        | mAP   | mAP-small | mAP-medium | mAP-large |\n    |-----------------|:------------:|:-----------:|:------------:|:-----------:|\n    | CLIP  | 40.46 | 19.11     | 44.46      | 62.79     |\n    | Cap          | 39.00 | 18.31     | 42.56      | 60.55     |\n    | CapPa        | 39.48 | 18.32     | 43.21      | 59.81     |\n    | LocCa        | 47.73 | 26.88     | 52.80      | 67.64     |\n\n\n3. **Q2: Visualization details**\n\n    During the LocCa pretraining process, each example specifically sampled one box-text pair for the AREF and GCAP tasks, individually (we also explored using multiple box-text pairs for each task during pre-training, it did not enhance performance and instead increased computational costs). We hypothesize that through multiple epochs of training, LocCa can observe all candidate box-text pairs in the training set.\n\n    When deploying the pretrained model for zero-shot predictions with the GCAP task, it executes multiple decodings for a single input image using beam search, as described in Section 4.3. By setting the beam search to a beam size of 1 (single beam) and a low temperature, the model consistently selects the most confident bounding box regions. While this sampling strategy yields high-confidence predictions, it also results in substantial bounding box overlaps without the application of Non-Maximum Suppression (NMS).\n\nReference:\n\n[1] Schuhmann et. al. LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs.\n\n[2] https://huggingface.co/datasets/mehdidc/yfcc15m\n\n[3] Minderer et.al. Scaling Open-Vocabulary Object Detection. \n\n[4] Chen et al. Pix2seq: A Language Modeling Framework for Object Detection.\n\n[5] Liu et al. SSD: Single Shot MultiBox Detector.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4O2cLMDJTm", "reviewer": "Reviewer_HN67", "summary": "This paper explored integrating region captions into caption-based language-image pertaining. Specifically, in addition to the autoregressive text modeling on global captions, the proposed LocCa also predicts constructed strings of \"{bounding box}-{region caption}\" and \"{region caption}-{bounding box}\". Although the method is conceptually simple, the author benchmarked LocCa in a variety of tasks and showed that it could achieve strong performance.", "questions": "N/A", "limitations": "N/A", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The methodology is straightforward and elegant. The proposed LocCa unifies global and region-level caption generation and doesn't require additional complex architectural modifications.\n\n- The benchmarking of downstream evaluation is quite comprehensive and the performance is strong.", "weaknesses": "1. **Objectives of LocCa**. Overall, the paper is well-written and easy to follow. However, the presentation on LocCa objectives in section 3.2 is a bit confusing starting from line 131. The \"dual-faceted\" loss is hard to interpret for first-time readers. If I understand correctly, it might mean that LocCa calculates loss on both box $b$ and region caption $c$. The author may consider rewriting this part to improve readability. Also, citing references regarding \"contrasting with traditional approaches\" could help readers understand the novelty. Additionally, how to construct a batch with three tasks should also be mentioned somewhere in section 3.\n\n1. **Box predictions in GCap**. In the grounded captioning (GCap) task, conditioned on the image, the model needs to first predict a bounding box, and then predict the caption. The task of predicting a box from the image looks like an object proposal generation task, but the supervision here is only one box per sample. However, it's not clear whether introducing this supervision of object proposal generation is desired by the authors since there are no relevant discussions.\n\n1. **Ablation of pretraining tasks**. Table 6 is very important for this paper. The authors ablated AREF and GCAP tasks respectively and showed that it leads to performance drops. To make the paper more complete, the authors may consider adding the ablation of only removing the box part and keeping the prediction to the region captions. This will help demonstrate the value of location awareness. Also, ablating the global caption and keeping AREF and/or GCAP may also be interesting.\n\n1. **String-based box representation**. Compared to special tokenizations for box coordinates, whether the string-based method require more tokens to represent one bounding box? Whether it lead to less efficient decoding? Also, the hyperparameter of box coordinates resolution also plays an important role here and needs to be discussed and studied with ablations. The authors may see the discussion and explorations in the following papers for reference on bounding box representations:\n    - [1] Shikra: Unleashing Multimodal LLM’s Referential Dialogue Magic\n    - [2] Kosmos-2: Grounding Multimodal Large Language Models to the World\n    - [3] RemoteCLIP: A Vision Language Foundation Model for Remote Sensing", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. **Weaknesses 1: Objectives of LocCa** \n\n    Thanks for pointing out the \"dual-faceted\" loss. The understanding is correct, and we will revise this section for clarity. By \"traditional approaches,\" we are referring to methods like OFA [1] and Florence-2 [2], which we will appropriately cite in the final manuscript. Additionally, we have explained how to construct a batch with three tasks in Lines 163-166, and we will include more details in Section 3 as suggested.\n\n2. **Weaknesses 2: Box predictions in GCap**\n\n    The introduction of object proposal generation supervision in the GCap task is indeed an intentional design choice. This approach is conceptually similar to the first RoI prediction in the Pix2Seq model [3]. Our hypothesis is that providing partial location information (such as the coordinates of the upper-left corner of an object) enables the model to predict the complete bounding box coordinates. This assumption justifies the introduction of additional supervision as a reasonable enhancement to the model’s learning process.\n\n    As detailed in Section 4.3 and Appendix C, one of the additional benefits of this design is the flexibility it offers during the prediction phase. For instance, we can input a specific coordinate to prompt the model to generate a corresponding regional caption. Alternatively, we can simply input the task prefix ('GCAP') and use beam search to allow the model to output various detection results. This flexibility enriches the model’s utility by accommodating different levels of input specificity, thereby enhancing its practical applicability in diverse scenarios.\n\n3. **Weaknesses 3: Ablation of pretraining tasks**\n\n    - We understand your intention and appreciate the suggestion regarding the ablation of the bounding box component. However, the integration of spatial and textual information is central to the LocCa model's design. Removing the box component would impede the model’s ability to learn about spatial coordinates, which are crucial for its zero-shot prediction capabilities and understanding the spatial context of objects. The LocCa framework is specifically developed to link region box locations with captions to enhance cross-modal associations. Thus, omitting the bounding box data would significantly diminish the model’s ability to comprehensively understand visual scenes. Therefore, we have focused our evaluations on scenarios that maintain the full scope of the model’s cross-modal learning functionality.\n    - We totally agree it would be interesting to see the impact of the global caption (Cap) task. The experimental setup is similar to that in Table 6 in the paper, but we used a larger decoder with 12 layers. Comparing Exp1 and Exp2 in **Table 3 in the rebuttal PDF file**, we observed a consistent performance drop without the Cap task on both RefCOCOs and holistic tasks. The drop was more pronounced on holistic tasks like COCO captioning and image classification, suggesting that the Cap task is beneficial for holistic image representation learning.\n\n4. **Weaknesses 4: String-based box representation**\n\n   - String-based tokenization requires more tokens than special tokenization, making it less efficient. For instance, using the c4 tokenizer [4], '498' requires two tokens for string representation but only one token with special tokenization.\n\n    - As shown in the **Table 3 in the rebuttal PDF file**, by comparing Exp 3,1,4, we did not observe a noticeable difference in performance across various box coordinate resolutions (224, 500, 1000) with image resolution of 224, indicating that LocCa remains robust despite changes in coordinate resolution. We set the coordinate resolution to 500 to adapt to different image resolutions (224 -> 384 -> 640), as shown in Figure 3. We believe that further lowering the coordinate resolution (such as 32 that much less than 224) could degrade the model’s accuracy, as each coordinate might then correspond to multiple pixels, leading to potential mapping errors.\n\n    - Thanks for providing Shikra, Kosmos-2, and RemoteCLIP as references for bounding box representations. We compared these with LocCa and noted distinct application scenarios. Shikra explored both string and special tokenization methods, ultimately opting for string tokenization, which we also employed. Shikra uses a LLM and a frozen visual encoder,  making it difficult to train a new coordinate vocabulary that aligns with the pretrained VL representation. In contrast, LocCa trains the entire model from scratch, allowing for easier adaptation. This explains the significant performance gain of around 2% on the RefCOCO+/g with string tokens in Shikra, but the gap in LocCa is much smaller (<0.3%). Additionally, Kosmos-2 introduces location tokens for each image grid rather than the coordinates. This allows a bounding box to be represented by just two tokens, enhancing efficiency. However, we believe these tokens are too coarse, potentially impairing the model’s ability to accurately learn and represent small objects. RemoteCLIP suggests converting the box coordinates into text strings and concatenating them with other textual data before encoding them with the text encoder. It avoids using complex region-text alignment (like RegionCLIP). However, without explicitly constructing the correspondence between regional visual features and box coordinates, achieving fine-grained region-wise alignment may be challenging.\n\nReference:\n\n[1] Wang et. al. OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework.\n\n[2] Xiao et. al. Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks.\n\n[3] Chen et al. Pix2seq: A Language Modeling Framework for Object Detection.\n\n[4] https://github.com/mosaicml/streaming/", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the valuable advice and generally positive feedback from all reviewers. Specifically, Reviewers 88rJ, ris8, and HN67 find our paper clear and well-written. Reviewer HN67 considers our methodology straightforward and elegant. Reviewers gCZj, ris8, and HN67 acknowledge that our experiment encompasses a wide range of downstream tasks. All reviewers agree that our performance is strong with a lightweight model.

We addressed the reviewers' concerns in a separate section. To provide detailed experimental results, we have included a PDF file containing three tables. Please refer to this PDF file, where the relevant information is indicated as **Table x in the rebuttal PDF file**.

Meta-review: This paper presents a novel pretraining framework for vision foundation models. To enhance  location awareness of a model, the proposed method incorporates two location-related tasks, bounding box prediction and location-dependent captioning, in addition to the standard holistic captioning.   
The reviewers appreciated the strong performance of the method on downstream tasks, extensive experiments, and good writing and presentation of the paper. Meanwhile, they also raised some concerns. Some major ones include; comparison and positioning of the proposed method with respect to other location-aware MLLMs, lack of evaluation on some downstream tasks, insufficient ablation studies, lack of evidence that the prosed method is more efficient than previous location-based pretraining  methods, etc.   
During the discussion phase, the authors addressed the above concerns carefully with many new experimental results as evidence. The reviewers have been generally satisfied with the authors rebuttal, and two of them decided to upgrade the scores, resulting in unanimous recommendation for acceptance of the paper. The AC agrees that the reviewers’ concerns have been properly addressed and cleared, and concludes that the paper is above the bar for acceptance. The AC does encourage the authors to take in the new results, analysis and discussion in the camera ready version as promised.
Decision: Accept (poster)