TIPS: Text-Image Pretraining with Spatial awareness
Authors: Kevis-kokitsi Maninis
Kaifeng Chen
Soham Ghosh
Arjun Karpur
Koert Chen
Ye Xia
Bingyi Cao
Daniel Salz
Guangxing Han
Jan Dlabal
Dan Gnanapragasam
Mojtaba Seyedhosseini
Howard Zhou
Andre Araujo
Venue: ICLR 2025 Poster (2025)
Keywords: image representations
image-text
vision-language
dense understanding
computer vision
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Vision+language learning with spatial awareness for off-the-shelf dense and global understanding tasks.

Abstract: While image-text representation learning has become very popular in recent years, existing models tend to lack spatial awareness and have limited direct applicability for dense understanding tasks.
For this reason, self-supervised image-only pretraining is still the go-to method for many dense vision applications (e.g. depth estimation, semantic segmentation), despite the lack of explicit supervisory signals.
In this paper, we close this gap between image-text and self-supervised learning, by proposing a novel  general-purpose image-text model, which can be effectively used off the shelf for dense and global vision tasks.
Our method, which we refer to as Text-Image Pretraining with Spatial awareness (TIPS), leverages two simple and effective insights.
First, on textual supervision: we reveal that replacing noisy web image captions by synthetically generated textual descriptions boosts dense understanding performance significantly, due to a much richer signal for learning spatially aware representations.
We propose an adapted training method that combines noisy and synthetic captions, resulting in improvements across both dense and global understanding tasks.
Second, on the learning technique: we propose to combine contrastive image-text learning with self-supervised masked image modeling, to encourage spatial coherence, unlocking substantial enhancements for downstream applications.
Building on these two ideas, we scale our model using the transformer architecture, trained on a curated set of public images.
Our experiments are conducted on $8$ tasks involving $16$ datasets in total, demonstrating strong off-the-shelf performance on both dense and global understanding, for several image-only and image-text tasks.
Code and models are released at https://github.com/google-deepmind/tips .

Published as a conference paper at ICLR 2025
TIPS: TEXT-IMAGE PRETRAINING WITH SPATIAL AWARENESS
Kevis-Kokitsi Maninis"
Kaifeng Chen"
Soham Ghosh
Arjun Karpur*
Koert Chen
Ye Xia
Bingyi Cao
Daniel Salz
Guangxing Han Jan Dlabal Howard Zhou André Araujo
Dan Gnanapragasam
Mojtaba Seyedhosseini
Google DeepMind
ABSTRACT
While image-text representation learning has become very popular in recent years, existing models tend to lack spatial awareness and have limited direct applicability for dense understanding tasks. For this reason, self-supprvised image-only pre- training is still the go-to method for many dense vision applications (e.g. depth estimation, semantic segmentation), despite the lack of explicit supervisory signals. In this paper, we close this gap between image-text and self-supervised learning, by proposing novel general- purepose image-text model, which can be effectively used off the shelf for dense and global vision tasks. Our method, which we refer Text-Image Pretraining with Spatial awareness (TIPS), leverages two sim- ple and effective insights. First, on textual supervision: we reveal that replacing noisy web image captions by synthetically generated textual descriptions boosts dense understanding performance significantly, due to much richer signal for learning spatially aware representations. We propose an adapted training method that combines noisy and synthetic captions, resulting in improvements across both dense and global understanding tasks. Second, on the learning technique: we propose to combine contrastive image-text learning with self-supervised masked image modeling, to encourage spatial coherence, unlocking substantial enhance- ments for downstream applications. Building these two ideas, we scale model using the transformer architecture, trained on curated set of public images. Our experiments conducted 8 tasks involving 16 datasets in total, demon- strating strong off-the-shelf performance on both dense and global understanding. for several image-only and image-text tasks. Code and models are released https: wwg.tu......../ggg com/ google le deeeeddeeeeeeiiieeeeeeeeeeeeeeee
INTRODUCTION
The quest for effective image representations has permeated much of the research work in computer vision over the past two decades: starting with hand-crafted techniques such SIFT (Lowe, 2004) and HOG (Dalal & Triggs, 2005), then moving into the deep learning era with supervised (Krizhevsky al., 2012; He al., 2016; Dosovitskiy al., 2021), weakly-supervisee (Radford et al., 2021; Mahajan et al., 2018) or self-superviser (Chen et al., 2020; Caron et al., 2021; He et al., 2020) techniques. Most computer vision tasks critically depend on capable image encodings, and for this reason the holy grail in image representation learning is generic model that can be used off the shelf for a variety of downstream tasks.
One of the most promising directions for such representation learning research is on leveraging noisy textual supervision, which is abundant on the web, as introduced by CLIP (Radford et al., 202 1) and ALIGN (Jia et al., 2021). Recent methods (Zhai et al., 2023; Sun et al., 2023) have also made progress on imat learning. However, these techniques have for the most part not been successful at dense image prediction tasks, such depth estimation semantic segmentation. On the other hand,
Authors contributed equally.
'Soham Ghosh is now with Mistral Al.
Correspondence: (kmani n s, franci schen, andrearaujo
@googgle.......ooo..oo @erogg com

---

Published as a conference paper at ICLR 2025
TIPS
Depth prediction
Image
Encoder
Segmentation
Two cats laying on a big bed
Image-text retrieval
One cat playing with small toy
Text Encoder
A photo of tabby cat
Image zero-shot classification
A photo Siamese cat
Embedding space
Figure 1:
We introduce TIPS: Text-Image Pretraining with Spatial awareness. TIPS general- purpose image-text encoder model, which can be effectively used for dense global understanding, vision-only or Vision-languag tasks.
self-supervised learning techniques (Caron al., 2021; Zhou et al., 2022), although lacking semantic signals to guide the training, can enforce feature consistency between distorted images or spatially adjacent patches, and result in effective pretraining techniques for dense image understanding (Oquab et al., 2024). These methods, however, do not use any text for training, and are by nature limited to vision-only tasks. In this work, build on top of both the image-text and self-supervised learning paradigms, to develop general-purpose image representations which can be used for a variety of image-only and image-text tasks. We build strong representations for dense prediction tasks, such as segmentation and depth estimation, and global prediction tasks that reason about the image as whole, such as image classification and image-text retrieval. We illustrate our method in Fig.
We address the limitations of image-text learning, which hinder its applicability to dense spatial understanding tasks, with two simple and effective ideas that improve weak supervision and incen- tivize image features to become spatially coherent: (1) Enhancing the textual labels via automated generation of image captions, leveraging recent multimodal generative model (Beyer et al., 2024). Such synthetic image captions tend to describe the visual contents more comprehensively than the noisy captions mined from the web, by capturing all the objects in the scene and their spatial rela- tionships, being rich supervision signal for dense understanding. However, at the same time, noisy web captions often contain fine aiee-graind details which can be helpful for global understanding tasks (as discussed in Fig. 3). Thus, devise effective method train model with both noisy and synthetic captions, with separate image-text contrastive losses for them, to achieve strong performance on both dense and global tasks. (2) Encouraging the learned image features be spatially coherent, inspired by lessons from the self-supervised literature. We incorporate self-distillation and masked image modeling (MIM) into the image-text learning framework, with carefully designed adaptations, resulting in substantial improvements for many applications, especially the dense ones.
Finally, we build on these ideas to scale Vision Transformer (Dosovitskiy al., 2021) with text aligmment on training dataset of 117M public images, leveraging their noisy web captions and synthetically-geet generrete ones. Our method showcases spatial understanding and textual alignment in the same model, essentially combining the strengths of the image-text and self-supervised literature. refer to our method as Text-Image Pretraining with Spatial awareness (TIPS), and thoroughly evaluate it across many downstream tasks. Specifically, we demonstrate that TIPS achieves strong and competitive performance off-the-shelf across computer vision tasks involving 16 datasets in total, comprising image-only image-text evaluations, for dense or image- predictions. We hope that our findings will inspire the community towards the development of next-generation image representations, enable multimodal and spatially grounded applications.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
General-purpose image representation models have been proposed for computer vision tasks, gen- erally leveraging self-supervised or weakly-supervised learning. Recent self-supervised techniques include DINO (Caron et al., 2021). 1), MAE (He et al., 2022), iBOT (Zhou et al., 2022), I-JEPA (Assran al., 2023) and the scaled-up DINOv2 (Oquab al., 2024), which employs large curated dataset. Our work differs from self-supervised approaches by learning with readily-available and public textual captions, which makes the model more capable as it can handle language inputs. Weakly-supervised learning of image representations generally leverages noisy textual captions, with early examples coming from Joulin al. (2016); Mahajan et al. (2018). Modern approaches include CLiP (Radford et al., 2021), ALIGN (Jia et al., 2021), COCA (Yu et al., 2022), OpenCIIP (Cherti et al., 2023), SigLIP (Zhai et al., 2023), Florence (Yuan et al., 2021), VL. (Chen et al., 2024) and the EVA series (Fang et al., 2023; 2024; Sun et al., 2023). Different from these image-text techniques, we design TIPS to offer frozen pretrained image features that are directly useful to a broad range of downstream vision tasks, without model fine-tuning. Generally, existing image-text models do not focus much on downstream dense prediction tasks with frozen features. Additionally, our technique differs from these by enhancing the core contrastive training common to all of these methods to obtain spatially-coherent representations, via improved captions and loss functions.
Image-text learning for dense understanding tasks. While the
existing image-
text learning approaches lead to powerful representations, they have not demonstrated clear benefits for dense image prediction tasks. As consequence, today the models learned with self-supervised techniques are preferred for these cases: for example, the recent DepthAnything (Yang et al., 2024) model is built on top of the self-supervised DINOv2 features, even though weakly-supervised image backbones are widely available. Similarly, Tong et al. (2024) have demonstrated shortcomings of CLIP-style models and incorporated DINOv2 to enhance the visual grounding of multimodal models. However, recent work has adapted image-text learning for dense prediction, e.g. for open-vocabulary detection (Kim et al., 2023; Minderer et al., 2022; Rao et al., 2022) and segmentation (Mukhoti et al., 2023; Wu et al., 2024; Wysoczanska et al., 2024). SLIP (Mu et al., 2021) combines an adapted SimCLR (Chen et al., 2020) self-supervised objective with CLIP for classification tasks. Closer to work, MaskCLIP (Dong et al., 2023) leverages masked image modeling with contrastive learning, and the very recent SILC method (Naeem et al., 2024) combines contrastive image-text training with self-distillation. Different from our goals, most of these methods are specifically tailored towards improving vision-language tasks that require spatial understanding and do not aim to learn a general- purpose vision encoder. Besides, most of these do not incorporate dense objectives during pretraining, and may require additional fine-tuning stages or dense supervision, which can be costly. MaskCLIP and SILCC propose to incorporate self-supervised losses into image-text training. but only employ either masked image modeling or self-distillation, respectively; in contrast, our TIPS technique goes beyond to combine both, which we show to boost performance significantly for dense image prediction. FLIP (Li et al., 2023) proposed to combine contrastive learning with masking, but without any reconstruction loss, aiming only at efficient language-image training. Altogether, our approach demonstrates for the first time a way to learn image-text models whose vision representations rival those of self-superrised approaches in dense image understanding tasks.
Synthetic data for image representation learning. One of our contributions is to show the power of synthetic textual captions for image representation learning, in particular for dense prediction. Previous work has explored the training of visual representation models with synthetic data, especially with synthetic images (Ren & Lee, 2018; Tian et al., 2023; Sariyildiz et al., 2023), which may be generated based LLM captions (Tian et al., 2024; Hammoud et al., 2024). CapsFusion (Yu et al., 2024) leverages synthetic captions to improve large multimodal models for generative text applications. More similar to our work, VeCLIP (Lai et al., 2024) and LaCLIP (Fan et al., 2023) generate synthetic captions for contrastive image-text training. However, their methods are only applied to global image understanding tasks such as image retrieval and classification, and there is no consideration related to dense prediction. In contrast, we reveal for the first time the power of synthetic captions to improve spatial understanding in image-text models. Additionally, we propose a new learning method to combine noisy web captions with synthetic descriptions, by introducing an additional vision transformer class token to better leverage synthetic descriptions, boosting performance in many tasks. Our technique goes beyond the sampling multi-text caption combination strategies proposed in LaCLIP, enabling more flexible learning with two image-level tokens which focus on different characteristics.
3

---

Published as conference paper at ICLR 2025
Masking
Loss
Contrastive Loss
Self-Distillation Loss
Patch Takens
Local Crops
Patch Tokens
Text Encoder
Image Encoder Teacher
Image Encoder Student
EMA
Synthetic capt.: kitchen with table, chairs, and painting."
Masked Patches
Local Crops
Web caption: "the westin spa puerto vallarta suite"
Captioner
Image Caption
Input Image
Figure 2: Block diagram of TIPS. From bottom to top: given an input image, we produce masked and cropped augmentations, along with synthetic descriptive captions from captioner model. They are fed into the text and image encoders, along with the noisy web caption, and the output tokens used in the losses. The contrastive loss makes use of the two captions, aligning them TIPS also employs self-distillation applied to the local crops and a masked image modeling loss applied to dense patch tokens, which encourage spatially-awar aware and discriminative image representations.
two (CIS) tokens obtained from image encoder.
TIPS
Our goal is create general-purpose image representation model, with text alignment, which can be used off-the-shelf for dense and global vision tasks. While image-text contrastive techniques (Radford et al., 2021; Jia et al., 2021) can effectively model global image information, they tend underperform for dense understanding tasks, where self-supervised models are the method of choice today (Oquab al., 2024). To bridge this gap, we propose Text-Image Pretraining with Spatial awareness (TIPS), illustrated in Fig. 2, which leverages enhanced weak supervision via synthetic image captions, as well as self-supervised masked modeling, improving image feature quality significantly, for both dense and global understanding.
Problem setup. Given a collection of image-text pairs {(4,, Te)), where Ti a noisy textual caption for image Ig. we aim to learn model which encodes images into dense and global embeddings that are useful to a variety of multimodal tasks. More concretely, we set out to train the function f. mapping image I to set of image embeddings {e" C1, ea. eN }. where e is the global embedding representation of the entire image and [e,) are patch embeddings corresponding to different image regions. The text associated with the images meaningful joint embedding space, leading to useful image features. We build on top of the standard CLIP method (Radford al., 2021), which learns text encoder g. mapping T to its embedding e by pushing e and e' close for corresponding images and captions, and far otherwise. CLIP uses cross-entropy loss with softmax normalization of cosine similarities, referred as InfoNCE (van den Oord et al., 2018), which we denote LCLIP. p. In this work, we model f as Vision Transformer (ViT) (Dosovitskiy et al., 2021) and obtain the image embeddings from the final layer's feature map, with el corresponding to the [CLS] token. The function y is modeled standard transformer (Vaswani al., 2017).
be leveraged train semantically
3.1 ENHANCING WEAK SUPERVISION WITH SYNTHETIC IMAGE CAPTIONS
limitation of standard image-text learning using large-scale web data is the quality the captions, which noisy and may accurately describe images. An example is shown in Fig. (top),
4

---

Published as a conference paper at ICLR 2025
where the words "for sale dealership S30k" not describing the image contents. While this may hinder model learning, the caption still captures the main object ('2007 Cadillac Escalade ) However, deeper issue we commonly observe is that these captions often only mention salient objects, without describing their arrangement in the scene. In other words, the captions usually serve as noisy image-level supervi- sion and generally tend be of limited use for learning spatially-awar representations. This motivates us to inves- tigate automated generation of synthetic captions, which could serve as useful pre-training weak supervision for dense tasks.
Noisy web caption: "2007 Cadillac Escalade for sale dealership $30K
We employ off-the-shelf,, publicly available models which can caption images effectively: given the image we gen- erate text T. particular, we leverage captioning models which tend to generate accurate and high-level image de- scriptions - am example is given in Fig. 3 (bottom). Note the use of the preposition "in front of", which indicates the spatial arrangement of the scene, the description of the background ("building") and the color of the object ("black"), all combined providing rich signals for dense image representation learning.
Synthetic caption: *A black SUV parked in front of building."
Figure 3: Example web image (CC BY-SA 2.0) with noisy caption (top) and synthetic caption by PaliGemma (Beyer et al., 2024) (bottom).
However, draw of the synthetically generated cap- tions their lack of detailed object semantics. Referring again Fig. 3, the cetthetically-generatee caption misses information of the specific car model (2007 Cadillac Escalade), which can be helpful to learn discriminative representations. For this reason, we propose combine the original and synthetic π captions, to aim for globally discriminative and spatially-aware image features.
Dual image-text embedding. We strive to leverage relevant information from both captions, and thus propose to modify the vision transformer I to learn from them, in an approach we call "dual embedding". We insert additional [ [CLS] token in the model, to be used for learning with the synthetic caption, obtaining an additional global embedding ê At training time, we feed both T and î into the text encoder, to obtain their text embeddings cx and 6 In addition to the LCLIP loss between e and e', we compute fCUI between 69 and é This introduces flexibility in the model learn an object-centric image embedding in e², and a more spatially-awarr image embedding in é Both back-propagate into the dense feature maps to learn improved patch embeddings (e, > At inference time, the model can have access to both types of global image embeddings, and the one to use may be decided based on the downstream task: generally spatially aware tasks will use è while object-centric ones will employ e².
3.2 INTEGRATING SELF-DISTILLAA AND MASKING TO BOOST IMAGE FEATURES
In addition improving training data quality and learning with different types of textual supervision, we propose to incentivize the model to learn spatially-aware representations via dedicated loss functions. We are inspired by recent self-supervised learning techniques, which produce features suitable to dense downstream tasks (Caron et al., 2021; Zhou et al., 2022; Oquab et al., 2024). We incorporate self-distillation and masking losses in our training setup, adapting them work in weakly- suakly--uperiieed image-text learning framework. Building on top of CLIP, we introduce teacher ViT model. ft, to help guide the training process, which processes the full image The teacher's weights are updated by Exponential Moving Average (EMA) of the main (student) ViT, fs = S. as per (He et al., 2020). Two additional loss terms are introduced, as described next.
Self-distillation loss. We create M local crops from the input image J, which are processed by fa. to obtain M local crop embeddings via their [CLS] tokens, {e TM -= During training, we enforce these embeddings to match predictions of the teacher's [CLS] ] token, e9.t which is obtained from a forward pass of through fc. This incentivizes the model to learn representations which are consistent across the local crops and the original (global) image. The embeddings are used to compute prototype scores using an MLP- projection head, on top of which softmax normalization and cross-entropy loss applied:
5

---

Published as a conference paper at ICLR 2025
Ldistill = Σ> m
softmax c)/T.) log(softmax(pp" /r.))
(I)
where b iterates over the images in the batch. p' = P.(e and pT' (e correspond to the teacher's 's and student's prototype scores respectively, which are computed with the teacher and student projections, Pt and Ps. where P. is updated with an EMA of P.- T. and т, correspond to the teacher's and student's temperatures, used for sharpening the scores, and which is updated with an EMA of P to encourage a uniform distribution.
centering variable
Masking loss. We introduce a masked image modeling loss in order encourage the learned patch embeddings to understand their spatial surroundings. The high-level idea is to have the visible patch representations recover the semantics of the masked patches. More concretely, we feed a masked version of I through fe. where the masked patches are replaced by mask tokens, {m, Jn. The encoded mask tokens, {e Ins are then projected to prototype scores and compared to the teacher's corresponding unmasked tokens, (er similarly to Eq. 1:
Lmaak =- --ΣE softmax((Pi,.. - (og//)//g(sf (pT)//T)) b
where, again, b iterates over the batch. P = (P,( and pT" = P'(( correspond respectively to the teacher's 's and student's prototype scores for patch n, which are computed with the teacher and student projections, Pf and Ps, where P% is updated with EMA of P Similarly as before, of and T/ correspond to the teacher's 's and student's temperatures and c' to the centering variable.
The total loss for our method is then: Liotal = (COLiP+ + LCLIP) + aL + BLmask-
Discussion. Our method builds ideas from the weakly and sulf--uper literature, and to the best of our knowledge is the first to demonstrate that simultaneously combining contrastive image-text learning with both self-distillation and masked image modeling leads improvements across many tasks, indicating positive synergies between these objectives. The closest existing techniques are MaskCLIP (Dong et al., 2023) and SILC (Naeem al., 2024), which combined CLIP with either masked image modeling or self-distillation. As we show in experimental ablations, though, combining the masked image loss with eef-distillation substantially improves performance across dense tasks, being critical for downstream applications. We also note some key differences compared to previous methods. Given that we use a CLIP loss, the self-supervised components can simplified, compared to the original formulations in DINO (Caron et al., 202 1) and iBOT (Zhou et al., 2022). A major difference is that we use a single global "crop", instead of two in DINO, iBOT and SILC, substantially comparatively
increasing throughput by 25%. In contrast to many self-supervised methods, simple data augmentations: the local crop is just a random crop of the original image, and the global crop is larger random crop with horizontal flips. This is similar Assran et al. (2023); Moutakanni et al. (2024) who argue that complex augmentations may not be necessary for representation learning. Finally, our masking approach simply random, in contrast to blockwise in iBOT.
3.3 SCALING TIPS
We aim at creating highly-capable and general-purpose model, and for this reason it is critical to scale a large model architecture and training dataset, aiming at enhanced image representations.
Model. The ViT architecture has been shown to scale well to billion- sized models in a variety of tasks (Zhai al., 2022; Oquab et al., 2024; Chen al., 2024). We scale our TIPS model the ViT-g architecture, with patch size 14, and use the SwiGLU (Shazeer, 2020) feed-forward network variant. Similar to Oquab al. (2024), we adapt the embedding dimension to 1536 with 24 heads. This makes our image encoder directly comparable to DINOv2-g, counting BB parameters in total. On the text side, we scale the transformer to 12 layers, with the same embedding dimension and number of heads the image encoder.
Data. We leverage the WebLI dataset (Chen et al., 2023), which a large and noisy web dataset of public images and associated alt-text containing 10B image-test pairs. We filter the dataset in successive rounds in order to enhance its quality for model training, similar to previous work in language (Gunasekar al., 2023; Wenzek et al., 2020) and vision (Oquab al., 2024; Parthasarathy

---

Published as a conference paper at ICLR 2025
et al., 2023). This is critical for model, since it is intended for off-the- use in many downstream applications. First, similar to Schuhmann et al. (2022), we filter the image-text pairs based their contents, by discarding those whose image-text similarities are low, as computed by a pretrained alignment model. Second, we filtered the resulting dataset to only keep pairs with English captions. These two initial steps result in dataset of 1.7B images. Finally, we follow similar curation process as previous work (Oquab et al., 2024; Parthasarathy et al., 2023) and select images that are similar enough those in curated datasets, leveraging pretrained model compute image embeddings: further details on the curated datasets and filtering strategies are reported in Appendix A.3. Note that we also remove near-duplicate images from our dataset they appeared in any of the evaluation datasets used in this paper. This process generates our main curated pretraining dataset, containing 116M image-text pairs in total.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Evaluation datasets and protocols. Our models are evaluated on suite of tasks involving 16 datasets in total, comprising images-only or images-and-text tasks. We assess the quality of the learned representations thoroughly in a wide range of conditions, covering indoor/outdoor scenes and object-centric captures. Note that, in all evaluations, our image-text representations are kept frozen, since our goal is to assess their applicability as off-the-shelff feature extractors. We evaluate dense prediction tasks, 2 holistic global image understanding tasks and 3 multimodal retrieval tasks. We introduce the tasks below, and provide further details about their evaluation protocols in the appendix.
Semantic segmentation is dense task evaluated on PASCAL VOC (Everingham et al., 2010) and ADE20k (Zhou et al., 2017) datasets, using mean Intersection over Union (mloU). We use a simple linear probe setup similar to (Oquab al., 2024), where classes are predicted from the spatial features.
Monocular depth estimation aims to predict the depth value for each pixel on the image. We benchmark depth estimation on the scene-centric NYUv2 (Silberman al., 2012), and the object- centric NAVI (Jampani et al., 2023), and we use the RMSE metric. For NYUv2, we use a linear probe setup similar to (Oquab al.. 2024), where patch tokens are concatenated the global embedding, on top of which linear classifier predicts among 256 quantized depth values. For NAVI, we follow (EI Banani et al., 2024) and probe with the DPT (Ranftl et al., 2021) decoder.
Surface normal estimation is the task of densely predicting the 3D surface normal direction of each pixel, and is also assessed using NYUv2 and NAVI. We use both datasets with the setup of (El Banani et al., 2024), and report angular RMSE.
Image classification is evaluated on the ImageNet-IK 1K dataset (Russakovsky et al., 2015), where we consider K K-Neerest-Neighbor (KNN) and linear probe evaluations on top of the learned features. We report 1 accuracy.
Fine-grained and instance-level retrieval is evaluated leveraging the Universal Embeddings Dataset (UnED) (Ypsilantis et al., 2023), which itself a benchmark combining datasets from 8 domains: food (Food2k dataset, Min et al. (2023)), cars (CARS196 dataset, Krause et al. (2013)), online products (SOP dataset, Song ct al. (2016)), clothing (InShop dataset, Liu et al. (2016)), natural world (iNat dataset, Van Horn et al. (2018)), artworks (Met dataset, Ypsilantis et al. (2021)), landmarks (GLDv2 dataset, Weyand et al. (2020)) and retail products (Rp2k dataset, Peng et al. (2020)). We report the average recall @1 @ 1) over the 8 domains, and per-domain results in the appendix.
Image-to-text (I+T) retrieval is assessed using the Flickr30K (Young et al., 2014), DOCCI (Onoe et al., 2024) and COCO (Chen et al., 2015) datasets, also reporting the R@1 1 metric.
Text-to-image (T-+) retrieval is similarly assessed using Flickr30K, DOCCI and COCO, with the R @1 metric.
Zero-shot classification is conducted on ImageNet-1K 1K by retrieving the class text embedding closest to the each test image's 's embedding, following Radford et al. (2021), and using top-1 accuracy.
Additional training dataset details. As discussed in Sec. 3.3, we use images from a set of curated datasets as queries for mining among large pool of web images. Following DINOv2 (Oquab et al.,
7

---

Published as a conference paper at ICLR 2025
Method
Segmentation (Pascal VOC)
4. Depth (NYUv2)
1 KNN classif. (ImageNot)
1 r--revettieval (Flickr)
TT retrieval (Flickr)
(A)
Baseline
CLIP (noisy captions)
64.4
0.620
76.9
79.1
62.9
(R; CLIP using synthttic captions
PaliGemma captions
74,5
0,544
70.0
79.8
60.1
Both captions, sampled
71.8
0.563
77.0
90.2
75.4
Both captions, multi-text
72.1
0,580
76.9
85.1
73,9
Both captions, dual
73.3
0.588
78.3
88.7
77.1
(C) Improved loss functionss, with noisy cuptions
CLIP + self-dist
70.3
0.589
79.1
81.5
CLIP sell-dist MIM
75.9
0.511
79.0
82.6
67.6
Ours:
combining improvve captions from (B) and losses from (C)
CLIP + sell-disst + MIM
Both captions, dual
79.0
0.478
78.8
89.2
Table I: Ablations for enhanced captions and improved losses, using the ViT-B hackbone on 5 representative dense, global and image-text tasks. Our final method presented in (D) achieves large gains in all tasks compared the baseline CLIP shown in (A).
2024), we use the training sets of some of our evaluation datasets as the curated queries (details in the appendix). This leads to a web-based training dataset with 116M image-text pairs, which we use for all released models. Additionally, for the ViT-g models, we experiment with adding the training set of the Mapillary SLS dataset (Warburg et al., 2020) as-is to our training set to compensate for the lack of street-level imagery in web images, and in the absense of any alt-text we use the generated synthetic caption for training both CLS tokens. This increases the number of images in our training set to 117M in total. A similar procedure is conducted by DINOv2 for their LVD-142M dataset.
Implementation details. We use global crop at resolution 224 and M = 6 local crops at resolution 98. We train the VIT-B models for 70 epochs at batch size 16k, which takes 4 days on 256 TPUv3 chips. For the ViT-g model we train for 15 epochs at batch size 16k, which takes 2 days on 512 TPUv5 chips, and results in our low-res model (TIPS-g/14 LR). For our high-res variant (TIPS-g/14 HR), we run an additional finetuning stage with global crops at resolution 448 and local crops at resolution 140, for 0.1 epochs batch size 4k. We use only random resize crops and horizontal flips as image augmentations. Models marked with the [rel] prefix correspond to those which are released publicly, whose training set does not include Mapillary SLS. See the appendix for more details.
Captioner model. We leverage the recent PaliGemma (Beyer et al., 2024) model for image caption- ing. Specifically, we use the version fine-tuned on COCO, with the 224 image size version used for the core pretraining run and the 448 version for the short high-resolution fine-tuning stage.
Compared techniques. We strive to provide a large number of comparisons against recent work. For each existing model family, we compare against the largest instantiation up to ViT sizes "g" or "G", at about 1.8B parameters or less in the image encoder. We benchmark TIPS against a wide range of methods, from the self-supervised, weakly-supervised and supervised literature. All methods are used off- off-the-shelf, with frozen weights, for fair comparisons. As sulf-supee methods, we compare against DINO (Caron et al., 2021), MAE (He et al., 2022), iBOT (Zhou al., 2022) and DINOv2 (Oquab et al., 2024). As weakly-supervisee methods, we compare against CLIP (Radford al., 2021), OpenCLIP (Cherti al., 2023), SigLIP (Zhai al., 2023), MaskCLII (Dong et al., 2023), SILC (Naeem et al., 2024) and EVA-CLIP (Sun et al., 2023). As supervised method, we benchmark against the ViT-g trained on JFT-3B, as per (Zhai al., 2022).
4.2 RESULTS
Ablations. We present in Tab. I ablative experiments on 5 different tasks to isolate the effect of the enhanced textual supervision and new losses, where ViT-B backbone is used. The baseline CLIP model with the noisy web captions is presented in (A).
Part (B) of the table ablates the contribution of enhanced textual supervision. Simply replacing the web captions by PaliGemma-generaate ones improves segmentation by 10.1 percentage points and reduces depth RMSE by 0.076, which are big positive gains. This shows the potential of synthetic captions for dense understanding with image-text models. However, at the same time global tasks show significant regressions, with KNN classification loss of 6.9 points. But the CLIP performance can be improved in all tasks by combining the web and synthetic captions: using our dual embedding
8

---

Published as a conference paper at ICLR 2025
Method
Segmentation
1 Depth
Notmals
Fine-grained retrieval (UnED)
1 ImugeNet classif.
PASCAL
ADE20k
NYUv2
NAVI
NYUv2
NAVI
KNN
lin
DINO-B
66.4
31.8
0.555
28.4
28.8
77.4
80.1
MAE-H/14
67.6
33.3
0.517
49.4
76.6
LBOTTL/11
44.6
0.417
24.5
72.9
82.3
JBT
70.7
37.5
0.605
0.096
24.4
26.7
59.5
85.1
87.4
DINOv2-g/14
49.0
0.344
20.5
83.5
CLIP-L
74.5
39.0
0.553
0.073
24.3
25.5
57,4
79.8
84.3
SigLIP-SO/14
67.8
35.8
0.580
0.074
25.6
25.7
70.8
84.4
86.4
OpenCLIP-G/14
39.3
83.2
TIPS g/14 LR (ours)
82.9
47.8
0.377
0.061
23.0
24.5
71.4
83.6
86.4
TIPS-g/14 (ours)
0.353
0.058
21.9
24.2
68.2
83.3
[rel] TIPS-g/14 LR (ours)
82.0
47.4
0.390
0.063
23.5
24.7
71.5
83.6
86.3
[tel] T1PS-g/14 HR (ours)
83.1
49.4
0.363
0.059
22.3
24.3
68.4
83.2
86.1
Image-only evaluations for dense and global prediction tasks. Experiments using the largest backbone available for each model variant, comparing recent self-supervised and image-texi models against TIPS. Rows that are highlighted refer to image-only models that are very good at dense prediction tasks, but are by nature unable perform text-related tasks. LR (HR) refers to our low-res (high-res) model variant. Models marked with the [rel] prefix refer to those relcased publicly. We also highlight the best and second-best number of each column. TIPS achieves the best or second- performance in out 9 evaluations.
Table 2:
approach, we achieve large gains across the board. We also compare our dual approach against two other caption combination options, inspired by the ones proposed by Fan et al. (2023): "sampled", where either the web or the synthetic caption is chosen at random; or "multi-text", where both captions are matched against the same image embedding. Our dual approach performs better than other caption combinations in 3 out cases and achieves competitive results on the other 2, which indicates its effectiveness.
Part (C) ablates the effect of the sulf-supe losses, using web captions. The addition of self- distillation brings improvements in all tasks. This is setup similar SILC (Naeem et al., 2024): confirm their findings for I-TT and T->I retrieval, and additionally show that the self-distillation loss is effective for image-only tasks, notably dense ones. With our additional masked image modeling (MIM) loss, significant improvements are observed in dense tasks, while maintaining high scores in the other tasks: 5.6 points gain in segmentation and 0.078 reduction in depth RMSE.
Part (D) combines the findings of (B) and (C) to deliver very substantial improvements against the baseline CLIP setup in all tasks, notably: 14.6 points gain in segmentation, 0.142 reduction in depth RMSE, 10.1 points gain in I-T retrieval and 14.4 points gain in T-*I retrieval. Additional ablations can be found in the appendix.
Comparisons against existing ceneral-purpond methods are provided in Tables 2 and 3, for tasks involving images only or images and text, respectively, where results for TIPS are provided for the model before ("LR") and after ("HR") high-resolution fine-tuning. The released TIPS models are marked with the "[rel]" prefix. Overall, TIPS achieves strong results, with competitive performance across a wide range of tasks, reaching the best or second-best numbers in 13 out of the 16 reported evaluations. Compared against existing image-text methods, TIPS improves on I and T-II retrieval, while also achieving substantial gains in dense prediction tasks, reaching the level of DINOv2 and surpassing it in some cases. It is interesting to note that while recent image-text models have achieved excellent results in multimodal retrieval or rero-shot classification, those gains do not translate to improved features for dense undertanding, whose performance lags substantially behind TIPS and self-supervised approaches. In particular, even CLIP-L, with much worse performance on image-level prediction tasks, outperforms the recent SigLIP-SO on all 6 dense evaluations. Another recent and much larger image model trained with contrastive learning, Intern ViT-6B (Chen al., 2024), achieves 47.2% on ADE20K, which is much worse than our 1.1B TIPS- g model. In terms of supervised methods, the ViT-g trained on JFT-3B also performs worse on dense tasks than CLIP-L. And even larger ViT-22B (Dehghani et al., 2023), also trained on JFT, achieves only 34.6% in ADE20k on the same setup, as reported by Chen al. (2024). In comparison to self-supervised techniques, TIPS achieves strong results, with numbers comparable to DINOv2 in most cases and surpassing them significantly in segmentation and retrieval, while at the same time enabling multimodal tasks which cannot be performed with self-supervised methods alone. Fig. 4 shows qualitative examples for our dense feature probes. Additionally, we provide results for TIPS models distilled to smaller VIT backbones in the appendix.
Application: Single-image to 3D. Modern large reconstruction models rely on high-quality pre-trained image encoders to produce image tokens for encoder/decoder transformer (Hong

---

Published as a conference paper at ICLR 2025
Method
I-II retrival Flickr
TT-I
ImageNet O-shot
COCO
DOCCI
COCO
Flickr
DOCCI
MaskCI.P-B/14
41.4
70.1
25.5
45.6
44.5
CLIP-L/14
56.3
85.2
44.4
36.5
65.2
40.4
75.5
OpeaCLIP--G/1
67.3
92.9
51.4
79.5
80.1
EVA g/14
70.2
91.0
27.5
$2.0
75.3
28.4
81.2
SILC-G/16
73.2
TIPS-g/14 LR (ours)
73.7
93.0
56.4
58.3
83,2
58.9
79.7
TIPS-g/14 HR (ours)
74.0
93.0
57.2
$9.4
84.5
38.8
79.9
[rel] TIPS-g/14 (ours)
55.6
57.0
79.6
TIPS- g/14 HR (ours)
74.0
93.8
57.0
59.2
83.8
59.4
79.7
Table 3:
Image-text evaluations for multimodal retrieval and zero-shot classification, where TIPS outper- forms others in out of cases. compare solely against ones are not naturally aligned with language. We highlight the best and second-bes number of cach column.
methods, since supervied
RGB image
PCA feat.
M. Depth
Normals
Sem. Segm.
Figure 4: Qualitative dense prediction results. For a given image (first column), we illustrate the principal components of the predicted spatial features (column 2). Depth (column 3) and normals (column 4) are trained NYUD, and for semantic segmentation (last column) used the model trained ADE20K. All dense tasks used the DPT decoder, with a frozen image encoder. More qualitative results can be found in appendix.
al., 2024; Wang al., 2024). For example, LRM (Hong et al., 2024) predicts pa- rameters of neural rendering model from the image features of a single input image. The authors choose the ViT-based DINO encoder over more semantic-aware ones (such CLIP) due to its knowl- edge of structural and texture information necessary for 3D tasks. To better understand model's capabilities for neural 3D reconstruction, we evaluate TIPS the LRM framework and compare DINO-BI B/16 to equivalently- sized TIPS-B/14. We opt to use DINO-B/16 to follow after 3D
Method 1 PSNR
DINO-B/16 21.13
TIPS-B/14 21.75
Table 4: LRM novel view synthesis using original DINO features TIPS features. TIPS yields improvements with model of similar size.
the original paper's implementation. Single-image results on the Objaverse (Deitke et al., 2023) dataset are presented in Tab. 4, showing that TIPS outperforms DINO as an image encoder for large reconstruction models, with enhanced novel view synthesis capabilities (0.62 increase in PSNR). Qualitative results can be found in the appendix.
CONCLUSIONS
We introduce TIPS (Text-Image Pretraining with Spatial awareness), new general-purpose image- text encoder. TIPS can be successfully applied off-the-shelf to a variety of computer vision tasks, enabling dense and image-level prediction, leveraging two simple and effective contributions, First, we employ existing multimodal generative models to produce high-quality synthetic image de- scriptions, which are used improve contrastive learning and boost performance on dense image prediction. We propose dual embedding approach to leverage both synthetic and noisy web captions, unlocking gains across wide range of tasks. Second, we combine contrastive image-texl learning with celf-distillation and masked image modeling, incentivizing the model to learn spatially- -wware representations. These two contributions are complementary and allow us to effectively scale our models to ViT-g architecture trained curated dataset of 117M images. Our comprehensive periments demonstrate strong off-the the--shelf results on tasks comprising 16 datasets in total, enabling wide variety of computer vision applications which involve only images, images and text.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We would like to thank Xiaohua Zhai, Xiao Wang, Lucas Beyer, Alexey Gritsenko, Matthias Minderer, Mohamed El-Banani, Muhammad Ferjad Naeem, Austin Stone, Hagen Soltau, Jonathon Shlens, Abhijit Ogale, Ming-Hsure Hsuan Yang and Huizhong Chen for thoughtful discussions and helpful pointers to datasets, experiments and related work.
REFERENCES
L Alabdulmohsin, X. Zhai, A. Kolesnikov, and L. Beyer. Getting ViT in Shape: Scaling Laws for Compute- Optimal Model Design. In Proc. NewrIPS, 2023. 16
M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y. LeCun, and N. Ballas. Self-Supervised Learning from Images with Joint-Embedding Predictive Architecture. In Proc. ICCV, 2023. 3,6
L. Beyer, A. Steiner, A. Pinto, A. Kolesnikov, X. Wang, D. Salz, M. Neumann, I. Alabdulmohsin, M. Tschannen, E. Bugliarello, T. Unterthiner, D. Keysers, S. Koppula, F. Liu, A. Gryener, A. Gritsenko, N. Houlsby, M. Kumar, K. Rong. J. Eisenschlos,
Kabra, M. Baucr, M. Bošnjak, X. Chen, M. Minderer, P. Voigtlaender, Bica, Balazevic, Puigcerver, P. Papalampidi, O. Henaff, Xiong, Soricut, J. Harmsen, and X. Zhai. PaliGemma: A versatile 3B VLM for transfer. arXiv:2407, 07726, 2024. 2, 5,
M. Caron, H. Touvron, Misra, H. Jegou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging Properties in Self-Supervised Vision Transformers. In Proc. ICCV, 2021. 3, 6,
Chen, S. Korbllith, M. Norouzi, and G. Hinton. Simple Framework for Contrastive Learning of Visual Representations. In Proc. ICMI, 2020. 1,3
X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollar, and C. L. Zitnick. Microsoft COCO Captions: Data Collection and Evaluation Server. arXiv:/504.00325, 2015. 7,
X. Chen, X. Wang, S. Changpinyo, AJ Piergiovanni, P. Padlewski, D. Salz, S. Goodman, A. Gryener, B. Mustafa, L. Beyer, A. Kolesnikov, Puigeerver, N. Ding, Rong, Akbari, G. Mishra, L. Xue, A. Thapliyal, Bradbury, W. Kuo, M. Seyedhosscini, C. Jia, B. Karagol Ayan, C. Riquelme, A. Steiner, A. Angelova, X. Zhai, N. Houlsby, and R. Soricut. PaLI: A Jointly-Scaled Multilingual Language-Image Model. In Proc.
ICLR, 2023. 6
Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang. X. Zhu, L. Lu, B. Li, P. Luo, T. Lu, Y. Qiao, and J. Dai. VL: Scaling up Vision Foundation Models and Aligning for Generic Linual-Lig Tasks. In Proc. CVPR, 2024. 6,9
M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, C. Schuhmann, L., Schmidt, and
J. Jitsev. Reproducible Scaling Laws for Contrastive Language- Image Learning. In Proc. CVPR, 2023. 3, 8
N. Dalal and B. Triggs. Histograms of Oriented Gradients for Human Detection. In Proc. CVPR, 2005. 1
Darcet, M. Oquab, J. Mairal, and Bojanowski. Vision Transformers Need Registers. In Proc. ICLR, 2024. 23
M. Dehghani, J. Djolonga, B. Mustafa, P. Padlewski. J. Heek, J. Gilmer, A. Steiner, M. Caron, R. Geirhos, Al- abdulmohsin, R. Jenatton, Beyer. M. Tschannen, A. Amab, X. Wang. C. Ruiz. M. Minderer, J. Puigeerver, U. Evci, M. Kumar, S. Van Steenkiste, G. Elsayed, A. Mahendran, F. Yu, A. Oliver, F. Huot, J. Bastings, M. Collier, A. Gritsenko, V. Birodkar, C. Vasconcelos, Y. Tay, T. Mensink, A. Kolesnikov, F. Pavetic, D. Tran, T. Kipf, M. Lucic, X. Zhai, D. Keysers, J. Harmsen, and N. Houlsby. Scaling Vision Transformers to 22 Billion Parameters. In Proc. JCML., 2023.
M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kembhavi, and A. Farhadi. Objaverse: A Universe of Annotated 3D Objects. In Proc. CVPR, 2023, 10, 22
X. Dong, J. Bao, Y. Zheng, T. Zhang, D. Chen, H. Yang, M. Zeng, W. Zhang, L. Yuan, D. Chen, F. Wen, and N. Yu. MaskCLIP: Masked Self- Distillitilain Advances Contrastive Image Pretraining. In Proc. CVPR, 2023. 3, 6
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer. G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition Scale. In Proc. ICLR, 2021. 1, 16
M. Fl Banani, A. Raj, K.-K. Maninis, A. Kar, Y. M. Rubinstein, D. Sun, I. Guihas, J. Johnson, and V. V.Jampnni Probing the 3D Awareness of Visual Foundation Models. In Proc. CVPR, 2024. 7, 20
11

---

Published as a conference paper at ICLR 2025
M. Everingham, L. Van Gool, C. Williams, J. Winn, and Zisserman. The Pascal Visual Object Classes (VOC) Challenge. IJC,, 2010. 7
I. Fan, D. Krishnan, P. Isola, D. Katabi, and Y. Tian. Improving CLIP Training with Language Rewrites. In Proc. NeurIPS, 2023. 3,9
Y. Fang, W. Wang, B. Xie, Q. Sun, L. Wu, X. Wang, T. Huang, X. Wang, and Y. Cao. EVA: Exploring the Limits of Masked Visual Representation Learning at Scale. In Proc. CVPR, 2023. 3, 18
Y. Fang, Q Sun, X. Wang. T. Huang. X. Wang, and Y. Cao. EVA-02: A Visual Representation for Neon Genesis. Image and Vision Computing, 2024. 3
S. Gadre, G. Ilharco, A. Fang, J. Hayase, G. Smyrnis, T. Nguyen, R. Marten, M. Wortsman, D. Ghosh, J. Z.Zann, E. Orgad, R. Entezari, G. Daras, S. Pratt, V. Ramanujan, Y. Bitton, K. Marathe, S. Mussmann, R. Vencu, M. Cherti, R. Krishna, P. Koh, O. Saukh, A. Ratner, S. Song. H. Hajishirzi. A. Farhadi, R. Beaumont, Oh, Schmidt. DataComp: Search of the Next Generation of Multimodal Datasets, In Proc. NeurlPS Datasets Benchmarks, 2023. 17
A. Dimakis, J. Jitsev, Y. Carmon, V. Shankar.
Gemini Team Google. Gemini: Family of Highly Capable Multimodal Models. arXiv:3312.11800 2023. 17
J.BB. Grill, E. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, and M. Gheshlaghi Azar. Bootstrap Your Own Latent A New Approach to Self-Supervised Learning. In Proc. NewrIPS, 2020. 18
S. Gunasckar, Y. Zhang, J. Aneja, C. Mendes, A. Del Giomo, S. Gopi, M. Javaheripi, P. Kauffmann, G. de Rosa, O. Saarikivi, A. Salim, S. Shah, H. Bchl, X. Wang, Bubeck, Eldan, A. Kalai, Lec, and Y. Li. Textbooks Are You Need. arXiv:2306. 1644, 2023.
H. Hammoud, H. Itani, F. Pizzati. P. Tor, A. Bibi, and B. Ghanem. SynthCLIP: Are We Ready for Fully Synthetic CLIP Training? axXiv:2402.01832, 2024.
K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. In Proc. CVPR, 2016. 1
K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum Contrast for Unsupervised Visual Representation Learning. In Proc. CVPR, 2020. 5,5
K. He, X. Chen, S. Xie, Y. Li, P. Dollar, and R. Girshick. Masked Autoencoders are Scalable Vision Learners. In Proc. CVPR, 2022. 3, 8
G. Hinton, O. Vinyals, and Dean. Distilling the Knowledge a Neural Networ. Proc. NeurlPS Workshops, 2015. 16
Hong, Zhang, J. Gu, Bi, Zhou, Liu, F. Liu, K. Sunkavalli, Bui, and Tan. LRM: Large Reconstruction Model for Single Image to 3D. In Proc. ICLR, 2024. 9, 10, 22
V. Jampani, K.-K. Maninis, Engelhardt, Brualla, K. Patel, D. Vlasic, V. Ferrari, A. Makadia, Liu, Y. Li, and Zhou. Navi: Category-agnostic image collections with high-quality 3d shape and pose annotations. In Proc. NeurlPS Datasets and Benchmarks, 2023. 7
Karpur, K. Truong, K. Sargent, S. Popow, A. Araujo, R. Martin-
C. Jia. Y. Yang, Y. Xia, Y. Chen, Z. Parekh, H. Pham, Q. Le, Y. Sung. Z. Li, and T. Duerig. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision. In Proc. ICML, 2021. 1, 3,
A. Joulin, L. van der Maaten, A. Jabri, and N. Vasilache. Learning Visual Features from Large Weakly Supervised Data. In Proc. ECCV, 2016. 3
D. Kim, A. Angelova, and W. Kuo. Contrastive Feature Masking Open-Vocabulary Vision Transformer. In Proc. ICCV, 2023. 3
Krause, Stark, Deng, and Fei-Fei. Object Representations for Fine-Grained Categorization. In Proc. ICCV Workshops, 2013. 18
A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet Classification with Deep Convolutional Neural Networks. In Proc. NewrIPS, 2012.
Z. Lai, Н. Zhang, B. Zhang, W. Wu, Н. Bai, A. Timofeev, X. Du. Z. Gan, J. Shan, C. Chuah, Y. Yang. and M. Cao. VcCLIP: Improving CLIP Training via Visual-enriched Captions. In Proc. ECCV, 2024. 3
Y. Li, H. Fan, R. Hu, C. Feichtenhofer, and K. He. Scaling Language-Image Pre-training via Masking, In Proc. CVPR, 2023.
12

---

Published as a conference paper at ICLR 2025
Z. Li, X. Wang. X. Liu, and Jiang. Binsformer: Revisiting Adaptive Bins for Monocular Depth Estimation. IEEE Transactions on Image Processing, 2024, 20
Z. Liu, P. Luo, S. Qiu, X. Wang, and X. Tang. DeepFashion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations. In Proc. CVPR, 2016. 7, 19
D. Lowe. Distinctive Image Features from Inverianvaiaant Keypoints. IJCV, 2004. 1
D. Mahajan, R. Girshick, Ramanathan, K. He, M. Paluri, Y. Li, A. Bharambe, and L. van der Maaten. Exploring the Limits of Weakly Supervised Pretraining. In Proc. ECCV, 2018. 1,3
B. Mildenhall, P. Srinivasan, M. Tancik, J. Barron, R. Ramamoorthi, and R. Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In Proc. ECCV, 2020. 22
W. Min. Wang. Y. Liu, M. Luo, Kang. X. Wei, X. Wei, and S. Jiang, Large Scale Visual Food Recognition. IEEE TPAMI, 2023. 7, 18
M. Minderer. A. Gritsenko, A. Stone, M. Neumann, D. Weissenborn, A. Dosovitskiy, A. Mahendran, A. Arnab,
M. Dehghani. Z. Shen, X. Wang. X. Zhai, T. Kipf, and N. Houlsby. Simple Open- Vocabulary Object Detection with Vision Transformers. In Proc. ECCV, 2022. 3
Moutakanni, M. Oquab, M. Szafraniec, M. Vakalopoulou, and P. Bojanowski. You Don't Need Data- Augmentation in Self- Supe-Superviee Learning. arXiv:2406.09294 2024. 6
N. Mu. A. Kirillov, D. Wagner, and S. Xie. SLIP: Self-supervision meets Language- Language-Image Pre-training. arXw:2111.12750, 2021.
J. Mukhoti, T.-Y. Lin, O. Poursaeed, R. Wang, A. Shah, P. Torr, and S.-N. Lim. Open Vocabulary Semantic Segmentation with Paich Aligned Contrastive Learning, In Proc. CVPR, 2023.
M. F. Naeem, Y. Xian, X. Zhai, L. Hoyer, L. Van Gool, and F. Tombari. SILC: Improving Vision Language Pretraining with Self- Distillation. In Proc. ECCV, 2024. 3, 6, 8.9
Y. Onoe, S. Rane, Z. Berger, Y. Bitton, J. Cho, R. Garg. A. Ku, Z. Parekh, J. Tuset, G. Tanzer, S. Wang. and J. Baldridge. DOCCI: Descriptions of Connected and Contrasting Images. In Proc. ECCV, 2024. 7, 21
M. Oquab, T. Darcet, T. Moutakanni, H. Vo. M. Szafraniec, V. Khalidov. P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, Howes, P. Huang, H. Xu, Sharma, S. Li, W. Galuba, M. Rabbat, M. Assran, N. Ballas, G. Synnaeve, I. Misra, H. Jegou, J. Mairal, Labatut, A. Joulin, and Bojanowski. DINOv2: Learning Robust Visual Features without Supervision. Transactions on Machine Learning Research, 2024. 2, 3, 5, 6, 7, 8, 20
N. Parthasarathy, M. Eslami, J. Carreira, and O. Henaff. Self superviue Vidco Pretraining Yields Human- aligned Visual Representations. Proc. NearlPS, 2023. 6,7
J. Peng. C. Xiao, and Y. Li. RP2K: A Large-Scale Retail Product Dataset forFine-Grained Image Classification. 2rX.v:2006.1200 12634, 2020. 7, 19
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning Transfcrable Visual Models from Natural Language Supervision. In Proc. ICML, 2021. 1, 1,3,4, 4, 7,8, 8, 21
R. Ranfil, A. Bochkovskiy, and V. Koltun, Vision Transformers for Dense Prediction. In Proc. ICCV, 2021. 7, 19, 20
Rao, W. Zhao, G. Chen, Tang, Z. Zhu, G. Huang, J. Zhou, and Lu. DenseCLIIP: Language-Guided Dense Prediction with Cowtext-Aware Prompting. In Proc. CVPR, 2022.
Z. Ren and Y. J. Lee. Cross-domain Seff-supervised Multi-task Feature Learning Using Synthetic Imagery. In Proc. CVPR, 2018.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJC,, 2015. 7
M. B. Sariyildiz, K. Alahari, D. Larlus, and Y. Kalantidis. Fake It Till You Make It: Learning Transferable Representations from Synthetic ImageNet Clones. In Proc. CVPR, 2023. 3
13

---

Published as a conference paper at ICLR 2025
C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, P. Schramowski, S. Kundurthy, K. Crowson, L. Schmidt, R. Kaczmarezyk, and Jitsev. LAION-50 5B: An Open Large-Scale Dataset for Training Next Generation Image-Tex: Models. In Proc.
NeurlPS, 2022. 7
N. Shazeer. GLU Variants Improve Transformer. arXiv:2002.05202 2020. 6
N. Shazeer and M. Stem. Adafactor: Adaptive Learning Rates with Sublinear Memory Cost. In Proc. ICML, 2018. 19
N. Silberman, D. Hoiem, P. Kohli, and R. Fergus. Indoor Segmentation and Support Inference from RGBD Images. In Proc. ECCV, 2012. 7
H. Song, Y. Xiang, S. Jegelka, and S. Savarese. Deep Metric Learning via Lifted Structured Feature Embedding. In Proc. CVPR, 2016. 7,
Q. Sun, Y. Fang, I.. Wu, X. Wang, and Y. Cao. EVA-CLIP: Improved Training Techniques for CLIP at Scale. arXiv:2303. /5389, 2023. 1,3,8
Y. Tian, L. Fan, P. Isola, H. Chang, and D. Krishnan. StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learners, In Pruc. NeurlPS, 2023. 3
Y. Tian, L. Fan, K. Chen, D. Katabi, D. Krishnan, and P. Isola. Learning Vision from Models Rivals Learning Vision from Data. In Proc. CVPR, 2024. 3
S. Tong, Z. Liu, Y. Zhai, Y. Ma, Y. LeCun, and S. Xie. Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs. In Proc. CVPR, 2024. 3
A. van den Oord, Y. Li. and O. Vinyals. Representation Learning with Contrastive Predictive Coding. wwXiv:187..03725 2018.
G. Van Horn, O. Mac Aodha, Y. Song, Y. Cui, C. Sun, A. Shepard. H. Adam, P. Perona, and S. Belongie. The iNaturalist Species Classification and Detection Dataset. In Proc. CVPR, 2018. 7, 19
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polosukhin. Attention All You Need. In Proc. NeurlPS, 2017. 4
Wang, H. Tan, S. Bi, Y. Xu, F. Luan, K. Sunkavalli, W. Wang, Z. Xu, and K. Zhang. PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Prediction. In Proc. ICLR, 2024. 10
Warburg. S. Hauberg, M. López-Antequera, Gargallo, Kuang, and Civera. Mapillary Street-Level Sequences: A Dataset for Lifelong Place Recognition. In Proc. CVPR, 2020.
G. Wenzek, M. Lachaux, A. Conneau, V. Chaudhary, F. Guzman, Joulin, High Quality Monolingual Datasets from Web Crawl Data. In Proc. LREC, 2020. 6
Grave. CCNet: Extracting
Weyand, A. Araujo, B. Cao, and Sim. Google Landmarks Dataset v2 Large-Scale Benchmark for Instance-Level Recognition and Retrieval. In Proc. CVPR, 2020. 7, 19
Wu, W. Zhang. I.. Xu, S. Jin, X. Li, W. Liu, and C. C. Loy. CLIPSelf: Vision Transformer Distills Itself for Open- Vocabulary Dense Prediction. Proc. ICIRR, 2024.
M. Wysoczanska, O. Simeoni, M. Ramamonjisoa, A. Bursue, T. Trzcinski, and P. Perez. CLIP-DINOiser: Teaching CLIP few DINO tricks for vocaablayy semantic segmentation. Proc. ECCV, 2024.
L. Yang. B. Kang, Z. Huang, X. Xu, J. Feng, and H. Zhao. Depth Anything: Unlcashing the Power of Large-Scale Unlabeled Data. In Proc. CVPR, 2024.
P. Young, A. Lai, M. Hodosh, and J. Hockenmaier. From Image Descriptions to Visual Denotations: New Similarity Metrics for Semantic Inference Event Descriptions. TACI, 2014. 7,21
N.-A. Ypsilantis, N. Garcia, G. Han, S. Ibrahimi, N. Van Noord, and G. Tolias. The Met Dataset: Instance-level Recognition for Artworks. In Proc. NeurIPS Datasets and Benchmarks Track, 2021.
19
N.-. Ypsilantis. K. Chen, B. Cao, M. Lipovský. P. Schönbeer,,,,,,,,, G. Makosa, B. Bluntschli, M. Seyedhos- seini, O. Chum, and A. Araujo. Towards Universal Image Embeddings: A Large-Scale Datasct and Challenge for Generic Image Representations. In Pro. ICCV, 2023.
J. Yu, 7. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu. CoCa: Contrastive Captioners Imate-Text Foundation Models, Transactions on Machine Learning Research, 2022. 3
14

---

Published as a conference paper at ICLR 2025
Q. Yu, Q. Sun, X. Zhang, Y. Cui, Zhang, Y. Cao, X. Wang, and J. Liu. CapsFusion: Rethinking Image-Text Data at Scale. In Proc. CVPR, 2024.
L. Yuan, D. Chen, Y. Chen, N. Codella, X. Dai, Gao, H. Hu, X. Huang, B. Li, Li, C. Liu, M. Liu, Z. Liu, Lu. Y. Shi, Wang. Wang, B. Xiao, Xiao. J. Yang, M. Zeng. L. Zhou, and P. Zhang. Florence: New Foundation Model for Computer Vision. arxiv:21 1432, 2021. 3
Zhai, A. Kolesnikov, N. Houlsby, and Beyer. Scaling Vision Transformers. In Proc. CVPR, 2022.
X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid Loss for Language Image Pre- Pre-Training. In Proc. ICCV, 2023, 3,
B. Zhou, H. Zhao, X. Puig. S. Fidler, Barriuso, and A. Torralb.. Scene Parsing Through ADE20K Dataset. In Proc. CVPR, 2017.
J. Zhou, C. Wei, H. Wang, W. Shen, C. Xie, A. Yuille, and T. Kong. Image Bert Pre-training with Online Tokenizer. In Proc. ICLR, 2022. 2, 3,5, 5, 6,
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 ADDITIONAL EXPERIMENTAL RESULTS
Model
Image
Text #
Total Params
[rel] TIPS-S/14 HR
21.6M
[re]] TIPS-B/14 HR
85.7M
109.6M
195.3M
[re]] TIPS- HR
303.2M
183.9M
487.1M
[re]] STPS-
412.4M
448. 3M
860.8M
[re]] TIPS-g/14 LR
1.1B
389. IM
1.5B
[rel] g/14 HR
Table 5: Number of parameters for all released TIPS model variants. We release different models, of 5 different sizes. For S. B, L and model sizes, we use fixed number of 12 layers in the text encoder; for the SO size, the same number of layers in both the image and text encoders.
Method
Segmentation
Depih
Narmals NYUv22
Fine-grained retrieval (UnED)
ImageNet classif.
PASCAL.
ADE20k
NYUv2
NAVI
NAVI
KNN
[eel] TIPS-S/14 HR
80.6
44.5
0.425
0.072
24.3
25,5
$7.7
75.1
77.7
[re]] B/14
0.379
22.9
80.0
bell TIPS-LL14 HR
83.9
49.5
0.372
0.061
22.3
24.4
67.8
82.5
85.2
[re]] TIPS SSSS:5//4410 IR
83.7
49.7
0.362
0.060
22.0
24.4
68.6
83.0
85.7
[re]] TIPS-g/141.R I.R.
82.0
47.4
0.390
0.063
23.5
24.7
71.5
83.6
86.3
[re]] TIPS-g/14 HR
83.1
49.4
0.363
0.059
22.3
24.3
68.4
83.2
86.1
Table 6: Image-only evaluations for all released TIPS model variants, showcasing strong performance in
different tasks across a variety backbone sizes.
Method
I-+T retrivaal
T T retrieval
1 ImageNNtt O-shot
DOCCI
DOCCI
Irell SIPS-S/14 HR
64.8
86.3
42,9
50.0
74.7
44.7
71.5
[te]] TIPS B/14 HR
91.3
50.3
54.7
79.4
50.7
76.3
[re]] TIPS-L/14 HR
73.3
93.6
55.1
58.4
83.5
57.1
79.2
[ell] TIPS-S0/14 HR
73.4
55,9
58.6
58.6
79.1
[tell TIPS-g/14 LR
73.3
93.4
55.6
58.1
82.1
57.0
79.6
[re]] TIPS-g/14 HR
74.0
93.8
57.0
59.2
83.8
59,4
79.7
Table 7: Image-text evaluations for all relcased TIPS model variants, showcasing strong performance in 3 different tasks across variety of backbone sizes
TIPS variants with smaller backbones. Our objective is the development of a high-performing image-text encoder with spatial awareness. While large models trained on extensive datasets are central to this goal, practical deployment often requires smaller models. Directly training these smaller models from scratch proves suboptimal due to capacity limitations. Therefore, we adopt a knowledge distillation strategy (Hinton et al., 2015) as follows: smaller student network is trained to reproduce the outputs of our largest model (TIPS ViT-g), particularly the outputs from the image encoder, by minimizing the difference between the teacher and student outputs for given input set. This is achieved by adapting the TIPS training process: the image encoder teacher, initialized with the TIPS ViT-g parameters, remains frozen throughout the distillation process; image encoder student is learned while using the original masking and local cropping strategies. The text encoder training is unchanged, being initialized from scratch. Additionally, we run a finetuning stage higher input image resolution, as described in Sec. 4.1 for the ViT-g model variant, using the same teacher model, and initializing the student image encoder and text encoder with the weights from the previous training stage.
This training strategy is applied across variety of backbone sizes: VIT-S, VIT-B, VYT-L (Dosovitskiy et al., 2021) 1) and S0-400m (Alabdulmohsin et al., 2023) (we refer to the latter as simply "SO" from here on). For the text encoder scaling, we adopt the same transformer parameterization as the image encoder, but keep the number of layers fixed at 12 (except for the SO variant, which uses its standard number of layers). As a teacher, we use our "[rel] TIPS-g/14 4 HR" model variant. Our preliminary studies demonstrated that this approach yields superior performance compared training models completely from scratch, even when the student a relatively large model like ViT-L or S0-400m.
We present the number of parameters for each of the models in Tab. - all of them are released, available via our project webpage. Tables and present their performance across range of tasks,
16

---

Published as a conference paper at ICLR 2025
involving only images or images and text. Our distillation approach is effective at preserving high performance even for models which are much smaller than the teacher. For example, the VIT-L variant, with only 487M parameters, achieves performance across the board which is quite close to the ViT-g teacher's, with 1.5B parameters (even surpassing it in segmentation). The SO variant, at 861M parameters, surpasses the ViT-g teacher in 5 image-only evaluations and is quite close the teacher's 's performance for image-text ones.
Ablation on synthetic caption versions. To understand in more detail the impact of synthetically- generated descriptions on different tasks, we create different variants to ablate the effect of the compo- sition of the description. "PaliGemma object list" is created by prompting Gemini 1.5 Flash (Gemini Team Google, 2023) to take the original PaliGemma caption and produce list of the objects that are mentioned in it, for example: "A black SUV parked front of a building" becomes "black SUV, building". "PaliGemma main object" is created similarly to the object list version, except that it is prompted to produce only the main object in the caption, for example: "A black SUV parked in front of building" becomes "black SUV". Results are presented in Tab, 8, using a CLIP model with VIT- backbone. First, note how "PaliGemma object list" already provides significant boost over the noisy captions for dense prediction tasks, which indicates that listing the multiple objects in the images, without noisy web terms, helps substantially. On top of this, the full "PaliGemma captions", including descriptions about object spatial arrangements, further improves dense understanding, notably for depth estimation. The results for "PaliGemma main object" show some improvement compared to noisy captions in depth, but not for segmentation. On I-I and T->I retrivall, "PaliGemma captions" also provide significant improvements compared against other PaliGemma caption variants.
Captioning version
Segmentation (PASCAL)
Depth (NYUVv2)
1 KNN classif. (ImageNet)
1 rettivetieval (Flickr)
T retrieval (Flickr)
Noisy captions
64.4
0.620
76.9
79.1
62.9
PaliGemmma captions PaliGemma object list
74.5
0,544
70.0
79.8
73.8
0.575
70.1
66.4
45.0
PaliGemmaa main object
61.9
0.598
67.7
8.7
4.7
Table 8: Synthetic caption ablations for representative evaluations, using a CLIP ViT-B model.
Ablation on dataset versions. We compare our final curated version of WebLI, with 116M image- text pairs in total, to 2 other versions: "raw" (10B unfiltered image-text pairs) and "EN quality- filtered (1.7B image-text pairs filtered with pretrained alignment model and keeping only English- captioned pairs). Results are presented in Tab. 9, using CLIP model with a ViT-B backbone, for a representative set of image evaluations. Our curated dataset helps improve performance across all of these evaluations, while being 1 or 2 orders of magnitude smaller.
Training set
t Segmentation (ADE20K)
Depth (NYUv2)
KNN classif. (ImageNet)
1 Fine-grained retrieval (UnED)
Raw (IOB)
29.1
0.698
68.4
45.8
EN quality-fitered (1.7B)
31.5
0.632
76.2
59.3
Ours curated (116M)
31.6
0.620
76.9
62.9
Table Training set ablations for representative image evaluations, using a CLIP VIT-B model.
Training on DataComp. We provide additional experimental results for the CLIP baseline and for TIPS, when training on the public DataComp dataset (Gadre et al., 2023), and compare against training on WebLI (as per the main paper results). We process the DataComp dataset following the same pipeline which we used for WebLI. Starting from the DataComp-I1 1B version (1.4B samples), which is already filtered based on image-text quality and English captions, we apply our curation process and remove near-duplicate images those in evaluation datasets used in this paper. This leads to DataComp dataset version with 115M image-text pairs, which is close to our WebLI- curated dataset of 116M pairs. We additionally compute PaliGemma synthetic captions for the final DataComp dataset version, which are used to train the complete TIPS method. Results are presented in Tab. 10, using a ViT-B backbone, for a representative set of image evaluations. Overall, training on these two datasets leads to very similar performance, both when training the baseline CLIP method and our TIPS technique. This is not surprising, as both datasets are collected with similar procedures, from the web. Importantly, note that significant performance gains are observed when changing from CLIP to TIPS, when training on both datasets. This indicates that the strong TIPS results can be attributed to the new training method introduced in this work, given the consistent gains.
17

---

Published as a conference paper at ICLR 2025
Method
Segmentation (PASCAL)
Depih (NYUV2)
KNN classif. (ImageNet)
1 11-IT retrieval (Flickr)
rettevviiv (Flickr)
(A) CLP
WebLI
64.4
0.620
76.9
62.9
DataComp
64.3
0.620
76.0
80.4
64.0
(B) TIPS
WebLi
79.0
0.478 0.479
78.8
89.2
77.3
DataComp
79.1
73.4
88,9
74.1
Table 10: Comparison between training sets (WebLI and DataComp), using the ViT-B backbone on representative dense, global and image-text tasks.
Method
Segmentation (PASCAL)
Depth (NYUv2)
1 KNN classif. (ImageNet)
1 TT-+Tretrieve (Flickr)
T retrieval (Flickr)
(A) TIPS complete method TIPS ViT-B (ours)
79.0
0.478
78.8
89.2
77.3
Varying masking ratio
Random, 50%
79.3
0
79.1
90.5
78.0
Random, 25%
78.8
0.533
79.3
90.5
77.9
(C) Varying marking approach
Blockwise, 75%
79.5
0.491
Blockwise, 50%
78.9
0,504
79.0
89.5
77.3
Blockwise, 25%
78.8
0.537
79.4
89.8
77.9
(D) Varying image augmontatiios
BYOL augs local crop
40.1
0.878
78.8
90.6
78.0
BYOL augs local+global crop
79.4
0.490
77.7
88.7
76.4
Resize global crop
79.4
(E) Successive CLIP MIM
CLIP MIM
75.2
0,550
67.2
78.0
65.9
Table 11: Ablations for design choices of self-supervised learning components, using the ViT-B backbone on 5 representative dense, global and image-text tasks.
Ablation on self-supervised learning components. We present ablations on design choices for the self-supervised learning component in Tab. 11, using ViT-B backbone model, for a representative set of image evaluations. The complete TIPS method is presented in row (A). First, we vary the masking approach (random masking or blockwise masking) and masking ratios (from 25% to 75%), as per rows (B) and (C). Note that TIPS employs random masking with 75% ratio. The results show that higher masking ratios tend to benefit dense tasks substantially, while only impacting global classification and image-text retrieval modestly. For example, depth RMSE improves significantly with higher masking ratios, with small impact on ImageNet KNN and Flickr I->T retrieval. As we image-tett model, we find that 75% is good trade-off. Blockwise masking
aim for a spatially- performs a little worse than random masking overall, and for this reason we adopt the simpler random masking approach. Next, we vary the image augmentation approach in row (D). Our method uses only crops and flips, and we compare against the more complex augmentations introduced in BYOL (Grill et al., 2020) (color jitter, gaussian blur, solarization) applied to only the local crops well as to both the global and local crops. We find that additional augmentations for only local crops benefits global and image-text tasks, but are significantly detrimental local tasks, while additional augmentations for both local and global crops are detrimental for all tasks but segmentation. We also consider replacing the random square crop for the global crop with resizing square, but find that it has mixed impact, decreasing depth performance significantly. The modest impact and performance trade-offs of complex image augmentations inform our choice to use simple strategy of only crops and flips. Finally, in row (E) we modify the learning process to use successive CLIP then MIM training, similar to EVA (Fang et al., 2023), instead of our proposed method which leverages contrastive and self-supervised losses simultancously. Compared to the CLIP baseline (Tab. I (A)), the results show improvements in dense tasks when using this strategy. This strategy also outperforms the approach that combines CLIP only with delf-distillation simultaneously for dense tasks (CLIP + dilf-dist in Tab. (C)). However, the combination of CLIP with both self-distillation and MIM simultaneously fares better across the board (CLIP + self-dist + MIM in Tab. (C)).
UnED detailed results can be found in Tab. 12, covering all of the 8 domains in the benchmark: food (Food2k dataset, Min et al. (2023)), cars (CARS196 dataset, Krause et al. (2013)), online products
18

---

Published as a conference paper at ICLR 2025
(SOP dataset, Song et al. (2016)). clothing (InShop dataset, Liu et al. (2016)), natural world (iNat dataset, Van Horn et al. (2018)), artworks (Met dataset, Ypsilantis et al. (2021)), landmarks (GLDv2 dataset, Weyand et al. (2020)) and retail products (Rp2k dataset, Peng et al. (2020)). We compare TIPS against the main competitors, outperforning SigI.P on average by 0.6 percentage point and DINOv2 with very significant 9.2 points gain. TIPS achieves the best score in domains, SigL.P in another 3 domains and DINOv2 in 2 domains.
Mothod
Food2k
CARS196
InShop
GLDv2
Rp2k
CLIP-L/14
46.7
89.8
63.5
61.1
12.4
30.8
36.7
58.3
57.4
JFT-3B-g/14
56.7
96.9
64.0
75.4
28.2
27.2
69.6
59.5
DINOv2-g/14
54.4
83.2
56.3
35.8
82.3
60.4
55,3
69.5
62.2
Sigl.P--CM/ S0/14
60.6
97.5
76.7
76.1
76.5
58.9
44.6
76.2
70.8
TIPS-p/I4 LR (ours)
63.6
94.9
23.8
83.3
55.6
41.9
74.4
71.4
TIPS-g/14 14 HR (ours)
57.0
94.8
73.2
81.3
80.8
48.2
40.8
72.4
68.6
Table 12: UnED detailed results over the 8 tine-grineediisttnceelee recognition domains, measured with
Recall 1. highlight the best second-best number each column.
A.2 ADDITIONAL IMPLEMENTATION DETAILS
Loss weight coefficients as in Sec. 3.2 & Stern, 2018) with a learning rate schedule of linear warm-up for 1.4 epochs up to 5e-4, and then linear decay down the student using a momentum on a cosine schedule from 0.994 to 1. The projection heads for self- distillation and masking are identical but unshared, and consist of 3-layer MIP, l2 normalization, and weight-normalized projection layer to a prototype dimension of 32k. We use sharpening and centering operations after the projection head to avoid collapse to either uniform or Dirac delta distributions. To center, subtract the student scores by the EMA of their means using constant momentum 0.9. To sharpen, we set temperatures T₃ = T = 0.1, Ti 0.07, and warm-up r/ along a linear schedule from 0.04 to 0.07. For TIPS-g/14 LR, stop the training early 15 epochs due to evaluation saturation. For TIPS- g/14 [4 HR, we start high-resolutton finetuning from the 13-epoch checkpoint of TIPS-g/14 LR.
= 2. We use the Adafactor optimizer (Shazeer
for the remaining epochs. The teacher model updated with the EMA of
A.3 DATASET CURATION
Table 13 lists the high-quality datasets used to curate WebLI beyond filtering based on image-text and English language. For each target dataset, we first extract image embeddings from pretrained model and perform k-means clustering; Table 13 includes the number of clusters chosen manually to avoid overelustering. WebLI images are assigned its closest cluster by image embedding distance, and probability distribution is defined over the clusters using cluster membership sizes. Then, sample from clusters accordingly, ignoring members that are sufficiently far from their assigned cluster center (90th percentile or above). We repeat this process for each target dataset independently and perform deduplication across all samples. We also perform deduplication with all evaluation data, which removes around 19k images.
A.4 DETAILED EVALUATION PROTOCOLS
In this section we provide the detailed evaluation protocols all evals used in this work. As general remark, identical protocols for low-res and high-res models, without modifying the input resolution, consistent with previous work (Oquab et al., 2024). The parameters of the pretrained transformer network remain frozen throughout the evals.
A.4.1 DENSE IMAGE TASKS
For dense prediction tasks, we evaluate the quality of the patch tokens. As is common practice, concatenate the [CLS] ] token to each of the patch tokens. For our method, we choose the [CLS] token that trained with the more spatially aware synthetic caption (@"). We attach two different types of probes to the image encoder: a simple linear layer (segmentation, depth) or a powerful DPT (Ranftl al., 2021) decoder (depth, normals).
19

---

Published as a conference paper at ICLR 2025
Dutaset Name
Datasel Size
a Clusters
# Imuges Sampled
MASCAL-VOC-20UT-BR
2,501
16.935,483
PASCAL-VOC-201
8,648
17.925.056
ADE20K-train
20,210
20
17.288,796
NYU-Depth-
24,231
1,004,740
ImageNet-20122rraaa
1.281.167
1000
17,792.551
ImageNet22K-train
12,720,275
1000
19,859,560
UnED FoodZk train
472.349
100
4,063,232
HEDD-ARRS99--tali
6,346
5,012.681
UnED-SOP-trin
48,942
48
15,246,219
UnFD-InShop-trii
20,897
UnED-iNatuualistttiil
273,929
188
10,017, 768
UnED-Metttrrin
397.121
397
8,010,633
UnED GLDv2-train
1,422,914
1000
12.323,999
MnFFD-Rp2k-trral
188.724
56 1695
16,888,254
159,65 159,654,074
Total (after self-dedup-)
16,888,254
115,913,373
Total (after selfreval-dedup.)
16,888,254
115,894,610
Table 13:
Dataset curation statistics. High-quality image datasets are used to filter WebLI. We use precom- puted image embeddings calculate image similarity between target datasets and raw WebLI data.
Semantic segmentation. For semantic segmentation, we train the network with images of resolution 512 x 512. We use batch size of 16 images, and train for 40k iterations. We attach a linear layer to the main network, and up-sample to the input resolution to apply the cross-entropy loss. At test time, we run inference on full-resolution images. We note that for DINOv2 (Oquab et al., 2024) concatenating the [CT.S] token the patch tokens consistently yields inferior results, so we report results using their original protocol that does not use the [CLS].
Monocular depth estimation. For NYUv2, we use the simple evaluation protocol of et al. (2024); Oquab et al. (2024). We train with resolution of 480 640. Similar segmentation evals, we concatenate the [CLS] token to the patch tokens, and we upsample by a factor of 4. We attach a linear layer and cast the prediction as classification into 256 uniformly distributed bins. Note that this version of NYUv2 is slightly different to the one used in (El Banani et al., 2024), and the numbers not directly comparable with that work.
For NAVI, we follow Banani et al. (2024) and attach the DPT decoder (Ranftl et al., 2021) to uniformly distributed layers (1 10, 20, 30, 40 for ViT-g/14). We train and test on center crops of objects, with resolution of 512 x 512.
For both datasets we use batch size of 8 and train for 50k iterations.
Surface normal estimation. We follow similar protocols to Banani et al. (2024) for both NYUv2 and NAVI, using their data and metrics. For both datasets, we probe with the powerful DPT decoder using outputs from 4 uniformly sampled transformer blocks. For NYUv2 we train with a resolution of 480 x 480, and we test with full-resolution 480 640 images. For NAVI, we train and test on center crops of the objects, with a resolution of 512 x 512. For both datasets we use batch size of 8 and we train for 50k iterations.
A.4.2 GLOBAL IMAGE TASKS
In global image tasks, we use the [CLS] token that corresponds to the noisy image label that often contains more fine-grained information (e²).
Image classification. We use ImageNet-Ik 1k for classification. We train a linear classifier on top of the ex embedding. We use input image resolution of 224 x 224 for 10 epochs, using an effective batch size of 1024 (smaller batch sizes achieve identical results). We run grid search with the search space of (Oquab et al., 2024), and we report the maximum accuracy, which is a standard practice. We also report the accuracy of soft KNN classification with 20 neighbours (Oquab et al., 2024), without training any weights.
Fine-grained and instance-level retrieval. We the UnED (Ypsilantis et al., 2023) dataset for retrieval. We use an input size of 224 x 224, and we l2- norooaale the resulting e embedding. As is standard for this dataset, we KNN with single neighbour. As also mentioned in the main text,
20

---

Published as a conference paper at ICLR 2025
RGB image
PCA feat.
M. Depth
S. Normals
Sem. Segm.
Figure 5: More qualitative results for dense prediction tasks. For given image (first column), we illustrate the principal components of the predicted spatial features (column 2). Depth (column and normals (column 4) are trained on NYUD, and for semantic segmentation (last column) we used model trained on ADE20k. dense tasks used the DPT decoder, while keeping the image encoder weights frozen.
UnED consists of 8 datasets of different domains. We report recall the final aggregated result, and report results per domain in Sec. A.I.
A.4.3 MULTIMODAL RETRIEVAL TASKS
Image-to-text (I-TT) and text-to-image (T-+)) retrieval. We the [CLS] token that trained with the synthetic caption (@9) as the image representation. We prefer synthetic captions because they tend describe image content more comprehensively than noisy web captions, making them better aligned with the image-text retrieval evaluation datasets used in this work: Flickr30k (Young al., 2014), DOCCI (Onoe et al., 2024) and COCO (Chen et al., 2015). Our text encoder handles maximum of 64 tokens. Texts longer than this limit are truncated, which happens more frequently in DOCCI. Notably, the text encoder SigL.I has small maximum token length of 16, which likely contributes to its relatively low performance in DOCCL.
Zero-shot classification. We use the [CLS] ] token that corresponds to the noisy web caption, which often contains more fine-grained information (e²), as image embedding. We adhere to the established protocol initiated by Radford et al. (2021), which utilizes 80 context prompts to transform each ImageNetlk class into 80 distinct texts. The class embedding is then formed by taking the average of the embeddings from these 80 texts. Classification of the image achieved by retrieving the nearest class embedding.
21

---

Published as a conference paper at ICLR 2025
Input Image
LRM-DIND
LRM-TIPS
Figure 6: Qualitative results on novel view synthesis from LRM (Hong et al., 2024) trained on top of DINO- B/16 and TIPS-B/14. The input image is used to generate parameters for neural rendering model, after image encoding. We visualize RGB and depth images rendered with the same camera parameters (intrinsics and extrinsics) using NeRF-style ray marching. We observe that LRM-TIPS is able predict the geometry of the captured object with higher degree of accuracy, indicated by the gain PSNR from Tab.
A.4.4 3D VISION TASKS.
Single-image to 3D. We use the baseline LRM (Hong et al., 2024) method and follow the architectural and training details of the original paper closely unless otherwise mentioned. We obtain images from Objaverse (Deitke et al., 2023) by sampling random rotations around each object and rendering with uniformly sampled focal lengths E [512, 1024] and nominal distance of 0.4. Cameras are oriented towards the origin and objects are centered at the origin. For each entry in Objaverse, render reference views for training and novel target views for evaluating view synthesis. We freeze the image encoder to extract patch features and directly compare the paper baseline (DINO-B) with ours the ViT-B/14 scale, for fair comparison. Figure 6 shows qualitative results of LRM trained on top of DINO-B/16 and our TIPS-B/14 We show the input single image used to generate neural rendering model parameters, along with RGB and depth images rendered via NeRF-style ray marching (Mildenhall et al., 2020).
A.5 ADDITIONAL QUALITATIVE RESULTS
Figure illustrates further qualitative results of the features and outputs obtained by our method for dense features, probing the image encoder features. Figure we show the qualitative improvement when substituting the default DINO-B with the TIPS-B model for novel view synthesis.
A.6 DUAL EMBEDDING ATTENTION MAPS
Our model generates two distinct image embeddings, an object-centric embedding (e#) and spatially- aware embedding (@"). These embeddings are designed to focus on different aspects of the image: the object-centric embedding captures tie-grained information about the main object, while the spatially-awarr embedding encodes compositional information about multiple objects in the scene. To validate these roles, examine whether their attention maps distributed consistently with
22

---

Published as a conference paper at ICLR 2025
33)
(5)
(6)
Figure 7: Visualization of the attention maps for the object-centric image embedding (e") and the spatially- aware image embedding (e%). Column I: Input images sampled from the MSCOCO test set. Column 2: Attention maps of the object-centric image embedding. Column Attention maps of the spatially-awan image embedding. While the attention maps in Columns 2 and
visually similar, they exhibit meaningful differences, as highlighted in Column 4. The attention peaks in these maps correspond to global proxy patches, as reported in Darcet et al. (2024), which encode global image information. Column 4: Differences between the two attention maps. Red regions indicate areas where the spatially-aware embedding assigns more attention, where the object-centric embedding assigns more attention. The spatially-aware
while blue regions indicate embedding focuses less on global proxy patches and distributes attention more broadly across different objects. Columns 5-6: Filtered attention maps for the object-cenne centric embedding and the spatially-awe aware embedding. respectively. Filtering was performed using median filter with kernel size of 3 on the 32x32 resolution attention maps, which dilutes the effect of the global proxy patches and highlights image regions which attended more prominently. Column Difference between the filtered attention maps. Red regions represent areas with greater attention from the spatially-aware emhedding, while blue regions represent areas with greater attention from the object-centric embedding. The spatially-aware embedding distributes attention more evenly across multiple objects in the scene, whereas the object-centric embedding focuses more the main object.
these expectations. We highlight two key findings: (1) as observed in Darcet et al. (2024), a few sparse patches in low-informative background areas receive high attention, functioning "global proxy patches" that encode global information; (2) the spatially-awan embedding attends less to these global proxy patches and distributes its attention more evenly across multiple objects in the image, which agrees with our expectations.
We randomly selected 500 images from the MSCOCO test set analyze the attention weights of e and 6 over the patch tokens. Example attention maps for both embeddings are shown in Figure 7, columns 1-3. Since e and 6 are not far apart in the embedding space (cosine distance around 0.15 generally), their attention maps are visually similar at a coarse level. However, their differences are highly informative.
First, both embeddings share a common pattern: sparse patches in low-informative background areas-global proxy patches-receive the most attention, contributing significantly to the global image embedding. Second, the spatially-aware embedding (êØ) differs by assigning less attention to these global proxy patches and distributing its focus more broadly across patches corresponding to different objects in the image. These differences are further visualized in Figure 7, column 4, where red indicates where the spatially-awar embedding focuses more and blue where the object-centric one focuses We further analyzed the attention maps by filtering out the peak attention global
23

---

Published as a conference paper at ICLR 2025
Embedding
Cetect-centrre
Spatially-aware
Weight
Deviation
Disribution Entropy
Percentage
10.0%
0.0%
0.04
0.08
0.12
0.16
0.003
0.004
0.005
0.006
5.25
5.50
5.75
6.00
Maximum Attention Weight
Attention Standard-peviation
Attention Distributiin Entropy
Figure 8: Distributions of statistical metries for the attention maps of the object-centric embedding (e*) and the spatially- embedding (ê"), computed over a sample of 500 images randomly selected from the MSCOCO test The green bars represent the attention of the object-centric embedding, while the red bars represent the attention the spatially-awart embedding. The results indicate that the spatially-aware embedding has lower maximum attention weights, lower standard deviations, and higher entropy values, demonstrating that its attention less concentrated on global proxy patches and more evenly distributed across the image compared to the object-centric embedding.
proxy patches using a simple median filter with a kernel size of 3, applied to the 32x32 resolution maps (Figure 7, columns 5-7). This filtering highlights that the spatially-awwa embedding allocates attention more broadly to multiple objects, while the object-centric embedding remains focused primarily the main object.
To quantify the differences between the two embeddings' attention maps, we measured three metrics: the maximum attention weight, the standard deviation of the attention weights, and the entropy of the attention distribution. Specifically, for each of the 500 test images, we computed these metrics for both embeddings' attention maps. The maximum attention weight, typically found proxy patch, indicates how concentrated the attention is on these patches. A higher value reflects greater focus on global proxy patches. Similarly, the standard deviation of the attention weights reflects the spread of attention; more evenly distributed attention map has a lower standard deviation. Finally, treating the attention weights as probability distribution, we computed the entropy, where higher values indicate more evenly distributed attention. The distributions of these metrics for the two embeddings are shown in Figure The results reveal that the spatially-aware embedding (θ") exhibits lower maximum attention weights, lower standard deviations, and higher entropy values, confirming that its attention more evenly distributed and less focused on global proxy patches compared to the object-centric embedding (e²).
global
These findings align with our observations that the object-centric embedding performs better on global tasks, while the spatially embedding excels at tasks requiring spatial awareness. Overall, the complementrry attention patterns of the two embeddings validate the effectiveness our dual embedding design in capturing diverse types of image information.
24


Reviews:
[{"review_id": "vnLp2icUJT", "reviewer": "Reviewer_6Wzj", "summary": "This paper presents a spatial-aware text-image pre-training method that combines contrastive image-text learning with self-supervised masked image modeling. Besides, the method proposes to combine the noisy web captions and synthetic captions that are more helpful to learn spatially aware representations. The method is evaluated on both zero-shot classification and dense prediction tasks.", "questions": "Please refer to my comments above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents a solid work on text-image pre-training: a large-scale synthetic caption dataset is created, the method is evaluated on both classification and dense prediction tasks, and extensive experimental studies are conducted for ablation studies and analyses. \n\n- The results look good. The proposed method can achieve good dense prediction and classification/retrieval performance simultaneously. Ablation results provided in the paper may be helpful for developing new text-image models.", "weaknesses": "- The general idea of combining contrastive image-text learning and masked image modeling is not new. Previous work like EVA-CLIP [r1] has already show that MIM can improve the spatial awareness or locality of CLIP features and improve CLIP performance. The core different between TIPS and the line of work is to combine MIM and CLIP successively or simultaneously. I think simultaneously perform the two tasks may be better to preserve the spatial awareness/locality, but it may also make the training more costly, or possibly unstable. It would be better to provide a comparison/analysis on the pros and cons of the two strategies.  \n\n[r1] EVA-CLIP: Improved Training Techniques for CLIP at Scale\n\n- The study use the proprietary WebLI dataset to train the model. Is it possible that the improvements over previous methods mainly come from better data sources?  How about the results if both the proposed model and the baseline use publicly available datasets like LAION, COYO or DataComp.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zSBCQNbWM8", "reviewer": "Reviewer_FXGp", "summary": "This paper introduces a novel pretrained image-text encoder with spatial awareness which is effective in a variety of downstream computer vision tasks. To achieve this, the author first employs pretrained multimodal generative models to generate high-quality synthetic image descriptions and develops a dual embedding approach that leverages both synthetic and noisy web captions in training. Additionally, contrastive image-text learning, coupled with self-distillation and masked image modeling, is introduced to encourage the model to learn spatially aware representations. Experiments conducted on eight downstream tasks validate the effectiveness of the proposed method.", "questions": "Will the pretrained model and the curation dataset with synthetic captions be released?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The author proposes an effective approach that enhances the utility of both synthetic and noisy web captions in training. They also introduce contrastive image-text learning with self-supervised masked image modeling, which effectively encourage the learning of spatial coherence.\n2. The author conduct a variety of experiments in 8 downstream tasks demonstrate the effectiveness of its spatial-aware text-image encoder.", "weaknesses": "The formatting of the paper needs improvement and there are  a lot of empty spaces around fig1 and fig2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y7dm8H5YOv", "reviewer": "Reviewer_BGwY", "summary": "This paper targets integrating the paradigms of both image-text representation learning and self-supervised learning to improve the spatial awareness of the former. For the SSL branch, the authors leverage the DINO V2 (iBOT) pre-training method; for the image-text branch, they propose the dual image-text embedding technique that learns from both noisy and sythetic captions while harnessing the distribution gap between two types of captions. The effeciveness of the proposed method is evaluated on several image-level multimodal tasks and comprehensive dense image prediction tasks.", "questions": "- As the motivation of this paper is to bridge the gap between image-text representation learning and SSL, although the ablation studies are provided, this paper lacks an in-depth analysis on how the two paradigms interact with each other. For example, how the SSL design choices such as augmentations (mask ratio, etc.) affect the image-text representation learning. \n\n- The idea of dual embedding is interesting. I'm curious about the different roles of the two embeddings, and how they interact with the network. Could the authors provide more empirical analysis on this? For example, visualization of the attention maps of the two different $[CLS]$ to see their focus areas.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This paper is well written.\n\n- The experiments on dense image prediction tasks are comprehensive and promising, outperforming DINO V2 on several tasks.\n\n- Improving the spatial awareness of image-text representation learning is an important direction, combining DINO v2 and CLIP, where both are foundational works in their respective fields, is intuitive and promising.", "weaknesses": "- The technical contributions are limited. The proposed method is a combination of existing methods, with the dual embedding technique being the only novel contribution. Nonetheless, I'm okay with this, since the proposed model effectively and adequately solves model's spatial awareness limitation.\n\n- As claim in Line 300: \n>Our method is the first to demonstrate that combining contrastive image-text learning with self-distillation and masked image modeling leads to improvements across many tasks\n\nHowever, both integrating CLIP with self-distillation and masked image modeling[1][2] have been proposed before. And this paper lacks a further discussion against these works.\n\n- Since this is a multimodal model with spatial awareness, only I$\\rightarrow$T and T$\\rightarrow$I retrieval tasks are not enough to evaluate the model's fine-grained spatial awareness under multimodal settings. Including more experiments like open-vocabulary segmentation would be beneficial.\n\nreference:\n\n[1] Maskclip: Masked self-distillation advances contrastive language-image pretraining.  CVPR 23.\n\n[2] Scaling Language-Image Pre-Training via Masking. CVPR 23", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PLo0rm3D8Q", "reviewer": "Reviewer_wggG", "summary": "This paper addresses dense and global vision tasks by enhancing textual supervision and integrating contrastive image-text learning with self-supervised techniques. The method combines noisy web captions with synthetically generated captions to improve spatial awareness and applies masked image modeling to promote coherence in spatial understanding. As a result, the model demonstrates robust performance across various tasks without the need for fine-tuning, showcasing its general-purpose applicability in both image-only and image-text applications.", "questions": "1.\tAuthors are suggested to add detailed ablation results to isolate the impact of the synthetic captions on different spatial tasks.\n2.\tHave you considered alternative ways of introducing spatial awareness besides synthetic captions and masking?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThe paper is well-structured and clearly articulated, with detailed experimental records. By cleaning and constructing a high-quality dataset and incorporating self-supervision, methods such as dual captioning and masked image modeling enable the model to achieve significant (albeit incremental) advancements in dense prediction tasks.\n2.\tThe trained model demonstrates strong generalizability across multiple tasks, indicating its broad applicability in vision tasks.\n3.\tThe paper includes a substantial amount of experimental comparisons and work.", "weaknesses": "1.\tThe work presents only a limited amount of novelty. The main critique lies in the lack of significant innovation. The paper largely repurposes existing techniques like synthetic captioning and contrastive learning, and while the results are solid, they do not represent a substantial leap forward in the field.\n2.\tThe improvements over existing models such as CLIP and DINOv2 are incremental, and the performance gains are sometimes marginal or context-specific. The originality in combining these techniques does not feel transformative.\n3.\tMore detailed ablation studies focusing on the contribution of each component (e.g., the specific impact of spatial coherence from the captions) could strengthen the claim of novelty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)