Characterizing Vision Backbones for Dense Prediction with Dense Attentive Probing
Authors: Timo Lüddecke
Alexander S. Ecker
Venue: Accepted by TMLR ()

Abstract: The paradigm of pretraining a backbone on a large set of (often unlabeled) images has gained popularity. The quality of the resulting features is commonly measured by freezing the backbone and training different task heads on top of it. However, current evaluations cover only classifications of whole images or require complex dense task heads which introduce a large number of parameters and add their own inductive biases. In this work, we propose dense attentive probing, a parameter-efficient readout method for dense prediction on arbitrary backbones – independent of the size and resolution of their feature volume. To this end, we extend cross-attention with distance-based masks of learnable sizes. We employ this method to evaluate 18 common backbones on dense predictions tasks in three dimensions: instance awareness, local semantics and spatial understanding. We find that DINOv2 outperforms all other backbones tested – including those supervised with masks and language – across all three task categories. Furthermore, our analysis suggests that self-supervised pretraining tends to yield features that separate object instances better than vision-language models.  
Code is available at http://eckerlab.org/code/deap.

Published in Transactions on Machine Learning Research (09/2025)
Characterizing Vision Backbones for Dense Prediction with Dense Attentive Probing
Timo Lüddecke and Alexander Ecker
Reviewed OpenReview: http:://pppaaeii......e/////ttta
Abstract
The paradigm of pretraining backbone large set of (often unlabeled) images has gained popularity. The quality of the resulting features is commonly measured by freezing the backbone and training different task heads on top of it. However, current evaluations cover only classifications of whole images or require complex dense task heads which intro- duce a large number parameters and add their own inductive biases. this work, we propose dense attentive probing, a parameter-effiiint readout method for dense prediction on arbitrary backbones independent of the size and resolution of their feature volume. To this end, we extend cross-altention with distance-based masks of learnable sizes. We employ this method to evaluate 18 common backbones dense predictions tasks in three dimensions: instance awareness, local semantics and spatial understanding. We find that DINOv2 outperforms all other backbones tested including those supervised with masks and language across all three task categories. Furthermore, our analysis suggests that self-supervised pretraining tends to yield features that separate object instances better than lision-language models. Code available at https://eckerlab org/ prg/code/eaap.
1 Introduction
Driven by the success of self-supervised learning (Chen et al,, 2020; He al., 2022; Oquab al., 2023) and vision-language training (Radford et al., 2021) training from scratch has largely been replaced by finc-tuning large pretrained backbones for many computer vision tasks. Ideally, the pretraining yields powerful features such that fine-tuning succeeds even with small datasets and by modifying only small number of parameters. Many factors influence the feature quality of pretrained backbones, including the pretraining paradigm, model architecture and the training data. Therefore, for both computer vision scientists and practitioners, it crucial to characterize strengths and weaknesses of large pretrained backbones through systematic benchmarks. For assessing the whole-image classification performance of backbones per image) such benchmarks exist, for example, ImageNet (Russakovsky al., 2014), VTAB (Zhai et al., 2019) and FGVC (Jia et al., 2022). They often rely on the established approaches of linear and attentive probing. Linear probing applies global average pooling and then linearly maps the resulting feature vector to class predictions while attentive probing uses cross-attention with a query token that predicts the class. Both techniques are not applicable for producing dense output due to limited resolution. For example, in linear probing the prediction has the same spatial resolution as the feature volume, which usually too low to capture object structures. Furthermore, the resolution of the feature volume varies across different backbones, preventing fair comparison. Therefore, often common task heads such UPerNet (Chen et al., 2024a) for semantic segmentation Faster- Faster-RCNN for object detection (Ren et al., 2015) are used at the price of introducing a large number of learnable parameters and additional inductive biases. As a consequence, the resulting backbone feature quality measurements are mediated by compatibility and performance of these heads.
predicting one label
In this work, we address the problem of assessing and comparing the representational quality of dense feature volumes from various backbones. To this end, we measure dense prediction performance of backbones as directly possible by proposing novel dense equivalent to attentive probing. Our model consists of

---

Published in Transactions on Machine Learning Research (09/2025)
5:
bacibooe
bankbrone
backk
depth (d)
learnable
Model
perameters
Conv Adapedr (Chen et al., 2024a)
>24M
ViT-Adapter (Chen al., 2023)
2.5M 23.7M
attention
Conv LoltA (Zhong al., 2024)
4M
heads
Dense Attentive Probing (ours)
0.05M 0.5M
Figure Feature volumes of different. backbones (black rect- angles) differ in feature depth (d) and spatial size (s). We assume that the region size of informative features for each lo- cation q (blue and orange rectangles) varies across backbones. To compensate for this, our model uses attention masks with belrnable region size (blue and orange lines) per attention head and decouples feature and output resolution.
Table Adapter methods for dense predic- tion with the number of learnable parame- ters.
single masked cross-attention layer, introduces only small number of parameters (less than 100K, often even less than 50K) and adds a relatively small computational overhead over the backbone. By using cross- attention, we decouple the size and resolution of the input image and encoder output from that of the dense output, i. c. generate outputs at any resolution. As the backbones that we evaluate were trained with different techniques (e.g. language-vision or masked image modeling), they might encode local information differently (Fig. 1). For example, information for predicting at output location q might not exclusively be encoded at the corresponding location p in the feature volume but be distributed over a large region around in the feature volume. This region can be defined as the set of spatial (i.j) and feature (d) indices dd) [(q) of the feature volume which contains relevant information to predict target function (such as segmentation mask) at location q. Idcally, probing model would capture this region I(g). Our model makes the simplifying assumption to define the region (()) smoothly by a Gaussian function which is used as attention mask in the cross-attention mechanism. By learning individual standard deviations of this function per attention head, the model can process different degroes of feature locality (or region sizes) each dimension.
Balestriero et al. (2023) note the lack of standardized evaluation protocol for self-supervised learning meth- ods on dense prediction tasks. We believe dense attentive probing method can address this gap and experimentally assess the quality of various supervised, self-supervised, vision-languar training methods. Specifically, we use our dense probing method to characterize features for dense tasks along these three dimensions: (1) instance disentanglement, e. how well are individual instances recoguizable from the fea- tures; (2) local semantics, evaluating how meaningful the features are for a local classificatio;; and (3) spatial understanding, which assesses how well is the 3D structure of the scene captured. Our method is better suitable for evaluation than prior work because it decouples feature and output resolution, adapts feature locality and uses only a small parameter budget.
2 Related work
Representation learning. Self-supervised representation learning has been a popular research topic with multiple approaches that can roughly be categorized into joint-embedding (Chen et al., 2020; 2021; Caron et al., 2021) and reconstruction- based (He et al., 2022). DINOv2 is based the iBot (Zhou et al., 2022) method which uses coint-cmbedding architocture in combination with sclf-distillation and reconstruction. VicRegL (Bardes et al., 2022) and many other recent methods explicitly addresses local features by modeling losses at the token level. There has been a discussion about which techniques lead to better and more efficient features for perceptual tasks (Balestricro & LeCun, 2024). The approaches discussod above mainly address classification. Another stream of research, called object-centric learning, focuses on learning disentangled object representations. While early methods only worked on synthetic data (Burgess et al., 2019; Locatello
2

---

Published in Transactions on Machine Learning Research (09/2025)
et al., 2020) more recent approaches succeed natural images (Zadaianchuk et al., 2024; Aydemir et al., 2023). Recently, a new method for evaluating such object-centric representations was proposed Didolkar et al. (2025). The main difference of our present work to object-centric methods is that we assume that object instances are encoded implicitly in feature volumes whereas object-centric methods explicitly represent objects in their architecture ge in slots). The seminal CLIP model (Radford et al., 2021) introduced another stream of research called vision-language models (VLMs), where the model is trained on aligning text-image pairs. Later, this training paradigm was simplified use sigmoid-based loss function (Zhai et al., 2023) instead of a softmax-based loss, making the method less dependent on the batch size. Recently, the role of data is investigated more closely in the context of vision-language models (Gadre et al., 2024; Xu et al., 2024; Fang et al., 2024b).
Feature evaluation. Evaluations on local features predate the deep learning era in computer vision, often addressing hard-coded descriptors which were computed around a sparse set of points of interest (Mikola- jezyk & Schmid, 2005). Recently, there have been numerous attempts at characterizing and comparing features from common deep learning backbones. This involves evaluating the 3D understanding of back- bones (El Banani et al., 2024; Man et al., 2024) as well as more specific aspects like physics (Zhan et al., 2024), language-binding (Lewis et al., 2024), compositionality (Thrush et al., 2022) or shape perception (Bonnen et al., 2024). Danier et al. (2025) compare numerous methods for depth estimation. Chen et al. (2024b) design a zero-shot benchmark for image encoders for the contrastive vision-language pretraining setting and propose the ViTamin architecture based their findings. Goldblum et al. (2024) evaluate clas- sification, instance segmentation, object detection and retrieval by training large task heads (Cascade Mask R- R-CNN) for dense prediction. Further efforts to characterize vision backbones include the timm leaderboard (Wightman, 2019) for image classification, CLIP benchmark (TAIONNAA,, 2022) for vision-language models, V-PETL, for parameter-offiiien fine-tuning (Xin et al., 2024) and CV-Bench for multimodal large language models (MLLMs; Tong et al., 2024). Our work differs in considering multiple pre-training paradigıns (self- supervised, languala etc.) and multiple feature qualities (instance awareness, local semantics and spatial understanding) while fine-tuning with small number of learnable parameters. Importantly, we avoid the use of standard task heads like UperNet (Chen et al., 2024a) for segmentation as they introduce own inductive biases and large number of parameters which interferes with measuring the backbone's performance.
Adapters and parameter--effciiet fine-tuning methods. Fine-tuning of large backbones using a small parameter budget is well-studied problem under the umbrella term parameter efficient fine-tuning (PEFT). Our work shares the goal of keeping the nunber of parameters minimal. A commno approach to address this problem is to use adapters (Houlsby et al., 2019; Hu et al., 2022; Lian et al., 2022), which fine-tune by adding Icarnable small, low- www-parmeteer sub-networks within larger backbones or finc-tune only on a subsct of weights. While these techuiques originate in natural language processing, they have recently been applied to computer vision. However, many adapters only address whole-image classification tasks Chen et al. (2022); Steitz & Roth (2024). Existing dense prediction adapters are not suitable for our goal of comparing backbones due to large number of parameters: ViT-Adapter (Chen et al., 2023) builds established task heads for segmentation (UperNet) and detection (Mask R-CNN and HTC++). The number of parameters introduced by this adapter depends on the backbone, ranging from 2.5M to 23.7M parameters. ConvAdapter (Chen et al., 2024a) propose an adapter specifically for convolutional networks. In case of dense prediction, their method uses standard task heads which require a large mumber of parameters. Zhong et al. (2024) apply low- rank adaptation to image segmentation tasks by introducing around 4M parameters. These methods require substantially more parameters than our method (Table 1). For an in-depth review of adapters we refer to the survey of Yu et al. (2024). In addition to adapters, there are other parameter-efficien fine-tuning techniques: Attentive probing (Yu et al., 2022; Bardes et al., 2023) is inspiration for our work but is limited to whole- image classification in its original formulation. Oquab et al. (2023) use bilincar interpolation in the feature volume to make dense predictions. However, this approach is limited in its output resolution and unsuitable for a comparison as the feature volume sizes vary across backbones. Their advanced multi-scale readout (called "ms") combines activations from multiple layers, adopting this would complicate a comparison across backbones. FeatUp (Fu al., 2024) is generic method to upscale feature volumes and thus also applicable here. We will use this method as a baseliee.
3

---

Published in Transactions on Machine Learning Research (09/202))
W
Backbone
Crovs-ttentiig Transformer Layer
F(H)
connection
Reys
Interpolate
Corn.
Input image (H)
mask computation
per head
scaled attention
2-Layer MIP
Der
Masks
Query (HoWaal)
B8h
queries
per tolin
Query
Prodiction: (aHo aWo, Daad
Frczen Concatenation
Addition
encoding (FE)
Figure 2: Dense attentive probing design: An input image x is processed by an arbitrary backbone ø and posilionally encoded (see P'), yielding the leal ure volume F(x). Queries (Q) in form of a position-encoded coordinate grid cross-attend to these features F(x). Analogous to the Transformer layer, queries, keys and values are projected (+b) before attending. The attention constrained to a local region by a mask per head. These masks are computed from pairwise spatial distances between locations in the feature volume and in the query grid. The standard deviation
learnable scalar parameter that controls the size of the mask in each attention head. The resulting tokens upscaled by small CNN to yield task-specific output. Since the query position grid determines the output size, predicted output size and size of the feature volume are decoupled.
3 Dense Attentive Probing
In this section, we introduce the Dense Attentive Probing (DeAP) method (Fig. 2). It is designed be a parameter and compute efficient readout that uses cross-attention make dense predictions based on features from frozen backbone. Instead of allowing all queries to attend to the complete feature volume (full cross-atiention) constrain the attentive region locally such that each query can only interact with local feature vector neighborhood. The size of this smooth local region is learnable to adapt to the characteristics of different backbones.
An arbitrary, frozen image backbone & receives an image x size (Hs, Ws, and generates features of size (H, W, D') with s indicating the backbone's stride (the factor by which the backbone reduces the spatial resolution of its input). These features are concalenated (denoted by []] with non-learnable, sinusoidal positional encoding (PE) of the grid coordinates P', yielding F(x) of size (HW,
F(x) [PE(p), ((x))
(1)
To generate dense output, generating output in local region. The spatial queries Q of size (HoWq, 16) are fixed (non-learnable), 8- dimensional positional sinusoidal encodings (Vaswani et al., 2017) of the (output) position grid. This position grid consists of integer positions, í.c. Qij = [PE(1), PE(r)]- The queries Q interact with the feature volume F(x) using a cross-allention mechanism. To enable the model to account for feature locality, we modify the cross-attention to consider spatial proximity by adding M(a) to the attention scores. The computation for each head h (out of H heads) described by
cross-attention-bart approach, where each query is responsible for
(h) 1/2 222 (QN (F(x))
(h) softmax
+ M(6) w (F(x)),
(2)
Vdime
with the functions 46 (b) (VV...)) flattening along the spatial dimensions followed by multiplications with matrices of sizes (D, 2P), (D, P) and (8, P) for values (V), keys (K) and queries (Q) respectively.
4

---

Published in Transactions on Machine Learning Research (09/2025)
Self-supervieed
Vision-
Training
Architecture Data
Supervized
Training Variant Architecture
Data
Training
Arch.
Date
MoCoV3
J
VrT-B (86M)
ImageNet
Meta
CLIP
VIT-B (86M) CLIP
ViT-B (86M) CC-400M
Tmg. Class. VYT-B (86M)
SAM2
Hiera B+ (69M) SA-1B
Not
Hiora
DINO
MAE
R
R
J
Hiera B+ (69M)
VIT-B (86M)
VIT-B (86M)
ImageNet
ImageNet
ImageNet
SigL.P
SigLIP 512
SigLIP (8O)
VIT-B (86M) Webli
VIT-B (86M) Webli
VIT (414M) Webli
DINO
&
VIT-B (86M)
LVD-142M
DINOv2
J & R VIT-L (304M)
LVD-142M
ViTamin--22
Aim2
own (300M)
(333M)
many
DataComp-1
Table 2: Models, their backbone architectures (with parameters) and their training datasets. Supervised means that numan-provided labels were used. and R denote boitt-ebeedding and reconstruction self- supervised learning methods.
The attention computes interactions between all backbone features F(x) and all query locations, therefore M(r) has a size of (Ho HW). Each element of My depends on the squared Euclidean distance dy between the i-th location in the query and the j location in the feature volume and the region size a through
di
the function My = ov2n
exp
(-
The learnable parameter a is a scalar in each attention head that 2n²²
adjusts the size of the region in the feature volume (locality) which each query interacts with. The output is obtained by a concatenation of all H head outputs T =
Similar to the transformer layer, after this layer, each token T₁ is processed independently by 2-layer multi-layer perceptron (or feedforward layer) FFN. In the hidden layer of FFN, the vector dimension is expanded by a factor of two.
Instead of using separate query for each output pixel, regions of size 8 are processed jointly for efficiency reasons, i. e. each query Qij generates 64 pixels of the output. This is realized by re-arranging the FFN output into spatial shape (HQ WQ,D), denoted by T', and processing it with a small CNN y which yields the final dense output (T).. The CNN y has three blocks, each composed of convolution and transposed convolution and ReLU non-linearity. These blocks are followed by final convolution layer, skip-connection enables efficient learning. The nunber of channels in the CNN y is given by DONN max (D/4, Dout), with the number of output channels Dout being task-dependent. Dense attentive probing consists of only a single masked cross attention layer.
Since the queries only receive a position as input and can attend to all features, the masked cross-attention architecture resembles implicit (or coordinate-based) net works (Mescheder et al., 2019; Park et al., 2019), especially PiFU (Saito et al., 2019) and PixelNerf (Yu et al., 2021) which combine implicit networks with feature volumes. The modification of the attention through bias term is similar GraphDINO (Weis et al., 2023), ALiBi (Press et al., 2022) and the work of Shaw et al. (2018). In contrast to these approaches, our bias term depends on 2D distances and has a learnable size parameter.
4 Experiments
4.1 Choice of models and training datasets
We select a broad range of feature backbones that encompass different training paradigms and were trained on different datasets (Tab. 2). This enables us to conduct controlled comparisons along several axes, for instance, datasets and pretraining task. In general, we differentiate between three broad classes of methods: supervised (Dosovitskiy et al., 2021; Ravi et al., 2024), self-supervised (Caron et al., 2021; He et al., 2022; Ryali et al., 2023), and vision-language (Radford et al., 2021; Xu et al., 2024; Fini et al., 2024; Chen et al., 2024b; Liu et al., 2022). The latter involves training on image-caption pairs often obtained from the internet, while self-supervised training operates dataset (Russakovsky et al., 2014), but there are several exceptions: All SigLIP models are trained on the Webli dataset, a Google-internal dataset of 10 billion images with 12 billion multi-lingual texl-image pairs. MetaCLIP uses a selection of the open LAION dataset (Schuhmann et al., 2021), CLIP is trained on the unpublished CLIP dataset by AI. DINOv2 (Oquab et al., 2023) is trained on the LVD-142M, a Meta- internal dataset of 142M images which were deduplicated and curated to be similar to ImageNet- -2k images. In our experiments, we use the V2 version with registers Darcet et al. (2024). The data mix of Aim2
images only. Many models are trained on the ImageNet
5

---

Published in Transactions on Machine Learning Research (09/2025)
Instance awareness
Local semanttcs
Spatial understanding
Image
Instance boundaries
Object centers
Instance discrimination
Semantic Segmentation
Depth Estimatiio
glass
background
Figure 3: Task overview. We use three tasks to probe instance awareness: object. detection based on CenterNet, instance boundaries and instance discrimination. Additionally, we assess local semantics and spatial understanding.
(Fini et al., 2024) contains DFN-2B (Fang et al., 2024a), COYO (Byeon et al., 2022), the proprictary HQITP dataset and synthetic data.
We decide to mainly focus on vision transformers as many approaches share the same architecture and checkpoints are available for a large number of training paradigms. To ensure comparability with other work and control for model architecture, we primarily use ViT-B/16 and similarly sized models our experiments. We also include larger models in some cases obtaín an estimate of how much performance can be improved simply by scaling-up model size. Fretrained-weightt. are obtained from the timm package (Wightman, 2019) the code repositories of the methods.
4.2 Experiment design
We provide images in the native resolution of the respective backbones to prevent nut-of-distribution input. For a fair comparison, we use images of size 224 x 224 as the basis for all tasks, if not indicated otherwise. When backbones require a higher native input resolution, we scale the 224 224 image to the respective resolution. For generating predictions, we make use of the capability of our model to decouple input and output resolution (see Sec. 3): The output size is fixed to 224 x 224 for all models (even for backbones with larger input image sizes), ensuring fair comparison across models and limiting the advantage of large input sizes for the backbone. In the masked cross attention, we use dimension D= 16 and eight altention heads (i. e. two dimensions per head). Although the readout could attend to backbone activations at different layors we opt for using only the last layer, motivated by Li et al. (2022) who found that a simple feature pyramid using the last layer without top-down connections works best.
We pursue straightforward approach comparison: We freeze the backbone features, train the small readout in supervised way and evaluate and capacity of the readout forces it to directly rely on the feature volume for making dense prediction. This setup differs from conventional task heads (c. g. in object detection) which are able to perform more complex computations on the features. For example, using Faster R-CNN on top of a ResNet50 backbone adds around 18 million parameters to the model.'
held-out test set. The rationale is that the low expressivity
4.3 Evaluation tasks
To characterize broad spectrum of traits of the backbones we implement three task categories: probing instance awareness, local semantics and spatial understanding (Fig. 3).
Instance awareness. In this task we evaluate how well the features are able to disentangle individual instances, for example multiple apples in bowl. In the field of object-centric machine learning (Burgess et al., 2019) models are designed to disentangle instances. Here we assess the extent to which instance discrimination is already encoded in the features of different backbones as a result their pretraining. We consider the following three notions how instances can be encoded:
These caleulations obtained using the Faster R-CNN implementation PyTorch vision (Paszke al., 2019), in more detail FPN: 3.3M, RPN: 0.6M, ROT heads: 14.3M parameters.

---

Published in Transactions on Machine Learning Research (09/2025)
Instance boundaries: The objective is to outline individual objects in the image. We frame this problem a binary segmentation and consequently use an output dimension Dout = 1 as well as the binary cross-entropy loss function on the output.
Object centers (CenterNet heatmaps): CenterNet (Zhou et al., 2019) is detector that uses class-wise heatmaps for detection. We ignore classification and use single center map. In second heatmap, bounding box sizes are regressed. Both maps are predicted using individual dense attentive probing readouts with Deetter = 1 and Doine = 2. As proposed in the original paper, we obtain detections by finding 9-neighborhood maxima in the center healmap and extracting the bounding box sizes at these locations. We further process the detections with non- non-maximum suppression. For comparison we report average precision of large objects due the small (in particular for detoction) input image size of 224 x 224.
Instance discrimination: Another way to encode instances is to generate a latent space where features within instances are the same (or similar) while being different to all other instances. If this works perfectly, clustering the latent vectors of all pixels would yield instances. This task is sometimes also called coloring (Novotny et al., 2018). We train on only 8,000 sample images and treat every instance individual class (resulting in around 60,000 classcs). We first use dense attentive probing map the features to latent space (in our case Dont = 32). Then, linear layer maps each local 32-dimensional feature to a probability over all instances in the dataset. Thus, the problem is essentially framed as semantic segmentation with 60,000 classes. This way, the latent features before the classification head learn to discriminate instances. For testing, we cluster these features obtained from unseen images. For clustering we use k-means and provide the ground-truth number of instances as well as a foreground mask. Then we compare the predicted foreground instances with the ground truth instance segmentation based on the adjusted rand index.
For these experiments, we use the COCO dataset (Lin et al., 2014), with the 5,000 images from the validation set being used for testing. For the instance discrimination task, we compute the ARI (adjusted rand index) test scores only on images with at least three large objects (resulting in a subset 754 images). Note, these tasks do not involve classifying the instances into object categories, unlike typically done in instance segmentation (this is assessed below in "local semantics").
Local semantics. A natural choice for evaluating local semantics is a semantic segmentation task. Here we rely on two benchmarks: Pascal VOC 2012 (Everingham al., 2015) and COCO Stuff (Caesar et al., 2018). The Pascal VOC 2012 encompasses a fairly small of only 1,464 training images. For COCO Stuff, train 100,000 images. We account for the larger number of classes in COCO Stuff by sctting the internal dimension of the CNN, DONN, to 32.
Spatial understanding To assess how well the features capture the 3D structure of the scene, we im- plement the well-known monocular depth estimation task: The models need to infer the depth ei.c. position along the z-axis) for every pixel of the visible scene based on the features provided by the backbone. We frame this as depth map estimation problem, e. Dont 1, relying on the NYUv2 dataset (Nathan Silber- man & Fergus, 2012) for training and testing the depth estimation readout. We first scale the input images to a resolution of 216 x 288 and then to the native resolution of the backbones.
5 Results
We begin this section by presenting our main result, the comparison of backbone performance and conclusions we draw from it. We also show that the results are accordance to existing findings. Then we validate the design of our method through an ablation and comparison to similar baselines. Lastly, we investigate the sprean-peformmnce tradeoff for practitioners.

---

Published in Transactions on Machine Learning Research (09/2025)
Inst. Disc.
Boundaries
Object Detection
Cat. Backbone
F
ARI Plearn
CE 4 IoU t
Plearn
AP t
Plearn
Random (untrained)
224
14
85.8
23.5
0.053
0.2544
0.8
0.043
0.0015
0.087
ImageNet
224
14
85.8
36.3
0.053
0.1787
16.0
0.043
0.1126
0.087
SAM V2 B
1024
80.8
50.0
0.028
0.1394
30.2
0.018
0.1759
0.035
MoCo V3
224
14
85.8
41.8
0.053
0.1613
21.4
0.043
0.2078
0.087
Dino
224
28
85.9
41.5
0.053
0.1433
28.3
0.043
0.2344
0.087
Dino V2
518
86.6
46.8
0.053
0.1297
32.4
0.043
0.2666
0.087
Dino V2 (VIT-L)
518
37
304.4
46.4
0.066
0.1450
33.8
0.056
0.2811
0.112
MAE
224 14
85.8
49.8
0.053
0,1500
25.3
0.043
0.2662
0.087
Hiera B
224
69.1
44.8
0.060
0.1717
17.1
0.050
0.2315
0.099
CLIP
224
14
85.9
39.5
0.053
0.1665
19.0
0.043
0.1391
0.087
CLIP (VIT-L)
336
24
303,6
40.6
0.066
0.15433
24.3
0.056
0.2005
0.112
MetaCIIP
224
85.9
38.4
0.053
0.1
19.1
0.043
0.1667
0.087
SigLIP-224
224
14
85.8
37.8
0.053
0.1718
17.2
0.043
0. 0.1677
0.087
SigLIP-384
384
24
86.1
39.1
0.053
0.1564
23.3
0.043
0.1836
0.087
SigLIP-512
512
86.5
38.9
0.053
0.1496
26.2
0.043
0.1910
0.087
SigLIP-SO
512
36
413.7
40.7
0.073
0,1496
25.7
0.062
0.1952
0.125
Aim2
336
24
309.6
40.0
0.066
0.1556
23.5
0.056
0.2011
0.112
ViTamin
384
333.0
42.1
0.066
0.1535
24.4
0.056
0.1648
0.112
Table 3: Instance awarenoss results in all three categories. 7: image size. F: Size of feature volume. P Number of all and learnable parameters, respectively, in millions. Metries: Adjusted rand index (ARI), cross-entropy (CE), intersection over union (IoU), average precision for large objects (AP1g).
and Beam:
Pascal VOC2012
COCO Stuff
Depth
Backbone
F
CE
IoU
Plearn
CE
loU
Piearn
RMSE
Piearn
Random (untrained)
224
14
85.8
2.7364
3.5
0.047
5.0889
0.1
0.088
1.757
0.043
ImageNet
224
85.8
0.3924
61.0
0.047
1.5140
28.1
0.088
0.713
0.043
SAM V2 B+
1024
64
80.8
0.6005
33.9
0.022
2.0630
12.6
0.062
0.714
0.018
MoCo V3
224
14
85.8
0.3274
63.3
0.047
1.4417
27.3
0.088
0.606
0.043
Dino
224
85.9
0.3397
63.8
0.017
1.3423
30.2
0.088
0.585
0.043
Dino V2
518
37
86.6
0.1259
83.4
0.047
1.0468
42.8
0.088
0.425
0.043
Dino V2 (VIT-L)
518
37
304.4
0.1173
85.3
0.060
1.03447
43.8
0.101
0.398
0.056
MAE
224
85.8
0.3471
60.2
0.047
1.4822
25.3
0.088
0.580
0.043
Hiera B+
224
7
69.1
0.3344
61.5
0.054
1.5007
25.7
0.094
0.523
0.050
CLIP
224
14
85.9
0.2710
68.3
0.017
1.3367
32.0
0.088
0.603
0.043
CLIP (VIT-L)
336
24
303.6
0.2228
74.9
0.060
1.2623
35.5
0.101
0.515
0.056
MetaCLIP
224
14
85.9
0.2580
69.4
0.047
1.3427
31.9
0.088
0.576
0.043
SigLIP-224
224
14
85.8
0,2898
67.1
0.047
1.3355
33.0
0.088
0.598
0.043
SigLIP-384
384
0.2137
73.9
0.047
1.2565
35.4
0.088
0.568
0.043
SigLIP-512
512
32
86.5
0.2203
75.7
0.047
1.2435
36.3
0.088
0.565
0.043
SigLIP-SO
512
36
413.7
0.1908
77.9
0.066
1.1737
38.8
0.107
0.488
0.062
Aim2
336
309.6
0.2244
75.2
0.000
1.1957
37.5
0.101
0.498
0.056
ViTamin
384
24
333.0
0.1843
77.9
0.060
1.1757
37.9
0.101
0.492
0.056
Table 4: Local semantics results on Pascal and COCO Stuff as well as spatial understanding on NYUv2 (right). I: image size. F: Size of feature volume. P and Pieam: Number of all and learnable parameters, respectively, in millions. Metrics: cross-entropy (CE), intersection over union (IoU), mean-squared error (MSE)
8

---

Published in Transactions on Machine Learning Research (09/202))
1.0
relaivv task perfor arreere
0.9
relative perfornance
0.8
0.8
0.6
0.7
0.4
0.6
=cb), det.: 0.35 (0.206)
ebb.. det.: 0.27 3..)
ebbl. det.: 0.21 (0.392)
0.5
Inst.
supervised
sem, 0eg.0.55 (0.035)
sem, wwg.0.61.......... (9.023)
seem. seg. (0.000)
Boundaries
self supervised
ddephh: 10.177
deepth (0.035)
deepth
0.4
X
Obj. Det.
vision-language
0.0
10:
25 spatial feature size
500
1000
log runtime factor over VIT-8/16
input image resolution
parameters 1eB
(a) Self-supervised methods consistently outper- form vision-language methods across all three in- stance awareness tasks.
(b) Spearman rank correlation of relative performance to spatial feature size, input image resolution and number of parameters. P-values are shown in brackets (with multiple comparison cor- rection using the Bemjamini-Hochber method).
Comparison of relative task performance. To show multiple tasks in single figure, we report the relative task performance, for which the original scores were mapped linearly into the interval [0,1].
Figure 4:
5.1 Comparison of backbone performance
A main goal of our work is to provide a fair comparison of different backbones regarding instance discrim- ination, local semantics and spatial understanding. We aim to highlight strengths and weaknesses each backbone. Results (Tab. 3) indicate that DINOv2 has the best instance awareness. This finding is consistent with recent work on whole-image classification (Oquab et al., 2023) and other tasks such as physics (Zhan et al., 2024) and 3D understanding (El Banani et al., 2024; Bonnen et al., 2024). The backbone of SAM2 does not outperform other backbones considerably, this is surprising given that it was trained instance discrimination. It suggests SAM2's mask decoder (which we did not use here) is crucial component. In- terestingly, self-supervised methods consistently outperform vision-language models in instance awareness (Fig. 4a). Despite following the same reconstruction--aase. training, MAE performs substantially better than Hiera in instance awareness while the opposite is true for semantic segmentation and depth estimation. Among the vision- lision-language models CLIP, MetaCLIP and SigLIP we did not find meaningful differences. We investigated the effect of backbone spatial feature size, backbone input resolution (images had a constant resolution and were upscaled to native backbone resolution for fairness) and number of parameters of the backbone by considering the Spearman rank correlation (Fig. 4b). While we do not find significant results for object detection, both semantic segmentation and depth benefit from larger feature volumes, better in- put resolution and more backbone parameters. Espocially, semantic segmentation and depth show a strong correlation of 0.87 and -0.79 to the number of backbone parameters.
Vision-language models tend show stronger local semantics (Tab. 4). While all vision-language models with 224px input size show similar performance, the larger versions of SigLIP (i.e. 384px and the shape optimized SO version) perform better but at higher cost. Also in this evaluation, DINOv2 achieves the best scores. All things considered, possibly the most striking finding is the dominance of DINOv2. While one might argue that this is due to large image sizes and feature volumes, the mediocre performance of SigLIP-512, Hiera-B+ and (partly) SAM V2 show that cannot be the only factor.
The evaluation on spatial understanding shows mixed results. Larger backbones tend perform better, with the exception of Hiera-B+. Again, DINOv2 performs best, in this case by a large margin.
5.2 DeAP performance correlates with performance of abject-centric methods and related benchmark
Next, we relate our findings to previous work. Specifically, we consider previous work in object-centric representation learning and classffcation-based evaluation (Fig. 5). Object-centric learning shares the goal of disentangling instances but achieves this through specific model architectures, whereas evaluate

---

Published in Transactions on Machine Learning Research (09/2025)
Kydemir et
Aydemir et BinsV2p44
DINOSAUR HBE
Battle of Backbores
CV-Bench-20 Frgtil-n
CV-Bench-3D
bost Hinedureed
60
44
27.5
51.0
25.0
FS-RR
50.5
score
Bine V2 SYTR 303000
HHAE
40
50.0
54
CUP
30
49.5
52
0.40
0:45
0.50
0.45
40
45
50
40
45
50
0.7
0.8
0.5
0.5
Insi. Disar. AUU
Inst Dact. ANL
Irst. Diace AMU
Inst. Discr. ARl
Pascaltou
RISE (depth:)
Figure 5: We compare our results with reported scores on two object-centric learning methods (three leftmost panels) (Aydemir et al., 2023; Seitzer et al., 2023)), fine-tuned Faster R-CNN backbones (Goldblum et al., 2024) and CV-Bench 2D and 3D (Tong et al., 2024).
model-agnostic features for instanco-specifie signals. In these experiments we compare how well the object- centric scores match our instance discrimination scores for the same backbones. Relating to the instance clustering foreground adjusted rand-index, which indicates how well instances are disentangled, by Aydemir et al. (2023) we find an almost linear relationship between their and our instance discrimination scores. Also compared DINOSAUR (Seitzer et al., 2023), we find an almost linear relationship over three backbones (Dino ViT-S, Dino ViT-B and MAF).
The evaluation of Goldblum et al. (2024) shares the goal of characterizing current backbones with our work but puts more emphasis their and our scores is recognizable, the trend is less pronounced than in previous experiments. CV-Bench- 2D and CV-Bench-3D (Tong et al., 2024) were recently introduced to assess the capabilities of multi-modal language models and provide scores for several vision backbones. While the variance is larger, we can see the same trends again: For the semantic score of CV-Bench- 2D there is a positive correlation with our semantic segmentation measurements while there negative correlation between the spatial scores of CV-Bench-3D and our monocular depth prediction errors (where lower is better). In summary, our method can be used to obtain similar insights on relative backbone performance as more expensive and slower evaluation methods.
aat-of-distribution and backbone architecture. While positive relation between
5.3 Ablation on readout design
We next explore design choices of our dense attentive probing readout by varying relevant hyperparameters (Tab. 5). The internal dimension of the mothod correlates positively with performance: while using 24 dimensions instead of 16 (base) improves performance but also requires 50 pp more parameters. An analogous observation can be made for the 8 dimensional version. The introduction of the parameter and its adaptivity per head is crucial for good performance as shown by the performance drop in versions where no masking is used ('no masking', i.e. attention between all locations is possible) and learnable is shared for all heads ('no indiv. o²) This suggests that information is organized at different levels of locality in the feature volumnes Ignoring all query information and thus attending only based on masks "only-mask') works well in some cases, in particular for the MAE backbone. The approach seems robust with respect to the number of heads, as the 4-head version performs similarly. The relevance of the feedforward network (FFN) varies across backbones and tasks in some cascs quite strongly. The CNN (7) accounts for small performance improvements but can be seen as non-critical component.
5.4 Comparison to dense readout baselines
Next, we ask whether alternative architectures with a similar parameter budget could perform competitively to dense attentive probing. To investigate this question, we implement three dense readouts capable of obtaining a high resolution prediction based on a low-resolution feature volume F(x) of size (HW, D). Specifically, we compare against the following baselines:
Bicubic Interpolation (Bicubic): A natural choice to increase resolution to interpolate in the feature volume F. This baseline replaces only the cross-attention of our model by projection to a low dimension followed by non-learnable bieubic interpolation operation. The rest of the baseline,
10

---

Published in Transactions on Machine Learning Research (09/2025)
rel.
Object Detection: AP
Semantic Segmentation: IoU
Depth: RMSE
names
Params.
Dino SigLIP-384
MAE
Dino V2 SigLIP-384
MAE
Diuo V2
SigLIP-384
MAE
base
100%
0.2666
0.1836
0.2662
83.4
73.9
60.2
0.4248
0.5684
0.5804
8-dim
51%
0.2448
0.1722
0.2529
82.8
70.4
55.6
0,4279
0.6041
0.5592
24-dim
154%
0.3150
0.2103
0,2871
84.0
75.3
60.2
0.4161
0.5746
0.5453
4 heads
100%
0.3027
0.1971
0.2633
84.9
74.2
59.4
0.4283
0.5894
0.5536
No masking
100%
0.1791
0.1196
0.1712
50.5
38.1
35.1
0.5992
0.7132
0.6801
No indiv.
100%
0.2409
0,1520
0.2429
83.8
70.2
58.6
0.4670
0,6070
0.5864
No FFN 94%
0.2409
0.0277
0,1234
83.7
73.2
56.4
0.4429
0,5829
0.5650
No CNN 97%
0.2969
0.1876
0.2632
83.2
73.4
59.7
0.4269
0.5711
0.5527
Table 5: Ablation of model parameters and components. The base model is the variant we use all other experiments. We vary our model's internal dimension (-dim), number of attention heads, use of masking, use of individual o, and removal of FFN or CNN 7.
Object Detection (COCO)
Semantic Segmentation (VOC)
Depth (NYUv2)
0.86
0.52
(ours)
0.30
0.84
0.50
Bicublc
AP1
0.80
0.48
CNN
0.70
0.45
FeatUp
0.20
0.78
0.44
0.15 parameters (M)
parameters (M)
parameters (M)
Figure 6: Dense attentive probing outperforms CNN, interpolation and Feat Up-bsed readouts Dino features (instances: = better; semantic segmentation f; depth 4). For semantic segmentation and depth estimation the lines represent averages over three runs. For the largest. bicubic run on object. detection we report the best score three runs due to training instability.
ie. the MLP and CNN, identical dense attentive pooling to isolate the effect of replacing the cross-altention.
Transposed Convolutions (CNN): A common component in many dense prediction architectures are transposed convolutions. This operation reverses downsampling by applying learned filters that generate spatially larger outputs. Analogous to the bicubic interpolation baseline, we replace the masked cross-attention with single transposed convolution layer, while use the same CNN for upsampling. A disadvantage of this baseline is that the output resolution depends the feature volume resolution, c.g. increasing the feature volume from by a factor of two would also increase the output size by a factor of two.
Recently, the FeatUp (Fu et al., 2024) method was introduced that up-scales feature volume under consideration (conditional on) of the input image. We employ this image-aware upsampling technique to up-scale the feature volumes F(x) and add a linear projection that maps to the task-defined output space.
Fea
We find dense atlentive probing to be more parameter elficient and to achieve better scores than the baselines across all three tasks (Fig. 6). An additional advantage of our method over CNNs is decoupling input and CNN, a larger feature volume size would result larger output. FeatUp is highly parameter efficient but has high memory demands and requires long computation times (factor compared to dense attentive probing).
output resolution,
11

---

Published in Transactions on Machine Learning Research (09/202))
Training setting
fcature Sze
10x10
90xB0
Figure 7: After training, our readout can be queried to output different resolutions from the same backbone. Here we use DINOv2 backbone trained on Pascal VOC.
Object Detection
Local Semantics
Depth
0.85
0.250
0.80
0.65
4P large
0.225
NcCo
Hert 69
0.75
vilaiin
50. P.50
0.60
0.200
SgUF
0.55
0.175
PNCECUP Squu--224
1lera
0.50
0.150
CLP
MacdPge Wetaka
0.45
-Dio V2
Divo Mil
25
30
15
20
25
10
runtime factor over VIT-B 224
runtime factor over WIT-B 224
runtime factor over VITB 224
self-supervised
vision-language
Figure 8: Inference speed over performance on three tasks, relative to the VIT-B/16 with 224px image input (the fastest and most frequent architecture in our evaluation). The lines depict the pareto fronts.
5.5 Variable output size
To ensure a fair comparison, the readout size is fixed in the previous experiments. However, it is possible generate outputs at an arbitrary resolution, because the adapter takes positions as inputs (similar implicit neural fields). Instead of using the standard query position grid for 224 px output, can sample different query coordinates at test time (Fig. 7). The results show that our method generalizes well to higher resolutions, with some details being resolved better at higher resolutions (see for example the head of the horse and the chair).
5.6 pereedddeffommacce tradeoff
For practitioners, runtime limitations constrain the selection of viable backbones, for example when minimal latency is required. Therefore, we explore the relationship between task performance and inference speed in more detail (Fig. 8). We measure the time to run len batches with eight samples each in inference mode (i.e. e. without gradient computation). The fastest model in our evaluation set is the VIT-B/1K 16 at a resolution of 224 pixels. Therefore, we indicate the factor by which the runtime extended with respect this model. For example, the slowest model, SigLIP-SO takes around 30 times as long as the ViT-B/16 reference model. The results show that, depending the task, there fast models that achieve good
12

---

Published in Transactions on Machine Learning Research (09/2025)
performance, namely MAE for object detection and Hicra B+ for depth prediction. For local semantics we find a stronger dependency between model sizes and performance, i.e. larger models are required for good performance.
6 Conclusion
In this work we proposed dense attentive probing, fast. and parameter-ellicint readout for evaluating the representational expressiveness of trained backbones dense prediction. For example, our standard training for readout on ViT-B/16 224 pixel backbone Pascal VOC adds less than 70,000 parameters and trains in less than 16 minutes (using single Nvidia RXT2080 GPU). We used dense attentive probing to systematically analyze common vision backbones with respect to the three complementary aspects: instance awareness, depth and local semantics. Our results suggest that the backbones of the DINOv2 family are highly capable: is the best ViT-B model across all experiments. This finding is in line with prior rescarch on complementary tasks (Oquab et al., 2023; Zhan et al., 2024; El Banani al., 2024; Bonnen et al., 2024). Using DINOv2 with ViT-I backbone improves performance further but at the cost of a three times longer runtime. Classic supervised image classification pretraining on ImageNet results in inferior performance compared to the self-supervised and vision-language paradigms. Excluding DINOv2, we identified the trend that local semantics is better captured by language-vision models while reconstruction-based self-supervised Icarning leads to features with better instance awareness. This indicates that combining both paradigms could be a promising direction for future research.
For practitioners, DINOv2 is natural choice if enough compute is available. For compute--cnnstraneed cases, the decision is more complex: MAE is recommendable for instance related tasks while CLIP- models (e.g. MetaCIIP) show good local semantics. For spalial understanding Hiera-B represents good trade-off between performance and inference speed. We plan to retain an online leaderboard where new backbones can easily be incorporated to help tracking future progress of dense prediction performance.
7 Limitations
While use fairly small readout (in terms of parameters) that adapt to multiple locality scales in the features through the learnable masks, even this readout has inductive biases and can favor certain backbones such that results might get distorted. We deliberately opted for small readout to directly measure the feature performance. Consequently, this limits the degree to which the features can be re- combined and processed. In fact, this could be the reason why the SAM2 backbone performs comparably poorly in instance awareness in our hands: it does rely on a relatively "heavy" decoder. A more direct comparison to object-centric approaches would be desirable, but is challenging as these approaches explicitly encode objects (e. g. in attention slots) which can be compared to ground truth.
The current selection of tasks we evaluated is limited to three broad categories and a few instances of those. Adding additional task categories (e. as in Taskonomy (Zamir et al., 2018)) would be desirable in the future for a broader characterization of backbones.
8 Acknowledgements
This publication was funded by the Deutsche Fosschungseemeinsccall (DFG, German Research Foundation) Project-ID 454648639 SFB 1528
References
Görkay Aydemir, Weidi Xie, and Fatma Guney. Self-supervised object-centric learning for videos. Advances in Neural Information Processing Systems, 36:32879 32899, 2023.
Randall Balestriero and Yann LeCun. Learning by reconstruction produces uninformative features for per- ception. XrXiv preprint ar arXiv:2402.114 11337, 2024.
13

---

Published in Transactions on Machine Learning Research (09/2025)
Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari S. Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Grégoire Mialon, Yuandong Tian, Avi Schwarzschild, Andrew Gordon Wilson, Jonas Geiping, Quentin Garrido, Pierre Fernandez, Amir Bar, Hamed Pirsiavash, Yann LeCun, and Micah Goldblum. cookbook of self-supervised learning. ArXiv, abs/2304.122I1 2023.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg; Variaance-nvvaiinneeccvvaaa regularization for self-supervised learning. International Conference on Learning Representations (ICLR), 2022.
Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mido Assran, and Nicolas Ballas. V-jepa: Latent video prediction for visual representation learning. 2023.
Tyler Bonnen, Stephanie Fu, Yutong Bai, Thomas O'Connell, Youi Friedman, Nancy Kanwisher, Joshua B Tenenbaum, and Alexei A Efros. Evaluating multiview object consistency in humans and image models. arviv preprint 2rx::22409.0m 05862,, 2024.
Christopher P. Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matthew M. Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. ArXiu, abs/1901.11390 2019.
Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. Coyo- 700m: Image-text pair dataset. https:// comm///akabbbrattcooooottaaa
2022.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. Tn Proceedings of the IEEE conference on computer vision and pattern recognition, 1209 1218, 2018.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVP international conference on computer vision, pp. (650-96600 2021.
Hao Chen, Ran Tao, Han Zhang, Yidong Wang, Xiang Li, Wei Ye, Jindong Wang, Guosheng Hu, and Marios Savvides. Conv-adapler: Exploring parameter ellicient transfer learning for convnets. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognétion, pp. 1551-1550 1551-156,, 2024a.
Jieneng Chen, Qihang Yu, Xiaohui Shen, Alan Yuille, and Liang- Ching Chen. Vitamin: Designing scalable vision models in the vision-language era. In Proceedings of the IEEE/CVI Conference on Computer Vision and Pattern Recognition, pp.
2024b.
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adapt- former: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35:16664-16678, 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for rastivvvvve learning of visual representations. In International conference on machine learning, pp. 1597-11978111 1607. PMLR, 2020.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVE /CVE international conference on computer vision, pp. 6640-9649, 2021.
Zhe Chen, Yuchen Dnan, Wenhai Wang, Junjun Hc, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. International Conference on Learning Representations (ICLR), 2023.
Duolikun Danier, Mehmet Aygün, Changjian Li, Hakan Bilen, and Oisin Mac Aodha. Depthcues: Evaluating monocular depth perception large vision models. Tn Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 20049 20059, 2025.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. International Conference on Learning Representations (ICI.R), 2024.
14

---

Published in Transactions on Machine Learning Research (09/2025)
Aniket Didolkar, Andrii Zadaianchuk, Anirudh Goyal, Mike Mozer, Yoshua Bengio, Georg Martius, and Maximilian Seitzer. Zero-shot object-centric representation learning. International Conference on Learning Representations (ICLR), 2025.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image worth 16x16 words: Transformers for image recognition at scale. International Conference Learning Repre- sentations (ICLR), 2021.
Mohamed EI Banani, Amit Raj, Kevis-Kokits Maninis, Abhishek Kar, Yuanzhen Li, Michael Rubinstein, Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani. Probing the 3d awareness of vi- sual foundation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21795-21880 2024.
Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 111:98-158 106,136,2220 2015.
Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. International Conference on Learning Representations (ICLR), 2024a.
Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt,, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. International Conference on Learning Representations (ICLR), 2024b.
Enrico Fini, Mustafa Shukor, Xiujun Li, Philipp Dufter, Michal Klein, David Haldimann, Sai Aitharaju, Victor Guilherme Turrisi da Costa, Louis Béthune, Zhe Gan, Alexander T Toshev, Marcin Eichner, Moin Nabi, Yinfei Yang, Joshua M. Susskind, and Alaaeldin El-Nouby. Multimodal autoregressive pre-training of large vision encoders, 2024.
Stephanie Fu, Mark Hamilton, Laura E. Brandt, Axel Feldmann, Zhoutong Zhang, and William T. Freeman. Fealup: A model-agnostic framework for features at any resolution. In International Conference Learning Representations (ICLR), 2024.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jicyu Zhang, et al. Datacomp: In scarch of the next gener- alion of multimodal datasets. Advances in Neural Information Processing Systems, 36, 2024.
Micah Goldblun, Hossein Souri, Renkun Ni, Manli Shu, Viraj Prabhu, Gowthami Somepalli, Prithvijit Chattopadhyay, Mark Ibrahim, Adrien Bardes, Judy Hoffman, et al. Battle of the backbones: A large- scale comparison of pretrained models across computer vision tasks. Advances in Neural Information Processing Systems, 36, 2024.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 16000 16009, 2022.
Neil Houlsby, Andrei Giurgin, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pp. 2790-220 2799. PMIRR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. International Conference Learning Representations (ICLR), 2022.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pp. Springer, 2022.
LATON-AL. Clip benchmark. https://githin httpps///gtttbbb.cmmAAAAAAAAAAAA htt.a///AALLL..ppbcarr....
benchmark, 2022.
15

---

Published in Transactions on Machine Learning Research (09/2025)
Martha Lewis, Nihal V. Nayak, Peilin Yu, Qinan Yu, Jack Merullo, Stephen H. Bach, and Ellie Pavlick. Does clip bind concepts? probing compositionality in large image models. ACL, 2024. URL https: //arxiv. (aaxxi.orgg/aaeee wttgg/////2...///////2 10537.
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In European conference on computer vision, pp. 280-222022 29002996.220 Springer, 2022.
Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang- Scaling & shifting your feal ures: A new baseline for efficient model tuning. Advonces in Neural Information Processing Systems,
2022.
Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft. coco: Common objects in context. In European Conference on Computer Vision, 2014.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11976 11986, 2022.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkorcit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in neural information processing systems, 33:11525 11538, 2020.
Yunze Man, Shuhong Zheng, Zhipeng Bao, Martial Hebert, Liangyan Gui, and Yu-Xiong Wang, Lexicon3d: Probing visual foundation models for complex 3d scene understanding. Advances in Neural Information Processing Systems, 37:76819 76744, 2024.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 4460 4470, 2019.
Krystian Mikolajezyk and Cordelia Schmid. A performance evaluation of local descriptors. IEEE transactions on pattern analysis and machine intelligence, 27(10))1615-163) 2005.
Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012.
David Novotny, Samuel Albanie, Diane Larlus, and Andrea Vedaldi. Semiiconvolutionall operators for in- stance segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 86-102, 2018.
Maxime Oquab, Timoth'ee ee Darcet, Th'eo Moutakanni, Huy Q Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacliin El-Nouby, Mahmoud Assran, Nicolas Ballas, Woj- ciech Galuba, Russ Howes, Po- Yao Huang, Shang- Wen Li, Ishan Misra, Michael G. Rabbat, Vasu Sharma, Gabriel Synnaeve, Huijiao Xu, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. ArXiv, abss/3304.06
2023.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 165-174, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zemiing Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Ofrr Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. International Conference on Learning Representations (ICLR), 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Πya Sutskover. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
16

---

Published in Transactions on Machine Learning Research (09/2025)
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint S.fe:2408.00711 2024.
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster r-cmn: Towards real-time object detection with region proposal networks. IEEE Transactions on Pottern Analysis and Machine Intelligence, 39: 1137-1149, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenct large scale visual recognition challenge. International Journal of Computer Vision, 115:211 1-222 2014.
Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaced, Judy Hoffman, et al. Hicra: A hierarchical vision transformer without the bells-and-whistles. International Conference on Machine Learning (ICML), 2023.
Shunsuke Saito, Zeng Huang, Ryota. Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel- aligned implicit function for high-resolution clothed human digitization. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 2304-2314, 2019.
Christoph Schuhmamn, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Open dataset of clip-filtered 400 million imago-text pairs. Data Centric AI NeurIPS Workshop, 2021.
Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominík Zictlow, Tianjum Xiao, Carl-Johann Simon- Gabriel, Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, et al. Bridging the gap to real-world object-centric learning. International Conference on Learning Representations (ICLRR, 2023.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Sclf-attention with relative position representations. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Línguistics: Human Language Technologies, Vol- ume 2 (Short Papers), pp. 464-468, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1 10153//11/188-. /v/11112214
Jan-Martin O Steitz and Stefan Roth. Adapters strike back. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 23449 23459, 2024.
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition, pp. 5238 023355248, 2022.
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Ji- han Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centrie exploration of multimodal 1lms. Advances in Neural Information Processing Systems (NeurIPS), 2024.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Neural Information Processing Systems, 2017.
Marissa A. Weis, Laura Pede, Timo Lüddecke, and Alexander S. Ecker. Self-supervised representation learning nouronal morphologies. Transactions on Machine Learning Research (TMLR), abs22112.1222 2023.
Ross Wightman. Pytorch image models. https://github: http://hhh.....oe....aaaaa/aarra
2019.
Yi Xin, Siqi Luo, Xuyang Liu, Haodi Zhou, Xinyu Cheng, Christina E Loe, Junlong Du, Haozhe Wang, MingCai Chen, Ting Liu, al. V-petl bench: A unified visual crrameter-efficien transfer learning benchmark. Advances in Neural Information Processing Systems, 37:80522- -00525, 2024.
17

---

Published in Transactions on Machine Learning Research (09/2025)
Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Yo- Huang, Russell Howes, Vasu Sharma, Shang Wen Li, Gargi Ghosh, Luke Zeutlemoyer, and Christoph Feichtenhofer. Demystifying clip data. In International Conference on Learning Representations (ICLR), 2024.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from few images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4578 4587, 2021.
Bruce XB Yu, Jianlong Chang, Haixin Wang, Lingbo Liu, Shijic Wang, Zhiyu Wang. Junfan Lin, Lingxi Xie, Haojie Li, Zhouchen Lin, et al. Visual tuning. ACM Compuling Surveys, 56(12):1-35 2024.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research (TMLR), 2022.
Andrii Zadaianchuk, Maximilian Seitzer, and Georg Martius. Object-centric learning for real-world videos by predicting temporal feature similarities. Advances in Neural Information Processing Systems, 36, 2024.
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling lask transfer learning. In Proceedings of the IEEE conference on compuler vision and pattern recognition, pp. 9712-3722, 2018.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of repre- sentation learning with the visual task adaptation benchmark. ar preprint ar Xrx:::1910. 190,0.0700,,7, 2019.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11975- 11986, 2023.
Guanqi Zhan, Chuanxia Zheng, Weidi Xie, and Andrew Zisserman. A general protocol to probe large vision models for 3d physical understanding. Advances in Neural Information Processing Systems, 37: 43468-43499. 2024.
Zihan Zhong, Zhiqqiang Tang, Tong He, Haoyang Fang, and Chun Yuan. Convolution meets lora: Parameter efficient finctuning for segment anything model. International Conference on Learning Representations (ICLR), 2024.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong, ibol: Image bert pre-training with online tokenizer. International Conference on Learning Representations (ICLR), 2022.
Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects points. ar preprint arXi::1904.075: 2019.
18

---

Published in Transactions on Machine Learning Research (09/2025)
A Appendix
A.1 Implementation
We use the Adam optimizer with learning rate of 0.001, except for boundary prediction and depth where it is set to 0.002. We use 8 attention heads in all models. On COCO and Pascal we use the validation sets for testing, while model selection is carried out on a separate part of the training set via validation loss.
BS
LR
WD
iterations
val intv.
img size
heads
dim
base size
Pascal VOC2012
32
0.001
0.010
6000
250
224
8
16
28
COCO Stuff
32
0.001
0.010
20000
250
224
8
16
28
NYUv2 Depth
32
0.001
0.010
3000
100
[216,
288]
8
16
28
Instance Discrimination
32
0.001
0.010
20000
-1
224
8
16
28
Boundariers
32
0.001
0.010
10000
250
8
CenterNet
32
0.001
0.010
20000
1000
448
8
16
56
BS,LR and WD correspond to batch size, learning rate and weight decay respectively.
A.2 Comparison in higher resolution
Some the evaluated backbones support higher native resolutions. Here we increase the provided resolution 448 448 and 432 576 for semantic segmentation and depth respectively. We find small improvements in performance for SigLIP-512 and DinoV2 due to the increased resolution.
NYUv2 Depth RMSE
Pascal VOC2012 IoU
Model
Inp. size
Original resolution
SigIIP-551
512
0.5648
75.7
ViTamin
384
0.4916
77.9
Dino V2
518
0.4248
83.4
Higher resolution
SigLIP-512
512
0.5571
78.0
ViTamin 384
0.4945
77.2
Dino V2
518
0.4208
85.0
A.3 Inference Speed
In Fig. 9 we show the inference speeds relative to the fastest model (VIT-B/16).
A.4 Feature Visualization
We visualize the backbones output features (Fig. 10) by reducing the number of feature dimensions three and interpreting these three dimensions as RGB color.
19

---

Published in Transactions on Machine Learning Research (09/202)
8
Random (untrained)
4
Dino V2 (VTT-L)
SAM V2 B+
Ima maggeNet
Dino V2
MetaCLip
SigLIP:512
sigLiP 50
Vilamin
Dino
MAE
Hiera
CUP
CLiP
Sigi
Aim2
Figure 9: Inference speed of selected methods. Light bars on the top represent runtime of the dense attentive probing.
Randoe bystrnest medh
SCN 94
MACO VS
Dita VOL
Diln VO PETTA
MAG:
Ba: #3627
CUP
224- 788/20
790 14
2224/22
758754
758726
SLLP-512 512
Stu-P-50 1152
Amc:
Condiete 1359
ConvMett
by:
Figure 10: PCA Visualization of the backbone features. The numbers below the backbone names denote feature dimension spatial size of the feature volume.
20


Reviews:
[{"review_id": "kc1DpK9AYd", "reviewer": "Reviewer_EEFu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n+ This paper proposes an efficient dense attentive probing strategy to evaluate the feature quality of pre-trained backbones for dense prediction tasks. Evaluating the feature quality of pre-trained backbone is an important and interesting topic to compare different self-supervised learning methods.\n+ The dense attentive probing is adaptive to arbitrary backbones and independent of the size and resolution of their feature volume, showing a better trade-off between efficiency and efficacy compared with the other readout architectures.\n+ This paper finds that DINOv2 outperforms all other backbones tested.\n\nCons:\n\n1. The motivation in the Introduction is not easy to understand. For instance, please first define what is the \"variable feature locality\", then please mention why it is important for assessing feature quality. After that, please explain how you come up with the idea of using a learnable masking radius (rather than other strategies) to enforce the feature locality.\n\n2. In the Introduction, please also clarify why the proposed readout can better handle the evaluation for SSL methods along three dimensions. It would be better to clearly present the connection between the readout and these three dimensions first.\n\n3. In Fig. 1, how to construct the \"Positions\" of [H,W,2] as the input (i.e., how to obtain the values at each position in H*W*2)? What is the relation between the \"Positions\" and positional encoding (PE) of the grid coordinates p in Eq. (1)?\n\n4. In Sec. 3, why can M_{ij} consider \"spatial proximity\"? The design of this mask is not easy to understand. Why can we directly add M to the cross-attention rather than multiplication?\n\n5. Please give more details in the settings of Tab. 5, e.g., what does \"only-mask\" mean? Which one denotes only cross-attention without the mask M?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes dense attentive probing to evaluate the feature quality of pre-trained backbones for dense prediction tasks like segmentation. The proposed probing is a parameter-efficient readout, which exploits a masked cross-attention layer with learnable mask sizes, to make dense predictions using arbitrary backbones independent of the size and resolution of their feature volume. Based on the proposed probing, this paper finds that DINOv2 outperforms all other backbones tested.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the questions in Cons and revise the manuscript accordingly.", "broader_impact_concerns": "There is no concerns on the ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Flzz67CmBc", "reviewer": "Reviewer_tki8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n* I think that the core idea is well-motivated and the designed module generally facilitates a fair comparison among backbones, given that in the conclusion the author proposes to maintain a leaderboard to further incorporate more, and new backbones I believe that this method could aid further research in this area.\n\n* The execution of the idea is decent with comprehensive experiments.\n\n**Weaknesses**\n\n* I could not find major flaws in this work. Rather I am more curious as to whether the findings summarized in the conclusion (including identifying that the DINOv2 is of a higher performing model, while conventional backbones trained on classification are less effective, and vision-language models could better capture local semantics) are entirely new findings -- it seems that these topics may have been well covered in previous works, or are somewhat a general consensus among the research community, though as I do not directly work on the domain of, say, linear probing and related methods, I cannot definitively confirm this suspicion.\n\n**Additional Comments**\n\nAs I do not directly work on this particular topic, I will later observe other reviewers' comments and potentially add more comments to my review.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies how to probe (efficiently assess) the feature quality given a backbone model in the context of dense prediction, which differs from conventional paradigms such as linear probing wherein the entire features are assessed for, e.g., image classification. The author proposes a method that is agnostic of the resolution of the feature.\n\nThe author has conducted rather comprehensive experiments to assess the validity of the proposed method on top of various backbones. The assessments comprise of instance awareness, local semantics and depth semantics.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The author could consider addressing the minor concerns I raised above.", "broader_impact_concerns": "No broader impacts were identified.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SKafVdLHGp", "reviewer": "Reviewer_AftN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. Establishing a probing protocol for evaluating pretrained vision models for dense prediction is important and timely\n2. There are experiments using the proposed probing approach to evaluate a variety of different vision models\n3. The introduction is well-written and compelling\n\n**Weaknesses**\n1. The writing and presentation need to be improved considerably\n  - There seems to be a mismatch between the related work and the goals specified in the introduction\n  - The figures and tables are hard to understand and do not really connect with main paper text\n2. Limited discussion of relevant prior work\n  - No discussion of simpler efforts on dense probing (e.g. dense probing in DINOv2)\n3. Due to the limited discussion of prior work, it was not clear whether this work compared to the appropriate prior methods and whether this work is this better in some way.  It's also not clear whether the selected tasks is better than evauating using more standard instance segmentation or 3D scene understanding for spatial understanding. \n4. Due to Table 1 (which is not referenced) and the section devoted to parameter efficient fine-tuning methods, the position of this paper became unclear: is this a paper focused on an adapter method for dense prediction or a probing paper?\n5. It's unclear whether the baselines in section 4.2 are true baselines or should they be considered as alternatives to upsampling the readout.  \n6. Limited discussion of the impact of various design choices.  What parameters are selected just for the experiments in this paper vs which parameters are recommended because of some desirable features?\n7. The experiments use images of size 224x224 for all tasks, and for image backbones that have higher native resolution, the images are scaled up.  As the outputs are also at 224x224, this setup does not seem to take advantage of the higher resolution offered by some of the backbones.  Would it be make more sense to separately evaluate backbones that are higher resolution with higher resolution inputs and outputs?\n8. The reviewer found the experimental results difficult to follow and the figures and tables hard to interpret\n  - For instance, there was too many different factors in Table 4 to see if there were any conclusions as to what at the important factors to having a good pretrained image model (training dataset, pretraining objective, image resolution, etc.).  It was also not clear what the row for ImageNet was - is that the original ViT model with supervised training on ImageNet?  Figure 3 was more useful.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes an architecture and methodology for probing vision models for dense prediction tasks consisting of 1) an dense attentive probe (DAP) and 2) a set of dense prediction task.  The dense attentive probe takes a grid of positional queries, applies attention with relative position bias to features from the vison backbone, and the upsamples that using a CNN to obtain a final dense readout.  The DAP is trained on a set of dense prediction tasks consisting of 1) instance awareness tasks (boundary detection, center detection, and instance discrimination on MS COCO), 2) local semantics - semantic segmentation (on Pascal VOC 2012 and COCO stuff), and 3) spatial understanding - monocular depth understanding on NYUv2.  Experiments are conducted to compare different pre-trained ViT-based vision models covering a range of training objectives (e.g. MAE, DINO, SigLIP, etc).\n\nThe main contributions are 1) architecture for probing vision models for dense prediction, 2) set of tasks for training the dense attentive probe, and 3) set of experiments comparing different pretrained models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The reviewer believe that considerable improvements to writing and presentation is needed before the word is ready for publication.\n\n1. The related work need to be considerably improved\n- For instance, there is a lot of information about PEFT, but it is not clear how relevant those methods are to this work.  \n- This reviewer did not found the section on \"Feature evaluation\" to be that relevant or informative.  Instead, the reviewer expected to see more on alternative methods for dense probing, feature evaluation (e.g. recent efforts in evaluating visual backbones [1,2] ) and potentially work on interpretability (e.g. GradCam [3] for visualizing features).   \n- It would be good to include a discussion between this work and work on direct image segmentation (e.g why would those studies be not appropriate for dense probing)\n2. Table 1 is not referenced\n3. Figure 1 is hard to follow and match with the caption\n- It's not clear what parts of the figure depict the queries, is the small CNN the CNN gamma, where are the dense readout, Gaussian function, etc.  \n- The attention part is also not described, even though there is is large parts of figure devoted to it\n- Use symbols in caption to connect to figure, and add names of components to the figure\n4. Table 2 is hard to understand.  Table needs columns headers.  What does the entry \"Supervised\" mean in the Supervised table?\n5. Figure 2 - why only include the instance understanding tasks?  Why not include the semantic segmentation and monocular depth estimation task? \n6. Section 3, when introducing $M(\\sigma)$, the key idea should be introduced vs just introducing the notation.  It seems also appropriate to relate this to other work that uses biases with attentino (e.g. ALiBi [4]).\n7. The experiment results (section 5) should be improved to have a better overview of the sections and the experiments.  While the tasks and a set of baselines was described in section 4, section 5 was hard to follow as it jumped into the results.  One issue is that section 4.2 (baselines) was introduced and then there was a set of experiments without considering this baselines.  There should be more context provided for each set of experiments (e.g. for section 5.2, why the specific models are selected, what does the different conditions mean, what is being measured, etc.)\n8. There should be more discussion of 1) design choices in probing approach (e.g. which parameters should be adjusted for different use cases), and 2) how the findings of this work relates to findings from prior work. \n\n**References**\n[1] V-PETL Bench: A Unified Visual Parameter-Efficient Transfer Learning Benchmark [Xin et al. NeurIPS 2024]\n\n[2] Lexicon3D: Probing Visual Encoding Models for Complex 3D Scene Understanding [Man et al. NeurIPS 2024]\n\n[3] Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization [Selvaraju et al. CVPR 2017]\n\n[4] Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation [Press et al. ICLR 2022]", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)