Attention, Please! Revisiting Attentive Probing Through the Lens of Efficiency
Authors: Bill Psomas
Dionysis Christopoulos
Eirini Baltzi
Ioannis Kakogeorgiou
Tilemachos Aravanis
Nikos Komodakis
Konstantinos Karantzalos
Yannis Avrithis
Giorgos Tolias
Venue: ICLR 2026 Poster (2026)
Keywords: Linear Probing
Attentive Probing
Vision Transformers
Vision-Language Models
Evaluation
Neural Networks
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: Full fine-tuning is impractical at scale, while linear probing overlooks the potential of models whose pre-training optimizes local representations. As an alternative, we propose efficient probing (EP), a lightweight cross-attention mechanism.

Abstract: As fine-tuning becomes impractical at scale, probing is emerging as the preferred evaluation protocol. However, standard linear probing can understate the capability of models whose pre-training optimizes local representations rather than an explicit global representation. This motivates attentive probing, an alternative that uses attention to selectively aggregate patch-level features. Despite growing adoption, attentive probing is still underexplored: existing approaches are often over-parameterized and computationally inefficient. In this work, we revisit attentive probing through the lens of the accuracy vs. parameter-efficiency trade-off. We present the first comprehensive study of existing methods, analyzing their design choices and benchmarking their performance. Building on these insights, we propose efficient probing (EP), a lightweight yet effective multi-query cross-attention mechanism that eliminates redundant projections and reduces the number of trainable parameters. Across multiple benchmarks and pre-training paradigms, EP consistently outperforms linear probing and previous attentive probing methods, and remains effective when combined with parameter-efficient fine-tuning. Beyond evaluation, our analysis uncovers emerging properties of EP, including complementary attention maps, which open new directions for leveraging probing beyond protocol design. Project page: https://vrg.fel.cvut.cz/ep/.

Published as a conference paper at ICLR 2026
ATTENTION, PLEASE! REVISITING ATTENTIVE PROBING THROUGH THE LENS OF EFFICIENCY
Bill Psomas' Dionysis Christopoulos 2.
Eirini Baltzi² loannis Kakogeorgiou'
Tilemachos Aravanis'
Konstantinos Karantzalos Yannis Avrithis Giorgos Tolias'
Nikos Komodakis
1 VRG, FEE, Czech Technical University in Prague
National Technical University of Athens
University Crete
4 *Archimedes, Athena RC 1IACM-FORTH IIARAI
IIT, NCSR "Demokritos"
ABSTRACT
As fine-tuning becomes impractical at scale, probing is emerging as the preferred evaluation protocol. However, standard linear probing can understate the capa- bility of models whose pre-training optimizes local representations rather than an explicit global representation. This motivates attentive probing, alternative that uses attention to selectively aggregate patch-level features. Despite growing adop- tion, attentive probing is still underexplored: existing approaches are often over- parameterized and computationally inefficient. In this work, we revisit attentive probing through the lens of the accuracy parameter-efficiency trade-off. We present the first comprehensive study of existing methods, analyzing their design choices and benchmarking their performance. Building these insights, we pro- pose efficient probing (EP), lightweight yet effective multi-query cross-attention mechanism that eliminates redundant projections and reduces the number of train- able parameters. Across multiple benchmarks and pre- pre-training paradigms, EP con- sistently outperforms linear probing and previous attentive probing methods, and remains effective when combined with parameter-efficient fine-tuning. Beyond evaluation, our analysis uncovers emerging properties of EP, including comple- mentary attention maps, which open new directions for leveraging probing beyond protocol design. Project page:
INTRODUCTION
The past few years have witnessed remarkable progress in representation learning, with pre-training paradigms ranging from sulf-supe learning (Chen al., 2020; Caron al., 2021; Kakogeor- giou et al., 2022). to vision-language models (Radford et al., 2021; Jia et al., 2021), and auto- regressive architectures (El-Nouby et al., 2024; Fini et al., 2025). These diverse approaches share a common goal: learning rich, transferable visual representations that minimize reliance on task- specific labels and scale large datasets. Evaluating the quality of such pre-trained representations is therefore central to measuring progress. Conventional evaluation protocols include k-NN clas- sification, linear probing (LP), and full fine-tuning (FT). While k-NN and LP assess the quality of the learned representations under a frozen backbone, FT measures the utility of pre-training as ini- tialization for downstream tasks. Although FT achieves the highest performance, it increasingly viewed as unsustainable and prohibitive at scale (Xin et al., 2024; Shuttleworth et al., 2024; Zou et al., 2023). As a result, probing is emerging as a practical evaluation protocol (Oquab et al., 2024; Bardes et al., 2024; Darcet et al., 2025).
However, probing protocols remain misaligned with many pre-training approaches. Standard L.P typically relies single global representation. e.g., the [CLS], which is well-suited for archi- tectures trained with global objectives, but poorly reflects the potential of models such as masked image modeling (He et al., 2022), autoregressive (EI-Nouby et al., 2024), diffusion (Ma et al., 2024), where valuable information is distributed across local representations. This gap motivates the rise of attentive probing (Darcet et al., 2025; Chen al., 2023; Bardes al., 2024). Despite its promise, attentive probing remains underexplored. Existing methods vary significantly in design,
"Equal contribution

---

Published as a conference paper at ICLR 2026
often suffering from excessive parameterization and inefficiency. Moreover, the connection between how attention aggregates features and why it improves predictive performance remains unclear.
In this work, we address these limitations by conducting the first comprchensive study of attentive probing, revisiting its design through the lens of the accuracy vs. parameter-effiienny trade-off. We introduce unified framework that encompasses wide range of attention-based aggregation methods iinluudig those proposed for probing (EI-Nouby et al., 2024; Bardes et al., 2024; Darcet al., 2025) and others from unrelated tasks (Psomas et al., 2023; Noh et al., 2017; Rymarczyk et al., 2021). Through this framework, we derive efficient probing (EP), a simple multi-querr cross- attention mechanism that eliminates redundant projections, reduces parameter count and computa- tional cost, while matching or surpassing prior state-of-the-ar performance. Moving beyond the parameter-effiieent probing (PEP), we compare EP against parameter-efffcient fine-tuning (PEFT) methods. We find that EP remains beneficial even when PEFT is allowed: combining HP with LoRA (Hu et al., 2022) yields configurations that dominate both pure probing and pure LoRA.
Beyond efficiency and accuracy, common ingredient of both existing methods and EP is the use of multiple independent attention predictors (e.g., heads or queries). We show that a predictor's con- tribution to classification accuracy correlates with its localization quality: predictors with sharper, foreground- focused attention drive larger accuracy gains. Rather than resorting to shortcut learning, e.g., leveraging background cues like water to classify a "fish", EP's predictors consistently attend to the object, improving interpretabilityy, robustness, and performance. Notably, the attention maps of EP are more diverse and complementary than those of existing methods, with different predictors specializing in distinct object regions. Our contributions are threefold:
1. We conduct the first systematic benchmark and analysis of attentive probing methods across diverse pre- pre-training paradigms, comparing their accuracy, efficiency, and design choices.
2. We introduce efficient probing (EP), which achieves state-of-the-art accuracy, while bring- ing substantial gains in compute, memory, and parameter efficiency.
3. We uncover a correlation between spatial localization and predictive performance, and show that EP produces diverse, complementary, and interpretable attention maps.
2 RELATED WORK
Self-supervised learning (SSL) has transformed visual representation learning, with evaluation typ- ically performed via (i) k-NN on frozen features, (ii) linear probing (LP) using a shallow classifier on frozen encoder, or (iii) fine-tuning (FT) the entire model. Although FT achieves the high- computationally expensive, motivating the evaluation under frozen backbones. Two dominant SSL paradigms are joint embedding architectures (JEA) and masked image mod- eling (MIM). JEA methods (e.g., DINO (Caron et al., 2021)) contrast or cluster augmentations to learn global representations via [ [CLS] token pooled features. In contrast, MIM methods (e.g., MAE (He et al., 2022)) reconstruct masked regions. yielding localized, mitchdisttrbuted representa- tions. This global vs. local distinction affects evaluation: LP is effective for JEA (Caron et al., 2021) but under-performs for MIM (He et al., 2022; Przewikezlikowski et al., 2025), where discriminative information not concentrated in single token.
est accuracy,
Beyond SSL, vision-language models (VLMs) pre-train on web-scale image-text corpora (e.g.. LAION (Schuhmann et al., 2022)) and optimize cross-modal alignment (e.g.. CLIP (Ilharco al., 2021; Radford et al., 2021)). Although these models expose strong global descriptors, much of the signal remains distributed across patch tokens, making attentive aggregation appealing at probe time. Likewise, auto- regressive (AR) families (e.g., AIM/AIMv2 (El-Noub) et al., 2024; Fini et al., 2025)) and diffusion- based transformers (e.g., DIT (Peebles & Xie, 2023)) are primarily trained for generation (next-token prediction or denoising), not representation learning per se; nevertheless, their frozen features can be probed to assess representation quality. In all these cases, protocols that assume single discriminative token may under-utilize the information spread across patches.
To address this, recent work explores attentive probing (El-Nouby et al., 2024; Chen et al., 2023; Bardes et al., 2024; Darcet et al., 2025), which learns attention to selectively aggregate patch tokens into informative descriptors for a linear classifier. While methods like AIM (El-Nouby et al., 2024), CAE (Chen et al., 2023), and V-JEPA (Bardes et al., 2024) adopt this idea, unified evaluation exists. We fill this gap with a comprehensive benchmark and introduce novel attention mechanism achieving strong eccuracy-efficieee trade-off. Additional related work appears in Appendix A.
2

---

Published as a conference paper at ICLR 2026
3 METHOD
3.1 ATTENTIVE POOLING
Preliminaries. Let X E RDx be the feature matrix obtained from pre-trained and frozen ViT backbone, where D; is the number of feature channels and N = W x И the number of features, one per image patch across the spatial dimensions W x H. Given the input features X, the goal R. by applying an attentive pooling mechanism.
is to generate an output image-level feature The output feature is used to train C-way linear classifier with D2 1) parameters.
We consider M atlention predictors, to be discussed in subsection 3.2. For each predictor j (1...., M,, let a E N be the nn-ormallized attention vector generates. Each vector, reshaped to W H, is attention map indicating the locations which the predictor focuses. Let V e RR R be the value features, commonly obtained by a linear transformation V = WvX, where Wy DDDxD is learnable projection matrix.
Let the output feature y, value features V and projection matrix Wy be partitioned into M subvec- tors submatrices according to
y/
Vi
Wv : Wim
=
V =
Wv
=
(1)
yM
VM
with y₁ E Rdd, Vj E Rd.NN, Wv; E RdoxD. and do
Do M
The attentive pooling operation then given by
y Vja, WV Xaj
Each attention predictor is responsible for the weighted pooling of features into a d,, di-mmesiiionll subspace of the final representation space. In the following. we explore existing and novel ways for designing these attention predictors. We focus on the number of additional parameters to be learnt on top of the frozen backbone and the computational complexity of the pooling operation.
3.2 ATTENTION PREDICTORS
Multi-Head Cross-Attention (MHCA). A vanilla approach is to perform multi-head cross- attention between the input features and an input vector u E TR.., where each head corresponds to separate attention predictor. The query query feature
R., and key features K IR DNN xN obtained by linear transformations = Wou, K = WKX with projection matrices Wo, WK D.
Let the query feature q and projection matrix Wo be partitioned into M subvectors submatrices according to
q1
Wº
q
: qM
Wq=
:
Wox
with q = Wo, u TRd,, Wo, E Rd and da projection matrix WK be partitioned according to
D. Similarly, let the key features K and
K1
WK. WK
K
=
: WKM]
(4)
KM
with K2 = WK, RdNN and WK, E R.
The attention vector for head is then given
a, = softmax((j)
(5)
with
à; = K. = (WK, (Wo,u))
(6)
3

---

Published as a conference paper at ICLR 2026
11 7
Wg
K
sugleszz
Wy
X
N
globed
global
regincanttaiio
patoo
Figure I: Comparison of multi-head cross-atie aatss-attettion (MHCA, left) vs. our multi-query cross-attention (EP, right). MHCA uses an input vector u projected into query space and interacts with key features K' in (two) separate subspaces, each corresponding to an attention predictor. Attention predictor outputs aj are used to aggregate value features into sub-vectors yj, forming the final output y. In contrast, EP employs (two) learnable queries І; one per attention predictor, to compute attention with input features directly in the full representation space. Attention predictor outputs aj are used as in MHCA to perform the aggregation.
That is, the input features and input vector u are projected to di-diimenioona subspaces where attention subvectors are computed via dot product followed by softmax normalization over patches. This attention predictor requires (2D2 + 1) parameters and has complexity O(ND.D,). As discussed in subsection 3.3, there several existing methods that fit within this generic framework.
MHCA with learnable query. If we consider input vector u to be learnable, then there is no set the query feature q be learnable,
need for the projection matrix Wi in (6). Instead, thus absorbing Wq and u.
ā = (Wx, x) = - - W
where the query feature qi E Rd is learnable.
We observe that WK, maps qj the D:-dimensional space of input features compute the atten- tion vector. Thus, standard MHCA ensures that each query subvector is interacting with the full representation space of the input features, despite being defined in a smaller dimensional space. Us- ing a learnable query feature directly simplifies the architecture, reduces the amount of computations and the number of parameters to (D, + 1).
In order to explore ways for reducing the total number of parameters and better understand the role of key transformation, we simplify the architecture by removing it. Letting WK be fixed to the identity matrix, (7) becomes
à =, qj
(8)
where the feature matrix X is partitioned into M submatrices according to
X1
X
(9)
XM
with X, E BR and d, = D. We thus observe that the query feature only interacts with d:- A dimensional subspace of the input features for M > 1, which is a limitation. We experimentally verify that setting WK identity matrix results in noticeable performance drop. Based on this observation, in the following, we revisit (7) and design attention predictors that have less parameters and require less compute but do not experience performance drop due mathematical equivalence.
Parameter-efficient Multi Sulti-Query Cross-Attention (MQCA). Instead of using the key submatri- WK, to project the query subvectors qj to the Dy-dimensional input feature space, propose learn M effective query features uj := W 9j R in that space directly (Figure 1). Thus, uy absorbs WK, and qj and attention prediction becomes
à = XT uj
(10)
for j E (1....,M, M). As a result, there are no projection matrices and there are no parameters other than the learnable query features uj.
4

---

Published as a conference paper at ICLR 2026
This choice reduces the number of additional parameters to be learned and saves from more matrix-vector multiplication. In particular, it requires D,M parameters for the attention compared to (D, + 1) for (6), while the number of operations drops to N N,M compared to ND, (D; + 1). Typically, M is one to two orders of magnitude smaller than D2 and Da, which are commonly equal to each other, making the proposed approach more efficient in parameters and operations.
There is a connection between EP and slot attention (Locatello et al., 2020), where slots are also multiple vectors in the input feature space. To derive EP from slot attention, one needs to perform only a single iteration; remove LayerNorm, GRU and MLP; make slot vectors learnable rather than initialized at random; and concatenate the output features into a global representation of appropriate dimension. Thus, EP can be seen as lightweight counterpart of slot attention, where the absence of interactions compensated by the query features being learned.
3.3 EXISTING VARIANTS
We analyze existing methods as instances of the presented framework (Table 1), and examine com- mon variants, considering their relationship to the framework despite slight deviations. Additional methods considered in experiments are presented in Appendix B.
AbMILP (Rymarezyk et al., 2021) the simplest variant. It fixes WK and Wy to identity and is equivalent to MHCA with a learnable query feature framework in (7), with single head (M = 1). It can also be seen as special case of our proposed method in (10) with one learnable query feature, i.e. M = 1. AbMILP requires only D; parameters and computes attention with single matrix- vector multiplication, but its performance is limited by the single head/query.
AIM (El-Nouby et al., 2024) is instance of MHCA with learnable query feature. It deviates from the generic framework by applying batch normalization on the input features. It does follow (2) and (7) with M heads and D. D; Do. but replaces X by BN(X). Batch normalization introduces minor additional parameters and slight computational overhead compared to the default variant of the framework.
DELF (Noh et al., 2017) feeds each of the N input features to a MLP whose output is a scalar attention value in [0, 1]. It can be viewed an instance of the MHCA with a learnable query feature and M- = with the following modifications. Key and value projection matrices share the same weights, i.e. WK Wy W, a non-linearity is introduced in equation (7) by á ReLU(W X) q. where subscript j is skipped due to M = 1, and softmax in (5) is replaced by wlement-wise softplus, = (a)).. In the context of DELF, the query feature q can be seen as the parameter of a x 1 convolutional layer. DOLG (Yang et al., 2021) adopts similar design choice.
SimPool (Psomas et al., 2023) can be seen as an instance of MHCA with a single head (M 1) that uses a dependee input vector u. Wy fixed to identity, and layer normalization on the input features. Specifically, the query feature is obtained as q = Wou, is replaced by LN(X) for key and value transforms. Compared to MHCA with a learnable query, SimPool saves D; parameters and has the same complexity.
=Wou, u = v XT (M = 1). X
V-JEPA. The first part of V-JEPA (Bardes et al., 2024) is identical to the MHCA framework but applies layer normalization on the input features for key and value transforms, like SimPool. Its second part is an MLP with GeLU activation (Hendrycks & Gimpel, 2016) and residual connections, making the overall process equivalent to a transformer block.
Table I: Attentive pooling variants as aligned algorithmic steps, fitting the framework of Sec. 3.1- 3.2. We list (i) how the query is formed, (ii) key/value transforms, (iii) attention predictor outputs, and (iv) pooling operation. o. softmax, Jp: softplus. Φ(z) := W2 Ge). blue: learnable.
METROD QUERY SOURCE
KEY TRANSFORM
VALUE TRANSFORM AFTENTION
POoLING
MHCA
2; E Rda
Ky=WK, x
V/= V3=Wi,
ay K3 q;) V V2a,
AbMILP
q R
K=X
V=X
a=am (K q)
y= Va
AIM
9; E Rda
WW, BN(X)
VJ=Wy, BN(X))
a, a,=0m(t (K) q:)
V V,
DELF
RER";
K K=ReLU(WXX
V=WX
a=am (K"
y=
SimPool
q= Wau E R u=
K WR LN))
V=LN(X)
a= (K' q)
y=Va Va
V-JEPA
q == E Rd RP,
K2 WK LN(X)
- (K) q:)
(WW))aa))
EP (ours)
2; Rd
K=X x
We3 X
a, a. (K" 9;) y Via,
5

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets. We evaluate attentive probing across diverse image classification benchmarks, including ImageNet- 1K (IN-IK) (Deng al., 2009), CIFAR-100 1-00 (Krizhevsky al., 2009), Places365 (Zhou et al., 2014), CUB-200 (Wah et al., 2011), FGVC Aircraft (Maji et al., 2013), Stanford Cars (Krause et al., 2013), and Food-101 101 (Bossard et al., 2014). More details in subsection C.I.
Pre-training methods. We conduct attentive probing experiments with frozen models from diverse pre-training paradigms: MIM (e.g.. MAE and CAPI), JEA (e.g., BYO and DINO), hybrid (e.g., iBOT and DINOv2), VLMs (e.g., CLIP and SigLIP), and generative (e.g., DiT and AIMv2). Model sizes vary from small (e.g. ViT-S for MAE) to extra large (e.g. DiT-XL for DiT).
Pooling/probing methods. We compare attentive probing against diverse set of methods, cov- ering different paradigms. First, we evaluate attentive poolings originally designed for probing, including AIM (El-Nouby et al., 2024), CAE (Chen et al., 2023), CAPI (Darcet et al., 2025), and V-JEPA (Bardes et al., 2024). Second, we include attentive poolings originally proposed in other contexts but applicable to probing, such AbMILP (Rymarczyk et al., 2021), SimPool (Psomas et al., 2023), CLIP (Radford et al., 2021), SigLIP (Zhai et al., 2023; Wightman, 2019), CoCa (Yu et al., 2022), CaiT (Touvron et al., 2021), 1), and DELF (Noh et al., 2017). Additionally, we include fea- ture re- reighttgg methods like CBAM (Woo et al., 2018), applying global average pooling to obtain the global descriptor. As baselines, we use [CLS], which corresponds to standard linear probing when the backbone provides classification token, and GAP, which serves as linear probing when no [CLS] J token is available. but can also be interpreted as a baseline attentive probing method with uniform attention over patch tokens. To establish a reference, we also evaluate ViT (Dosovitskiy et al., 2021) block, applying global average pooling to extract the global representation, and MHCA. All methods operate on the same input features-namely namely, the patch tokens extracted from the frozen backbone-ensuring a fair and consistent comparison. Unless otherwise stated, D,=D;=Do.-
Parameter-efficient fine-tuning (PEFT) methods. We compare attentive probing against PEFT methods on ImageNet- IK: LoRA (Hu et al., 2022), BitFit (Ben Zaken et al., 2022), and LayerNorm tuning (Zhao et al.).
Evaluation protocols. Attentive probing is performed for 90 epochs. We evaluate top-1 classi- fication accuracy the validation set of each dataset. Additionally, compute the number of parameters and measure the FLOPs to assess computational efficiency and scalability.
4.2 EXPERIMENTAL RESULTS
Accuracy vs. parameters. In Figure 2. we compare efficient probing (EP) with baseline and com- petitor methods using MAE ViT-B, BEÎTv2 ViT-B, and CAPI ViT-L on ImageNet- and MAE ViT-B on Food-101. We plot top-1 accuracy against the number of trainable parameters, including both attentive pooling and classifier parameters, and overlay the Pareto frontier to highlight optimal trade-offs. The two primary baselines, [CLS] and GAP, are the most parameter-efffiient,, they introduce no overhead beyond the classifier parameters, but yield noticeably lower accuracy. In con- trast, methods like V-JEPA, CaiT, SigLIP, and the reference ViT block employ significantly more parameters, though within the attentive probing setting, their increased complexity provides mostly marginal accuracy improvements. Among the existing attentive probing or pooling methods, Sim- Pool provides moderate accuracy but is not particularly purameter-efficien while CAE and CLIP achieve stronger performance at the cost of higher parameter counts. AbMILP, DELF, AIM, and MHCA lie primarily on the Pareto frontier, striking the optimal balance.
EP consistently achieves the best accuracy-paramett trade-off, positioning itself on the left or upper-left region of the Pareto frontier across pre-training methods and datasets. A key factor is its flexibility in controlling the number of queries M and the output dimensionality D, (because of Wr). allowing adaptation to different parameter constraints. Notably, on ImageNet-IK with MAE VIT-B, EPs4 (64 queries) achieves a state-of thettherrt top-1 accuracy of 75.6% with less than 1.4M parameters. EP 18 with 48 queries and D. D achieves 70.3% top-1 accuracy, while having little more than 200k parameters, i.. almost 4x less than linear probing (ICLS) ). On ImageNet-IK KK with BEiTv2 VIT-B and with CAPI VIT-I., FP and F.P4 achieve state-of-the-ar accuracy of

---

Published as a conference paper at ICLR 2026
MAE VIT-B ImageNet- IK
(b) MAE VIT-B Food-101
76
EP
VUEPANIT
RPoG
AIMMMICA
VIT
85
BP
AIMMMCA
FP10
CAF
CLIP
VJER
BP
CoCa
CAF
SIPLIP
74
CLIP
top-1 accuracy (5e
EP
DELF
CoCa
DELF
EP4
80
IRP
72
P
SimPool
13
ALMILP
SimPool
70
JEPg
[P2
Do Dr
Do
Da/2
Do D₁
68
[CLS
Dz/4
GAP [CLS
Do
Dz/4
GAP
CRAM
DL/8
70
Do Dz/16
66
10
107
10
10c
107
(c) B VIT-B ImageNet-Il IK
(d) CAPI VIT-L ImageNet-IK
EPaz
EP
top- accuracy (%)
81
EPa2 EP1
1 RP.
AIMMHCA
CLIP
82
EFg
EI's
ARMIL
ICLST
DELF
ALMMICA
CLIP
Smu VT
VJEPA
Carl
Stgt
80
BP2
CoCa
Do
Du D2
[CLS)
12a
D2/2
78
Do D2/2
D, D
Do D:/4
GAP
GAP
D./8
D. D:/5
CRAMM
76
10
10
10
107
number of parameters
number of parameters
Figure 2: Top-/ classification accuracy number of parameters for various self-supervised pre- training methods across different datasets. We evaluate both dedicated probing mechanisms (e.g., V-JEPA) and repurposed attentive pooling methods (e.g.. CLIP). EP variants
marked with dif-
ferent colors for different output dimensionalities D EP m: efficient probing with M learnable queries. [CLS]: linear probing using the classification token; GAP: global average pooling over patch tokens; MHCA: multi-head attesnatte ViT: default transformer block.
81.7% and 83.7% respectively. On 101, a dataset smaller than ImageNet- IK, we observe that reducing D, even D,/16 does not significantly hurt performance. More results in subsection C.2.
Accuracy vs. computational cost. In Figure 3, we compare different pooling/probing methods in terms of top-1 accuracy against computational cost, measured in GFLOPs. Baselines [CLS] and GAP the most efficient but less accurate, while self-attention methods (e.g., ViT, CLIP) in- higher cost due additional attention putations. EP achieves better accuracy than a ViT block with over 10x less compute, and consis- tently lies on the Pareto frontier, combining high accuracy with low cost. By adjusting the output dimensionality D.. EP scales to different compu- tational budgets without significant accuracy loss. Note that backbone FLOPs (e.g., 17.58 GFLOPs for MAE ViT-B) are not shown, since they are identical across all methods and thus omitted for fairness. Here we only compare the additional cost of the pooling/probing methods.
76
IP
84
AIMMMICA
VIT
top-1 accuracy (%)
74
EPaa
JJEPA
CLIP
72
SimPool
A-MIP
70
Do
68ICLS
D.
D. D,/4
66
0.5
1.5
GFLOPs (beyond backbone)
Figure Top-I classification accuracy vs. GFLOPs for MAE ViT-B with different prob- ings on ImageNet-IK.

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of pre-training methods in terms of different evaluation protocols on ImageNet-IK: PAR.: number of trainable parameters including classifier: 'Results reported from the official paper under different than probing setup (c.g., augmentations); 'Evaluated using GAP.
METROD
ARCH
PRE. TRAINING k NN
LP
# PAR.
EP
# PAR.
[AIN
FT
PAR.
5/16
+17.2
MAE (He etal. 2022)
V/1-
IN-IK
46.1
67.7
0.8M
75.6
1.4M
+79
83.6
87M
MIM
VIT-L/16
58.2
76.0
1.0M
79.3
2.1M
+3.3
85.9
304M
BF/Tv2 (Peng al. 2022)
VIT-B/16
IN-IK
74.8
79.0
0.8M
81.7
14M
+2.7
85.0
87M
SimMIM (Xie et al., 2022)
VIT-B/16
IN-1K
15.1
51.5
0.8M
65.1
14M
+13.6
83.8
87M
CAPI (Darect et al, 2025)
VIT-L/44
IN-IK
76.7
81.5
1.0M
83,6
2.1M
+2.1
304M
BYOL BYrll et al., 2020)
RN-50
IN-IK
64.8
74.3
2.0M
75.1
6.3M
+0.8
77.7
26M
DINO (Caron et al, 202.)
VIT-B/16
IN-IK
76.1
77.3
0.8M
77.8
1.4M
+0.5
82.8
87M
iBOT (Zhou et al., 2022)
V/I
IN-IK
77.0
78.7
0.8M
79.2
1.4M
+0.5
84.0
87M
HYBRID
DINOv2 (Qquab et al,, 2024)
VIT-B/14
WIT-L114
LVD-142M
83.5
81.8
85.2
83.2
0.8M
1.0M
85.6
84.0
2.1M
1,4M
+0.4
+0,8
X
X 304M
87M
Franca (Venkataramanan et al. 2025)
VIT-1/14
IN-2IK
82.2
83.8
1.0M
84.3
2,1M
+0.5
X
304M
DINOv3 (Simćoni et al., 2025)
B/T
VIT-L/16
LVD-1689M
85.3
83.0
84.0
86.6
0.8M
1.0M
87.1
84.4
2.1M
1,4M
+0,4
+0.5
X
304M
S7M
CLIP (Radford al., 2021)
VIT-L144
WIT
77.2
82.3
0.8M
83.4
2,1M
+1.1
305M
SigT.IP (Zhai ol,, 2023)
VIT-116
WebLI
83.7
84.1
1.0M
86.1
2.1M
+2.0
305M
SigL.PP (Tschannen et ol., 2025)
VIT-L/66
WebLI
84.4
85,2
1.0M
87.0
2.1M
+1.8
305M
DIT (Poobles Xie, 2023)
DIT-XL/2
IN-IK
8.3
32.7
1.2M
57.0
+24.3
676M
ATMv2 (Fini et ol., 2025)
VIT-I/14
custom
80.8
84.8F
1.0M
85.9
2.1M
+1.1
304M
Note: custom": DFN-2B (Pang et al., 2024), COYO (Byeon et al,, 2022), HQITP (Fini et al., 2025), Default EP
EPs2-
78
Baseline probings
Baseline Paretn
top-1 accuracy (%)
76
Eg Pp/2
EPD
AIM
various
X0x
74
x LoRA at layer
LoRA layer
72
kpD,/4
DELF
LoRA layer 12
AMM
X LoRA on all layers
70
PP/s
BirFir
LayerNorm tuning
68
LoRA EP
(CT.S)
LoRA EP Pareto
2.5 10
5 10
10
2 10
number of parameters
Figure 4: securacy-paramett trade-off of probing and parameter-effiiien fine-tuning methods on ImageNet- IK using frozen MAE V backbone. The hybrid LoRA + EP configurations achieve a strictly better trude-off than both pure EP and pure LoRA.
Comparison of pre-training methods. Table 2 compares methods from diverse pre-training paradigms under multiple evaluation protocols ImageNet-IK. Fine-tuning (FT) achieves the highest accuracy but is compute-intensive and unsustainable at scale, making it increasingly rare in recent evaluations (x). A-NN consistently underperforms, reflecting the weak separability of raw features. Linear probing (LP) offers stronger bascline, yet EP provides consistent gains across all paradigms, often with only small increase parameters. Notably, EP changes the relative ranking of methods compared to LP and k-NN: for example, MAE surpasses BYOL, and CAPI outperforms CLIP, challenging the view that MIM methods are weaker. In general, methods whose pre-training optimizes representations of patch tokens rather than an explicit global representation are benefit the most from attentive probing (e.g., SimMIM +13.6%, DiT +24.3%). We further analyze this trend in subsection C.3. Across the board, EP narrows the gap to full fine-tuning while remaining lightweight and scalable, demonstrating both its generality and effectiveness.
Comparison against PEFT methods. In Figure 4 we compare the most efficient baseline probing methods (ICLS), AbMILP, DELF, AIM), EP at different output dimensionalities D., more than 40 LoRA variants, BitFit, and LayerNorm tuning, all on a frozen MAE ViT-B/16. We sweep LoRA across individual transformer layers (4, 8, 12) as well as all layers, and across multiple configurations (e.g. Wnlonly, W onlyn,y, Wy-only, Wa Wxx K. Wo+W Ww++Wv) with ranks p {8, 16, 32, 64).
Overall, LoRA applied to a small subset layers (red, cyan, magenta crosses) lies roughly on or slightly above the baseline Pareto front, but is consistently dominated by EP in the aceu-
8

---

Published as a conference paper at ICLR 2026
racy-parameter plane. For instance, EPD./2 attains 74.9% top-1 accuracy with only 750K param- eters, whereas the best single-layer LoRA configuration around that region requires around 1.2M parameters. LayerNorm tuning and BitFit (green and brown crosses) also outperform the pure prob- ing baselines, yet EP remains strictly more efficient. All-layer LoRA configurations (orange crosses) move closer to the EP Pareto curve and some even surpass it in accuracy (up 76.7%). In subsec- tion C.2, we present an analysis with in- and out-of-domain k-NN experiments on frozen features, showing that all-layer LoRA modifies the representation more strongly than EP, consistent with their roles as dask-adaptin low-rank fine-tuning (LoRA) versus representation preseevvng probing (EP).
Motivated by these observations, we finally combine one of the most parameter- LoRA variants (LoRA on all Wy matrices across layers) with EP for different output dimensionalities Do (orange star-cross markers). The resulting LoRA+EP configurations form a new dominant plane, strictly improving over both pure EP and pure LoRA, For
region in the accuracy-paramest example, a hybrid setting with 850K parameters achieves 76.99% top-1 accuracy, improving over both the best pure EP configuration (75.5%% with 1.38M parameters) and the best all-layer LoRA variant 72% with 1.95M parameters). At the low-parameter end, a LoRA+EP configuration with only 250K parameters already reaches 71.99 71.99% accuracy, i.e. about 4.3% above [CLS] linear probing (67.66%) while using over 3x fewer parameters. These results indicate that EP captures information that LoRA alone does not, and vice versa: rather than being redundant with PEFT, EP provides a complementary, parameter-efficiett probing (PEP) mechanism that remains beneficial.
4.3 EXPERIMENTAL ANALYSIS
Impact of WK and Wy. Our analysis in subsection 3.2 posits that while single learnable query effectively absorbs the key transformation WK in single-head attention, the same does not hold in multi-head. To empirically validate this, we probe MAE VIT-B pre-trained on ImageNet-IK
with four variants: single-head (M-1) vs. multi-head (M>1), each with (7) vs. without Wx (8). Specifically, we evaluate AbMILP and AIM for the angle-head and multi-head (AIM12 with 12 heads) case, respectively. In single-head attention, removing Wr has minimal impact on perfor- mance (71.8% + 71.7%), while in multi-head the drop is noticeable (75.1% - 72.9%), since each query now interacts only with subspace. Note that MHCA (7), that natively includes WK. in AIM12 and our MQCA (10), that natively does not includes Wr in EP configured with the same number of heads/queries (M= attain identical accuracy (75.1%), reflecting their mathematical equivalence. However, EP12 reaches this performance with far fewer parameters (1.36M for EP12 1.95M for AlM12), making its design more parameter-efficient. We also ablate the effect of the value transformation Wv, which operates on patch tokens (2), by adding or removing it across pooling methods. Introducing Wy GAP results in top-1 accuracy improvement from 66.7% - 68.0%. Conversely, removing Wy from EP degrades performance from 75.1% + 72. %. A sim- ilar accuracy drop observed for other methods, such AIM (75.1% + 72.0% and CAE (74.9% + 72.2%), confirming that Wv is a critical component.
2.5
2.5
3
3
accuracy
2
1.5
1.5
2
2
AIM
AIM
D2 =
1
D. D.
0.5
EP
0.5
UP
Do D2/2
D2/2
VUEES
0
Do D./4
0
0.8
0.9
4.5
5
5.5
0.8 0.9
4.8
5
6
localization
entropy
localization
entropy
Figure 5: Classification accuracy vs. attention quality on ImageNet- 1K. Each point corresponds to an attention predictor (head query). A accuracy measures the drop when replacing attention predictor's distribution with uniform. Plots show relations to localization quality (Ist, 3rd) and entropy (2nd, 4th). Left: different attentive probing methods; Right: varying D, for EP.
Classification vs. localization. We investigate whether the quality of attention maps in terms of localization contributes positively to classification accuracy. Localization quality is measured by (i) the attention mass within the ground-trun -ound-truth bounding box (Deng et al., 2009) and (ii) the entropy of the attention distribution, averaged over the validation set. To estimate each predictor's contribution

---

Published as a conference paper at ICLR 2026
Figure 6: Attention maps of E.g. Each query is assigned distinct color. Semantic correspondences emerge (e.g., tails, beaks, feet), with each query capturing complementary regions and enabling a structured decomposition of visual cues. MAE ViT-B pre-trained IN-1K, probed with EP.
Complementarity
0.8
0.70
0.00
0.87
0.6
NIDSA
0.46
0.4
ADM
0.21
0:22
0.2
0.18
MAE
BETTv2 SimMIM iROT DINOv2 DINOVá Francn CLIP
SigLIP VIT-L
MAE VIT-B
VIT-L
VIT-B
VIT-B
VIT-B
VIT-B
VIT-B
VTL
VIT-L
VITL
Figure 7: Complementarity scores of attention maps across different backbones and probings. We compare the diversity of the internal MHSA heads in the last block against the external EP queries (left), V-JEPA heads, and AIM heads (right). Number of predictors are matched for fairness.
to classification, we measure the accuracy drop when replacing its attention with a uniform distribu- tion. Figure shows a clear correlation: predictors with better localization and lower entropy exert stronger influence accuracy. This holds across different attentive methods as well as EP with duced output dimensionality D., As also seen in Figure 2, lowering Dr degrades performance; the rightmost plots further suggest that this effect arises not only from reduced representational capacity but also from diminished attention quality, as reflected in higher entropy.
Attention complementarity. Figure 6, we jointly visualize the attention maps of EPs. An emerg- ing property of EP is that its queries specialize in different object regions, yielding complementary and interpretable attention patterns. Queries consistently attend distinct parts, producing seman- tic correspondences across images and structured decomposition of visual cues. To quantify this diversity, we define a complementarity metric measuring how differently attention predictors dis- tribute mass over patch tokens. For standard backbones, we extract the [CLS] ->patch attention from the internal last-block MHSA heads, and for EP the learned queries, with matched counts for fairness. We L2-normalize each distribution, compute pairwise cosine similarities, and define com- plementarity as one minus the average off-diagonal similarity. Higher values indicate more diverse (less redundant) attention. As shown in Figure achieves significantly higher complementarity than MHSA and outperforms other probing approaches. More experiments in subsection C.3.
5 CONCLUSION
We revisit evaluation protocols for pre-training methods and introduce efficient probing (EP), scalable alternative to fine-tuning. Lightweight yet expressive, EP delivers interpretable attention, strong generalization across paradigms, and consistent gains over linear probing-up to +24.3% on ImageNet- 1 K. By comparing EP to standard parameter-efficien fine-tuning (PEFT) baselines, we show that EP is not only competitive on its own but also complementary: hybrid configurations achieve the best accuracy-parameter trade-offs, hinting at new research directions.
Acknowledgments. This work was supported by the EU Horizon Europe programme MSCA PF RAVIOLI (No. 101205297) and the Junior Star GACR GM 21-28830M. We acknowledge VSB-Technical University of Ostrava, IT4Innovations National Supercomputing Center, Czech Re- public, for awarding this project (OPEN-33-6) (3-67) access to the LUMI supercomputer, owned by the EuroHPC Joint Undertaking, hosted by CSC (Finland) and the LUMI consortium, through the Min- istry of Education, Youth and Sports of the Czech Republic via the e-INFRA CZ project (ID: 90254). AWS resources were provided by the National Infrastructures for Research and Technology GRNET and funded by the EU Recovery and Resiliency Facility.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Maximilian Augustin, Syed Shakib Sarwar, Mostafa Elhoushi, Yuecheng Li, Sai Qian Zhang, and Barbara De Salvo. Petah: Parameter efficient task adaptation for hybrid transformers. In Pro- ceedings of the Computer Vision and Pattern Recognition Conference, 2025. 17
Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mido Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. 1, 2, 5, 6, 17, 18
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. BitFit: Simple parameter-efficient fine- tuning for transformer-based masked Linguage-modele In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 2: Short Papers), 2022. 6, 17
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative com- ponents with random forests. In European Conference on Computer Vision, 2014. 6, 19
Minwoo Bycon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Back, and Sachoon Kim. Coyo-700m: Image-text pair dataset, 2022. 8
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021. 1, 2, 8, 19
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang. Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 2022. 17
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, 2020.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representa- tion learning. International Journal of Computer Vision, 132(1), 2023. 1, 6, 18
Junsuk Choe, Scong Joon Oh, Seungho Lee, Sanghyuk Chun, Zeynep Akata, and Hyunjung Shim. Evaluating weakly supervised object localization methods right. Conference on Computer Vision and Pattern Recognition, 2020. 23
Junsuk Choe, Seong Joon Oh, Sanghyuk Chun, Seungho Lee, Zeynep Akata, and Hyunjung Shim. Evaluation for weakly supervised object localization: Protocol, metries, and datasets. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 2022. 23
Timothée Darcet, Federico Baldassarre, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Clus- ter and predict latents patches for improved masked image modeling. Transactions on Machine Learning Research, 2025. 1, 6, 8, 19
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, 2009. 6, 9, 19
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. 6, 17
Alaaeldin El-Nouby, Michal Klein, Shuangfei Zhai, Miguel Ángel Bautista, Vaishaal Shankar, Alexander T Toshev, Joshua M. Susskind, and Armand Joulin. Scalable pre- pre-training of large autoregressive image models. In Proceedings of the 41st International Conference on Machine Learning, 2024. 2, 6, 18
11

---

Published as a conference paper at ICLR 2026
Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander T Toshev, and Vaishaal Shankar. Data filtering networks. In The Twelfth International Conference on Learning Repre- sentations, 2024.
Enrico Fini, Mustafa Shukor, Xiujun Li, Philipp Dufter, Michal Klein, David Haldimann, Sai Aitharaju, Victor G Turrisi da Costa, Louis Béthune, Zhe Gan, et al. Multimodal autoregres- sive pre-training of large vision encoders. In Proceedings of the Computer Vision and Pattern
Recognition Conference, 2025. 2, 8, 20
Cheng Fu, Hanxian Huang, Xinyun Chen, Yuandong Tian, and Jishen Zhao. Learn-to-shar: A hardware-friendl transfer learning framework exploiting computation and parameter sharing. In Proceedings of the 38th International Conference on Machine Learning, 2021. 17
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a approach to self-supervised learning. Advances in Neural Information Processing Systems, 2020. 8, 19
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016. 17
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 1, 2, 8, 27
Xin He, Yushi Chen, Lingho Huang, Danfeng Hong, and Qian Du. Foundation model-based multi- modal remote sensing data classification. IEEE Transactions on Geoscience and Remote Sensing, 62:1-17, 2024. 17
Dan Hendrycks and Kevin Gimpel.
Gaussian
error linear units (gelus).
arXiv preprint
arXiv: 08415, 2016. 5
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con-
ference on Learning Representations, 2022. 2, 6, 17
Leiyi Hu, Hongfeng Yu, Wanxuan Lu, Dongshuo Yin, Xian Sun, and Kun Fu. Airs: Adapter in remote sensing for parameter-efficient transfer learning. IEEE Transactions on Geoscience and Remote Sensing, 62:1-18. 2024. 17
Chia-Chien Hung, Lukas Lange, and Jannik Strötgen. Tada: Efficient agnk-tgn domain adap- tation for transformers. In The 6lst Annual Meeting Of The Association For Computational Lin- guistics, 2023. 17
Gabriel Ilbarco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip. July 2021. 20
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, 2021.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge J. Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, 2022. 17
Ioannis Kakogeorgiou, Spyros Gidaris, Bill Psomas, Yannis Avrithis, Andrei Bursuc, Konstantinos Karantzalos, and Nikos Komodakis. What to hide from your students: Attention-guided masked image modeling. In European Conference Computer Vision, 2022.
Samar Khanna, Medhanie Irgau, David B. Lobell, and Stefano Ermon. Explora: Parameter-efficient extended pre-training adapt vision transformers under domain shifts, 2025. 17
12

---

Published as a conference paper at ICLR 2026
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei Li. 3d object representations for tne-grained categorization. ICCVW, 2013. 6, 19
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 6, 19
Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ra- manujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, et al. Matryoshka representation learning. Advances in Neural Information Processing Systems, 2022. 27
Minglei Li, Peng Ye, Yongqi Huang, Lin Zhang, Tao Chen, Tong He, Jiayuan Fan, and Wanli Ouyang, Adapter-x: A novel general parameter-cfficient fine-tuning framework for vision. CoRR, abs/2406.03051, 2024. 17
Tao Li, Zhengbao He, Yujun Li, Yasheng Wang. Lifeng Shang, and Xiaolin Huang. Flat-lora: Low- rank adaptation over flat loss landscape, 2025. 17
Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. arXiv preprint arXiv:1312.4400, 2013. 17
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot atten- tion, Advances in Neural Information Processing Systems, 2020. 5
Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Fric Vanden-Fijndee and Sain- ing Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In European Conference on Computer Vision, 2024. 1
S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013. 6, 19
Hyeonwoo Noh, Andre Araujo, Jack Sim, Tobias Weyand, and Bohyung Han. Large-scale image retrieval with attentive deep local features. In Proceedings of the IEEE International Conference Computer Vision, 2017. 2, 5, 6, 17
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without super- vision. Transactions on Machine Learning Research, 2024. 1, 8, 19
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in Neural Information Processing Systems, 2019.
20
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023. 2, 8, 20
Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. Beit v2: Masked image modeling with vector-quantized visual tokenizers. CoRR, abs/2208.06366 2022. 8, 19
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapter- fusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Computational Linguistics, 2021. 17
Marcin Przewikežlkkowki.. Randall Balestriero, Wojeiech Jasiński, Marck Šmieja, and Bartosz Zieliński. Beyond [cls]: Exploring the true potential of masked image modeling representations. In Proceedings of the IEEE/CVF International Conference Computer Vision, 2025. 27
Bill Psomas, loannis Kakogeorgiou, Konstantinos Karantzalos, and Yannis Avrithis. Keep it sim- pool: Who said supervised transformers suffer from attention deficit? In Proceedings of the IEFF/CVF International Conference Computer Vision, 2023. 2, 5, 6, 17
13

---

Published as a conference paper at ICLR 2026
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, Proceedings Machine Learning Research, 2021. 1, 2, 6, 8, 17, 18, 19
Dawid Rymarezyk, Adriana Borowa, Jacek Tabor, and Bartosz Zieliñski. Kernel self-attention in deep multiple instance learning, 2021. 2, 5, 6, 17, 18
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mchdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 2022. 2
Reece Shuttleworth, Jacob Andreas, Antonio Torralba, and Pratyusha Sharma. Lora vs full fine- tuning: An illusion of equivalence. arXiv preprint arXiv:2410.21228, 2024.
Oriane Siméoni, Huy V Vo. Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Mare Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3. arXiv preprint arXiv:2508.. 2025. 8
Romain Thoreau, Valerio Marsocci, and Dawa Derksen. Parameter-efficiet adaptation of geospatial foundation models through embedding deflection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025. 17
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In IEEE/CVF International Conference Computer Vision, 2021. 6, 17,
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Nacem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual sgion-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint wrXiv:2502.14788 2025. 8, 19
Shashanka Venkataramanan, Valentinos Pariza, Mohammadreza Salehi, Lukas Knobel, Spyros Gi- daris, Elias Ramzi, Andrei Bursuc, and Yuki M Asano. Franca: Nested matryoshka clustering for scalable visual representation learning. arXiv preprint drXiv:2507.14137 2025.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech- UCSD Birds-200-2011 Dataset. Technical Report CNS-TR ANS-TR-2001-00 California Institute of Technology, 2011. 6,
Ross Wightman. Pytorch image models, 2019. 6, 17, 18
Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon. CBAM: convolutional block attention module. In European Conference Computer Vision, 2018. 6, 17
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: simple framework for masked image modeling. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 8, 19
Yi Xin, Siqi Luo, Haodi Zhou, Junlong Du, Xiaohong Liu, Yue Fan, Qing Li, and Yuntao Du. Parameter-efficien fine-tuning for pre-trained vision models: A survey, arXiv e-prints, pp. arXiv- 2402, 2024. I
Min Yang, Dongliang He, Miao Fan, Baorong Shi, Xuetong Xue, Li, Errui Ding. and Jizhou Huang. Dolg: Single-stage image retrieval with deep orthogonal fusion of local and global fea- tures. Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021. 5, 17
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv: 7708. 03888, 2017. 20
14

---

Published as a conference paper at ICLR 2026
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learn- ing Research, 2022. 6, 17, 18
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In IEEE/CVF International Conference on Computer Vision, 2023. 6, 8, 18, 19
Bingchen Zhao, Haoqin Tu, Chen Wei, Jieru Mei, and Cihang Xie. Tuning layernorm in atten- tion: Towards efficient multi-modal Ilm finetuning. In The Twelfth International Conference Learning Reprexentations. 6, 17
Houqiang Zhong, Shaocheng Shen, Ke Cai, Zhenlong Wu, Jiangchao Yao, Yuan Cheng, Xuefei Li, Xiaoyun Zhang, Li Song, and Qiang Hu. Serial low-rank adaptation of vision transformer. In IEEE International Conference on Multimedia and Expo, 2025. 17
Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learning deep features for scene recognition using places database. In Advances Neural Information Process- ing Systems, 2014. 6, 19
Jinghao Zhou, Chen Wei, Huiyu Wang. Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. International Conference on Learning Representa- fions, 2022. 8, 19
Yitao Zhu, Zhenrong Shen, Zihao Zhao Sheng Wang, Xin Wang, Xiangyu Zhao, Dinggang Shen, and Qian Wang. Melo: Low-rank adaptation is better than fine-tuning for medical image diagno- sis. In IEEE International Symposium on Biomedical Imaging, 2024. 17
Wentao Zou, Qi Li, Jidong Ge, Chuanyi Li, Xiaoyu Shen, Liguo Huang, and Bin Luo. A comprehen- sive evaluation of parameter-efficient fine-tuning on software engineering tasks. arXiv preprint arX1: 2023.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
Additional Related Works
17
A.1 Pooling
17
A.2
Parameter-efffcient Fine-tume tinettunin
17
Additive.
17
Selective.
17
Low-rank adaptation.
17
Additional Methods
18
CLIP.
CAiT.
18
SigLIP.
18
CAE.
18
CoCa
18
C
Additional Experiments
19
C.I Experimental Setup
19
Datasets.
19
Pooling/probing methods.
19
Implementation details.
19
C.2 Experimental Results
20
Accuracy parameters.
................................
20
Layer-wise probing.
21
Low-shot probing.
22
In- and out-of-domall k-NN evaluation.
22
Object localization.
23
Image retrieval.
23
................................
C,3 Experimental Analysis
................................
23
Attention complementarity.
23
Attention complementarity as a loss.
24
Entropy.
26
Impact of attention predictors, Da. and D
................................
26
EP convergence.
27
Attention mixing.
27
Matryoshka representation learning.
27
Visualizations.
28
Limitations and Future Work
32
Use of Large Language Models
32
16

---

Published as a conference paper at ICLR 2026
A ADDITIONAL RELATED WORKS
A. POOLING
Pooling reduces spatial resolution while retaining semantic information. In CNNs, fixed pooling (e.g., global average pooling (Lin et al., 2013; He et al., 2016)) is standard; in vision transformers (ViTs) (Dosovitskiy et al., 2021), the [CLS] token aggregates features via self-attention.
Recent work proposes attention-based pooling to enhance representation quality. SimPool (Pso- mas al., 2023) replaces global average pooling using trainable attention in both CNNs and ViTs. Vision-language models such as CLIP (Radford et al., 2021), SigLIP (Wightman, 2019), and CoCa (Yu et al., 2022) use attentive pooling or cross-attention fuse modalities. V-JEPA (Bardes et al., 2024) applies cross-attention pooling for probing pretrained representations. In image re- trieval, DELF (Noh al., 2017) and DOLG (Yang al., 2021) spatial attention to focus salient regions. CaïT (Touvron et al., 2021) improves class-token attention, AbMILP (Rymarezyk al., 2021) uses single-query pooling for multiple-instance learning, and CBAM (Woo al., 2018) combines channel and spatial attention to recalibrate features. Although these poolings are origi- nally introduced in diverse contexts, we repurpose them for probing of frozen pre-trained models, enabling a fair and comprehensive benchmark.
A.2 CREAMETER-EFFICEREE FINE-TUNING
Parameter-efficciett fine-tuning (PEFT) adapts large pre-trained models to downstream tasks without updating all model parameters. PEFT techniques broadly include additive, selective, and low-rank adaptation methods.
Additive. Additive methods introduce small, task-specific modules into the frozen backbone, leav- ing the pre-trained weights untouched. These modules often reside within the transformer blocks and are trained to specialize the model for a new domain. Notable examples include Adapter- Fusion (Pfeiffer et al., 2021), LeTS (Fu et al., 2021), and TADA (Hung et al., 2023) in natural language processing (NLP), VPT (Jia et al., 2022), AdaptFormer (Chen al., 2022), and Adapter- X (Li et al., 2024) in computer vision (CV), and FMA (He et al., 2024), AiRs (Hu et al., 2024), and DEFLECT (Thoreau et al., 2025) in remote sensing (RS).
Selective. Selective methods fine-tune only specific subsets of parameters, typically chosen based on their functional role or estimated importance. Examples include BitFit (Ben Zaken et al., 2022), which adjusts only the bias terms, and norm-tuning approaches (Zhao et al.) which up- date only the normalization layers. These techniques avoid introducing new components, making them lightweight, though sometimes at the expense of performance.
Low-rank adaptation. Low-rank adaptation methods like LoRA (Hu al., 2022) in NLP sume that parameter updates lie in a low-dimensional subspace. They inject trainable low-rank matrices into existing layers, yielding strong performance with minimal parameter growth. In the vision domain, LoRa and its variants have been effectively adapted to vision transformers (ViTs), often rethinking where and how low-rank modules are inserted align with the spatial and hier- archical nature of visual representations. Notable examples include structure-awart methods like Serial LoRA (Zhong et al., 2025) and Flat-LoRA (Li et al., 2025). layer-wise extensions such as AdaptFormer (Chen et al., 2022), and task specific designs like PETAH (Augustin et al., 2025) and MeL.o (Zhu al., 2024), which adapt LoRa to mobile inference and medical imaging, respectively. Continued pretraining approaches such as ExPLoRA (Khanna et al., 2025) further extend low-rank adaptation to domain-shifted self-supervised settings.
EP naturally fits the additive PEFT family. introduces a compact learnable query set interacting with frozen tokens via multi-head cross-attention Unlike typical prompt-based methods, it avoids backbone modifications and focuses training on minimal parameters. Thus, EP efficiently combines additive PEFT simplicity with task-specific attentive pooling.
17

---

Published as a conference paper at ICLR 2026
B ADDITIONAL METHODS
We provide here additional attentive pooling/probing methods evaluated in subsection 4.2 but not de- scribed in detail in subsection 3.3. These approaches represent variations of the MHCA framework, highlighting only their key deviations from the default design.
CLIP CLIP (Radford et al., 2021) differs from MHCA by employing self-attention rather than cross-attention. Specifically, CLIP prepends global average pooled (GAP) token to the layer- normalized imput features, treating this token as a global representation. All tokens, including the GAP token, are augmented with learnable positional encodings and processed through sin- gle self-attention block (which includes query projection matrix Wº). The global representation extracted from the output corresponding to the GAP token. Additionally, CLIP includes linear projection matrix Wp after attention aggregation. These modifications enable interactions across all tokens but increase parameter count and computational complexity.
CAiT. CAiT (Touvron et al., 2021) adapts the catenating the learnable query token with the input features and applying self-attention rather than cross-attention. It retains the query projection matrix Wo and includes a linear projection matrix Wp after attention aggregation, followed by an MLP block with GELU activations, residual connec- tions, and LayerScale parameters, similar to V-JEPA. The global representation is obtained from the updated query token after these operations, thus increasing complexity and parameter count relative to the default MHCA variant.
formulation by con-
SigLIP. SigLIP (Zhai et al., 2023; Wightman, 2019) remains close to the MHCA--iit-llarmablle query formulation but retains the query projection matrix Wq. After the attention aggregation, SigLIP incorporates an output projection Wp, followed by a transformer-style MLP block with GELU activation and residual connections, similar to V-JEPA and CAiT. Optional layer normaliza- tion can also be applied before the MLP. These changes add further parameters and computational overhead compared the baseline MHCA design.
CAE. CAE (Chen al., 2023) follows the MHCA- wwwthlearraale-peeeee template closely but tains the query projection matrix Wq and applics separate layer normalization both input features and the query token prior attention. After attention aggregation, it employs additional out- put projection matrix Wp. These modifications introduce additional parameters and computational complexity.
CoCa CoCa (Yu et al., 2022) is aligned with the MHCC--wit-leaammble-que framework but re- tains the query projection matrix Wo and layer-normalizes the query token before computing atten- tion. Attention and value aggregation both occur in peduced-dimensional space, with dimension D. = D, < D;- A final linear projection matrix Woo is then applied to restore the feature di- mension to the original backbone dimension D.. These choices introduce controlled amount of additional complexity and parameters.
In Figure 8 we present a visual comparison of three selected attentive pooling/probing techniques: AbMILP (Rymarczyk et al., 2021), AIM (El-Nouby al., 2024), and V-JEPA (Bardes al., 2024). AbMILP (top-left) serves as a lightweight method, employing a single-head learnable query without additional linear projection matrices, thus requiring only D; parameters. AIM (top-right) extends this by adopting multi-head cross-attention, operating within multiple subspaces. This approach introduces linear projection matrices for keys and values, increasing the number of parameters, yet allowing more expressive query-key interactions. V-JEPA (bottom) represents significantly more complex and computationally intensive architecture. Beyond multi-head attention and multiple linear projections for queries, keys, and values, it integrates an additional projection step, followed by a multi-layer perceptron (MLP) featuring GeL.U activation and residual connections.
18

---

Published as a conference paper at ICLR 2026
a
R
K
N
X
N
MLP
Zproj
X
Figure 8: Visual comparison of three attentive pooling/probing methods. AbMILP (top-left) em- ploys a single-head, learnable query without linear projections, minimizing complexity. AIM (top- right) extends the approach by introducing multi-head attention, operating in multiple subspaces, and applies linear projections keys and values. V-JEPA (bottom) offers more comprehensive architecture by integrating multi-head attention with extensive linear projections and an additional MI.P block with a residual connection, increasing representational capacity.
C ADDITIONAL EXPERIMENTS
C.I EXPERIMENTAL SETUP
Datasets. We evaluate attentive probing across diverse image classification benchmarks. As large-scale dataset, ImageNet-1K (Deng al., 2009) serves the primary benchmark, taining 1.28M images across 1,000 categories. CIFAR-100 (Krizhevsky et al., 2009) provides smaller yet challenging 100-class task with 60K images. To assess scene understanding, we use Places365 (Zhou et al., 2014), comprising 1.8M images spanning 365 scene types. For fine-grained classification, we evaluate on CUB-200 (Wah al., 2011) (11,788 images, 200 bird species), FGVC Aircraft (Maji et al., 2013) (10K images, 100 aircraft models), Stanford Cars (Krause et al., 2013) (16K images, 196 car types), and Food-101 (Bossard et al., 2014) (101K images, 101 food cate- gories). Together, these datasets span wide spectrum of scales and challenges large- scale small-scale, generie vs. fine-graing are-grdi and object- vs. scene-centric taniocccrric-proviiing a comprehensive testbed for probing methods.
Pooling/probing methods. We adopt AbMILP with depth=1, which reduces AbMILP to the formulation described in Table 1 and subsec- tion 3.3. Table 3, we evaluate AbMILP on ImageNet- IK with larger depths (depth-2, 3), which correspond to the MLP-based variants and scale O O(D?), exploring their parame- ver-accurayy tradeoffs. Although deeper MLPs introduce substantially more parameters (+O(D))), the gains in accuracy are marginal. To ensure setting, we consider the most competitive
Table 3: AbMILP architecture ablation increas- ing MLP depth. # PAR.: number of parameters.
METHOD
# PAR.
ACCURACY
AbMILP (depth 1)
769,769
71.74
AbMILP (depth 2)
1.360,361
72.25
AbMILP (depth 3)
1,950,953
72.84
fair comparison in the AbMILP variant hdepth=1) default, which successfully lies on the Pareto front.
Implementation details. We evaluate 15 models spanning five pre-training paradigms: four masked image modeling (MAE (He et al., 2022), BEiTv2 (Peng et al., 2022), SimMIM (Xie et al., 2022), CAPI (Darcet et al., 2025)), two joint-embedding (BYOL (Grill al., 2020), DINO (Caron et al., 2)21)), two hybrid (iBOT (Zhou et al., 2022), DINOv2 (Oquab et al., 2024)), three vision- language (CLIP (Radford et al., 2021), SPgL.P (Zhai et al., 2023), SigL.PP (Tschannen al., 2025)),
19

---

Published as a conference paper at ICLR 2026
and two generative DiT (diffusion) (Peebles & Xie, 2023) and AIMv2 (autoregressive) (Fini et al., 2025). Architectures range from small (e.g., ViT-S for MAE) to extra-large (e.g., DiT-XL for DiT).
To ensure fair comparison of pooling/probing methods, mostly adopt the LARS optimizer You et al. (2017) and conduct a learning rate search in the range [0.1. 5.0] with a step size of 0.1 for each model. For large-scale datasets such as ImageNet- to 0.1 due to the computational cost of an exhaustive search. All models are trained for 90 epochs with 10 warm-up epochs, ensuring consistent training schedules even though many models converge much earlier (e.g.. SigLIP within 15 epochs). The effective batch size is set to 4096 for all datasets except FGVC Aircraft, where reduced to 512 due the smaller dataset size. Data augmen- tation follows standard PyTorch (Paszke et al.. 2019) image pre- pre-processing: RandomResizedCrop horizontal flipping, and normalization. For vision-language models, we adopt their official prepro- cessing pipelines (e.g., OpenCI.IP (Ilharco et al., 2021) transforms for CLIP, SigI.I,, and SigI.IP2) to ensure alignment with pre-training distributions. Experiments are conducted on cluster of NVIDIA A100 GPUs (40 GB VRAM each) and on the LUMI supercomputer, on clusters of 8 AMD Instinct MI250X GPUs (128 GB HBM2e memory each).
1K KK and Places365, we fix the learning rate
C.2 EXPERIMENTAL RESULTS
Accuracy vs. parameters. We extend the benchmark presented on subsection 4.2 include ad- ditional pre-training methods and datasets. Figure 9 and Figure 10 present the trade-off between top-1 accuracy and the number of trainable parameters (including the classifier) for various pool- ing/probing methods integrated into MAE and SimMIM with different backbone sizes (VīT-S, ViT-B and ViT-L). The evaluation spans multiple datasets, including FGVC-Aireraft, CUB200, Places365, CIFAR-100 and 19rs196.
As shown in Figure 9a and Figure 9b, EP consistently outperforms standard linear probing across MAE VIT-S and ViT-L. Notably, on MAE ViT-L, EP with D. = D./2 achieves an accuracy boost of 79.1% surpassing linear probing by 3.1% while maintaining the same number of trainable parameters. Furthermore, EP 128 reaches 79.4%, outperforming SigLIP, while reducing the number of trainable parameters by over 11M.
In Figure 9c, we benchmark attentive probing on SimMIM ViT-B pre-trained on ImageNet- IK. Baselines such as [CLS] and GAP remain parameter-efficient but yield relatively low accuracy. Classical attention modules like CBAM and AbMILP do not improve this trade-off, while methods such as ViT, V-JEPA, and SigLIP achieve higher accuracy but at the cost of orders-of-magnitude more parameters. EP strikes a favorable balance: scaling the number of queries (EP2-EPo4) consis- tently increases accuracy, while reducing D, effectively lowers parameter count with only moderate drops in performance. Notably, EP2 with D, D;/2 achieves 60.6% top-1 accuracy using fewer than 0.7M parameters, outperforming GAP and DELF under similar budgets. At the high end, EPsa reaches 65. 15% % accuracy, closing the gap to heavy-weigh, probing methods while remaining lighter.
In Figure 9d, our EP24 variant, for the FGVC-Aircrat dataset, achieves a remarkable accuracy boost of 61.2% (+19.5%) while maintaining lower parameter count than linear probing (41.7%). Simi- larly, in Figure 9e for the CUB200 dataset, our EP& with D. D./ variant achieves comparable accuracy (75.9% with computationally costly poolings such as SigLIP (77.8%) with around 7M trainable parameters less.
Finally, in Figure 9f, GAP and [CLS], the two primary baselines, exhibit high parameter efficiency but low classification accuracy. In contrast, methods like SigL.P (54.39%), ViT (54.84%), and V- JEPA (52.44%) achieve higher accuracy, albeit at the cost of increasing the number of trainable parameters. EP has the best trade-off between accuracy and parameters, achieving top- classifica- tion accuracy of 53.7% with just 1M extra trainable parameters (EP64).
Moving to the additional datasets shown in Figure 10, observe consistent benefits of EP. On CIFAR-100 (Figure 10a), EP128 achieves 78.9%, which is close to the best-performing attention pooling methods, while using significantly fewer parameters. On 1ars196 (Figure 10b), our EP variant achieves 82.7% top-1 accuracy, clearly surpassing repurposed pooling methods such as DELF, while requiring far fewer parameters. Smaller variants (c.g.. EP and EP2) already provide substantial gains over linear probing, showing that even lightweight configurations of our method maintain strong performance on fine-grained datasets.
20

---

Published as a conference paper at ICLR 2026
(a) MAE ViT-S ImageNet-Ik
(b) MAE VIT-L ImageNet-IK
70
VIT
EP
SigLi
AIMCLIP
CAR
65
EF64
SigL.I
DELF
top-1 accuracy (56
Call
AbMIL
SimPool
EPys
EP4
CoCa
EPag EIPs
60
1PP4
EP
[CLS
55
EP
AbMILLY
DELF SimP
BP
Do D₁
Do D₁
Do
Da/2
Do Da/2
50
FP
Dz/4
D. Dz/4
GAP
GAPCRAM
DL/8
Do
45
10
10
10t
107
(c) SimMIM ViT-B ImageNet-IK
(d) MAE VIT-B PGVC-Aireral
VIT
SigLiP
70
EP64
UJRPAA
64
65
AIM
top-1 accuracy (%
EP64
EPs
EP64
EPg
CAB
CLIP
Ca
KP
1P21
8P4
DE
ATM
CLIIP
CAR
60
EP4
CxCh
AbMLLP
LP's
SinaRool
EP
55
EP49
EPa
GAP
DELF
EP's
[CLS AAMILP
Simpheol
SPg
50
Du
12/2
[CES)]
BRA
Du D2
Du
12/4
D. D2/4
CBAM
D3/8
GAP
D2/16
45
10
10
10
10
10
(e) MAE VIT-B CUB200
(I) MAE VIT-B Places365
80
CAR
EP
var CLIP
FPL6
EP
AIM
VJJEPW
Elfes
CAB
Sigt.P
top-1 accuracy (%)
70
ADMILP
SimFFol
CLII
52
V-ЛEPA
60
CCSS
50
SimPoool
50
AbMILe
D.
CBAM
D.
/
48
40
D. D
[CLS]
105
10
107
10c
number parameters
numher of parameters
Figure 9: Top-1 classification accuracy vs. number of trainable parameters (including the classifier) for two self-supervised learning methods, with backbones of varying size (a, b) and across various datasets (d, e, f). EP variants are marked with different colors for different output dimensionalities D,, F.P efficient probing with M learnable queries. [CLS] linear probing using the class token; GAP: global average pooling over patch tokens; VIT: default transformer block.
Layer-wise probing, Figure 11 presents layer-wise comparison between standard linear probing (LP) and efficient probing (EP) using patch token representations from intermediate layers of pre-trained and frozen MAE with ViT-B. While LP exhibits a clear degradation in per- formance as we move toward earlier layers (dropping from 67.7% at layer 12 just 45.8% at layer 6), EP demonstrates remarkable robustness. It maintains high accuracy from lower layers, with performance stabi-
Figure 11: ImageNet- I accuracy from intermediate MAE VIT-B layers for LP vs. EP, with EP-LP gains.
LAYER
LP
EP
GAIN
67.7
75.6
+7,9
10
66.2
75.9
+9,7
64.5
75.4
+10.9
45.8
69.6
+23.8
21

---

Published as a conference paper at ICLR 2026
(a) MAE V CIFAR- 100
(b) MAE VIT-B Cars- 196
VIT
FF EP
ADSEAE
FF
EPas
FP
AIM
CLIP
CAE
SigL CuT
80
10
EP3
CoCa
CLIP
SigL.IP IP
top-1 accuracy (%)
78
EP4
EP's
EP4
CoCa
V-Japu
LP EPs
2-
AbMILP
DELF
76
EP3
DELF
70
EP4
SirnPool
SimPual
74
A
EPp
60
72 EP2
D. D2
0 Do D2
70
[CLS
GAP
3AM
Do D./4
50
CRAM
0 Do Dz/4
D. D2/16
GGAP
Do D,/16
10
10
107
10a
10c
107
number of parameters
number of parameters
Figure 10: Top-/ classification accuracy vs. number of parameters for MAE ViT-B on two datasets (a, b). We evaluate both dedicated probing mechanisms (c.g.. V-JEPA) and repurposed attentive pooling methods (e.g.. CLIP), EP variants are marked with different colors for different output di- mensionalities D.. EP as: efficient probing with M learnable queries. [CLS]: linear probing using the classification token; GAP: global average pooling over patch tokens; VIT: default transformer block.
lizing beyond layer 9. Notably, EP yields significant relative improvement of +23.8% at layer over LP, underscoring its ability to extract and utilize meaningful representations from less seman- tically enriched stages of the encoder. These results highlight the effectiveness of EP in unlocking information from carlier layers that standard LP fails exploit.
Low-shot probing. Table 4 evaluates the per- formance of LP, EP, and FT under limited super- vision, using only 5% and 10% of the ImageNet- 1K training set, stratified by class. Although substan-
Table 4: Top-1 accuracy ImageNet-Ik with limited training data for MAE ViT-B. Results for linear probing (LP), efficient probing (EP), and fine-tuning (FT) on 5% and 10% subsets. The last column shows the percentage of the LP-FT performance gap closed by EP. For ref- erence, the gap closed by EP on the full training set (100%) is 49.7%.
L.P struggles in this low-shot regime, tially bridges the gap toward FT. Specifically, EP closes 74.8% and 71.5% of the -P perfor- mance gap for the 5% and 10% subsets, respec- tively. These improvements are particularly im- pressive given that EP remains significantly more parameter-efficient than FT, with a complexity comparable to that of LP. These findings highlight the strong data efficiency of EP.
SUBSET L.P EP
FT
% GAP
5% 10%
49.6
60.9 64.7 55.9
74.8%
65.2
68.9
71.5%
In- and out-of-domain k-NN evaluation. To further examine how EP behaves relative to LoRA- based fine-tuning, perform cross-dataset k-NN evaluation on MAE ViT-B features, using ImageNet-IK: StanfordCars, and Food101 as target datasets (Table 5). The first row reports the baseline k-NN accuracy obtained directly from the frozen MAE backbone. The next three rows evaluate features produced by EP when the probe is trained on each dataset independently; the diag- onal entries (c.g., 70.5% on ImageNet-IK, 70.0% on StanfordCars, 75.2% on Food101) correspond to in-domain performance, while off- off-diagonal entries measure cross-dataset generalization. The last three rows report the corresponding results when MAE is adapted with the best-performing LoRA configuration for each dataset. As expected, LoRA generally achieves the strongest in-domain ac- curacy (e.g., 72.3% on ImageNet-11. 75.4% on StanfordCars, 80.3% on Food101), reflecting the benefit of supervised feature adaptation. However, EP consistently provides stronger comparable out-of-donme -o-mad----minn performance: for instance, when trained on StanfordCars, EP features achieve 51. on ImageNet- 1K and 42.0% Food101, compared to 45.8% and 38.7 7877% for LoRA-tuned features; similarly, EP trained on Food101 yields 58. 2% on ImageNet- IK and 23.4% on StanfordCars, ver- 56.7% and 15.2% for LoRA. These results suggest complementary behavior: LoRA excels
22

---

Published as a conference paper at ICLR 2026
specializing the backbone specific task, while EP preserves more of the original pre- tre-trained structure and thus offers more robust cross-datase generalization.
k-NN evaluation on MAE ViT-B using frozen, EP-proeed, and LoRA-tuned features. Default EP corresponds to 32, while the "best LoRA" configuration applies LoRA to all 12 layers the Wq, WK. Wv, and Wo projection matrices with rank p=8.
Table 5:
FEATURES
IMAGENET-IK
STANFORDCARS
FOOD101
frozen MAE backbone
46.1
9.5
28.8
EP probed on ImageNet. IK
70.5
31.4
64,2
+ probed on StanfordCars
42,0
EP probed Food-101
58.2
23.4
75.2
best LoRA tuned ImageNet-IK
72.3
24.7
63.6
+ best LoRA tuned on StanfordCars
45.8
75.4
38.7
+ LoRA luned Food-101
Object localization. A key empirical property of EP that its queries specialize comple- mentary and semantically meaningful object parts (subsection 4.3). To examine EP utility beyond standard classification, mance in an unsupervised object localization set- ting, ImageNet-1K (Table 6), using the WSOL protocol (Choe et al., 2020; 2022). We average EP 's attention maps, without any modification or additional training, and compare them to the standard last-layer [CLS] > patch attention. Across five backbones and three model sizes, EP consistently improves MaxBox V2 by +9.8% on average. This demonstrates that EP 's attention maps act as strong unsupervised localizers "out- of-the-boxx..
Table 6: Unsupervised ImageNet-IK localiza- tion (MaxBoxAccV2). EP substantially im- proves localization quality across five back- bones and three model sizes, with an average gain of +9.8% over baseline attention.
evaluate its perfor-
MODEL
ARCH
LP
EP
AAcC, (%)
MAE
VIT-S/16
46.0
58.5
+12.5
MAE
VIT-B/16
54.2
60.4
+6,2
+14.3
BETTv2
VIT-B/16
47.0
61.2
+14,2
SimMIM
VIT
45.2
50.0
+14.8
iBOT
VIT-R/16
57.6
63.7
46.1
SigLIP
VIT-L/16
44.0
44.2
40,2
AVERAGE
48.7
58.5
+9.8
Image retrieval. Table 7 and Table 8 evaluate EP in zero-shot image retrieval setting, following standard @ protocol on two fine-grained datasets, CUB200 and Cars196, respectively. the same EP mechanism trained once ImageNet-IK, without any further
In both cases, adaptation on the target dataset. Across all retrieval experiments, we utilize the entire datasets (full set of images and classes), all of which remain fully unseen during the training stage on ImageNet- 1K. Every image serves as a query, ensuring a robust zero-shot evaluation protocol. On CUB200, EP consistently improves performance across five models (e.g., +40.8% 8% for MAE, +1 1.2% for BEiTv2, +10.3% for iBOT, +5.3% for CLIP, +15.9% for SigLIP on @ metric). Similarly, on Cars196 the gains remain significant in most cases (e.g., +21.5% for MAE, +8.6% for iBOT, +4.6% for SigLIP). These findings show that EP 's features can be used "out-of-the-be for image retrieval, consistently outperforming the features of the frozen backbones.
C.3 EXPERIMENTAL ANALYSIS
Attention complementarity. To further expand our analysis of attention complementarity. we complement the average-based metric reported in Figure with an additional measure: 1- max off- diagonal similarity, which reflects the strongest redundancy between predictors. As shown in Fig- ure 12a and Figure 12b, EP achieves substantially higher complementarity than the internal MHSA heads across all backbones, regardless of whether we use the average or max metric. A natural con- cern is whether this effect is biased by considering only the last block. To address this, compute complementarity for MAE VIT-B across all 12 blocks (Figure 12c). The best scores obtained inter- nally (0.37 for average, 0.10 for max) remain far below those of EP (0.65 and 0.22, respectively), confirming that the gap is not specific to the last block. Another question whether the low diversity arises from self-attention itself. However, SigL.P SigL.IP_hhich uses cross-attention mechanism in
23

---

Published as a conference paper at ICLR 2026
Table 7: Zero-shot image retrieval performance (Recall@ K) on CUB200 dataset (images: 11788,
classes: 200) across five different backbones with and without Efficient Probing (EP).
FEATURES
R@1
R@2
R@4
R@8
R@16
R@
frozen MAE backbone
16.3
23.7
33.3
45.0
58.0
70.8
on ImageNet-IK
57.1
69.5
79.5
87.5
93.0
96.2
frozen BEITV2 backbone
57.0
68.2
78.0
85.8
90.9
94,2
ImageNet-IK
68.2
78.7
86.7
92.3
frozen iBOT backbone
51.8
74.7
90.0
EP ImageNet-IK
62.1
73.7
82.4
89.7
94.0
96.8
frozen CLIP backbone
75.0
84.3
91.3
95.3
97.5
98.7
+ EP on ImageNet-1K
80.3
88.0
92.9
95.8
97.6
98.6
frozen SigL.IP backhone
60.8
72.1
81.6
88.8
93.5
96.4
+ on ImageNet- IK
76.7
85.7
92.0
95.7
97.8
98.8
Table 8: shoo-shot image retrieval performance (Recall@ K) on Cars196 dataset (images: 16185, classes: 196) across five different backbones with and without Efficient Probing (EP).
FEATURES
frozen MAE backbone
12.7
17.6
23.7
31.8
42.9
55.7
ImageNet- IK
34.2
44.8
55.6
65.9
76.4
85.2
frozen BETTV backbone
48.3
72.3
EP on ImageNet-18 IK
44.2
56.4
68.5
79.1
87.3
93.7
frozen IBTT backbone
31.5
40.7
50,2
60.0
69.9
79.0
+ EP on ImageNet-IK IK
40.1
50.6
61.0
70.7
79.5
87.6
frozen CLIP backbone
79.6
88.9
94.6
97.8
99.2
99.7
+ ImageNet-IK
79.8
94.8
99.2
frozen Sigl IP hackhone
85.8
92.3
96.1
98.0
99.1
99.6
ImageNet-IIK
90.4
95.4
97.8
its last block- still yields lower complementarity than EP. suggesting that the effect is not explained by self-attention alone.
Figure 12d and Figure 12c further compare EP against other attentive probing methods (V-JEPA, AIM). Interestingly, all attentive probing mechanisms achieve higher complementarity than the ternal MHSA heads, with AIM coming close to EP. This pattern might suggest that attentive probing, when designed effectively, encourages predictors to specialize in complementary regions, likely be- cause probing operates on frozen backbones and must learn to aggregate features as efficiently as possible. This evidence indicates that complementarity may be an inherent property of attentive probing. rather than byproduct of backbone architecture or attention type, opening new directions for future work.
loss. We Attention complementarity further examine the role of query com- plementarity by encouraging diversity be- tween queries/heads via an auxiliary attention- similarity loss. Specifically, for AIM, V-JEPA,
validation accuracy
60
and EP we add to the cross-entropy loss, an extra term Lamn that penalizes similarity be-
tween the attention maps of different queries. On MAE VIT-B probed on ImageNet- 1K, this yields small gain for V-JEPA (74.1% + 74.3%), while AIM and EP retain essentially identical final accuracy but converge faster (e.g. EP reaches 64.1% at epoch 10 with Catn vs. 60.0% without it, before both converge to the same final score). These results suggest that complementarity is a beneficial property that can modestly help or accelerate training. How- ever, methods such AIM and EP, learn highly
atto-sim
EP
0
20
40
60
80
epoch
Validation top-1 accuracy on ImageNet- over training epochs for EP, with and without the auxiliary attention-similarii loss. The additional loss slightly accelerates early train- ing but both variants converge to identical final performance.
Figure 13:
24

---

Published as a conference paper at ICLR 2026
Complementarity
0.8
0.09
0:77
9:75
0.96
MISA
0.75
0.75
(0.03
0.01
0.6
0.44
0.99
0.4
0.24
0.97
0.24
0.37
0.2
0.18
0:18
MAE
MAE
IBOT
DINOv2
DINOy2
DINOv3
DINOv3
Franca VIT-
CLIP
SiglIP VIT-L
VIT-L
(a) Complementarity measured as minus average off-diagonal similarity.
0.3
Complementarity
0.2
0.19
0.17
0.15
0.1
0.07
0.06
0.004
0.004
0.004
0,005
MAE VIT-B
MAE
IBOT VIT-B
DINOv2 VIT-B
DINOv2 VIF-
DINOv3 VIT-B
DINOv3 VIFL
Francu
CLIP VIHL
SigLIP WTFL
VIFL
WITL
(b) Complementarity measured as 1 minus max off diagonal similarity.
0.4
Complementarity
maxs
0.3
01.23
01.29
0.20
0..24
0.24
0.2
0.19
0.14
0.1
0.04
0.04
0:07
0.08
0.00
0.10
0.:02
0.00
(c) Complementarity MAE ViT-B across blocks, using both average and max.
0.8
0.76
Complementariyy
0.6
0.50
0.63
0.65
0.7
V-EBA
0.4
0.36
0:33
EP
0.2
MAE
BEITv2 ViT-B
SimMIM VIT-B
CAPI VIT-L
VIT.B
(d) Complementarity measured as minus average off-diagonal similarity.
0.3
Complementariy
0.2
0.2
0.10
MISA
0.17
0.17
V-JEPA
0.00
0:08
0.11
0.1
ATM
CAPI VIT-L (e) Complementarity measured as minus max off-diagonal similarity.
MAE
BEITv2 WIT-B
SimMIM VIT-B
Figure 12: Complementarity scores of attention maps across different backbones (a, b. c) and prob- ing methods (d, e). We compare the diversity of internal MHSA heads in the last block against the external V-JEPA heads, AIM heads, and EP queries.
25

---

Published as a conference paper at ICLR 2026
complementary queries, without Laun. via the attention mechanism that naturally discovers this structure on its own.
Entropy. Figure 14 examines the relation between last block attention entropy and probing per- formance across different pre-training methods. Models with lower entropy exhibit more concen- trated and focused attention distributions, correlating with stronger probing accuracy under EP. This is particularly evident for methods like DINOv2 and DINOv3, which couple low entropy with strong LP and EP performance. In contrast, models like MAE VIT-S (MAE-S) or SimMIM VīT-B (SimMIM-B) show higher entropy, reflecting more diffuse attention and correspondingly weaker probing under LP. Crucially, EP consistently boosts accuracy across all entropy levels, with bubble sizes indicating particularly large gains for the high-entropy models. This aligns with our broader finding that methods optimizing patch-level representations rather than explicit global tokens benefit most from attentive probing, as EP effectively compensates for diffuse attention and makes probing more robust to the quality of backbone distributions.
EP
90
DINDWM.
DINOv2-L
Franca-L.
DINOVL-B
DINDw2-B
85
80
CAPI-L
BRITV2-R
IBCTEB
MAE
DINO-B
80
70
60
Simmim-B
70
50
MAF-S
40
65
3
3.2
3.4
3.6
3.8
4.2
4.4
4.6
4.8
[CLS] patch tokens entropy (last block)
Figure 14: Entropy analysis of last-block attention distribution across different pre-training meth- ods. Bubble color indicates efficient probing (EP) accuracy, bubble size encodes A accuracy (EP-LP). Lower entropy corresponds to more focused attention and higher accuracy under EP.
Impact of attention predictors, Da, and Do. We analyze the effect of increasing the number M of heads (in AIM) and the number M of queries (in EP) on probing performance. Figure 15 shows that both lead to accuracy improvements. For AIM,
EPss
AlMB4
ATM
EP'
AlM
75
1G
AIM
top-1 accuracy
EP
AIM
BP16
increasing the number of heads incurs no additional cost terms of parameters, but its effectiveness depends the presence of WK. In contrast, EP achieves similar better performance by leveraging addi- tional queries, while removing WK- AIM introduces an additional attention dimen- sionality D... since its query is Icarnable and interacts with WK. Lowering D. re- duces the parameters but leads to greater accuracy drop (green points), indicating that the learned query formulation benefits from a large attention space. We also eval- uate the impact of reducing the output di- mensionality D, (blue points). On EP, we observe that lowering D, to D2/2 reduces
74
AlM
AIM
FP
AIM;
AIMg
73
FP
ATM2
Do D.
AIM
D. D./2
0,/2
72
1. 10
1.5 10
2-10
number of parameters
Figure 15: Efffect of varying the number of heads/queries M, Da. and output dimension D, probing accuracy. Black: standard (D, D2); blue:
reduced classifier dimension (D, D./2); green; re-
duced attention dimension (D₂ = D:/2y; half-blue- half- green: simultaneous reduction of both D. and D..
26

---

Published as a conference paper at ICLR 2026
parameters while maintaining competitive performance. Interestingly, this strategy also general- izes well to AIM, demonstrating that extracting lower-dimensional features can achieve comparable accuracy with reduced computational cost.
EP convergence. Although all probing methods are trained for 90 epochs for fairness (following the protocol of He et al. (2022) and Przewikežlikowski et al. (2025)), we also examine how quickly EP converges. As shown in Table 9, EP trained for only 10 epochs (EP@ 10) already matches or even surpasses the performance of standard linear probing trained for the full 90 epochs (LP@90) across 12 models. Moreover, EP recovers on average more than 97% of its final accuracy within these first 10 epochs. These results highlight that in practical scenarios, EP requires only small fraction of the standard training budget to achieve near-optimal probing performance and produce highly discriminative features.
Table 9: EP convergence
Me. E 10 is EP accuracy after 10 epochs; LP@90 and E are final probing results after 90 epochs.
MODEL
EP@
L.P
E VS. LP EP@ 10 Vs. EP
MAE VIT-B/16
60.0
67.7
75.6
88.6%
79.4%
BETTV VIT B/16
80.0
79.0
81.7
101.3%
97.9%
CAPI VIT-L/14
82.3
81.5
101.0%
98.4%
BOT VIT-B116
78.7
DINOv2 VIT-B/14
83.6
83,2
84.0
100.5%
99.5%
DINOv2 VIT-L/14
85.0
85,2
85.6
99.8%
99.3%
Frunca VIT-B/16
83.8
83.8
84.3
100.0%
99.4%
DINOv3 VIT-B/16
84.4
84.0
84.4
100.5%
100.0%
DINOV3 VIT-L/14
86.7
86.6
87.1
100.1%
99.5%
VIT-L/14
82.3
SiPLIP VIT-L/16
85.6
84,1
86.1
101.8%
99.4%
AIMv2 VIT-L/14
85.5
84.8
85.9
100.8%
99.5%
Attention mixing. We fur- ther explore attention mix- ing variant of EP, where a larger set of query maps is linearly projected into a smaller num- ber of effective maps. For ex- ample, projecting 64 queries into 8 maps achieves 75.6% ac-
Table 10: Comparison of EP variants using different numbers of original maps (queries) and projected maps ImageNet-IK K with MAE ViT-B. Attention mixing refers to projecting multiple query maps into smaller set via linear+softmax layer. While projec- tion improves over directly training with few queries (e.g., 64-+8 8 original), it does not outperform the un-projected setting with the same number of queries, and adds extra parameters.
curacy, outperforming the 8- query baseline (74,%% and suggesting that mixing can im- prove the expressivity of a small query budget. How- ever, compared to the un- projected 32 64 query vari- ants (75.6%), attention mixing offers no clear advantage while introducing additional param- eters through the projection layer. For this reason, we do not include attention mixing the final design of EP, though may inspire future extensions exploring alternative aggrega- tion strategics.
# ORIGINAL EP MAPS
# PROJECTED EP MAPS
ACCURACY
ORIGINAL EP MAPS
ACCURACY
32
8
75.4
75.6
128
8
75.5
64
16
75.5
128
16
75.8
16
75.0
256
75.6
75.6
128
32
75.6
32
75.6
256
32
75.7
128
64
75.8
256
64
75.8
75.6
384
64
75.9
256
128
75.8
128
75.5
384
128
76.0
Matryoshka representation learning. In our accuracy vs. parameter trade-offss (e.g.. Figure 2). each evaluation scale (D,=D;, Dz/2, D,/4, Dz/8) requires training a separate classifier, and the resulting accuracies potentially serve as upper bound for that dimensionality. since each probe is fully specialized to its target output size. To reduce training cost and enable a single probe that op- erates across multiple scales, investigate whether Matryoshka representation learning (Kusupati
27

---

Published as a conference paper at ICLR 2026
al., 2022) originally proposed for training from seratch fine-tuning- be applied to atten- tive probing under EP. Matryoshka jointly optimizes probing across multiple output dimensionalities by summing their losses (e.g.. L=>Ln+LD/2) We study two variants: (i) Efficient Matryoshka, which uses a single classifier with nested subspaces spanning multiple dimensionalities, and thus adds extra parameters; (ii) Vanilla Matryoshka, which trains separate classifiers for each dimen- sionality, increasing the parameter count. Table 11 reports results with MAE VIT-B probed using EP. Without Matryoshka, probing D (D,=D,) achieves the highest accuracy (75. 6%). but perfor- mance drops sharply at smaller dimensions (50.7% 7% at D/2, 20.1 at D/4, 5.5% at D D/8). Efficient Matryoshka substantially improves performance at reduced dimensions (e.g.. +22.5% D /2 while maintaining accuracy at D. Accuracy at D slightly decreases as À (the weight of the full-dimensional loss) reduced, reflecting the trade-off between full- and low-dimensional performance. Vanilla Matryoshka alleviates the performance drop at 1) but requires more parameters. To further illustrate these effects, we evaluate the standard linear probing (LP) baseline without Matryoshka under the same D/2, D/4, D/ evaluation protocol, also extending LP with Matryoshka. In Table 12 observe that standard LP (without Matryoshka) suffers from an even sharper performance collapse than EP (e.g., 34.5% D/2, 10.8% D/4, and 1.4% D/8). This confirms that the drop reduced dimensions is not specific fic to EP, but is an inherent limitation of evaluating ViT representa- tions at truncated dimensionalities without explicit multi-scale optimization. Overall, Matryoshka probing provides a promising way to probe across multiple scales simultancously, improving effi- ciency, and udaptability of evaluation. However, it does not yet match the upper-boumd performance of dimension -seciifc probes, thus we present it here as an exploratory analysis that highlights an exciting direction for future work.
Table 1I: Comparison of EP with and without Matryoshka representation learning on ImageNet- (MAE ViT-B). Without Matryoshka, probing at (D,=D,) achieves the best accuracy but collapses smaller dimensions. Efficient Matryoshka enables multi-scale probing without extra parameters, improving performance at reduced dimensions with relatively small degradation at D. Vanilla Matryoshka partly restores performance at D but the cost of additional parameters.
WITHOUT MATRYOSHIKA
EFFICIENT MATRYOSHIA
VANILLA MATRYOSHKA
D/2
D/4
D/8
D. D/2,
D/4, D/8
D. D/2, D/4, D/8
BVAL. ON
A=
A=0.6
2=0.4
0002
A=0.6
D
75.6
75.0
74.7
74.3
73.9
75.4
75.2
D/2
50.7
74.4
72.2
73.1
73.6
73.8
73.8
74.4
D/4
20.1
72.4
70.9
71.4
71.6
72.2
D/8
5.5
69.8
65.7
67.3
67.9
68.2
68.6
68.9
1.4M
1.4M
14M
1.4M
PAR.
1.4M
2.1M
5.5M
Table 12: Comparison of LP with and without Matryoshka representation learning ImageNet- K (MAE ViT-B). Without Matryoshka, linear probing collapses sharply at reduced output dimension- alities. Matryoshka improves low-dimensional performance, without increasing parameters, though the accuracy at D decreases as (the weight of the full-dimensional loss) is reduced.
WITHOUT MATRYOSHKA
EFFCIENT MATRYOSHKA
À 0.6
0.4
0.25
66.4
66.1
65.8
D/2
34,5
59.2
60.6
61.2
61.4
D/4
10.8
52.5
54.5
55.3
55.7
D/8
1.4
43.4
46.1
47.0
47.4
Visualizations. To better understand the behavior of different attentive pooling/probing methods, present visualizations of attention maps across various configurations.
Figure 16 explores the effect of varying the number of queries in EP, visualizing configurations with 2, 4, and more queries. When only single query (EP:) is used, the attention map tends to capture a coarse, global representation of the object. As the number of queries increases, the attention becomes more fine-graine and spatially distributed, with each query specializing in distinct object
28

---

Published as a conference paper at ICLR 2026
original image
EP f
BPP
EP,
EP4
EP:
EP4
q1
42
41
42
q3
q4
16
16
EP
EPib
91
42
94
95
90
47
iP
EPin
EP
EP.6
EP
EP.6
EP15
EP
qa
410
411
412
4:3
41
9:.5
9:6
Figure 16: Attention maps of efficient probing (EP) variants grouped by the number of queries (1, 2, etc.). MAE ViT-B pre-trained on ImageNet-IK, probed with EP. Images: ImageNet- Ik validation set.
regions. This highlights the flexibility of EP in controlling the granularity of attention: fewer queries encourage holistic coverage, while more queries promote detailed, part-based localization.
Figure 17 shows the attention maps obtained from four single-head attention probing methods (CBAM, AbMILP, DELF, and SimPool) using an ImageNet- pretrained MAE ViT-B model. Among them, CBAM exhibits poor localization, often failing to focus on the target object, which consistent with its low classification accuracy across datasets. In contrast, AbMILP, DELF, and SimPool produce more precise and meaningful attention, highlighting relevant object regions while suppressing background noise. Due to their single-head nature, these methods are compelled to concentrate all semantic information into single attention vector, which encourages global view of the input image rather than fine-grained discrimination.
Figure 18 compares attention maps from multi-head probing methods. Rather than visualizing just the average attention across heads, which can obscure useful per-head behavior, we show the min- imum, maximum, and standard deviation across attention heads. The first column contains maps from the [CLS] ] token of the pretrained MAE ViT-B model. The remaining columns display maps from CAE, CaïT, CLIP, CoCa, VIT, V-JEPA, SigLIP, and AIM, alongside EP using 16 learnable queries (EP16). Notably, EP produces high-quality attention maps that rival the best pest-performing methods both clarity and relevance, while retaining computational efficiency.
Figure 19 presents the attention maps corresponding to each individual query in EP. We observe that each query qi attends distinct, complementary regions of the object (e.g., head, torso, bound- aries), illustrating how EP distributes attention cooperatively across salient features without redun- dancy. This diversity among queries reveals the model's capacity to decompose complex objects into meaningful sub-parts.
29

---

Published as a conference paper at ICLR 2026
input image
AbMILP
SimPool
Figure 17: Attention maps of single-head attention pooling methods. MAE ViT-B pre-trained on ImageNet-Ie 1k. Images: ImageNet-Il 1k validation set.
30

---

Published as a conference paper at ICLR 2026
mensn
min
std
min
maa
CafT
CLIP
CoCa
V-JEPA
SigLIP
EP
Figure 18: Artention maps of multi-head attention pooling methods for different attention predictor aggregators: mean, standard deviation (std), minimum (min), and maximum (max). MAE VIT-B pre-trained on ImageNet- KK Images: ImageNet- Ik validation set. EP1e: efficient probing (EP) with 16 queries.
$s
Figure 19: Attention maps of efficient probing (EP) with 8 queries. Each query qi learns focus on distinct and complementary regions, capturing diverse spatial and semantic information. MAE VIT-B pre-trained on ImageNet- K, probed with EP.. Images: ImageNet- validation set.
31

---

Published as a conference paper at ICLR 2026
D LIMITATIONS AND FUTURE WORK
While our study provides the first systematic benchmark of attentive probing and introduces a lightweight yet effective alternative, several limitations remain. First, our evaluation focuses ex- clusively on frozen backbones. Although this setting isolates the effect of probing mechanisms, it leaves open how efficient probing might interact with lightweight fine-tuning strategies or adapter- based methods. Exploring the synergy between probing and parameter-efficte fine-tuning could offer a broader view of scalable evaluation.
Second, our current experimental protocol always performs full forward pass of the backbone during training, even though the backbone frozen. practice, one could pre-compute and store patch-token features once and run probing purely on top of these cached features. Systematically studying this "frozen-feature" protocol-its memory/IO trade-offs, impact on optimization dynam- ics, and applicability across architectures--s an interesting direction for future work.
Third, while we introduced complementarity metrics (average and max similarity) to study the diver- sity of attention predictors, analysis remains largely diagnostic. A deeper theoretical understand- ing of why attentive probing tends to yield more complementary maps-and and whether this property can be explicitly optimized-coe further extend probing beyond evaluation into representation refinement.
Fourth, experiments concentrate image classification benchmarks. Attentive probing may also benefit other tasks that naturally require part-level reasoning, such as detection, segmentation, retrieval, where its complementary attention maps could act as implicit part detectors. Extending probing to such structured tasks is a promising direction.
Fifth, although examined variants such attention mixing and Matryoshka probing, both were limited to ImageNet-scale experiments. Their potential on larger models and multimodal settings (e.g., vision-language tasks) remains underexplored. Similarly, the alternative protocol of probing with pre-stored frozen features, requiring only one backbone forward pass, is left for future work.
Finally, the broader implications of probing as more than an evaluation protocol deserve attention. Our results suggest that attentive probing exhibits emerging properties- such as diversity and inter- pretability-ther are not trivially inherited from the backhone. Understanding whether these prop- erties generalize across modalities and can be exploited for tasks like explainability, robustness, or adaptive computation opens up an exciting line of futurre research.
E
USE OF LARGE LANGUAGE MODELS
This paper has made limited use large language models (LLMs), specifically to aid in the pol- ishing and refinement of writing. LLMs were not used for ideation, technical contributions, exper- imental design, analysis, or related work retrieval. All research ideas, methodology, experiments, and conclusions presented are solely the work of the authors.
32


Reviews:
[{"review_id": "nTKNF1VqnH", "reviewer": "Reviewer_4JLQ", "summary": "This paper addresses the challenge of efficiently evaluating large-scale, pre-trained vision models. The authors argue that as \"fine-tuning becomes increasingly impractical at scale\", probing is emerging as the preferred evaluation protocol. However, standard linear probing (LP) \"fails to adequately reflect the potential of models\" like Masked Image Models (MIM) or autoregressive models, where information is distributed across patch tokens rather than a single global [CLS] token. This motivates \"attentive probing\", which uses attention to aggregate patch-level features. The paper notes that existing attentive probing methods suffer from \"excessive parameterization and poor computational efficiency\". The aothors present the \"first comprehensive study of existing methods\"  for attentive probing, benchmarking their accuracy and efficiency. They also propose \"efficient probing (EP)\" , a \"simple yet effective multi-query cross-attention mechanism\" that \"eliminates redundant projections\"  to reduce parameter count and computational cost.", "questions": "1. **Method Justification (EP vs. AIM w/o $W_K$)**: The ablation in 4.3 shows that removing $W_K$ from multi-head AIM causes a significant performance drop ($75.1→72.9$). Your proposed EP also lacks a $W_K$ yet achieves high performance (75.6%). Does this imply that EP's design (using full-dimensional $D_i$ queries $u_j$) successfully compensates for the lack of $W_K$, whereas simply dropping $W_K$ from a standard MHCA (like AIM) fails? A direct comparison of (AIM - $W_K$) vs. EP ($EP_{12}$) would clarify if EP's \"transformation-free\" design is truly superior or just more efficient.\n\n2. **Code Implementation**:\n    * **Paper (Table 1)**: The \"QUERY SOURCE\" for V-JEPA is listed as $q_j = W_{Q_j} u$, with $u \\in \\mathbb{R}^{D_i}$. This categorizes it with SimPool, where $u$ is *data-dependent* (derived from the input $X$, e.g., Global Average Pooling). This is explicitly contrasted with methods like AIM or EP, which are listed as having a learnable query $q$.\n    * **Code (`poolings/jepa/attentive_pooler.py`)**: The `AttentivePooler` class (used for \"jepa\" in `main_linprobe.py`) implements its query as a *data-independent, learnable parameter*: `self.query_tokens = nn.Parameter(torch.zeros(1, num_queries, embed_dim))`. The paper's experimental performance for V-JEPA is based on the *code's* learnable-query architecture, but the paper's *analysis* (Table 1) attributes this performance to a *data-dependent* query architecture. Will this invalidates the paper's own framework for comparing V-JEPA against other methods like AIM or EP?\n\n    * **Paper (Table 1 & Sec 3.3)**: The \"ATTENTION\" mechanism for AbMILP is explicitly defined as dot-product attention: $a = \\sigma_m(K^{\\top} q)$, where $K=X$ and $q$ is a single learnable query vector. Section 3.3 further claims this method \"requires only $D_i$ parameters\" (the size of $q$).\n    * **Code (`poolings/abmilp.py`)**: The `ABMILPHead` class seemingly does *not* implement dot-product attention. Instead, it uses a multi-layer MLP (`self.attention_predictor` is `nn.Sequential`) to *predict* attention weights directly from the input features: $a = \\text{softmax}(MLP(X))$. This is an MLP-based attention mechanism, not a dot-product one. The number of parameters in a two-layer MLP (such as `abmilp_depth=2`) is on the order of $D_i \\times D_{hidden} + D_{hidden} \\times 1$, which is far more than $D_i$ parameters (in `main_linprobe.py`, `abmilp_depth` defaults to 2). See details below:\n\n  * Original Code (Supplementary Material `poolings/abmilp.py`)\n```python\n# This is the MLP-based implementation from the supplementary material.\n# Note 'self.attention_predictor' is an nn.Sequential (MLP).\n# This implementation's parameters are O(D^2), not O(D).\nclass ABMILPHead(nn.Module):\n    def __init__(\n            self,\n            dim: int,\n            activation: str= \"tanh\",\n            depth: int = 2, # Default depth=2\n            **kwargs\n        ):\n        super().__init__()\n        self.ATTENTION_BRANCHES = 1\n        attn_pred_layers = []\n        for i in range(depth-1):\n            attn_pred_layers.extend([\n                nn.Linear(dim, dim),\n                (nn.Tanh() if activation == \"tanh\" else nn.ReLU()),\n            ])\n        # The MLP: e.g., Linear(D, D) -> Tanh -> Linear(D, 1)\n        attn_pred_layers.append(nn.Linear(dim, self.ATTENTION_BRANCHES))\n        self.attention_predictor = nn.Sequential(*attn_pred_layers)\n        # ... (other initializations) ...\n\n    def forward(self, x: torch.Tensor) -> torch.Tensor:\n        # ... (self-attn logic) ...\n        predictor_input = x # (Simplified for clarity)\n        \n        # a = softmax(MLP(X))\n        attn_map = self.attention_predictor(predictor_input) # [B, N, 1]\n        attn_map = F.softmax(attn_map, dim=1)\n\n        # y = V * a (where V=x)\n        out = (x * attn_map).sum(dim=1)\n        return out\n```\n   *  Based on Paper's Description, My Understanding: \n\n```python\n# This implementation strictly follows the paper's description\n# (Table 1 & Sec 3.3): a = softmax(X^T q).\n# It uses dot-product attention with a single learnable query 'q'.\n# This implementation has O(D) parameters.\nclass ABMILPHead(nn.Module):\n    def __init__(self, dim: int):\n        super().__init__()\n        \n        # The learnable query vector 'q' (D_i parameters)\n        self.query = nn.Parameter(torch.zeros(1, dim, 1))\n        nn.init.xavier_uniform_(self.query)\n\n    def forward(self, x: torch.Tensor) -> torch.Tensor:\n        # x (K and V) shape: [B, N, D]\n        \n        # 1. K^T q (using batch matmul)\n        # (B, N, D) @ (B, D, 1) -> (B, N, 1)\n        attn_logits = x @ self.query\n        \n        # 2. a = softmax(K^T q)\n        attn_map = F.softmax(attn_logits, dim=1) # [B, N, 1]\n\n        # 3. y = V a (where V=x)\n        # (B, N, D) * (B, N, 1) -> sum(dim=1) -> (B, D)\n        out = (x * attn_map).sum(dim=1)\n        return out\n```", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method, EP, is a \"simple multi-query cross-attention mechanism\" that is derived by \"eliminat[ing] redundant projections\"  from standard MHCA. This simplification is well-illustrated in Figure 1.\n\n2. This work provides the \"first comprehensive study\"  of attentive probing methods. The evaluation is extensive, covering \"diverse pre-training paradigms\" (MIM, JEA, VLMs, generative) and multiple datasets.\n\n3. EP consistently achieves a state-of-the-art \"accuracy-parameter trade-off\" , as shown in Figure 2. It achieves \"state-of-the-art top-1 accuracy of 75.6% with less than 1.4M parameters\" on MAE ViT-B, outperforming more complex methods. The gains are largest for the target model classes, such as \"SimMIM +13.6%\" and \"DiT +24.3%\" over LP.", "weaknesses": "1. **Placement of Key Ablations**: The justification for EP's specific design is partially buried in Section 4.3. For instance, EP's design is \"transformation-free\", notably lacking the $W_K$ projection. The empirical justification for this comes from an ablation showing that while removing $W_K$ from multi-head AIM hurts performance ($75.1\\% \\rightarrow 72.9\\%$), EP's design performs well without it. This key comparison, which validates EP's specific design choice, should be more central to the method's motivation in Section 3.\n\n2. **Confusing Terminology**: The method is named \"Transformation-free Multi-Query Cross-Attention\" , and the text states \"there are no projection matrices\". However, the value projection $W_V$ is still a critical component of the pooling operation (Equation 2) and the EP method. An ablation in 4.3 confirms that \"removing $W_V$ from $EP_{12}$ degrades performance from $75.1\\% \\rightarrow 72.1\\%$\". The \"transformation-free\" label applies only to the key/query interaction, not the value projection, which could be made clearer.\n\n3. **Matryoshka Probing**: The Matryoshka analysis (Appendix C.3, Table 5) is very compelling. It shows that standard EP, when evaluated at smaller dimensions, \"collapses\" (e.g., 5.5% at $D/8$ ). In contrast, \"Efficient Matryoshka\" resolves this and provides robust \"multi-scale probing without extra parameters\". Given this, should \"Efficient Matryoshka EP\" be the recommended protocol for the community, rather than the standard EP described in the main paper?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B9GZPQNoi8", "reviewer": "Reviewer_Ph7P", "summary": "This paper takes a hard look at how we evaluate giant, pre-trained vision models. The core idea is that full fine-tuning is just too expensive , but the standard alternative, linear probing (LP), doesn't do justice to models (like MIM) that spread information across patch tokens instead of a single [CLS] token. The paper champions \"attentive probing\"—using an attention head to smartly aggregate patch features—as the right middle ground.", "questions": "1. The main question is about the PEFT comparisons. Could you provide data on how EP stacks up against training a LoRA module (with a frozen backbone) using an equivalent number of trainable parameters?\n2. Regarding the complementarity analysis: do you have any experiments to suggest this is a causal driver of performance? For example, what happens if you explicitly add a loss term to force the queries in AIM or V-JEPA to be more complementary? Would their performance increase?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The biggest strength, in my view, is the systematic benchmark. The field of attentive probing has been a bit all over the place, with different papers (AIM, V-JEPA, etc.) all proposing their own one-off solutions .\n2. EP is a nice piece of engineering. It's not flashy, but it's simple, well-motivated (why have redundant projections?), and it just plain works. It consistently lands on the Pareto frontier for both parameter count and GFLOPs, which is exactly what you want from something called \"efficient.\"\n3. The experimental setup is comprehensive. The authors didn't just test on one model; they covered the gamut of pre-training paradigms (MIM, JEA, VLMs, generative) across a ton of datasets. The extra analyses, like the low-shot probing (Table 3) and layer-wise probing (Fig. 10), really strengthen the case that EP is a robust tool.", "weaknesses": "1. While EP is effective, its novelty is a bit thin. The core idea is essentially an ablation study on Multi-Head Cross-Attention (MHCA),\n2. The paper frames this entire problem as \"probing for evaluation.\" But what they're doing—freezing the backbone and training a few extra parameters—is exactly what Parameter-Efficient Fine-Tuning (PEFT) is. The authors even acknowledge EP fits into the PEFT family in Appendix A.2. So, why is there no comparison to mainstream PEFT methods like LoRA, Adapters, or even simpler baselines like BitFit (tuning biases) or just tuning the LayerNorms?\n3. The analysis showing that EP's queries are \"complementary\" is interesting. But the paper never really closes the loop. Is this complementarity the reason EP performs well, or just a neat side effect? Do other methods (like AIM, which Figure 6 shows is also pretty complementary) also benefit? It's presented as a key finding but feels disconnected from the main efficiency argument.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EbSF5IFjp2", "reviewer": "Reviewer_Vb6E", "summary": "This paper addresses the limitations of standard linear probing (LP) for evaluating pre-trained visual representations, particularly for models like those based on Masked Image Modeling (MIM) that distribute information across patch tokens rather than relying on a single global representation (like a [CLS] token). The authors present a systematic study of attentive probing (AP), a protocol that uses attention to selectively aggregate these patch features. They introduce a novel, parameter-efficient AP method called Efficient Probing (EP), which uses a simple multi-query cross-attention mechanism without redundant projections. EP demonstrates state-of-the-art accuracy compared to LP and prior AP methods across various benchmarks and pre-training paradigms while being significantly more efficient in terms of parameters and computational cost. Furthermore, the analysis reveals that EP's high performance correlates with the localization quality of its attention maps, which are shown to be highly diverse and complementary.", "questions": "1. Can EP be adapted for non-classification tasks, and what modifications would be necessary for such applications?\n\n2. How does EP scale when applied to very large backbone models or extremely large datasets, and are there cases where efficiency or accuracy gains diminish?\n\n3. Considering that modern large models often use a vision backbone as a component in multimodal architectures, does the probing result truly reflect the backbone’s capacity, and is probing genuinely necessary in this context?\n\n4. If probing is indeed necessary, would it be possible to supplement the analysis with comparisons to zero-shot and LoRA performance?", "limitations": "", "rating": 2, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper correctly identifies and addresses the misalignment between standard LP and modern pre-training paradigms (MIM, auto-regressive, diffusion) where discriminative information is distributed across patch tokens. Attentive probing is established as the necessary alternative.\n\n2.  This is presented as the \"first comprehensive study\" of attentive probing, offering a unified framework that categorizes existing methods (including those from unrelated tasks) and thoroughly benchmarks their performance against parameter and computational efficiency.\n\n3. The proposed Efficient Probing (EP) method is simple, yet highly effective. By reformulating the attention mechanism as a transformation-free multi-query cross-attention, it significantly improves the accuracy vs. parameter efficiency trade-off, consistently outperforming baselines and competitors.\n\n4. The results show that EP provides consistent and substantial gains across diverse pre-training methods (MIM, JEA, Hybrid, VLM, Generative), with the largest gains seen in models whose pre-training optimizes patch tokens (e.g., DiT, SimMIM).", "weaknesses": "1. The paper positions its contribution against a backdrop where attentive probing is described as \"underexplored\" and existing methods \"suffer from excessive parameterization and poor computational efficiency.\" While EP solves these issues, the baseline comparison set may be inherently weak due to the newness of the protocol, potentially overstating the relative accuracy gain compared to a hypothetical future efficient method.\n\n2. The motivation for removing the key transformation ($W_{Kj}$) in EP is primarily efficiency. While the poor performance of a partially-shared-subspace design (Equation 8) is mentioned, the core reasoning for why learnable queries $u_j \\in \\mathbb{R}^{D_i}$ directly interacting with the full feature space $X$ (Equation 10) works better than projected queries $q_j \\in \\mathbb{R}^{d_a}$ interacting with projected keys $K_j$ (Equation 7) could be elaborated with a more formal argument about representation capacity or gradient flow beyond just empirical verification. The section on the impact of $W_K$ and $W_V$ in MHCA is incomplete in the provided text.\n\n3. While the probing overhead (GFLOPs) is much smaller than the backbone, the overall evaluation still requires training for 90 epochs. The abstract/introduction states that full fine-tuning (FT) is \"unsustainable and prohibitive at scale,\" but the computational cost of an attention-based protocol, even an efficient one, is still much higher than standard LP. This is mitigated but not entirely eliminated, especially when comparing against \"frozen and pre-stored features\" (mentioned in Figure 3 caption) which is a crucial future direction not implemented here.\n\n4. The analysis is incomplete because it omits a comparison with crucial modern, low-cost alternatives that are widely used for large-scale model assessment, such as Zero-Shot classification for Vision-Language Models (VLMs) and PEFT techniques like LoRA. These methods represent highly relevant alternatives to EP's parameter efficiency and performance trade-off in the modern evaluation landscape.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)