Title: Attention, Please! Revisiting Attentive Probing Through the Lens of Efficiency
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: Linear Probing, Attentive Probing, Vision Transformers, Vision-Language Models, Evaluation, Neural Networks

Abstract: As fine-tuning becomes increasingly impractical at scale, probing is emerging as the preferred evaluation protocol. Yet, the standard linear probing fails to adequately reflect the potential of models whose pre-training optimizes representations of patch tokens rather than an explicit global representation. This motivates the need for attentive probing, an alternative that uses attention to selectively aggregate patch-level features. Despite its growing adoption, attentive probing remains underexplored, with existing methods suffering from excessive parameterization and poor computational efficiency. In this work, we revisit attentive probing through the lens of the accuracy vs. parameter efficiency trade-off. We present the first comprehensive study of existing methods, analyzing their design choices and benchmarking their performance. Building on this, we propose efficient probing (EP), a simple yet effective multi-query cross-attention mechanism that eliminates redundant projections and reduces the number of trainable parameters. Despite its simplicity, EP outperforms linear probing and prior attentive probing approaches across seven benchmarks, generalizes well to diverse pre-training paradigms, and delivers strong low-shot and layer-wise gains. Beyond evaluation, our analysis uncovers emerging properties of EP, such as complementary attention maps, which open new directions for leveraging probing beyond protocol design.

TL;DR: Linear probing overlooks the potential of models with distributed patch-token representations. We propose efficient probing (EP), a lightweight attention-based method that is faster, more parameter-efficient, and outperforms prior probing approaches.