Masked Cross-attention Adapters Enable the Characterization of Dense Features

Authors: Timo Lüddecke, Alexander S Ecker
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: image features, image backbones, ViT, instance segmentation

Abstract:
Learning meaningful representations is a core topic of deep learning. Throughout the last decade, many strategies for learning image representations have been proposed involving supervision and self-supervision and various data sources. 
In most current work, evaluation is focused on classification tasks while neglecting dense prediction tasks, possibly because linear probing is more challenging in the latter case.
Furthermore, dense prediction heads are often large and come with specific inductive biases that distort performance measurement further.
In this work we propose masked cross-attention adapters (MAXA), a minimal adapter method that is capable of dense prediction independent of the size and resolution of the encoder output. This allows us to make dense predictions using a small number of additional parameters ($<0.3 $%) while allowing for fast training using frozen backbones.
Using this adapter, we run a comprehensive evaluation assessing instance awareness, local semantics and spatial representation of a diverse set of backbones. 
We find that DINOv2 outperforms all other backbones tested - including those supervised with masks and language - across all three task categories.  
Code is available at https://to.be.released.

Decision: reject
Reviewer scores: 3, 3, 3, 5

Reviews and discussion:
<thread>
creation date:
1730169563008

thread_type:
official_review

<thread_object>
creator: reviewer_6ajn

<content>
summary:
This paper presents the Masked Cross-Attention Adapter (MAXA), a lightweight solution for making dense predictions on frozen vision backbones. By utilizing cross-attention, MAXA decouples the encoder output's resolution from the final prediction, filling a gap in the evaluation of feature extractors for dense tasks such as segmentation and depth estimation. The authors assess multiple vision backbones along three dimensions: instance awareness, local semantics, and spatial understanding, with DINOv2 emerging as the top performer across all tasks. The study emphasizes MAXA’s effectiveness in characterizing dense features while requiring minimal parameters and training effort.

Overall, the concept is straightforward and well-presented. The authors explore various aspects of the capabilities of frozen features, providing a solid basis for evaluating the adapter's effectiveness. However, the paper falls short due to a limited literature review and insufficient experimental depth, which may impact its chances of acceptance.

strengths:
- (1) The idea is simple and easy to read.
    
- (2) This research has potential to become a standard for evaluating the dense awareness of frozen features.
    
- (3) The authors explore various aspects of the capabilities of frozen features, providing a solid basis for evaluating the adapter's effectiveness.

weaknesses:
Major Weaknesses

- (1) The literature cited is outdated. For instance, the authors state, “At the other end of the spectrum, using complex dense task heads, for example, Faster R-CNN (Ren et al., 2015) for object detection, adds a large number of parameters and introduces its own inductive biases.” However, Faster R-CNN is nearly a decade old. The authors should clearly differentiate their approach from more recent works like ViTDet [1], ViT-Adapter [2], and Segmenter [3] in both the introduction and related work sections, as these studies also focus on developing lightweight dense task heads. Although ViTDet is briefly mentioned in the “Experiment Design” section, this reference is insufficient for establishing the distinction.
    
- (2) Lack comparison with zero-shot dense prediction using frozen features. Zero-shot segmentation using frozen features [4, 5]  from foundation models has been extensively studied. These models [4, 5] demonstrate strong segmentation performance with training-free dense heads.
    
- (3) The experimental comparison is insufficient. In the CLIP setting, the authors focus solely on ViT-based architectures (SigLIP for example), whereas ConvNet-based or hybrid architectures might be more appropriate for dense tasks. It is highly recommended that the authors include experiments with Hybrid CNN-Transformer architecture like ViTamin [6] and ConvNet architecture like CLIP-ConvNeXt [7]. These additional experiments are crucial, and I would consider raising the score if they are incorporated during the rebuttal phase.  
    


Minor Weaknesses

- (1) In Figure 1, it is unclear how the M(q, \sigma) is generated.
    
- (2) Section 3 mentions that “spatial queries Q of size (HQWQ, 16)”. It is unclear why the query dimension is chosen to be 16.
    
- (3) Concerns regarding reproducibility arise due to the unclear descriptions. Section 3 mentions that “This is realized through a small CNN operating on the output of all queries using transposed convolutions to increase spatial size.”. Please specify the details of "small CNN". CNN usually refer to Convolutional Neural Network consisting with convolutional layer and non-linear activation layer. Please specify the type and number of layers, and the kernal size of each convolution operator.


[1] Li Y, Mao H, Girshick R, et al. Exploring plain vision transformer backbones for object detection[C]//European conference on computer vision. Cham: Springer Nature Switzerland, 2022: 280-296.
    
[2] Strudel R, Garcia R, Laptev I, et al. Segmenter: Transformer for semantic segmentation[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2021: 7262-7272.
    
[3] Chen Z, Duan Y, Wang W, et al. Vision transformer adapter for dense predictions[J]. ICLR, 2023.
    
[4] Sun S, Li R, Torr P, et al. Clip as rnn: Segment countless visual concepts without training endeavor[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 13171-13182.
    
[5] Wang F, Mei J, Yuille A. Sclip: Rethinking self-attention for dense vision-language inference[C]//European Conference on Computer Vision. Springer, Cham, 2025: 315-332.
    
[6] Chen J, Yu Q, Shen X, et al. ViTamin: Designing Scalable Vision Models in the Vision-Language Era. CVPR. 2024. https://huggingface.co/jienengchen/ViTamin-XL-384px
    
[7] https://huggingface.co/laion/CLIP-convnext_large_d_320.laion2B-s29B-b131K-ft-soup

questions:
As raised the weakness section, it is highly recommended to include experiments with Hybrid CNN-Transformer architecture like ViTamin [6] and ConvNet architecture like CLIP-ConvNeXt [7]. These experiments are essential, and I would consider increasing the score if they are added during the rebuttal phase.

The authors should also address the major concerns and minor concerns detailed in the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730551711704

thread_type:
official_review

<thread_object>
creator: reviewer_62gx

<content>
summary:
The article introduces a new method called Masked Cross-Attention Adapters (MAXA) for evaluating and characterizing the performance of different visual feature extractors (backbone networks) in dense prediction tasks. MAXA employs a cross-attention mechanism that enables effective feature extraction and dense prediction without relying on the size and resolution of the backbone network's output. This method introduces a learnable masking radius, allowing the model to adapt to the locality of various features, thereby achieving fast training while maintaining a low number of parameters.

strengths:
- MAXA can adapt to downstream tasks with only a small number of additional parameters (less than 0.3%).
- Since the backbone network is frozen during training, MAXA achieves faster training speeds.

weaknesses:
- Although MAXA has been evaluated across three main task categories, these categories may not fully cover all possible visual tasks.
- There is a lack of comparisons with other fine-tuning methods, such as Adapters Strike Back.
- The novelty is relatively weak.

questions:
Is there more discussion on the reasons behind DINOv2's superior performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730699798531

thread_type:
official_review

<thread_object>
creator: reviewer_phnn

<content>
summary:
This paper explores vision adapter for pretrained ViTs on dense visual tasks like Segmentation and Depth Estimation.  The paper detailedly benchmark the performance with the tasks and also ablates some adapter designs.

strengths:
- Comprehensive Benchmark for Dense Vision Tasks: The paper makes a valuable contribution by introducing a comprehensive benchmark specifically designed for evaluating the dense prediction capabilities of pre-trained vision encoders, addressing a notable gap in the current landscape of benchmarks that primarily focus on classification tasks.   
- Methodological Rigor: The authors employ a rigorous methodology, including the use of masked cross-attention adapters (MAXA) to enable fair comparisons across different encoders. The choice of MAXA is well-justified, as it allows for fast training and evaluation.   
- Clarity and Insightful Presentation: The paper presents a clear and well-organized set of experiments, covering a diverse range of pre-trained models and dense tasks. The results are presented in a readily understandable and comparable manner, providing valuable insights into the relative strengths and weaknesses of different encoders for dense prediction tasks.

weaknesses:
- Limited Insight into Learned Representations: While the benchmark effectively compares the performance of different encoders, it lacks deep analysis regarding the specific representations learned by each encoder. Simply stating that "DINOv2" achieves the highest numbers isn't sufficient; the paper would benefit from a more in-depth investigation into the characteristics of the learned representations that contribute to performance differences.   
- Overlooking Architectural Biases: The paper does not explicitly address how architectural biases in different encoders might contribute to their performance on dense tasks. A discussion on this aspect would be valuable, as it could help disentangle the effects of pre-training from those inherent to the encoder architectures.   
- Potential Bias from MAXA: Although the authors justify the use of MAXA, the paper could be strengthened by exploring whether the findings hold consistent with other adapter methods or lightweight dense heads. This would provide further validation of the results and address potential concerns about biases introduced by the specific choice of MAXA.   
- Missing Comparisons with Key Adapter Methods: The paper lacks a direct comparison with other relevant adapter methods, such as ViT-Adapter and FeatUp. Including these methods in the evaluation would offer a more complete picture of the adapter landscape for dense prediction tasks.

questions:
Please see weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730707184723

thread_type:
official_review

<thread_object>
creator: reviewer_sk5i

<content>
summary:
This work introduces the Masked Cross-Attention Adapter (MAXA), a method designed to provide a cost-effective approach for probing transformer backbones for their dense prediction capabilities. The method employs a masked cross-attention readout layer that uses positional encodings as fixed query vectors. This is followed by a second unmasked cross-attention layer and a deconvolution network. Additionally, each attention head incorporates a learnable locality bias term during cross-attention. The authors evaluate MAXA's performance using multiple pretrained backbones (such as MAE, DINO, and DINOv2) across tasks including instance awareness, semantic segmentation, and monocular depth estimation.

strengths:
The ability to perform a meaningful cost-effective evaluation of backbones for their dense prediction capabilities is currently lacking, and as a result, only a limited number of studies offer meaningful evaluations. The proposed method is well-motivated and clearly described, making it easy to follow. The authors provide a solid introduction to the problem and include an informative related work section. Additionally, the ablation studies demonstrate that the advantages of certain design choices are consistent across multiple tasks and different pretrained backbones.

weaknesses:
The relevance of a cost-effective dense prediction evaluation largely depends on its high correlation with currently optimal but more resource-intensive evaluation techniques. However, the experiments provided are insufficient to establish confidence in this correlation. While the authors evaluate MAXA across multiple tasks and backbones, they do not present a statistical analysis of its correlation with fine-tuned results, nor do they quantify the trade-off between training cost and performance compared to state-of-the-art techniques. Such context is necessary to make the presented evaluations meaningful.

The choice to follow simple FPNs and rely solely on information from the last layer appears questionable. Although Vision Transformers (ViTs) maintain the same resolution across all layers, it is doubtful that the final layer alone contains all the necessary information without fine-tuning the backbone. For instance, MAE demonstrated that linear probing performance of ViTs is not always a reliable indicator of fine-tuning performance. Furthermore, [1] showed that employing cross-attention readouts from every layer leads to significant performance improvements compared to using simple FPNs.

minor: Lines 260-270 contain some incomplete sentences, such as "SAM2 also good.".

[1] Chen, Zhe, et al. "Vision Transformer Adapter for Dense Predictions." The Eleventh International Conference on Learning Representations.

questions:
The ablations show that a feature dimension of 8 performs better than a dimension of 16. Why only report a change in one direction? How did a feature dimension of 4 perform? Similarly, without the second cross-attention layer MAXA performs worse. Does a third cross-attention layer improve performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732891290356

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
## General Comments and Updates

We thank the reviewers for their comments. For our revised version we incorporated the following changes:

1. **Goal of our work.**  
The primary goal is not to outperform other adapter method in terms of performance. Instead, we propose a fast and minimal adapter to *directly* assess feature quality in dense tasks. 
Our method can be seen as a dense equivalent to attentive probing (as in CoCa, DinoV2 and Aim2). To the best of our knowledge no such method exists. We clarified the goal of our work in the introduction as this was not well communicated before.

2. **Extended comparison.**  
The main motivations behind our work is to address the lack of low-parameter, dense adapter methods for frozen backbones. Several related methods were proposed by the reviewers, below we clarify differences to these methods:
    * *ViT-Adapter* [1]: Requires more parameters (for ViT-B: 14M vs. ~0.3M in our case). Uses common task heads with specific inductive biases (HTC++ for detection, Mask2Former for segmentation).
    * *Adapters strike Back* [2]: Addresses whole-image classification. Compares different adapter types but all are based on ImageNet-trained ViT-B. In this paper we focus on dense tasks and compare different backbones.
    * *Segmenter* [3]: A transformer for semantic segmentation. Uses many more parameters as the backbone is not frozen.
    * *ViTDet* [4]: A transformer for object detection model. Uses more parameters as the backbone is not frozen.
    * *FeatUp* [5]: We agree that FeatUp is a meaningful baseline and added this to our comparison. While FeatUp allows for linear probing with a very small parameter budget, it has larger memory requirements and around 4 times longer runtimes than our method.
    
    We agree that our literature section did not explain these differences well and updated it accordingly.

3. **Correlation with fine-tuning techniques.**  
We concur that relating our findings on dense task to established results is important. We already compare to work in object-centric learning (relevant specifically for instance discrimination) and whole-image classification (Fig. 5).
In the revised version, we added a comparison with matching timm backbones and zero-shot dense prediction models (as requested by reviewer 6ajN). Both experiments are reported in the appendix.

### Minor changes
- For reproducibility we report on hyperparameters and dataset splits in the appendix. 
- Furthermore, we added a description of CNN-baseline used in Fig. 3 to the appendix
- We added a visualization of feature output to the appendix.


[1] Chen et al.: Vision Transformer Adapter for Dense Predictions. ICLR 2023.  
[2] Seitzer et al.: Bridging the gap to real-world object-centric learning, ICRL 2023.  
[3] Strudel et al.: Segmenter: Transformer for semantic segmentation, ICCV 2021.  
[4] Li et al.: Exploring plain vision transformer backbones for object detection, ECCV 2022.  
[5] Fu et al.: Featup: A model-agnostic framework for features at any resolution, ICLR 2024.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work introduces masked cross-attention adapters method for dense predictions that operates independently of encoder output size, enabling efficient training with frozen backbones.

The reviewers expressed concerns about the manuscript's statistical correlation between MAXA and state-of-the-art methods, the reliance on simple FPN, and the need for comprehensive ablation studies. They called for deeper analysis of learned representations, comparisons with key adapter methods, and consideration of architectural biases. Additionally, they noted the limited scope of evaluated tasks and suggested including comparisons with other fine-tuning methods and updating the literature review. In response, the authors defended their choices, agreed to conduct ablation studies, and committed to updating the literature and expanding comparisons, though some gaps remain regarding the evaluation's breadth.

Al the reviewers remained their original scores. Given the unanimous opinion of all reviewers, AC decide to reject the paper.