MaskInversion: Localized Embeddings via Optimization of Explainability Maps

Authors: Walid Bousselham, Sofian Chaybouti, Christian Rupprecht, Vittorio Ferrari, Hilde Kuehne
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: localized embedding, fondation models, test-time optimization

Abstract:
Vision-language foundation models such as CLIP have achieved tremendous results in global vision-language alignment, but still show some limitations in creating representations for specific image regions. 
To address this problem, we propose MaskInversion, a method that leverages the feature representations of pre-trained foundation models, such as CLIP, to generate a context-aware embedding for a query image region specified by a mask at test time.
MaskInversion starts with initializing an embedding token and compares its explainability map, derived from the pretrained model, to the query mask.
The embedding token is then subsequently refined to approximate the query region by minimizing the discrepancy between its explainability map and the query mask. During this process, only the embedding vector is updated, while the underlying foundation model is kept frozen
allowing to use MaskInversion with any pre-trained model. 
As deriving the explainability map involves computing its gradient, which can be expensive, we propose a gradient decomposition strategy that simplifies this computation.
The learned region representation can be used for a broad range of tasks, including open-vocabulary class retrieval, referring expression comprehension, as well as for localized captioning and image generation. We evaluate the proposed method on all those tasks on several datasets such as PascalVOC, MSCOCO, RefCOCO, and OpenImagesV7 and show its capabilities compared to other SOTA approaches.

Decision: reject
Reviewer scores: 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730508249423

thread_type:
official_review

<thread_object>
creator: reviewer_rbtp

<content>
summary:
The paper proposes a new method that uses explainability maps from pretrained models to generate localized embeddings. These embeddings can represent object properties while capturing the broader image context. The paper further demonstrates that these learned region representations are versatile and can be applied to various tasks, including retrieval, grounding, captioning, and image generation.

strengths:
The paper introduces a novel approach leveraging explainability methods to enable the model to focus on specific regions within an image. Unlike traditional techniques like clipping, blurring, or masking, this approach allows the model to retain access to global image information. The method is clearly outlined and validated through comprehensive downstream tasks, demonstrating its effectiveness.

weaknesses:
The paper primarily focuses on single-object scenarios, lacking analysis on multiple objects and their interactions. Including experiments and analysis on multi-object scenarios would strengthen the study and provide a more comprehensive evaluation of the method's effectiveness. For instance, datasets like MSCOCO, with complex captions involving multiple objects, could offer valuable insights; sharing examples from such datasets would further illustrate the model's performance in these scenarios.

questions:
1. What type of global image context does this method capture? Could the authors provide visualizations, like attention map, to illustrate how the global context influences localized embeddings across different scenarios? This would clarify the method’s effectiveness in capturing and utilizing global context for downstream tasks.
2. In referring expression retrieval tasks, MaskInversion with ViT-B/16 underperforms compared to Masked Crop in RefCOCO+. Could the authors provide a detailed analysis investigating the reasons for this discrepancy?
3. Minor comment: In the related work section, "maks" should be corrected to "masks".
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716286236

thread_type:
official_review

<thread_object>
creator: reviewer_zd2g

<content>
summary:
The paper introduces MaskInversion, a method designed to generate localized embeddings for specific image regions using pre-trained vision-language foundation models like CLIP. This approach leverages the feature representations of these models to create context-aware embeddings for a query image region specified by a mask at test time.

strengths:
1. This paper is overall well-written.
2. The paper provides a comprehensive set of experiments and results, including quantitative metrics and qualitative visualizations, which helps in understanding the method's effectiveness and behavior.
3. MaskInversion operates in a zero-shot setting, which means it can handle tasks without requiring additional training data for specific tasks, leveraging the knowledge embedded in pre-trained models.

weaknesses:
1. This paper may be a bit short on innovation, as it actually uses the explainability map obtained from LeGrad to improve the feature extraction of the pre-trained models. Besides, some of the methods section is devoted to reviewing LeGrad, reinforcing the perception that this article is not innovative enough.
2. The regulaization loss seems very important to avoid trivial solutions. However, I find no ablation study on the hyper-paramter $\alpha$, which modulates the influence of the regularization loss. 
3. The performance of MaskInversion is heavily dependent on the quality of the input masks. In practical applications, obtaining high-quality masks might be challenging, which could limit the method's real-world applicability.
4. The paper could benefit from a deeper analysis of scenarios where MaskInversion might fail or underperform, and how such cases could be addressed.

questions:
If the pre-trained model itself does not have strong local feature capture capability, then post-training can give limited improvement. I'm curious if this idea of mask-guided feature capture can be applied to the training phase to improve the fine-grained perception of pre-trained VL models.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730781584220

thread_type:
official_review

<thread_object>
creator: reviewer_gqg2

<content>
summary:
This paper introduces MaskInversion, a method that leverages pre-trained vision-language models (such as CLIP) to generate context-aware embeddings for specific image regions by optimizing explainability maps. It aims to improve localized image representation tasks, such as referring expression comprehension and captioning, while employing a gradient decomposition strategy to reduce computation.

The contributions of this paper include:
1) a new method that is able to learn localized embeddings for given queries;
2) an efficient gradient decomposition approach for multi-query masks;
3) improved performance on various downstream tasks.

strengths:
1. The motivation is interesting. The problem of poor localization capabilities does exist in CLIP.
2. The proposed method is intuitive.
3. The performance is good. MaskInversion achieves superior results on a wide range of vision-language benchmarks.

weaknesses:
1. Very important baselines are missing. I noticed that you have discussed the paper of MaskCLIP [1] but did not compare with it in the experiments. Actually, CLIP's localization issues can be addressed in a very simple way. You just need to reform the last layer's self-attention in the fasion of MaskCLIP (removing Q and K), SCLIP [2] (Q-to-Q and K-to-K attention), or CLIPSurgery [3] (V-to-V attention with dual paths). I believe by simply modifying CLIP with these methods (they are all training-free), the performance can be improved by a very large margin.

2. Given these baselines are missing, it's difficult to evaluate whether the new method is effective enough. As MaskInversion involves a much more complex process, I expect it to perform significantly better than those three baselines.

3. The other contribution of the paper, gradient decomposition, is not that significant. As shown in Table 5, It makes clear speed improvements only if we have >10 masks/image. What is the general case of the number of masks involved in your tasks?

4. Minor comments: there are some typos in the paper such as in Line 481, what does Table 4.5 refer to?

[1] Extract free dense labels from clip, in ECCV 2022

[2] SCLIP: rethinking self-attention for dense vision-language inference, in ECCV 2024

[3] A closer look at the explainability of contrastive language-image pre-training.

questions:
See Weaknesses.

---- updates after rebuttal ----
I appreciate the authors' response and additional experiments for the metioned baselines. While MaskInversion outperforms the training-free approaches in most cases, some of my concerns are addressed. Howerver, the authors did not discuss the new results in the revised paper, which may cause misleading for readers. Overall, I still think this is a boarderline paper and have changed the score to 5. I still have concerns about the scalability of the method, as on OpenImagesV7, which is relatively more complex and has more masks in the images, MaskInversion performs worse than CLIPSurgery.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper addresses the poor localization problem of contrastive image-text pretraining (CLIP) models, which is a critical issue when using CLIP models in practice. The paper is well-written and motivated, and experiments show improved performance on the target localization tasks.

However, as Reviewer gqG2 pointed out, the paper lacks comparison and discussion with training-free methods (MaskCLIP, SCLIP, and CLIPSurgery, as the reviewer suggested) that share the same (or similar) objective (i.e., resolving poor localization ability of CLIP). In addition, the paper lacks a discussion about additional costs compared to these training-free approaches. While the authors showed additional comparisons with those baselines in their rebuttal, these results were not reflected in the revised version of the paper, which remains the same concerns from the initial review.

Initial reviews also highlighted concerns about complex design, lack of ablation studies, and required optimization steps compared to other baselines. Additionally, the AC agrees with Reviewer zd2G that the methodology heavily relies on LeGrad's explainability method and lacks technical novelty. 

The AC considers this a borderline paper with both strengths and weaknesses. Given the reviews and rebuttal, and the highly competitive nature of ICLR submissions, the AC recommends rejection.