Segmentation From Attention: Training-Free Layer Selection and One-Shot Tuning for Segmentation in VLMs
Authors: Mir Rayat Imtiaz Hossain
Mennatullah Siam
Leonid Sigal
James J. Little
Venue: Accepted by TMLR ()

Abstract: Large-scale vision-language models (VLMs), trained on extensive datasets of image-text pairs, exhibit strong multimodal understanding capabilities by implicitly learning associations between textual descriptions and image regions. This emergent ability enables zero-shot object detection and segmentation, using techniques that rely on text-image attention maps, without necessarily training on abundant labeled segmentation datasets. However, performance of such methods depends heavily on prompt engineering and manually selected layers or head choices for the attention layers. In this work, we propose a training-free entropy-based measure, InfoScore, to identify the best image-text attention layers for segmentation, providing a more flexible and scalable solution for training-free open-vocabulary segmentation, reducing the additional burden of hyperparamter search. We empirically show that our training-free selection strategy is superior to naive selection strategies. Additionally, we demonstrate that instead of solely relying on text prompts, fine-tuning the image-text attention layer with a single visual example of each class significantly improves segmentation without the need of additional parameters or decoders. Moreover, we show that our methods and findings are general and can be applied across various vision-language models (VLMs).

Published in Transactions on Machine Learning Research (02/2026)
Segmentation From Attention: Training-Free Layer Selec- tion and One-Shot Tuning for Segmentation in VLMs
Mir Rayat Imtiaz Hossain Deportment of Computer Science University of British Columbia Vector Institute
royat137@cs..e ubc.ca
Mennatullah Siam
memaauu@@ullertt..cm
Department of Computer Science Universily of British Colambia
Leonid Sigal
Isigal www.llcc...b..... on
Department of Computer Science University of British Columbia
Vector Institute
CIFAR AI Chair
James J. Little
lätle
Department of Computer Science University of British Columbia
Reviewed on OpenReview: https: // openreview. net/ forum? id= a5 lAwubXro
Abstract
Large-scale vision-language models (VLMs), trained on extensive datasets of image-text pairs, exhibit strong multimodal understanding capabilities by implicitly learning associa- tions between textual descriptions and image regions. This emergent ability enables zero- shot object detection and segmentation, using techniques that rely on text-image attention maps, without necessarily training abundant labeled segmentation datasets. However, performance of such methods depends heavily on prompt engineering and manually selected layers or head choices for the attention layers. In this work, we propose a training-fre entropy-based measure, InfoScore, identify the best image-lext atlention layers for seg- mentation, providing more flexible and scalable solution for training- open-vocabulary segmentation, redueing the additional burden of hyperparamter search. We empirically show that our training-free free selection strategy is superior to naive selection strategies. Ad- ditionally, we demonstrate that instead of solely relying text prompts, fine-tuning the image-text attention layer with single visual example of each class significantly improves segmentation without the need of additional parameters or decoders. Moreover, we show that our methods and findings are general and can be applied across various lasiunnla models (VLMs).
1 Introduction
In recent years, deep lcarning rescarch has incrcasingly focused on foundation models (Bommasani et al.
2021 Li et al. 2024 trained on broad datasets to support generalization across wide variety of down-
Source code: http:///tthubccoo////aaat111//// Prw.iiaatiiiooo--tttteeeeee Prom-Attentioon

---

Published in Transactions on Machine Learning Research (02/2020)
Output
Image-fext Attentions
Random Selection
All Layers
InfoScore (Top-2)
60
Optional I-
Support Image!
shot tuning Top-K
Vision Langunge
Model Laperi
50
layers
Select Top-K
using InfoScure
Layend
30
Gmnge of Hond Emage of Penson]
10
Immage of Car Text Prompts
Training-Free
1-shot
Test Image
(a)
(b)
Figure 1: Overview of Proposed Approach. (a) Overview: Given vision-language model, class- specific lext prompls, and query image, we rank layers using the proposed InfoScore measure, select the top-K attention layers, and aggregate their text-to-image attention maps to generate the final prediction. Optionally, the top-K layers can be fine-tuned with a single support image, enabling both training-free zero- shot and 1-shot settings. (b) Effectiveness of InfoScore: selecting only the top-2 layers yields significantly superior segmentation compared random or all-layer aggregation in training- setting. A single support image further boosts the performance. In this 1-shot setting, InfoScore- based layer selection also outperforms random or all-layer aggregation, although the performance gap is reduced.
stream tasks, typically through self-supervised learning He et al. 2022 Oquab et al. 2024 Caron et al.
2021 or vision-language modeling (Radford et al. 2021 Xu et al. 2023 Li et al. 2022b 2023a Alayrac
ct al. 2022). Vision-language pre-training, in particular, has seen significant advancements with models like CLIP Radford al., 2021) and BLIP (Li et al., 2022b), which leverage image-text contrastive learning imago-text matching. Another family of vision-language models, such as Flamingo (Alayrac et al. 2022) and LLaVA (Liu et al.] 2024b), builds upon the capabilities of large language models (LLMs) to support vision-text reasoning tasks, such as visual question answering (VQA) and image captioning. These models are commonly referred to as multi-modal LLMs (MLLMs). Additionally, task-specific foundation models like Segment Anything (SAM) (Kirillov et al., 2023 Ravi et al. 2024) and LLaVA descondants (Chen et al. 2023)) Zhang et al. 2024b) focus on pixel-level or region-level understanding, but they lypically require complex decoders and extensive datasets with pixel-level annotations to train.
A growing body of rescarch focuses training-free methods for pixel-level tasks, particularly image seg- mentation. These approaches leverage vision-language models to enable segmentation in an open-vocabulary
setting |Wang et al. 2024a Hajimiri et al.) 2025 Luo et al. 2024 Zhou et al. 2022 Cha et al.
2023
Barsel-
lotti et al. 2024 W Wysoczańska et al. 2024b Lan et al.
2024
Li et al. 2023b Karazija et al.] 2024
However,
most existing methods rely on a single type of VLM often CLIP-based models (Radford et al. 2021 that
form segmentations using similarity between CLIP visual patch features and text features (ysoczanska
et al. 2024b Wang al. 2024a Hajimiri et al. 2025 Zhou et al.] 2022 Cha et al.
2023 Lan et al. 2024]
Others rely text-conditioned diffusion models (Barsellotti et al. 2024 Li et al.
2023b: Karazija et al.
2024), typically extracting text-image cross-attention maps from diffusion backbones (Rombach et al. 2022
PNP-OVSS (Luo et al. 2024) extends this idca to VLMs trained with image-text matching (ITM) losses
by extracting cross-atlention maps, but it does not apply to modern LLM-based VLMs (e.g.. LLaVA [Lu et al. 2024bb) that lack ITM heads. Moreover, its layer selection relies on a CLIP-based reward computed using imago-level ground-truth class names, while inference additionally requires an external multimodal LLM (c.g., GPT-4o) identify the of classes present in each image. Overall, such approaches remain closely tied to specific architectures and external supervision, limiting their generality.
2

---

Published in Transactions on Machine Learning Research (02/2026)
Hence, propose model-agnostic framework for open-vocabulary segmentation using vision-language mod- els that operates in both training- free and one-shot fine-tuning modes, as illustrated in Figure [ Our met hod is designed to generalize across diverse VLM families, including both cross- muss-attentionn-tee based architectures and modern LLM-based vision-language models. We introduce three core components:
1. An entropy-based, training-fres free measure, InfoScore, which ranks and selects the most relevant lext-to-image attention layers in unsupervised manner.
false-positive filtering mechanism that re-weights attention maps using image-text scores derived from the model's own language modeling or image-text matching logits without relying on external models or annotations.
3. A one-shot tuning strategy that selectively fine-tunes the attention layers identified by InfoScore, without introducing additional decoders requiring large-scale dense annotations.
In the training-free mode, given class-specific text prompts and a query image, we aggregate the text-to- image attention maps from the top-K layers identified by InfoScore make final predictions (Figure left).
This layer selection yields more effective segmentation than other alternatives (Figure 1 right).
In the one-shot mode, we fine-tune the InfoScore-selected layers together with the corresponding word embeddings using single support image per class. This helps disambiguate text prompts and better align visual concepts with language. As shown in Figure I fine-tuning with even a single visual example yields substantial improving mloU by nearly 10 points on PASCAL VOC (Everingham et al. 2010).
We evaluate our approach across four benchmark datasots and demonstratc its extensibilitt across both cross-at-ention--aaeed vision-language models(e.g- ALBEF (Li et al. 2021), BLIP (Li et al. 2022b) and modern LLM-based rision-languag models (e.g., LLaVA 1.5 (Liu et al. 2024a Our training free method outperforms prior works, and we further show consistent improvements with onc-shot fino-tuning.
Contributions. In summary, contributions include:
model-agnostic framework for open-vocabulary segmentation that generalizes across both cross- attention-based and LLM-based vision-language models.
InfoScore, an entropy-based measure for training-frre free, unsupervieed selection of informative atten- lion layers.
false-positive class filtering mechanism that leverages the model's own image-text scores without requiring external models.
A one-shot tuning strategy that selectively fine-tunes InfoScore-identtffed attention layers without introducing additional decoders requiring large-scale annotations.
State-of-the-art performance on three out of four benchmark datasets in the training- free setting, and significant additional gains from one-shot fine-tuning, surpassing all existing approaches.
2
Related Work
2.1
Semantic Segmentation
Semantic segmentation Arbelácz et al. 2012 2014
Carrcira & Sminchiscscu, 2011
Uijlings et al. 2013;
Chen et al. 2014 2017 7a Long el al.] 2015 Cheng el al. 2021 2022 Hossain el al.
2024 has evolved from
early methods based on graph cuts over seeded regions Boykov & Funka-Less 2006
Boykov & Kolmogorov
2004 Shi Malik 2000) and proposal-basee mask classification (Arbelécz et al.
2014 Carreira Sminchisescu 2011 Uijlings al. 2013 Carreira et al. 2012 Dai al. (015), deep learning approaches
2012
that frame segmentation as per-pixel classification task (Chen et al. 2014
201 7a
Long et al. 2015
3

---

Published in Transactions on Machine Learning Research (02/2026)
More recently, models like MaskFormer (Cheng et al. 2021 and Mask2Former (Cheng al. 2022 have reintroduced the idea of mask classification plassficationn-prdiiciiin label establishiii a new paradigm. This formulation underlies many modern methods that incorporate textual and visual prompts. OneFormer (Jain et al. 2023). for example, uses text-conditioned prompts to unify semantie, instance, and panoptic segmentation. SAM (Kirillov el al., 2023). SAM2 (Ravi el al.] 2024) and SEEM (Zou et al., 2023) similarly adopt a prompt-driven mask prediction framework, enabling flexible interaction via text, clicks, or bounding boxes.
set of binary masks, each paired with class
2.2 Open Vocabulary Segmentation
Zero-shot. semantic segmentation has been extensively studied to segment unseen, novel classes using text descriptions (Bucher et al. 2019 Zhou et al. 2023). However, these methods do not allow overlap between base and novel classes. Ooca-loca segmentation addresses this by allowing such overlap and leveraging large-scale vision-language pre-training (Li et al. 2022a; Wu et al. 2024). Training with language data allows the models to handle large, extensíble vocabularios, unlike traditional zcro-shot scttings with prodefined base
calegories. Recent advances in vision-language pre-training (Radford et al., 2021 Li et al. 2022b) have further advanced open- opeabvocabulary image segmentation.
2.3 Vision-language Pretraining
Vision-language models (VLMs) can broadly be categorized into two types: (i) models that use cross- attention between image and text modalities for multi-modal reasoning, such as BLIP (Li et al. 2022b) and ALBEF (Li et al. 2021); and (ii) models that integrate Large Language Models (LLMs) by projecting image embeddings into the language space, where they are treated as pscudo-text tokens and processed through self-attention alone (Liu el al. a).
BLIP (Li et al., 2022b) jointly trains image and text encoders using a contrastive image-text loss to align the modalitics. These aligned representations are then passod to a multi-modal encoder trained with an image-text matching (ITM) loss, further reinforeing cross-modal aligument. ALBEF (Li et al., 2021 p adopts similar two-stage strategy: contrastive pretraining phase followed by multi-modal fusion using both ITM and masked language modeling (MLM) objectives. In both cases, text-to-image interactions are explicitly handled via cross-attention layers in the transformer architecture.
More recent multi-modal LLMs such as LLaVA (Liu et al. 2024b ap take a different approach. These models project image embeddings into the language embedding space, where they are concatenated with textual input and processed entirely through causal self atffaattention This design eliminates explicit cross-attention layers; instead, the self-attention mechanism implicitly models cross-modal interactions. These models are typically pre-trained and then instruction-tunet using only the next-token prediction objective, similar to large language models.
2.4 Prompting VLMs for Training-Free Segmentation
Unlike segmentatioon-spediic models such as SAM (Krillov et al. 2023 or SEEM Zou et al. 2024 vision- language models (VLMs) are not explicitly trained for segmentation ion. Nevertheless, prior work prior work has shown that they can be prompted to produce segmentation maps without additional training (Wang
ct al. 2024a Hajimiri et al.
2025 Luo al., 2024 Zhou et al., 2022 Cha et al., 2023 Barsellotti et al.,
2024 Wysoczańska et al.]
2024b
Lan al.]
2024
Li et al. 2023b
Karazija
al. 2024). 1) Most of these
approaches, however, are limited to a single type or family of VLMs, such as CLIP Wysoczańska et al.
2024b Wang et al. 2024a Hajimiri et al.
2025 Zhou et al. 2022 Cha et al. 2023 Lan et al. [2024)
models with explicit imago-text cross-attention layers, including ciftusion-basee models (Luo et al. 2024
Barsellotti et al. 2024 Li el al. 2023bb Karazija et al. 2024 Rombach et al. 2022 In CLIP-based mods segmentation is typically derived from the similarity between visual features and text prompts. In contrast, cross-attention--aeedd methods use image-text attention maps as proxics for segmentation. However, (Luo) et al. 2024) observed that cross-attention maps tend to over-segment objects and generate numerous false positives, and therefore used GPT-4o to filter out categories not present. in the image. In contrast, we observe
4

---

Published in Transactions on Machine Learning Research (02/2026)
that can refine the heatmaps using image-text scoring that comes with these VLMs, without the need for any additional models beyond the VLM itself.
Another key insight from approaches that utilize image-text cross-attention maps, or Grad-CAM, for tasks such as segmentation (Luo et al. 2024h, visual grounding (He et al.) 2024 Li et al. 2021 or visual question answering (Tiong et al. 2022 Li et al. (021) that. the quality of text-to-image grounding highly sensitive to the choice of cross-attention layer or head from which attention maps are extracted. Typically, these methods assess layer-wise, task-specific performance on validation sets with ground-truth annotations
to select the layer that maximizes performance (Tiong et al. 2022 Li et al., 2021 He et al. 2024). or they use ground-truth class information from validation images (Luo et al. 2024). J. Both approaches, however, rely on some form of ground truth, which limits their practicality in truly training-free, open-vocabulary setting where annotations are unavailable, making it challenging to determine which layer's cross-attention map would yield strong segmentation performance. To address this limitation, we propose an entropy- measure called InfoScore, designed to identify the most suitable layer combinations in a vision-language model (VLM), without. annotations.
Furthermore, these methods do not incorporate few-shot or one-shot demonstrations, making them vulnerable to training distributions and vocabularies of the underlying VLMs. In contrast, our approach extends open- vocabulary segmentation by incorporating such demonstrations to guide VLMs in producing segmentation maps. Crucially, our method is applicable to a broad range of VLMs, including recent multimodal large language models (MLLMs) like LLaVA (Liu et al. 2024b).
2.5 Quantifying Heatmap and Attention-Map Properties
Regarding determining the desirable propertics of heatmaps or attention maps, a line of work has studied the intrinsic properties of saliency maps, aiming to quantify their informativeness or uncertainty. Several approaches (Wang et al. 2010 Kümmerer et al.] 2015 Pardyl et al. 2023) adopt information-thcorriie frameworks such as entropy, entropy-rate, or information gain to characterize saliency distributions (Wang
et al. 2010 Kümmerer et al. 2015) and attention maps (Pardyl et al. 2023).
Beyond information-thoorett formulations, prior work has examined other intrinsic propertics of saliency maps, including sparsily, dispersion, and calibration, and proposed quantitative metrics for their evalua- tion (Gomez et al. 2022 Gupta et al., 2022). A complementary line of research evaluates heatmap quality through perturbation based and causal criteria, testing whether highlighted regions genuinely influcnce model predictions. Representative approaches include meaningful and extremal perturbations (Fong & Vedaldi, 2017 Fong et al. 2019), randomized input. masking (Petsiuk et al. 2018): and senchmark-based faithfulness and sensitivity analyses (Hooker et al. 2019 Yeh et al. 2019).
Other works have investigated the reliability of saliency methods, showing that some explanations may be insensitive to model parameters or training data (Adebayo et al., 2018 Hedström et al., 2024). Tn transformer-based models, several studies have further questioned whether attention weights can be directly interpreted as explanations, motivating analyses beyond raw altention visualization (Jain & Wallace 2019 Serrano & Smith 2019 Chefer al. 2021).
While these approaches provide valuable tools for analyzing and comparing heatmaps and attention maps, they are not designed to identify informative layers for segmentation. Our work builds on this literature by leveraging information-theorett properties of attention maps for training-free layer selection in VLM-based segmentation.
2.6 Few-shot Segmentation
Few-shot segmentation has been extensively studied in prior works, focusing primarily on traditional ap- proaches that do not leverage recent advances in vision-language models (Wang el al.] 2019 Siam el al., 2019 Shaban et al. 2017 Min et al. 2021). These approaches typically address 1-way or N N-way segmentation of novel classes against a background but do not evaluate performance on the base classes seen during pre- training. Recent studies have proposed a generalized few-shot segmentation setting that enables evaluation
both base and novel classes
TTian et al. 2022 Hajimiri et al. 2023 Hossain et al. 2024 Liu et al. 2023
5

---

Published in Transactions on Machine Learning Research (02/2026)
Class Presence Probubility (Class VLM Score)
Segmenter
Support Prediction Maps
MiBeydde
P(Person)
P(Cac)
Pifferse)
0.56
0
Image-test Matching! I Language Modelling
Dicycle
Person
Top-K leyer map per
Prompt
Rank
Attention Lover I-1
Rank
Mean of top-K Layer Maps
CE Loss
InfeScore
Query Prediction Maps
Attertion Layer 2
Rank
Support GT Masks
Attentonn Layer
Rank
Image-test Attention
Layers
Segmentation
[Image Bicycle]
Image Encoder
[Image Person]
[Image Cat]
1-shot Support Images
[Image Hooess]
Query Image
Test Prompts
Figure Model Overview. Our segmentation framework leverages VLMs trained on imago-text pairs, supporting training-free inference and one-shot fine-tuning. For training-fres free inference, given class names and query image, extract text-to-image attention maps from top-K layers (c.g., Layer 2 and Layer L), selected via InfoScore (see Sec. 3.2 These maps we-ieighted with class VLM scores to filter irrelevant categories (see Sec. 3.1 and used for prediction. In one-shot fine-tuning, we adjust text embeddings and top-K attention layer parameters (sce Sec. 3.4 to further improve the performance.
A recent approach (Hossain et al. 2024) uses a multiscale visual prompting technique to improve segmnnn- tation. However, none of generalized approaches utilize advanced vision foundational models, particularly VLMs.
A recent work Xiao al. 2024 has explored few-shot adaptation of segment anything (SAM) model Kir- illov et al. 2023). However, SAM is already trained with dense mask annotations over millions of data points. In contrast, we utilize VLMs that were only trained with image-text pairs, and not trained for segmenta- tion. Another approach (Zhu et al.) 2024a) explored few-shot segmentation with LLMs and an additional mask decoder by instruction-tuning the LLM on pixel-level annotations to generate 16-point polygons, and training the mask dccoder on a large datasct pixcl-level annotations for base categorics.
Our approach instead directly mines segmentation maps from VLM text-to-image atlention maps, without requiring any external models (e.g., ChatGPT),, additional decoders parameters, and can be applied broadly across different VLMs themselves trained without any pixel-level annotations.
3 Method
Our proposed approach leverages text-to-image attention maps from vision- language models (VLMs). We focus on VLMs that are pre-trained on largo-scale image-text pairs without any sogmentation supervision, except for the optional use of a single visual exemplar per class (i.e., the one-shot setting). Figure provides overview of our method, which operates in two modes: training-free mode and one-shot fine-tuning

---

Published in Transactions on Machine Learning Research (02/2026)
mode. In the training-free setting, convert the provided class names into natural language prompts and pass them as inputs alongside query image. We then extract attention maps from the kop- text-to- image attention layers ranked by our proposed InfoScore measure and re-weight them using image-text similarity scores (see Sections 3.1 and 3.2 respectively). In the onc-shot setting, we further utilize one labeled visual exemplar per calegory, including segmentation mask, to learn an ensemble of allention maps from multiple layers and/or prompts (sce Section 3.4).
3.1 Training-free free Prediction
As shown in Figure 2 during inference we are given a query image 7 and a class vocabulary V = fci,..., cc). Each class name e E V converted into a natural language prompt of the form [Image of c]. The full set of prompts is input to the vision language model (VLM) together with the image I, which is tokenized into grid of P P visual patches.
We consider a cross-allention layer of the multimodal encoder. Let T be the number of text lokens per prompt. For each class prompt E V, the cross-attention operation produces attention maps over the image patches, one per attention head. Since each cross-attention layer contains multiple heads, we compute the element-wise maximum attention score across all heads for each prompt yielding a prompt-specific attention map:
A E TRXXXP
Because single word in the prompt may be decomposed into multiple sub-word tokens by the tokenizer, we further aggregate across the token dimension to obtain single spatial attention map per class. Specifically, compute the mean over the T prompt tokens:
T Σ T
1
A
(t)
We adopt this aggregation strategy to remain parameter-free and model-agnostic. Among simple parameter- free choices, mean and max pooling are the most common. We choose mean aggregation as it provides a stable estimate across all prompt tokens and avoids over-emphasizing individual tokens, which can occur with max pooling. Stacking these maps over all classes yields a layer-specific attention tensor:
A RXPxP
Normalizing across the C' classes using a softmax operation produces a per- per-patch class probability estimate M PxP
In cases where we use related words enhance the class description, similar to previous works (Hajmmiii et al. 2025 Wang al. 2024a), e.g., man and woman added as related words to the class person, we pass them separate prompts. To compute the heatmap for the corresponding class, we simply take the maximum across the heatmaps of all the related prompts. We call this the multi-prompt per class setting. The per class setting, we do not
details of the prompts are provided in Appendix A In the single-prone use any related words for a particular class and instead use only the class name as is.
Once per-class attention maps are obtained, we select the top-K cross-attention layers according to the InfoScore ranking described in Scction 3.2 Let Lx C L denote the selected layers. Attention maps extracted from transformer-basee based VLMs are often noisy due to atlention sinks, partieularly in models that leverage large language models (LLMs) such as LLa VA-1.5 (Liu et al. 2024a), observed in prior work (Kang) et al. 2025 Darcet et al.) 2024 Sun et al. 2024). This issue, together with aggregation across multiple prompls, can lead to spurious activations for classes that are not present in the image, leading to false positive predictions.
To mitigate this effect, we introduce class VLM score S(c I), which estimates the likelihood of class being present in image I. For VLMs trained with an mmage-text matching (TTM) objective, we observed that given prompt [Image of cl, image-tex text matching (TTM) scores from ITM head gives estimate of

---

Published in Transactions on Machine Learning Research (02/2026)
whether class is present in the image not. Hence, S(c I) is obtained directly from the ITM scores. For models without ITM head (e.g,, LLaVA-1.5); we instead prompt the model with [Is there class in the image? Answer in Yes or No.] and estimate S(e) I) using the probability assigned to the token Yes.
For each selected layer I E LK, the class-specific attention maps are weighted by the corresponding class VLM scores,
M S(c 1) A
(1)
upsampled to the image resolution, and averaged across layers to form an ensemble attention map for each class
1 Me Ue E.
(2)
K KECK
Normalizing across classes at each pixel yields per-pixel class probabilities
uc(r,y)
Pa(x,y) = (e
(3)
where æ; y denotes pixel location.
While these probabilities can be used directly for segmentation, this approach often underperforms near
object boundaries (Luo et al.
2024). Instead, we use them as unary potentials for Conditional Random defined as U,(x) = - log (x)) the Fully Connected CRF (Full- the conditional distribution given by
Field (CRF) (Tafferty et al.]
2001
CRF) Krähenbühl & Koltun
2011
P(y II) x exp ( Ely I)),
(4)
with energy
Ely 1)
Ug)
Σ xn
(x, + woki ((okolE (x), Lo) Lx)) yvl.
(5)
unary term
pisirwise term
Here, k2 a spatial Gaussian kernel and hs bilateral (position+ Gaussian kernel computed from the input image I, with w, and wy being learnable weights. Since exact. inference of the per-pixel probabilities Plye c I) scales quadratically with the number of pixels, it is computationally intractable. FulICRFs approximate these probabilities using mean-field inference with permutohedral lattice filtering, while Con- vCRFs (Tecchmann & Cipolla, 2018) achieve the same via convolutional Gaussian filtering, enabling efficient GPU-parallel inference. Let qe(2 (x) denote this approximation of P(yz = c T). The final segmentation is computed by:
((x) arg max qa.))
(6)
We outline the procedure of training-fie free inference for a single image in Algorithm 2
3.2 Ranking Layers Using InfoScore
As noted in previous works [Luo et al. 2024 He et al. 2024 Tiong et al. 2022), and as observed in Figure 5 segmentation performance can vary significantly depending on the layer from which the attention map is extracted. In practice, when performing training-fre free segmentation in-the-wild ground-truth dense annotations or information about object presonce in the images unavailable. To address this, we propose an entropy-based measure, InfoScore, which can automatically identify the optimal layer(s) for training-free segmentation without requiring ground-truth annotations.
8

---

Published in Transactions on Machine Learning Research (02/2026)
Algorithm Computation of InfoScore
Input: Unlabeled image set D with [D] = N; class vocabulary with [V] C; list of candidate layers C.
Output: Ranking of layers R in descending order.
Initialize for each layer 1 L::
S +
Sum of image entropies
S + 0 RC
Sum of image-level class distribution P((Y,) > Running mean image-level class distribution > Running 2nd moment for variance of image-level class distribution
"¹ + 0 E RC
M2 + RC
for each image I D do Query the VLM with (I. for each layer / E L do Extract image-text attention maps Aj from layer I. 6: Scale Ai with class- VLM scores Sj.
5:
Normalize across classes at each pixel to obtain per-pixel class probabilities Pt.,, (Y.) e R Spatially average Pi (Yc) to get image-level class distribution P\(Y.) E ERC
8:
Compute image-level entropy (I) HP)(())).
9:
10:
SH + S + H (I)
S + P/(A
+ ((Y))-p)
11:
"' + J//N
B Welford update for running mean and variance
8' p((Y)) -r
M + M2 + 2000 8'
12:
end for
13: end for
14: for each layer E L do
15:
H Hmmae ////1 N
b mean image-level entropy
16:
(Y.)) Sp/N
P Dataset-level marginal distribution
17:
H Dataset = HPP) (Y.))
18:
R' Dataset H
Image
b EutropyRatio of e RIV
19:
o v MM/(N 1)
20:
Cov' Ecev ot
21:
St Rt Cov
InfoScore
22: end for
23: Rank layers E L in descending order of S!.
24: return Ranked list of layers.
Given set of unlabeled images, D, and vocabulary of class names, the proposed InfoScore measure evaluates the predictive uncertainty of text-to-image attention maps from each layer individually. It is designed with the intuition motivated by the Inception Score (Salimans et al. 2016) that layers suitable for segmentation should make confident predictions within each image while maintaining diversily across the dataset.
Design desiderata. To guide the design of InfoScore in the absence of ground-trut taund-truth annotations, we define the following desiderata for layer-selection measure:
1. Confidence: Predictions should be confident within each image, favoring sharply peaked class distributions.
Diversity: Across the dataset, predicted classes should be diverse and not collapse to single outcome.

---

Published in Transactions on Machine Learning Research (02/2026)
Algorithm 2 Segmentation From Attention with SeloScoor--Gudder Layer Selection
Input: Unlabeled image Drnkk for layer scoring; class vocabulary V; candidate layer set test image I; number of selected layers K; CRF module CRF (c.g., Gaussian CRF).
Output: Predicted segmentation masks for images in Dang.
1: Step 1: Select informative layers via InfoScore
2: R - INFOSCORE(Prand) C, L)
Alg P e.g. K 2, layers [0,3
3: LK < TR K)
4: Step 2: Segmentation for a single image
5: Query the VLM with (I,C) to obtain class-VLM scores S and image text attention.
6: Initialize ensemble attention map:
for each layer E LK do
8: Extract cross-attention maps A for tokens in C.
Reduce over text tokens to obtain attention map per-class (by taking mean) A(()) E RCPPx
10: Reseale per-class attention maps by class-VIA VLM scores S: M A0 E RRCxPP
11:
Upsample image resolution: M Upsample(MMM) HI W).
12:
Accumulate into ensemble:
u < M
13: end for
14: Average over selected layers:
U + U/K
15: Normalize across classes at each pixel obtain a class probability map Pe (see Equation
16: Apply CRF refinement using the input image: Port <- CRF(P,
17: Obtain final segmentation by per-pixel argmax:
y(h, w) = arg max, Porfee, h,,,,,,) 18: return
Image sensitivity: Predictions should vary meaningfully across images, reflecting image-specific content rather than global statistics.
These desiderata directly motivate the three components InfoScore described next.
The three components of InfoScore are as follows:
The average per-image entropy of the label marginal distribution summed over pixels. captures how confident the model within each image.
Mean Image-Level Entropy:
Dataset-Level Entropy: The entropy of the class-wise wise marginal distribution across the entire dataset. It captures the diversity of class predictions across all images.
Coefficient of Variation (CoV): The ratio of the standard deviation to the mean of the predicted class probabilities across the dataset. It quantifies how much the class probabilities vary across different images.
Given the per-pixel probability predictions for an image I, we compute the image-level classwise marginal distribution using layer 1 cross-attention maps, P((Y.) as,
1 hw
p\().)
pl Pl (Y.):
(7)
1=
10

---

Published in Transactions on Machine Learning Research (02/2026)
where h, w are the height and width of the predicted heatmap (correspondent to h = P and w = P in Section 3.1 PI (Yo) is the probability that the pixel (x, v) belongs to class e which is computed based on the attentont maps from layer L. Then we compute the image-level entropy H (I) as,
H'(I= = - Po P((.))
(8)
cEV
where V is the vocabulary of the classes segment, To compute the mean image-level entropy, we take the average over the set of unlabeled images in dataset D,
H image (D)
Σ H())..
(9)
D
IED
In order to compute the dataset-level entropy over the dataset DD we first calculate the classwise marginal distribution over the entire dataset, P1 (Yo), as,
1
P((Y.)
(10)
D
LED
The datasct-level entropy Heataset is then given by:
Hadtsset (D) = Σ
()((Yo) Po P\()))
(11)
<EV
We define the ratio of dataset-level entropy to image-level entropy as the Entropy Ratio:
EntropyRatio()) = Halataset (D)
(12)
H Himage (D)
The EntropyRatio can be viewed as assessment of the classification ability of each text-to-image attention layer. At the image level, desirable layers should make confident predictions, resulting in sharply peaked class distributions for only a few classes within each image and hence low mean image-level entropy. However, minimizing image-level entropy alone can yield trivial solutions, such as always prodicting the same class across all images (e.g., a background-biasee classifier). At the dataset level, by contrast, we want predictions to cover diverse set of classes. Under the assumption of independent and identically distributed (i.i.d.) images where all classes are equally likely, high dataset-level entropy indicates that the layer produces a balanced distribution over classes across the dataset, i.c., it is not biased toward single outcome regardless of input. In essence, high Entropy Ratio identifies layers that balance confident, low-entropy predictions within each image, against diverse, high-entropy predictions across the dataset.
Nevertheless, dataset-level entropy, on its own, does not capture how predictions vary between images. Since aggregates marginal class probabilities across the dataset, can be high even the same set of classes is predicted for every image-prnn producigo globally balanced distribution without meaningful image specificity.
To address this limitation, incorporate the Coefficient of Variation (CoV) multiplicative factor to the CnttropyRatio. The CoV measures the relative fluctuation of class probabilities across images, normalized by their mean, thereby favoring layers that produce distinct predictions for different inputs. Therefore, max- imizing CoV promotes image-sensitive and diverse predictions, complementing the global balance captured by dataset-level entropy. Figure illustrates this with a toy example of five images and six classifiers, ranging from trivial (Classificr 1: always class 1; Classifier 2: uniform) to idcal (Classifier 6: GT-like). As shown, Entropy Ratio alone ranks Classifier 4 above Classifier 5. Although Classifier is confident and predicts two of the three classes, it completely misses class 3. This ranking ignores the meaningful image-to-image
11

---

Published in Transactions on Machine Learning Research (02/2026)
Images
Classifier 1
Classifier 2
Classifler
Classifler 4
Clatsifier
Classiffer 6
Classifier
Description
EntropyRatio
InfoScore
Rank
Value
Rank
Classifier
Dead I blased towards class 1.
0.883
6th
0.000
5th (tie)
Classifier
Dead classifier Fully uniform prediction
1.000
5th
(tie))
Classifier
Bad Classifier - Highly biased towards class with minor variation prediction.
1.006
0.294
Classifier
Bad Classifier II Sensitive input never predicts class
1.374
2
1.600
31d
Classifier
Good Classifier - Image sensitive and moderately accurate classifier.
1.203
2.299
n
Classifier
Ideal Classifier GT Like prediction.
2.546
1"
7.851
1
Figure 3: Toy demo of EntropyRatin vs. InfoScore. We illustrate how InfoScore differs from Entropy- Ratio using synthetic examples. Shown are five images and six classificrs exhibiting diffcrent behaviors, from trivial (Classifier 1: always class 1; Classifier mistakenly ranks Classifier 4 above Classifier 5, despite Classifier never predicting class 3. In contrast, InfoScore-by combining EntropyRatio with the Coefficient of Variation highlghts imagc-sensitive and ideal classifiers. Moreover, while EntropyRatio prefers Classifier 2 over Classifier 1 despite both being non-informative, InfoScore assigns both a value of 0, correctly identifying them as non-informative. Overall, InfoScore balances confidence, diversity, and variability reliable layer ranking.
uniform) to ideal (Classifier 6: GT-like). EntropyRatio
variation captured by Classifier 5. InfoScore corrects this by incorporating CoV, correctly ranking Classifier above Classifier 4, emphasizing its image sensitivity. Moreover, while EntropyRatio ranks Classifier 2 (uniform/random prediction) above Classifier 1 (biased class 1), despite both being non-informative and equally poor, InfoScore assigns them value of zero, indicating their equally bad performance.
The Coelficient of Variation (CoV) for layer is computed as:
CoV(!) Σ of
(13)
where of and nd, denote the standard deviation and mean, respectively, the image-level class- wise marginal distribution (Y.) aggregated across the dataset, for each class at layer l.
12

---

Published in Transactions on Machine Learning Research (02/2026)
nDCG for Layer Ranking with Uncertainty (# Std Dev)
1.000
0.975
0.950
0.925
nDCG
0.900
0.875
0.850
nDCG@2
0.825
nDCG@3
nDCG@6
0.800
20
40
60
80
100
Number Random Samples
Figure 4: Convergence of InfoScore. The InfoScore measure converges for the top-3 ranked layers with as few as 50 random samples and for the top-6 layers with approximately 100 samples. The shaded region indicates uncertainly across repeated trials. The metric nDCG@k (normalized discounted cumulative gain) measures the quality of the ranking for the top-k layers, where a perfect ranking compared to the ground- truth ordering yiclds score of 1.0.
Finally, the InfoScore for layer / is defined as:
InfoScore(() = Entropy(atio(() x CoV(()
(14)
By maximizing the proposed InfoScore measure, we prioritize layers that balance image-level confidence, dataset-level diversity, and image-specific variability leading to more reliable layer selection for training- free segmentation. The procedure for computing InfoScore and ranking layers is detailed in Algorithm !
Finally, we note key assumption underlying InfoScore: object classes are assumed to occupy roughly similar spatial extents within images. In scenarios with consistent size disparities across classes, InfoScore may erroneously favor layers that overlook smaller objects, resulting in suboptimal rankings.
3.3 Convergence of InfoScore with Respect to Number of Samples
We do not need to evaluate InfoScore on the full test/validation set of images. In practice, it sufficient. to use a small number of random samples that include images from diverse classes. In Figure 4 we show how the InfoScore converges with respect to the number of random samples. We use the normalized Discounted Cumulative Gain metric (nDCG) (Järvelin & Kekäläinen, (2022) quantify the ranking quality. The metric nDCG@k measures the quality of the predicted ranking up to the top-k positions, where score of 1.0 indicates a perfect match with the ground-truth ordering. The nDCG is formally defined as:
rel,
DCG@k
(15)
loga +
13

---

Published in Transactions on Machine Learning Research (02/2026)
where rel, denotes the relevance of the item at position i. DCG@k
The normalized version is given by:
nDCG = IDCG@k
(16)
where IDCG@k is the maximum possible DCG achievable with the ideal (ground-truth) ranking.
We compute InfoScore and rank the layers using n randomly selected samples from the PASCAT-211 Fv- eringham et al. 2010) validation set. For each selection of n random samples, we repeat the experiment live times to account for randomness and report the standard deviation. As observed, as few as 50 random image samples are sufficient for convergence the top-3 layer ranking (nDCG@3), and approximately 100 samples suffice for top-6 ranking.
3.4 One-shot fine-tuning
So far, we have discussed training-frec inference using standard VLMs. However, we argue that class names alone not sufficient to properly ground or segment the corresponding objects. They can be ambiguous or confusing without the appropriate context. For example, the COCO-Obj dataset includes a class name tie, which can be ambiguous unless additional context is provided to indicate it refers to a piece of clothing, Therefore, we hypothesize that fine-tuning certain parameters of VLM pretrained on an image-text retricval lask, using one visual example with dense annotations, can improve the overall segmentation quality.
In few-shot segmentation literature, whether performing 1-way or k-way classification, classes are typically divided into non-overlapping splits of base and novel categories. Models are first trained on base categorics with large number of annotated examples to learn how to perform segmentation, and are then subsequently adapted to novel classes in a few-shot setting. In contrast, our approach does not involve base classes, as the models we use were never explicitly trained for segmentation, meaning all classes are novel. To ensure fairness, only one novel category is present in each selected image. However, in practice, certain novel classes may co-occur. If a one-shot example image for one class contains other classos, we assign them to either the background (if the datasct includes background category) or to an ignorod class.
During one-shot tuning, we minimize per-pixel cross-entropy loss between the predicted segmentation map and the ground-truth annotations of the support image, updating only the word embeddings and the parameters of the top-K ranked attention layers.
It is worth noting that we do not introduce any additional parameters or decoders for fine-tuning, merely encouraging the attention maps to align with the support image ground truth. Moreover, optimizing only a subset of parameters leads to more data-cfficient training and limits overfitting, which is necessary given that we are only leveraging a single (or very few) labeled examples for training.
4 Experimental Results
4.1 Experimental Setup
Datasets. We evaluate our method on four commonly used segmentation datasets: PASCAT-21 (Eeringg- ham et al. 2010), COCO-Obj (Lin et al. 2014), COCO-Stuffs-177 (Caesar et al.) 2018) and ADE-20K (Zhou et al. 2017). PASCAL-21 and COCO-Obj include a background category, while COCO-Stuffs. 171 and ADF-20K do not, but are the most challenging due to the inclusion of stuff classes. Following previous works (Wang et al., [2024a] Hajimiri et al., 2025), the background class is represented as list of possible background categories. Details about the class names used for single and multi-prompt setting are provided in the supplementary. For one-shot evaluation, conduct five separate runs and average their mloU.
Implementation Details. Implementation details regarding the model weights used for BLIP [L et al. 2022b), LLaVA (Liu et al.) 2024a) and ALBEF (Li et al. 2021] are provided in the supplementary. We report results on images resized to maximum side length of 512 for BLIP and ALBEF and 448 for LLaVA. By default, we use top-2 layers for all datasets and models except for ADE-20K where top-1 performs slightly better. For ConvCRF, we use a kernel size of 15 15 and perform 20 iterations per image. We follow (Teichmann & Cipolla. 2018) for other hyper-parameters of ConvCRF.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Method
VLM
PASCAL-21
COCO-Obj
COCO-171
ADE-20K
Weekly Supervised Training
Group VTT Xu et
2022
CLIP
10.4
ReCo Cha
CLIP
30.4
Ground Everythnnng [ouuselham al. 2024
MetaCIII
46.8
17.1
SAM-CLIP Wang el 2024b
CLIP
31.5
17.1
Olip-DINOiser (Wysoezarska
2024a
CLIP
62.1
34.8
24.6
20.0
Training Free
PNP-OVSS Luo et al.
2024
BLIP
51.3
36.2
17.9
14.2
SCLIP Wang et al.
2024a
CLIP
61.7
33.2
23.9
17.8
ProxyCIIB CLIP
Lan et al. 2024
CLIP+DIN
61.3
37.5
26.5
20.2
NACLIP Hajimiri et al.
2025
CLIP
64.1
36.2
25.7
19.1
Ours
BLIP
60.2
42.8
28.1
22.8
One-shot Supervision (Average Across runs)
Ours
BLIP
70.1
0.89
45.3
1.32
29.3
0.05
24.5+ 0.13
Table 1: Comparison of our approach with state-of-the-art methods for weakly-supervised, training- free, and one-shot supervised open-vocabulary semantic segmentation (OVSS). The results reported on this table are using BLIP top-2 layers (selected by InfoScore) in multi-prompt per class setting. Best results for training-Tree setting and the 1-shot results are in bold.
For one-shot supervision we use batch size of two. We fine-tune all models (BLIP, ALBEF, LLaVA) on a single A40 GPU with 48GB memory. We use a learning rate of 2 10 for PASCAL-21 and COCO-Obj, and x 10 for COCO-171 and ADE-20K. We train COCO-Obj for two epochs, COCO-171 and ADE-20K for three epochs and PASCAL-221 for five epochs. Further details are provided supplementary materials.
Source code for our approach is publicly available at https:/ fttps://gittub..omm// http://aayyyaaleegggeeeeeaattooa Seginettation-Pron Attention
4.2
Quantitative Results
4.2.1
Comparison to the State of the Art
We compare against the state-of-the-art methods in open vocabulary segmentation. We mainly compare against training-free free methods (Luo et al. that further fine-tune pre-trained VLMs large-scale image-text pairs pseudo annotations through weak supervision (Xu et al. 2022 Cha et al.
2024 Wang et al.) 2024a Hajimiri al. 2025) and the ones
2023 The choice for open vocabulary segmentation setup is motivated for the sake of fair comparison, since previous training-free methods had access to aligned vision- language data similar to our approach in its training free mode. In Table I it is shown that our training-free free open-vocabulary method outperforms the state-of-the-art ProxyCLIP (Lan et al., 2024) on three challenging benchmark datasets COCO-Obj, COCO-171 and ADE-20K by 5.3%, 1.6% and 2.6%, respectively, while achieving competitive performance PASCAL-21. Additionally, when provided with single visual example and only few iterations of fine-tuning significantly improves across all four datascts, with an improvement of 9.9%, 2.5%, 1.2% and 1.7% for PASCAL-21, COCO-Obj, Coco-171 and ADE-20K respectively. This emphasizes the importance of visual aids even single visual example can reduce ambiguities that may arise from text prompts alone. Examples include the stuff classes COCO-171 that are inherently ambiguous (c.g., solid or structural). The results in this table are reported using BLIP top-2 layers multi multi-prompt setting. Single vs. Multi-prompt performance and extensibility other VLMs is discussed next.
small number of parameters, the segmentation perfromance
15

---

Published in Transactions on Machine Learning Research (02/2026)
Method
Base Training
SO
S1
S2
$3
Average
Few-shot segmentation approackes strictly separating base and novel classes
LLaFS (CVPR 2024) Zhm et al.
202
71.2
78.8
72.3
68.5
73.5
HMNet (NeurIPS 2024] (uu BL.
2224
72.2
75.1
70.0
63.9
70.4
AMFurmer (NeurIPS 2023] Waug ts ml.
20233
71.3
76.7
70.7
63.9
70.7
HDMNeet (CVPR 2023) Peng al.
71.0
75.4
62.0
69.3
ACBC (CVPR 2021) Zhu 27210
73.0
76.0
69.2
72.0
OCNet (ICCV 2025) Wen 2025
73.5
75.9
71.1
64.9
71.4
Ours
70.4
73.7
76.4
72:2
72.9
Ours w/ base training
71.1
77.7
75.2
73.5
75.2
Calegories in Inaining ooeer cutegories in leaking
Graphi'SS (NeurIPS 2024) (Zhang ot al.
2024n
72.1
Matcher' (ICLR 2024) (Liu ét al. 2024
68.1
Comparison with SOTA 1-shot segmentation approaches on Pascal-5', Results are reported under the standard 1-shot setting using four disjoint splits (S0-S3), where each split treats different set of five classes as novel and the remaining classes as base. Our method without basc-class training remains competitive with prior work, achieving the second-best average performance after (Zhul et al. 2024a Incorporating base-class pretraining yields consistent improvements across all splits and achioves statc-of-tho-art average performance. denotes mothods that do not strictly enforce base-novel class separation. Best results are shown in bold.
Table 2:
4.2.2 Comparison to 1-shot Segmentation Approaches
As mentioned section 3.4 in the conventional few-shot segmentation protocol, semantic classes are parti- tioned into disjoint base and novel sets across multiple dataset splits. Models are first trained on base classes using abundant segmentation annotations and are subsequently adapted to novel classes using only single annolated support example per class. Our method the other hand does not involve any pre-training on base classes with abundant data.
In Table 2 we compare our approach against a range recent state-of-the-art few-shot segmentation meth- ods that strictly follow the standard few-shot protocol, which evaluates performance over four disjoint splits (SO-S3), each defining a different set of novel classes. As observed, our method achieves competitive perfor- mance even without any base-class segmentation training, placing it in substantially more challenging and ammotation-effcient setting.
For completeness, we also report results when our model is trained under the standard few-shot protocol, including base-class training with segmentation annotations. Under this fair setting (strictly following few- shot segmentation protocol), our approach achieves taate--f-thh--r performance, outperforming several prior methods.
We further distinguish methods that do not enforce strict base novel separation and allow novel-class expo- during base training, and report their results separately for clarity. Overall, these results demonstrate that our comparisons are fair and comprehensive, and that the proposed approach remains effective across both training-fres free and standard few-shot settings.
4.2.3 Liser-wise mloU and Layer Ranking using InfoScore
Figure presents the training-free segmentation performance of BLIP (in the single-prompt per class set- ting) on the Pascal-21 validation set when each layer is independently used for segmentation, f.e., layer-wise mloU (in red). As shown, segmentation performance varies substantially across layers; the best-performing layer achieving an mIoU of 55.6% and the lowest-perforiing layer achieving an mIoU of only 6.3%.
We also report the InfoScore (in blue) for each of the 12 layers of BLIP along with their corresponding rankings. As observed, the proposed InfoScore effectively identifies the top-performing layers on Pascal-21. In most cases, the InfoScore ranking closcly aligns with the mloU ranking, with only minor deviations. This demonstrates the importance of selecting or extracting attention maps from layers that exhibit strong grounding between text. and image, which in turn leads to improved segmentation performance.
16

---

Published in Transactions on Machine Learning Research (02/2026)
Layer Index
Layer3
Layero
LayerB
Layer1l
Layerd
Layer10
Layer7
Layers
Layer2
Layerl
Layers
Layer5
200
60
150
50
InfoScore
miou
100
30
50
10
mloU Infos Rank (1st)
(2nd)
(4th)
(3rd)
[LOIN]
(1218)
(8th)
(Pth)
[8AN]
(9th)
151M
(11th)
Figure 5: Illustration of the InfoScore measure on BLIP. The mloU Rank reflects the descending order of mIoU values (in red) derived from cross-attention maps for cach standalone layer (labeled LayerN, top) the PASCAL VOC 2012 validation set (1449 images), compared the predicted InfoS Rank (bottom) based our InfoScore measure (in blue) requiring annotations. Most InfoScore rankings align with mIoU Rankings, with minor displacements of =2 positions highlighted in bold, except for four layers. Empirically, the top-1 and top-2 layers are correctly identified and consistently deliver better performance across four datasets and three VLMs. Results shown here are in single-prompt per class setting.
Layer Selection
PASCAL-21
COCO-Obj
COCO-171
Training Pree
All layers (12 layers)
42.8
35.9
25.1
Random (1 layer)
21.5
15.6
11.0
Naive (First layers)
48.2
37.2
25.5
Naive (Last 2 layers)
19.6
10.0
6.8
InfoScore (Top-1)
55.6
39.0
25.9
InfoScore (Top-2)
58.0
42.6
28.3
InfoScore (Top-3)
51.3
38.0
27.4
InfoScore (Top-6)
49.2
37.0
25.2
One-shot Supervision
InfoScore (Top-1) InfoScore (Top-2)
66.8
44.3
26.6
67.5
45.4
28.9
Table 3: Analysis InfoScore-basel layer selection in BLIP. Using an ensemble of attention maps from the top-1, top-2, or top-3 layers ranked by the InfoScore measure outperforms ensembling all layers, random selection, naive selection. Results are reported under the single-prompt per class setting: best results are shown in bold.
In Table 3 we further demonstrate the benefits and effectiveness of our layer selection stralegy using InfoScore compared to naive or random selection strategies on BLIP. The results show that using an ensemble of the top attention maps, ranked by the proposed InfoScore, significantly outperforms simple aggregation strategies such as averaging attention maps from all layers, well as naive and random baselines across all three datasets: Pascal-21, COCO-Obj, and COCO-171.
17

---

Published in Transactions on Machine Learning Research (02/2026)
PASCAL-21
COco-Obj
COCO-171
Method
Training-Fres
One-Shot
Training-Fro
One-Shot
Training- Free
Oue-Shot
Single
58.0
67.5
42.6
45.4
28.3
28.9
Multiple 60.2
70.1
42.8
45.3
28.1
29.3
Table 4: Ablation of single vs, multiple prompts with related words. Multiple prompts consistently improve segmentation accuracy on PASCAL-21 and COCO-171, while their effect on COCO-Obj less pronounced. Best results are in bold.
PASCAL-21
COCO-Obj
Method
Layers
Training- Freeing-Frree
One-Shot
Training- Free
One-Shot
Cross-Attentio Based VLMs
BLIP
Top-1
55.6
66.8
39.0
44.3
BLIP
Top-2
58.0
67.5
42.6
45.4
ALBEF
Top-1
37.9
65.1
28.7
37.7
ALBEF
Top-2
43.2
65.5
31.7
38.7
LLM-Based VLMs
LLaVA-155-BB
Top-1
50.1
63.7
31.3
41.0
LLaVA-1.5-7B
Top-2
51.6
64.2
32.3
42.2
Table Segmentation performance across different vision-language models (VLMs). Fino-tuning with single example significantly improves performance for BLIP (Li et al. 2022b), ALBEF (Li et al. 2021) and LLa VA-1.5-BB (Liu et al. 2024a). Results shown are for the single-prompt per class setting. Best results are bolded.
Additionally, the results show that using the top-2 layers consistently outperforms using only the top-1 layer. Performance tends to saturate at the top-3 layers and declines when more layers are added (e.g., top-6). As noted before, our ranking also surpasses other naive strategies, including selecting the first or last two layers, aggregating attention maps across all layers, or randomly selecting layer. For the Random baseline, we evaluate every layer once (12 runs), compute the mIoU for cach layer individually, and report the average (expectation) across all runs, mimicking unbiased uniform sampling.
4.3 Single versus Multi-prompt setting
In Table ! we compare the performance difference between using a single prompt corresponding to each class name against the use of multiple prompts of related words that specific class name. shows that multiple prompts either provide better on-par performance to the single prompt across both training free and onc-shot supervision modes.
4.3.1 Extensibility Across VLMs
Table demonstrates the extensibility and practicality of our approach, which relies on our proposed In- foScore for layer selection, across diverse vision-language models (VLMs), including recent LLM-based ar- chitectures such as LLaVA-1.5. Across all three VLMs, incorporating single visual example consistently leads to substantial performance gains without introducing additional parameters classificrs. Among the evaluated models, BLIP achieves the best overall performance on both the PASCAL-21 and COCO-Obj benchmarks, outperforming ALBEF and LLaVA-1.5 in both the training free and one-shot supervision set- tings. Additionally, using the Top-2 layers selected by the InfoScore mcasure yields better performance than using only the Top-1 layer in the training-fres scenario across all VLMs, although this advantage diminishes after fine-tuning.
18

---

Published in Transactions on Machine Learning Research (02/2026)
Layer Selection
BLIP (12 layers)
ALBEF (6 layers)
LLaVA-1.5-7B (32 layers)
All layers
42.8
34.3
49.7
Random (I layer)
21.5
20.6
39.3
Naive (First 2 layers)
48.2
38.4
26.3
Naive (Last 2 layers)
19.6
11.0
34.4
InfoScore (Top-1)
55.6
37.9
50.1
InfoScore (Top-2)
58.0
43.2
51.6
InfoScore (Top-3)
51.3
39.6
49.8
InfoScore (Top-6)
49.2
34.3
49.5
Table Extensibility and effectiveness InfoScore-based layer selection in the training-free set- ting on PASCAI-21. Across three distinct architectures BI.I, ALBEF, and ---.1.5-7B-InooSoe consistently identifies the most informative layers. In all cases, selecting the Top-2 layers ranked by In- foScore achieves the best mloU, oulperforming random selection, naive heuristics, and averaging all layers. Results shown are in single-prompt per class setting.
70
Accuracy (%)
60
PASCAL (Single)
PASCAL (Multi)
COCO-Obj (Single)
55
COCO-Obj (Multi)
50
0-shot
1-shot
3-shot
5-shot
Shots
Figure 6: Performance analysis with increased number of shots for single and multi-prompt setting. Our model scales the number of shots increases from 1 to 5 on both COCO-Obj and PASCAL- 21 datasets. Dashed lines denote the single-prompt per class setting, and solid lines denote the multi-prompt per class setting. The benefit of using multiple prompts diminishes with more shots.
4.3.2 Effectiveness of InfoScore-baadd based layer selection across VLMs
To demonstrate that our InfoScore-based layer selection strategy generalizes VLMs beyond BLIP, Table 6 reports results ALBEF and LLaVA-L.5-7B in addition to BLIP. Across all three VLMs, the Top-2 layers selected by InfoScore consistently achieve the highest mIoU, outperforming random selection, naive heuristics, and averaging attention maps over all layers.
Since LLaVA-L.5-7B has 32 layers, a full sweep would be computationally expensive. Instead, we perform 5 independent runs: in each run, we uniformly sample one layer, compute its mIoU, and report the average across the 5 runs. The sampled layers in our experiments were Layer 7, Layer 0, Layer 16, Layer and Layer 27. For ALBEF, follow the same approach BLIP (see Sec. 4.2.3
19

---

Published in Transactions on Machine Learning Research (02/2026)
Unseen mloU
Percent of Seen Classes
S1
S2
S3
S4
Ave
0% (zero-shot)
63.6
57.2
59.9
59.5
60.1
25% (one-shot)
64.8
60.0
59.0
62.8
61.7
Table 7: Comparing fine-tuning with 1-shot learning on 25% of the classes (75% unseen) against zero-shot performance on the unseen classes in PASCAL-21. The seen and unseen classes are divided into four non-overlapping splits. Fine-tuning with just one visual example per class for 25% of the classes improves performance on the 75% unseen categories. Results shown are in multi-prompt per class setting.
4.3.3 Extensibility Beyond One-Shot
We further examine the scalability of our method beyond the one-shot setting. Figure i compares the shot and 5-shot performance against the 1-shot bascline both single-prompt and multi-prompt settings the PASCAT-21 and COCO-Obj datasets. All results averaged five runs to ensure stability. As expected, increasing the number of shots leads to consistent improvements of approximately 1 1-2% on both datasets. Notably, the performance gap between single-prompl and multi-prompl settings narrows the number of shots increases, suggesting that our method does not strongly depend on prompt engineering when a few visual demonstrations are provided beyond the one-shot scenario.
4.3.4 One-shot Tuning with Partial Vocabulary
Table 7 illustrates the impact of one-shot fine-tuning on small subset of the vocabulary and how this adaptation transfers to unscen categories. Specifically, we fine-tune the model using single visual example for only 25% of the total classes, leaving the remaining 75% entirely unseen during fine-tuning. The classes are divided into four non-overlapping splits; for each split, we fine-tune on one-shot examples from 25% of the classes and evaluate performance on the remaining 75%.
The results show that even with such minimal supervision just one example per class for quarter of the vocabulary-the fine-tuned model significantly outperforms the zero-shot baseline the unseen categories. This demonstrates the model's ability to generalize from a small, partially labeled subset to much larger set of unseen classes, likely benefiting from the shared semantic structure among categories.
Overall, this finding underscores the practicality of leveraging limited labeled data to boost performance settings where many classes remain unlabeled, offering scalable solution for resource-constrained scenarios.
4.4 Ablation Study
4.4.1 Impact of Image- Text Text Scoring
Table 8 Janalyzes the impact of re-weet re-weighting the image-text attention maps using the class VLM scores already computed by the VLMs. Incorporating this re-weighting in our design lcads to significant improvements in segmentation performance for both training-free and one-shot settings. While the benefit of using class VLM scores is slightly reduced in the one-shot setting, it remains crucial for achieving strong training-free performance in both aattentioo-basee VLMs like BLIP and LLM-based VLMs like LLaVA.
4.4.2 Ablation with Individual Components of InfoScore
In Table 9 we compare the layer rankings of the BLIP model obtained using individual components of the InfoScore measure on the PASCAL-21 and ADE-20K datasets. As shown, relying solely on either Image Entropy or Dataset Entropy leads suboptimal layer selection and consequently poor segmentation performance.
20

---

Published in Transactions on Machine Learning Research (02/2026)
PASCAL-21
COCO-Obj Training-
Method
Class VLM Score
Training- Free
One-Shot
Preee Pree
One-Shot
BLIP
25.1
62.4
26.0
38.7
BLIP
58.0
67.5
42.6
45.4
LLaVA-L5--B
X
36.8
55.0
20.1
34.4
LLaVA-155-BB
51.6
63.7
32.3
42.1
Table 8: Ablation on the Importance of image-Text Scoring. Re-weighting the image-text attention maps using the class VLM scores significantly enhancos performance in the training-free scenario and provides additional gains in the one-shot setting. The benefit of this scoring is more pronounced without fine- line-Luning, though it still contributes to improved results with one-shot supervision. Results reported for the single- prompt per class setting. Best performances are bolded.
PASCAL-21
ADE-20K
Ranker
nDGG@2
mloU@2
nDCG@2
mloU@2
Image Entropy
0.43
9.2
0.57
8.9
Dataset Entropy
0.67
19.1
0.57
8.9
EntropyRatio
1.00
58.1
0.93
16.8
InfoScore
1.00
58.1
0.98
22.8
Table 9: Ablation of InfoScore Components for Layer Ranking of the BLIP Model on PASCAL- 21 and ADE-20K. We report nDCG@2 and mloU@2 for each ranker on both datasets, which respectively indicate the ranking quality and the training-free segmentation performance (mloU) of the top-2 layers. Results are shown for single-prompt per class setting.
Using only Image Entropy favors layers biased toward a few dominant classes, while Dataset Entropy favors layers predicting many classes per image but with low confidence. Both lead to poor segmentation. Entropy- Ratio (Sec. 3.2 balances these effects by promoting layers with confident per-image predictions and global class diversity, yielding strong results on PASCAT.-21. However, overlooks image-to-image variation and can mis-rank layers when global balance masks low variability, seen on ADE-20K.
InfoScore resolves this by multiplying EntropyRatio with the Coefficient of Variation (CoV), explicitly mod- eling prediction variability across images. As Table g shows, this consistently produces the best rankings and significantly improves training- segmentation, especially on ADE-20K.
Fine-Tuned Parameters
PASCAL-21
COCO-Obj
Top-2 Cross-Attention Layers Only
65.0
44.4
Word Embeddings Only
63.2
43.4
+ Random-2 Cross-Attention Layers
66.1
42.4
+ Top-2 Cross-Attention Layers
67.5
45.4
+ All Cross-Attention Layers
64.1
40.6
+ All VLM Parameters
60.9
41.9
Table 10: Fine-tuning Strategies. The top section shows the standalone impact of fine-tuning the top-2 cross-attention layers. The bottom section presents the progressive addition of parameters starting from fine-tuning word embeddings. Fine-tuning both the word embeddings and the top-2 layers yields the best performance, while further increasing the number of tunable parameters leads to reduced performance. Results are reported in the single-prompt per class setting. Best results are bolded.
21

---

Published in Transactions on Machine Learning Research (02/2026)
O-shot w/o Class VLM Score
O-shot Class VLM Score
Image
GT
s-shot
Figure 7: Qualitative Results PASCAL-21: Shown results from zero-shot training-free w/o image-text scoring (3rd column), zero-shot model w/ image-text scoring (4th column), and one-shot fine- tuning (5th column). The final row shows an example where the zero-shot prediction outperformed the fine-tuned one-shot model. For all variants, we ensemble the top-2 layers from BLIP ranked by InfoScore. All the variants use multi-prompt per class setting.
4.5 Parameter Selection for One-Shot Fine- Tine-Tuning
Table 10 demonstrates the effectiveness of our selective fine-tuning strategy, where fine-tune only the word embeddings and the top-2 image-text cross-attention layers in BLIP. We compare six fine-tuning variants: (i) fine-tuning only the top-2 cross-attention layers selected by InfoScore, (ii) fine-tuning only the word embeddings, (iii) fine-tuning both the word embeddings and two randomly picked cross attention layers (over runs), (iv) fine-tuning both the word embeddings and the top-2 cross-attention layers, (v) fine-tunig all the cross-attention layers alongwith word embeddings, and (v) fine-tuning all VL.M paramters. The results show that finc-tuning both the word embeddings and the top-2 cross-attention layers selected by InfoScore consistently yields the best performance. Notably, increasing the number of tunable parameters beyond this point leads to reduced performance, highlighting the effectiveness of our selective fine-tuning.
Overall, BLIP consists of approximately 146.13 million parameters. In our selective fine-tuning strategy, we tune only the lop-2 cross-allention layers and the word embeddings corresponding to the prompts. Although BLIP's vocabulary size is 30,524 words, in practice, we fine-tune only the embeddings of the words that appear in the prompts-typinn million parameters out of 446.13 million, which is just 1.28% of the total parameters. Importantly, we do not introduce any additional parameters in either the training- free few-shot. fine-tuning settings. Our method
up to around 250 word embeddings. This results in fino-tuning approximately 5.7
22

---

Published in Transactions on Machine Learning Research (02/2026)
leverages the existing VLM architecture to generate segmentation maps efficiently, without increasing model size, demonstrating both scalability and parameter efficiency.
4.6 Qualitative Results
Figure 7 presents the qualitative results of our approach across different settings, illustrating the impact of image-text scoring and one-shot fine-tuning. As shown, the model without image-text scoring consistently under-segments across all images, labeling most of the pixels as background. Applying image-text scoring for filtering notably improves performance across all images, as helps the model better focus on relevant areas. However, there a tendency to over-segment certain categories, as seen in the first couple rows. One- shot fine-tuning further enhances segmentation accuracy, consistently producing more precise segmentations. However, the last row highlights a failure case for the one-shot fine-tuning, where potential bias from the single example of the class couch led to mis-classification.
We provide additional qualitalive results on the COCO-Obj and ADE-20K datasets in Figures 10 and respectively, with further discussion in Section A.4 of the Appendix.
5 Limitations
Our method is applicable to wide range of modern vision-language models that expose explicit multimodal interactions between image and text tokens. Its effectiveness depends on the quality of the resulting mul- timodal attention maps. In particular, successful segmentation requires attention maps that exhibit strong text-image grounding and sufficient spatial resolution. Architectures that aggressively merge or downsample visual tokens may produce attention maps that are too coarse for fine-grained pixcl-level sogmentation.
In addition, our framework is not applicable to CLIP-like encoder-only vision language models that compute image and text representations independently and do not expose multimodal attention during inference. Such models fall outside the scope of this work.
6 Conclusion
Our work extends state-of-the-arl approaches that perform segmentation by using vision loundation models by eliminating the need for abundant segmentation labels. We leverage the strength of a single visual example to better disambiguate categorics beyond their textual names. This, combined with our proposed InfoScore measure, reduces reliance on intensive prompt engineering or tuning the layers/heads selected for segmentation in VLMs. Our approach can operate in both training-free and one-shot fine-tuning settings, with the latter achieving significant gains on four benchmarks and demonstrating compatibility across three different VLMs.
References
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanily checks for saliency maps. Advances in neural information processing systems, 31, 2018.
Cean-Baptiste Alayrac, Jeff Donahne, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716 23736, 2022.
Nikita Araslanov and Stefan Roth. Single-stage semantic segmentation from image labels. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp.
2020.
Pablo Arbeláez, Bharath Hariharan, Chunhui Gu, Saurabh Gupta, Lubomir Bourdev, and Jitendra. Malik. Semantic segmentation using regions and parts. In CVPR, pp.
IEEE, 2012.
Pablo Arbeléez, Jordi Pont-Tuset, Jonathan T Barron, Ferran Marques, and Jitendra Malik. Multiscale combinatorial grouping. Tn CVPR, pp. 328 335, 2014.
23

---

Published in Transactions on Machine Learning Research (02/2026)
Luca Barsellotti, Roberto Amoroso, Lorenzo Baraldi, and Rita Cucchiara. Fossil: Free open-vocabulary semantic segmentation through synthetic references retrieval. In Winter Conference on Applications of Computer Vision, pp. 1464 1473, 2024.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bossclut, Emma Brunskill, et al. On the opportunities and risks of foundation models. Xiv preprinl arXic:2108.0720 2021.
Walid Boussclham, Felix Petersen, Vittorio Ferrari, and Hilde Kuchne. Grounding everything: Emerging localization properties in vision-language transformers. In CVPR, 2024.
Yuri Boykov and Gareth Funka-Lea. Graph cuts and efficient nd image segmentation. IJV,, 70(2):109-15 131, 2006.
Yuri Boykov and Vladimir Kolmogorov. An experimental comparison of min-cut/max--low algorithms for energy minimization in vision. PAMI, 26(9):1124-1137, 2004.
Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. Zero-shot semantic segmentation. Aduances in Newral Information Processing Systems, 32, 2019.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In IEEE Conference on Computer Vision Pattern Recognition, pp. 1209 1218, 2018.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In International Conference on Com- puter Vision, pp. 9650-990000 160009660,200 2021.
Joao Carreira and Cristian Sminchisescu. Cpme: Automatic object segmentation using constrained para- metric min-cuts. PAMI, 34(7):1312-1328, 2011.
Joao Carreira, Rui Caseiro, Jorge Batista, and Cristian Sminchisescu. Semantic segmentation with second- order pooling. In ECCV, pp. 430 443. Springer, 2012.
Junbum Cha, Jonghwan Mun, and Byungseok Roh. Learning to generate text-grounded mask for open-world semantic segmentation from only image-text pairs. In IEEE Conference on Computer Vision Pattern Recognilion, pp. 11165 111544 2023.
Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 782-78 182-791, 2021.
Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan I. Yuille. Semantic image segmentation with deep convolutional nets and fully connected crfs. arXiv preprint arXiv: 7062, 2014.
Liang-Chise Chigh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connectod crfs. PAMI, 40(4):834 848, 2017a.
Slang-Chich Chen, George Papandrcon, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution 2017b.
for semantic image segmentation. arXiv preprint
Wei-Ge Chen, Irina Spiridonova, Jianwei Yang, Jianfeng Gao, and Chunyuan Li. Llava-interactive: An all-in-one demo for image chat, segmentation, generation and editing. arXiv preprint arXiv:2311.0057,, 2023.
Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. NIPS, 34:17864 17875, 2021.
Bowen Cheng, Ishan Misra, Alexander Schwing, Alexander Kirillov, and Rohit Girdhar. Maskod-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1290-1299, 2022.
24

---

Published in Transactions on Machine Learning Research (02/2026)
Jifeng Dai, Kaiming He, and Jian Sun. Convolutional feature masking for joint object and stuff segmentation. In CVPR, pp. 3992-4000, 2015.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In ICLR, 2024.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized Ilms. Advances in neural information processing systems, 36:10088-10115, 2023.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the assaciation for compulational linguisties: human language technologies, volume 1 (long and short papers), pp. 4171 4186, 2019.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88:303 338, 2010.
Ruth Fong, Mandela Patrick, and Andrea Vedaldi. Understanding deep networks via extremal perturbations and smooth masks. In Proceedings of the IEEE/CVF internalional conference on compuler vision, pp. 2019.
Ruth Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In Proceedings of the IEEE international conference on computer vision, pp. 3429 3437, 2017.
Tristan Gomez, Thomas Fréour, and Harold Mouchère. Metrics for saliency map evaluation of deep learning explanation methods. In International Conference on Pattern Recognation and Artificial Intelligence, pp. 84-95. Springer, 2022.
Arushi Gupta, Nikunj Saunshi, Dingli Yu, Kaifeng Lyu, and Sanjeev Arora, New definitions and evaluations for saliency methods: Staying intrinsic, complete and sound. Advances in Neural Information Processing Systems, 35:33120 33133, 2022.
Sina Hajimiri, Malik Boudiaf, Ismail Ben Ayed, and Jose Dolz. A strong bascline for generalized few-shot semantic segmentation. In IEEE Conference on Compuler Vision Pallern Recognition, pp. 11269 11269-11278. 2023.
Sina Hajimiri, Ismail Ben Ayed, and Jose Dolz. Pay attention to your neighbours: Training vocabulary semantic segmentation. Winter Conference on Applications of Computer Vision, 2025.
fraeeing-free free open-
Kaiming Hc, Xinlci Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In IEEE Conference on Computer Vision Pallern Recognition, pp. 16000 16009, 2022.
Ruozhen He, Paola Cuscante-Bonilla, Ziyan Yang, Alexander C Berg, and Vicente Ordonez. Improved visual grounding through self-consistent explanations. In IEEE Conference on Computer Vision Pattern Recognition, pp. 13095 13105, 2024.
Anna Hedström, Leander Weber, Sebastian Lapuschkin, and Marina Höhne. A fresh look at sanity checks for saliency maps. In World Conference on Explainable Artificial Intellignne,, pp. 403-420 Springer, 2024.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods deep neural networks. Advances neural information processing systems, 32, 2019.
Mir Rayat Imtiaz Hossain, Mennatullah Siam, Leonid Sigal, and James J Little. Visual prompting for generalized few-shot segmentation: A multi-scale approach. In IEEE Conference on Computer Vision Pallern Recognition, pp. 23470 23480, 2024.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Alu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
25

---

Published in Transactions on Machine Learning Research (02/2026)
Jitesh Jain, Jiachen Li, MangTik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. OneFormer: One Transformer to Rule Universal Image Segmentation. 2023.
Sarthak Jain and Byron C Wallace, Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 3543 3556, 2019.
Kalervo Järvelin and Jaana Kekäläinen Cumulated gain- based evaluation of ir techniques. ACM Transac- tions on Information Systems (TOTS), 20(4):422-44) 446, 2002.
Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. See what you are told: Visual attention sink in large multimodal models. In ICLR, 2025.
Laurynas Karazija, Iro Laina, Andrea Vedaldi, and Christian Rupprecht. Diffusion models for open- Springer, 2024.
vocabulary segmentation. In European Conference on Computer Vision, pp.
Alexander Kirillov, Fric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Yen Lo, et al. Segment anything. In International Conference on Compuler Vision, pp. 4015 4026, 2023.
Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected crfs with gaussian edge potentials. Advances in newral information processing systems, 24, 2011.
Matthias Kümmerer, Thomas SA Wallis, and Matthias Bethge. Information-theoer model comparison unifies saliency metrics. Proceedings of the National Academy of Sciences, 112662):16054416055, 2015.
John Lafferty, Andrew McCallum, and Fernando CN Pereira. Conditional random fields: Probabilistic
models for segmenting and labeling sequence data. 2001.
Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang. Litong Feng, and Wayne Zhang. Proxyclip: Proxy attention improves clip for open-vocabulary segmentation. In ECCV. Springer, 2024.
Boyi Li, Kilian Q Weinberger, Serge Belongic, Vladlen Koltun, and René Ranftl. Language-driven semantic segmentation. arXiv preprinl ar arXiv:2201.033 03546, 2022a.
Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, Jianfeng Gao, et al. Multimodal foundation models: From specialists to general-purpose assistants. Foundations and Trends0 in Computer Graphics and Vision, 16(1-2):1 214, 2024.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in Neural Information Processing Systems, 34:9694-9708 97005, 2021.
Jumnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pp. 12888 12900. PMLR, 2022b.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference Machine Learning, pp. 19730 19742. PMLR, 2023a.
Ziyi Li, Qinye Zhou, Xiaoyun Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xic. Open-vocabulany object seg- mentation with difflusion models. In Proceedings of the IEFF/CVV International Conference on Compuler Vision, pp. 7667-7676, 2023b.
Tsung-Y: Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pp. 740-755. Springer, 2014.
26

---

Published in Transactions on Machine Learning Research (02/2026)
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVE Conference Computer Vision and Pottern Recognition, pp. 26296- 26306, 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in Newral Information Processing Systems, 36, 2024b.
Sun-Ao Liu, Yiheng Zhang, Zhaofan Qiu, Hongtao Xie, Yongdong Zhang, and Ting Yao. Learning orthogonal prototypes for generalized few-shot semantic segmentation. In IEEE Conference on Computer Vision Pattern Recognition, pp. 11319-11328, 2023.
Yang Liu, Muzhi Zhu, Henglao Li, Hao Chen, Xinlong Wang, and Chunhua Shen. Matcher: Segment anything with one shot using all-purpose feature matching. In ICLR, 2024c.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In CVPR, pp. 14410 2015.
Jiayun Luo, Siddhesh Khandelwal, Leonid Sigal, and Boyang Li. Emergent open-vocabulary semantic seg- mentation from off-the-shelf dision-langual models. In IEEE Conference on Computer Vision Pattern Recognition, pp. 4029-4040, 2024.
Juhong Min, Dahyun Kang, and Minsu Cho. Hypercorrelation squeeze for few-shot segmentation. In International Conference on Computer Vision, pp. 6941-6952, 2021.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alazeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024.
Adam Pardyl, Grzegorz Rypeśé, Grzegorz Kurzejamski, Bartosz Zieliński, and Tomasz Trzciúski. Active visual exploration based on attention-map entropy. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pp. 1303 1311, 2023.
Bohao Pong, Zhuotao Tian, Xiaoyang Wu, Chengyao Wang, Shu Liu, Jingyong Su, and Jiaya Jia. Hicrarchical dense correlation distillation for few-shot segmentation. In Proceedings of the IEEE/CVP conference on computer vision and pattern recognition, pp. 23641-23655 2023.
Vitali Pelsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. arXiv preprint arXiv:1806.074. 07401, 2018
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkinn, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748 8763. PMLR, 2021.
Nikhila Ravi, Valentin Gabeur, Yuan- Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint ar Xi::2240 00714, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684 10695, 2022.
Tim Salimans, Ian Goodfellow, Wojeicch Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in Newral Information Processing Systems, 29, 2016.
Sofia Serrano and Noah A Smith. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Associalion for Compulational Linguislics. Association for Compulational Linguisties, 2019.
Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. XrXiv preprint or Xive 709.03410, 2017.
27

---

Published in Transactions on Machine Learning Research (02/2026)
Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. PAMI, 22(8))888 905, 2000.
Mennatullah Siam, Boris N Oreshkin, and Martin Jagersand. Amp: Adaptive masked proxies for few-shot
segmentation. In International Conference on Computer Vision, pp.
2019.
Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. COLM, 2024.
Marvin TT Teichmann and Roberto Cipolla. Convolutional crfs for semantic segmentation. arXiv preprint arXiv:1805.04 2018.
Zhuotao Tian, Xin Lai, Li Jiang, Shu Liu, Michelle Shu, Hengshuang Zhao, and Jiaya Jia. Generalized few- shot semantic segmentation. In IEEE Conference on Computer Vision Pattern Recognition, pp. 11563- 11572, 2022.
Anthony Meng Huat Tiong, Junnan Li, Boyang Li, Silvio Savarese, and Steven CH Hoi. Plug-ande legg-and-plyy vqa: Zero-shot vqa by conjoining large pretrained models with zero training. In Findings of the Empiricul Methods in Natural Language Processing 2022, pp. 951 967, 2022.
Jasper RR Uijlings, Koen EA Van De Sande, Theo Gevers, and Arnold WM Smeulders. Selective search for object recognition. IJCV, 104(2):154-17) 171, 2013.
Feng Wang, Jieru Mei, and Alan Yuille. Sclip: Rethinking self-attention for dense vision- language inference. In European Conference on Computer Vision, pp. 115-332. Springer, 2024a.
Haoxiang Wang, Pavan Kumar Anasosalu Vasu, Fartash Faghri, Raviteja Vemulapalli, Mchrdad Farajtabar, Sachin Mehta, Mohammad Rastegari, Oncel Tuzel, and Hadi Pouransari. Sam-clip: Merging vision foun- dation models towards semantic and spatial understanding. In CVPR, 2024b.
Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng. Panet: Few-shot image semantic segmentation with prototype alignment. In International Conference on Computer Vision, pp. 2019.
Wei Wang, Yizhou Wang, Qingming Huang, and Wen Gao. Measuring visual saliency by site entropy rate. In CVPR, pp.
IEEE, 2010.
Yuan Wang, Naisong Luo, and Tianzhu Zhang. Focus on query: Adversarial mining transformer for few-shot segmentation. Advances in neural information processing systems, 36:31524 31542, 2023.
Chunlin Wen, Yu Zhang, Jie Fan, Hongyuan Zhu, Xin-Shen Wei, Yijun Wang, Zhiqiang Kou, and Shuzhou Sun. Object-level correlation for few-shot segmentation. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 23689-23663 2025.
Jianzong Wu, Xiangtai Li, Shilin Xu, Haobo Yuan, Henghui Ding, Yibo Yang, Xia Li, Jiangning Zhang, Yunhai Tong, Xudong Jiang, et al. Towards open vocabulary lcarning: A survey. IEEE Transactions on Pallern Analysis and Machine Intelligence, 2024.
Monika Wysoczańska, Oriane Siméoni, Michaël Ramamonjisoa, Andrei Bursuc, Tomasz Trzciński, and Patrick Pérez. Clip-dinoiser: Teaching clip few dino tricks for open-vocabulary semantic segmenta- tion. In ECCV, pp. 320 337. Springer, 2024a.
Monika Wysoezańska, Oriane Siméoni, Michaël Ramamonjisoa, Andrei Bursuc, Tomasz Trzciáski, and Patrick Pérez. Clip-dinoiser: Teaching clip a few dino tricks for open-vocabulary semantic segmenta- tion. In ECCV. Springer, 2024b.
Aoran Xiao, Weihao Xuan, Heli Qi, Yun Xing, Ruijic Ren, Xiaoqin Zhang, Ling Shao, and Shijian Lu. Cat-sam: Conditional tuning for few-shot adaptation of segment anything model. Europeon Conference on Computer Vision, 2024.
28

---

Published in Transactions on Machine Learning Research (02/2026)
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Bycon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Semantic segmentation ion emerges from text supervision. In IEEF Conference on Computer Vision Pattern Recognition, pp. 18134 18144, 2022.
Groupvit:
Qianxiong Xu, Xuanyi Liu, Lanyun Zhu, Guosheng Lin, Cheng Long, Ziyue Li, and Rui Zhao. Hybrid mamba for few-shot segmentation. Advances in Newral Information Processing Systems, 37:73858-73883 2024.
Xiao Xu, Chenfei Wu, Shachar Rosenman, Vasudev Lal, Wanxiang Che, and Nan Duan. Bridgetower: Building bridges between encoders in lision-language representation learning, In Association for the Aduancement of Artificial Intelligence, volume 37, pp. 10637-10647, 2023.
Chih-Kuan Ych, Cheng- Yu Hsich, Arun Suggala, David Inouye, and Pradeep K Ravikumar. On the (in) fidelity and sensitivity of explanations. Advances in neural information processing systems, 32, 2019.
Anqi Zhang, Guangyu Gao, Jianbo Jiao, Chi Liu, and Yunchao Wei. Bridge the points: Graph-based few- shot segment anything semantically. Advances in Neural Information Processing Systems, 01.33222-33261 2024a.
Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shumping Ji, Chen Change Loy, and Shuicheng Yan. Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding. Advances in Newral Information Processing Systems, 2024b.
Bolci Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k datasel. In Proceedings of the IEEE conference on compuler vision and pallern recognilion, pp. 633 633-641, 2017.
Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference on Computer Vision, pp. 896-712. Springer, 2022.
Ziqin Zhou, Yinjie Lei, Bowen Zhang, Lingqiao Liu, and Yifan Liu. Zegelip: Towards adapting clip for zero-shot semantic segmentation. In IEEE Conference on Computer Vision Pattern Recognition, pp. 11175-11185, 2023.
Lanyun Zhu, Tianrun Chen, Deyi Ji, Jieping Ye, and Jun Liu. Llafs: When large language models meet few-shot segmentation. In IEEE Conference on Computer Vision Pattern Recognition, pp. 3065 3075, 2024a.
Lanyun Zhu, Tianrun Chen, Jianxiong Yin, Simon See, and Jun Liu. Addressing background context bias in few-shol. segmentation through iterative modulation. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 3370 3079,379,4 2024b.
Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. Advances in neural information processing systems, 36:19769 19782, 2023.
Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. Advances in Neural Information Processing Systems, 36, 2024.
A Appendix
A.1 Further Implementation Details
For BLIP Li et
al.
2022bb and LLaVA Liu et al. 2024a), we roly on the HuggingFace 2 library. For
ALBEF Li et al. 2021;, we use LAVIS
29

---

Published in Transactions on Machine Learning Research (02/2026)
Layer Index
Layer1
Layer2
Layero
Layer3
Layer5
Layer4
40
InfoScore
200
30
mlou
20
100
10
mloU Rank (1st)
(2nd)
(3rd)
(4th)
(5th)
(6th) (5th)
InfoS Rank
(1st)
(2nd)
(3rd)
(4th)
(6th)
Figure 8: Illustration of the InfoScore measure on ALBEF. The mloU Rank reflects the descending order of mIoU values (in red) derived from cross-attention maps for cach standalone layer (labeled LayerN, top) on the PASCAL VOC 2012 validation set (1449 images), compared to the predicted InfoS Rank (bottom) based on our InfoScore measure (in blue) requiring no annotations. InfoScore rankings align with mIoU Rankings except for the worst two layers with rankings interchanged between 5th and 6th. Results shown here are single-prompt per class setting.
BLIP. For BLIP pretrained model weights of Sallefforee/bllpp------------
It uses ViT-L/16
backbone pre-trained on ImageNet vision encoder and BERT (Devlin et al. 2019 image-grounee text encoder. The cross-attention layers of BERT take image embeddings as key, value. There are 12 such cross- atlention layers and the number of heads is 12. The BLIP model is trained for image-text understanding using a combination of Image- Text Matching (TTM) and Image-Text Contrastive (ITC) Loss on 129M image-text pairs The ITM head of BLIP model used for class-scoring.
ALBEF. For ALBEF we use albef-retrieval VIT-B/16 pretrained on ImageNet. Similar to BLIP they also trained with ITM and ITC Loss but much smaller number of image-text pairs of 14.1M. It has six multi-modal layers where there explicit cross-attention between image and text.
chockpoint pre-trained model weights. uses
VA. For LLaVA use 1lava-1.5-7b-be as pre-trained model weights. It uses CLIP VIT-L/14 vision encoder which is pre-trained with 400M image-text pairs. The image embeddings are first passed into a projection layer which are then passed to a large languago model relying on Vicuna-7B.
One-shot fine tuning details. For COCO-Obj, PASCAL-21 and ADE-20K we use training images cropped maximum width of 512, while for COCO-171 use resolution of 256 for training but 512 during inference time. In order to finc-tune LLaVA efficiently, employ Low-Rank Adapters (LoRA) (Hu et al. 2022 with 4-bit quantization (Dettmers al.) 2023), and selectively fine-tune the LoRA adapters only in the layers identified by InfoScore, along with the word embeddings.
PostProc
Inference Time
PASCAL-21
COCO-Obj
ConvCRF TTichmann & Cipolla. 2018
0.4s
60.2
42.8
PAMR Araslanov & Roth 2020
3.6s
59.6
42.2
Table 11: Ablation with two different post-proccssing methods in the training-free setting. Con- vRRF (Tecchmann & Cipolla 2018 marginally outperforms PAMR (Araslanov & Roth 2020) while being significantly laster at inference. Even with PAMR post-processing, our training-free set setting outperforms state-of the--rrtrrr training-free free approaches like NACLIP (Hajimiri et al. 2025) on COCO-Obj by nearly 6%. Results in multi-prompt per class setting. Best results are in bold.
30

---

Published in Transactions on Machine Learning Research (02/2026)
58
56
54
mlou
52
50
Layer Info5 Rank
Layero (2nd)
Layera (9th)
Layer2 (6th)
Layer4 (10th)
Layers
Layers (5th)
Layer? (8th)
Layerl (4th)
Layers (7th)
Layer20 (12th)
Layer11 (3rd)
(11th)
Layer palred with top-1 (Layer3)
Figure 9: Performance analysis of pairing layers with the top-1 layer. We compare the mloU achieved by pairing the top-1 layer (Layer 3) with each remaining layer using BLIP, evaluated on Pascal-21.
A.2 InfoScore Additional Analysis
Layerwise analysis of mloU and InfoScore on ALBEF. Following the analysis conducted for BLIP (see section 4.2.3 we perform a layerwise study on ALBEF by independently evaluating each layer for training-free segmentation on the Pascal-21 validation set under the singlc-prompt per class sctting. As illustrated Figure segmentation accuracy exhibits noticcable variation across layers.
We further compute the corresponding InfoScore measure for each layer and compare the inclued ranking against the oracle mloU-based ranking. We observe that InfoScore correctly identifies the top-4 layers, with only minor swap in the ordering of the last two layers. This close alignment with oracle rankings demonstrates that InfoScore reliably captures layer informativeness and grounding quality, consistent. with our observations on BLIP
Performance of top-1 paired with others on BLIP. We ablate a challenging setting where we use the top-1 layer of BLIP identified with InfoScore measure paired with all the other layers in BLIP evaluated PASCAL-21 dataset. Figure shows that pairing the top-1 layer (Layer3) with the second best layer (Layer0), out performs all the other pairs. Thus, it confirms the benefits from our proposed approach even in the challenging setting.
A.3 Choice of Post-processing Algorithm
In this section, we compare post-processing with ConvCRF TTichhaann & Cipolla 2018 and PAMR [Araslanov & Roth 2020), used by recent methods (Wang et al.) 2024a| Hajimiri et al.) 2025 Ta-
ble 11 shows that our approach is robust to the choice of posi-processing, with ConvCRF providing slight improvement and being nine times faster than PAMR. In the zero-shot setting on COCO-Obj, our method outperforms the second-best. approach, NACLIP (Hajimiri et al. 2025), by 6%, regardless of post-processing, These results confirm that our approach consistently outperforms state-of the-art methods, independent of the post-processin strategy.
A.4
Additional Qualitative Results
A.4.1 Qualitative Results on COCO-Obj
We show additional qualitative results on COCO-Obj dataset in Fig. 10 It shows eight successful scenarios, where the variant without class VLM Score based re-weighting either under-segments or mis-classifies the objects in the scene. On the other hand, the one-shot variant. with only single visual example shows
31

---

Published in Transactions on Machine Learning Research (02/2026)
O-shot
O-shot
w/o Class
W Class
Image
GT
VLM Score
VLM Score
1-shot
Figure 10: Qualitative Results on COCO-Obj: Shown are results from zero-shot model w/o image- text scoring for filtering (3rd columm), zero-shot model w/ image-text scoring (4th column), and one-shot fine-tuning (5th column). The final row shows an example where the zero-shot prediction outperformed the fino-tuned onc-shot model. For all variants, we ensemble the top-2 layers ranked by InfoScore. All variants multi-prompt per class setting.
32

---

Published in Transactions on Machine Learning Research (02/2026)
O-shot
O-shot
w/o Class
w Class
Image
GT
VLM Score
VLM Score
1 s-shot
Figure 11: Qualitative Results on ADE-20K: Shown are results from zero-shot model w/o imago-text scoring for filtering (3rd column), zero-shot model w/ image-text scoring (4th column), and one-shot fine- tuning (5th column). The final shows example where the zero-shot prediction outperformed the fine-tuned one-shot model. For all variants, ensemble the top-2 layers ranked by InfoScore. All variants use multi-prompt per class setting.
considerable improvement in segmenting the objects in the scene and overcomes the aforementioned issues. Looking specifically at the second and seventh rows we see that the one-shot variant resolved the confusion on whether to sogment the clothes as part of class Person or not. Since by definition of the word itsclf it might exclude the clothes as another class, but with the class definition of COCO-Obj these are to be considered part of Person class.
33

---

Published in Transactions on Machine Learning Research (02/2020)
A.4.2 Qualitative Results ADE-20K
We present qualitative results on ADE-20K in Fig. I1 comparing zero-shot inference without class VLM score based re-weighting, zero-shot inference with class VLM score- based re-weighting, and one-shot fine- tuning. As observed in general, the zero-shot variant without class VLM score often suffers from fragmented regions, under-segmentation,
incorrect scmantic assignments. Incorporating class VLM scores consistently improves region coherence and semantic aligument, reducing spurious predictions across diverse scenes. The one-shot variant further refines object boundaries and resolves remaining ambiguities, producing segmenta- tions that more closely align with the ground truth.
For example, in the first row, the zero-shot variant without class conditioning produces fragmented and noisy regions, while class-aware scoring substantially improves semantic coherence; the one-shot model fur- ther sharpens object boundaries. In the third row, class-aware scoring helps suppress spurious background predictions around the foreground structure, which are further refined after one-shot adaptation.
Finally, the last row highlights case where zero-shot inference with class-aware scoring outperforms one- shot fine-tuning, indicating that strong class-conditioned priors can already yield competitive segmentations without adaptation.
A.5 List of Prompts
In this section, we provide the detailed prompts used in the multiple-prompt setting. Our prompt format is [Image of {class}.] J], where {class} represents the class name in the single-prompt setting. For the pungleppoompttper--ceeeeoo setting, we made minor modifications to the names of certain categories. For example, in COCO-Obj, we corrected the misspelling of hair drier hair dryer. In COCO-171, we removed ambiguous suffixes (e.g., -other, -stuff) and renamed classes like floor-wood to wooden floor. In the multiple-prompe variant, we additionally used synonyms, hyponyms, and/or plurals of the class names.
Table 13 shows the prompts used for PASCAL--21 in the multiple-prompt setting, while Table 12 presents the prompts used for COCO-Obj (80 classes) and COCO-171 (171 classes). The table includes both the 'things' classes, which are common in both datasets, and the 'stuff" classes, which are exclusive to COCO-171.
Note that the list. classes for the background adopted from the implementations of SCLJPP (Wang et al. 2024a and NACLIP (Hajimiri al. 2025 5 and used consistently for both the single and multi-prompt settings. For background class aggregation, in cross-attention-basel models like BLIP and ALBEF, sum the predictions of all background classes, whereas for LLM-based models like LLaVA, simply take the maximum prediction among the background classes.
fttp:///iaacaloooeaaaa///////aar object.
34

---

Published in Transactions on Machine Learning Research (02/2026)
Table 12: List of prompts for Single and Multi-Prompt Per Class Settings for COCO-Obj (Things Classes) and COCO-171 (Things + Stuff). The symbol for "background" indicates that additional prompts used to represent the back- ground classes for the single prompt setting, following (Wang et al, 2024a Hajimiri et al. 2025). The list of classes for the background borrowed from the implementation SCLIP (Wang et al. 2024a and NACLIP (Hajimiri et al. 2025). Our prompt takes the form [Image of {class}.1 J, where the {class) is given below. It. to be noted that COCO-171 does have any background class and the background prompts are used for COCO-Obj only.
Class Id and Prompt
Additional Prompts (Multiprooppl)
sky, wall, tree, wood, grass, road, sea, river, mountain, sands, desk, building, cloud, lamp, door, window, wardrobe, ceiling, shelf, curtain, stair, floor, hill, rail, fence. people, man, woman, child, children, boy, girl bicycles, bike cars.
Background
background"
1: person
2: bicycle
3: car
4: motorcycle
motorcycles, motorbike airplanes, aeroplane, aircraft buses, coach
5: airplane
bus
train
8: truck
trucks, lorry ship, boats, yacht, sailboat, speedboat
9: boat
10: traffic light
11: fire hydrant
12: stop sign
Things Classes
13: parking meter
14: bench
benches
15: bird
birds
16: cat
cats, kitten dogs, puppy
17: dog
18: horse
horses
19: sheep
20: cow
cows, cattle
21: elephant
elephants
22: bear
bears
23: zebra
zebras
24: giraffe
giraffes
25: backpack
backpacks
26: umbrella
parasol, umbrellas
27: handbag
28: tie
necktie
29: suitcase
30: frisbee
31: skis
ski
32: snowboard
anowboarda
33: sports ball
ball, sports balls
35

---

Published in Transactions on Machine Learning Research (02/2026)
Class Id and Prompt
Additional Prompts (Multi-prompp)
34: kite
kites
35: baseball bat
36: baseball glove
37: skateboard
skateboards
38: surfboard
surfboards
39: tennis racket
racket, tennis rackets, racquet bottles
40: bottle
41: wine glass
42: cup
cups
43: fork
forks
44: kmife
knives
45: spoon
46: bowl
dish
47: banana
bananas
48: apple
apples
49: sandwich
sandwiches
50: orange
oranges
51: broccoli
52: carrot
carrots
53: hotdog
hotdogs, sausage
54: pizza
Things Classes
55: donut
donuts
56: cake
cakes
57: chair
chairs
58: couch
sofa
59: potted plant
indoor plant
60: bed
61: dining table
62: toilet
63: tv
television, tvs, television screen
64: laptop
65: mouse
computer mouse
66: remote
remote control, remotes
67: keyboard
68: cell phone
cell phones, mobile phone
69: microwave
70: oven
71: toaster
72: sink
73: refrigerator
fridge
74: book
books
75: clock
76: vase
77: scissors
78: teddy bear
teddy
79: hair dryer
blow dryer
80: toothbrush
81: banner
82: blanket
83: branch
branches, tree branch
84: bridge
85: building
buildings
36

---

Published in Transactions on Machine Learning Research (02/2026)
Class Id and Prompt
Additional Prompts (Multi-prompt)
86: bush
bushes storage, wall cabinet
87: cabinet
88: cage
89: cardboard
90: carpet
91: ceiling
92: tile ceiling
93: cloth
94: clothes
95: clouds
96: counter
97: cupboard
98: curtain
99: desk
100: dirt
101: door
102: fence
103: marble floor
104: floor
105: stone floor
106: tiled floor
107: wooden floor
Stufis Classes
108: flower
109: fog
110: food
111: fruit
112: furniture
113: grass
114: gravel
115: ground
116: hill
117: house
118: leaves
119: light
120: mat
door mat
121: metal
metal surface, metallic object
122: mirror
123: moss
spores, mosses
124: mountain
125: mud
126: napkin
127: net
128: paper
129: pavement
sidewalk, footpath
130: pillow
131: plant
plants
132: plastic
133: platform
134: playing field
playground
135: railing
136: railroad
137: river
138: road
37

---

Published in Transactions on Machine Learning Research (02/2026)
Class Id and Prompt
Additional Prompts (Multi-prompt)
139: rock
140: roof
141: rug
142: salad
143: sand
144: sea
145: shelf
146: sky
147: skyscaper
148: snow
149: solid material
150: stairs
151: stone
Stufis Classes
152: straw
153: structure
154: table
155: tent
156: textile
157: towel
158: tree
159: vegetable
160: brick wall
161: concrete wall
162: wall
163: wall panel
164: stone wall
165: tiled wall
166: wooden wall
167: water
168: drops of water
169: window blind
170: window
171: wood
timber
38

---

Published in Transactions on Machine Learning Research (02/2026)
Class Id and Prompt
Additional Prompts
0: background"
sky, wall, tree, wood, grass, road, sea, river, mountain, sands, desk, building, cloud, lamp, door, window, wardrobe, ceiling, shelf, curtain, stair, floor, hill, rail, fence
1: airplane
aeroplane, jet, airplanes, plane, aeroplanes, jets, planes
2: bicycle
bicycles, bike, bikes
3: bird
birds
4: boat
boats, yacht, ship, ships, speedboat, speedboats,
yachts
5: bottle
bottles
6: bus
buses, coach, coaches
7: car
cars
8: cat
cats
9: chair
chairs, dining chair
10: cow
cows, cattle
11: dining table
dining tables dogs
12: dog
13: horse
horses
14: motorcycle
motorcycles, motorbike, motorbikes
15: person
people, man, woman, men, women, boys, girls, child, children, boy, person in shirt, person in jeans, person in dress, person in sweater, person in skirt, person in jacket
potted plants, indoor plants, house plants
16: potted plant
17: sheep
18: couch
sofa, couches trains, railcar, railcars television, television set, television monitor, tv monitor, monitor, television screen, TVs
19: train
20: tv
Table 13: List of prompts for Single and Multi-Prompt Per Class Settings for PASCAL-21. The symbol for 'background' indicates that additional prompts are used to represent the background classes for the single-prompt setting, following (aag et al. 2024a Hajimiri al. 2025). The list of background classes is borrowed from the implementations of SCLIP (Wang et al. 2024a and NACLIP (Hajimiri et al. 2025). Our prompt format is [Image of {class) J, where {class} is listed above.
39


Reviews:
[{"review_id": "xtWLRPDMD2", "reviewer": "Reviewer_guZd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. Proposal of InfoScore (**Entropy-based Layer Selection**): Unlike existing methods that rely on manual layer selection or ground-truth annotations , the paper proposes InfoScore, an entropy-based unsupervised metric. This metric enables the automatic selection of optimal attention layers in a training-free manner. The authors demonstrate that this approach achieves superior performance compared to naive selection strategies.\n\n2. False-Positive Filtering (**Class VLM Score**) Mechanism: The authors introduce a false-positive filtering mechanism that suppresses noise from classes not present in the image. This method utilizes the VLM's inherent probabilities, such as image-text matching scores or LLM token probabilities, without requiring external models or additional annotations.\n\n3. **Efficient One-shot Fine-tuning Strategy**: The paper proposes an efficient one-shot fine-tuning strategy that fine-tunes only the text embeddings and the top-K attention layers selected by InfoScore. This approach does not introduce additional decoders or parameters. The results show that performance is significantly improved using only a single visual example.\n\n4. **Generalizability across VLM Architectures (Model-Agnostic)**: The authors demonstrate the generalizability of their proposed framework by experimentally verifying its effectiveness across different VLM architectures. This includes both standard cross-attention-based VLMs (such as BLIP and ALBEF) and modern LLM-based VLMs (such as LLaVA).", "claims_and_evidence_yn": "Yes", "claims_explanation": "1. Proposal of InfoScore (Entropy-based Layer Selection)\nEvidence: The experiments empirically validate that the InfoScore-based selection is superior to other methods. As shown in Table 2 and Table 5, selecting the Top-2 layers using InfoScore consistently outperforms naive strategies (such as selecting the first/last layers) and random selection baselines across multiple datasets and VLM architectures. Furthermore, Figure 5 demonstrates that the layer rankings predicted by InfoScore align closely with the ground-truth mIoU performance, proving its reliability as an unsupervised metric.\n\n2. False-Positive Filtering (Class VLM Score) Mechanism\nEvidence: The effectiveness of the proposed filtering mechanism is clearly evidenced in the ablation study presented in Table 7. Removing the Class VLM Score results in a drastic performance drop (e.g., from 58.0% to 25.1% mIoU on PASCAL-21 for BLIP in the training-free setting). Qualitative results in Figure 7 further illustrate that without this scoring, the model suffers from severe under-segmentation by failing to suppress irrelevant classes, confirming the mechanism's critical role in reducing noise.\n\n3. Efficient One-shot Fine-tuning Strategy\nEvidence: The efficiency and effectiveness of the one-shot strategy are substantiated by the results in Table 1 and 9. As shown in Table 1, the method achieves a state-of-the-art mIoU of 70.1% on PASCAL-21, significantly outperforming the training-free baseline (60.2%). Additionally, Table 9 confirms that selectively fine-tuning only the top-2 attention layers and word embeddings yields superior performance compared to fine-tuning more parameters, highlighting the method's parameter efficiency.\n\n4. Generalizability across VLM Architectures (Model-Agnostic)\nEvidence: The model-agnostic nature of the framework is supported by Table 4, which shows consistent performance gains across diverse architectures, including cross-attention-based models (BLIP, ALBEF) and LLM-based models (LLaVA).\n\n**Concern1**: However, a significant limitation is that the method relies on explicit cross-attention maps (or self-attention with multimodal tokens). This design renders it inapplicable to independent two-tower architectures like **CLIP** (or newer variants like **SigLIP**), which do not utilize cross-attention layers for inference. This limits the framework's ability to leverage the rapid advancements and scalability of these powerful CLIP-style backbones.\n\n**Concern2**: While the paper compares the method against naive baselines (e.g., random selection) on BLIP but not on ALBEF and LLaVA. Extended experiments are needed to verify that the method consistently provides significant gains over naive sampling across all VLMs.", "audience_yn": "Yes", "audience_explanation": "Zero-shot segmentation is a field that has been consistently researched, and this paper proposes a simple and accessible method to improve the zero-shot segmentation.\n\nFurthermore, the authors have empirically verified their proposed contributions through extensive experiments.\n\nTherefore, relevant researchers in the community would likely be interested in this paper.", "requested_changes": "There are some typos. For example, \"off\" -> \"of\" The very last line in Page 11.", "broader_impact_concerns": "N/A", "additional_comments": "Please see the concerns in the above comments.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZYIflia2oM", "reviewer": "Reviewer_ZrbZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose InfoScore, a training-free entropy-based metric that automatically ranks and selects the most relevant image-text attention layers for open-vocabulary segmentation. The authors introduce a false-positive filtering mechanism that re-weights attention maps using the model's own image-text scores to reduce noise without external supervision. The paper presents a one-shot tuning strategy that selectively fine-tunes the identified attention layers using a single support image, significantly improving performance without adding complex decoders. Experiment results show that the proposed framework is model-agnostic and consistently outperforms naive selection strategies across diverse vision-language model architectures.", "claims_and_evidence_yn": "Yes", "claims_explanation": "- The authors addressed an important topic on how to do zero-shot and few-shot segmentation from vision-language models. Demonstrating that this is possible is an important empirical result.\n- In the paper, the models used for experiments are BLIP, LLaVA and ALBEF. These models are not the state-of-the-art vision-language models. Could the authors experiment with more recent multimodal vision-language models, like Qwen2/3-VL or Gemma 3.\n- I think analysis like Figure 5 is very important. I think it would be good to provide this analysis for more models and datasets. I am curious if there is any pattern from the selected layers, can this be used to help understand which layers in vision-language models are better at localization/segmentation ability.", "audience_yn": "Yes", "audience_explanation": "I think the research topic and question studied in this paper should be of interest to the TMLR community.", "requested_changes": "I think the authors need to provide more qualitative analysis in the experiments. Currently in the main paper, only Figure 7 illustrates the qualitative segmentation results. I think this kind of results is more intuitive and can help readers understand the proposed methods better. It would be good to supplement the paper with segmentation results on more diverse image datasets and compare with other state-of-the-art segmentation methods, e.g, SAM-3 [1].\n\n[1] SAM 3: Segment Anything with Concepts. 2025.", "broader_impact_concerns": "I do not have concerns on broader impact of this paper.", "additional_comments": "\"To compute the overall attention score for a prompt, we take the mean across the words in the prompt, resulting in an attention score of dimension P × P for that prompt. \" Why use mean attention score over the prompt here? I don't think this is well justified. Could the authors explain more here? I think using mean attention score is susceptible to the prompt format. For example, one could use \"Class *Something*\" instead of \"*Image of Class 1*\".", "extra_scores": {}, "extra_text": {}}, {"review_id": "lxVZJTjueI", "reviewer": "Reviewer_tJRW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors study the problem of obtaining training-free segmentation maps from attention layers in vision-language models (VLMs). To this end, the authors (i) introduce an entropy-based measure, called InfoScore, that can quantify the relevance of an attention layer for a class and (ii) propose selecting most informative layers (either in a training free fashion or for one-shot training) through their InfoScore measure. Experiments on several datasets suggest improvements over the baselines.\n\nStrengths:\n\n+ Training-free acquisition of segmentation maps is an important research direction in computer vision.\n+ The paper proposes a novel measure for quantifying the relevance of attention layers in a VLM.\n+ Positive results on several benchmarks.\n\n\nKey Weaknesses:\n- Some assumptions are not justified.\n- Experimental evaluation needs to be more comprehensive.", "claims_and_evidence_yn": "No", "claims_explanation": "6. Experimental evaluation should be strengthened:\n\n6.1. Comparison to other few-shot/one-shot methods is missing.\n\n6.2. Fig 4 caption: \"The metric nDCG@k (normalized discounted cumulative gain) measures the quality of the ranking for the top-k layers, where a perfect ranking compared to the groundtruth ordering yields a score of 1.0.\" => How do you know the ground truth ranking of the layers? This is not explained in the text either.\n\n6.3. The comparison with NACLIP is not fair since NACLIP uses CLIP whereas the proposed methods uses BLIP. It is important that the proposed method is evaluated and compared with other methods using CLIP as well.\n\n6.4. Table 4: How did you choose these VLMs?", "audience_yn": "Yes", "audience_explanation": "TMLR appears to have segmentation papers published before.", "requested_changes": "Weaknesses:\n\n1. The paper's methodology is difficult to follow owing to mostly verbal and informal (notation-less) explanations of the steps.\n\n2. Some comparisons or arguments are not substantiated:\n\n2.1. Figure 1 (right): \n\n2.1.1. What is the performance of existing layer selection methods? The comparison to random selection and including all layers shows an incomplete picture. \n2.1.2. What would be the performance of random selection and including all layers with one-shot tuning?\n\n3. It would be helpful to add a related work section on measuring segmentation map properties.\n\n4. The properties of the new measure should be justified by introducing desiderata for the new measure.\n\n5. The new measure is called a metric. Metricity requires satisfying certain properties. Please provide that these properties are satisfied or call it a measure.\n\n6. Experimental evaluation should be strengthened:\n\n6.1. Comparison to other few-shot/one-shot methods is missing.\n\n6.2. Fig 4 caption: \"The metric nDCG@k (normalized discounted cumulative gain) measures the quality of the ranking for the top-k layers, where a perfect ranking compared to the groundtruth ordering yields a score of 1.0.\" => How do you know the ground truth ranking of the layers? This is not explained in the text either.\n\n6.3. The comparison with NACLIP is not fair since NACLIP uses CLIP whereas the proposed methods uses BLIP. It is important that the proposed method is evaluated and compared with other methods using CLIP as well.\n\n6.4. Table 4: How did you choose these VLMs?\n\n\nMinor comments:\n- \"on foundation models Bommasani et al. (2021); Li et al. (2024)\" => If a citation is a part of the sentence (e.g., \"Bommasani et al. (2021) proposed a novel method..\"), then the citation should be included as you did. However, if the citation is not a part of the sentence, it should stay within parentheses. The correct way to write the selected text is: \"on foundation models (Bommasani et al., 2021; Li et al., 2024)\". One of the cite commands in Latex handles this correctly. Try cite, citep or citet.\n- Figure 1 (right): Please export/save your plots as scalable graphics, e.g. SVG, PDF, and include them as such in Latex.\n- Section 3.1: Start with the first paragraph, I would suggest introducing symbols for tensors and modules/models, e.g., query image, prompt, layer, attention score, words, class, ... In its current state, the method is very difficult to follow without proper notation.\n- Fig 2: Font sizes can be increased.\n- \"the probability that the pixel (x, y) belongs to class c\" => Previously you used x to represent a pixel and y to represent a label. Please make the notation more coherent.\n- Eq 4: The summations run over i and j but the subscripts run over x and y.\n- Eq 10: So, this reduces to \\sum_c sigma_c/mu_c, right?", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)