Improving Visual Discriminability of CLIP for Training-Free Open-Vocabulary Semantic Segmentation
Authors: Jinxin Zhou
Jiachen Jiang
Zhihui Zhu
Venue: Accepted by TMLR ()

Abstract: Extending CLIP models to semantic segmentation remains challenging due to the misalignment between their image-level pre-training objectives and the pixel-level visual understanding required for dense prediction. While prior efforts have achieved encouraging results by reorganizing the final layer and features, they often inherit the global alignment bias of preceding layers, leading to suboptimal segmentation performance. In this work, we propose LHT-CLIP, a novel training-free framework that systematically exploits the visual discriminability of CLIP across \emph{layer}, \emph{head}, and \emph{token} levels. Through comprehensive analysis, we reveal three key insights: (i) the final layers primarily strengthen image–text alignment with sacrifice of visual discriminability (e.g., last 3 layers in ViT-B/16 and 8 layers in ViT-L/14), partly due to the emergence of anomalous tokens; (ii) a subset of attention heads (e.g., 10 out of 144 in ViT-B/16) display consistently strong visual discriminability across datasets; (iii) abnormal tokens display sparse and consistent activation pattern compared to normal tokens. Based on these findings, we propose three complementary techniques: semantic-spatial reweighting, selective head enhancement, and abnormal token replacement to effectively restore visual discriminability and improve segmentation performance without any additional training, auxiliary pre-trained networks, or extensive hyperparameter tuning. Comprehensive experiments on eight widely used semantic segmentation benchmarks demonstrate that LHT-CLIP achieves substantial performance improvements across diverse scenarios, underscoring its effectiveness and practicality for real-world deployment.

Published in Transactions on Machine Learning Research (04/2026)
Improving Visual Discriminability of CLIP for Training-Free Open-Vocabulay Semantic Segmentation
Jinxin Zhou
zhou.3820@osu..d
Deportment of Computer Science The Ohio State University
Jiachen Jiang
jiang. Borng.28800oor edu
Deportment Computer Science The Ohio State University
Zhihui Zhu
zlu.34400osu.co edu
Department of Computer Science The Ohio State University
Reviewed on OpenReview: /ttps:/oopenws wwt.reveeppr.v.ee..om net/ forum? 93pNW3
Abstract
Extending CLIP models to semantic segmentation remains challenging due to the misalign- ment between their image-level pre-training objectives and the pixel-level visual understand- ing required for dense prediction. While prior efforts have achieved encouraging results by reorganizing the final layer and features, they often inherit the global alignment bias of pre- ceding layers, leading suboptimal segmentation performance. In this work, we propose LHT-CLIP, novel training-free framework that systematically exploits the visual discrim- inability of CLIP across layer, head, and loken levels. Through comprehensive analysis, we reveal three key insights: (i) the final layers primarily strengthen image text alignment with sacrifice of visual discriminability (e.g., last 3 layers in VīT-B/16 and 8 layers in ViT- L/14), partly due to the emergence of anomalous tokens; (ii) subset of attention heads (e-g,, 10 out of 144 in ViT-B/16) display consistently strong visual discriminability across datasets; (iii) abnormal tokens display sparse and consistent activation pattern compared to normal tokens. Based these findings, propose three complementary techniques: semantic-spatian reweighting, selective head enhancement, and abnormal token replacement to effectively restore visual discriminability and improve segmentation performance without any additional training, auxiliary pre-trained networks, or extensive hyperparameter tun- ing. Comprehensive experiments on eight widely used semantic segmentation benchmarks demonstrate that LHT-CL.PP achieves substantial performance improvements across diverse scenarios, underscoring effectiveness and practicality for real-world deployment. The source code is available https://jiixxizzie w.t.....t......... tthuu.io/LTH-CLIP.
1 Introduction
Recent advances in vision-language pretrained models, such as CLIP (Radford et al., 2021 have demon- strated remarkable generalization and open-vocabulary recognition capabilities at the image level, thereby opening up possibilitics for transferring image-text alignment to pixel-level tasks. Despite this progress, they often underperform in dense prediction tasks like semantic segmentation, primarily due to their limited capacity to localize fine-grained visual details [Rao et al. 2022 Wang et al. (024)). To address these limita- tions, several studics have incorporated trainable modules into CLIP, typically relying on additional forms of supervision such dense annotations for a restricted of categories (Xu et al.] 2022b a Xing et al.
2023
Cho al. 2024
Li et. al. 2024a b supplementary image-text, pairs (Cha al. 2023 Luo et al.

---

Published in Transactions on Machine Learning Research (04/2026)
2023a; Ren et al. 2023 Xu et al. 2023b Zhang et al. 2023 Although these approaches have demonstrated
improved segmentation performance, they incur significant computational and annotation costs. Further- more, the dependence on limited supervision undermines the generalizability of the model, making it prone to overfitting the training distribution.
These challenges have sparked increasing interest in training-free methods (Wang et al. 2024 Li et al. 2023 Zhou et al.
2022a Lan et al.]
2024a
Hajimiri et al.) 2025 Lan et al. 2024b] Shao et al. 2024
2024 which aim to adapt CLIP's pro-trained representations for semantic segmentation without additional training, while preserving its generalization capability. A key dif- ficulty in this direction is enhancing visual representations for accurate pixel-level predictions. For instance, MaskCLIP (Zhou et al. 2022a) computes similarity between key features in the final attention layer to enrich patch embeddings. SCLIP (Wang et al.) 2024) replaces the standard query-key attention with correlative self-attention (query-query and key-key). ClearCI.IP (Lan et al. 2024a) further removes residual connec- tions and discards the FFN in the final layer to reduce noise and improve spatial alignment. ResCLIP (Yang jet al. 2024) incorporates attention maps from earlier layers to refine final-layer atlention map. However, these methods largely focus on modifying the final-layer attention, often leading to suboptimal ambignous local relationships and noisy segmentation. To address spatial limitations, some approaches incorporate
Bousselham et al.
2024 Yang et al.,
features from auxiliary backbones such as DINO (Wysoczańska et al., 2024 Lan et al. 2024b). SAM (Lan
et al. 2024b Zhang et al. 2025 Shi et al., 2025), or diffusion models (Corradini et al. 2024 Sun et al. 20244
Zhang et al. 2025). ) While effective, these methods incur significant computational and memory overhead.
Motivated by these limitations, we begin with a layer-wise analysis of visual discriminability and text- semantic alignment within the CLIP vision encoder (see Section 2.2 for detailed definitions). As shown in Figure [ the final layers exhibit a clear trade-off: visual discriminability drops sharply while semantic alignment improves only marginally. To understand the cause of this phenomenon, we further examine internal token interactions and structural patterns across layers. Attention map visualizations reveal that abnormal tokens emerge in deeper layers, attracting disproportionately high attention from nearly all spatial positions. This behavior causes the majority of lokens to converge on a small subset, thus disrupting the spatial coherence. Further analysis reveals that these abnormal tokens have sparse, high-magnitude activa- tions that remain consistent across positions, layers, and samples. Complementary to prior assumptions that such tokens primally encode global semantic content, our findings suggest they act more bias components ofg global mean features, thereby facilitating alignment with text embeddings.
Based on the analysis, we propose LHT-CLIP, training- free framework that leverages the inherent properties of CLIP to enhance the visual discriminability while preserving semantic alignment. LHT comprises three complementary strategies: abnormal token replacement (ATR), spatial-semantic reweighting (SSR), and selective head enhancement (SHE). Specifically, ATR identifies abnormal tokens via sparsity threshold- ing and replaces them with neighboring tokens. SSR mitigates the degradation of visual discriminability in the final layers by upweighting residual pathways, thereby restoring balance between spatial coherence and semantic alignment. Finally, SHE further enhances visual discriminability by selectively aggregating fea- tures from high-discrininablliy attention heads, using them as soft pseudo-masks to refine output features. Experimental results show that LHT-CLIP consistently improves performance when integrated into diverse baselines, achieving significant improved results on eight benchmark datasets.
Comparison with previous work. While prior work (Shao et al., 2024 Bai et al. 2024 has similarly
identified the presence of anomalous tokens, our study provides a more comprehensive characterization of their distinguishing properties, demonstrating that these tokens elicit sparse, high-magnitude activations that remain consistent across positions, layers, and samples. Moreover, existing approaches rely sophis- ticated moomaly-detection techniques and require careful tuning of multiple hyperparameters, such as the
neighborhood radius and minimum cluster size in DBSCAN (Ester et al. 1996) as used in Shao et al. (2024) and the neighborhood size and contamination level in LOF (Breunig et al. (000) as used in Bai et al. (002
In contrast, our ATR method directly leverages the consistent property of sparsity in anomalous tokens, thereby reducing the need for intricate hyperparameter selection and showing hyperparameter robustness across diverse datasets, as shown in Table [ Furthermore, to improve open-vocabulary semantic segmen- tation performance, prior work has explored leveraging earlier layers, either by reorganizing the final-layer
2

---

Published in Transactions on Machine Learning Research (04/2026)
computation using attention maps derived from preceding layers Yang et al. 2024 or by designing multi- layer fusion strategies (Yang et al. 2024 Bai et al. 2024). However, these approaches leave the underlying inference behavior of earlier layers unchanged, which can lead to suboptimal results due to the degraded visual discriminability typically exhibited by intermediate representations. In contrast, our proposed SSR stralegy explicitly modifies the inference procedure by upweighting the residual components, thereby sub- stantially enhancing the visual discriminability of intermediate features, as illustrated in Figure Morcover, SSR is orthogonal to these prior methods and can be naburally integrated with them.
Contributions. Our contributions can be summarized as follows:
We conduct a throughout analysis of visual discriminability at the token, head, and layer levels.
We propose, novel training-free approach, terms LHT-CLIP. To the best of our knowledge, this is the first work to explicitly modify the inference procedure prior to the final layer, enabling improved spatial coherence without compromising semantic alignment.
The extensive experiment results on open-vocabulary semantic segmentation tasks consistently demon- strate the effectiveness of the proposed method.
2
Analysis of Visual Discriminability and Semantic Alignment
2.1 Preliminaries
CLIP employs a Vision Transformer (ViT) (Dosovisskiy et al. 2020 as its image encoder to generate visual representations that are aligned with correspouding textual descriptions. The vision encoder first tokenizes an imput image of size H x W 3 by dividing it into a grid of non-overlapping patches of size P P. yielding h H/P rows and dimensional embedding space, 2, RP, and augmented with positional embeddings. An additional learnable [CLS] loken is prepended to the sequence and is later used for image-level prediction. The resulting token sequence is denoted as X - xf...., .....,,%, 2hun] E g(+hh)xxD, This sequence is passed through a stack of L Transformer encoder layers, cach consisting of a multi-head self-attention (MSA) module followed by a forwadd network (FFN). Let LN(.) denotes layer normalization, the token representations are updated at each layer as follow:
W/P columns of patches. Each patch is then linearly projected into D-
X = X + (X'((N(('-))),,
(1)
x ='=X" X FFN(LN(XX)) )).
(2)
The CLIP model is originally trained large-scale image text pairs for open-vocabulary image recognition tasks. To extend it to semantic sogmentation, natural approach is to compute the similarity between the from the final Transformer layer and the textual embeddings of C' category names, denoted by t E RCXD This results in a patch-texs text similarity map of size hw x C. Denote te as the embedding of the e-th class name, the final segmentation prediction is obtained by applying an argmax operation over the class dimension of this similarity map, as follows:
visual tokens X L = jaf ...., x
(at te)
e(x:)) = arg max
(3)
ttll
Ideally, for effective semantic segmentation, the vision encoder should produce feature representations that satisfy two key properties:
Visual discriminability: token features should exhibit high internal consistency within the same se- mantic category while remaining clearly distinguishable from those of other categories, thereby enabling accurate and clean segmentation results.
Semantic alignment: token features should be well-aligned with their corresponding textual embeddings to enable semantically meaningful segmentation results.
3

---

Published in Transactions on Machine Learning Research (04/2026)
0.72
Shoal (lortty rebiity
3.74
Sonann akgiired
Sevorri akaperees
0.70
2
0.72
0.68
D2
8.70
Wusi
0.60
0.00
1
Lager 10
Lagar
Layar
(a) VOC (ViT-B)
(b) Context (ViT-B)
(c) ADE (ViT-B)
(d) COCO-Stuff (ViT-B)
Schanic Arginen
5o 7.750
baee mineally
Senviikk digmeet
0.75
0.2
0.70
50.00
5 33 35
10
0.00
31 13 15
33 15
23 80
31 33 29:22 23
(e) VOC (VTT-L)
(f) Context (VIT-L)
(8) ADE (VTT-L)
(h) COCO-Stuff (VTT-L)
Figure 1: Laye analysis of visual discriminability (blue) and semantic aligument (orange) within the CLIP vision encoders across different datasets. The final layer excluded from the analysis to avoid dis- crepancies caused by prior modifications to the last-layer different methods.
Beyond their importance in open-vocabulary semantic segmentation, these two properties are also highly rel- evant to the development of multimodal large language models. To preserve strong generalization capability, the vision encoder of CLIP is often directly employed to extract visual representations without additional training, which are then used as inputs to downstream language models, as exemplified by LLaVA (Liu et al. 2023
2024
2.2
Analysis of visual discriminability and semantic alignment
Measures of visual discriminability and semantic alignment. To quantify visual discriminability, we adopt the evaluation protocol proposed by Mukhoti et al. (2023.. Specifically, let x E R and x R denote the features of two image patches i and j extracted from the 1-th layer. Each feature vector is fz-orrmalize,, and their cosine similarity is computed to serve the prediction of binary classifier that determines whether the two patches belong to the same semantic calegory. Given the corresponding semantic labels t(x;) and t(æ;), the target label for classification is set to 1 if t(æ;) = t(æ,), and 0 otherwise. Performance on this binary classification task provides a mcasure of visual discriminability, since effective representations should yield high cosine similarity for patches from the same semantic class and low similarity otherwise. To evaluate semantic alignment, we extract the intermediate representations x E TR. from each individual visual token at layer I, and project them into the final visual text aligned space using the last ViT layer. Based on these projected features, semantic alignment is quantified as the average accuracy between predicted and ground-truth semantic labels, following Equation (3) To prevent contamination from extrancous semantic information and noisy integration during the final attention computation, we follow prior work Lan et al. 2024a; Zhou et al. 2022a) by replacing the attention matrix with an identity matrix and removing both the FFN and residual connections of the last ViT layer.
Sharp decline in visual discriminability with marginal gains in semantic alignment in the final layers. To analyze the layer-wise dynamics of visual discriminabilltty and semantic alignment, we investigate on four datasets: Pascal VOC (Everingham & Winn 2011), PASCAL Context (Mottaghi al., 2014), ADE20K (Zhou et al.] 2017), and COCO-Stuff (Caesar al.J 2018). For each dataset, evaluate the VIT-B/16 and VIT-L/14 variants of the CLIP vision encoder using 1,000 randomly selected training samples. As shown in Figure we observe a consistent pattern across datasets:
Visual discriminability follows an inverted U-shaped curve across layers: it increases in the early stages but declines sharply in the deeper layers. For instance, in the ViT-B/16 16 model, the last two layers preceding
4

---

Published in Transactions on Machine Learning Research (04/2026)
layer I
layer
layer 7
layer 9
layer 11
Figure 2: Abnormal token phenomenon in attention maps across different layers of the ViT-B/16 model used the CLIP vision encoder. Attention maps are computed with respect to specific visual token positions, denoted by x (e.g,, the "child" token in the top row and the "car" token in the bottom row). Representative abnormal tokens are highlighted with orange boxes.
the final layer exhibit pronounced reduction in visual discriminability, while in the VIT-L/114 model, similar decline is observed over the last seven layers.
Semanlic alignement exhibils an upprozimately monolonic increase across layers: improves substantially in the early stages but gradually saturates in the later layers, yielding only marginal gains thereafter.
These observations offer a nuanced understanding of why CLIP has proven effective for open-vocabulary semantic segmentation. In particular, the strong semantic alignment observed the final layers explains why prior work can leverage last-layer features for aligning visual tokens with textual categories. However, the significant decline in visual discriminability in last layers reveals key limitation as they may lack the fine-grained visual distinctions necessary for producing accurate and precise segmentation masks. In this work, aim to mitigate this limitation by proposing methods that jointly improve visual discriminability and preserve semantic aligmment. Before introducing approach, we first investigate the underlying causes of the decline in visual discriminability.
Emergence of abnormal tokens with high norms and sparse activations. To understand the cause of the sharp decline observed in the final layers, we analyse the attention maps among visual tokens across different layers. As shown in Figure 2 deeper layers exhibit small set of dominant tokens that receive disproportionately high attention from nearly all visual tokens, causing most tokens to focus on this subset, consistent with prior observations (Darcet et al. 2023 Shao et al. 2024). The presence of dominant tokens attracts visual tokens to become similar, thereby reducing visual discriminability and degrading segmentation result. To further characterize these dominant tokens, we compare their features with those of normal tokens. As illustrated in Figure 3 dominant tokens exhibit sparse activation patterns, with only a few channels maintaining high activation. To quantify this sparsity, we adopt the hoyer score (Hoyer 2004
vD
| hh
H(x)
VD 1 [0.1].
x
la
(4)
Let x E DRD denote the feature vector of the i-th token at layer 1. A higher score corresponds to sparser activations. We employ this metric to quantify sparsity and to visualize its distribution across layers and token positions in Figure 3(d). As shown in the figure, sparse high-norm abnormal tokens begin to emerge in the middle layers and persist at the same positions in subsequent layers.
A bnormal tokens encode global information while being dominated by a bias-like component.
To further explore the information encoded in abnormal tokens, analyze their pairwise cosine similarity
5

---

Published in Transactions on Machine Learning Research (04/2026)
Adterior Cuied
Addector Chirrel
(a) Attention Map
(b) Normal Token
(c) Abnormal Token
(d) Abmormal
Token
[e) Hoyer Map
Figure 3: Illustration of the sparsity and high-norm characteristics of abnormal tokens. Figure (a) shows the attention map of the red anchor token x. Figures (b) (d) depict the channel activations of a normal token (red x) ) and two abnormal tokens (orange and blue *) highlighted in Figure (a). Figure (e) presents the hoyer score distribution across layers and token positions.
with normal tokens and the [CLS] token on the same images from the ImageNet (Deng et al. 2009) validation set. As shown in Figure a). the cosine similarity with [CLS] increases while the similarity with normal tokens decreases as the layer depth increases, suggesting that abnormal tokens progressively align with global information and discard local details. Interestingly, when analyzing their pairwise cosine similarity across different positions, layers, and data samples, we find that these tokens exhibit consistently high similarity, with cosine values exceeding 0.98, as shown in Figure g Complementary to prior discovery, this indicates that abnormal tokens primarily act as bias components that offset global mean features, thereby facilitating text alignment in a manner analogous to the bias term in final-layer classifiers under neural collapse (Zhu et al. 2021 Zhou et al.) 2022b). However, both global information and bias-like components in abnormal tokens are detrimental for segmentation tasks, which rely on fine-gre ine-grained local understanding.
A subset of attention heads exhibits consistently strong visual discriminability. To enhance the visual discriminability of last-layer features, natural strategy is to exploit the more discriminative intermediate features to construct a pseudo mask, guiding the reorganization of the final-layer features, as indicated by the analysis in Figure [ Inspired by recent studics (Gandelsman et al. 2023 Kang et al.) 2025) showing that different attention heads capture distinct visual concepts, such as number, shape and texture, we take a step further by investigating whether specific heads are particularly responsible for encoding visual discriminability. To identify such heads, we follow the formulation introduced in Elhage al. (2021); Gandelsman et al. (2023), which rewrites the MSA output
summation over H independent attention = A V. W E R((+hw)xD, where A and VA, denote the attention and value matrices for the hith head layer and W is the output projection matrix shared across all heads. Accordingly, the features of the h-th head at layer I are expressed as:
heads: MSA(LN(X)))
(5) show the distribution of visual discriminability across attention heads for ViT-B/16. From the figure, we observe that the output features of certain attention heads, such the 11th head in the 6th layer, consistently exhibit high visual discriminability across different datasets, suggesting that there are a subset of heads which are more effective in capturing locally distinguishable features.
X - A W
In Figure 0
3 Method for Improving Visual Discriminability
In this section, we introduce our training-fres framework, which comprises three components: Abnormal Token Replacement (ATR) in Section 3.1 Spatial-Semantic Reweighting (SSR) in Section 3.2. and Selective
Head Enhancement (SHE) in Scction 3.3 Each component is complementary, and together they work
synergistically to enhance the visual discriminability of the CLIP model.
3.1 Abnormal token replacement (ATR)
Identify and replace abnormal tokens with their neighbors. To mitigate the adverse effects of anomalous tokens, we propose simple and effective strategy to suppress their impact. As shown in earlier

---

Published in Transactions on Machine Learning Research (04/2026)
10
lerma--oomal Tomes
Abeomaaltsme Rear
Costre Seatetty
0.5
Layer 1oles
Layer trdox
Layor mndox
(a) Inter-taken Similarity
(b) Inter-layer Similarity
(c) Inter-sample Similarity
Figure 4: Layer-wise cosine similarity among abnormal tokens across positions, layers and samples.
0.75
0.42
0.14
6.30
20.73
Voun Grormmen
0.71
4.40
7.70
0.72
0.70
10.35
5.24
0.73
2.03
0.70
0.33
CLP
0.00
0.60
01P
C.19
CLP
1338
1358
0.51
CLP
+504
CLP
1351
+558
CLP7
+350
6.20
0.87
Lager Indee
Layer In:ee
Lager Indes
Lager Indea
(a) VOC (VIT-L)
(b) Context (ViT-L)
(c) ADE (ViT-L)
(d) COCO-Stuff (ViT-L)
Figure 5: Layer-wise visual discriminability (blue) and semantic alignment (orange) for VIT-L/14 on four benchmarks. Solid lines denote the baseline CLIP (Raw) performance, while dashed lines indicate results after applying the proposed SSR strategy. The application of SSR notably improves visual discriminability in final layers and consistently enhances semantic alignment across all datasets.
analysis, these tokens characterized by high norms and sparse activations. To identify them systemati- cally, we compare norm-based and sparsity- based criterion. Empirically, we find that the sparsity- approach is more robust to hyperparameter selection (see Section A.3.3); (herefore, we adopt the hoyer score H(x{) defined before as a sparsity-based criterion. Tokens with scores exceeding a predefined threshold r are deemed anomalous and grouped into the set Ar = {i H(x)) > 7}. To mitigate their impact, each anomalous token at spatial position (m,n) n) E substituted with a weighted aggregation of its eight nearest neighbors:
Emtl Di=m- クe=n- (((ig) A)ai,,
2 m,
v(m,n) n) E.A
(6)
1 ンズー ((i,)) & A)
Here, I(-) ensure that only normal tokens contribute to the replacement of anomalous ones. Empirically, we find that applying this strategy before the penultimate layer leads to performance drop, likely due to the removal of inherent biases encoded in abnormal tokens, which substantially alters the inference process. Therefore, we apply it only at the penultimate layer, i.e, with 1 = I-11
3.2 Spatial-semantic reweighting (SSR)
Upweight residual component to improve visual discriminability. After mitigating the impact of anomalous tokens at the penultimate layer, the final features exhibit improved visual discriminability (see Figure 8 in Section A.2.2 However, critical challenge remains: anomalous tokens appeared in earlier layers alrcady degrade visual discriminability. limiting the effectiveness of final-layer refinements. To address it, we propose a spatial semantic reweighting stralegy, based on our layer-wise analysis showing that the final layers provide only marginal gains semantic alignment while substantially reducing visual discriminability. Overall, the proposed strategy aims to enhance the modcl's spatial coherence while preserving its semantic alignment. Specifically, given the feature representation X at the l-th layer within the final few layers (e.g., layers 10-11 in ViT-B/16 and layers 17-23 in VIT-I./14), we reweight the forward pass by upweighting
7

---

Published in Transactions on Machine Learning Research (04/2026)
Layen/leed
Lagate
(a) Context (VIT-B)
(b) ADE (VIT-B)
(c) COCO-Stuff (VIT-B)
Figure 6: Head-wise visual discriminability analysis across multiple datasets using the ViT-B/16 backbone. The dashed lines in different colors denote the corresponding layer-wise visual discriminability scores. For clarity, heads from three layers (i.e., the 6th, 7th, and 8th layers) are displayed.
the residual pathway and downwcighting the attention and MLP submodules, as follows:
x'= = (1 + + ( M))MA(()(("))))
(7)
x' = (1+a) X' + +(1 - (LL=)FFN(LN(("")
(8)
where a E [0, 1] a reweighting coefficient that controls the emphasis on the residual signal. As a increases, the I-th block preserves more visually discriminative features from earlier layers through the residual pathway, while reducing the adverse impact of noisy semantic aggregation in the MSA and FFN submodules. To the best of our knowledge, prior work has primarily focused on reforming the final layer modifying its representations to improve performance. However, such approaches inevitably inherit the global semantic alignment bias of the proceeding layers, leading to suboptimal segmentation due to substantially reduced visual discriminability. In contrast, our SSR strategy explicitly addresses this limitation by rebalancing residual and semantic contributions, thereby improving the visual discriminability of intermediate features.
Effectiveness of SSR in enhancing visual discriminability of preceding layers. Since our SSR strategy is applied not only to the penultimate layer but also to earlier layers, we further present the layer- wise curves of visual discriminability and semantic alignment after applying SSR Figure 5 As illustratod in the figure, the visual discriminability of the final few layers is significantly enhanced, demonstrating the effectiveness of the proposed SSR strategy for visual discriminability improvement. Moreover, semantic align- ment also exhibits consistent improvements, as improved visual discriminability reduces spurious semantic aggregation from other noisy tokens.
3.3 Selective head enhancement (SHE)
Construct pseudo masks from discriminative heads to further enhance penultimate features. Based on the head-wise analysis, we leverage high-performing heads to construct soft pseudo masks, which are used to enhance the visual discriminability of the penultimate features. Specifically, let VD denote the visual discriminability score of the h-th head in the 1-th layer for dataset E {Context, ADE, Stuff).
To obtain dataset- aggnssicc measurement, we compute the average score (described in Section 2.2 across datasets, denoted as VD,,A- We then rank heads by their VDL values (see Figure and Figure 10 and select the top-k heads form the set H4. The corresponding features are aggregated X 2(.)EH X
and a similarity map is constructed as S X X|| to caplure the pairwise similarity among visual tokens. To suppress the spurious interactions between tokens from different semantic categories, we apply a thresholding operation with a parameter B3, yiclding the filtered similarity map Sg, where Sg(i,) S(i, j) if S(i,j) 2 8, and (i. otherwise. This filtered similarity map S3 serves as soft pseudo-mask, which is column-normnllied and used to refine the final-layer features XL-1 Norm(Sg)
8

---

Published in Transactions on Machine Learning Research (04/2026)
4 Experiment Results
Evaluation Protocol. We follow the evaluation protocol from prior works (Wang et al. 2024 Lan et al.
2024a; Hajimiri et al. 2025 and assess our method on eight widely used semantic segmentation benchmarks. We group them into two categories and use abbreviated names for clarity. The first category excludes background and includes Pascal VOC (Everingham & Winn 2011) (VOC20), Pascal Context (Mottaghi
et al. 2014) (C59), COCO-Stuff (Caesar et al.) 2018) (Stuff), ADE20K (Zhou et al.) 2017) (ADE), and Cityscapes (Cordts et al., 2016) (City). The second includes background and consists of VOC21, C60, and COCO-Object (Caesar et al,, 2018) (Object). We use ViT-B/18 and ViT-L/14 as the vision encoder of CLIP Radford et al.) 2021), and report results using the mcan Intersection-over-Uu Union (mloU). All hyper- paramelers are tuned on 1,000 randomly sampled training images from (Context, ADE, Stuff) } datasets and kept fixed without task-specific tuning during evaluation, based on ablation study in Section 4.2 Additional
details are provided in the appendix.
4.1 Comparison with existing methods.
We compare our approach against a comprehensive set of open-vocabulary semantic sogmentation mothods, including the direct baseline CLIP (Radford et al.] 2021). as well as several state-of-the-arl training-Tres [ree
Li et al. 2023), SCLIP (Wang et al.) 2024). NA- CLIP Hajimiii ct al., 2025), , ClcarCLIP (Lan et al. 2024a LAVG (Kang & Cho. 2024), and ResCLIP (Yang)
approaches: MaskCLIP (Zhou et al. 2022a), CurgSyrgery
et al. 2024). We also inchude several influential weakly supervised methods, such as Group ViT (Xu et al,
2022a), ReCo (Shin et al., 2022), and TCT. (Cha et al. 2023). Unless otherwise specified, all reported results are taken directly from the respective original papers and ResCLIP (Yang et al.] 2024). As our method is orthogonal to approaches that primarily target improvements in the final-layer attention, we eval- uate its effectiveness when integrated with recent state-of-the-arl methods that employ specialized attention mochanisms in the last layer, including SCLIP (Wang et al.) (024)), ClearCLIP (Lan et al., 2024a). and ResCLIP (Yang et al., 2024). For fair comparison, we exclude the Semantic Feedback Refinement module in ResCLIP, as it relies on the computationally expensive PAMR (Araslanov & Roth 2020) post-processing, which is inconsistent with our evaluation setting.
In Table 1 we summarize the performance of various previous methods on benchmark datasets using the ViT-B backbone. Our proposed LHI-CLIP consistently enhances the performance of state-of-thh-art approaches, including SCLIP (Wang et al.] 2024), ClearCLIP (Lan et al. 2024a), and ResCLIP (Yang) ct al. 2024). Notably, when integrated with ResCLIP (Yang et al.) (2024). LHT-CLIP achieves significantly improved results, outperforming leading weakly supervised methods. As plug-and-play solution, LHT-- CLIP yields consistent improvements across all clatasets compared to the respective baselines, demonstrating its strong generalization capability. For comprehensiveness, results on the ViT-L/14 backbone are provided in Tableg In line with observations from Yang et al. (2024), existing methods generally exhibit a performance drop exceeding 2% mIoU when adapting to a different backbone; for instance, ClearCLII (Lan et al. (024a) suffers a notable decline of 2.7% mloU. In contrast, when augmented with LHT-CLIP, this performance degradation is significantly alleviated, highlighting the robustness of our approach. Across both backbones, LHT-CIIP delivers substantial improvements over baseline methods, validating its effectiveness.
4.2 Experimental analysis
In this section, we conduct comprehensive ablation studies to validate the effectiveness of our proposed method. We adopt SCLIP (Wang et al., 2024) as the baseline, which enhances spatial correlation by modi- fying the attention mechanism in the final layer, replacing the standard QKT attention with a combination
of QQ + KKT. In addition, following prior work (Lan et al. 2024a Yang et al. 2024), we remove the residual connections and FFN from the final transformer layer. For experiments in this part, we highlight
the optimal hyperparameter settings in gray
Analysis of the hoyer threshold parameter 7. Our method relies on hoyer sparsity to identify anoma- lous tokens, making the sparsity threshold a critical hyperparameter. In Table We conduct systematic evaluation. At 7 0.2, many normal tokens are misclassified, leading to excessive smoothing and degraded
9

---

Published in Transactions on Machine Learning Research (04/2026)
Table Performance comparison of our approach with other methods on eight semantic segmentation benchmarks. Existing methods with our improvement are marked in gray.
With background class
Without background class
Methods
Training
Avg.
VOC21
C60
Object
VOC20
City
C59
ADE
Stuff
ReCo
25.1
19.9
15.7
57.7
21.1
22.3
11.2
14.8
23.5
ViT
62.3
18.7
27.5
79.7
18.5
23.4
10.4
15.3
30.7
TCL
V
51.2
24.3
30.4
77.3
23.1
30.3
14.9
19.6
33.9
CLIP
16.2
7.7
5.5
41.8
5.5
9.2
2.1
4.4
11.6
MaskCLIP
38.8
23.6
20.6
74.9
16.4
26.4
9.8
14.8
28.2
CLIPSurgery
55.2
18.7
27.5
79.7
18.5
23.4
10.4
15.3
31.1
LaVG
62.1
31.6
82.5
26.2
15.8
23.2
38.8
NACLIP
58.9
32.2
79.7
35.2 35.5
23.3
39.4
SCLIP
X
59.7
31.7
33.5
81.3
32.3
34.5
16.5
22.7
39.1
+LHT-CLIP (ours)
64.8
34.8
36.6
86.3
36.1
37.6
18.0
24.9
42.4
(+3.3)
ClearCLIP
57.0
32.2
32.5
82.3
32.8
35.8
17.3
24.0
39.2
+LHT-CLIP (ours)
63.8
35.2
35.6
85.7
37.8
38.8
19.2
25.8
42.7
(43.5)
ResCLIP
60.0
32.7
34.0
85.5
35.6
35.8
17.7
23.8
40.6
+LHT-CLIP (ours)
63.9
35.5
86.9
38.2
19.1
25.5
(+2.2)
Table Study of hoyer sparsity threshold т.
Table Study of (Istarts lend, o:) in SSR module.
C60
Stuff
C59
ADE
Avg
(Istart, Lend,
C60
Stuff
C59
ADE
Avg
0.2
1.3
1.2
1.5
0.6
1.2
baseline
32.4
24.0
36.0
17.6
27.5
0.4
33.0
24.4
36.6
17.9
28.0
(9, 11, 0.1)
32.7
23.7
36.5
17.7
27.7
0.5
33.0
24.4
36.7
17.9
28.0
(10, 11, 0.1)
33.1
24.3
36.9
18.0
28.1
0.8
33.0
24.4
36.6
17.9
28.0
(11, 11, 0.1)
32.7
24.4
36.4
18.0
27.9
0.9
32.4
24.0
36.0
17.6
27.5
(10, 11, 0.05)
32.8
24.3
36.4
18.0
27.9
baseline
32.4
24.0
36.0
17.6
27.5
(10, 11, 0.2)
32.6
23.9
36.5
17.8
27.7
performance. As F increases 0.4, performance steadily improves, but plateaus between 0.5 and 0.8, with a decline observed beyond this range. The broad stable region indicates clear sparsity gap between normal and abnormal tokens, highlighting the robustness of ATR to threshold selection. Based on this analysis, we fix T= = 0.5 for all experiments.
Analysis spatial-semantic rewcighting paramcters a and number of layers. To evaluate the impact of the reweighting strength a and the range of layers involved, from lstart to lend, we perform a comprehensive sensitivity analysis. The results are summarized in Table 3. We observe that the best performance is obtained when reweighting is applied to layers 10 11 the VTT-B/16 backbone. This aligns with our carlier findings that these layers experience marked decline in visual discriminability while yiclding only marginal improvements in semantic alignment. Extending reweighting to include layer 9 results in a slight gain in visual discriminability but introduces noisy semantic signals, ultimately leading to reduction in segmentation performance. In addition, we examine the effect of varying the reweighting threshold parameter a. As a increases from 0 to 0.1, performance improves steadily, indicating a beneficial balance between visual and semantic cues. However, further increasing a leads a performance drop, as it incorporates inaccurate semanttc information from earlier layers and significantly perturbs the input distribution of subsequent layers.
Analysis of the number of selected heads k. In Table 4 we study the effect of varying the number of top-A attention heads selected for enhancement. For ViT-B/16 backbone, increasing k from 5 to 10 improves sogmentation accuracy, as aggregating multiple visual discriminative heads helps suppress spurious correlations. However, performance declines when becomes too large due to the inclusion of noisy or less informative heads, which introduce noisy cross-category interactions. We also compare head- and layer- level selection (best = 8), finding that head-level selection consistently performs better, as discriminative heads are distributed across layers, while entire-layer selection introduces irrelevant heads and degrades performance.
10

---

Published in Transactions on Machine Learning Research (04/2026)
Table 4: Study of number of selected heads k.
Table 5: Combination of three strategies.
k
C60
Stuff
C59
ADE
Avg
Module
mloU
baseline
32.4
24.0
36.0
17.6
27.5
ATR
SSR
SHE
layer(1 = 8)
32.8
24.8
36.4
17.7
28.0
baseline
27.5
k 5
32.8
24.6
36.3
17.8
27.9
4
28.6
11.1
k 10
33.0
25.1
37.0
18.3
28.4
28.9
+1.4
k 30
33.0
25.0
37.1
18.2
28.3
4
28.9
+1.4
k 50
32.8
25.0
37.0
18.1
28.2
4
29.2
+1.7
Image
GT
SCLIP
+LHT-CLIP
ClearCLIP
+LHT-CLIP
Figure 7: Qualitative comparison between CLIP-based training-fres free bascline methods and their counterparts integrated with LHT-CLIP.
Study of combination effect. In prior parts, we show the effectiveness of each individual strategy in Ta- ble 2 Table and Table g The Table 5 further presents results of different combinations under the same settings. The result highlights the complementary contributions of each strategy to the overall segmentation performance, yielding average 1.7 mloU improvement on these four datasets.
4.3 Qualitative results
In Figure 7 we present a qualitative comparison between CLIP-based training-(ree baseline methods and their counterparts integrated with LHT-CLIP. As shown in the figure, the integration of LHT-CLIP consistently leads to accurate and visually coherent segmentation results. The enhanced models produce cleaner segmentation maps with reduced noise, improved spatial consistency among same-category objects, and better delineation of object boundaries, all while preserving the original text-image alignment capabilities of the baseline methods.
4.4 Applicability beyond CLIP
11

---

Published in Transactions on Machine Learning Research (04/2026)
Table 6: Performance of the ViT-B variant as the SigLIP vision encoder. The best results are obtained when combined with our proposed LHT-CLL method, highlighted in gray
Method
VOC20
City
C59
ADE
Stuff
Avg
SigLIP
48.0
20.5
18.6
11.5
12.1
22.1
SigLIP+ ClearCLP
5.7
2.9
0.6
2.3
59.1
23.2
23.1
13.8
15.3
26.9
Table 7: Efficiency comparison of individual strategies.
Models
FLOPs(G) 4 Params(M)
Speed
CLIP
106.10
149.6
13.7
ResCLIP
141.34
149.6
3.0
baseline
100.70
149.6
13.9
+XTR
100.88
149.6
12.9
+SSR
100.88
149.6
11.7
+SHE
102.65
149.6
8.2
To evaluate the generalizability of our approach beyond CLIP, we further assess it on the SigLIP model (Zaii et al. 2023). We adopt the ViT-B variant as the SigLIP vision encoder and conduct experiments on five datasots without a background category, as summarized in Table ( Similar to CLIP, the vanilla SigLIP model performs poorly, achieving an average score of only 22.1. Notably, prior training-free methods such as ClearCLIP (Lan et al. 2024a] perform even worse than vanilla SigLIP. We attribute this to architectural differences between CLIP and SigLIP, in particular the use of an AttentionPool projector in SigLIP instead of linear projection. In this setting, modifying the final transformer layer significantly disrupt the input distribution to the projector and degrade performance. We therefore adopt vanilla. SigLIP as the baseline.
In contrast to prior approaches that focus exclusively on modifying the final transformer layer, our method is guided by a fine-gratd -raigra analysis visual discriminability and semantic alignment across token, head, and layer levels. Rather than restricting improvements to the last stage, we introduce targeted modifications at carlier layers of the model, Icading to more robust and generalizable inference. Despite the architectural and oraining-objective differences, LHT-CLIP consistently improves open-vocabulary segmentation performance SigLIP without modifying the model introducing additional training. As shown in Table 6 our approach delivers notable gain of 4.8 points in average segmentation performance, improving from 22.1 to 26.9. This result demonstrates that our LTP-CLIP framework is not unique to CLIP, but reflects more general properties of ViT-based languaeee models and captures model-agnostic structural phenomena rather than CLIP-specific artifacts. Importantly, the same hyperparameter settings (e.g., sparsity threshold 7, SSR layer range and coefficient (Astart a), number of selocted heads k, and similarity threshold /) are reused when transferring from CLIP SigLIP, without task- or model-specific tuning.
4.5 Efficiency Comparison
In Table we report efficiency results on the Context59 dataset using an A5000 GPU with the ViT-B/16 backbone, taking ClearCLIP as the baseline. The results demonstrate that the inference-time overhead introduced by LHT-CLIP is modest. As shown in Table 11, integrating all three components (ATR, SSR, and SHE) Icads to only a minor increase in inference latency compared to standard CLIP-based basclines, while remaining substantially more efficient than the prior slate-of-the-arl ResCLIP. Notably, by removing the computationally expensive PAMR post-processingg step used in ResCLIP's Semantic Feedback Refinement module, LHT-CLIP simultancously improves sogmentation performance and reduces computational cost. As a result, the inference speed increases from 3.0 FPS to 8.2 FPS, and the total computational cost is reduced from 141 44G to 102.65G FLOOPs.
12

---

Published in Transactions on Machine Learning Research (04/2026)
5 Conclusion
In this paper, we present comprehensive analysis of the visual discriminability of pretrained CLIP mod- els at the token, layer, and head levels. Based on the analysis, we introduce LHT-CLIP, training-free framework that enhances visual discriminability while preserving semantic alignment. LHT-CLIP comprises three complementary components: (1) abnormal token replacement, (2) spatial-semantic reweighting, and (3) selective head enhancement. Our approach provides lightweight, plug-and-play modules compatible with existing architectures. Extensive experiments on multiple segmentation benchmarks show the effectiveness of LHT-CLIP outperforms strong baselines. Furthermore, since CLIP vision encoders are often frozen in MLLM training, our findings offer practical guidance for improving visual understanding in broader MLLMs.
Acknowledgment
The authors would like to acknowledge support from NSF grants IIS-2312840 and IIS-2402952.
References
Nikita Araslanov and Stefan Roth. Single-stage semantic segmentation from image labels. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 4253-4253 2020.
Sule Bai, Yong Liu, Yifei Han, Haoji Zhang, and Yansong Tang. Self-calibrated clip for training-free open- vocabulary segmentation ion. Xrxiv preprint or arxiv:241 11.1111.188 2024.
Danicl Bolya, Po-Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Rasheed, et al. Perception encoder: The best visual embeddings are not at the output of the network. arXiv preprint arXiv:2504.13188 2025.
Walid Bousselham, Felix Petersen, Vittorio Ferrari, and Hilde Kuehne, Grounding everything: Emerging localization properties in vision-langunge transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3928-3837, 2024.
Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jörg Sander. Lof: identifying density-based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pp. 93 104, 2000.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1209-1215 2018,
Junbum Cha, Jonghwan Mun, and Byungseok Roh. Learning to generate text-grounded mask for open- world semanntc segmentation from only image-text pairs. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recogrition, pp. 11165 11174, 2023.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitscv. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 2818 2829, 2023.
Scokju Cho, Heeseong Shin, Sunghwan Hong, Amurag Arnab, Paul Hongsuck Sco, and Seungryong Kim. Cat-seg: Cost aggregation for open-vocabulary semantic segmentation. Tn Proceedings of the IEFE/CVP Conference on Computer Vision and Pattern Recognition, pp. 4113 4123, 2024.
MMSSegmentation Contributors, MMSSggeenattin Openmmlab semantic segmentation toolbox and bench- 2020.
mark. http:///gg.htaa.ccco/o//aaaaaaaa
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiclo. The cityscapes datasct for semantic urban scene understand- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-30 12233,
2016.
13

---

Published in Transactions on Machine Learning Research (04/2026)
Barbara Toniella Corradini, Mustafa Shukor, Paul Couairon, Guillaume Couairon, Franco Scarselli, and Matthieu Cord. Freeseg-diff: Training-free open-vocabulary segmentation with diffusion models. ArXin, abs/ abs/2243.20115, 2024. URL
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint ar Xit:2309 16588, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. Ieee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ArXiu, abs/2010. 11929 2020. URL https://api semanticschollar /tt.atps:///pp...mmaniiiiasaatie htt.a/aar.r.rrrrrraai:0:22250000
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Trans- former Circuits Thread, 11)):22, 2021.
Martin Ester, Hans- Peter Kriegel, Jörg Sander, Xiaowei Xu, el al. A density-based algorithm for discovering clusters in large spatial databases with noise. In kdd, volume 96, pp. 226-231, 1996.
Mark Everingham and John Winn. The pascal visual object classes challenge 2012 (voc2012) development kit. Pattern Analysis, Statistical Modelling and Computational Learning, Tech. Rep, 8(5):2 5, 2011.
Yossi Gandelsman, Alexei A Efros, and Jacob Steinhardt. Interpreting clip's 's image representation via text- based decomposition. arXiv preprint ar arXiw:2310.0591 2023.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, (32(():581155)) 595, 2024.
Sina. Hajimiri, Ismail Ben Ayed, and Jose Dolz. Pay attention to your neighbours: Training-free open- vocabulary semantic segmentation. In 2025 IEEE/CVE Winter Conference Applications of Computer Vision (WACV), pp. 5061-5077 IEEE, 2025.
Patrik O Hoyer. Non-negative matrix factorization with sparseness constraints. Journal of machine learning research, 5(Nov):1457 1469, 2004.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pp. 4904-4916 PMLR, 2021.
Dahyun Kang and Minsu Cho. In defense of lazy visual grounding for open-vocabulary semantic segmenta- tion. In European Conference on Computer Vision, pp. 144-164. Springer, 2024.
Seil Kang, Jinyeong Kim, Junhyeok Kim, and Scong Jae Hwang. Your large vasion-language model only needs few attention heads for visual grounding. arXiv preprint arXiv:2503.062: 2025.
Philipp Krähenbühl and Vladlen Koltum. Efficient inference in fully connected crfs with gaussian edge potentials. Advances in newral information processing systems, 24, 2011.
Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang, Clearelip: De- composing clip representations for dense vision-lanne langual inference. In European Conference on Computer Vision, pp. 143-160. Springer, 2024a.
Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Proxyclip: Proxy attention improves clip for open-vocabulary segmentation. In European Conference on Computer Vision, pp. 70 88. Springer, 2024b.
14

---

Published in Transactions on Machine Learning Research (04/2026)
Yi Li, Hualiang Wang, Yiqun Duan, and Xiaomeng Li. Clip surgery for better explainability with enhance- ment in open-vocabtlary tasks. ar arXiv e-prinls, pp. arXiv-2304, 2023.
Yongkang Li, Tianheng Cheng, Bin Feng, Wenyu Liu, and Xinggang Wang. Mask-adapter: The devil is in the masks for open-vocabulary segmentation. arXiv preprint or 0rXi:2412.04553) 2024a.
Yunheng Li, ZhongYu Li, Quansheng Zeng, Qibin Hou, and Ming-Ming Cheng. Cascade-clip: Cas-
caded vision-language embeddings alignment for zero-shot semantic segmentation. arXiv preprint arXiv:2406.00 00670, 2024b.
Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip, In Proceedings of the IEEE/CVP Conference Computer Vision and Pattern Recognition (CVPR), pp. 7061 7070, June 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 26:34492-34416, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pottern Recognition, pp. 26296- 26306, 2024.
Huaishao Luo, Junwei Bao, Youzheng Wu, Xiaodong He, and Tianrui Li. Segelip: Patch aggregation with learnable centers for open-vocabulary semantic segmentation. In International Conference on Machine Learning, pp. 23033-23044. PMLR, 2023a.
Jiayun Luo, Siddhesh Khandelwal, Leonid Sigal, and Boyang Albert Li. Emergent open-vocabulary semantic segmentation from off-the-shelf vision-language models. 2024 /EFE/CVF Conference on Computer Vision URL https://api semanticscholar. org/
and Pattern Recognition (CVPR), pp. 4029 4040, 2023b.
Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whie Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 891-898, 2014.
Jishnu Mukhoti, Tsung- Yu Lin, Omid Poursaeed, Rui Wang, Ashish Shah, Philip HS Torr, and Ser-Nam Lim. Open vocabulary semantic segmentation with patch aligned contrastive learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 19413-19423, 2023.
Reza Qorbani, Gianluca Villani, Theodoros Panagiotakoponlos, Mare Botet Colomer, Limus Harenstam- Nielsen, Mattia Segu, Pier Luigi Dovesi, Jussi Karlgren, Daniel Cremers, Federico Tombari, and Matteo Poggi. Semantic library adaptation: Lora retrieval and fusion for open-vocabulary semantic segmentation. Arxiv,
2025. URL https://api iimmanttsccholarr. org///oosssssss::07770999999
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkinn Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PmLR, 2021.
Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guider dense prediction with context-aware prompting. In Proceedings of the IEEE/CVE conference on computer vision and patlern recognition, pp. 18082-18000 2022.
Pengzhen Ren, Changlin Li, Hang Xu, Yi Zhu, Guangrun Wang, Jianzhuang Liu, Xiaojun Chang, and Xiao- dan Liang. Viewco: Discovering text-supervised segmentation masks via multi-view semantic consistency. arXiv preprint ar Xic:2302. 10307, 2023.
15

---

Published in Transactions on Machine Learning Research (04/2026)
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural information processing systems, 2022.
Tong Shao, Zhuotao Tian, Hang Zhao, and Jingyong Su. Explore the potential of clip for training-free open vocabulary semantic segmentation. In European Conference on Compuler Vision, pp. 139 156. Springer, 2024.
Yuheng Shi, Minjing Dong, and Chang Xu. Harnessing vision foundation models for high-performance, training- free open vocabulary segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23487-23497, 2025.
Gyungin Shin, Weidi Xie, and Samuel Albanie. Reco: Retrieve and co-segment for zero-shot transfer. Advances in Newral Information Processing Systems, 35:33754 33767, 2022.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In European conference on computer vision, pp. 746-760. Springer, 2012.
Julio Silva-Rodriguez, Sina Hajimiri, Ismail Ben Ayed, and Jose Dolz. A closer look at the few-shot adaptation of large vision-language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pullern Recognilion, pp. (1881-23690, 2024.
Lin Sun, Jiale Cao, Jin Xie, Xiaoheng Jiang, and Yanwei Pang. Cliper: Hierarchically improving spatial representation of clip for open- vocabulary semantic segmentation. arvvv preprint arXtv:8411.18888 2024.
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Vl-adapter: Parameter-effiient transfer learning for vision-and language tasks. In Proceedings of the IEFE/CVE conference on computer vision and pattern recognition, pp. 5227 5237, 2022.
Feng Wang, Jieru Mei, and Alan Yuille. Sclip: Rethinking self-attention for dense vision- langual inference. In European Conference on Computer Vision, pp. 315-332 Springer, 2024.
Monika Wysoczaíska, Oriane Siméoni, Michaäl Ramamonjisoa, Andrei Bursuc, Tomasz Trzciński, and Patrick Pérez. Clip-dinoiser: Teaching clip a few dino tricks for open-vocabulary semantic segmenta- tion. In European Conference on Compuler Vision, pp. 320 337. Springer, 2024.
Yun Xing, Jian Kang, Aoran Xiao, Jiahao Nie, Ling Shao, and Shijian Lu. Rewrite caption semantics: Bridging semantic gaps for language-supervieed semantic segmentation. Advonces in Neurol Information Processing Systems, 36:68798-68800 2023.
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonnin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVE con- ference on computer vision and pattern recognition, pp. 18134-18144, 2022a.
Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. 2023 IEEE/CVF Conference on Computer Vision and Pallern Recognilion (CVPR), pp. 2955 2966, 2023a. URL https://app.iii semanticcchonnlc.rr.r........
org/CorpusID: : 257405338
Jilan Xu, Junlin Hou, Yuejie Zhang, Rui Feng, Yi Wang, Yu Qiao, and Weidi Xie. Learning open-vocabulary semantic segmentation models from natural language supervision. In Proceedings of the IEFF/CVF con- ference on computer vision and pattern recognition, pp. 2935 2944, 2023b.
Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model. In European Conference on Computer Vision, pp. 736 236-753. Springer, 2022b.
Yuhang Yang, Jinhong Deng, Wen Li, and Lixin Duan. Resclip: Residual attention for training-free dense vision-language inference. Xin preprint ar 11. 15.1551,, 2024.
16

---

Published in Transactions on Machine Learning Research (04/2026)
Tao Yu, Zhihe Lu, Xin Jin, Zhibo Chen, and Xinchao Wang. Task residual for tuning vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Poltern Recognilion, pp- 10899 10909, 2023.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre- training. In Proceedings of the IEEE/CVP international conference computer vision, pp. 11975-11986, 2023.
Dengke Zhang, Fagui Liu, and Quan Tang. Correlip: Reconstructing patch correlations in clip for open- vocabulary semantic segmentation, ion. In Proceedings of the IEEE/CVP International Conference Com- puter Vision, pp. 24677-24688 2025.
Fei Zhang, Tianfei Zhou, Boyang Li, Hao He, Chaofan Ma, Tiajjiao Zhang, Jiangchao Yao, Ya Zhang, and Yanfeng Wang. Uncovering prototypical knowlodge for weakly open-vocabulary semantic segmentation. Advances in Newral Information Processing Systems, 36:73652 73665, 2023.
Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hong- sheng Li. Tip-adapter: Training-fres clip-adapter for better vision-language modeling, orXiv preprint arXiv:2111.0300 2021.
Renrui Zhang, Ziyao Zeng, Ziyu Guo, and Yafeng Li. Can language understand depth? In Proceedings of the 30th ACM International Conference on Multimedia, pp. 6868 6874, 2022.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 633-641, 2017.
Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference on Computer Vision, pp. 196-712. Springer, 2022a.
Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, and Zhihui Zhu, On the optimization landscape of neural collapse under mse loss: Global optimality with unconstrained features. In International Conference on Machine Learning, pp. 27179-27202. PMLR, 2022b.
Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu A geomet- ric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 34:29820 298344, 2021.
17

---

Published in Transactions on Machine Learning Research (04/2026)
A Appendix
In the appendix, we first provide additional related literature in Section A.1 We then present extended experimental details and results for both the ViT-B/16 and ViT-L/14 vision encoders in Section A.2 For each model, we report the distribution of average head visual discriminability scores across multiple datasets and conduct a component-wist analysis of their contributions to visual discriminability. In Section A.3 we address key concerns, including hyperparameter selection, applicability beyond CLIP, and a comparison between SSR and direct skip methods.
A.1 Additional literature
Vision-language pre-training models. Deep learning is experiencing significant paradigm shift driven with the emergence of large-scale vision-lagguage models. Among these, the Contrastive Language- Image Pre-Training (CLIP) framework (Radford et al. 2021) has achieved remarkable success, largely attributed to its strong zero-shot and few-shot generalization capabilities in visual recognition tasks, particularly image classification. Building on these strengths, an expanding body of research has sought to further improve CLIP's zero-shot performance through enhanced training on large-scale image-text pairs (Jia et al.) 2021
Cherti et al. 2023 Schuhmann et al., 2022 Zhai et al.] 2023) or by enabling its efficient adaptation novel
downstream tasks using limited labeled data (Gao et al. 2024 Silva-Rodriguez et al. 2024 Sung et al. 2022 Yu et al. 2023 Zhang et al. 2021). Nevertheless, as CLIP is pre-trained predominantly al the image level, its learned representations, particularly the [CLS] token, are optimized to capture global semantics. This coarse-grained supervision inhorently limits their utility in dense prodiction tasks, where fino-grained spatial localization is critical.
Open-vocabulary Semantic Segmentation. Driven by the remarkable generalization ion capabilities of
large-scale vision-language models (Radford et al. 2021
Jia et al.
2021 Cherti et al.
2023
Schuhmann
ct al.
2022 growing line of research has focused on open-vocabulary semantic segmentation (Ren et al.
2023
Liang et al. 2023 Luo et al.] 2023a b Shin et al.) 2022 Xu et al. 2023b a Li al. 2024a Qorbani
et al. 2025), which seeks to extend global eross-modal alignment. to fine-grained, pixel-level predictions.
Existing approaches can be broadly categorized into three groups: fully supervised, weakly supervised, and
training-free methods, based on the level of auxiliary supervision required for adaptation. Fully-superoised
approaches (Liang et al. 2023 Luo et al. 2023b Li et. al. 2024a Qorbani et al. 2025) adapt pretrained CLIP models to semantic segmentation by leveraging large-scale datasets with dense pixel-wise annotations for a predefined, yet limited, set of categories. Although such supervision facilitates the learning of fine-grained spatial representations, these methods often struggle to generalize to novel categories. Furthermore, their reliance labor-intensive, densely annotated datasets poses significant challenges to scalability, limiting their applicability in real-world scenarios. Instead of requiring pixel-wise annotations, weakly-supereee approaches leverage auxiliary datasets with image-level annotations to adapt pre-trained CLIP models for semantic segmentation. These approaches commonly employ large-scale image-text corpora, wherein textual descriptions explicitly reference the object categorics present in the corresponding images. Adaptation is achieved by enforcing cross-modal aligument through a contrastive loss (Xu et al., 2022a], analogous to the original CLIP pre-trainnn pre-training objective. Although such strategies alleviate the burden of dense supervision, they still rely on access to large-scale annotated datasets. However, the auxiliary datasets are substantially smaller than those employed during CLIP pre-training, which limits the generalization capability. Additionally, these methods presuppose prior knowledge of the categories present in each image, thereby constraining their applicability in genuinely open-world segmentation scenarios.
Training-fit free ooca-voca semantic segmentation. Training-fres methods explore the feasibility of employing frozen CLIP models to generate segmentation masks in the absence of additional data for adap- tation. Some approaches rely on auxiliary models pre-trained on large-scale datasels such as DINO (Wysozza- īiska et al. 2024 Lan et al. 2024b), SAM (Zhang et al.) 2025). or Stable Diffusion (Corradini et al. 2024 Sun et al., 2224), which incur significant computational and memory overhead. An alternative line of rescarch soeks to enhance CLIP's capability for dense visual representation by modifying the inference pipeline of its visual encoder. For instance, MaskCT.IP (Zhou et al.] 2022a) removes the self-attention module in the
18

---

Published in Transactions on Machine Learning Research (04/2026)
final Transformer layer and demonstrates that the resulting value embeddings encode local visual features that align effectively with textual prompts. CLIP-Surgery (Li et al. 2023) introduces a dual-path structure that replaces the original self-attention with value-value attention to better preserve semantic consistency, mitigating the tendency to attend to unrelated regions. Building on this idca, GEM (Bousselham et all, 2024) and SCLIP (Wang el al.] 2024) generalize the approach by incorporating correlative sell-attention mechanisms, such as query-query and key-key interactions. ClearCLIP (Lan et al., 2024a) further simplifies the architecture by removing the final feed-forward layer and associated residual connections, which were found to contribute to noisy segmentation outputs. Additionally, NACLIP (Hajimiri et al. 2025) imposes explicit spatial regularization, encouraging each token to primarily attend to its neighbors to improve the
spatial coherence.
Taking into account both generalization ability and computational costs, this work aims to advance existing training-free methods for open-vocabulary semantic segmentation. While prior training-free approaches prodominantly rely on features extracted from the final layer of the visual encoder, we instead conduct a comprehensive analysis across loken, head, and layer levels. Building on this multi-level analysis, we propose recipe at each level to enhance the spatial representation capacity of CLIP, thereby improving segmentation performance within a training-froe framework.
A.2
Additional experimental results
A.2.1 Additional Implementation Details.
We adopt CLIP (Radford et al. 2021 with VIT-B/16 and ViT-L/14 backbones, implemented in MMSeg- mentation (Contributors) 2020). Following the protocol of Yang et al. (2024), input images are resized to 336 pixels on the shorter side, except for Cityscapes, which is resized to 560 pixels to accommodate higher resolution. Inference is conducted with a sliding-window strategy using 224 x 224 crops and a stride of 112 pixels. Consistent with TCL (Cha et al., 2023), we avoid computationally intensive post-processing methods that hinder fair comparison, such as PAMR (Araslanov & Roth 2020) (used in TCL (Cha et al., 2023 NACLIP (Hajimiri et al., 2025)) and DenseCRF' (Krähenbühl & Koltun 2011) (used in ReCo (Shin) ct al. 2022)) ). For textual inputs, we employ the standard ImageNet prompts (Radford et al., 2021) without
additional prompting strategies. Based on the ablation study in Section 4 and Section A.2.4 the following
configuration is fixed across all dataset evaluations without task-specific tuning
VIT-B/16: The hoyer threshold is set to T = 0.5, and the reweighting operation is applied to layers [10, 11] (i.e., the two layers preceding the final one), with a reweighting coefficient of 0.1. The top-10 attention heads with the highest visual discrininability are selected using a filter threshold of 8 0.7, corresponding to the following (layer, head) pairs: (8,9), (8,8), (7,10), (9,12), (7,3), (9.4), (5.1), (9.6). (4,11), and (8,6).
ViT-L/14: The hoyer threshold is set to 0.4 and the reweighting is applied to layers [17, [1] with coefficient of 0.1. Top-perforning heads are selected using the same threshold B = 0.7, with the top-30 (layer, head) pairs identified as: (11, 3), (9, 3), (7, 9), (11, 6), (10, 10), (9, 13), (3, 10), (4, 14), (10, 6), (6, 9), (7, 12), (14, 16), (11, 8), (10, 13), (8, 4), (8, 8), (10, 8), (9, 4), (2, 11), (9, 6), (8, 1), (14, 1), (16, 2). (4, 13), (13, 11), (11, 14). (7, 4), (14, 11), (13, 13), and (3, 13).
All experiments are conducted using eight NVIDIA RTX A5000 GPUs, each with 24 GB of memory.
A.2.2 Effects of individual components effects on visual discriminability
Complementary and effective for visual discriminability. The ablation study in Section the effectiveness of each individual strategy for segmentation performance. As shown in Figure g further analyze their contributions to visual discriminability by examining how progressively integrating these strate- gies improves the quality of the final representations. Similar to the definition of visual discrininability we treat the token-wine wise similarity score as the output of a binary classifier predicting whether two tokens belong to the same category, and plot the ROC curves of penultimatc-layer features. A higher area under the curve (AUC) indicates stronger visual discriminability. The results show that each strategy individually improves AUC, while their combination yields substantial gains in visual discriminability. For the ViT-B/ /16 backbone,
1 confirms
19

---

Published in Transactions on Machine Learning Research (04/2026)
1.0
1.0
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.6
Base (AUC- - 0.73501
Base (AUC 0.7436)
Base (AUC- - 0.7205)
Base (AUC 0.7278
0.2
ATR (AUC- 0.7973)
0.2
ATR (ALIC- - 0.7809)
0.2
ATR (AUIC 0.76231
02
ATR (ALIC 0.7697)
+55R 0.8026)
(AUC 0.7002)
+55R 0.7720)
+SSR 0.7789)
0.0
+HE (AUC- 0.8320)
0.0
+SHE (AUC- 0.8270
0.0
+SHE IAUC 0.8282)
0.0
+SHE (AUC 0.8275)
02
0.6
10
0.0
0.4
0.6
0.8
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
FPR
FPR
FPR
FPR
(a) VOC (ViT-B)
(b) Context (ViT-B)
(c) ADE (VIT-B)
(d) Stuff (ViT-B)
1.0
0.8
0.8
0.8
0.8
0.6
0.5
0.6
0.6
g8
g
Base (AUC- - 0.70811
Ease (AUC 0.6574)
Ease (AUC- - 0.63831
Ease (AUC - 0.5447)
0.2
+ATR (AUC= 0.7254)
0.2
+ATR (AUC = 0.6717)
0.2
+ATA (AUC- 65281
0.2
+ATR (AUC 05650))
+55R (AUC 0.7662)
+55R (AUC 0.7325)
+S5R 0.7245)
+55R (AUC 0.7238)
0.0
SHE [AUCC- n.8460)
0.0
15HE (AUC- 0.8126)
0.0
+SHE (AUC 0.8208)
0.0
+SHE (AUC 0.8:74)
0.2
0.6
10
0.0
0.4
0.3
0.4 0.6
1.0
0.2
0.6
1.0
FPR
FPR
FPR
FPR
(e) VOC (VTT-L)
(f) Context (ViT-L)
(x) ADE (ViT-L)
(b) Stuff (VIT-L)
Figure 8: ROC curves of the penultimate-layer output features for VIT-B/16 and ViT-L/14 backbones across four datasets. Each curve corresponds to an incremental combination of strategics. The results demonstrate consistent improvements in AUC with each added strategy, culminating in significantly higher visual discriminability when all are applied.
AUC increases from 0.7560 to 0,8320 VOC, 0.7406 to 0,8270 Context, 0.7205 to 0.8282 on ADE, and 0.7278 to 0.8175 on COCO-Stuff. Similarly, for VIT-I//14, AUC rises from 0.7081 to 0.8460 on VOC, 0.6574 to 0.8126 on Context, 0.6383 to 0.8208 on ADE, and 0.6447 to 0.8174 on COCO-Stuff. These improvements highlight the complementary nature of the proposed strategies and their collective effectiveness in enhancing visual discriminability across diverse benchmarks.
A.2.3 Head AUC distributions
In Section we presented the distribution of subset of head-level visual discriminability scores across mul- tiple datasets, employing ViT-B/16 as the visual encoder. In practice, we observed that the discriminability of head features is also degraded by the presence of abnormal tokens in the final layers. To fully realize the potential of head features, therefore, we apply abnormal token replacement to head features. Similarly, we consistently find that cortain heads exhibit significantly higher visual discriminability than others. Since we select the top-k heads based on their average visual discriminability across multiple datasets, we additionally present the distribution of average visual discriminability scores for all heads in descending order in Figure g for ViT-B/IL 16 and Figure 10 for ViT-L/14, to facilitate reference and reuse in future research.
A.2.4 Additional experiments on VIT-L/14
In Table 8 we present detailed comparison of open-vocabulary semantic segmentation models on multiple benchmarks using the ViT-L/14 backbone. LHT-CLIP consistently improves the performance of lcading methods, including ClearCLIP (Lan et. al.) 2024a). NACLIP (Hajimiri et al.] 2025), ; and ResCLIP (Yang) et al. )024). In particular, integrating LHT-- CLIP with ResCLIP achieves ofate--f-thheart results. As a plug- and-play module, LHT-CLIP provides consistent gains across all datasots, highlighting its robustness and generalization. As noted in Yang et al. (2024), many methods suffer performance drops exceeding 2% mloU when switching from ViT-B/16 to VIT-L//14; for example, ClearCT.IP declines by 2.7%. In contrast, L.TT-
20

---

Published in Transactions on Machine Learning Research (04/2026)
0.8
0.7
Vigual Discriminabliy
0.6
0.5
0.4
0.3
0.2
0.1
0.0
8-9
0 8
7-10
9-12
7-
9-6
4:11
8-6
7-31
3-8
6-1
8.8
9.5
5:5
B-4
6:11
8:12
9-10
5.8
9.8
$100
6:2
3-1
2-6
10:3
8-10
7-4
9.9
233
7.9
7-1
6-4
10.5
1:2
6-
10-3
7-6
10.8
10-11
5:11
10:3
75
6-10
2-11
11-10
10-7
11-4
11:3
4-8
5-7
11.9
6.5
10-4
4-4
11-7
11:1
4.2
6-7
6.3
5.9
Layer-Head
(a) Heads 1-66
0.7
Visual Discriminabiity
0.2
0.0
11-12
11.2
11:11
6-6
3-7
9-7
8-3
8-11
11-6
9.2
10:12
3:2
1-7
3-3
3.9
4-12
3.11
4.6
3.17
22
9:11
11-5
6:17
4-1
1-6
5F2
2:10
10:10
244
7-12
8-7
2-8
1-4
2-5
1-12
3-
1:10
1.9
3-4
12
2-7
5:12
1-1
1.5
7-7
1-8
8-1
2.1
4.10
5:3
Layer-Head
(b) Heads 67-172
Figure 9: Distribution of average visual discriminabllity score for individual attention heads across three datasets (Context, ADE, and Stuff) using the ViT-B/16 model, with the heads from final layer excluded. Bars are sorted in desconding order of average visual discriminability.
CLIP largely mitigates this degradation, demonstrating strong cross- backbone robustness. These results further validate its effectiveness in enhancing open-vocabulary segmentation across both ViT-B/16 and ViT- L/14 architectures.
Additionally, in Table 9 Table 10 Table and Table 12 we present detailed analyses of the hoyer threshold parameter, the spatial-scmantic rewcighting configuration, the number of selected attention heads, and the ablation analysis of individual components, respectively, for the ViT-L/14 model. Consistent with the ViT-B/16 results, the optimal configuration achieved with a moderate sparsity level (7 = 0.4), a reweighting coefficient a = 0.1 applied to layers 17 23, and the top-k heads with = 30. Therefore, we adopt these settings as fixed hyperparameters for ViT-L/14 across all benchmarks, without any dataset- specific tuning. Moreover, the ablation study confirms the effectiveness of each individual component and their complementary effect. As shown in Table I2 their combination leads to substantial improvement of 5.1 mloU, achieving final score of 30.1 mloU across the four benchmark datasets.
A.3 Further Discussion
A.3.1 Hyperparameter Selection
While LHT-CLIP introduces several hyperparameters, these are guided by universal patterns consistently observed across datasots, such as the sparsity of abnormal tokens, the tradc-off between visual discriminabil- ity and semantic alignment, and the presence of shared visually discriminative heads. Building on these consistent phenomena, emphasize three key points:
21

---

Published in Transactions on Machine Learning Research (04/2026)
Table 8: Performance comparison of our approach with other methods on eight semantic segmentation Our results are marked in gray.
benchmarks following the evaluation protocol in Section
With background class
Without background class
Methods
Training
Avg.
VOC21
C60
Object
VOC20
City
C59
ADE Stuff
CLIP
9.8
4.2
3.9
4.0
5.6
MaskCLIP
24.4
10.0
9.9
30.0
11.
12.6
7.8
10.1
CLIPSurgery
X
47.7
27.2
27.5
80.5
32.1
31.5
16.4
17.3
35.0
SCLIP
X
44.4
22.3
24.9
60.3
32.2
20.5
7.1
13.1
28.1
ClearCIIP
48.7
28.3
29.7
80.0
27.9
29.6
15.0
19.9
34.9
+LHT-CLIP (ours)
62.8
33.9
33.5
40.5
39.0
42.7
(+7.8)
NACLIP (Hajimiri et 2025
52.2
28.7
30.4
78.7
31.4
32.1
17.3
21.4
36.5
+LHT-CLIP (ours)
63.4
34.0
33.4
84.6
40.8
39.0
20.5
25.9
42.7
(46.2)
ResCLIP Yang al.
2024
55.7
29.7
30.8
81.3
32.8
33.6
17.9
22.9
38.1
LHT-CLIP (ours)
X
63.4
34.0
33.4
85.9
40.9
39.0
20.5
26.0
42.9 (14.8)
Table 9: Study of hoyer sparsity threshold
Table 10: Study of (7start, lend; Q) in SSR module.
C60
Stuff
C59
ADE
Avg
(latert, lend,
C60
Stuff
C59
ADE
Avg
T = 0.2
2.4
2.8
3.8
2.3
2.8
baseline
28.6
21.7
32.5
17.3
25.0
T = 0.4
29.7
22.5
33.6
17.8
25.9
(14, 23, 0.1)
30.8
21.2
33.9
16.9
25.7
T = 0.5
29.2
22.2
33.3
17.6
25.6
(17, 23, 0.1)
31.4
23.1
34.7
17.8
26.8
= 0.8
28.9
22.0
33.0
17.4
25.3
(19, 23, 0.1)
30.9
23.0
34.2
17.9
26.5
0.9
28.9
22.0
33.0
17.4
25.3
(17, 23, 0.05)
30.3
22.5
33.6
17.5
26.0
baseline
28.6
21.7
32.5
17.3
25.0
(17, 23, 0.2)
31.6
22.1
34.8
17.8
26.6
1. Generalizable. The hyperparameters are generalizable across datasets. In practice, we found that values selected on a small subset of training data (e.g., 1,000 randomly selected training samples from {Context, ADE, COCO-Stuff)) also perform well on other datasets, indicating that they are not overfitted to specific datasets. As shown in Section 4.1 and Section A.2.4 we apply the same hyperparameters to all datasets without specific tuning, underscoring their strong generalizability.
2. Robustness.
The method is robust to a wide range of hyperparameter values, which means that precise tuning is not required to achieve strong performance. This is demonstrated in our ablation studies Sec- tion and Section A.2.4 where the results remain consistent in various hyperparameter settings.
3. Reliable initialization. Additionally, our analysis provides interpretable heuristics for selecting hyper- parameters. For example, the case of an SSR starting layer, one can use the observed turning point in visual discriminabilityy semantic alignment a reliable initialization.
Taken together, while several hyperparameters are introduced, we believe these properties support the prac- tical usability of our method without requiring extensive or fragile hyperparamcter tuning.
A.3.2 Comparison between SSR and direct skip methods
Recent work on the Perception Encoder (Bolya et al. 2025) highlights the effectiveness of directly selecting an earlier layer's output for segmentation. This approach conceptually related to our semantic reweighting (SSR) stratogy, which emphasizes residual pathways by amplifying their contribution to coun- teract the dominance of deeper layers. In Table 13 present systematic comparison between SSR method and the direct-skip strategy, using ViT-L/Ie the CLIP vision encoder and adopting ClearCLIP as the baseline.
Although directly leveraging earlier layer outpuls provides some improvement over baseline CLIP perfor- mance, our experiments show that SSR consistently achieves superior results. For example, when augmented with ATR and SHE, directly skipping layers 20-23 of the VIT-L encoder and fooding the output of layer into layer 24 yields an average performance of 40.8, higher than the bascline ClearCLIP score of 34.9, but still below the performance of our full LHT-CLIPP method (42.7), as reported in Table
22

---

Published in Transactions on Machine Learning Research (04/2026)
Table 11: Study of number of selected heads
Table 12: Combination of three strategies.
b C60
Stuff
C59
ADE
Avg
Methods
Module
mIoU
baseline
28.6
21.7
32.5
17.3
25.0
ATR
SSR
SHE
k =
32.0
24.3
35.0
18.6
27.5
bascline
25.0
k 10
32.8
24.6
36.0
19.2
28.2
28.1
131
30
33.1
24.9
36.3
19.5
28.5
4 29.2
+4.2
60
33.1
24.9
36.3
19.5
28.5
4
29.0
+4.0
k 100
33.0
24.8
36.4
19.5
28.4
Ours
30.1
+5.1
Table 13: Comparison between SSR and direct skip methods using the ClearCLIP as the baseline method and ViT-L./14 as the CLIP vision encoder. The best results are obtained when combined with our proposed LHT-CLIP method, as highlighted in gray
Method
VOC21
C60
Obj
VOC20
City
C59
ADE
Stuff
Avg
ClearCLIP
48.7
28.3
29.7
80.0
27.9
29.6
15.0
19.9
34.9
60.3
33.7
32.1
77.3
40.8
38.8
20.1
23.7
40.8
ClearCLIP + LHT-CLIP
62.8
33.9
33.5
85.9
40.5
39.0
20.3
26.1
42.7
A.3.3 Comparison of Norm-Based and Sparsity-
Sarsitty-Based Based Criteria for ATR
We regard ATR flexible framework that functions by detecting and replacing emergent abnormal tokens. As discussed in Scction [ these tokens exhibit two distinctive characteristics: unusually large activation magnitudes and sparse activation patterns. To evaluate the effectiveness of magnitude-based criterion for abnormal token detection, we conduct additional experiments using ViT-B/16 as the vision encoder, with ClearCLIP as the baseline method.
The norm-based approach identifies abnormal patches by computing the activation norm of each visual patch, classifying those above threshold its best average performance of 27.9 across the four datasets used in Table [4] when the threshold is set to = 14, which is comparable to the performance of our sparsity-based ATR method (28.0). However, the norm-based method is more sensitive to threshold selection. For example, the overall performance drops to 26.8 when & = 12 and 27.6 when 7 = 16, whercas the sparsity-based approach remains more stable under hyperparameter variations in Table
abnormal. Empirically, find that this method achieves
A.3.4 Extension to Monocular Depth Estimation
While this work primarily focuses on training-fre open- vpen-vocabulary semantic segmentation, we further inves- tigate the generality of our findings by extending LHT-CLIP to monocular depth estimation. In particular, we conduct additional experiments based on DepthCLIP (Zhang el. al. 2022) training-Tree framework that adapts CLIP representations for depth prediction. This extension allows us examine whether our analysis of visual discriminability and the proposed components remain effective beyond semantic segmentation.
Local and Global Roles of LHT-CLIP Components. From methodological perspective, the three components of LHT-CLIP can be categorized according to their operational scope:
Table 14: Study of norm threshold 7. The best results and corresponding & are highlighted gray
r
C60
Stuff
C59
ADF
Avg
7
31.7
16.9
26.8
7
32.7
23.9
17.5
27.7
7
32.8
24.2
17.8
27.9
7
32.6
24.3
36.7
17.8
27.8
7 == 16
32.2
24.1
36.6
17.6
27.6
haseline
32.4
24.0
36.0
17.6
27.5
23

---

Published in Transactions on Machine Learning Research (04/2026)
Table 15: Effect of individual LHT-CLIP components on monocular depth estimation performance. Red
indicates improvement over the baseline, and blue indicates degradation.
Method
< 1.25
1.25
< 1.25
rel.
logio
rimee
DepthCLIP
0.373
0.672
0.850
0.367
0.160
1.191
DeptaCLIP+AIR
0.378
0.675
0.861
0.375
0.159
1.182
SSP
0.346
0.640
0,833
0.353
0.169
1.246
0.383
0.675
0.850
0.405
0.159
1.171
DepthCLIP SHE( 0.5)
0.383
0,664
0,838
0.445
0.161
1.175
DepthCLIP / 0.9)
0.384
0.675
0,851
0.404
0.159
1.170
(1). Abnormal Token Replacement (ATR) acts as a local mechanism by replacing anomalous tokens through neighborhood aggregation, thereby enhancing spatial coherence.
(2). Spatial-Semantic Reweighting (SSR) functions as a global mechanism that improves visual discrim- inability by rebalancing semantic and visual signals across layers.
(3). Selective Head Enhancement (SHE) can operate in either a local or global manner depending on the similarity threshold. A larger threshold emphasizes localized, while a smaller threshold enforces broader semantic coherence across spatially distant regions.
consistency,
This categorization provides a useful lens for understanding how different dense prediction tasks benefit from local versus global mechanisms.
Task-Specific Effects on Depth Estimation. To evaluate our method on monocular depth estimation,
we fellow the default settings of DepthCT.I (Zhang et al. 2022 on NYU Depth v2 dataset. (Silberman ct al. 2012) and adopt ViT FYT/B-16 model as the vision encoder. We follow the same evaluation protocol of absolute relative error (rel), root mean squared error (rmse), absolute error in log space (log10), and threshold accuracy (8.).
DepthCLIP (Zhang al. 2022) and assess performance using four standard metries:
These metrics are defined as follows:
U/
dpl
rel
(9)
V/
p
rmse
(Up 0p)
(10)
log10
log (Up)) log1o
(11)
Up 0p
8 percentage of pixels p satisfying max
thr,
(12)
fp yo
where thr (1.25, 1.252, 1.253) and n denotes the number of valid pixels.
As shown in Table 15 monocular depth estimation, similar to semantic segmentation, requires spatially co- herent and locally discriminative visual features across both foreground and background regions. Consistent with this requirement, ATR and SHE with large similarity threshold (e.g., 0.9), both operating primarily at local level, lead to clear improvements in depth prediction quality across most evaluation metrics, with the exception of the mean absolute relative error (rel). These gains stem from enhanced pixel-level visual discriminability and improved spatial consistency.
However, depth estimation differs fundamentally from semantic segmentation its reliance on global visual discriminability. While semantic segmentation benefits from globally consistent features that enable uniform
24

---

Published in Transactions on Machine Learning Research (04/2026)
labeling of spatially separated regions belonging to the same object category, monocular depth estimation depends more heavily on local geometric continuity. Regions that share similar semantic appearance may exhibit significantly different depth values. For example, multiple desks in a classroom may belong to the same semantic category but lie at different distances from the camera. As a result, enforcing strong Teature similarity across distant regions, via global mechanismn such as SSR SHE with a smaller similarity threshold (e.g., 0.5), can be detrimental for depth estimation, as may oversmooth depth variations that are geometrically meaningful.
Conclusion. These findings highlight that the effoctiveness of local and global mechanisms is task- dependent. While global semantic consisteney is crucial for semantic segmentation, depth estimation benefits more from localized discriminability and spatial contimuity. Importantly, the observed behaviors are consis- tent with our core analysis of visual discriminability at the token, head, and layer levels, further validating the generality of our findings beyond a single dense prediction task. This provides additional insight into how different dense vision lasks selectively benefit. from local and global representations.
A.3.5 Discussion of Failure Case:
In Figure our qualitative results highlight typical improvements in spatial coherence and boundary quality, We acknowledge that LHT-CLIP may still struggle in certain scenarios, including (i) small or fine-grained objects, where individual patch predictions are dominated by majority voting (e-g., the green car in the ground-truth segmentation map shown in the fourth row of Figure 7 and (ii) inherently high semantic ambiguity, where CLIP representations ure which exhibits semantic ambiguity and is misclassified into other categories). Importantly, such failure modes are not unique to our method but are shared by most training-free free CLIP-based segmentation approaches that rely on frozen representations.
insufficiently discriminative (e.g., the black ground region in Fig-
25

---

Published in Transactions on Machine Learning Research (04/2026)
0.8
Visual Discriminabiii lity
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
Layer-Head
(a) Heads 1-88
0.7
0.6
Visual Discriminebini
0.5
0.3
0.2
0.1
Layer-Head
(b) Heads 89-176
0.7
0.6
Visual Discliinability
0.5
0.4
0.3
0.1
0.0
Layer Head
(c) Heads 177-264
0.6
Visual Discriminabiiity
0.5
0.4
0.3
0.1
0.0
490000
Layer Head
(d) Heads 265-352
Figure 10: Distribution of average visual discriminability score for individual attention heads across three datasets (Context, ADE, and Stuff) using the ViT-L/14 model, with the heads from final layer excluded. Bars are sorted in descending order of average visual discriminability.
26


Reviews:
[{"review_id": "vVCFguKYNC", "reviewer": "Reviewer_DSCb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores a fundamental limitation of CLIP-based training-free open-vocabulary semantic segmentation, i.e., although deeper layers improve image–text semantic alignment, they significantly degrade patch-level visual discriminability, thus leading to degradation on open-vocabulary semantic segmentation. Through a systematic analysis on token, head and layer perspectives, the authors present several key observations: (i) the sparse and high-norm abnormal tokens the  dominate attention and reduce spatial coherence, and (2) a small subset of attention heads that consistently exhibit strong visual discriminability across.\nBased on these observations, the paper proposes LHT-CLIP, a new training-free and plug-and-play framework consisting of Abnormal Token Replacement (ATR), Spatial-Semantic Reweighting (SSR), and Selective Head Enhancement (SHE). Extensive experiments on eight standard benchmarks show that LHT-CLIP consistently improves multiple state-of-the-art training-free methods (e.g., SCLIP, ClearCLIP, ResCLIP) across backbones, demonstrating strong generalization ability of the proposed approach.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes. \n- The authors observe that features in deep layers of CLIP significantly degrade patch-level visual discriminability, thus leading to degradation on open-vocabulary semantic segmentation;\n- The authors analyze the problem in token, head and layer perspectives, and present several key observations;\n- Based on the observation, the authors propose a new training-free and plug-and-play framework consisting of Abnormal Token Replacement (ATR), Spatial-Semantic Reweighting (SSR), and Selective Head Enhancement (SHE), which can generally improve multiple CLIP-based models;", "audience_yn": "Yes", "audience_explanation": "Yes. The paper provides a detailed analysis of visual discriminability in CLIP-based models and proposes training-free techniques that are relevant to researchers working on representation learning and open-vocabulary segmentation.", "requested_changes": "Weakness & Questions:\n- Lack of failure case analysis. The paper focuses primarily on showing successful scenarios and quantitative performance comparison. A discussion of failure cases would provide more insights for future work;\n- Although the proposed method is positioned as training-free, its inference-time computational overhead is not analyzed or justified. Specifically, LHT-CLIP introduces multiple additional operations during inference, including layer-wise re-weighted forward passes (SSR), abnormal token detection and neighborhood-based replacement (ATR), as well as head-wise feature aggregation and token–token similarity computation for pseudo-mask construction (SHE). In particular, the SHE module requires computing pairwise token similarities, which scales quadratically with the number of visual tokens and may become prohibitively expensive for high-resolution inputs. Moreover, applying SSR across multiple late layers further increases inference latency compared to standard CLIP-based baselines. However, the paper does not report any runtime, FLOPs, or latency comparisons against existing training-free methods. As a result, it is unclear whether the proposed approach maintains the efficiency advantages, especially in large-scale or real-time deployment scenarios;\n- The proposed framework is extensively validated on CLIP-based models, but it remains unclear whether the key observations, such as the emergence of sparse abnormal tokens, the optimal residual reweighting strategy, and the existence of consistently discriminative attention heads, are effective for other vision–language or vision-only models (e.g., SigLIP, EVA-CLIP, DINOv3). Since several components rely on empirically chosen hyperparameters (e.g., sparsity threshold τ, SSR layer range and coefficient α, number of selected heads k, and similarity threshold β), it is uncertain whether these settings can be transferred directly to other architectures or would require nontrivial re-tuning. This raises concerns about whether the method captures a general property of pre-trained vision-language models or is specifically designed to CLIP’s characteristics;\n\nOverall, I think this is a solid paper with interesting findings and effective solutions. The main concerns lie in the lack of failure case analysis, the insufficient discussion of inference-time computational overhead, and the unclear generality of the proposed method beyond CLIP-based models.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Dgq4FPvECo", "reviewer": "Reviewer_gXWC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides a systematic analysis of CLIP’s limitations in dense prediction tasks, revealing that degraded visual discriminability and semantic misalignment significantly hinder segmentation performance.\n\nTo address these issues, the paper introduces LHT-CLIP, a training-free and lightweight framework built on three complementary strategies:\n\n(1) Abnormal Token Replacement (ATR): mitigates token degradation by substituting abnormal tokens with spatially neighboring ones\n\n(2) Spatial-Semantic Reweighting (SSR): adaptively balances spatial and semantic cues to enhance visual discriminability\n\n(3) Selective Head Enhancement (SHE): exploits the most effective attention heads to refine feature representations for segmentation.\n\nExtensive experiments on eight benchmark datasets demonstrate that LHT-CLIP consistently delivers substantial performance gains and effectively boosts existing commonly used baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are supported by thorough empirical evaluations across multiple benchmark datasets, clear ablation studies validating each proposed component, and both quantitative and qualitative results that convincingly demonstrate performance gains and improved segmentation quality.", "audience_yn": "Yes", "audience_explanation": "The paper addresses fundamental limitations of vision–language models and proposes a simple, training-free solution that is broadly applicable, making its findings relevant and interesting to a substantial portion of the TMLR audience working on representation learning, multimodal models, and dense prediction tasks.", "requested_changes": "1.**Semantic Alignment.** For semantic alignment evaluation, intermediate visual tokens from layer $l$ are projected using the final ViT layer. However, the final layer is trained on a different input distribution (i.e., features from the penultimate layer). Feeding intermediate-layer features may introduce a distribution shift, potentially biasing the alignment scores and confounding representation quality with the generalization behavior of the final projection layer. It would be helpful to clarify or justify the robustness of the final-layer projection under this distribution \nshift.\n\n2.**Fig.1** While Table 1 reports the highest semantic alignment at the final layers across datasets and backbones, this trend should be interpreted with caution. Because intermediate-layer features are projected using the final ViT layer, differences across layers may partially stem from distribution mismatch with the projection layer, especially for early layers. Additional analysis could help disentangle true semantic alignment from projection-induced effects.\n\n3. While ATR, SSR, and SHE are empirically effective, they are primarily derived from experimental observations and heuristics. In addition, the framework relies on several hyperparameters, and it is unclear how sensitive the performance is to their selection across different datasets and backbones. This may limit the robustness and reliability of the method when applied to new datasets or architectures.\n\n4. The selection of top-k attention heads in SHE is based on averaged visual discriminability scores. It is unclear how stable these rankings are across datasets, random seeds, or backbone variants, and whether a fixed set of heads generalizes well.\n\n5. While the approach is claimed to be lightweight, the paper lacks a detailed analysis of inference-time overhead introduced by SSR reweighting and SHE-based similarity computation, especially for high-resolution inputs.", "broader_impact_concerns": "The methodology does not discuss potential failure cases, such as scenarios where suppressing semantic aggregation may harm performance (e.g., scenes with strong contextual dependencies), which would help practitioners better understand the method’s limitations.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7ZCT0zU00B", "reviewer": "Reviewer_REZ6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Contributions:\nThis paper addresses a limitation of CLIP-based models in training-free open-vocabulary semantic segmentation, which is the mismatch between a good image–text alignment obtained during CLIP pre-training, but a poor pixel-level visual discriminability. The authors conduct an analysis of CLIP’s vision encoder at the layer, attention-head, and token levels. They found that deeper layers substantially degrade visual discriminability while providing only marginal gains in semantic alignment. A central finding is the emergence of abnormal tokens with sparse, high-norm, and bias-like activations, which dominate attention and impair spatial coherence.\n\nMotivated by these observations, the paper proposes LHT-CLIP, a training-free framework that enhances visual discriminability while preserving semantic alignment. They also conduct extensive experiments on eight standard benchmarks, demonstrating that LHT-CLIP obtains significant improvements over several state-of-the-art training-free baselines, across different CLIP backbones, without relying on additional supervision, training, or auxiliary models.\n\n\nStrength:\n1. The paper provides a thorough empirical analysis of CLIP representations together with a reasonable motivation. It provides convincing evidence that degradation in visual discriminability, rather than insufficient semantic alignment, is a key bottleneck for dense prediction.\n2. The proposed method has a training-free setting, avoiding additional datasets, optimization, or heavy computation. It aligns well with the motivation of open-vocabulary and general-purpose deployment.\n3. The identification and mitigation of abnormal tokens via sparsity-based criteria is simple, interpretable, and seems to be more robust than prior anomaly-detection-based approaches.\n4. The paper evaluates on a broad set of benchmarks and backbones, providing several new insights to the community.\n\n\nWeakness:\n1. The proposed method requires inference-time architectural modification. Modifying the intermediate transformer during inference may complicate integration with highly optimized or fixed inference pipelines.\n2. The evaluation is limited to open-vocabulary semantic segmentation. I wonder if the proposed method can generalize well to other dense prediction tasks, like Monocular Depth Estimation [1,2,3] ? For example, in DepthCLIP [1], authors also mentioned that one weakness of CLIP-based monocular depth estimation might be a poor pixel-level visual discriminability, since features that are not important to image classification might be neglected during CLIP’s feature extraction. As a result, depth estimation for the background is not as good as for the foreground. Related experiments and discussion would be appreciated.\n\n\n[1] Zhang, Renrui, et al. \"Can language understand depth?.\" Proceedings of the 30th ACM International Conference on Multimedia. 2022.\n[2] Auty, Dylan, and Krystian Mikolajczyk. \"Learning to prompt clip for monocular depth estimation: Exploring the limits of human language.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n[3] Son, Eunjin, and Sang Jun Lee. \"CaBins: CLIP-based adaptive bins for monocular depth estimation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors conducted extensive experiments on eight standard benchmarks, demonstrating that LHT-CLIP obtains significant improvements over several state-of-the-art training-free baselines, across different CLIP backbones, without relying on additional supervision, training, or auxiliary models. Those experiments also back up the argument in the introduction.", "audience_yn": "Yes", "audience_explanation": "People who work on dense prediction (segmentation, object detection, or depth estimation) could benefit from this paper, especially people who work on applying vision-language models to those tasks.", "requested_changes": "As stated above in the weakness part, the evaluation is limited to open-vocabulary semantic segmentation. I wonder if the proposed method can generalize well to other dense prediction tasks, like Monocular Depth Estimation [1, 2, 3] ? For example, in DepthCLIP [1], the authors also mentioned that one weakness of CLIP-based monocular depth estimation might be a poor pixel-level visual discriminability, since features that are not important to image classification might be neglected during CLIP’s feature extraction. As a result, depth estimation for the background is not as good as for the foreground. Related experiments and discussion would be appreciated.\n\n[1] Zhang, Renrui, et al. \"Can language understand depth?.\" Proceedings of the 30th ACM International Conference on Multimedia. 2022.\n[2] Auty, Dylan, and Krystian Mikolajczyk. \"Learning to prompt clip for monocular depth estimation: Exploring the limits of human language.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n[3] Son, Eunjin, and Sang Jun Lee. \"CaBins: CLIP-based adaptive bins for monocular depth estimation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "broader_impact_concerns": "No concerns on ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)