OPMapper: Enhancing Open-Vocabulary Semantic Segmentation with Multi-Guidance Information
Authors: Xuehui Wang
Chongjie Si
Xue Yang
Yuzhi Zhao
Wenhai Wang
Xiaokang Yang
Wei Shen
Venue: NeurIPS 2025 poster ()
Keywords: Open-vocabulary semantic segmentation
Primary area: applications
TL;DR: A lightweight, plug-and-play mapper to boost the performance of OVSS with minimal computational overhead

Abstract: Open-vocabulary semantic segmentation assigns every pixel a label drawn from an open-ended, text-defined space. Vision–language models such as CLIP excel at zero-shot recognition, yet their image-level pre-training hinders dense prediction. Current approaches either fine-tune CLIP—at high computational cost—or adopt training-free attention refinements that favor local smoothness while overlooking global semantics. In this paper, we present OPMapper, a lightweight, plug-and-play module that injects both local compactness and global connectivity into attention maps of CLIP. It combines Context-aware Attention Injection, which embeds spatial and semantic correlations, and Semantic Attention Alignment, which iteratively aligns the enriched weights with textual prompts. By jointly modeling token dependencies and leveraging textual guidance, OPMapper enhances visual understanding. OPMapper is highly flexible and can be seamlessly integrated into both training-based and training-free paradigms with minimal computational overhead. Extensive experiments demonstrate its effectiveness, yielding significant improvements across 8 open-vocabulary segmentation benchmarks.

OPMapper: Enhancing Open-Vocabullay Semantic Segmentation with Multi-Guidance Information
Xuehui Wang!* Chongjie Si
Xue Yang²
Yuzhi Zhao Wei Shen
Wenhai Wang
Xiaokang Yang*
'Mo Key Lab of Artificial Intelligence, Institute, School Computer Science, SJTU School of Automation and Intelligent Sensing, SJTU
City University of Hong Kong *MMLab, Chinese University of Hong Kong
{wangxuehui, wei. (ra.sshen)ssjuu edu. en
Abstract
Open-vocabulary semantic segmentation assigns every pixel label drawn from open-ended, text-defined space. Vision-language models such as CLIP excel at zero-shot recognition, yet their image-level Current approaches either fine-tune CLIP-at high computational cost-or adopt training-free attention refinements that favor local smoothness while overlooking global semantics. In this paper, we present OPMapper, lightweight, plug-and and- play module that injects both local compactness and global connectivity into attention maps of CLIP. It combines Context-awart Attention Injection, which embeds spatial and semantic correlations, and Semantic Attention Alignment, which iteratively aligns the enriched weights with textual prompts. By jointly modeling token dependencies and leveraging textual guidance, OPMapper enhances visual understanding. OPMapper is highly flexible and can be seamlessly integrated into both training-based and training- paradigms with minimal computational overhead. Extensive experiments demonstrate its effectiveness, yielding significant improvements across 8 apecbboccyylary segmentation benchmarks.
hinders dense prediction.
Introduction
open-vocabulary semantic segmentation (OVSS) aims assign every pixel in image any category rather than to a fixed, closed set labels. In real-world applications, exhaustively annotating all possible classes is infeasible. Hence, OVSS has become touch-stone for measuring model's ability to generalize beyond its training categories. Large-scale vision-language pre-trained models (VLMs) 35 7 38). epitomised by Contrastive Language- Image Pre-training (CLIP) (40), have 50 52). However, leveraging CLIP for dense prediction is far from straightforward, Because CLIP optimised under image-level super- vision, its training paradigm is intrinsically misaligned with the pixel-wise granularity demanded by open-vocabulan detection 22 54
unlocked promising avenues for OVSS [14] 20 51
33
48
Consequently, naive adaptations typically yield noisy, spatially inconsistent masks and unsatisfactory performance, un- derscoring the need for principled strategies that bridge the supervision gap between image-level pre-training and pixel-level inference.
30
31 and segmentation
seemingly direct remedy finetune the entire CLIP backbone the mask generator 10 49 as shown in Figure (C). Although this strategy can inject task-specific inductive bias, it comes at a steep cost: (i) demands large-scale pixel-level annotations and significant GPU hours: (ii) gradient updates inevitably distort CLIP's ' carefully calibrated image-text embedding space, diminishing the
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Mask generase
Mask generntor
Maxk generatoe
Mask generator
siin weight
weights
value
weights
value
weights
A
Estra module
value
OPNiupper
valuc
CLIP Image Encoder
CLIP Image Encoder
CLIP Image Encoder
CLIP Image Enceder
(a) Triiiing based
(b) Training free
(c) Extra trainable module
(d) Our adapter plugin
Figure 1: Different approaches for achieving open-set segmentation using CLIP. (a) illustrates the training-basee paradigm, where all or part of CLIP's and mask generator's 's parameters are finetuned. (b) demonstrates the training- frreeng--reeeeee manner, where the key idea is directly enhancing the query-key attention to boost performance. (c) illustrates a scheme where an additional trainable module is inserted and jointly trained with the mask generator to achieve better performance. (d) highlights the versatility of our mapper, which are trained offline and can be applied across (a), (b), and (c), or their hybrid paradigms. For simplicity, we illustrate its application only in training-free methods here.
open-vocabulary generalization it was prized for. To sidestep these drawbacks, a parallel line of training-free methods, shown in Figure has emerged 45 28 [5]. Their key insight is to re- engineer the self-attention computation rather than alter model parameters. Concretely, they impose matrix transformations on the query-key similarity matrix-e.g. introducing spatial priors-to craft task-aware attention weights, which in turn remix the unmodified value embeddings into pixel- discriminative features. Because the underlying value embeddings (and all learned parameters) their aggregation is reweighted- the original image-text alignment is
remain unchanged-ool preserved, and no additional training or labelled data are required [28] [5 45). Nevertheless, most existing designs hinge on the heuristic of locality compactness assuming that neighbouring tokens possess the strongest semantic affinity- while undervaluing global connectiviry, the broader relational dependencies among distant tokens that underpin high-level semantic structures. This limitation compromises holistic understanding of image context; for example, recognizing small patch of knitted fabric alone may be insufficient to distinguish whether it belongs to coat or towel.
The foregoing analysis naturally prompts a question: Can we avoid redesigning the entire attention pipeline or updating millions of parameters? ? In this paper, we answer this question by proposing OPMapper (Object-to-Pixx Mapper) that re-calibrates only the frozen last-layer query-key pair to produce dense-aware attention weights, as shown in Figure dd). This mapper shifts CLIP's s focus from objects to pixels while balancing local compactness and global connectivity. To construct the OPMapper, we design one operative module and one training-only auxiliary module: Context-awe Attention Injection (CAI), which processes the query and key from the frozen last-layer of CLIP and injects local and global cues into them to predict attention weights; Semantic Attention Alignment (SAA), which nudges the enriched attention produced by CAI to stay aligned with textual prompts, allowing effective use of textual information to refine the representation of attention. Specifically, in the CAT module, we manually design an idealized set of attention weights guided by two principles: (1) tokens in close proximity receive higher attention, reinforcing local compactness; (2) tokens within the same semantic category also receive higher attention, promoting global connectivity. Aligning the predicted attention weights from CAI with this prior distribution effectively integrates both local and global dependencies into the attention mechanism. The SAA module, implemented as an iterative cross-modal fusion network, refines these CAI-enhanced attention weights by aligning them with textual features, thus facilitating a better text- text-image alignment. Once training is complete, the SAA module can be discarded, leaving CAI as the sole component of OPMapper.
Since OPMapper can convert CLIP's native outputs into dense-aware representations without altering parameters of the CLIP, it is highly flexible and can be seamlessly integrated into existing methods with minimal computational overhead, as shown in Figure dd), serving as a modular enhancement to improve both training-based/ffree approaches. Extensive experiments demonstrate its ability to significantly enhance the performance of existing baselines. Our contributions are as follows:
We design a lightweight OPMapper that transforms CLIP's object-level attention into pixel-level attention, without modifying any of its parameters.
To train the mapper, we devise two novel modules that inject both local compactness and global connectivity while preserving CLIP's inherent rross-modal alignment.
Our OPMapper was integrated into 11 Case- models and consistently delivered significant performance improvements across 8 benchmarks.

---

2 Related works
2.1 Vision-language foundation model
Vision-language models 440 35
7 38] have greatly advanced the ability to address open-set visual tasks. As a representative contrastive learning-based approach, CLIP [40 widely employed in open-set recognition for its effective alignment of visual and linguistic representations. CLIP establishes the relationship between vision and language by leveraging large-scale image-text pairs, resulting in two parallel encoders: one for encoding textual descriptions into text embeddings and the other for encoding images into context-aware global visual embeddings, thus enabling zero-shot recognition capabilities for various visual tasks.
2.2 Open vocabulary semantic segmentation
Open-vocabulary semantic segmentation [28 15 45 20 29), also referred to as zero-shot semantic segmentation, aims to segment images into arbitrary categories defined by textual descriptions. This task has gained significant attention due to its potential to generalize beyond fixed class sets, leveraging the capabilities of VLMs such CLIP. However, the pretraining objective of CLIP, which focuses on image-level aligmment between visual and textual features, makes it inherently unsuitable To address this limitation, various methods have been proposed to extend CLIP for dense segmentation, which can be broadly categorized into training-baseed based approaches and training-free free approaches
for dense prediction tasks [14 20 33
50 52].
Training-based methods typically involve optimizing all parameters of CLIP. CAT-Seg and SED [66] employ cost aggregation to refine the pixel-text cost volume and generate per-pixel classification predictions. SAN [49] enhances CLIP with an adapter, enabling both mask proposal generation and recognition. Meanwhile, FC-CLIP [53) leverages CLIP as a shared backbone for unified mask generation and classification. Mask-Adapte 32 extracts semantic activation maps from proposal masks, enriching contextual information and enhancing the alignment. SCAN [17] integrates CLIP's generalized semantic prior into the proposal embedding, preventing it from collapsing onto known categories. While these approaches achieve promising results, the substantial computational resources required for CLIP training pose significant challenges for practical deployment.
Training-free methods effectively address the aforementioned resource constraints by keeping CLIP's parameters fixed. Instead of fine-tuning the model, these approaches focus on designing more effective attention weights--seccffccllly optimizing the interaction between queries and keys within the attention mechanism to guide the aggregation of value tokens, thereby enhancing the final output. Specifically, SCLIP [45] reformulates the g-k attention weight as combination of q-q weight and k-k weight to better capture local semantic coherence. ProxyCLIP 28 leverages the output features from other vision foundation models, such as DINO 33 and MAE 23 to generate attention weightss. MaskCLIP [15,, on the other hand, enforces the q attention weights to be an identity matrix, thereby directly guiding visual feature aggregation. However, discussed in the introduction, these methods predominantly focus on token-level local compactness while overlooking global connectivity, leading to incomplete understanding of the image. To address this limitation, in the following section, introduce a novel approach that effectively leverages both local compactness and global connectivity, enhancing the overall comprehension of visual content.
3 Preliminary on CLIP
The CLIP model comprises an image encoder, denoted as Enc,, and a text encoder, denoted as Enc, Given an input image and a series of textual descriptions = (a photo of a {class} } E R2,, where class represents the categories present in the image and L is the number of classes, the visual feature F R x XC and the text embeddings T LxC are obtained as follows:
(I) where CLS E RC represents the class token with C representing the channel. aggregates global contextual information and is used for classifying the entire image.
CLS, F-FFF=EEn,. EEc,, (T),
= Enc,
For semantic segmentation, given an input image, we first utilize CLIP to extract the corresponding text embedding and the query, key, and value features from the final layer of its visual encoder, then process the query and key to generate a q-k attention weights, which combined with the value to produce the final visual features. The output of the final layer of Enc,, is employed to perform
3

---

Eygt
Oppapper
Laice Lacr
Context-aware Attention Injection
K"
Mared
Aligament
Cross-Att
ResAttn Block
ResAtt Block
W gt
Wreed
pred
Cross-Atn
Context-aware
Semantic Attention
MLP
Attention Injection
Alignment
ResAltn Block
ResAlin Block
Cruss-Atta
mask
soloct
CLIP Emmgge Encoder
CLIP Teat Encoder
T
CLS
(b)
(c)
Dropout after training
infomation stream
Appended
Figure 2: Overall framework of our proposed model. (a) illustrates the architecture of
OPMapper, which consists of two modules. (b) depicts the Context-aware Attention Injection (CAI) module with detailed network structure provided in the supplementary materials. (c) represents Semantic Attention Alignment (SAA) module, where the vellow-green gradient indicates the fusion of visual and textual information. Note that after training, the SAA module will be dropout.
clement-wise classification. This is achieved by leveraging the given text embedding T to generate a segmentation mask M x xL.
M = 4(FT*) h(Attn K, V)T*).
(2)
where is the argmax function and Attn is the attention mechanism. Q. K. and V denote the query, key, and value, respectively. These representations are obtained by applying linear mappings the previous layer's outputs, transforming input features for the attention mechanism. The text embedding T is typically kept fixed during training to preserve their semantic richness, effectively functioning as comprehensive set of classifiers.
Methodology
As illustrated in Figure OPMapper consists of two key modules: (1) Context-aware Attention Injection (CAD), which injects an idealized attention weight distribution that encodes both locality and global semantic coherence, effectively guiding the attention mechanism to capture meaningful token interactions. CAI is the sole element appended to the frozen CLIP stream, thus constituting the entire OPMapper proposed in this work. (2) The auxiliary Semantic Attention Alignment (SAA) module, which further refines the enriched attention by aligning it with textual prompts through an iterative fusion strategy, ensuring We will elaborate the overall loss functions that jointly optimize these modules and details about integration with other methods in Sec 4.3
comprehensive integration of visual and textual information.
4.1
Context-aware attention injection
4.1.1
Manually designed attention weights
Firstly, local compactness ensures that features at adjacent spatial locations exhibit smooth transitions, which erucial for fine-graid ane-ggraneed tasks such as object boundary delineation. Mcanwhile, global connectivity captures long-range semantic relationships, enabling the model to recognize holistic object structures even when parts of an object are spatially distant. We argue that a prior attention weight matrix should simultaneously account for both local compactness and global connectivity to achieve well- bell-balanced representation. However, in a training-free setting, it nearly impossible for the model to naturally incorporate both aspects using only existing supervision signals and standard computational mechanisms. To address this limitation, we introduce an idealized attention weight matrix that explicitly encodes both local and global relationships. This matrix serves guiding

---

reference for the attention mechanism, providing an inductive bias that encourages a more structured and semantically coherent attention distribution.
Given a ground truth label map L E R#*W where each pixel is assigned to one of C semantic classes, we first down-sample it to align with the output resolution of the CLIP image encoder:
L= [a(L, a:) E BR
where a is the scaling factor, ensuring that the segmentation label map is compatible with the feature grid of CLIP. fa represents the nown-sampling function, and H, W down-sampled label map. Next, we construct the prior attention weights based on two key principles: (I) spatially adjacent tokens should receive higher attention scores to preserve local compactness, ensuring smooth feature transitions, and (2) tokens that share the same semantic category should be more strongly connected, reinforcing global connectivity and capturing long-range dependencies.
W. represent the size of the a
Local Compactness. To preserve local compactness, we define the local attention matrix W E I pffW xfW which assigns higher scores to spatially closer tokens within the same semantic category. The attention score between two positions i and j is computed as:
Wili.j] [exp(_d(i,)) if = L5 0,
otherwise.
where d(i,j) represents the Euclidean distance between positions i and in the 2D spatial domain. This ensures that nearby tokens receive higher attention weights while distant tokens within the same class still maintain meaningful connection. To prevent self-loops, diagonal elements (i.e., i = j) are set to zero. The matrix is then row-wise normalized using a softmax function, similar to q-k attention computation in standard self-attention mechanisms.
Global Connectivity. To capture global connectivity, we construct global attention matrix W, w x where each entry () represents the semantic similarity between two spatial positions:
- f1. W - {0, otherwise.
i;
(5)
This binary mask ensures that only tokens belonging to the same semantic category are connected, even they are far from each other. We then normalize each row by a mean operation to obtain a probabilistic attention distribution, giving all related tokens an equal share of attention.
4.1.2 Unified attention weight construction
To achieve a balanced integration local and global dependencies, we combine the two attention matrices into unified attention weight matrix as shown in Figure 3
W qk AW, + A)W
(6)
where À is empirically determined scaling factor (set 0.3) to adjust the relative contribution of local and global attention components. This fused attention weight matrix serves as a reference target for OPMapper, guiding the model to learn optimal attention distribution that balances spatial smoothness and semantic coherence.
4.1.3 Supervising attention learning
OPMapper is integrated into the q-k attention computation of CLIP's vision encoder. Specifically, given the final layer's query Q and key K, OPMapper transforms them into enhanced representations:
Q' Mapper, (Q). K' = Mapper, & (K),
(7)
where Mapper, and Mapper,, are two mapper modules, each consisting of two cross-attention blocks identical those used in CLIP. The refined attention weights Wp Q'K'T are then computed using these transformed queries and keys.
To ensure OPMapper effectively learns to align with the designed local-global balanced attention, employ KI. divergence loss between the predicted weights W,, p and the reference weights W Wat:
W pi
LKL Σ
W op log
(8)
W p [[
5

---

onl fonality comparciooss
only global connectiitty
Jonality compactions global cunnentivity
Figure 3: Illustration of attention map construction. The left column shows how attention weights are assigned between patches, while the right column visualizes the resulting attention map. Red dots mark the selected tokens, and blue dots indicate the tokens whose similarity is computed, For simplicity, distances are represented by numerical values, with smaller distances implying higher weights. The example uses the bear category, visualizing attention relationships among all tokens.
By minimizing this loss, OPMapper learns to adjust the attention mechanism to better integrate local compactness and global connectivity, ultimately enhancing segmentation performance while maintaining computational efficiency.
4.2 Semantic attention alignment
While the CAI module injects local and global token relationships into the attention mechanism, it does not explicitly ensure alignment with textual prompts, which are crucial for apen-vocabulary segmentation. Without proper aligmment, the refined attention may fail to fully leverage semantic cues from language supervision, limiting its effectiveness in distinguishing visually similar objects belonging different categories. To address this, introduce the Semantic Attention Alignment (SAA) module, which further refines the enriched attention representation by aligning it with textual features. By iteratively fusing visual attention with text embeddings, SAA enhances the semantic consistency of attention weights, ensuring that object regions are not only spatially and globally coherent but also correctly associated with the corresponding textual descriptions.
As shown in Figure c), SAA operates in a three-stage process:
1. Text-Guided Visual Enhancement: Text embeddings refine visual attention to enhance semantic consistency.
2. Visual-Guided Text Refinement: The enriched visual features reciprocally influence textual embeddings for better alignment.
3. Cross-Modal Attention Refinement: A final fusion step iterates between the two modalities, ensuring deep semantic integration.
4.2.1 Text-guided visual enhancement
To inject textual cues into visual features, we use text embeddings as keys and values a cross- attention layer, with visual features as queries. This enables the model to reweight visual features based on their semantic proximity to textual categories. The operation formulated as
Fi CA(y = F, T, T) IR AWxC
where F; represents the text-refined visual features, F represents the visual representation obtained using W p as the attention weights, i.e., the result of applying this attention to the value embeddings. T denotes the feature generated by the text encoder from the encoded text prompt, and CA represents the cross attention mechanisms.
To further suppress the influence of irrelevant categories, we compute the cosine similarity between the CLS token and text embeddings, and between visual features and text embeddings, determining the relevance of each visual token to different categories:
SaLs CosSim CLS, T) E R
(10)
Sv T CosSim E IR #WXL
From these similarity scores, we select the top-k categories for each visual token by identifying the highest similarity values in Sas-T and Sy-T. The final category selection is obtained by taking the

---

union of the top-k categories from both scores:
Cselected (Sps(Sass-r) (opk(Sv-T).
(11)
Using this selected category set, we construct an attention mask Mottn E R x,, where each column determines the valid textual categories for the corresponding visual token. Specifically, if the j-th category is present in Cselevned we assign M[: J] otherwise, remains zero. Finally, we apply the computed mask to restrict attention weights only to the selected textual categories:
F CA(g F, T, T, mask Mattn).
(12)
This process ensures that visual features are selectively enriched with relevant textual context rather than being influenced by unrelated categories.
4.2.2 Visual-guided text refinement
To establish bidirectional alignment, the roles of visual and textual features reversed. Instead of text refining vision, updated visual features F1 now serve as keys and values, while text embeddings queries. This enables the textual features to dynamically adjust based on the enriched visual information:
Ti CA(q T,k F1, F1).
(13)
ensures that textual embeddings incorporate relevant spatial and semantic cues, reinforcing visual- text consistency.
4.2.3 Cross-modal attention refinement
Finally, we establish cross-modal feedback loop, integrating the refined textual representation T, back into the visual feature space through another cross atteenioo layer:
F2 CA(1 F,, Tı =T))
(14)
The resulting feature F2 represents the final semantically-alinee visual representation, where each token effectively captures both local-global dependencies and textual associations. The final segmentation prediction computed via:
Mpred Sigmoid(F,TT).
(15)
This iterative fusion strategy ensures that object regions are not only structurally coherent but also correctly mapped to textual descriptions, improving segmentation accuracy in an open-vocabulary setting. Since we continue to use the CLIP text encoder's original text embeddings T without incorporating the refined textual features T, from the visual refinement process, SAA is only applied during training. During inference, the model relies solely on the CAI module, making OPMapper lightweight, plug-and-play enhancement that imposes minimal computational overhead. This design ensures that OPMapper can be seamlessly integrated into existing segmentation frameworks.
4.3
Loss function and adaptation details
Loss function. In addition to LKL, the loss function for OPMapper combines the DICE loss, to address class imbalance, and the binary cross-entropy (BCE) loss, improve foreground bacoounduddbakoouudd discrimination. The overall loss function is:
L = WKL Lxi. + WDICE WPCCe
(16)
where WKL. WDICE, and WBCE hyper-parameters
balance the contributions of each term.
Adaptation details. Applying OPMapper to other methods is straightforward. Since the SAA module is discarded after training, OPMapper effectively reduces to the CAI module at inference time. To integrate it into other methods, we only need feed the output query and key embeddings from the final layer of the CLIP visual encoder of the method into OPMapper, which maps them to attention weights suitable for dense prediction tasks. These attention weights are then used in the original computation flow to interact with the value representations, yielding the final visual features.
7

---

Table 1: Quantitative comparison 8 benchmarks. Gains over the base model are shown in green. "Avg." indicates the mean performance across 8 datasets. t: Results refined with a post proceproessiing toolkit such DenseCRF. I: Models trained from seratch under identical batch sizes for faimess. All implementations are based on official codes.
Medie
Imagee Famddr
VOC-21
VOC-20
Contest-00
Contest-39
Object-171
Stif-171
AIEE-150
CitySrapos
Aug.
Yroining Jasso!
VIT-B/16
52.3
24.1
22.4
23.4
24.3
10,6
3
VIT
30.3
30.4
Former
VITB//66
89.5
SV
VIE
92.6
53.3
24.5
SAN
#CPMagpe:
VIT
95.2 (40.5
52.7 (40.)
41.3
(40.2)
Seg
L0
VIT
OPMappes
VIE
74.800.4
(40.40.))
562 (1.0)
(40.))
(40.6)
SCAN D7
BIL
94,9
56.8
44.1
30,5
+PPMapper:
VIEW
(+1.)
1..5
(40.7)
31.0040551
Tmiling.fee free
Vinila CLIP [0)
VIT-216
16.4
41.9
844
56
4.4
2.9
5.0
119
MasKCLIP
38.8
23.6
20.6
12.6
27,89
+3PMogger
VIF-2166
50.7
50.7.11199 19
(+4.)
57.5 (-8.)
158 ()9.4
3188-1122
246
(4700
(-189)
3766 (-4.9)
VIT-216
332
310
37.65
MoPMopper
VIRIN16
86.1
(+5.9
(-12
98.
56.8
(+5.6)
(4.(4
183(+1.5)
53.6-200
41 (-340
ClarCLIP
VIE-216
$1.8
80.9
32.4
35.9
35.2
23.9
16.7
3000
38.11
40PMappe
VIE
35.6(4..))
84.7 (43.8)
54.8 (-2.4)
38.(4227)
3655 (63.3
243.42.0)
18.1 (41.9
(-2.)
(-2.78)
ProsyCU JP 21
VIT-216
61.3
80.3
35.3
90.1
37.5
26.5
202
161
42.79
+0PMap
VIEB216
628(41.5)
84.3 (44.0)
56.0 (40.7)
40.1
10100..00
(61.3)
27.1 (40.6)
20.3 (40.))
37.2
43.35
(-104
LPOSS
VIT
60.2
80.2
35.0
36.9
34.7
25.3
21.2
37.6
41.39
VIE-216
63.744355
84.9 (44.7)
55.7 (40.7)
37. (+10)
(40.5)
26.4
(40.9)
40.0 (-2.4)
33224 (-135)
CASS
VIE
36.9
38.1
39.8
44.16
+0PMogper
VIT-2166
67.0
900 (+1.7)
(-1.3)
46 (40.5)
58. (40.7)
27.1 (40.9)
(40
(-122
45.87
(-1.1)
Vasilla CLIP 44
V/1
4.1
2.7
5.2
CaR 13
676
91.4
30.5
39.5
36.6
17.7
MukCLIP
15
VIEL/14
65.1
24.5
26,5
26.4
17.6
15.1
212
29,68
+0PMagpes
50.9-99.9
30.6 (+6.1)
(+7.0)
22.3 (+4.7)
(4.3433.2)
(-8.)
(-7.6)
SCLIP 49
VIEL/14
47.4
79.3
27.8
30.6
30.1
20.5
15.6
27.8
34,59
*0PMapper
VIE
(0,4
81.6 (12.5)
(-1.))
32.6
(42.0)
335 (+3.4)
(1.3)
165 (40.9)
(-25)
37.03 (-2.44
ClarCLPP 17
VIRE/1
46.1
268
29.6
30.1
19.9
27.
34.33
*PPMappe
VIEL/14
(9.2
$1. (+1.5)
28.7 (+1.5)
317(42.)
33.1 (+3.0)
(1.4)
16.1 (+E.)
29.5 (-1.9)
36.43 (-2.00)
PLP
35.1
38.2
+0PMogger
VIE-/14
$4.4 (+18)
38.(-0.66
369 (41
W.1 (40.0)
25.1 (409)
21.7 (4000400)
Table 2: Ablation study on CAI and SAA.
Table Ablation study on the construction of local and global attention weights.
Method
ADE
City
Context59
Object
VOC
Avg.
CLIP
2.9
5.0
9.2
5.6
41.9
12.9
Settings
ADE
City
Context-59
Object
VOC
Avg.
wlo SAA.
17.8
32.0
37.9
35.2
83.5
41.3
W
17.6
36.9
32.2
79.6
39.4
wio CAL.
18.3
31.3
37,5
35
83.7
41.2
w/o W.
17.5
32.4
38.3
35.1
83.8
41.4
OPMapper
18.1
32.9
38.6
36.5
84.7
42.2
OPMapper
18.1
32.9
38.6
36.5
84.7
42.2
5
Experiments
5.1
Datasets and baselines
We evaluate our method on 8 widely used semantic segmentation benchmarks, ensuring compre- hensive assessment. The datasets include PASCAL VOC20 (VOC21, with one additional background class) [8) PASCAL Context59 (Context60, with background) (39], COCO Object (Object) (2). COCO-Stuff (Stuff) (2). Cityscapes (City) 13, and ADF20K (ADE) 55 Mean Intersection-over- Union (mloU) is used as the evaluation metric across all datasets. Our mapper is trained the COCO-Stuff dataset, which contains 1 18k densely annotated images spanning 171 categories. We then integrate our lightweight mapper into two types of existing methods: training-based methods, including SAN [49]. SCAN [77 and CAT-Seg [10], and training-five methods, including Proxy-
CLIP (28), ClearCLIP
27
SCLIP 45). LPOSS 42 CASS 26 and MaskCLIP (15] We replace
the standard attention operation with our mapper to construct the attention weights.
5.2 Implementation details
All experiments were conducted using MMSegmennation were resized to have a 384-pixel short edge, followed by proportional scaling and random cropping to 384 x 384. Data augmentation strategies aligned with CAT-Seg, including random flipping and patterned noise injection. We used the Adam W optimizer with learning rates of 5 x 10-1 for the SAA and x 10-5 for the CAL Our mapper is highly memory-effiiiet, requiring I 2GB of GPU memory per batch, and was trained on four NVIDIA A100 GPUs (batch size: 8 per GPU) for 80,000 iterations,
During training, COCO-Stuff images
8

---

(a) Image
(b) GroundTruth
(c) CLIP
(d) ClearCLIP
(e) CLIP OOPMappe
Figure 4: Some visualization results evaluated on COCO-Stuff dataset. When equipped with our OPMapper, CLIP can obtain incredible performance. We provide more results in Appendix.
completing in 3-4 hours. Notably, OPMapper is trained entirely on the officially pre-trained CLIP model, keeping all CLIP parameters frozen. WKL, WDICE, and WBCE are set to 10, 5, 10, respectively. Further details are provided in the Appendix
5.3
Main results
The main results are presented in Table demonstrating the effectiveness of our lightweight OPMap- per across various benchmarks. In particular, when combined with CASS, OPMapper achieves superior performance across most benchmarks while substantially improving computational effi- ciency. By directly refining attention weights and integrating them with the feature maps from the vision- vision-language model, OPMapper enhances segmentation quality in a training free paradigm. Fur- thermore, when applied to trainable methods, OPMapper still yields notable gains, albeit with a less pronounced effect. This likely because fine-tuning inherently optimizes attention weights for dense prediction, reducing the extent of additional improvements brought by our mapper. OPMapper consis- tently delivers significant performance improvements while maintaining exceptional efficiency, with parameter size of only 0.8M. By eliminating redundant computations, OPMapper also facilitates faster inference, reinforcing its scalability and adaptability for diverse segmentation architectures.
To further illustrate OPMapper's effecctiveness, we provide qualitative results in Figure 4 showcasing its ability to enhance segmentation through improved attention alignment. More experimental results can be referred to Appendix A
5.4 Further analysis
To efficiently validate the effectiveness of each component, we conduct series of ablation studies, primarily employing CLIP with the VÝT-B/16 image encoder as the baseline, unless otherwise specified. This choice ensures that the evaluation remains unbiased and is unaffected by potential performance enhancements introduced by other CLIP variants.
Due to space limitations, we sincerely encourage readers to refer to the Appendix A B for more detailed analyses, including more ablation studies, the visualization of the learned attention weights, the motivation behind our design, the evidence for our claim.
Ablation different modules Table present the ablation study of OPMapper. It is obvious that both CAI and SAA demonstrate substantial improvements over the baseline, with the average performance improving from 12.92 mloU (baseline) to 41.28 mloU (CAl) and 41.16 mloU (SAA), respectively. Finally, by combining both modules, the average performance is further enhanced to 42. 16 mloU. These results confirm the complementary nature of the two modules.
blation on local and global attention weights To validate the effectiveness of the Local and Global Attention Weights (W; and W constructed in the CAI module, we conduct ablation experiments where each component is used independently. The results, presented in Table clearly demonstrate that the most substantial performance gains achieved when both global and local attention weights

---

are jointly considered. This highlights the necessity of balancing local and global dependencies for optimal segmentation performance. To validate the rationale behind our attention weights design, we visualize the predicted attention weights and compare them with those from fully trained models in the supplementary materials. The results indicate that the attention weights generated by OPMapper closely resemble those from fully trained models, highlighting its effectiveness.
Ablation on mixing coefficient (A). We conduct an ablation study on the choice of À in Equation 6 of our paper, which serves to balance the relative contribu- tions of local compactness and global connectivity when manually constructing the prior weight map. As
Table Ablation on mixing coefficient (A).
Settings
ADE
City
Context59
Object
Avg.
A- 0.1
17.8
32.5
18.2
35.9
83.8
41.6
==
32.9
38.6
36.5
0.5
18.0
32.6
38.1
36.0
83.7
41.7
shown in Table E the best performance is achieved when = 0.3. As increases beyond this point, performance gradually deteriorates, suggesting that the current fusion strategy between local and global cues is optimal for generating pixel-level weight maps. In contrast, relying solely on either local compactness or global connectivity leads to noticeable drop in performance.
0.
17.7
32.3
38.0
36.0
83.4
41.5
A-
31.7
37.3
35.6
Ablation on the weight of three loss functions. To investigate the impact of the three losses used dur- BaaninngKLLDICE/BBBC
Table 5: Ablation on the weight of three loss functions. Here, the setting (5. 5, 5) indicates that the WKL, WDICE and WACE are 5, 5, and 5, respectively. Other settings follow the same convention.
losses we explore various weight combinations within prede- fined range of 5 to 10, chosen to match the learning rate of the whole model. As shown in Table 5 the best performance is achieved when the weights are set to WKL = 10, WDICE = 5, and WBCE Reducing the weights of both WKL and WBCE to 5 results in noticeable performance drop. Interest- ingly, increasing the WDICE to also leads to a decline in performance, suggesting that an overem- phasis on DICE loss may amplify its role in handling class imbalance to extent that disrupts the CAI module's ability to properly transform the object- aware query and key embeddings from the final layer of the CLIP's vision encoder into pixel-aware.
ing OPMapper
Settings
ADE
City
Context59
Object
VOC
Avg.
10.
(5,5.5)
16.2
30.7
36.2
34.5
81.6
39.8
(5,5,10)
17.4
31.7
38.1
35.7
83.5
41.3
(5.10.5)
15.2
35.4
809
(5,10,10)
16.7
31.3
37.6
35.3
83.3
40.8
(10,5.5)
17.7
31.9
38.4
35.9
83.9
41.6
(10.5.10)
18.1
38.6
(10,10.5)
16.5
31.1
38.0
35.2
82.6
40.7
(10,10,10)
17.6
31.7
38.1
35.7
83.5
41.3
6 Conclusion
In this paper, we introduced OPMapper, a lightweight and flexible plug-and-play module designed to enhance vven-vocabulary segmentation by integrating local compactness and global connectivity into the attention mechanism. OPMapper introduces Context-aware Attention Injection (CAI) to explicitly encode both spatial proximity and semantic coherence, and Semantic Attention Alignment (SAA) to iteratively refine attention by aligning it with textual features. Our approach is computationally efficient and requires no additional training of CLIP parameters, making it seamlessly compatible with both training-baate based and training-free free paradigms. Extensive experiments demonstrate that OPMapper significantly improves segmentation performance across multiple benchmarks while maintaining minimal computational overhead. By effectively balancing local and global information within the attention mechanism, OPMapper offers scalable and adaptable solution for advancing open-vocabulary segmentation.
Limitations. OPMapper's ' benefit diminishes when the CLIP vision encoder has been heavily fine- tuned (its parameter distribution drifts markedly from the original model). Currently, OPMapper excels only when this shift is moderate. We think (i) quantify the distributional gap between the fine-tuned and base encoders, then correct to better accommodate OPMapper, and (ii) explicitly tie OPMapper's 's parameters to the statistics of the fine-tuned model are promising directions.
Broader impacts. Our method targets object segmentation beyond the training distribution, enabling more generalizable perception in robotics, autonomous driving, and related domains, ultimately advancing socially beneficial technologies.
Acknowledgmentss
This work was supported by the NSFC under Grant 62322604, 62176159, Natural Science Foundation of Shanghai 225/R4002266 and part by the Shanghai Municipal Science and Technology Major Project under Grant 2021SHZDZX0102.
10

---

References
[1] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang. Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 12966, 2023.
[2] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari, Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 1209-1218, 2018.
[3] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-super vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pages 9650 26609966, 2021.
[4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[5] Junbum Cha, Jonghwan Mun, and Byungscok Roh. Learning to generate text-grounded mask for open- world semantic segmentation from only image-text pairs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11165-11177 11174, 2023.
[6] Liang-Chich Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan I. Yuillc. Deeplah: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysix and machine intelligence, 40(():834 848. 2017.
[7] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing. Muyan Zhong, Qinglong Zhang. Xizhou Zhu, Lewei Lu,, et al. Internvl: Scaling up vision foundation models and aligning for generic vinual-linguistic tasks. In Proceedings of the TEEEJCVF Conference on Computer Vision and Pattern Recognition, pages 24185-24198, 2024.
[8] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2022.
[9] Seokju Cho, Sunghwan Hong, Sangryul Jeon, Yunsung Lee, Kwanghoon Sohn, and Seungryong Kim. Cats: Cost aggregation transformers for visual correspondence. Advances Neural Information Processing Systems, 14:9011-9023, 2021.
[10] Scokju Cho, Heeseong Shin, Sunghwan Hong, Anurag Arnab, Paul Hongsuck Seo, Seungryong Kim. Cat-seg: Cost aggregation for vecabuucyy semantic segmentation. In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Pattern Recognition, pages 41 13-4123, 2024.
[11] MMSegmeenation Contributors. MMSegmentation:: Openmmlab semantic segmentation toolbox and benchmark. https://ggthheii cttp:/cccoooooooao/ooooeeececccc wwwpwwwwmop.ppect..tc.c..co...e www.m/mlmmmaammmm wtt.a//mmmmmmmmmmiiinn..no 2020.
[12] Marius Cordts, Mohamed Omran. Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bemt Schiele. The cityscapes dataset for semantic urban understanding. In Proceedings of the IEFF conference computer vision and pattern recognition, pages 3213-3223, 2016.
[13] Marius Cordis, Mohamed Omran, Sebastian Ramos, Timo Scharwächter, Markus Enzweiler. Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset. In CVPR Workshop on the Future of Datasets in Vision, volume 2, page 1, 2015.
[14] Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11583- 11592, 2022.
[15] Xiaoyi Dong. Jianmin Bao, Yinglin Zheng, Ting Zhang, Dongdong Chen, Hao Yang. Ming Zeng. Weiming Zhang, Lu Yuan, Dong Chen, Maskelip: Masked self distilisttln advances contrastive language-imas pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10995-1 1005, 2023.
[16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 11929, 2020.
[17] Liu et al. Open- vocabulaay segmentation with semantic-assisted calibration. In CVPR 24, 2024.
11

---

[18] M. Everingham. L. Van Gool. C. K. I. Williams. J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2):303-338, June 2010.
[19] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88:303-308, 2009.
[20] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung- Yi Lin. Scaling open-vocabulary image segmentation with image-level labels. Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Isruel, October 23 27, 2022, Proceedings, Part XXXVI, pages 540 1500-57.55 Springer, 2022.
[21] Xiuye Gu, Tsung-Yi Yi Lin, Weicheng Kuo, and Yin Cui. language knowledge distillation, arXiv preprint arXiv:2 13391, 2021.
object detection via vision and
[22] Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[23] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 16000-16009, 2022.
[24] Dong Un Kang, Haycon Kim, and Se Young Chun. Class distrbbution-indueed attention map for open- vocabulary semantic segmentations. The Thirteenth International Conference on Learning Representa- tions, 2025.
[25] Tsung- Wei Kc, Jyh-Jing Hwang, Ziwei Liu, and Stella X Yu. Adaptive affinity fields for semantic segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 587-602, 2018.
[26] Chanyoung Kim, Dayun Ju, Woojung Han, Ming- Hsuan Yang, and Seong Jae Hwang. Distilling spectral graph for object-context aware open-vocabulary semantic segmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15033-15042, June 2025.
[27] Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang. Litong Feng, and Wayne Zhang. Clearelip: Decomposing clip representations for dense vision-language inference. arXiv preprint arXiv:2407.1244 12442. 2024.
[28] Mengeheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Proxyclip: Proxy attention improves clip for open-vocabulary segmentation. arXiv preprint arXiv:2408.04883 2024,
[29] Boyi Li, Kilian Q Weinberger. Serge Belongie, Vladlen Koltun, and René Ranftl. Language-driven semantic segmentation. arXiv preprint arXiv:2201.0333 035560 2022.
[30] Yan Li, Weiwei Guo, Xue Yang, Ning Liao, Dunyun Hc, Jiaqi Zhou, and Wenxian Yu. Toward open vocabulary aerial object detection with clip-activated student-teacher Icarning. In European Conference on Computer Vision, pages 431-448. Springer, 2025.
[31] Yan Li, Weiwei Guo, Xue Yang. Ning Liao, Shaofeng Zhang, Yi Yu, Wenxian Yu, and Junchi Yan. Exploiting unlabeled data with multiple expert teachers for open vocabulary aerial object detection and its orientation adaptation. arXiv preprint orXiv:241 1.02057, 2024.
[32] Yongkang Li, Tianheng Cheng, Wenyu Liu, and Xinggang Wang. Mask-adapter: The devil in the masks for open-vocabulary segmentation. arXiv preprini wrXiv:2412.045533 2024.
[33] Feng Liang. Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. arXiv preprint 2rxi::22 10,04100, 2022.
[34] Feng Liang. Bichen Wu, Xiaoliang Dai, Kumpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculeseu. Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7061-7071 2023.
[35] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lec, Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[36] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pages 10012-10022, 2021.
12

---

[37] Jonathan Long, Evan Shelhamer. and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In Proceedings of the IEEE conference on computer vision and patiern recognition, pages 3431-3440, 2015.
[38] Gen Luo, Xue Yang. Wenhan Dou, Zhaokai Wang. Jifeng Dai, Yu Qiao, and Xizhou Zhu. Mono-internvl: Pushing the boundaries of monolithic multimodal large language models with endogenous visual pre- training. arXiv preprint xrXiv:2410.08202. 2024.
[39] Roozbeh Mottaghi, Xianjic Chen, Xiaobai Liu, Nam-Gyu Cho, Scong- Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The rolc of context for object detection and semantic segmentation in the wild. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2014.
Scong-Whan
[40] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763 17688 PMLR, 2021.
[41] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng. Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Proceedings of the IEEE/CVF international conference computer vision, pages 1-3008839,2300 2019.
[42] Vladan Stojnié, Yannis Kalantidis, Jifi Matas, and Giorgos Tolias. Lposs: Label propagation over patches and pixels for open-vocabulary semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9794-9803, June 2025.
[43] Shuyang Sun, Runjia Li, Philip Torr, Xiuye Gu, and Siyang Li. Clip mnn Segment countless visual concepts without training endeavor, In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 13171-13182, 2024.
[44] Feng Wang, Jieru Mei. and Alan Yuille. Sclip: Rethinking self-attention for dense vision-languee inference. arXiv preprint arXiv22112.01597. 2023.
[45] Feng Wang, Jieru Mei, and Alan Yuille. Sclip: Rethinking self-attention for dense vision- -angugle inference. In European Conference on Computer Vision, pages 315-332. Springer, 2025.
[46] Bin Xie, Jiale Cao, Jin Xie, Fahad Shahbaz Khan, and Yanwei Pang. Sed: A simple encoder-decoder for open-vocahulary semantic segmentation. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 3426-3436, 2024.
[47] Jiarui Xu, Shalini De Mello, Sifci Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18134 18144, 2022.
[48] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang. and Shalini De Mello. Open- vocabulary panoptic segmentation with text-to-image diffusion models. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 2955-2966, 2023.
[49] Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. San: Side adapter network for open- vocabulary semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[50] Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open- vocabulary semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2945-2954, 2023.
[51] Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. simple baseline for open- vocabulary semantic segmentation with pre-trained vision-language model. European Conference on Computer Vision, pages 736-753. Springer, 2022.
[52] Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chich Chen. Convolutions die hard: Open- vocabulary segmentation with single frozen convolutional clip. arXiv preprint arXiv:2308.0 02487, 2023.
[53] Qihang Yu, Ju Hc, Xueqing Deng, Xiaohui Shen, and Liang-Chich Chen. Convolutions die hard: Open- vocabulary segmentation with single frozen convolutional clip. Advances in Neural Information Processing Systems, 36:32215 322344, 2023.
[54] Yuhang Zang. Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. -ocabbocyy detr with conditional matching. In European Conference on Computer Vision, pages 106-122. Springer, 2022.
13

---

[55] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler. Adela Barriuso, and Antonio Torrala. Scene parsing through ade20k dataset. In Proceedings of the IEEE/CVF conference computer vision and puttern recognition, pages 633-641, 2017.
[56] Bolei Zhou, Hang Zhao, Xavier Puig. Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through ade20k dataset. International Journal of Computer Vision, 127:302-321, 2019.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction accurately demonstrate model's contributions in open-vocabure vocabulary semantic segmentation.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss potential limitations in the section of Conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
This study does not present theoretical results that necessitate formal deriva-
tions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All of experimental details can referred to the section Experiments and the Supplementary Materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: We do not use any additional proprietary data; all datasets employed in this work are publicly available and can be accessed freely. Our code will be released GitHub upon acceptance the paper (if accepted).
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c ec/
public/guides///
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
7 //nips. hhaps..cc/ubllic/uiddes
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: All of experimental details can referred to the section of Experiments and the Supplementary Materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Our method is plug-in approach designed to enhance the performance of existing models. We conducted experiments across 88 combinations involving 11 different models and 8 distinct benchmarks scale that is substantial. Repeating all experiments multiple times would be prohibitively expensive for us. We believe the experiments con- ducted on a wide range of models and datasets can provide strong empirical evidence of the generalizability and effectiveness of our approach.
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: All of experimental compute resources can be referred the section of Experiments and the Supplementary Materials.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We conduct our code based on public available code toolbox. Our code will be released on GitHub upon acceptance of the paper (if accepted).
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We provide broader impacts in the section of Conclusion.
18

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out, For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have cited all necessary resources.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.com//aaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not791 involve LLMs as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neuril) hatt:e///eerrrppccc///ooeereeee/ /cC////oefeeeenee/222////.. for what should or should not be described.
21

---

Technical Appendices
More experiments
A.1
Qualitative results
As shown in Figure 5 we provide additional segmentation results for different datasets, including ADE20K [56]. Cityscapes [12, VOC [88] as well as Pascal Context [9]]. As illustrated in the
Figure [] inserting OPMapper into ClearCLIP(27 leads to more coherent segmentation results by connecting semantically related regions and reducing fragmented or noisy mis-seggmentttions.
COCO-Stuff 171
Pascal VOC 21
Pascal Context 59
ADE 150
Cityscape 19
GroundTruth
CL.IP
(d) ClearC1IIP
CLIP+ODAdapter
Figure 5: Visualization results on different datasets.
22

---

A.2 Additional ablations
Ablation on the choice of top k within SAA. In our SAA module, the parameter l: of topk operation (Equation TT controls how many of the most relevant text embeddings are allowed to interact with the visual features and subsequently influence the textual features through feedback. As shown in Table 6 when & is set greater than or equal to the total number of categories, the mechanism degenerates into standard aross-altention, allowing all textual features interact with the visual features. Conversely, when is set to the minimum value of I, the interaction is limited to the single textual token corresponding to CLIP's 's global classification output. As shown in Table setting &s too small (e.g.. k = 16) or too large (e.g., h = 128) leads to performance degradation Eegradatioonikkely due to the omission of relevant categories the inclusion of redundant ones, respectively. These results highlight that a properly chosen k value enhances the predicted attention map of CAI by improving pixel-level text-image alignment.
Table 6:
Ablation on the choice of top k of SAA.
Settings
ADE
City
Context59
Object
VOC
Avg.
k=16
16.9
31.7
37.2
35.3
83.2
40,9
k=32
17.6
32.4
38.2
36.2
83.6
41.6
k=64
18.1
32.9
38.6
36.5
84.7
42.2
k=96
17.7
32.5
38.4
36.3
84.1
41.8
1-128
17.1
32.2
37.9
35.8
83.7
41.3
Ablation on the components of SAA. We also conduct an ablation study on the three key steps of the SAA module. For convenience, we refer to Text-Guided Visual Enhancement, Visual-Guided Text Refinement, and Cross-Modal Attention Refinement as TGV, VGT, and CMA, respectively. As shown in Table both the text-to-image and image-to-ters to-text fusion steps contribute to improved alignment between modalities. This bidirectional interaction enables the injection of more accurate dense representations into CAI while preserving the original open-vocabulary capability of CLIP.
Table 7: Ablation on the components of SAA. TGV = Text- Guided Visual Enhancement, VGT Visual-Guidee Text Refinement, CMA Cross-Modal Attention Refinement.
Settings
ADE
City
Context59
Ohject
Avg.
TGV
17.5
32.2
37.8
36.0
83.3
41.4
TGV CMA
17.6
32.6
38.1
36.2
83,8
41.7
TGV + VGT + CMA
18.1
32.9
38.6
36.5
84.7
42.2
Ablation on the choice of training dataset for OPMapper. In addition to training OPMapper on COCO-Stuff, we also explore the use of other datasets commonly employed in dense prediction tasks. Specifically, we replace the original COCO-Stufe segmentation annotations with mixture of annotations from Object365 Detection [1] and LVIS Segmentation [22]. This modification
necessitates some adjustments to the training pipeline: 1). When constructing the prior attention maps for CAL, we treat all pixels within each bounding box in the detection datasets as belonging to single class, and proceed to compute the prior attention maps in the same manner as with segmentation data. 2) During training, the SAA module is supervised using both segmentation and detection annotations. In contrast to supervision derived solely from segmentation, introduce a parallel detection head to predict bounding boxes. As shown in Table 88, even when using the mixture data, the model is still capable of effectively learning mapper that transforms object-level query-key representations into pixel-level ones. These results suggest that training standalone mapper appended to CLIP outputs is viable strategy for enhancing its compatibility with open-vocabulary dense prediction tasks.
additionally
Table 8: Ablation on the choice of training dataset for OPMapper.
Training Data
ADE:
City
Context59
Object
VOC
Avg.
cOCO-Stuff
18.1
32.9
38.6
36.5
84.7
42.2
I.VIS O365
18.9
33.5
39.2
37.1
85.5
42.8
23

---

(a) Image (d) q-k attention w/ SAA
(b) Designed q-k attnention
(c) DINO V-v attention q-k attention w/All
(e) q-k attention w/ CAI
(a) Image (d) q-k attention w/ SAA
(b) Designed q-k attnention
(c) DINO v-v attention
(c) q-k attention w/ CAI
(f) q-k attention w/All
Figure 6: The visualizations different types of q-k attention weights. We applied normalization to the designed q-k attention (Figure and DINO attention weights (Figure which enhances the brightness of the visualization and makes the patterns easier to observe (with the sum of each row exceeding 1). In contrast, the results produced by OPMapper retain their original form, where the sum of each row equals This design choice aims to clearly reflect the magnitude of the original values. For better clarity, it recommended view the corresponding figures an enlarged format.
24

---

A.3 Visualization Results of attention weights
In this section, we present visualizations of the predicted q-k attention map from CAI, the corre- sponding designed attention map, and the prediction results, illustrated in Figure Specifically, we examine the following scenarios:
Only CAI applied (w/CAT).
Only SAA is applied (w/SAA).
Both CAI and SAA are applied (w/All), alongside comparison with the u-v attention map generated by DINO 4
From these visualizations, it obvious that the predicted attention weights under the supervision of CAI closely resemble the learned attention weights indicating their alignment. Our OPMapper produces attention maps that closely mirror those of the vision foundation model, such as DINO, further substantiating our approach and supporting claims.
B Further analysis
B.1 In-depth analysis of the motivation
The motivation of OPMapper: existing methods for computing attention weights mainly emphasize local compactness (q-q, k-k, q-q & k-k) while neglecting down-weighting global connectivity. To overcome this, we propose the CAI module, which explicitly combines local compactness and global connectivity to produce pixel-aware attention maps. In parallel, the SAA module guides CAI in preserving CLIP's cross- moossl alignment, thereby enabling open-vocabulary segmentation.
To validate the limitations of existing methods regarding global connectivity and to demonstrate that our approach better addresses this aspect, we conducted an experiment on 50 randomly selected images. Using the attention matrices from training-free methods such as MaskCLIP, we measured the attention between each token and same-category tokens at different spatial distances. Tokens were defined as "long-range" if their distance exceeded the average distance to all same-category tokens. For each image, we summed the attention values of these long-range tokens and then averaged across the image to obtain the mean range attention value. A higher value indicates greater emphasis on global connectivity.
Table 9: Long-range attention values for different models.
Model
Mean long-range atlention value
MaskCLIP [5
0.084
SCLIP 44
0.142
ClearCLIP 27
0.117
ProxyCLIP 28
0.289
CDAM 24
0.231
OPMapper
0.373
As shown in Table 9 our method achieves the highest mean long-range attention strength (0.373), while most other methods fall between 0.08 and 0.15, with the maximum only 0.289 (when in- corporating external models such as DINO). These results indicate that our method captures more information from same-category but distant tokens, ensuring stronger global connectivity. This supports our argument that existing methods emphasize local compactness while underrepresenting global connectivity, whereas our CAI module explicitly strengthens global connectivity to construct more effective attention relationships. In addition, we visualize both our designed attention weights and the v-v attention produced by DINO in Figure For clearer comparison, we applied brightness enhancement (so the sum of each row not equal to 1). The visualization shows that in DINO's v-v attention, brighter values mainly appear near the diagonal of the matrix, indicating that local compactness dominates when merging information from other tokens, even though DINO's training emphasizes global connectivity more than other training-fire -frec methods.
25

---

B.2 Deeper analysis of the prior attention weight
We clarify that our prior attention matrix design is not a simple intuition but fundamentally grounded on two well-established theoretical principles widely recognized in semantic segmentation literature:
Local Compactness:
Fundamentally, semantic segmentation can be conceptualized as a problem of spatially continuous local clustering, wherein adjacent pixels are not independent but instead exhibit strong categorical coherence due to the inherent spatial smoothness of natural images. This property underscores the expectation that meaningful segmentation should leverage local contextual regularities, thereby reinforcing the robustness and consistency of the learned representations. This is commonly agreed in classical segmentation literature [37 (6. In the main paper, Equation 4 explicitly incorporates the Euclidean distance into the attention computation to reinforce local semantic smoothness.
High-quality semantic segmentation fundamentally relies on the capacity to capture long-range semantic dependencies, beyond local contextual cues. This capability ensures that spatially distant pixels of the same semantic category- suchh as occluded regions of an identical object-are jointly recognized and consistently segmented [25]. Without modeling such non-local associations, segmentation results risk fragmenting semantically coherent regions, thereby undermin- ing both accuracy and robustness. Vision transformers [36 16 have also demonstrated this principle in modeling semantic relations between distant tokens and our Equation 5 precisely formulates a semantic-awee aware global connectivity pattern that aligns perfectly with this principle.
Global Connectivity:
To further provide rigorous empirical evidence, we conducted the following experiment: we randomly selected 50 images containing open-set categories. For each image, we generated three attention maps using our proposed construction method, Cat-Seg (which was trained on other close-set categories but not trained these open-set categories), and Mask2Former [8] (which was trained both close-set and open-set categories, and thus its attention map is treated as the "optimal" one), respectivlly. We then computed both the average cosine similarity and average Structural Similarity Index (SSIM) to measure the similarity between two attention map. As shown in the Table TO, ours (prior attention matrix), average cosine similarity 0.834 and average SSIM 0.808, shows consistently high similarity compared to Cat-Seg. rigorously demonstrating that our prior attention construction is much more similar to the supervvuee attention structure.
Table 10: Comparison on cosine similarity and SSIM.
Setting
Cosine Similarity
SSIM
Ours vs Mask2Former 0.834
0.808
Cat- Seg-Seg vs Mask2Former
0.785
0.771
B.3 Discussion on the difference with related works
In general, the originality of OPMapper and its distinctions from other related methods reflected in the following aspects:
We propose an explicitly designed local-global composite attention weight construction that directly maps q to obtain pixel-aware attention weights. In contrast, other methods compute attention either by incorporating external modules or by relying on self-self attention that primarily enhances local compactness. These two categories of approaches only achieve pixel-aware features indirectly or approximately.
We introduce the cross-modal semantic alignment module (SAA), which leverages alternat- ing text-image fusion as an implicit supervision signal to guide the learning of the mapping module (i.e., CAI), while preserving CLIP's original vision-language alignment. Other methods, however, neglect maintaining this alignment and disregard the utilization of the CLIP text encoder.
* Our method is modular, lightweight, and incurs no additional overhead during inference, making it adaptable to a wide range of approaches. In contrast, other methods lack such flexibility and generalizability, limiting their applicability across diverse settings.
26

---

One similar solution as a plug-and-plat modular for other open- vecaboocyyllary semantic segmentation method is CDAM [24]. It leverages the category distribution correlations in the image-text similarity map to construct attention better suited for precise localization, and it achieves strong results in experiments. However, our method and CDAM differ significantly terms of motivation and innovations, model framework, adaptation process, and experimental characteristics. In the tableX, we summarize the approaches of both methods across these dimensions and highlight the key differences.
Table 11: Comparison between OPMapper and CDAM.
Dimension
OPMapper
CDAM
Motivation innovations
We observe that frozen CLIP features are inherently object-aware. Under consistent value representations, transforming the at- tention into pixel-aware form facilitates direct transfer to dense tasks. The learn- ing the mapper, however, must balance local compactness and global connectiv- ity, while preserving CLIP's inherent vi- sion language alignment.
They observe that for each token (corre- sponding to a patch), its classification vec- tor distribution exhibits category depen- dency. Leveraging this distributional property allows them compute more robust inter-token correlations, ultimately yielding an attention mechanism better suited for segmentation tasks.
Architecture
A Plugin Mechanism. CAI is trained with manually designed ground-trun attention and guided by the SAA module to learn how to map and k. In inference, the & and k from the final CLIP layer are mapped into g' and K, which yield pixel-aware attention. Together with the frozen v, these form image features, which are then integrated with text features to produce the final mask.
A Rewriting Mechanism. Using the frozen from the final CLIP layer and the en- hanced text, CDAM derives a category distribution vector for each token. At- tention is then computed by mcasuring the differences among these token-level distributions. With this attention, v is recalculated obtain a new v', which is subsequently combined with the original (non-enhanced) text features to generate the mask.
Image-Text Alignment
SAA performs iterative alignment of im- age and text features, implicitly guiding OPMapper and ultimately influencing q and k.
CDAM lacks both explicit and implicit vi- sion-language alignment mechanisms. result, the attention generated from category distribution similarity, when used together with the value features compute v may not be compatible with the frozen text fea- tures.
Adaptation process
Adapt OPMapper to and & of the final CLIP layer to obtain new o' and K', and then caleulate image features through con- ventional dot-product attention with frozen v.
Compute similarity between text and image fcatures, and then calculate attention by ap- plying JS divergence the similarity map. Finally, the attention map and frozen to calculate the final image features.
Threshold sensitivity in complex
OPMapper requires no threshold or heuris- tic hyperparameter and learns attention weights in a fully end-to-end manner.
CDAM relies on a manually selected entropy-based threshold to filter category distributions. As acknowledged hy the au- thors, this threshold is difficult determine in complex scenes.
Differences in global seman- tie modeling
OPMapper, by design, assigns high atten- tion weights explicitly to distant same- category tokens, ensuring more consis- tent global connectivity.
CDAM enhances correlations among nearby same-category patches using JS divergence, but its handling of long-range same-category tokens still depends thresholding.
Applicability scope
OPMapper is compatible with both training- free and training-basce segmentation frame- works, showing broader versatility.
CDAM currently only applicable to training-free methods.
27

---

C Details of Mapper
We provide detailed description of the Mapper's architecture in Figure 7 As shown in Figure 2bb, the CAI consists of two parallel components, Mapper A and Mapper B, which share the same structure but operate with independent parameters. Taking Mapper A as an example, it incorporates two modified ResAttn blocks compared to the original CLIP [40] architecture. These modifications are designed to enhance the mapper's functionality while maintaining computational efficiency.
Input Reatures
Linear, Qu 768 Cou 996
Linear, C.e 7668 Coue
Layer Cu Guue
Linear, Ci Capp= =3072
Attn Layer. Ccp Couz-6
Gel.Ii
Linear. Coe 96 Cauz 768 766
Linear, 3072 Cons 2
Layer Norm, Cap Cou 7668
Lineur, C.e Coai-66
Low-rank MLP,, Can Gom 768
Output features
Figure 7: The detailed architecture the modified ResAttnBlock used in our CAI is illustrated. The components highlighted in red indicate the differences compared to the original ResAttBllock. Additionally, the green Linear represents low-rank linear transformation with rank of r = 2.
28


Reviews:
[{"review_id": "dyHZZKazJZ", "reviewer": "Reviewer_ohL3", "summary": "This paper tackles the problem of open-vocabulary semantic segmentation. It presents OPMapper, a lightweight, plug-and-play module that injects both local compactness and global connectivity into attention maps of CLIP. It introduces two core modules, Context-aware Attention Injection to embed spatial and semantic correlations, and Semantic Attention Alignment to iteratively align the enriched weights with textual prompts. The method shows promising performance on standard benchmarks.", "questions": "see weaknesses", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n+ The method is well motivated, and the proposed module is trained-once and applied-everywhere. \n+ Extensive experiments are conducted, and results are promising. \n\nWeaknesses: \n- The core task in building the CAI block is to construct an ideal attention matrix. However, the proposed method is mostly based on intuitive insights, and lacks a rigorous analysis to justify why the learned attention matrix  $W_{qk}$ can be considered  ideal. \n- The method introduces several hyperparameters (e.g., in Eq. (6) and Eq. (16)), but the robustness and sensitivity of these parameters are not analyzed. \n- Equation (9) appears to be redundant and potentially misleading, as Equation (12) is the one actually used in the computation.\n- The method demonstrates smaller overall improvements when using ViT-L/14 as the encoder compared to ViT-B/16. Please provide deeper insights regarding this result.", "was_revised": true, "final_justification": "The paper presents OPMapper, a lightweight, plug-and-play module that injects both local compactness and global connectivity into attention maps of CLIP. The method is well motivated, and the proposed module is trained-once and applied everywhere. Results are promising. The rebuttal helps clarify some key points. I appreciate it, and decide to maintain the borderline accept score.", "rebuttal": "Dear reviewer ohL3,\n\nWe sincerely thank you for your thoughtful comments, valuable insights, and the considerable time and effort you have dedicated to carefully reviewing our manuscript. The concerns and suggestions raised by you have provided us with an excellent opportunity to significantly enhance the quality and clarity of our paper. In the following, we address each concern point-by-point and present additional analyses and experiments to thoroughly resolve your concerns.\n\n&nbsp;\n\n---\n\n### 1. W1: Deeper analysis of the ideal attention matrix\n\nWe appreciate the reviewer for pointing out this concern regarding the theoretical rigor behind our ideal attention matrix design. We would first like to clarify that **the term 'ideal' used in our paper should be more appropriately understood as a prior attention matrix. Since it represents the target we expect the model to learn, we referred to it as 'ideal' in the current version**. However, to avoid potential confusion with an objective ideal, we will henceforth refer to it as a prior attention matrix (e.g., $W_{qk}$) in this response and update the paper accordingly.\n\nThen, we clarify that our prior attention matrix design is not a simple intuition but  fundamentally grounded on two well-established theoretical principles widely recognized in semantic segmentation literature:\n\n1. Local Compactness: Fundamentally, semantic segmentation can be conceptualized as a problem of spatially continuous local clustering, wherein adjacent pixels are not independent but instead exhibit strong categorical coherence due to the inherent spatial smoothness of natural images. This property underscores the expectation that meaningful segmentation should leverage local contextual regularities, thereby reinforcing the robustness and consistency of the learned representations. This is commonly agreed in classical segmentation literature (FCN [Long et al., CVPR’15], DeepLab [Chen et al., TPAMI’18]). In our paper, Equation 4 explicitly incorporates Euclidean distance into attention computation to reinforce local semantic smoothness.\n\n2. Global Connectivity: High-quality semantic segmentation fundamentally relies on the capacity to capture long-range semantic dependencies, beyond local contextual cues. This capability ensures that spatially distant pixels of the same semantic category—such as occluded regions of an identical object—are jointly recognized and consistently segmented (AAF [Ke et al., ECCV'18]). Without modeling such non-local associations, segmentation results risk fragmenting semantically coherent regions, thereby undermining both accuracy and robustness. Vision transformers (ViT, Swin Transformer) have also demonstrated this principle in modeling semantic relations between distant tokens and our Equation (5) precisely formulates a semantic-aware global connectivity pattern that aligns perfectly with this principle.\n\nTo further address the reviewer’s concern with rigorous empirical evidence, We conducted the following experiment: we randomly selected 50 images containing open-set categories. For each image, we generated three attention maps using our proposed construction method, Cat-Seg (which was trained on other close-set categories but not trained on these open-set categories), and Mask2Former (which was trained on both close-set and open-set categories, and thus its attention map is treated as the \"optimal\" one), respectivlly. We then computed both the average cosine similarity and average Structural Similarity Index (SSIM) to measure the similarity between two attention map. As shown in the table:\n\n| Setting  |  Cosine Similarity  |  SSIM  |\n| --------- | -------------------- | ---------|\n|Ours vs Mask2Former   |     0.834      |          0.808 |\n|Cat-Seg vs Mask2Former  |     0.785    |     0.771 |\n\n**Ours (prior attention matrix), average cosine similarity = 0.834 and average SSIM = 0.808, shows a consistently high similarity compared to Cat-Seg, rigorously demonstrating that our prior attention construction is much more similar to the fully-supervised attention structure.**\n\nFurthermore, as clearly shown in Supplementary Figure 2, the attention patterns generated by CAI supervision remarkably resemble those produced by vision foundation models (e.g., DINO), providing strong qualitative validation.\n\nThese analyses collectively provide rigorous justification beyond mere intuition. Therefore, we hope these can fully address the reviewer’s concern.\n\n&nbsp;\n\n---\n\n### 2. W2: More ablations for the choice of hyperparameters\n\nTo address concerns regarding hyperparameter sensitivity and robustness, We have already conducted experiments in Tables 2 and 3 of the supplementary material for the choice of $\\lambda$ and weights of different losses. We paste the results from the supplementary material below, and we kindly encourage the reviewer to refer to the supplementary material for detailed settings and explanations. The performance observed therein substantiates the rationality of our chosen hyperparameters and reinforces the reliability of the conclusions drawn in this work.\n\n> The choice of $\\lambda$ in Eq.(6)\n\n|  Settings | ADE | City | Context59 | Object | VOC | Avg |\n|-----------|-------|------|-------------|---------|-------|------|\n|$\\lambda$=0.1|17.8|32.5|38.2|35.9|83.8|41.6|\n|$\\lambda$=0.3|18.1|32.9|38.6|36.5|84.7|42.2|\n|$\\lambda$=0.5|18.0|32.6|38.1|36.0|83.7|41.7|\n|$\\lambda$=0.7|17.7|32.3|38.0|36.0|83.4|41.5|\n|$\\lambda$=0.9|17.0|31.7|37.3|35.6|82.6|40.8|\n\n> The choice of $w$ for different loss in Eq.(16). Settings denotes for ($w_{KL}$, $w_{DICE}$, $w_{BCE}$)\n\n|Settings | ADE  | City | Context59 | Object | VOC  | Avg. |\n|---------|------|------|-----------|--------|-------|------|\n|(5,5,5) | 16.2 | 30.7 | 36.2 | 34.5 | 81.6 |  39.8  |\n| (5,5,10) | 17.4 | 31.7 | 38.1 | 35.7 | 83.5 |  41.3  |\n| (5,10,5) | 15.7 | 30.2 | 35.4 | 34.1 | 80.9 |  39.3  |\n| (5,10,10) | 16.7 | 31.3 | 37.6 | 35.3 | 83.3 |  40.8  |\n|(10,5,5) | 17.7 | 31.9 | 38.4 | 35.9 | 83.9 |  41.6  |\n| (10,5,10)| 18.1 | 32.9 | 38.6 | 36.5 | 84.7  | 42.2 |\n| (10,10,5) | 16.5 | 31.1 | 38.0 | 35.2 | 82.6 |  40.7 |\n| (10,10,10) | 17.6 | 31.7 | 38.1 | 35.7 | 83.5 |  41.3 |\n\n&nbsp;\n\n---\n\n### 3. W3: Confusion about Eq 9 and Eq 12\n\nWe respectfully clarify that Equation 9 and Equation 12 are not redundant. Equation 9 introduces the **general formulation** of text-guided enhancement, establishing the theoretical basis of our approach. Equation 12, on the other hand, specifies the **implementation that integrates an attention mask** which further filters out irrelative categories before merging text guidance, thereby operationalizing the general form under our proposed framework. To avoid any potential confusion, we will further clarify their relationship in our paper, ensuring that readers can clearly distinguish between the general formulation and its masked instantiation.\n\n&nbsp;\n\n---\n\n### 4. W4: Smaller improvements for ViT-L/14\n\nWe appreciate the reviewer for raising this insightful question. The relatively smaller improvement observed with ViT-L/14 can be reasonably attributed to its inherently stronger global modeling capability due to its larger number of parameters and deeper architecture. Specifically, larger Vision Transformers (e.g., ViT-L/14) inherently exhibit more robust global connectivity, as evidenced by previous studies which consistently demonstrate improved semantic consistency and stronger long-range attention modeling as transformer models scale up in size and depth.\n\nSince our proposed OPMapper is explicitly designed to enhance both local compactness and global connectivity—particularly addressing the deficiency of global semantic connections in smaller vision transformers—the room for improvement naturally diminishes as the global modeling capability of the base encoder strengthens. **This observation is in fact a positive indication of OPMapper’s core motivation: it significantly boosts performance precisely when global connectivity is inherently limited**, such as in smaller transformer variants like ViT-B/16. Thus, the reduced improvement margin observed in ViT-L/14 experiments further validates and highlights OPMapper’s effectiveness in explicitly compensating for inadequate global connectivity.\n\n&nbsp;\n\nWe hope that our response has satisfactorily addressed your concerns, and we thank you again for your thoughtful feedback.\n\n&nbsp;\n\nBest regards,\n\nAuthors"}, {"review_id": "nWe99sNZMS", "reviewer": "Reviewer_Jwvm", "summary": "This paper presents OPMapper, a lightweight and effective module designed to enhance open-vocabulary semantic segmentation by improving the attention mechanisms in CLIP-based models. The authors point out that existing CLIP-based OVSS approaches either lack sufficient global semantic connectivity or demand substantial computational and data resources. OPMapper addresses these issues through two key components: Context-aware Attention Injection, which integrates both local and global semantic information into attention calculations, and Semantic Attention Alignment, which iteratively aligns attention maps with text embeddings to promote semantic coherence. The SAA module is only used during training and removed at inference time. Experiments show that this method has certain effectiveness.", "questions": "Please refer to the weaknesses proposed in strength and weaknesses section.\n\nAnd I have some minor questions:\n\n1. Discussion on the diminishing returns of recent training-based models on some of benchmarks. What can be the causes? How could this be mitigated in the future? This will further enhance the practicability of this work.\n2. I wonder that for ProxyCLIP is the DINO encoder still kept at inference with OPMapper?\n3. Are there results on the ADE-847 and PC-459 datasets?", "limitations": "Yes, authors proposed two promising directions to address their limitations.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths：\n\n1. Good motivation and easy to read. The authors pinpoint (i) the compute/annotation burden of training-based CLIP adaptations and (ii) the loss of global semantic connectivity in training-free attention patching. This crisp problem statement immediately clarifies why another method is needed and what precise gaps in the current landscape OPMapper fills.\n2. Proposes a unifying, low-overhead remedy. By reframing “object-to-pixel” mapping as the sole task of a tiny adapter that re-calibrates the frozen last-layer query–key pair, the paper shows a path that eliminates heavy fine-tuning yet still injects both local compactness and global connectivity into attention maps. This makes the solution both practically attractive and conceptually elegant.\n3. Solid empirical coverage. The authors graft OPMapper onto 11 distinct CLIP-based baselines (both training-free and training-based) and test on 8 public datasets.\n4. Exaustive ablations. Section 5.4 and Sections in supplementary have systematically ablated each component and hyper-parameters used in this paper, demonstrating the effectiveness of its design.\n5. The idea could generalise to detection, panoptic segmentation, or video.\n\nWeaknesses：\n\n1. Dense prose / heavy notation. Equations 3-8 introduce three attention matrices plus down-sampling without an illustrative toy example, which may cause many readers to re-derive to internalise the idea.\n2. The paper directly introduces open-vocabulary semantic segmentation without first providing an overview of semantic segmentation itself. It is recommended to include a brief introduction to some SOTA semantic segmentation methods[1,2,3,4].\n3. All experiments are trained on COCO-Stuff and then transferred. There is no analysis of domain shift (using different training data) or scaling to larger ViT-H/14 backbones.\n4. Marginal gains on strong baselines. For the best published model (ProxyCLIP-L/14), OPMapper yields only +0.96 mIoU on average. Authors should claricy this in their rebuttal.\n\n[1] Segmenter: Transformer for semantic segmentation. ICCV 2021.\n\n[2] SegNeXt: Rethinking convolutional attention design for semantic segmentation. NeurIPS 2022.\n\n[3] AUCSeg: AUC-oriented Pixel-level Long-tail Semantic Segmentation. NeurIPS 2024.", "was_revised": true, "final_justification": "After reading the authors’ response, my concerns have been resolved. I believe it is a strong paper that deserves to be published at NeurIPS. Therefore, I have maintained my positive evaluation.", "rebuttal": "Dear reviewer Jwvm,\n\nWe sincerely thank you for your thoughtful comments, valuable insights, and the considerable time and effort you have dedicated to carefully reviewing our manuscript. The concerns and suggestions raised by you have provided us with an excellent opportunity to significantly enhance the quality and clarity of our paper. In the following, we address each concern point-by-point and present additional analyses and experiments to thoroughly resolve your concerns.\n\n&nbsp;\n\n---\n\n### 1. W1: Dense prose / heavy notation without an illustration\n\nWe sincerely thank the reviewer for the insightful suggestion. Equations (3–8) are each associated with a specific meaning, and the three attention operations correspond to three distinct processes in our framework. In line with your suggestion, we will provide a more detailed explanation of these processes in the revised manuscript. In particular, we will add an illustrative figure to make the workflow more transparent and to ensure that readers can clearly grasp the rationale and implementation of our approach.\n\n&nbsp;\n\n---\n\n### 2. W2: Provide an overview of semantic segmentation for easily understanding\n\nWe sincerely thank the reviewer for the valuable suggestion. In the revised version, we will incorporate an introduction to the semantic segmentation task and cite relevant related works. This addition will help provide readers with a clearer understanding of the background and motivation for open-vocabulary semantic segmentation.\n\n&nbsp;\n\n---\n\n### 3. W3: Scaling experiments about training data and model size\n\nWe fully agree with the reviewer that the proposed scaling experiment is indeed necessary. In fact, **we have already provided results in the Table 6 of our supplementary material showing CAI trained on different datasets**, and we kindly invite the reviewer to consult the supplementary material for more details.\n\nRegarding the adaptation to larger CLIP image encoders such as ViT-H/14, we believe it will bring performance gains, though the improvement margin is expected to be narrower compared to ViT-B. In our Response 4 to Reviewer ohL3, we explained in detail why the improvement from ViT-L is smaller than that from ViT-B, and the same reasoning applies to ViT-H. Nevertheless, we sincerely appreciate this suggestion, and we plan to adapt CAI to ViT-H in our subsequent work and update the manuscript with the corresponding experimental results.\n\nAdditionally, as noted in our Response 6 to Reviewer  WxDe, we have included an experiment where CAI is integrated into Cat-Seg with a ViT-L backbone, and the results further demonstrate that CAI continues to improve performance in this setting.\n\n&nbsp;\n\n---\n\n### 4. W4: Marginal gains on strong baselines\n\nWe thank you for highlighting this interesting point, which was also observed by both you and Reviewer ReDz. First, we would like to clarify that our CAI module is extremely lightweight, and importantly, its training process and loss functions are entirely independent of the segmentation modules and categories, respectively.\n\nThen, in our experiments, we observe clear improvements in average performance among 8 benchmarks across most models. The only exception is ProxyCLIP, where the improvement is about 1%. This is because ProxyCLIP already leverages more powerful external models (e.g., DINO, MAE) to optimize its attention weights. It is consistent with the explanation we mentioned in our Response 4 to Reviewer ohL3. The parameter scale and training cost of these models are substantially larger than those of our CAI. Therefore, the relatively limited improvement (~1%) observed on ProxyCLIP is reasonable and consistent with expectations.  Importantly, **excluding ProxyCLIP, our method achieves gains ranging from 2% to 18.9% on other methods among 8 benchmarks, highlighting both the stability and the broad applicability of our approach**.\n\n&nbsp;\n\n---\n\n### 5. Q1: Discussion on the diminishing returns of training based methods\n\nWe sincerely appreciate your valuable suggestion, which will further enhance the practicability of this work. We posit that the limited, and in some cases negative, performance improvements observed on training-based methods may stem from two primary factors. (1) Their training datasets often overlap with the benchmark categories, leading the attention to become increasingly tailored to the local–global composition favored by those methods. (2) Many of these approaches fine-tune certain backbone parameters, which in turn induces substantial changes in the distributions of q, k, and v. This limitation has also been explicitly acknowledged in our manuscript.\n\nWe think (i) quantify the distributional gap between the fine-tuned and base encoders, then correct it to better accommodate OPMapper, and (ii) explicitly tie OPMapper’s parameters to the statistics of the fine-tuned model are promising directions. In the revised manuscript, we will explicitly discuss this phenomenon and make efforts to address this issue in our future work.\n&nbsp;\n\n---\n\n### 6. Q2: Inference details when appending CAI to ProxyCLIP\n\nYes, the DINO encoder is still kept at inference, otherwise, the ProxyCLIP essentially degenerates into an approximate version of ClearCLIP. The outputs of CAI and DINO will be combined as the attention weights for ProxyCLIP.\n\n&nbsp;\n\n---\n\n### 7. Q3: Results on ADE-847 and PC-459 benchmarks\n\nWe thank you for your valuable suggestion about extending experiments to more benchmarks. Typically, training-free methods do not provide results on ADE-847 and PC-459, as their performance on these large-scale open-vocabulary benchmarks is often extremely limited. Ensuring reliable performance in such large-category open-set tasks remains a challenging and valuable research direction. In our revised paper, we will make an effort to include experiments on these two benchmarks to further substantiate our approach.\n\n&nbsp;\n\nWe hope that our response has satisfactorily addressed your concerns, and we thank you again for your thoughtful feedback.\n\n&nbsp;\n\nBest regards,\n\nAuthors"}, {"review_id": "KdnPw5cvoX", "reviewer": "Reviewer_ReDz", "summary": "This paper proposes OPMapper, a lightweight, plug-and-play module designed to enhance open-vocabulary semantic segmentation by improving the pixel-level attention maps of frozen CLIP models. OPMapper achieves this by injecting both local compactness and global connectivity into attention weights via Context-aware Attention Injection (CAI) and aligning them with textual prompts using Semantic Attention Alignment (SAA) during training. Integrated seamlessly into existing methods, OPMapper increases performance across various benchmarks with minimal computational overhead at inference time.", "questions": "1. A detailed analysis is needed to determine whether previous methods indeed suffer from weak global connectivity.\n\n2. Please analyze whether OPMapper effectively addresses this global connectivity issue.\n\n3. Despite incorporating training, the performance improvement on training-free methods is limited. Could the authors explain why?\n\n4. Table 1 lacks comparison with more recent methods. Please justify the selection of baseline methods.\n\n5. Why is KL divergence used in Equation (8)?\n\n6. Is it valid to use W_qk as the reference target for OPMapper? Please provide justification.\n\n7. Would it not be better for OPMapper to update both q and k, similar to conventional self-self attention, to create q–q and k–k attention?\n\n8. In Equation (10), shouldn't the dimensions be R^C and R^(HW x C), instead of R^L and R^(HW x L)?", "limitations": "Yes. They discussed the limitations and broader impact.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1.OPMapper improves segmentation performance through various methods.\n\n2.OPMapper is compatible with both training-based and training-free approaches, enhancing its versatility.\n\nWeakness\n\n1. Despite incorporating training part, the performance gains for training-free methods remain relatively limited.\n\n2. The paper lacks a clear analysis of the limitations of existing methods and does not sufficiently justify the proposed approach. For instance, issues such as the lack of global connectivity in previous methods are not clearly examined.\n\n3. Due to weak analysis and motivation, it is difficult to assess the significance of the paper’s main claims.\n\n4. Existing methods also aim to improve attention maps, so it is unclear how OPMapper distinguishes itself in terms of originality.", "was_revised": true, "final_justification": "All of my concerns have been resolved and I change the score from 3 to 4.", "rebuttal": "Dear reviewer ReDz,\n\nWe sincerely thank you for your thoughtful comments, valuable insights, and the considerable time and effort you have dedicated to carefully reviewing our manuscript. In the following, We have summarized the related concerns into one unified concern, and address each unified concern point-by-point.\n\n&nbsp;\n\n---\n\n### 1. W1+Q3: limited performance improvement on training-free methods\n\nFirst, we would like to clarify that our CAI module is extremely lightweight, and importantly, its training process and loss functions are entirely independent of the segmentation modules and categories, respectively.\n\nThen, in our experiments, we observe clear improvements in average performance among 8 benchmarks across most models. The only exception is ProxyCLIP, where the improvement is about 1%. This is because ProxyCLIP already leverages more powerful external models (e.g., DINO, MAE) to optimize its attention weights. The parameter scale and training cost of these models are substantially larger than those of our CAI. Therefore, the relatively limited improvement (~1%) observed on ProxyCLIP is reasonable and consistent with expectations.  Importantly, **excluding ProxyCLIP, our method achieves gains ranging from 2% to 18.9% on other methods among 8 benchmarks, highlighting both the stability and the broad applicability of our approach**.\n\n&nbsp;\n\n---\n\n### 2. W2+W3+Q1+Q2: In-Depth Analysis of the Motivation: Why Prior Methods Lack It and How Our Approach Validates It\n\nWe first clarify the motivation of this work: existing methods for computing attention weights mainly emphasize local compactness (q–q, k–k, or q–q & k–k) while neglecting or down-weighting global connectivity. To overcome this, we propose the CAI module, which explicitly combines local compactness and global connectivity to produce pixel-aware attention maps. In parallel, the SAA module guides CAI in preserving CLIP’s cross-modal alignment, thereby enabling open-vocabulary segmentation.\n\nTo validate the limitations of existing methods regarding global connectivity and to demonstrate that our approach better addresses this aspect, we conducted an experiment on 50 randomly selected images. Using the attention matrices from training-free methods such as MaskCLIP, we measured the attention between each token and same-category tokens at different spatial distances. Tokens were defined as “long-range” if their distance exceeded the average distance to all same-category tokens. For each image, we summed the attention values of these long-range tokens and then averaged across the image to obtain the mean long-range attention value. A higher value indicates greater emphasis on global connectivity. The results are shown in the table below.\n\n|Model|mean long-range attention value|\n|-|-|\n|MaskCLIP|0.084|\n|SCLIP|0.142|\n|ClearCLIP|0.117|\n|ProxyCLIP|0.289|\n|Ours | 0.373|\n\nAs shown in the table, our method achieves the highest mean long-range attention strength (0.373), while most other methods fall between 0.08 and 0.15, with the maximum only 0.289 (when incorporating external models such as DINO). These results indicate that our method captures more information from same-category but distant tokens, ensuring stronger global connectivity. This supports our argument that existing methods emphasize local compactness while underrepresenting global connectivity, whereas our CAI module explicitly strengthens global connectivity to construct more effective attention relationships.\n\nIn addition, we kindly refer the reviewer to Figure 2 in our supplementary material, where we visualize both our designed attention weights and the v–v attention produced by DINO. For clearer comparison, we applied brightness enhancement (so the sum of each row is not equal to 1). We sincerely apologize for the typo in Figure 2 and will correct it in the paper. The visualization shows that in DINO’s v–v attention, brighter values mainly appear near the diagonal of the matrix, indicating that local compactness dominates when merging information from other tokens, even though DINO’s training emphasizes global connectivity more than other training-free methods.\n\n&nbsp;\n\n---\n\n### 3. W4: Difference with other methods that aim to improve attention maps\n\nWe thank the reviewer for raising this concern. The originality of OPMapper and its distinctions from other methods are reflected in the following aspects:\n\n1. We propose an explicitly designed local–global composite attention weight construction that directly maps q–k to obtain pixel-aware attention weights. In contrast, other methods compute attention either by incorporating external modules or by relying on self–self attention that primarily enhances local compactness. These two categories of approaches only achieve pixel-aware features indirectly or approximately.\n\n2. We introduce the cross-modal semantic alignment module (SAA), which leverages alternating text–image fusion as an implicit supervision signal to guide the learning of the mapping module (i.e., CAI), while preserving CLIP’s original vision–language alignment. Other methods, however, neglect maintaining this alignment and disregard the utilization of the CLIP text encoder.\n\n3. Our method is modular, lightweight, and incurs no additional overhead during inference, making it adaptable to a wide range of approaches. In contrast, other methods lack such flexibility and generalizability, limiting their applicability across diverse settings.\n\n&nbsp;\n\n---\n\n### 4. Q4: Baseline and comparison with recent methods\n\nWe thank the reviewer for the valuable suggestion. The baselines we selected are widely cited, come with publicly available code, and are commonly used as SOTA benchmarks (e.g., ClearCLIP). We also chose CLIP as it is the foundational training-free model on which most subsequent methods build.\n\nBeyond training-free methods, we integrated our approach into training-based SOTA models as well, where we similarly observed consistent performance improvements.\n\nWith respect to more recent methods, we identified two training-free approaches (LPOSS [CVPR’25], CASS [CVPR’25]) from CVPR 2025. Incorporating OPMapper into these methods, we achieved performance improvements, and we will include the corresponding experimental results in the revised manuscript.\n\n|model|VOC21|PC60|Object|VOC20|PC59|Stuff|City|ADE|\n|-|-|-|-|-|-|-|-|-|\n|LPOSS|60.2|35.0|34.7|80.2|36.9|25.3|37.6|21.2|\n|+OPMapper|+3.5|+0.7|+0.5|+4.7|+1.0|+1.1|+2.4|+0.9|\n|CASS|64.3|36.9|38.1|88.3|39.6|26.2|39.8|20.1|\n|+OPMapper|+2.7|+1.3|+0.7|+1.7|+0.5|+0.9|+1.2|+0.7|\n\n&nbsp;\n\n---\n\n### 5. Q5+Q6: The choice of using KL loss and the effectiveness of $W_{qk}$\n\nKL divergence is inherently well-suited for measuring the difference between two probability distributions, which makes it particularly appropriate for normalized probability-based representations such as attention matrices. In our implementation, both attention matrices—$W_{qk}$ and $W_{p}$—are row-wise normalized via softmax, thereby meeting the requirements of a probability distribution. We kindly refer the reviewer to our Response 3 to Reviewer WxDe, where we provide a more detailed explanation of the distribution of $W_{qk}$ and $W_{p}$.\n\nIn contrast, using MSE loss would treat the attention weights as continuous values without accounting for their probabilistic nature. While MSE penalizes absolute deviations, it does not capture the asymmetric information divergence between two distributions. This limitation is especially critical in attention modeling, where preserving the relative probability structure across tokens is more important than minimizing element-wise differences. Thus, KL divergence provides a more principled objective by emphasizing the distributional discrepancy rather than raw magnitude differences.\n\nTo further substantiate the validity of $W_{qk}$, we respectfully refer the reviewer to our Response 1 to Reviewer ohL3, in which we provide a detailed and rigorous justification.\n\n&nbsp;\n\n---\n\n### 6. Q7: q-k weights vs. (q-q)+(k-k) weights\n\nWe respectfully note that the reason we adopt the approach of mapping object-aware query and key embeddings into pixel-aware representations to construct a more effective attention map is because self–self attention, by design, tends to allocate the majority of attention weights to the diagonal and adjacent tokens, thereby being dominated once again by local compactness. Given that we have designed a ideal attention map as the target distribution, directly computing q–k attention provides a more principled and learnable formulation than first computing q–q and k–k attentions and then aggregating them.\n\nTo further validate this intuition, we performed an additional experiment on CLIP, where CAI was used to transform q and k to generate q–q + k–k attention. **The results demonstrate that our current approach achieves an average performance of 42.2, compared to only 40.8 for the q–q + k–k variant**, thereby confirming the effectiveness of our design.\n\n&nbsp;\n\n---\n\n### 7. Q8: Dimensions of Equation 10\n\nThank you for pointing out this potentially confusing issue. We would like to clarify the misunderstanding. As shown in Eq. 10,    $\\mathbf{S}_{\\texttt{CLS}-\\mathbf{T}}$ \n\nis obtained by computing the cosine similarity between $\\texttt{CLS}$ (with dimensions $1 \\times C$) and $\\mathbf{T}$ (with dimensions $C \\times L$, where $L$ denotes the number of categories input to the CLIP text encoder). Similarly, $\\mathbf{S}_{\\mathbf{V-T}}$ is computed between $\\mathbf{F_1}$ (with dimensions $\\hat{H}\\hat{W} \\times C$) and $\\mathbf{T}$. Therefore, their resulting dimensions are indeed $\\mathbb{R}^L$ and $\\mathbb{R}^{\\hat{H}\\hat{W}\\times L}$, respectively.  In the revised paper, we will clarify the dimensional relationships more explicitly to prevent any possible confusion.\n\n&nbsp;\n\nWe hope that our response has satisfactorily addressed your concerns, and we thank you again for your thoughtful feedback.\n\n&nbsp;\n\nBest regards,\n\nAuthors"}, {"review_id": "1VdsBDTZ5z", "reviewer": "Reviewer_WxDe", "summary": "This paper proposes a plug-and-play module, which is trained with the help of pixel-level annotated data to improve the dense representation capability of CLIP. This paper argues that training-based methods consume large amounts of resources, and training-free methods favor local smoothness while overlooking global semantics. The key contribution is to design self-attention labels to supervise the last layer of CLIP’s self-attention matrix to focus on local-global relationships.", "questions": "please refer to weakness.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The methods section is clearly written.\n2. Fig.2 is clear.\n\nWeaknesses:\n\n1.\tThe proposed method seems not to address the motivation that training-based method demands large-scale pixel-level annotations and significant GPU hours (Line.31). KL loss between manually designed attention and CLIP’s attention seems to require more GPU memory than the regular pixel-wise cross entropy loss (Because the self-attention matrix is very large). Please offer more training cost analysis.\n2.\tThe claim that training-free method favor local smoothness while overlooking global semantics (Line.40-45,5-6) needs to be more explored. Please provide some quantitative or qualitative insights to support the rationale for the motivation of the proposed method.\n3.\tWhy not calculate KL loss on the normalized distribution? Is Eq.6 softmax normalized? In addition, $W_p=Q^\\prime K^{\\prime \\top}$ seems not to softmax normalized, is the training stable if KL loss is calculated directly without normalization?\n4.\tMore experiment exploration about $\\lambda$. Such as setting it to learnable scalar. For high frequency areas like boundaries, does $\\lambda$ need to be larger? Please provide more insights on the local-global tradeoff (as this is one of the motivations).\n5.\tThe proposed method seems to aim to train CAI and discard SAA when using, so why not just use a linear layer instead of SAA? Please provide some insights. If the proposed method hopes to provide a trained segmentation model in addition to training a plug-in CAI, then the experimental results of the whole framework should be given to compare with the existing training-based methods.\n6.\tScaling experiments about training-based method needs to be provided.", "was_revised": true, "final_justification": "The author has provided a very detailed and reliable analysis of the issues that concern me. I decided to raise my score to 4.", "rebuttal": "Dear reviewer WxDe,\n\nWe sincerely thank you for your thoughtful comments, valuable insights, and the considerable time and effort you have dedicated to carefully reviewing our manuscript. The concerns and suggestions raised by you have provided us with an excellent opportunity to significantly enhance the quality and clarity of our paper. In the following, we address each concern point-by-point and present additional analyses and experiments to thoroughly resolve your concerns.\n\n&nbsp;\n\n---\n\n### 1. W1: More training cost analysis\n\nWe thank the reviewer for raising concerns regarding training efficiency. It is worth noting that training-based approaches typically require updating the backbone or relatively heavy fusion/segmentation modules, which constitute the most time- and memory-intensive components. In contrast, our method, when training OPMapper, only requires training two lightweight modules, CAI and SAA.\n\n**Both CAI and SAA are designed to be extremely efficient**, such that the entire training can be completed with less than 2GB of GPU memory while processing 8 images per GPU. Therefore, **the difference in GPU memory consumption between using KL divergence and pixel-wise cross-entropy is practically negligible in our current setting**. Moreover, pixel-aware cross-entropy loss is also not well-suited for measuring the discrepancy between two distributions.\n\nUnder current settings, the training takes approximately 3 hours with ViT-B and 4 hours with ViT-L. Furthermore, once trained, these modules can be seamlessly integrated into other models without additional retraining.\n\nTo provide a more comprehensive comparison, we present the training costs of representative training-based methods in the table below. The results clearly indicate that, while enhancing generalizability, our method achieves a substantial reduction in training costs, including both training time and GPU memory consumption.\n\n|model|backbone|Batch size x GPUs|Training time|VRAM|iterations|\n|-----|--------|-----------------|-------------|----|----------|\n|Cat-seg|ViT-B|4*4| ~8h | ~23GB | 80k |\n|SCAN | ViT-L |4*8 | ~18h | ~30GB | 80k |\n|SAN | ViT-B | 4*8 | ~18h | 6.6GB | 60k |\n|Ours | ViT-B | 8*4 | ~3h  | <2GB  | 80k |\n|Ours | ViT-L | 8*4 | ~4h  | <4GB | 80k |\n\n&nbsp;\n\n---\n\n### 2. W2: In-Depth Analysis of the Motivation\n\nWe first clarify the motivation of this work: existing methods, when computing attention weights, primarily emphasize local compactness (q-q, k-k or q-q&k-k attention) while neglecting or down-weighting global connectivity. To address this limitation, we introduce the CAI module, which explicitly integrates both local compactness and global connectivity to directly output pixel-aware attention maps. In parallel, we employ the SAA module to guide CAI in preserving CLIP’s inherent cross-modal alignment, thereby enabling open-vocabulary segmentation.\n\nTo validate the drawbacks of other methods in regard to global connectivity and demonsrate that our method better accounts for global connectivity compared to existing approaches, we conducted the following experiment on 50 randomly selected images. Specifically, using the attention matrices from training-free methods such as MaskCLIP and ProxyCLIP, we measured the attention value between each token and same-category tokens at varying spatial distances. We defined tokens as “long-range” if their spatial distance exceeded the average distance from the given token to all same-category tokens. For each image, we aggregated the attention values of these long-range tokens and computed their total value, followed by averaging across the entire image to obtain the mean long-range attention value. A higher value indicates greater emphasis on global connectivity. The results are presented in the table below.\n\n| Model | mean long-range attention value |\n|-|-|\n|MaskCLIP|         0.084     |\n|SCLIP |    0.142            |\n|ClearCLIP|     0.117              |\n|ProxyCLIP|      0.289             |\n|Ours |        0.373               |\n\nAs shown in the table, our method achieves the highest mean long-range attention strength, reaching 0.373, while most other methods fall within the range of 0.08 to 0.15, with the maximum being only 0.289 (achieved when incorporating external models such as DINO to provide global connectivity). These results demonstrate that our method effectively captures more information from same-category but spatially distant tokens, thereby ensuring more comprehensive global connectivity. This further substantiates our argument that existing methods tend to emphasize local compactness while underrepresenting global connectivity, whereas our CAI module explicitly enhances global connectivity, thereby achieving a more effective construction of attention relationships.\n\nIn addition, we kindly refer the reviewer to Figure 2 in our supplementary material. There, we visualize both our designed attention weights and the v-v attention produced by DINO. To facilitate a clearer comparison, we applied brightness enhancement (so that the sum of each row is not equal to 1). We also sincerely apologize for the typo appearing in the Figure 2 and will correct this in our paper. From the visualization, it is observed that in DINO’s v-v attention, the brighter values predominantly appear near the diagonal of the attention matrix, indicating that local compactness is the dominant factor when merging information from other tokens, even though global connectivity is heavily emphasized during DINO’s training compared with other training-free methods. \n\n&nbsp;\n\n---\n\n### 3. W3: Distribution of $W_{qk}$ and $W_{p}$ in Eq 6\n\nThank you for highlighting this important point. We agree that clarification regarding the normalization of Eq. (6) is necessary. Specifically, the components $W_l$ and $W_g$ in Eq. (6) are indeed normalized distributions: $W_l$ is normalized via a standard softmax operation (line 173), ensuring each row sums to 1. Similarly, $W_g$ is constructed as a uniform distribution (line 179) among pixels of the same category, which can also be viewed as a special case of softmax normalization, guaranteeing each row sums exactly to 1 as well.\n\nSince the weights in Eq. (6) (i.e., $\\lambda$ and $1-\\lambda$) sum to 1, the resulting combined attention matrix $W_{qk}$ is inherently normalized, ensuring each row also sums to 1. Regarding the predicted attention $W_p$ , it is also normalized via a softmax operation during training. However, we acknowledge that the explicit softmax symbol was omitted in the manuscript, potentially causing confusion. We will clearly include this explicit normalization step in future manuscript revisions to avoid any misunderstanding.\n\n&nbsp;\n\n---\n\n### 4. W4: The choice of $\\lambda$\n\nWe have conducted additional experiments on the choice of $\\lambda$ in Table 2 of the supplementary material to further validate the rationale behind our selection. We kindly invite the reviewer to refer to these results for more details. Moreover, following the reviewer’s suggestion, we also performed a quick experiment by treating $\\lambda$ as a learnable scalar. The result shows that when $\\lambda = 0.3372$, the average performance reaches 41.9, which is very close to the manually set value of 0.3. This further substantiates the effectiveness of our ablation study.\n\nRegarding high-frequency regions (e.g., boundaries), since they violate the assumption of local smoothness, the corresponding local weight $\\lambda$ is set to be **smaller** in these areas.\n\n&nbsp;\n\n---\n\n### 5. W5: The design of SAA\n\nFirst, we would like to clarify that our method does NOT rely on providing a trained segmentation module to achieve the final segmentation. The ONLY component inserted into other methods is the CAI module.\n\nSecond, we address the question of why we designed the SAA module instead of simply using an MLP. During the training of CAI, the most direct supervision signal comes from our manually constructed “ideal” attention matrix, i.e., the q-k weight map, which serves as an explicit form of supervision. The purpose of SAA, however, is to provide an indirect, implicit supervisory signal. Specifically, the predicted q-k weights are combined with the original v from the frozen CLIP to form the input to SAA, which then generates a segmentation map for loss computation. In the process of transforming input into output, SAA iteratively performs image-to-text and text-to-image fusion, ensuring proper alignment of the two modalities in CLIP. This alignment is propagated back to CAI through the predicted q-k weights. Therefore, SAA functions more like a mechanism for computing the loss, enabling CAI outputs to not only achieve the transition from object-level to pixel-level representations but also preserve the original cross-modal alignment. A simple MLP cannot fulfill this design objective.\n\nTo further verify this, we conducted an experiment where we replaced SAA with MLPs of approximately the same parameter size, applying it to transform the q-k weights, while keeping the subsequent segmentation generation process identical. When using ViT-B, this model achieved an average mAP of 40.5 across five datasets (ADE, Cityscapes, Context59, Object, and VOC-20). In comparison, the CAI+SAA combination achieved an average mAP of 42.2, while CAI without SAA yielded 41.3. These results demonstrate that directly using an MLP is suboptimal, as it even suppresses part of the improvement brought by CAI itself (41.3 → 40.5).\n\n&nbsp;\n\n---\n\n### 6. W6: Scaling experiments\n\nFollowing reviewer's advice, we conducted a model scaling experiment by inserting the ViT-L version of CAI into Cat-Seg. As shown, this integration led to improvements of +0.6, +1.5, +0.4, and +0.5 mIoU on the VOC, Context-59, Stuff-171, and ADE-150 benchmarks, respectively.\n\n&nbsp;\n\nWe hope that our response has satisfactorily addressed your concerns, and we thank you again for your thoughtful feedback.\n\n&nbsp;\n\nBest regards,\n\nAuthors"}]

Author rebuttal:

Meta-review: This paper presents work on open vocabulary semantic segmentation.  The core contribution is a module for pushing local compactness and global connectivity into a semantic segmentation method.

The reviewers initially raised concerns over the motivation and performance gains of the proposed method, while appreciating the simplicity and general applicability of the plug-in module.

After considering the authors’ responses, the reviewers converged on a general recommendation to accept the paper.  The author responses helped to clarify details of the paper and provided numerous additional pieces of empirical evidence validating the claims.

Overall, the paper provides a module for semantic segmentation that improves results and can be utilized in different paradigms for this task and is recommend for acceptance in NeurIPS.
Decision: Accept (poster)