Title: OPMapper: Enhancing Open-Vocabulary Semantic Segmentation with Multi-Guidance Information
Venue: NeurIPS 2025
Accepted: yes
Keywords: Open-vocabulary semantic segmentation
Abstract: Open-vocabulary semantic segmentation assigns every pixel a label drawn from an open-ended, text-defined space. Vision–language models such as CLIP excel at zero-shot recognition, yet their image-level pre-training hinders dense prediction. Current approaches either fine-tune CLIP—at high computational cost—or adopt training-free attention refinements that favor local smoothness while overlooking global semantics. In this paper, we present OPMapper, a lightweight, plug-and-play module that injects both local compactness and global connectivity into attention maps of CLIP. It combines Context-aware Attention Injection, which embeds spatial and semantic correlations, and Semantic Attention Alignment, which iteratively aligns the enriched weights with textual prompts. By jointly modeling token dependencies and leveraging textual guidance, OPMapper enhances visual understanding. OPMapper is highly flexible and can be seamlessly integrated into both training-based and training-free paradigms with minimal computational overhead. Extensive experiments demonstrate its effectiveness, yielding significant improvements across 8 open-vocabulary segmentation benchmarks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a plug-and-play module, which is trained with the help of pixel-level annotated data to improve the dense representation capability of CLIP. This paper argues that training-based methods consume large amounts of resources, and training-free methods favor local smoothness while overlooking global semantics. The key contribution is to design self-attention labels to supervise the last layer of CLIP’s self-attention matrix to focus on local-global relationships.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The methods section is clearly written.\n2. Fig.2 is clear.\n\nWeaknesses:\n\n1.\tThe proposed method seems not to address the motivation that training-based method demands large-scale pixel-level annotations and significant GPU hours (Line.31). KL loss between manually designed attention and CLIP’s attention seems to require more GPU memory than the regular pixel-wise cross entropy loss (Because the self-attention matrix is very large). Please offer more training cost analysis.\n2.\tThe claim that training-free method favor local smoothness while overlooking global semantics (Line.40-45,5-6) needs to be more explored. Please provide some quantitative or qualitative insights to support the rationale for the motivation of the proposed method.\n3.\tWhy not calculate KL loss on the normalized distribution? Is Eq.6 softmax normalized? In addition, $W_p=Q^\\prime K^{\\prime \\top}$ seems not to softmax normalized, is the training stable if KL loss is calculated directly without normalization?\n4.\tMore experiment exploration about $\\lambda$. Such as setting it to learnable scalar. For high frequency areas like boundaries, does $\\lambda$ need to be larger? Please provide more insights on the local-global tradeoff (as this is one of the motivations).\n5.\tThe proposed method seems to aim to train CAI and discard SAA when using, so why not just use a linear layer instead of SAA? Please provide some insights. If the proposed method hopes to provide a trained segmentation model in addition to training a plug-in CAI, then the experimental results of the whole framework should be given to compare with the existing training-based methods.\n6.\tScaling experiments about training-based method needs to be provided.\n\nQUESTIONS:\nplease refer to weakness.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes OPMapper, a lightweight, plug-and-play module designed to enhance open-vocabulary semantic segmentation by improving the pixel-level attention maps of frozen CLIP models. OPMapper achieves this by injecting both local compactness and global connectivity into attention weights via Context-aware Attention Injection (CAI) and aligning them with textual prompts using Semantic Attention Alignment (SAA) during training. Integrated seamlessly into existing methods, OPMapper increases performance across various benchmarks with minimal computational overhead at inference time.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1.OPMapper improves segmentation performance through various methods.\n\n2.OPMapper is compatible with both training-based and training-free approaches, enhancing its versatility.\n\nWeakness\n\n1. Despite incorporating training part, the performance gains for training-free methods remain relatively limited.\n\n2. The paper lacks a clear analysis of the limitations of existing methods and does not sufficiently justify the proposed approach. For instance, issues such as the lack of global connectivity in previous methods are not clearly examined.\n\n3. Due to weak analysis and motivation, it is difficult to assess the significance of the paper’s main claims.\n\n4. Existing methods also aim to improve attention maps, so it is unclear how OPMapper distinguishes itself in terms of originality.\n\nQUESTIONS:\n1. A detailed analysis is needed to determine whether previous methods indeed suffer from weak global connectivity.\n\n2. Please analyze whether OPMapper effectively addresses this global connectivity issue.\n\n3. Despite incorporating training, the performance improvement on training-free methods is limited. Could the authors explain why?\n\n4. Table 1 lacks comparison with more recent methods. Please justify the selection of baseline methods.\n\n5. Why is KL divergence used in Equation (8)?\n\n6. Is it valid to use W_qk as the reference target for OPMapper? Please provide justification.\n\n7. Would it not be better for OPMapper to update both q and k, similar to conventional self-self attention, to create q–q and k–k attention?\n\n8. In Equation (10), shouldn't the dimensions be R^C and R^(HW x C), instead of R^L and R^(HW x L)?\n\nLIMITATIONS:\nYes. They discussed the limitations and broader impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents OPMapper, a lightweight and effective module designed to enhance open-vocabulary semantic segmentation by improving the attention mechanisms in CLIP-based models. The authors point out that existing CLIP-based OVSS approaches either lack sufficient global semantic connectivity or demand substantial computational and data resources. OPMapper addresses these issues through two key components: Context-aware Attention Injection, which integrates both local and global semantic information into attention calculations, and Semantic Attention Alignment, which iteratively aligns attention maps with text embeddings to promote semantic coherence. The SAA module is only used during training and removed at inference time. Experiments show that this method has certain effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n\n1. Good motivation and easy to read. The authors pinpoint (i) the compute/annotation burden of training-based CLIP adaptations and (ii) the loss of global semantic connectivity in training-free attention patching. This crisp problem statement immediately clarifies why another method is needed and what precise gaps in the current landscape OPMapper fills.\n2. Proposes a unifying, low-overhead remedy. By reframing “object-to-pixel” mapping as the sole task of a tiny adapter that re-calibrates the frozen last-layer query–key pair, the paper shows a path that eliminates heavy fine-tuning yet still injects both local compactness and global connectivity into attention maps. This makes the solution both practically attractive and conceptually elegant.\n3. Solid empirical coverage. The authors graft OPMapper onto 11 distinct CLIP-based baselines (both training-free and training-based) and test on 8 public datasets.\n4. Exaustive ablations. Section 5.4 and Sections in supplementary have systematically ablated each component and hyper-parameters used in this paper, demonstrating the effectiveness of its design.\n5. The idea could generalise to detection, panoptic segmentation, or video.\n\nWeaknesses：\n\n1. Dense prose / heavy notation. Equations 3-8 introduce three attention matrices plus down-sampling without an illustrative toy example, which may cause many readers to re-derive to internalise the idea.\n2. The paper directly introduces open-vocabulary semantic segmentation without first providing an overview of semantic segmentation itself. It is recommended to include a brief introduction to some SOTA semantic segmentation methods[1,2,3,4].\n3. All experiments are trained on COCO-Stuff and then transferred. There is no analysis of domain shift (using different training data) or scaling to larger ViT-H/14 backbones.\n4. Marginal gains on strong baselines. For the best published model (ProxyCLIP-L/14), OPMapper yields only +0.96 mIoU on average. Authors should claricy this in their rebuttal.\n\n[1] Segmenter: Transformer for semantic segmentation. ICCV 2021.\n\n[2] SegNeXt: Rethinking convolutional attention design for semantic segmentation. NeurIPS 2022.\n\n[3] AUCSeg: AUC-oriented Pixel-level Long-tail Semantic Segmentation. NeurIPS 2024.\n\nQUESTIONS:\nPlease refer to the weaknesses proposed in strength and weaknesses section.\n\nAnd I have some minor questions:\n\n1. Discussion on the diminishing returns of recent training-based models on some of benchmarks. What can be the causes? How could this be mitigated in the future? This will further enhance the practicability of this work.\n2. I wonder that for ProxyCLIP is the DINO encoder still kept at inference with OPMapper?\n3. Are there results on the ADE-847 and PC-459 datasets?\n\nLIMITATIONS:\nYes, authors proposed two promising directions to address their limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper tackles the problem of open-vocabulary semantic segmentation. It presents OPMapper, a lightweight, plug-and-play module that injects both local compactness and global connectivity into attention maps of CLIP. It introduces two core modules, Context-aware Attention Injection to embed spatial and semantic correlations, and Semantic Attention Alignment to iteratively align the enriched weights with textual prompts. The method shows promising performance on standard benchmarks.

STRENGTHS AND WEAKNESSES:
Strengths: 
+ The method is well motivated, and the proposed module is trained-once and applied-everywhere. 
+ Extensive experiments are conducted, and results are promising. 

Weaknesses: 
- The core task in building the CAI block is to construct an ideal attention matrix. However, the proposed method is mostly based on intuitive insights, and lacks a rigorous analysis to justify why the learned attention matrix  $W_{qk}$ can be considered  ideal. 
- The method introduces several hyperparameters (e.g., in Eq. (6) and Eq. (16)), but the robustness and sensitivity of these parameters are not analyzed. 
- Equation (9) appears to be redundant and potentially misleading, as Equation (12) is the one actually used in the computation.
- The method demonstrates smaller overall improvements when using ViT-L/14 as the encoder compared to ViT-B/16. Please provide deeper insights regarding this result.

QUESTIONS:
see weaknesses

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03