Relationship Prompt Learning is Enough for Open-Vocabulary Semantic Segmentation
Authors: Jiahaoli
Yang Lu
Yuan Xie
Yanyun Qu
Venue: NeurIPS 2024 poster (2024)
Keywords: Open-vocabulary semantic segmentation
Zero-shot semantic segmentation
Vision-Language Model
Prompt learning
Mixture-of-Expert
Primary area: machine_vision

Abstract: Open-vocabulary semantic segmentation (OVSS) aims to segment unseen classes without corresponding labels. Existing Vision-Language Model (VLM)-based methods leverage VLM's rich knowledge to enhance additional explicit segmentation-specific networks, yielding competitive results, but at the cost of extensive training cost. To reduce the cost, we attempt to enable VLM to directly produce the segmentation results without any segmentation-specific networks. Prompt learning offers a direct and parameter-efficient approach, yet it falls short in guiding VLM for pixel-level visual classification. Therefore, we propose the ${\bf R}$elationship ${\bf P}$rompt ${\bf M}$odule (${\bf RPM}$), which generates the relationship prompt that directs VLM to extract pixel-level semantic embeddings suitable for OVSS. Moreover, RPM integrates with VLM to construct the ${\bf R}$elationship ${\bf P}$rompt ${\bf N}$etwork (${\bf RPN}$), achieving OVSS without any segmentation-specific networks. RPN attains state-of-the-art performance with merely about ${\bf 3M}$ trainable parameters (2\% of total parameters).

Relationship Prompt Learning is Enough for Open-Vocabulary Semantic Segmentation
Jiahao Li', Yang Lul.2, Yuan X,5 and Yanyun Qu
'ccoool of Informatics, Xiamen University
Institute of Artificial Intelligence, Xiamen University
3 Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University
"Shoool Computer Science and Technology, East China Normal University
Chongging Institute of East China Normal University
Abstract
Open-vocabulary semantic segmentation (OVSS) aims segment unseen classes without corresponding labels. Existing Vision-Language Model (VLM)- based methods leverage VLM's rich knowledge enhance additional explicit segmentation--seccinn networks, yielding competitive results, but at the cost of extensive training cost. reduce cost, we attempt to enable VLM to directly produce the segmentation results without any segmentation-speciff networks. Prompt learning offers a direct and parameter-efficien approach, yet falls short in guiding VLM for pixel-level visual classification. Therefore, propose the Relationship Prompt Module (RPM), which generates the relationship prompt that directs VLM extract pixel-level semantic embeddings suitable for OVSS. More- over, RPM integrates with VLM to construct the Relationship Prompt Network (RPN), achieving OVSS without any segmentation-specte networks. RPN attains state-of-the-an performance with merely about 3M trainable parameters (2% total parameters).
Introduction
Open-vocabulary semantic segmentation (OVSS) aims to segment novel classes without corresponding training images, which is still challenging task in computer vision. Vision-Language Model (VLM) [5]7] has emerged powerful approach, acquiring comprehensive knowledge large-scale image-caption matching training. Several VLM-based OVSS methods [8|[0] achieve promising results. These methods employ the rich image-text representation knowledge inherent in VLM to improve segmentation performance and are categorized into two types: two-stage and one-stage methods. Two-stage methods [11 first generate image-level masks without semantics via well-designed mask proposal network 115 and then classify these masks via the image-level classification ability VLM. One-stage methods [19 20 employ a semantic decoding network to distill VLM's comprehensive knowledge from the image to the pixel level, thereby producing pixel-level segmentation results.
However, both of these VLM-based methods rely additional explicit segmentation-speen specific net- works to obtain segmentation results, resulting in extensive training cost. To reduce the training cost, an intuitive idea is to make VLM directly produce segmentation results without these segmentation-
CCorresponiing author: yxie caae@cc.ccu.dd.crr yyqu @dm..duu.c
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Layer
Layer
Layer
Layor
Layer
Layer 6
Layer
Layer
Layer 9
Layer 10
Layer 11
Figure 1: Visualization of relationship attention map m from the well-trained RPN. The degree of attention from low to high is marked by colors from dark blue to red. The more attention, the darker red; the less attention, the darker blue. As the layers deepen, the attention maps exhibit increasingly precise pixel-level semantics. The images correspond to the attention maps for an aeroplane, sheep, and a train, respectively.
specific networks] In this context, prompt learning 55 21 23
emerges as a practical approach, guiding VLM to transform image-text pair embeddings into pixel-level semantic embeddings suitable for OVSS, thereby directly achieving OVSS.
Without explicit segmentation-speen specific networks, applying prompt learning solely to VLM for OVSS straightforward yet challenging. Typically, existing prompt learning methods fine-tune task-specific models to enhance performance on that task, yet they fail to secure cross- task performance gains. These methods employ either fixed or trainable vision prompt tokens (24], or they construct complex ViT-based networks for generating prompt [25 26, 20 27 The limitations of such prompt include: 1) providing only image-level granularity, which restricts VL.Ms from performing tasks related to pixel-level visual classification, and 2) lack of an image-text relationship, which hinders the exploration of VLMs' potential for open-vocabulary scene understanding. Therefore, it is difficult for these methods to enable VLM suitable for image-level classification to achieve open-vocabulary semantic segmentation directly.
In summary, addressing the above- mentioned issue lies in refining the gran- ularity of prompt and strengthening the image-text relationship within them, By analyzing the outputs of VI.M's encoder layer, we find they can construct an image-text relationship attention map via the attention mechanism, guiding the en- coder to focus relevant pixels. Thus, we propose the Relationship Prompt Module (RPM) that utilizes the outputs of image and text encoding layers to enable pixel-level relationship prompt- ing, enhancing prompt's granularity and image-text relationship. Moreover, implement a layer-by-laye guidance mode in VLM, enabling a progressive transfer of embeddings from the image to pixel level. As illustrated Figure [ each layer's relationship attention map is continuously refined following an image-to-pixel attention scheme. To obtain the segmentation results, we propose the Linear Projection Module (LPM) comprising merely two individual linear layers, which maps the image and text feature into shared space, and then computes their Matrix product to produce the results. Finally, we propose the Relationship Prompt Network (RPN). which consists of RPM. LPM and VLM. Figure 2 shows the comparison between RPN and other OVSS methods. RPN employs VLM to directly output pixel-level predictions by prompt learning, while other methods use VI.M to assist explicit segmentation-speziin these VLMMbbased methods, VLM transfers its rich knowledge to the mask proposal network by knowledge distillation or enables the semantic decoder to output segmentation masks by feature adaptation. The key difference between RPN and existing VLM-based methods is that RPN does
Existing VLM--ased methods
Our method
Mask Proposal
Prompt Learning
Semantic Segmentation
Netwoork
VLM
VLM
Semantic Decoder
fine-tune
feature adaptation
output results
knowledge distillation
Figure 2: Our method vs existing VLM-based methods. Our method employs VLM to directly perform OVSS by prompt learning, while the other methods relies on additional explicit segmentation-speciff networks.
networks to obtain predictions. In
'To achieve the idea, some works train the VLM from scratch using pixel-level supervision, which relies on large-scale pexel-level pre-training. Note that we aim to employ parameter- efficceen fine-tune tuning method to reduce the need for additional segmentatioon-speeifi networks and large-scale pre- traininn

---

not require any explicit segmentation-speeifcc networks; it only adapts VLM to perform OVSS. Our contributions are summarized as follows:
We propose the Relationship Prompt Module (RPM), which generates pixel-level relation- ship prompt to guide VLM in transforming image-level embeddings to pixel-level ones suitable for OVSS.
We propose the Relationship Prompt Network (RPN), employing prompt learning solely to adapt VLM for OVSS without explicit segmentation specific networks.
RPN attains state-of-the-art results four public benchmarks by optimizing about 3M trainable parameters (2% of total parameters).
2
Related Works
Open-Vocabulee Vocabulary Semantic Segmentation. Open- open-vocabulary semantic segmentation [28 29
aims to leverage the knowledge from representation distributions of seen categories to classify unseen categories. Existing methods can be divided into two types: generative and discriminative. Generative methods [30 [ [ [2 (32] require the segmentation network to aware of which categories are unseen during training, while discriminative methods [TT] 8 19 33 directly transfer semantics from to unseen categories, which is more straightforward. SPNet [33] introduces the zero-shot task for the first time and proposes an end-to-end training paradigm, in which the visual embeddings composed with the uniform semantic word embeddings to obtain the semantic logits. ZS3Net [31] utilizes generative approach to project the text embeddings into visual space and generate visual embeddings for unseen categories. Subsequently, many works following the generative method be present during training the model and adopts the self-training strategy to optimize the model for classifying the unseen categories. GaCNet (30) proposes novel context-aware feature generation method based on ZS3Net, in which pixel-wise contextual knowledge can be utilized to guide the feature generation process of unseen categories. CLIP-based approaches have also made great progress. ZegFormer [1I] proposes two sub-tasks, i.c., class-agnosti grouping and segment-level zero-shot classification and presents the CLIP-based method for the first time. MaskCLIP g utilizes the frozen CLIP to make minimal adaptation by fine-tuning lightweight classifier and replacing it with that of the segmentation network. Zsseg [2] proposes a two-stage CLIP-based method, which a proposal generator is used to generate binary masks and CLIP is required to classify them. ZegCLIP [19 presents one-stage method in which CLIP directly transfers knowledge to a lightweight decoder.
have been proposed. STRICT [34] assumes that the pixels of unseen categories
Vision-Langguag Models for Vision Tasks. Vision-language models for vision tasks are optimized with large scale of image-text pair data on the internet. There are three categories: contrastive, generative, and aligned objectives. CLIP (5) first proposes the paradigm of pre- pre-trained vision-language model. DeCLIP
351 argues that CLIP is data-intensive and proposes data-efficient training paradigm. UniCL [36] combines the two data sources to build new image-text-latt field and proposes unified contrastive learning. ZeroVL (21) proposes debiased sampling to deal with biased representation distributions and a new mixup method for the image and text models. OTTER [2 uses optimal transport to find the soft label for contrastive learning and handle the problem of noisy image-text pairs.
is technique that assists in adapting CLIP-like vision- langun-a models for various visual tasks. CoOp [40] adopts trainable vectors as word prompt to adapt CLIP for vision classification. VP 41 utilizes perturbations as visual prompt.
Visual Prompt Learning. Visual Prompt Learning 37-39] 39
VPT [24 proposes trainable visual prompt to adapt each layer of the visual embeddings. UPT [38 constructs unified prompt modeling to extract trainable visual and textual prompt for adapting CLIP. MaPLe (39 adopts trainable prompt to guide both visual and textual embeddings and proposes coupling function as a bridge to build a multi-modal prompt. DenseCLIP (26) uses the contextual
information from the image to prompt the language model. Probabilistic prompt 20 applies multiple prompt sampled from probabilistic text embeddings to better understand the image. SegPrompt [27 proposes category- level prompt to improve the model's class- segmentation ability.
3

---

Patch Partition
Image Layer 1
Embeddlings
Image Layer 11
db Embbeddings
Image Layer 12
Plain Encoder
patch
patch
Frozen
Training-free
Relationshipp
Relationship Prompt
Concatenate
M
MZ2E black
ITP block
RPM
RPM for layer
LPM
APG block
for
(persnn)
(bicycle)
Glass Embeddings
Class Embedddings
{car)
Text
Text
Text
Tokenize
Layer
Layer
Layer 12
Encoder
[cow)
Figure 3: Overview of RPN. The end-to-end architecture is delineated into four principal compo- nents: 1) the frozen plain encoder, which adopts ViT architecture to encode visual knowledge with relationship prompt: 2) the frozen text encoder, which adopts CLIP encoder architecture to encode class knowledge with text templates; 3) Relationship Prompt Module (RPM), which generates relationship prompt guide the plain encoder to output pixel-Ievel semantic embeddings: 4) Linear Projection Module (LPM), which consists of two individual linear layers to output OVSS results.
3 Approach
Overview. Our objective is to adopt prompt learning to develop VLM-based OVSS method without any explicit segmentation-spenn specifiee networks, thereby reducing training cost. As shown in Figure 3 the Relationship Prompt Network (RPN) is end-to-end system comprising text, image and image-text relationship Aoompt branches. In the text branch, the text encoder inputs text to yield the class embeddings respectively. In the image branch, the plain encoder inputs images to obtain the image embeddings (i)xxd which includes the patch embeddings p E I xd and the [CLS] token g E I 1x with N representing the number of patches. Concurrently, in the image-text relationship prompt branch, the proposed Relationship Prompt Module (RPM) alongside the encoder takes the class and image embeddings to generate pixel-level relationship prompt, which is subsequently concatenated with the image embeddings to serve as the input for the next image layer. To obtain segmentation results, the last class and image embeddings are fed into the proposed Linear Projection Module (LPM) to calculate their Matrix product.
R where C and d represent the number of classes and dimension,
Relationship Prompt Module. RPM can guide the plain encoder of VLM to directly produce pixel-level semantic embeddings suitable for OVSS, due its acquisition of three distinct types of knowledge. Firstly, it acquires multi-scale vision knowledge to locate objectives at distinct scales. Secondly, its image-text relationship knowledge enables the plain encoder learn open- vocabulary semantics from text features. Thirdly, it introduces dynamic pixel-level knowledge, which adaptive for the relationship knowledge, enabling pixel-Ievel relationship prompt learning and thus transforming VLM's image-level embeddings into pixel-level ones. Therefore, RPM comprises three blocks, each dedicated capturing one the aforementioned knowledge types.
is crucial for OVSS to obtain multi- scale image embeddings locate targets of different scales. How- ever, the image embeddings main- tain consistent scale across each image layer. Therefore, we pro- pose multi-scale mixture-of-exper (M2oE) block to aggregate patch embeddings across distinct scales. As illustrated in Figure 4 M2oE com- prises gating network and several
M2oE
Expert
Expert
Up-Samplo
Up-Sample
Expert
Expert 2
Expert 3
Expert
DW
600)
G(x)
Interpolate
0.5x Interpoliate
Gating
Selected
Unselected
Figure 4: M2oE. @ and + denote matrix product and addi- tion.

---

expert networks as in 442 43 The gating network aims to dynamically activate different experts, each responsible for scaling the input to various extents. M2oE of the i-th layer is formulated as follows:
n p p
EG(')) j=1
(I)
r-d
where G(-) and E-)-) represent the gating network and the j-th expert of all n experts, respectively. Note that the patch embeddings first reduce the dimension from to = 3. And Linear() maps the
r-+d
output of each expert back to the original dimension d. See Appendix for more details on M2oE.
It is crucial for OVSS to fuse the image embeddings with open-vocabulary semantics of the class embed- dings. The key is to construct the image-text rela- tionship that bridges the class and image embeddings. To achieve this, we propose image-to-pixc seman- tic attention (ITP) block, which utilizes the image embeddings and the last class embeddings to form the relationship attention map m E R NxC, As illus- trated in Figure 5 we first calculate the Hadamard product between the [CLS] token g and the last class embeddings / after dimension alignment. Then, the Matrix product between the Hadamard product result and the patch embeddings p yields the relation- ship attention map m. Consequently, the relationship attention map m' of the i-th layer is formulated as follows:
Patch
Embeddings
p'
m
Mui
ht
[CI3] Token
Learnable Parameter
Relationship Prampt
dls
Einsum
Class Embeddingg
Expand
ITP
APG
Figure 5: ITP and APG. Expand, Einsum
and Mul denote expanding class dimension, Hadamard product and Matrix product.
m pp''(tt')) O
Intuitively, the first Hadamard product operation assigns weights to images in a batch, with their sum being one (the more important the image, the larger the weight), by fusing the class embeddings used to identify different classes and the [CLS] token used to identify each image in a batch, thus attaining image-level attention. The subsequent Matrix product operation weights the pixels, with the sum of pixel weights normalized to one (the more important the pixel, the larger the weight), by integrating the patch embeddings that contain pixel-level visual information, thus securing pixel-level attention. Therefore, we refer to the training-free -frenng--feeeeo operation (i.e., Eq. 2 as the image-to to to-pxel attention scheme, in which the first and the subsequent products extract image-level and pixel-level information, respectively. As illustrated in Figure]] | the relationship attention map construction process from the shallow the deep layer demonstrates the effectiveness of the image-to-pixe attention scheme.
The construction of the adaptive image-text relationship for each pixel enables VLM to directly output pixel-level semantic embeddings. To achieve pixel-level dynamic tuning, propose adaptive prompt generation (APG) block. As illustrated in Figure we first initialize a trainable parameter h E Xxdxd representing the dynamic pixel-level knowledge. The adaptive relationship prompt m E Cxdd for each pixel is then derived from the Matrix product between the dynamic pixel-level knowledge h and the relationship attention map m. Consequently, the adaptive relationship prompt m' of the i-th layer is formulated as follows:
m' m' hh
The role of the dynamic pixel-level knowledge is twofold: 1) it projects the relationship attention map from lower to a higher dimension to serve an input for the plain encoder, 2) it fine-tunes the relationship prompt for each pixel. It is the fine-tuning of the relationship prompt for each pixel in a high-dimensional space that enables the plain encoder to directly obtain pixel-level semanties.
In addition, we integrate RPM in parallel within each layer of VLM. The image embeddings and the last class embeddings are fed into RPM to generate the relationship prompt, which is then merged with the image embeddings and fed into the next image layer. The prompt output from the image layer is discarded. The processing of each image layer is formulated as follows:
= Layer' ([v* .mm 'I)
(4)
The notation [.] represents the concatenation operation. See Appendix C for more details RPM.
5

---

Linear Projection Module. Given that RPM enables the plain encoder to directly obtain pixel-level seman- ties, LPM aims to map the last image and class embeddings into common space and calculate their Matrix prod- uct as the segmentation results. To this end, there are three intuitive designs illustrated in Figure All three de- signs share a common structure, con- sisting of an image branch and a text branch, each equipped with a linear layer and normalization layer. The image branch processing (the blue solid line) remains consistent, with the last patch embeddings sequentially passing through the linear layer and the normalization layer. The difference lies in the text branch processing (the orange solid line). In LPM, (as shown in Figure (a)), only the last class embeddings are factored into the text branch. In LPM, (as shown in Figure (b)), the last class embeddings t and the last [CLS] token first produce the Hadamard product betore entering into the linear layer. LPMc (as shown in Figure (c)), the last class embeddings is concatenated with the Hadamard product result and then fed into the linear layer. Thus, the segmentation results O., Og and O₂ of them defined as follows:
Linear
Linear
Linear
Layer Norm
Layer
Layer Norm
Layer Norm
Layer Norm
Layer
Linear
Linear
Linear
Text Branch
Image Branch
(a) LPM,
(b) LPMp
(c) LPM
Figure 6: Three kinds of LPMs. @ and @ denote element-
wise product and matrix product.
O, Linear(p) Linear(()
O, Linear(p) Linear(1 ④ g)T
(6)
Og = Linear(p) Linear([ @ g.g.,. g))
For simplicity, the normalization layers are omitted. We select LPM, as the proposed LPM experi- mentally (as shown in Table
Optimization. There are two types of loss functions: the cross-entropy loss with the Softmar function and the combination loss between the focal loss and the dice loss with the Sigmoid function. The former employs one-hot encoding to render the class distribution as mutually exclusive in the embedding space, while the latter utilizes multi-label encoding to permit class distribution overlap. Practically, these losses are typically selected based on the used semantic decoders, such as the former with FPN 44 and the latter with transformer decoder 45). Considering that we employ LPM (comprising just two individual linear layers) to obtain the final segmentation results without loss functions. We refer to the former as the Sof tmax loss and the latter as the Sigmoid loss. We select the Sigmoid loss experimentally (as shown Table 6)..
any well-designed semantic decoders, we evaluate the aforementioned
The Sof tmax loss is formulated as follows:
hxw
Csoftmax
Uk log is
(8)
where y and ie are the ground truth and the prediction vectors, respectively, and h and w represent the height and width of the input image. The Sigmoid loss is formulated as follows:
hxw
focat
Σ a (1
9T 7. yk log(ūk)
(9)
k=1
2 2 k= 2 x IN Ui
Ldice 1
(10)
===1 DJ OJ + Ek=1 yk
hxw
Uk
Lasgmosd d X1 + >y
(11)
where a, 7, An and X2 are hyperparameters. In Eq. when y/ equals the zero vector, ys and in are substituted with 1 ys and 1 D. respectively.

---

Experiments
We evaluate our method in both zero-shot and open-vocabulary settings. See Appendix D for more details the settings.
4.1 Implementation Details
Datasets and Evaluation Metrics. ADE20K [46) consists of 25k images for training and 2k images for valida- tion. Pascal VOC (47) includes 10,582 augmented training images and 1,449 validation images. COCO- Stuff164K[48] contains 118,287 train- ing images and 5,000 validation im- ages, with 171 classes total. Pas- cal Context 49 consists of 10,100 im- ages, of which 4,996 are used for train-
Table 1: Efficiency comparison with state-of-the-art meth- ods. #Params(M) represents the total number of trainable parameters.
Methods
#Params(M)
FLOPs(()
FPS
VOC
ZegFormer
60.3
1829.3
7egCLIP [1]
13.8
110.4
9.0
RPNNours)
3.2
84.2
10.6
COCO
ZegFormer
60.3
1875.1
1.5
ing and 5,104 for validation, cover- ZegCLIP 19 RPN(ours)
14.6
123.9
6.7
3.2
101.3
10.2
ing 60 classes. We employ pixel-wise classification accuracy (pAcc) and the mean of class-wise intersection over union (mloU) for seen classes 2), unseen classes (mloU,,, and their harmonic mean (hloU).
conduct experiments on eight NVIDIA GTX 3090 GPUs using the MMSegmentation tool [50] If not specified, we employ the pre-trained CLIP VIT-B/16 model for both the plain encoder and the text encoder. We set the batch size of 4 for each GPU and set input resolution to 512 x 512 pixels. The data augmentation strategy adheres to the default settings in MMSegmentation, which includes random image resizing with a short-side range of [256, 1024] and crop size of 512 x 512. The optimizer is AdamW, initialized with learning rate of 2 x 10 and weight decay of x 10-2. The learning rate follows a polynomial decay schedule with a power of 0.9. The number of iterations is set to 20K for the VOC dataset, 80K for the COCO dataset, and 40K for the Context dataset. We set d and X2 in Eq. to 100 and 1, respectively.
Training Strategy.
4.2 System Level Comparison
Efficiency Comparison. We present an efficiency comparison with sate-of-the-ar methods in Table The results of compared methods derived from [19] To ensure fair comparison, report our results based on the open-source code from [9] and evaluate them with an input resolution of 512 512 single NVIDIA GTX 1080 Ti GPU. Our method outperforms the other methods in efficiency, achieving the lowest number of trainable parameters and the smallest FLOPs.
Comparison in the Zero-Shot Setting. We show the performance comparison with the state-of- the-art methods in the zero-shot setting in Table and conduct the comparison under three scenarios: without self- aff-trrining, with self self-training, and fully supervised. In the absence of self-training, our method surpasses FreeSeg [10] with +6.0% mIoU,, on the VOC dataset and +0.6% mloU,, on the COCO dataset. With self-training our method outperforms ZegCLIP [19] with +3.7% mIoU,, on the VOC dataset, +1.3% mloU,, on the COCO dataset, and +2.3% mloU on the Context dataset. Under the fully supervised scenario, our method exceeds ZegCLIP I9 with an average of +2.0% mloU,, across the three datasets. We attribute the modest improvement on the COCO dataset the bias from the extremely unbalanced training and validation set ratios (more training data and less validation data), which contrasts with the performance on the Context dataset (less training data and more validation data), reflecting the robust zero-shot learning ability of our method.
Comparison in the Open- Vocabulcyy Setting. We show the performance comparison with the state-of-the-art methods the open-vocabulary setting Table Our method does not require an additional training dataset. The results indicate that no method can consistently outperform others across all validation datasets; however, our method attains state-of-the-art performance on the A-847, A-150 and the PAS-20 datascts. As analyzed in [53], the Context dataset and the ADE20K dataset
7

---

Table 2: Performance comparison in the zero-shot setting (unit: %). Here, the best results shown in bold and the second results are underlined. The self-training represents applying self-training via generating pseudo labels on all unlabeled pixels like in [19 8. The symbol "f"
indicates pseudo labels are merely annotated on unseen classes pixels excluding the ignore part.
VOC
COCO
Context
Methods
pAec
mloU mloU,
hoU
pAce
mlol, mloU,
hot
pdec
mlol, mlot . blot.
self-training
ZegFormer
86.4
63.6
73.3
36.6
33.2
34.8
ZogFoonr+MAFF
91.3
85.2
36.4
38.1
ZSSeg
[2
90.0
83.5
72.5
77.5
60.3
39.3
36.3
37.8
75Seg +MAFT
51
87.1
76.1
81.2
36.1
35.9
36.0
ZegCLIP
94.6
91.9
77.8
84.3
62.0
40.2
4144
40.8
76.2
46.0
54.6
49,9
FreeSeg
10
91.9
78.6
84.7
42.4
42.2
42.3
RPN(ours)
95.8
93.1
84.6
88.6
64.4
40.8
42.8
41.8
76.4
47.7
58.7
52.6
self-training
ZegCLIP [9
95.1
91.8
82.2
86.7
68.8
40.6
54.8
46.6
77.2
46.6
65.4
54.4
MuskCLIPt g
88.8
86.1
87.4
38.1
54.7
45.0
44.4
66.7
53.3
ZegCLIPt 1
96.2
92.3
89.9
91.1
69.2
40.7
59.9
48.5
77.3
46.8
68.5
55.6
RPN+(ours)
97.1
93.1
93.6
93.3
69.3
40.6
61.2
48.8
78.3
48.1
70.8
57.3
Fully supervised
ZegCTTP [9]
96.3
92.4
90.9
91.6
69.9
40.7
632
49.6
77.5
46.3
18.7
56,9
RPN(ous))
94.0
94.3
70.8
41.1
50.5
78.7
48.5
60.4
Table 3: Performance comparison in the open-vocabulary setting (unit: %). Here, the best results shown in bold and the second-boc results underlined.
Methods
VLM
Training Set
A-847
PC-459
A-150
PC-59
PAS-20
OVSeg [14
V/T-B/16
1OCO-Stut+-CCCH Caption
7.1
11.0
24.8
53.3
92.6
CAT-Seg 52
VIT-B/16
COCO-Stuff
8.4
16.6
27.2
57.5
93.7
SAN
VIT-B/16
COCO-Stuff
10.1
12.6
27.5
53.8
94.0
SED (44
ConNNeXt-B
COCO-Stuff
11.4
18.6
31.6
57.3
94.4
RPN(ours)
V/T
COCO-Stuff
11.4
17.3
31.5
57.1
95.2
OVSeg [44
VIT-L/14
COCO-SutF=COCO Caption
9.0
12.4
29.6
55.7
94.5
CAT-Scg 52
VIT-1/14
COCO-Stuff
10.8
20.4
31.5
62.0
96.6
SAN[53]
VIT-L/14
COCO-Stuff
13.7
17.1
33.3
60.2
95.5
FC-CLIP[55
ConvNeXi-L
COCO Panoptic
14.8
18.2
34.1
58.4
95.4
SED
ConvNeXt-L
COCO-Stuff
13.9
22.6
35.2
60.6
96.1
RPN(ours)
ViT-L/14
COCO-Stuff
14.9
22.1
36.4
61.9
96.6
exhibit the highest and lowest label-set similarities with the training dataset, respectively. Therefore, our method showcases a more comprehensive gren-vocabulary learning ability.
4.3 Ablation Study
We conduct the ablation experiments on the VOC and the COCO datasets. If not specified, most are conducted on the VOC dataset. See Appendix D for more details on the experiments.
Image-to-Pixel A Attention Scheme. RPM aims to transform image-level embeddings from VLM into pixel-level semantic embeddings, enabling direct OVSS. To illuminate its functionality, we employ the Mean Attention Distance (MAD) [56] (57) information aggregated within the self-attention head. As illustrated in Figure we analyze MAD of each self-attention head during the initial and advanced stages of training. A higher point indicates larger receptive field, and greater point spacing signifies richer feature diversity. In the initial training phase, shallow and deep layer information exhibit marked differences: the former concentrates the local field with fine granularity and high diversity, while the latter focuses on the global field with coarse granularity and limited diversity. During training with RPM, deep layer information maintains attention on both local and global fields without sacrificing granularity or diversity. Clearly, deep layer information, augmented with relationship prompt learning, is more suitable for pixel-Ievel semantic segmentation tasks, thereby diminishing the need for additional addition, we visualize the relationship attention map Figure [ The first line denote the relationship attention maps for seen classes across each layer; the last two lines for unseen classes. For seen classes, the model has prior pixel-level semantic, so the relationship attention map only needs to focus on a few pixels to guide the model to make predictions for these pixels (e.g., the relationship attention map for airplane has fewer highlighted areas). For unseen classes, the model lacks corresponding semantic, so the relationship attention map needs to focus on more pixels provide the model with
metrie, reflecting the granularity of
gegmentationnsseeiiie
networks. In

---

Mean Attention Distance
Mean Attention Distance
200
202
200
head1
heed2
head1
head2
Depth
Depth
(a) Early Training
(b) Well Training
Figure Mean Attention Distance of each self-attention head.
Table 4: Impact of different modules (unit: %).. Methods without LPM represent eliminating the linear layers in LPM, ie., discarding Linear(-) in Eq 7
RPM
VOC
COCO
LPM
w/ M2oE
w/o M2oE
pAcc
mloU,
mloU,.
hloU
PAcc
mloU,
mloU,
hloU
77.1
76.3
14.3
24.1
48.3
31.8
16.4
21.6
94.7
92.1
81.2
86.3
63.3
39.2
40.3
39.7
95.1
92.4
81.7
86.7
39.5
40.3
39.9
88.8
87.3
45.3
59.6
49.3
31.3
20.3
24.6
95.6
92.9
83.4
87.9
64.1
39.7
41.6
40.6
95.8
93.1
84.6
88.6
64.4
40.8
42.8
41.8
more sufficient pixel-level semantic (e.g., the relationship attention map for sheep highlights the complete semantic shallow layers).
Impact of Different Modules. Table shows the impact of various modules. RPM (the first line) denotes the combination of M2oE, ITP and APG. RPM without M2oE (the second line) denotes the ablation about APG and ITP. Utilizing RPM and LPM yields the best performance. The performance improvement attributed to RPM (+62.6% is significantly greater than that of LPM (+35.5%). Furthermore, integrating LPM on top of RPM yields a modest performance gain of 1.9%. In contrast, incorporating RPM based on LPM results in a substantial 29% improvement. Therefore, we conclude that VLM with relationship prompt learning is enough for OVSS without any explicit segmentation-sseecfic networks.
Exploration of Different Designs in RPM. Firstly, we explore vari- ous designs of the ITP block. As discussed in Section 3 removing the patch embeddings, which contain pixel-level visual information, results in loss of pixel-level attention for ITP. Conversely, omitting the [CLS] token, responsible for identifying each image, results a loss of image- level attention. To assess the sig- nificance of the image-to-pixel atten- tion scheme, we define the 'without pixel-level attention' and the 'without image-level attention' scenarios by ex- cluding the patch embeddings p' and the [CLS] token g" from Eq. [2] re- spectively. Note that employing the 'without lixel-level attention' scenario necessitates altering the dimension of
Table 5: Ablation study on different designs in RPM (unit: %).
Attention
pAce
mloU,
mloU,.
hloU
w/o image-level
73.1
74.2
21.4
33.2
w/o pixel-level
87.7
77.6
68.1
72.5
w/ both
95.8
93.1
84.6
88.6
Modes
pAcc
mloU,
mloU,
hloU
APG
nn.Lincar
95.1
92.1
80.6
86.0
Parameter
95.8
93.1
84.6
88.6
Dimension
pAcc
mloU.
mloU,.
hloU
95.8
93.1
84.6
88.6
93.4
84.5
88.7
M2oB
93.8
84.3
88.8
Modes
pAce
mloU,
mloU,
hloU
Multi-Scale M2oB
94.9
92.8
84.1
88.2
95.8
93.1
84.6
88.6

---

the trainable parameter h' from I Nxd to Rdxd The results presented in Table 5 (TTP) corroborate the efficacy of the image-to-pixel attention scheme. Secondly, we explore different configurationss of the APG block and evaluate two distinct trainable modes: nn.linear and nn. Parameteer The outcomes in Table 5 (APG) advocate for the implementation of the nn. Parametee mode, Thirdly, we explore the dimension of M2oE and different multi-scale aggregation modes in Table (M2oE). The results show that performance enhancements are marginal with increasing dimensions. Given the trade-off' between performance gain and parameter increase, we adopt = 3. The 'Multi-Scale' mode refers to removing the gating network and directly aggregating the features processed by all expert networks (See Appendix
B for more details).
Exploration of Different Designs in LPM. We evaluate three different LPM designs (shown in Figure (6) using Softmax and Sigmoid losses. The results in Table G reveal that
Table 6: Ablation study on three different LPMs using Softmax and Sigmoid losses (unit: %).
LPM. LPM,
LPM.
pAce
mloU,
mloU.
hloU
Using Sof tmax loss
70.6
67.3
19.6
30.3
LPM, with Sigmoid loss is the most effective strategy. In addition, using Sigmoid loss is significantly better than using Sof tmax loss. This reflects that the relationship prompt does not directly focus on pixels, but follows the to-ge-to-piell process.
89.4
88.6
55.5
68.2
90.4
89.5
59.3
71.4
Using Sigmoid loss
87.5
84.6
61.4
71.2
95.3
93.1
83.6
88.1
95.8
93.1
84.6
88.6
Impact of Training-Fres Projec- tion Modules. Note that our method comprises two trainable modules: RPM and LPM. To further explore the performance of RPM, we eliminate all linear layers in LPM to construct three variants of training-free free projec-
Table 7: Ablation study on three different training-free projection modules (unit:%).
TFPM. TFPM. TFPM.
pAcc
mloU.
mloU.
hloU
86.8
83.5
58.7
68.9
V
94.5
92.4
77.6
84.4
95.1
92.4
81.7
86.7
tion networks. Following the sequence depicted in Figure 6 we denote the training- free projection modules TFPM.. TFPM, and TFPM.- The results in Table demonntrate that both TFPM, and TFPM, can achieve the state-of-the-art performance. The suboptimal results with TFPM, suggests that the relationship prompt guides the plain encoder to perform lixel-level classifications by encoding semantics in the [CLS] token, which should not be disregarded when obtaining segmentation results.
5
Conclusion
In this work, we propose the Relationship Prompt Module (RPM) to guide VLM to transform its image-level embeddings into pixel-level semantic ones. RPM and VLM combine to form Relationship Prompt Network (RPN), VLM-based OVSS method that directly performs OVSS without any explicit segmentation-sppecife networks. To the best of our knowledge, we are the first to give a straightforward solution for OVSS that applies prompt learning solely to VLM. We evaluate our method on four public benchmark datasets in both zero-shot and open-vocabulary settings, and achieve the -tate-of-the-art performance with only about 3M trainable parameters (2% of total parameters). Therefore, it is concluded that VLM with relationship prompt learning is enough for open-vocabulary semantic segmentation without any explicit eggmenatiionsspeiif networks.
Limitations. Although we meticulously design an effective prompt learning method for directly using VLM to achieve pixel-level OVSS, there are several ways for prompt learning to achieve further improvemen: 1) directly acting on the attention map (more direct); 2) dynamically reorganizing multi-head attention map (more lightweight).
Acknowledgements.
This work supported by the National Natural Science Foundation of China (No. 62176224, 62176092, 62222602, 62306165, 62106075, 62476090, 62376233, 62431004); Natural Sci- ence Foundation of Shanghai (23ZR (23ZR142044) 1420400); Natural Science Foundation of Chongqing Natural Science Foundation of Fujian Province under Grant 2024J09001; China Computer Federation (CCF) Lenovo Blue Ocean Research Fund; China Academy of Railway Sciences No.2023Y1357; Xiaomi Young Talents Program.
INO.CSTB2023NSCQQ-OOO
10

---

References
[1] Shuting Hc, Henghui Ding, and Wei Jiang. Primitive generation and semantic-related alignment for universal zero-shot segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11238-11247, 2023.
[2] Hang Zhao, Xavier Puig, Bolei Zhou, Sanja Fidler, and Antonio Torralba. Open vocabulary scene parsing. In Proceedings of the IEFE International Conference on Computer Vision, pages 2002-2010, 2017.
[3] Peike Li, Yunchao Wei, and Yi Yang. Consistent structural relation learning for zero-shot segmentation. Advances in Neural Information Processing Systems, 33:10317- 15:1017-10027, 2020.
[4] Feihong Shen, Jun Liu, and Ping Hu. Conterfactual generative zero-shot semantic segmentation. arXiv preprint ar0iv:2106.00. 06360, 2021.
[5] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on Machine Learning, pages 8748-8763. PMLR, 2021.
[6] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le. Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling visual and vision- representation learning with noisy text supervision. In International conference on machine learning, pages 4904-4916. PMLR, 2021.
[7] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages (6638-16500, 2022.
[8] Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference on Computer Vision, pages 696-712 Springer, 2022.
[9] Xinyu Liu, Beiwen Tian, Zhen Wang, Rui Wang, Kehua Sheng, Bo Zhang, Hao Zhao, Guyue Zhou. Delving into shape-aware zero-shot semantic segmentation. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pages 2999-3009, 2023.
[10] Jic Qin, Jie Wu, Pengxiang Yan, Ming Li, Ren Yuxi, Xuefeng Xiao, Yitong Wang. Rui Wang. Shilei Wen, Xin Pan, et al. Freeseg: Unified, universal and open-vocabulary image segmentation. In Proceedings of the IEED/CVF Conference Computer Vision and Puttern Recognition, pages 19446-19455, 2023.
[11] Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11583- 11592, 2022.
[12] Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. simple baseline for open- ppnn-vocabulary semantic segmentation with pre-trained vision-language model. In European Conference on Computer Vision, pages 736-753. Springer, 2022.
[13] Seonghoon Yu, Paul Hongsuck Seo, and Jeany Son. Zero-shot referring image segmentation with global- local context features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19456-19465, 2023.
[14] Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabular semantic segmentation with mask-adaptee clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7061-7070, 2023.
[15] dlang-Chich Chen, Yukun Zhu, George Papandrcou, Florian Schroff, and Hartwig Adam. Encoder-decode with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 801-818, 2018
[16] Changqian Yu, Jingbo Wang, Changxin Gao, Gang Yu, Chunhua Shen, and Nong Sang. Context prior for seene segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 22416-1225, 2020.
[17] Wenwei Zhang, Jiangmiao Pang. Kai Chen, and Chen Change Loy. K-net: Towards unified image segmentation. Advances in Neural Information Processing Systems, 34:10326-- 2021.
[18] Bowen Zhang, Zhi Tian, Quan Tang, Xiangxiang Chu, Xiaolin Wei, Chunhua Shen, et al. Segvit: Semantic segmentation with plain vision transformers. Advances in Neural Information Processing Systems, 35:4971- 4982, 2022.

---

[19] Ziqin Zhou, Yinjie Lei, Bowen Zhang, Linggiao Liu, and Yifan Liu. Zegelip: Towards adapting clip for zero-shot semantic segmentation. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11 11175-11185, 2023.
[20] Hyeongjun Kwon, Taeyong Song, Somi Jeong, Jin Kim, Jinhyun Jang, and Kwanghoon Sohn. Probabilistic prompt learning for dense prediction. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 6768-6777, 2023.
[21] Quan Cui, Boyan Zhou, Yu Guo, Weidong Yin, Hao Wu, Osamu Yoshie, and Yubo Chen. Contrastive vision-language pre- training with limited resources. In European Conference on Computer Vision, pages 233-2533. Springer, 2022.
[22] Bichen Wu, Ruizhe Cheng, Peizhao Zhang, Peter Vajda, and Joseph E Gonzalez. Data efficient language- supervised zero-shot recognition with optimal transport distillation. arXiv preprint arXiv:2 2021.
[23] Timo Lüddecke and Alexander Ecker. Image segmentation using and image prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7086-7096, 2022.
[24] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pages 709-727. Springer, 2022.
[25] Lingbo Liu, Jianlong Chang, Bruce XB Yu, Liang Lin. Qi Tian, and Chang- Ween Chen. Prompt-matched semantic segmentation. arXiv preprint arXiv:2208. 10159, 2022.
[26] Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context-aware prompting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2022.
[27] Muzhi Zhu, Hengtao Li, Hao Chen, Chengxiang Fan, Weian Mao, Chenchen Jing, Yifan Liu, and Chunhua Shen. Segprompt: Boosting open-world segmentation via category-level prompt learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 999-1008, 2023.
[28] Jiaxin Cheng, Soumyaroop Nandi, Prem Natarajan, and Wael Abd-Almageed. Sign: Opatial-informan incorporated generative network for generalized zero-shot semantic segmentation. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 9556-9566, 2021.
[29] Donghyeon Back, Youngmin Oh, and Bumsub Ham. Exploiting a joint embedding space for generalized zero-shot semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9536 056669955,111 2021.
[30] Zhangxuan Gu, Siyuan Zhou, Li Niu, Zihan Zhao, and Liqing 7hang. Context-aware feature generation for zero-shot semantic segmentation. In Proceedings of the 28th ACM International Conference on Multimedia, pages 1921-1929, 2020.
[31] Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. Zero-shot semantic segmentation. Advances in Neural Information Processing Systems, 32, 2019.
[32] Weijia Wu, Yuzhong Zhao, Mike Zheng Shou, Hong Zhou, and Chunhua Shen. Diffumask: Synthesizing images with pixcl-level annotations for semantic segmentation using diffusion models. arXiv preprint arXiv:2303. 1 1681, 2023.
[33] Yongqin Xian, Subhabrata Choudhury, Yang He, Bernt Schiele, and Zeynep Akata. Semantic projection network for zero-and and lewelabel semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2256-8265.. 2019.
[34] Giuseppe Pastore, Fabio Cermelli, Yongqin Xian, Massimiliano Mancini, Zeynep Akata, and Barbara Caputo. A closer look at self-training for zero-label semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2693-2702, 2021.
[35] Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjic Yan. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. arXiv preprint / 10,00500,, 2021.
[36] Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Unified contrastive learning in image-text-lan label space. In Proceedings of the IEEE/CVF Conference on Computer 2022.
Vision and Pottern Recognition, pages
12

---

[37] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16816-16825, 2022.
[38] Yuhang Zang. Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Unified vision and language prompt learning. arXiv preprint arXi::2110007225, 2022.
[39] Muhammad Uzair Khattak, Hanoona Rashced, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 19113-19122, 2023.
[40] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-lange -ainugla models, International Journal of Computer Vision, 13009) 2022.
[41] Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint ariiv: 2022.
[42] Zihan Zhong. Zhiqiang Tang, Tong He, Haoyang Fang, and Chun Yuan. Convolution meets lora: Parameter efficient finetuning for segment anything model. arXiv preprint arXiv:2401.17865 2024.
[43] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Lee, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gate gatteely-gatdd mixture-of-experts layer. arXiv preprint arXiv:1701.0633 2017.
[44] Tsung-Y Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 21 17-2125, 2017.
[45] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is all you need for semantic segmentation. Advances in Neural Information Processing Systems, 14:17884477775 2021.
[46] Bolci Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k datasct. International Journal of Computer Vision, 127:302-321, 2019.
[47] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http:/www.pascall
[48] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Computer Vision and Pattern Recognition (CVPR), 2018 IEEE conference on. IEEE, 2018
[49] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam- Gyu Cho, Seong- Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
[50] MMSegmentation Contributors. MMSegmentatiinn: Openmmlab semantic segmentation toolbox and benchmark. https:/// com/ open mm ab/ www.aaaammmmmmtattiiooooo 2020.
[51] Siyu Jiao, Yunchao Wei, Yaowei Wang, Yao Zhao and Humphrey Shi. Learning mask-aware clip representations for zero-shot segmentation. Advances in Neural Information Processing Systems, 36:35631- 35653, 2023.
[52] Scokju Cho, Heescong Shin, Sunghwan Hong, Scungjun An, Scungjun Lee, Anurag Amab, Paul Hongsuck Seo, and Seungryong Kim. Cat-seg: Cost aggregation for open- vocabulary semantic segmentation. arXiv preprint arXiv:2303.1 1179,, 2023.
[53] Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open- vocabulary semantic segmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2945-2954, 2023.
[54] Bin Xie, Jiale Cao, Jin Xie, Fahad Shahbaz Khan, and Yanwei Pang. Sed: A simple encoder-decoder for open- vocablocy semantic sogmentation. arXiv preprint 2r..5.2311.15537, 2023.
[55] Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chieh Chieh Chen. Convolutions die hard: Open- vocabulary segmentation with single frozen convolutional clip. Advances in Neural Information Processing Systems, 36, 2024.
13

---

[56] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai. Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 11999, 2020.
[57] Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in neural information processing systems, 34:12116-12128, 2021.
[58] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 00665500..09685. 2021.
[59] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Language-driver semantic segmentation. arXiv preprint vrXiv:2201005455 2022.
[60] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Scaling open-vocabulary image segmentation with image-level labels. In European Conference on Computer Vision, pages 540-557. Springer, 2022.
[61] Kunyang Han, Yong Liu, Jun Hao Liew, Henghui Ding, Jiajun Liu, Yitong Wang. Yansong Tang. Yujuu Yang. Jiashi Feng, Yao Zhao, et al. Global knowledge calibration for fast open-vocabulary segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 797-807, 2023.
[62] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9404-9413, 2019.
[63] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18134-18144 2022.
[64] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt- dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguisties ( (Volume 1: Long Papers), pages 2018.
[65] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfee100m: The new data in multimedia research. Communications of the ACM, 59(2):44 73, 2016.
[66] Jordi Tuste, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. Connecting vision and language with localized narratives. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part V 16, pages
Springer, 2020.
[67| Zheng Ding, Jicke Wang, and Zhuowen Tu. Open-vocabulary panoptic segmentation maskclip. 2022.
[68] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang. and Shalini De Mello. Open- vocabulary panoptic segmentation with text-to-image diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2955-2966, 2023.
[69] Xudong Wang. Shufan Li. Konstantinos Kallidromitis. Yusuke Kato, Kazuki Kozuka, and Trevor Darrell. Hierarchical vee-bvocabllary universal image segmentation. Advances in Neural Information Processing Systems, 36, 2024.
[70] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple efrfmeter-efficient fine-tuning for transformer- based masked language-modet models. arXiv preprint arXiv:2106. 2021.
[71] Neil Houlsby. Andrei Giurgiu. Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790-277000 2799. PMLR, 2019.
[72] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Lst: Ladder side-tuning for parameter and memory efficient transfer learning. Advances in Neural Information Processing Systems, 15:12991-13005, 2022.
[73] Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Scaling & shifting your features: A new baseline for efficient model tuning. Advances in Neural Information Processing Systems, 35:109 123. 2022.
[74] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages (6000-16000 2022.
14

---

Appendix
A Our Structure Existing Structures
We show the comparison of our method and the existing VLM-based methods in Figure In our method, we abandon the explicit segmentation-speiiff networks, i.c. the mask proposal network and the semantic decoder and guide VLM to directly output the segmentation results. existing VLM- based methods, the entire OVSS framework consists of VLM, the mask proposal network and the semantic decoder, incurring extensive training cost. As illustrated in Section[]] the two-stage methods usually adopt knowledge distillation to transfer zero-shot learning ability to the segmentation-spect network, and the one-stage methods utilize feature adaptation to train the semantic decoder. In summary, prompt learning Conversely, the existing VLMMbased methods, VLM acts as the engine to drive the segmentation- specific network indirectly achieve OVSS. Therefore, our method is more straightforward and more parameter-efficient.
the engine drive VLM directly achieve OVSS in method.
Existing VLM-based methods
Our method
Proposai
Prompt
Semantic Segmentation
Sulti-Scale
Learning
VLM
Expert 1
Expert
Expert
Expert
feature adaptation
output results
knowledge distillation
Figure 8: Our method existing VLM-based methods.
Figure 9: Multi-Scale.
B M2oE
Details The role of the gating network to select the right expert for each sample batch. Thus, we need to calculate the gating scores G(z) Bx i.e., the scores for B samples in batch to experts. For simplicity, let the input, i.e., the patch embeddings be denoted by x E IR Bxxxd where N h x w. The input æ is first directed into a common linear reduce the dimension
and then is reshaped to the size of R Bxr after global average pooling (as Eq. 12 Like in 42 the value (z)) E R Bxn is calculated as follows:
43
Fa
(12)
d
H(x)= (x. Wg) + StandardNormal() Softplus(xa Wnoise)
(13)
where W, E Rrxa and W Wnoise E gr represent the trainable gating weight and the noise term, respectively. We select only the top values (xx) and set the rest to -o0. operation, the gating scores G(x) can be calculated as follows:
After 'Softmax'
G(x) = Softmax(KeeTon)(((((()
(14)
The role of the experts extract feature with multi-scale, which consist of Interpolate operation at specific scale, DWConvaxa and Upsample operation to map the feature to the original scale. Let i-th expert be E, with scale 8; z where 1, 1,2,3, 4. The expert E, processes the input x as follows:
Ex(x) = DD (L)near()).))
(15)
d-
Finally, obtain the output according to Eq.
An intuitive alternative applying a multi-scale strategy, which utilizes n branches to extract features with different scales. 'hee structural difference with M2oE is its lack of the gating network, as illustrated in Figure 9 The absence brings a higher parameter consumption, due the fact that the gating network helps selectively activate sparse experts.
Multi-Scale
15

---

image Weights
Entredingg
Prompe
Weights
(LEamaate) Paramittr
Weights
A
Image Weights
Weights
das pnoredngg
Extuu
Image Weights
Text Weights
Experd)
Trainfret free
(n) (b) VLM LoRA
Relationship Prompt
(d)
Figure 10: Relationship prompt. (a) LoRA. (b) VLM LoRA in common-bypan mode. (c) VLM relationship prompt tuning. (d) Details of A and B. Note that A and B refer ours IPT and APG.
C Motivation behind the design of RPM
The motivation behind the design of RPM inspired by LoRA (58]. As illustrated in Figure 10 (a), considering that the main components of large models lie in a low intrinsic dimension, LoRA introduces dimensionality reduction matrix and dimensionality enhancement matrix into bypass to fit the frozen weights. This approach enables fine-tuning the model with minimal training costs, thereby improving performance for specific tasks without changing the original parameters. Given that VLM has two branches to process image and text data, we initially adopt common- byyaass mode as shown in Figure (b). The distinction between LoRA and VLM LoRA lies in the fact that matrix A in VLM LoRA accepts two types of modal inputs and it is training- free. It is important to note that we do not connect the output of matrix B with the embeddings from the text module for two reasons: their dimensions do not match and the relationship attention map from matrix A performs more effectively as a visual prompt rather than a textual prompt, as illustrated in Figure | However, the performance of implementing VLM LoRA was suboptimal. We discovered that direct skip connection with the output of the current image weights that bypasses matrices A and B, achieves the same effect. Our analysis suggests that merely tuning the lightweight matrix B in post-tuning model is found to be insufficient for adapting parameter-intensive weights. Consequently, we propose a prefix-tuning architecture as illustrated in Figure [0 (c). We position the two matrices between the preceding and current weights and established a skip connection with the original output of the preceding image weights rather than the current image weights. This circumvents the need fit the parameter- intensive patesiver-intensive image weights. In light of this prefix-tuning architecture, we categorize this model into the category of prompt learning rather than adaptation, and thereby naming relationship prompt learning. In addition, we consider three operations between the output of the preceding image weights and the output matrix B: concatenation, addition, and Hadamard product. Following a quantitative analysis (shown in Table we select the concatenation operation.
D More Details about Experiments
Zero-Shot Semantic Segmentation Setting We adopt the popular zero-shot setting follows in[13] 9 3 (1). In the setting, we divide all classes into seen and unseen, only the seen classes are used training. For the VOC dataset, we select 15 seen classes and unseen classes, with the "background' class excluded. For the COCO dataset, we divide all classes into 156 seen classes and 15 unseen classes. For the Context dataset, we select 50 seen classes (including 'background") and 10 unseen classes. These unseen classes are defined as follows:
Dataset
Unseen Classes
VOC
pottedplant, sheep, sofa, train, tymonitor
cow, giraffe, suitcase, frisbee, skateboard carrot, scissors, cardboard, clouds, grass playingfield, river, road, tree, wall concrete cow, motorbike, sofa, cat, boat, fence bird, tv monitor, keyboard, aeroplane
COCO
Context
Open- Open-Vocabulae Vocabulary Semantic Segmentation Setting We employ the open-vocabulary setting as follows in [4
51 53 In this setting, we train our model on the COCO dataset and evaluate it on other datasets, including the VOC dataset with classes (PAS-20), the Context dataset with 59 and 459 classes (PC-59 and PC-459), the ADE20K dataset with 150 and 847 classes (A-150 and A-847).
12
16

---

Text Prompt Templates For the VOC dataset, we apply single template 'A photo of {}'. For the Context and COCO dataset, we apply multiple templates follows:
photo of a. "A photo of small [ photo medium
photo of large
"This is photo of a (). "This photo of small II." "This is photo of medium (). "This is photo of large []."
"A {} in the scene." 'A photo of [} in the scene."
"There a ()
scene." "There "This is the {} in the scene." 'This is one {} the seene."
() scene."
"This is () in the scene."
Full Experiment Results We show the full performance comparison of existing methods in the zero-shot and apen-vocabulary semantic segmentation settings in Table 8 and Table 9
Comparison with PEFT Methods We explore some parameter-efficien fine-tuning (PEFT) meth- ods with baseline (i.e., the CLIP with LPM): 1) fine-tuning entire baseline; only fine-tuning LPM; 3) BitFit [70], a sparse-fine- fine-tuning method where only the bias-terms of the model (or subset of them) are modified; 4) Adapter [7]], which inserts a trainable adapter module between the transformer layers. 5) VPT (24), which inserts trainable tokens to the input feature of each transformer layer; 6) LST [72), which trains a ladder-side network, a small and separate network that takes intermediate activations as input via shortcut connections (called ladders) from the backbone networks and makes predictions. 7) SSF (73), which trains scale and shift parameters to modulate the visual features. 8) LoRA 58 which inserts trainable dimensionality reduction matrix and dimensionality enhancement matrix parallel the frozen weights. As illustrated Table TO our method shows significant i improvements compared to PEFT methods. This is attributed to the fact that these PEFT methods only focus on how to train the baseline with fewer parameters, suitable only for image-level feature representation tasks, while our method not only trains the baseline with fewer parameters, but also focuses on applying capabilities suitable for image-level feature representation tasks directly to pixel-level semantic segmentation tasks. In summary, existing PEFT methods mainly focus on fine-tuning the task task-specific model to improve performance on that task, while our method enables VLM to directly perform semantic segmentation.
Combination with VPT We explore the combination our method and VPT in Table TT] When using VPT alone and in combination with our method, the number trainable tokens each layer is 400 and 40 respectively. Considering the impact of parameter initialization in VPT, the performance change after the combination is negligible.
Table 8: Full performance comparison in the zero-shot setting (unit: %). Here, the best results are shown in bold and the second-best results are underlined.
VOC
COCO
Contest
Methods
pAee mlnU,
mloU,
hall
pAce
mloli, mlnU, hall
pAce
minUi, mloti, hlo
w/o self-training
SPNet
33
78.0
15.6
26.1
35.2
8.7
14.0
ZS3Net
31
77.3
17.7
28.7
34.7
15.0
52.8
20.8
12.7
15.8
CaGNet
30
80.7
78.4
26.6
39.7
56.6
33.5
12.2
18.2
24.1
18.5
21.2
SIGN
28
75.4
28.9
41.7
32.3
15.5
20.9
Joint
29
77.7
32.5
45.9
33.0
14.9
20.5
Zormorerr
86.4
63.6
73.3
36.6
33.2
34,8
7eg Fomer-MFFAAFT
5
91.5
80.7
85.7
36.4
40.1
38.1
ZSSeg
[1]]
90.0
83.5
77.5
60.3
39.3
37.8
7S5eg +MAFT
87.1
76.
81.2
36.1
35.9
36.0
ZegCLIP
19
94.6
91.9
77.8
84.3
62.0
40.2
41.4
40.8
76.2
46.0
54.6
49.9
FreeSeg
91.9
78.6
84.7
42.4
42.2
42.3
RPN(ours)
95.8
93.1
84.6
88.6
64.4
40.8
42.8
41.8
76.4
47.7
58.7
52.6
w/ tell-training
SPNct
77.8
25.8
38.8
34,6
26.9
30.3
ZS5Net
31
78.0
21.2
33.3
34,9
10.6
16.2
49.5
27.0
20.7
23.4
CaGNer
81.6
78.6
30.3
43.7
35.6
13.4
19.5
STRICT
82.7
35.6
49.8
35.3
30.3
34.8
DiffMask
33
71.4
68.1
75Seg
88.7
79.2
78.1
79.3
63.8
39.6
43.6
41.5
ZeyCLIP
L9
95.1
91.8
82.2
86.7
68.8
40.6
54.8
46.6
77.2
46.6
65.4
54.4
MaskC1.1 IPF
88.8
86.
87.4
38.
54.7
45.0
44.4
66.7
53.3
ZegCLIPI
19
96.2
92.3
89.9
91.1
69.2
40.7
59.9
48.5
77.3
46.8
68.5
55.6
(PNY(ours)
97.1
93.1
93.6
93.3
69.3
40.6
61.2
48.8
78.3
48.1
70.8
57.3
fully supervised
ZegCLIP [9
96.3
92.4
90.9
91.6
69.9
40.7
63.2
49.6
77.5
46.5
78.7
56.9
RPN(ours)
97.2
94.0
94.3
70.8
41.1
50.5
78.7
48.5
60.4

---

Table Full performance comparison in the open-vocabulary setting (unit: %). Here, the best results shown in bold and the second-best results underlined.
Methods
VLM
Training Set A-347
PC-459
A-150
PC-59
PAS-20
SPNct 133
VOC
24.3
18.3
ZS3Net 31
VOC
19.4
38.3
LSeg
VIT-B/32
VOC-15
47.4
LScg+ 60
ALIGN
COCO-Stuff
2.5
5.2
13.0
36.0
Han ef al. 61
VIT-B/16
COCO Panoptic
04
3.5
7.1
18.8
45.2
83.2
ViT 63
VIT-S/16
GCC (64)+
4.3
4.9
10.6
25.9
50.7
ZegFormer
VIT-B/16
COCO-Stuff
5.6
10.4
18.0
45.5
89.5
OpenSeg 60
ALIGN
COCO Punoptic+Loc. Narr.
4.4
7.9
17.5
40.1
FreeSeg [0
COCO-Stuff
7.1
6.4
17.9
34.4
85.6
FreeSeg+MAFT
S1
COCO-Stuff
10.1
12.8
29.1
53.5
90.0
OVSeg 14
VII-B/16
COCO-Stutf+COOC Caption
7.1
24.8
53.3
92.6
CAT-Scg
VIT-B/16
COCO-Stuff
8.4
57.5
SAN 53
VIT-B/16
COCO-Stuff
10.1
12.6
27.5
53.8
94.0
SED 54
ConvNcXt-B
COCO-Stuff
11.4
18.6
31.6
57.3
94.4
RPN(ours)
VIT-B/16
COCO-Stuff
11.4
17.3
31.5
57.1
95.2
LSeg
VrT-B/32
VOC-T5
52.3
OpenSeg M
ALIGN
COCO Panopticel Loc. Narr.
8.1
11.5
26.4
44.8
OVSeg 14
ViT-L/14
COCO-Stul++COCS Caption
9.0
12.4
29.6
55.7
94.5
Ding
VIT-1/14
COCO Panoptic
8.2
10.0
23.7
45.9
ODISE
ViT-1/14
COCO Panoptic
$7.3
6I
BERT-B
COCO Panoptic
29.0
59.3
CAT-Segle
VIT-1/14
COCO--Stuff
10.8
20.4
31.5
62.0
96.6
SAN
ViT-L/14
COCO-Stuff
13.7
17.1
33.3
60.2
95.5
FC-CLIPES5
Conv-NeXt-L
COCO Pamoptic
14.8
58.4
SED S4
Co
COCO-Stu StCOfStuff
13.9
22.6
35.2
60.6
96.1
RPN(ours)
VIT-L/14
COCO-Stuff
14.9
22.1
36.4
61.9
96.6
Table 10: Performance comparison with PEFT methods (unit: %)..
Baseline represents the CLIP model with LPM. #Params(M) represents the number of trainable parameters during training.
VOC
COCO
Methods
#Params(M)
pAce
mloU,
mloU.
hlT
pAce
mloU,
mloU,
hloU
Baseline
154.5
84.1
83.5
31.2
45.4
47.8
30.1
19.6
23.7
LPM-only
0.8
88.8
87.3
45.3
59.6
49.3
31.3
20.3
24.6
BitFit[70
4.0
89.7
79.3
51.2
62.2
50.6
35.6
23.3
28.2
Adapter 71
3,9
90.3
79.7
51.6
62.6
51.4
36.0
24.1
28.9
2P
4.0
90.9
81.0
52.9
64.0
51.9
37.5
25.9
30.6
LST
11.5
88.6
78.7
50.4
61.4
50.1
34.8
27.4
$SF
4.4
90.8
80.8
52.7
63.8
51.7
37.3
25.6
30.4
LoRA 58
4.0
91.3
82.2
53.1
64.5
52.9
38.7
27.0
31.8
RPN(ours)
3.2
95.8
93.1
84.6
88.6
64.4
40.8
42.8
41.8
Table 1I: Performance of combining our method and VPT (unit: %). #Params(M) represents number of trainable parameters during training.
VOC
COCO
Methods
#Params(M)
pAce
mloU,
mloU,
hloU
pAce
mloU,
mloU,
hloU
VPT
4.0
90.9
81.0
52.9
64.0
51.9
37.5
25.9
30.6
RPN(surs)
3.2
95.8
93.1
84.6
88.6
64.4
40.8
42.8
41.8
RPN+VPT
3.6
95.8
93.1
84.2
88.4
64.4
40.3
43.1
41.7
Table 12: Impact of different pre-trained weights for the plain encoder (unit: %).
VOC
COCO
Weights
pAce
mloU
mloU
hloU
pAcc
mloU,
mloU
hloU
ViT [56]
85,9
81.0
42.9
56.1
46.7
29.3
19.9
23.7
MAE 74
87.1
82.4
43.7
57.1
48.2
30.8
21.5
25.3
CLIP 5
95.8
93.1
84.6
88.6
64.4
40.8
42.8
41.8
18

---

Table 13: Impact of RPM in different layers (unit: %).. 'Number' represent the number of layer which RPM is applied. Note that 'all' do not include the last layer.
Datasets
Number
pAcc
mloU.
mloU,.
hloU
93.1
88.4
74.1
80.6
11
92.8
88.7
70.3
78.4
VOC
(1.3,5.7.9.11)
93.9
89.4
80.9
84.9
(2,4,6,8,10)
93.9
89.1
80.1
84.4
all
95.8
93.1
84.6
88.6
1
60.2
39.8
35.8
37.7
11
60.1
39.4
33.6
36.3
COCO
(1.3,5.7.9.11)
61.9
40.2
38.5
39.3
(2,4,6.8,10)
61.9
40.1
38.4
39.2
all
64.4
40.8
42.8
41.8
Table 14: Ablation study three kinds of prefix-tuning operations (unit: %).
Datasets
Mul
Add
Cat
pAcc
mloU,
mloU,
hloU
88.8
87.3
45.3
59.6
VOC
90.3
83.2
69.5
75.7
95.8
93.1
84.6
88.6
49.3
31.3
20.3
24.6
COCO
56.8
36.6
25.7
30.2
64.4
40.8
42.8
41.8
Impact of Different Pre-trained Weights for the Plain Encoder We explore different pre- pre-trained weights the plain encoder in Table 12 These weights are from supervised learning, unsupervised learning and weakly-supervised learning of text signals. ViT [56] applies supervised learning to learn visual representation. MAE 774 applies self-supervised learning learn rich semanties. The two methods learn uni-modal knowledge. CLIP(5) applies text as supervisory signal and adopts contrastive learning to learn general visual representation. It multi-modal learning method. The results show that our method has limitations in mining uni-modal visual knowledge. In addition, compared to ViT and MAE, CLIP uses more data to pre-train the plain encoder. This also shows that even though our method is superior, it still cannot get rid of the dependence large amount of data.
Impact RPM in Different Layers We adopt a layer-by-layer guidance mode (i.e., all) as shown in Table [3 Our method is evaluated across various network depths, including shallow layers, deep layers, and interval layer mode. We find that the performance of single-layer mode (such as layers or 11) are significantly worse than those of multi-layer modes. The layer-by-layer mode demonstrates optimal performance on both the VOC and COCO datasets.
Impact of Different Prefix-tuning Operations We compare three prefix-tuning operations as shown in Figure 10 (c): concatenation, addition, and Hadamard product. Results from Table [4] indicate that the relationship prompt should not directly modify the original feature (like addition or Hadamard product), but rather should influence the model indirectly by computing the attention between the prompt and the original feature (like concatenation).
E
Visualization
Baseline represents the frozen CLIP model with LPM.
19

---

(a) Image
(b) Ground truth
(c) Ours
(d) Baseline
Figure 11: Qualitative analysis. The unseen classes include tree frishee grass
and
road
20

---

(a) Image
(b) Ground truth
(c) Ours
(d) Baseline
Figure 12: Qualitative analysis.
The unseen classes include tree clouds
skateboard
playingfield wall-concrete
grass and road
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We summarize contributions and scope in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: We show that even if our method is superior, it still cannot get rid of the dependence large amount of data in Appnedix D
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
22

---

Justification:
We provide the full set of assumptions and a complete (and correct) proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide details of our settings in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification:
We provide an anonymous URL in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/dde/eeee Cooeeeoodeesssseeeeee 7/nips.com / ssiniiiiccypyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We provide details of our settings in the paper.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide theses information in our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://aaarag.cccp//aa/a//caaaaa
Answer: [Yes]
Justification: Yes.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our paper belongs to foundational research.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper belongs to foundational research.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: They properly credited and respected.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "OTNgoUVfog", "reviewer": "Reviewer_DSMk", "summary": "This paper primarily studies Open-Vocabulary Semantic Segmentation. The main contribution of this paper is the proposal of RPN, which employs relationship prompt learning solely to perform OVSS without any segmentation-specific networks. The authors claim that RPN attains state-of-the-art results on four public benchmarks.", "questions": "I have placed all the questions I want to ask in the weaknesses box. Overall, the paper doesn't seem to have any major issues, but personally, I find the work uninteresting and believe it offers limited insights to the field of open vocabulary semantic segmentation. However, if other reviewers feel that this work is worth accepting, I wouldn't strongly oppose it.", "limitations": "The authors did not discuss the limitations of their proposed algorithm in either the main text or the supplementary materials. \nI have highlighted some limitations of the paper in the weaknesses box. The authors can refer to these points to further improve the quality of the paper.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The strengths of this paper can be listed as follows,\n\n- The paper clearly expresses its main research content and the proposed algorithm.\n\n- The proposed algorithm achieved good results on multiple benchmark datasets.", "weaknesses": "The weaknesses of this paper can be summarized as follows,\n\n- The authors claim that their method achieved state-of-the-art results, but the results in Table 3 and Table 8 do not support this claim. Additionally, the methods listed in Tables 2, 3, 7 and 8 are somewhat limited. More recent works should be included to substantiate the claim that the proposed method achieves state-of-the-art results. (refer to https://paperswithcode.com/task/open-vocabulary-semantic-segmentation)\n\n- The authors should provide a more detailed explanation of the content in Figure 2, either in the caption or in the introduction section. (Or, they could reference this figure in a more suitable location, such as the first paragraph of the introduction section. Or, the authors could mention that a more detailed explanation is provided in the supplementary materials.)\n\n- According to the authors' definition, there have already been many works that achieve open vocabulary semantic segmentation without segmentation-specific networks, e.g., \"Image Segmentation Using Text and Image Prompts\", \"GroupViT: Semantic Segmentation Emerges from Text Supervision\", \"CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation\", \"SegGPT: Towards Segmenting Everything In Context\", to name a few. These works, to some extent, diminish the novelty of the authors' contribution. (To be clear, I am not saying that the authors' work lacks novelty entirely, but its novelty is not as strong as claimed in the paper.)\n\n- Did the authors conduct any ablation studies on the hyper-parameters that appear in Equations 9 to 11?\n\n- Why do the authors use class embeddings from only text layer 12, while using image embeddings from various image layers? Can the authors provide some analysis or experimental support for this choice?\n\n- When visualizing the attention map in Figure 1, the authors could also include a color legend to represent the response intensity.\n\n- Figures 4 to 6 should include annotations to explain the meanings of the various symbols used in the images.\n\n- Did the authors conduct any ablation studies on how to select the scale and the number of experts used in M2oE?\n\n- Is the Hadamard product in Equation 2 significant? Would using a matrix product between $p^i$​ and $g^i$ instead affect the results?\n\n- Why is pixel-level information used instead of patch-level information?\n\n- Does the proposed algorithm produce block artifacts when generating segmentation maps for high-resolution images?\n\nOverall, the paper is quite dull and does not stand out in the field of open vocabulary semantic segmentation. However, for a conference paper, the amount of work presented seems sufficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer DSMk\nThank you so much for acknowledging the strength of our method. We have carefully considered your constructive and insightful comments and here are the answers to your concerns.\n\n**Q1. The claim that our methods achieved SOTA results lack persuasiveness.**  \nThanks for your suggestions, please refer to **General Response-Q2**.\n\n**Q2. More detailed explanation of the content in Fig.2.**  \nThanks for your suggestion. We will modify the description in L.63 as: *Fig.2 shows the key difference between RPN and other OVSS methods. RPN makes VLM directly output pixel-level predictions by prompt learning, while other methods use VLM to assist a complex segmentation-specific networks where VLM transfers its rich knowledge to the mask proposal network by knowledge distillation or make the semantic decoder output segmentation masks by feature adaptation.*\n\n**Q3. Some works that achieve OVSS without segmentation-specific networks diminish the novelty of our contribution.**  \nThanks for your suggestions, please refer to **General Response-Q1**.\n\n**Q4. Ablation studies about the hyper-parameters $\\lambda_1, \\lambda_2$ in eq.11.**  \n$\\lambda_1, \\lambda_2$ control the weights between $\\mathcal{L}\\_{focal}$ and $\\mathcal{L}_{dice}$. We explore three common combinations. The results is shown as:\n|$\\lambda_1$|$\\lambda_2$|A-847|PC-459|A-150|PC-59|PAS-20|\n|:-|:-:|:-:|:-:|:-:|:-:|:-:|\n|20|1|11.0|16.9|31.1|56.7|95.1|\n|1|1|10.4|16.7|29.6|55.4|94.8|\n|**100**|**1**|**11.4**|**17.3**|**31.5**|**57.1**|**95.2**|\n\nThe results show that $\\lambda_1=100$ and $\\lambda_2=1$ are the best.\n\n**Q5. Why using class embeddings from only text layer 12, not others?**  \nTo explore which text layer of class embeddings to use, we conduct studies as:\n||pAcc|mIoU$_s$|mIoU$_u$|hIoU|\n|:-|:-:|:-:|:-:|:-:|\n|one-to-one|69.7|65.4|17.3|27.4|\n|only last|95.8|93.1|84.6|88.6|\n\nOne-to-one denotes using the same-layer embeddings from image and text encoder, and only last denotes using the last class embeddings, i.e., layer 12. We find that the shallow text embeddings cannot effectively guide VLM to achieve OVSS. The reason is that the shallow embeddings cannot accurately represent the class information, thereby constructing wrong relationship prompt.\n\n**Q6. A color legend is needed in Fig.1**  \nThanks for your suggestion. We will add the color legend in the future version. The degree of attention from low to high is marked by colors from dark blue to red. The more attention, the darker red; the less attention, the darker blue.\n\n**Q7. Explanation about some symbols in Fig. 4 to 6 is needed.**  \nWe will add these explanation in caption as: *$\\odot$, $\\otimes$ and $\\oplus$ denotes element-wise product, matrix product and addition. The symbols Expand, Einsum and Mul denote expanding class dimension, element-wise product and matrix product.*\n\n**Q8. Ablation studies about the scale and the number of experts used in M2oE.**  \nNote that the scale and number of experts are same parameter, which control the size of feature maps. Due to we crop images into $512\\times512$, the scale can be set to 1/8 of the input feature maps at most. Thus, the scale $s_i = \\frac{1}{2^{i-1}}$, where $i \\in [1,4]$, i.e., the maximum number of experts is 4. The ablation studies is shown as:\n|$i$ |pAcc|mIoU$_s$|mIoU$_u$|hIoU|\n|:-|:--:|:-:|:-:|:-:|\n|(1,2,3,4)|95.8|93.1|84.6|88.6|\n|(1,2,3)|95.6|92.9|83.8|88.1|\n|(1,2)|95.6|92.8|83.5|87.9|\n\nThe results show that embeddings with more diverse scales can improve OVSS performance.\n\n**Q9. Is the Hadamard product in eq.2 significant? Would using a matrix product between $p^i$​ and $g^i$ instead affect the results?**  \nWe rewrite eq.2 as: ${{m}}^i = {p}^i \\cdot  (t \\odot {g}^i)^\\top$, where $p^i \\in \\mathbb{R}^{n\\times d}$, $t\\in \\mathbb{R}^{c\\times d}$ and $g^i \\in \\mathbb{R}^{1\\times d}$. Please note the calculation process figure in our top-level pdf for more details.  \nHadamard product have an important effect in eq.2. Eq.2 (including Hadamard product and matrix product) can construct pixel-text relationship attention map to guide image encoder to transform image-level semantic to pixel-level. Firstly, Hadamard product assigns class weights to images in a batch by fusing $t$ used to identify different classes and $g^i$ used to identify each image in a batch, thus attaining image-text relationship. Based on this relationship, matrix product gets the class distribution of a pixel, which is normalized to one. Because $p^i$ contains pixel-level visual information, eq.2 achieves pixel-text relationship. Fig.1 visualizes the guidance process from image-text to pixel-text relationship. To demonstrate the image-to-pixel guiding scheme, we also conduct related interpretability experiments (i.e., MAD evaluation, L.271-282). The MAD results show that with the guidance of relationship prompt, the deep features of VLM gradually have a wider MAD value range, which indicates fine-grained semantic properties.  \nDue to that we need relationship map $m^i \\in \\mathbb{R}^{n\\times c}$, thus matrix product between $p^i$​ and $g^i$ cannot output map of this shape. In fact, we have explored matrix product between $p^i$​ and $t$ in L.291-311 and Tab.5. The results show that missing any of the product operations will have a significant impact on performance.\n\n**Q10. Why is pixel-level information used instead of patch-level information?**  \nIn fact, pixel-level information for feature maps used refers to patch-level information for original images. Due to that ViT usually divides the image into multiple patches for encoding, a pixel of the feature describes a patch of the image.\n\n**Q11. Does the proposed algorithm produce block artifacts when generating segmentation maps for high-resolution images?**  \nNo, for high-resolution images, we adopt *slide inference* mode, i.e., set stride to $341\\times 341$, and crop size to $512\\times 512$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TSQx9CBAyr", "reviewer": "Reviewer_2wFX", "summary": "This paper proposes relationship prompt module (RPM), which generates relationship prompt that directs VLM to extract pixel-level semantic embeddings suitable for OVSS. Moreover, RPM integrates with VLM to construct relationship prompt network (RPN), achieving OVSS without segmentation-specific networks. RPN attains state-of-the-art performance with merely about 3M trainable parameters (2% of total parameters).", "questions": "Please refer to the concerns and issues raised in the \"Weaknesses\".", "limitations": "Please refer to the concerns and issues raised in the \"Weaknesses\".", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.  This paper proposes RPM, which generates pixel-level relationship prompt to guide VLM to transform image-level embeddings to pixel-level ones suitable for OVSS.\n\n2. This paper proposes a simple yet parameter-efficient OVSS method, i.e., RPN, employing relationship prompt learning solely to perform OVSS without any segmentation-specific networks.\n\n3. RPN attains state-of-the-art results on four public benchmarks by optimizing about 3M trainable parameters (2% of total parameters).", "weaknesses": "1. In the experiment tables, there is no Efficiency comparison for ADE20K and Context dataset. \n\n2. The ablation studies in Table 4 are insufficient; they only include M2oE and LPM ablation studies, and do not cover ITP and APG.\n\n3. The performance is not state-of-the-art compared to some previous works, such as CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer 2wFX\nThank you so much for acknowledging the strength of our method. We have carefully considered your constructive and insightful comments and here are the answers to your concerns.\n\n**Q1. No efficiency comparison for ADE20K and Context dataset.**  \nWe have verified the efficiency on two benchmarks as illustrated in L.245-249. According to your comments, we add the efficiency for ADE20K and Context as:\n|     | #Params(M) | FLOPs(G) |\tFPS |\n| :------- | :------------: | :-------: |:-------: |\n|ADE20K|3.2|117.5|10.4|\n|Context|3.2|95.5|10.7|\n\n**Q2. The ablation study in Tab.4 do not cover ITP and APG.**  \nWe would like to re-clarify the ablation study in Tab.4. Tab.4 mainly shows the impact of M2oE, ITP, APG and LPM (we use RPM to denote the combination of M2oE, ITP and APG). Although we do not directly mark APG and ITP in Tab.4, RPM without M2oE denotes the ablation about APG and ITP. Therefore, we have conducted the component of APG and ITP. In addition, due to APG and ITP together form our relationship prompt (ablating one of them will cause bug), it is notable that APG and ITP can not be separated for ablation.\n\n**Q3. The performance is not SOTA compared to some previous works.**  \nThanks for your suggestions, please refer to **General Response-Q2**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ekF5XvxRzD", "reviewer": "Reviewer_Rcdd", "summary": "The paper proposes a training and inference-efficient Relationship Prompt Network (RPN). This network leverages a layer-wise Relationship Prompt Module (RPM) utilizing tuning methods similar to VLM LoRA and an improved Linear Projection Module (LPM) without relying on a segmentation model. The authors conduct extensive experiments, demonstrating the solid efficiency and effectiveness of their approach.", "questions": "See the weakness.", "limitations": "Please provide a more detailed analysis on the correlation between the relational abilities of the model and its performance in open-vocabulary tasks (separating seen and unseen classes).", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper proposes a method for continuously injecting pixel-level image-text relationships into the layers of the image encoder using efficient tuning techniques. The experiments conducted are very comprehensive.", "weaknesses": "The paper lacks a discussion on region-text relationships, focusing instead on directly leveraging pixel-text relationships.\nAdditionally, there should be a discussion and experiments with other visual prompt tuning methods applied to the baseline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer Rcdd\nThank you so much for acknowledging the strength of our method. We have carefully considered your constructive and insightful comments and here are the answers to your concerns.\n\n**Q1. Lack of a discussion on region-text relationships, focusing instead on directly leveraging pixel-text relationships.**  \nThanks for your valuable suggestion. The discussion on region-text relationship is worthy.  \nFirst, we clarify the difference between pixel-text and region-text relationship: the former links image pixel with text description; the latter links image mask with text description. Since image mask contains more information, it can more intuitively represent things or stuff, while the image pixels are the opposite. Thus, image mask can be more easily associated with text description. For example, region-text relationship can be established by directly matching image mask and text description, while when we need to establish pixel-text relationship after featuring image mask and text description.  \nSecond, we consider the impact of region-text on OVSS. Although constructing region-text relationship seems to be simpler, is it more effective for OVSS? As illustrated in MaskFormer[1*], pixel-level classification is not necessarily better than mask-level classification for semantic segmentation, which seems to show that region-text relationship enable the VLM to achieve OVSS better than pixel-text relationship. However, adopting region-text relationship means that VLM need to output mask proposals, which are usually output by a DERT-like[1*,2*] decoder. This shows that VLM need a DERT-like decoder, and goes against our intended goal of achieving OVSS using only VLM without any segmentation-specific networks.  \nThird, we consider the impact of region-text relationship on open-vocabulary segmentation. VLM with region-text relationship can output class-agnostic mask, thus making it feasible to achieve instance segmentation and object detection. This is a very interesting direction worth exploring, and it gives us some inspiration for future work.\n\n[1*] Per-Pixel Classification is Not All You Need for Semantic Segmentation.  \n[2*] End-to-End Object Detection with Transformers.\n\n**Q2. There should be a discussion and experiments with other visual prompt tuning methods applied to the baseline.**  \nThanks for your valuable suggestion. In fact, we have discussed seven current PEFT methods (including other visual prompt tuning) applied to the baseline as illustrated in L.618-638, and show the comparison in Tab.9 and 10. We show the part of Tab.9 as:\n|(VOC)|#Params(M)|pAcc|mIoU$_s$|mIoU$_u$|hIoU|\n|:-|:-:|:-:|:-:|:-:|:-:|\n|Baseline|154.5|84.1|83.5|31.2|45.4|\n|VPT|4.0|90.9|81.0|52.9|64.0|\n|LoRA|4.0|91.3|82.2|53.1|64.5|\n|**RPN(ours)**|**3.2**|**95.8**|**93.1**|**84.6**|**88.6**|\n\nWe would like to re-emphasize the conclusion as: our method shows significant improvement (at least 20% mIoU improvement in VOC), due to that existing PEFT methods mainly focus on fine-tuning the image-supervised VLM to improve performance on classification. But, our method enables VLM to directly achieve OVSS.\n\n**Q3. Please provide a more detailed analysis on the correlation between the relational abilities of the model and its performance in open-vocabulary tasks (separating seen and unseen classes).**  \nThanks for your valuable suggestion. We provide our analysis as follows:  \nThe relational capability of the model is reflected in guiding the model to make pixel-level predictions for unseen classes by establishing relationship attention between text class information and pixel-level visual information, making its performance close to that of seen classes. We visualize the relationship attention map in Fig.1. The first line denote the relationship attention maps for seen class across each layer; the last two lines for unseen classes. For seen class, the model has prior pixel-level semantic information, so the relationship attention map only needs to focus on a small number of pixels to guide the model to make class predictions for these pixels (e.g., the relationship attention map for airplane has fewer highlighted areas). For unseen class, the model lacks corresponding pixel-level semantic information, so the relationship attention map needs to focus on more pixels to provide the model with more sufficient pixel-level semantic information (e.g., the shallow relationship attention map for sheep highlights the complete semantic information). In addition, to demonstrate the guidance mechanism, we conduct related interpretability experiments (i.e., MAD evaluation), as illustrated in L.271-282. The MAD results show that with the guidance of relationship prompt, the deep features of VLM gradually have a wider MAD value range, which indicates fine-grained semantic properties. We also conduct system level comparison (separating seen and unseen classes) as shown in Tab.2 and 7.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RoRbl1vit1", "reviewer": "Reviewer_Rvjz", "summary": "This paper tackles open-vocabulary semantic segmentation, a task aiming at per-pixel predictions of image inputs on any given classes. Existing methods are built on pre-trained vision-language models like CLIP. Different from CLIP that achieves coarse open-vocabulary classification, open-vocabulary segmentation extends this understanding capability to pixel level. This paper claims that existing approaches take much compute for training. Authors develop a Relationship Prompt Network that features dense image-text correlations across all layers. Also, their approach includes some training techniques, including mixture-of-experts, multi-scale input and prompt learning. Experimental results show that proposed RPN achieves performance gains against published methods like ZegCLIP [19], MaskCLIP [8] and FreeSeg [10].", "questions": "- Line 32, “it is challenging to apply prompt learning solely to VLM for OVSS”. Has this claim been checked? Why applying prompt learning to VLM for OVSS is challenging? Does “Challenging” refer to “technically challenging”, right?\n- Line 49-51, “we find they can construct an image-text relationship attention map via the attention mechanism”. Image-text relationship attention map is a confusing claim. This should refer to a well defined phrase in this field, such as language-aware deep fusion in [A], which is an influential paper in this field.\n- Figure 1, how are these attention maps plotted?\n- Line 58-61, we propose linear projection module comprising merely two individual linear layers, which maps the image and text feature into a shared space, and then computes their matrix product to produce the results. What should matrix product refer to? Element-wise or column-wise?\n- Typos. Line 67 and 69, RPN to RPM. There is potential confusion between RPM and RPN.\n- Line 113-114, reducing parameter consumption. I do not think this is a correct claim. Adding dense cross-attentions will surely increase parameters for inference. I do not see techniques in this paper that reduces parameters like quantization.\n- Line 113-125, refer to a figure here is better for illustration purpose.\n- Figure 2, what is M2oE, ITP and APG block? A bit confusing.\n- Table 4, what does the first line mean and what is the difference between the first and the second line?\n- Figure 7, only legends of head 1 and 2 are given, while others are missing. In such case, I cannot understand what is presented in this figure.\n- Table 1, instead of params, this should be Learnable params, right?\n\n[A] Grounded Language-Image Pre-training. CVPR 2022.", "limitations": "Authors have included limitations of proposed RPN in Appendix D.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- **Performance.** The proposed RPN outperforms several published methods, such as ZegCLIP, MaskCLIP and FreeSeg.\n- **Inference efficiency.** Table 1 shows that proposed RPN is trained with fewer learnable parameters. In inference, RPN shows less computation in FLOPs.", "weaknesses": "- **Unclear unique contributions.** I am not quite understand about the unique contribution from this paper. Does this paper give new insights to the community? The proposed relationship prompt is a combination of powerful tuning methods that have been proven effective, including dense image-text attentions, mixture-of-experts, multi-scale inputs and prompt learning. None of these modules are emphasized and I cannot see the unique value of applying these techniques. Performance gains from these techniques does not surprise me that much.\n- **Many typos and confusing claims**. Please refer to Questions 1-11 for details. I think these issues should be taken care of before considered for publication.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer Rvjz\nThank you so much for acknowledging the strength of our method. We have carefully considered your constructive and insightful comments and here are the answers to your concerns.\n\n**Q1. Unclear unique contributions.**  \nPlease refer to **General Response-Q1**.\n\n**Q2. Confusing claim that *it is challenging to apply prompt learning solely to VLM for OVSS* should be argued.**  \nWe would like to re-clarify the claim as:  \nWe point out two reasons for the claim (in L.32-41): (1) Current prompt provides only image-level granularity, which restricts VLMs from performing tasks related to pixel-level visual localization. (2) A lack of an image-text relationship, which hinders the exploration of VLMs’ potential for open-vocabulary scene understanding. To demonstrate this, we compare seven current PEFT methods applied to the baseline (in L.618-638), and show the comparison in Tab.9 and 10. We show the part of Tab.9 as:\n|(VOC)|#Params(M)|pAcc|mIoU$_s$|mIoU$_u$|hIoU|\n|:-|:-:|:-:|:-:|:-:|:-:|\n|Baseline|154.5|84.1|83.5|31.2|45.4|\n|VPT|4.0|90.9|81.0|52.9|64.0|\n|LoRA|4.0|91.3|82.2|53.1|64.5|\n|**RPN(ours)**|**3.2**|**95.8**|**93.1**|**84.6**|**88.6**|\n\nWe would like to re-emphasize the conclusion as: it is challenging to apply prompt learning solely to VLM for OVSS, and our method shows significant improvement (at least 20% mIoU improvement in VOC).\n\n**Q3. Concerns between image-text relationship attention map and language-aware deep fusion in [1\\*].**  \nWe analyzed their differences from three perspectives as follows:  \n(1) Motivation: We aim to directly output pixel-level predictions using only the frozen VLM, while [1*] aim to proposal a grounded language-image pre-training method.  \n(2) Structure: We use a train-free method to construct our map, while [1*] uses a trainable cross-attention module to construct it; we only need to introduce our map for the image encoder, while [1*] also introduces it for the text encoder. Thus, our method is more concise and direct.  \n(3) Training cost: we do not need additional training sets, parameter-intensive adapter and large-scale pre-training, which are all required by [1*].  \nIn summary, our method achieves OVSS with lower training cost and simpler structure.\n\n[1*] Grounded Language-Image Pre-training.\n\n**Q4. Fig.1, how are these attention maps plotted?**  \nWe directly visualize the results of eq.2, ${{m}}^i = {p}^i \\cdot  (t \\odot {g}^i)^\\top$, using a heat map.\n\n**Q5. What should matrix product refer to? Element-wise or column-wise?**  \nLet the image and text feature denote $p \\in \\mathbb{R}^{n\\times d}$ and $t\\in \\mathbb{R}^{c\\times d}$. The matrix product $O\\in \\mathbb{R}^{n\\times c}$ between them $O_{ij} = \\sum_{k=1}^d p_{ik}t^\\top_{kj}$.\n\n**Q6. Typos. Line 67 and 69, RPN to RPM. There is potential confusion between RPM and RPN.**  \nIn fact, RPM and RPN refers to **R**elationship **P**rompt **M**odule and **R**elationship **P**rompt **N**etwork. As illustrated in L.61-62, RPN consists of RPM and VLM.\n\n**Q7. Line 113-114, concerns about reducing parameter consumption.**  \nWe clarify the sentence of L.113-114 that our method only add 3M trainable parameters with the frozen VLM to achieve OVSS, so our method is low-cost during training. As illustrated in Tab.1, we conduct efficiency experiment to demonstrate our method indeed consume very small amount of additional trainable parameters to achieve OVSS. In addition, we add the efficiency for ADE20K and Context as:\n||#Params(M)|FLOPs(G)|FPS|\n|:-|:-:|:-:|:-:|\n|ADE20K|3.2|117.5|10.4|\n|Context|3.2|95.5|10.7|\n\n**Q8. L.113-125, refer to a figure here is better for illustration purpose.**  \nThanks for your suggestion.\n\n**Q9. Fig.2, what is M2oE, ITP and APG block? A bit confusing.**  \nActually, Fig.2 has nothing to do with these blocks. Are you referring to Fig.3? If yes, we have mentioned these blocks in the legend of Fig.3. RPM consists of M2oE, ITP and APG block. M2oE refers to multi-scale mixture-of-experts, which aims to propose multi-scale vision knowledge (L.134-150). ITP refers to image-to-pixel semantic attention, which aims to enable the image encoder to learn open-vocabulary semantics from text features (L.151-168). APG refers to adaptive prompt generation, which aims to construct the adaptive image-text relationship for each pixel (L.169-179).\n\n**Q10. Tab.4, what does the first line mean and what is the difference between the first and the second line?**  \nRPM (the first line) denotes the combination of M2oE, ITP and APG. Therefore, RPM without M2oE (the second line) denotes the ablation about APG and ITP.\n\n**Q11. Concerns about Fig.7. Only legends of head 1 and 2 are given, while others are missing.**  \nFirst, we give all heads in Fig.7 (MAD evaluation) instead of head 1 and 2, and use different color point to denote different heads; due to the number of heads is large, only the symbols for the two heads are given in the legend, and the remaining heads are indicated by ellipses (note the legend at the bottom right).  \nSecond, MAD can be used to explore the range of attention of each attention head, similar to the receptive field in convolutional neural networks (CNNs) (e.g., Fig.7 in ViT[1*] is also represented to valuate MAD). It is a common metric. A higher point indicates a larger receptive field, and greater point spacing signifies richer feature diversity. Fig.7 show that with the guidance of relationship prompt, the deep features of VLM gradually have a wider MAD value range, which indicates fine-grained semantic properties.\n\n[1*] An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.\n\n**Q12. Tab.1, instead of params, this should be Learnable params, right?**  \nYes, params denotes learnable params.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response

We would like to thank all reviewers for providing constructive feedback that helped us improved the paper. We are encouraged that reviews think our paper:
* Clear expression: *The paper clearly expresses its main research content and the proposed algorithm.* (Reviewer DSMk)
* Good performance: *The proposed algorithm achieved good results on multiple benchmark datasets.* (Reviewer DSMk) *RPN attains state-of-the-art results on four public benchmarks.* (Reviewer 2wFX) *The proposed RPN outperforms several published methods.* (Reviewer Rvjz) *RPN shows less computation in FLOPs.* (Reviewer Rvjz)
* Comprehensive experiments: *The experiments conducted are very comprehensive.* (Reviewer Rcdd)

We have been working diligently on improving the paper on several fronts, addressing your critique. Below, we summarize the changes that we have made in an updated draft.
* More explanation about the diagrams. (Please refer to Response to Reviewer DSMk-Q2,6-7)
* More discussion on region-text relationship. (Please refer to Response to Reviewer Rcdd-Q1)
* More analysis on the correlation between the relational abilities of the model and its performance. (Please refer to Response to Reviewer Rcdd-Q3)

We respond to each of the reviewers in detail, and to some common issues below.

**(Reviewer DSMk and Rvjz) Q1. Concerns about contributions.**  
We would like to re-emphasize our novelty as follows:  
(1) Motivation: There are two main ways to achieve OVSS using image-level supervised VLM: pre-training the VLM from scratch using pixel-level supervision; using knowledge distillation or feature adaptation to assist in training parameter-intensive segmentation-specific networks with VLM. However, the training cost of these methods is high (the former needs additional training sets and large-scale pixel-level pre-training; the latter needs additional parameter-intensive segmentation-specific networks). We aim to froze VLM and just make prompt learning to achieve OVSS. Thus, the training is low-cost.  
(2) Difference: Although there exist OVSS methods which do not rely on segmentation-specific networks, they do not just rely on frozen VLM to achieve OVSS. For example, CLIPSeg[1*], SED[3*] and CAT-Seg[2*] all rely on an additional parameter-intensive decoder adapter; GroupViT[4*] and SegGPT[5*] both need additional segmentation training sets. We would like to re-emphasize our contribution as: **we only use VLM with extremely low training cost (almost 3M trainable parameters) to directly achieve OVSS.** The comparison of important aspects such as, training sets, decoder adapter and pixel-level pre-training, between RPN and the other OVSS methods without segmentation-specific networks are shown as:
||Additional Training Sets|Parameter-intensive Decoder Adapter|Large-scale Pixel-level Pre-training|
|:-|:-:|:-:|:-:|
|CLIPSeg|❌|✔|❌|
|SED|❌|✔|❌|
|CAT-Seg|❌|✔|❌|
|GroupViT|✔|❌|✔|
|SegGPT|✔|❌|✔|
|RPN(ours)|❌|❌|❌|

[1*] Image Segmentation Using Text and Image Prompts  
[2*] CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation  
[3*] SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentation  
[4*] GroupViT: Semantic Segmentation Emerges from Text Supervision  
[5*] SegGPT: Towards Segmenting Everything In Context

**(Reviewer DSMk and 2wFX) Q2. The claim that our methods achieved SOTA results lack persuasiveness.**  
There are usually two types of experimental settings (zero-shot and open-vocabulary settings) for OVSS, involving multiple benchmarks. To fully evaluate the performance, we conducted experiments in the two settings.

Firstly, in zero-shot settings, our method outperforms existing methods on all benchmarks in Tab.2 and 7. Secondly, in open-vocabulary setting, our method outperforms CAT-Seg[2*] (the latest OVSS method) on three benchmarks (A-847, PC-459 and A-150), and achieve the highest results on three benchmarks (A-847, PAS-20 and A-150) in Tab.3 and 8. Thirdly, in both the two settings, we show the comparison as:
||A-849|pc-459|A-150|PC-59|PAS-20|VOC(mIoU$_u$)|COCO(mIoU$_u$)|Context(mIoU$_u$)|
|:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
|SPNet|-|-|-|24.3|18.3|15.6|8.7|-|
|ZS3Net|-|-|-|19.4|38.3|17.7|9.5|12.7|
|ZegFormer|5.6|10.4|18.0|45.5|89.5|63.6|33.2|-|
|FreeSeg|7.1|6.4|17.9|34.4|85.6|78.6|42.2|-|
|RPN(ours)|**11.4**|**17.3**|**31.5**|**57.1**|**95.2**|**84.6**|**42.8**|**58.7**|

The result show that our method achieve SOTA on all benchmarks in both the two settings.

Meta-review: Overall the reviews found the contribution in terms of technique too small, and the evaluation of benefit lacking for several reasons.  There was some confusion about acronyms that seems reasonable given the authors' choice of using coining two, RPM and RPN, that are similar to each other and to other acronyms used in the field.  The ability to adapt a VLM, without an additional explicit segmentation model, to come close to the state of the art is hopefully sufficient with some careful clarification in the paper text.
Decision: Accept (poster)