Relationship Prompt Learning is Enough for Open-Vocabulary Semantic Segmentation

Authors: jiahao  li
yang  lu
yuan  xie
yanyun  qu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-29 10:37:40

Abstract: Open-vocabulary semantic segmentation (OVSS) aims to segment unseen classes without corresponding labels. Existing Vision-Language Model (VLM)-based methods leverage VLM's rich knowledge to enhance additional explicit segmentation-specific networks, yielding competitive results, but at the cost of extensive training cost. To reduce the cost, we attempt to enable VLM to directly produce the segmentation results without any segmentation-specific networks. Prompt learning offers a direct and parameter-efficient approach, yet it falls short in guiding VLM for pixel-level visual classification. Therefore, we propose the ${\bf R}$elationship ${\bf P}$rompt ${\bf M}$odule (${\bf RPM}$), which generates the relationship prompt that directs VLM to extract pixel-level semantic embeddings suitable for OVSS. Moreover, RPM integrates with VLM to construct the ${\bf R}$elationship ${\bf P}$rompt ${\bf N}$etwork (${\bf RPN}$), achieving OVSS without any segmentation-specific networks. RPN attains state-of-the-art performance with merely about ${\bf 3M}$ trainable parameters (2\% of total parameters).

Main paper: 
 introduction: 
 Open-vocabulary semantic segmentation (OVSS) [1] [2] [3] [4] aims to segment novel classes without corresponding training images, which is still a challenging task in computer vision. Vision-Language Model (VLM) [5] [6] [7] has emerged as a powerful approach, acquiring comprehensive knowledge via large-scale image-caption matching training. Several VLM-based OVSS methods [8] [9] [10] achieve promising results. These methods employ the rich image-text representation knowledge inherent in VLM to improve segmentation performance and are categorized into two types: two-stage and one-stage methods. Two-stage methods [11] [12] [13] [14] first generate image-level masks without semantics via a well-designed mask proposal network [15] [16] [17] [18] and then classify these masks via the image-level classification ability of VLM. One-stage methods [19, 20] employ a semantic decoding network to distill VLM's comprehensive knowledge from the image to the pixel level, thereby producing pixel-level segmentation results.However, both of these VLM-based methods rely on additional explicit segmentation-specific networks to obtain segmentation results, resulting in extensive training cost. To reduce the training cost, an intuitive idea is to make VLM directly produce segmentation results without these segmentation- The degree of attention from low to high is marked by colors from dark blue to red. The more attention, the darker red; the less attention, the darker blue. As the layers deepen, the attention maps exhibit increasingly precise pixel-level semantics. The images correspond to the attention maps for an aeroplane, a sheep, and a train, respectively. specific networks 1 . In this context, prompt learning [5, [21] [22] [23] emerges as a practical approach, guiding VLM to transform image-text pair embeddings into pixel-level semantic embeddings suitable for OVSS, thereby directly achieving OVSS.Without explicit segmentation-specific networks, applying prompt learning solely to VLM for OVSS is straightforward yet challenging. Typically, existing prompt learning methods fine-tune task-specific models to enhance performance on that task, yet they fail to secure cross-task performance gains. These methods employ either fixed or trainable vision prompt tokens [24] , or they construct complex ViT-based networks for generating prompt [25, 26, 20, 27] . The limitations of such prompt include: 1) providing only image-level granularity, which restricts VLMs from performing tasks related to pixel-level visual classification, and 2) a lack of an image-text relationship, which hinders the exploration of VLMs' potential for open-vocabulary scene understanding. Therefore, it is difficult for these methods to enable VLM suitable for image-level classification to achieve open-vocabulary semantic segmentation directly.Figure 2 : Our method vs existing VLM-based methods.Our method employs VLM to directly perform OVSS by prompt learning, while the other methods relies on additional explicit segmentation-specific networks.In summary, addressing the abovementioned issue lies in refining the granularity of prompt and strengthening the image-text relationship within them. By analyzing the outputs of VLM's encoder layer, we find they can construct an image-text relationship attention map via the attention mechanism, guiding the encoder to focus on relevant pixels. Thus, we propose the Relationship Prompt Module (RPM) that utilizes the outputs of image and text encoding layers to enable pixel-level relationship prompting, enhancing prompt's granularity and image-text relationship. Moreover, we implement a layer-by-layer guidance mode in VLM, enabling a progressive transfer of embeddings from the image to pixel level. As illustrated in Figure 1 , each layer's relationship attention map is continuously refined following an image-to-pixel attention scheme. To obtain the segmentation results, we propose the Linear Projection Module (LPM) comprising merely two individual linear layers, which maps the image and text feature into a shared space, and then computes their Matrix product to produce the results. Finally, we propose the Relationship Prompt Network (RPN), which consists of RPM, LPM and VLM. Figure 2 shows the comparison between RPN and other OVSS methods. RPN employs VLM to directly output pixel-level predictions by prompt learning, while other methods use VLM to assist explicit segmentation-specific networks to obtain predictions. In these VLM-based methods, VLM transfers its rich knowledge to the mask proposal network by knowledge distillation or enables the semantic decoder to output segmentation masks by feature adaptation. The key difference between RPN and existing VLM-based methods is that RPN does not require any explicit segmentation-specific networks; it only adapts VLM to perform OVSS. Our contributions are summarized as follows:• We propose the Relationship Prompt Module (RPM), which generates pixel-level relationship prompt to guide VLM in transforming image-level embeddings to pixel-level ones suitable for OVSS.• We propose the Relationship Prompt Network (RPN), employing prompt learning solely to adapt VLM for OVSS without explicit segmentation-specific networks.• RPN attains state-of-the-art results on four public benchmarks by optimizing about 3M trainable parameters (2% of total parameters). 
related works: 
 Open-Vocabulary Semantic Segmentation. Open-vocabulary semantic segmentation [28, 29] aims to leverage the knowledge from representation distributions of seen categories to classify unseen categories. Existing methods can be divided into two types: generative and discriminative. Generative methods [30, 31, 28, 32] require the segmentation network to be aware of which categories are unseen during training, while discriminative methods [11, 8, 19, 33] directly transfer semantics from seen to unseen categories, which is more straightforward. SPNet [33] introduces the zero-shot task for the first time and proposes an end-to-end training paradigm, in which the visual embeddings are composed with the uniform semantic word embeddings to obtain the semantic logits. ZS3Net [31] utilizes a generative approach to project the text embeddings into visual space and generate visual embeddings for unseen categories. Subsequently, many works following the generative method have been proposed. STRICT [34] assumes that the pixels of unseen categories can be present during training the model and adopts the self-training strategy to optimize the model for classifying the unseen categories. GaCNet [30] proposes a novel context-aware feature generation method based on ZS3Net, in which pixel-wise contextual knowledge can be utilized to guide the feature generation process of unseen categories. CLIP-based approaches have also made great progress. ZegFormer [11] proposes two sub-tasks, i.e., class-agnostic grouping and segment-level zero-shot classification and presents the CLIP-based method for the first time. MaskCLIP [8] utilizes the frozen CLIP to make a minimal adaptation by fine-tuning a lightweight classifier and replacing it with that of the segmentation network. Zsseg [12] proposes a two-stage CLIP-based method, in which a proposal generator is used to generate binary masks and CLIP is required to classify them. ZegCLIP [19] presents a one-stage method in which CLIP directly transfers knowledge to a lightweight decoder.Vision-Language Models for Vision Tasks. Vision-language models for vision tasks [5, [21] [22] [23] are optimized with a large scale of image-text pair data on the internet. There are three categories: contrastive, generative, and aligned objectives. CLIP [5] first proposes the paradigm of pre-trained vision-language model. DeCLIP [35] argues that CLIP is data-intensive and proposes a data-efficient training paradigm. UniCL [36] combines the two data sources to build a new image-text-label field and proposes unified contrastive learning. ZeroVL [21] proposes debiased sampling to deal with biased representation distributions and a new mixup method for the image and text models. OTTER [22] uses optimal transport to find the soft label for contrastive learning and handle the problem of noisy image-text pairs.Visual Prompt Learning. Visual Prompt Learning [37] [38] [39] is a technique that assists in adapting CLIP-like vision-language models for various visual tasks. CoOp [40] adopts trainable vectors as word prompt to adapt CLIP for vision classification. VP [41] utilizes perturbations as visual prompt.VPT [24] proposes trainable visual prompt to adapt each layer of the visual embeddings. UPT [38] constructs unified prompt modeling to extract trainable visual and textual prompt for adapting CLIP. MaPLe [39] adopts trainable prompt to guide both visual and textual embeddings and proposes a coupling function as a bridge to build a multi-modal prompt. DenseCLIP [26] uses the contextual information from the image to prompt the language model. Probabilistic prompt [20] applies multiple prompt sampled from probabilistic text embeddings to better understand the image. SegPrompt [27] proposes a category-level prompt to improve the model's class-agnostic segmentation ability.  
approach: 
 Overview. Our objective is to adopt prompt learning to develop a VLM-based OVSS method without any explicit segmentation-specific networks, thereby reducing training cost. As shown in Figure 3 , the Relationship Prompt Network (RPN) is an end-to-end system comprising text, image and image-text relationship prompt branches. In the text branch, the text encoder inputs text to yield the class embeddings t ∈ R C×d , where C and d represent the number of classes and dimension, respectively. In the image branch, the plain encoder inputs images to obtain the image embeddings v ∈ R (N +1)×d , which includes the patch embeddings p ∈ R N ×d and the [CLS] token g ∈ R 1×d , with N representing the number of patches. Concurrently, in the image-text relationship prompt branch, the proposed Relationship Prompt Module (RPM) alongside the encoder takes the class and image embeddings to generate pixel-level relationship prompt, which is subsequently concatenated with the image embeddings to serve as the input for the next image layer. To obtain segmentation results, the last class and image embeddings are fed into the proposed Linear Projection Module (LPM) to calculate their Matrix product.Relationship Prompt Module. RPM can guide the plain encoder of VLM to directly produce pixel-level semantic embeddings suitable for OVSS, due to its acquisition of three distinct types of knowledge. Firstly, it acquires multi-scale vision knowledge to locate objectives at distinct scales. Secondly, its image-text relationship knowledge enables the plain encoder to learn openvocabulary semantics from text features. Thirdly, it introduces dynamic pixel-level knowledge, which is adaptive for the relationship knowledge, enabling pixel-level relationship prompt learning and thus transforming VLM's image-level embeddings into pixel-level ones. Therefore, RPM comprises three blocks, each dedicated to capturing one of the aforementioned knowledge types. It is crucial for OVSS to obtain multiscale image embeddings to locate targets of different scales. However, the image embeddings maintain a consistent scale across each image layer. Therefore, we propose multi-scale mixture-of-experts (M2oE) block to aggregate the patch embeddings across distinct scales. As illustrated in Figure 4 , M2oE comprises a gating network and several expert networks as in [42, 43] . The gating network aims to dynamically activate different experts, each responsible for scaling the input to various extents. M2oE of the i-th layer is formulated as follows:EQUATIONwhere G(•) and E j (•) represent the gating network and the j-th expert of all n experts, respectively. Note that the patch embeddings first reduce the dimension from d to r = 3. And Linear To achieve this, we propose image-to-pixel semantic attention (ITP) block, which utilizes the image embeddings and the last class embeddings to form the relationship attention map m ∈ R N ×C . As illustrated in Figure 5 , we first calculate the Hadamard product between the [CLS] token g and the last class embeddings t after dimension alignment. Then, the Matrix product between the Hadamard product result and the patch embeddings p yields the relationship attention map m. Consequently, the relationship attention map m i of the i-th layer is formulated as follows:m i = p i • (t ⊙ g i ) ⊤(2) Intuitively, the first Hadamard product operation assigns weights to images in a batch, with their sum being one (the more important the image, the larger the weight), by fusing the class embeddings used to identify different classes and the [CLS] token used to identify each image in a batch, thus attaining image-level attention. The subsequent Matrix product operation weights the pixels, with the sum of pixel weights normalized to one (the more important the pixel, the larger the weight), by integrating the patch embeddings that contain pixel-level visual information, thus securing pixel-level attention. Therefore, we refer to the training-free operation (i.e., Eq. 2) as the image-to-pixel attention scheme, in which the first and the subsequent products extract image-level and pixel-level information, respectively. As illustrated in Figure 1 , the relationship attention map construction process from the shallow to the deep layer demonstrates the effectiveness of the image-to-pixel attention scheme.The construction of the adaptive image-text relationship for each pixel enables VLM to directly output pixel-level semantic embeddings. To achieve pixel-level dynamic tuning, we propose adaptive prompt generation (APG) block. As illustrated in Figure 5 , we first initialize a trainable parameter h ∈ R N ×d , representing the dynamic pixel-level knowledge. The adaptive relationship prompt m ∈ R C×d for each pixel is then derived from the Matrix product between the dynamic pixel-level knowledge h and the relationship attention map m. Consequently, the adaptive relationship prompt mi of the i-th layer is formulated as follows:EQUATIONThe role of the dynamic pixel-level knowledge h is twofold: 1) it projects the relationship attention map from a lower to a higher dimension to serve as an input for the plain encoder, 2) it fine-tunes the relationship prompt for each pixel. It is the fine-tuning of the relationship prompt for each pixel in a high-dimensional space that enables the plain encoder to directly obtain pixel-level semantics.In addition, we integrate RPM in parallel within each layer of VLM. The image embeddings and the last class embeddings are fed into RPM to generate the relationship prompt, which is then merged with the image embeddings and fed into the next image layer. The prompt output from the image layer is discarded. The processing of each image layer is formulated as follows:EQUATIONThe notation [, ] represents the concatenation operation. See Appendix C for more details on RPM. EQUATIONEQUATIONEQUATIONFor simplicity, the normalization layers are omitted. We select LPM c as the proposed LPM experimentally (as shown in Table 6 ).Optimization. There are two types of loss functions: the cross-entropy loss with the Softmax function and the combination loss between the focal loss and the dice loss with the Sigmoid function.The former employs one-hot encoding to render the class distribution as mutually exclusive in the embedding space, while the latter utilizes multi-label encoding to permit class distribution overlap.Practically, these losses are typically selected based on the used semantic decoders, such as the former with FPN [44] and the latter with a transformer decoder [45] . Considering that we employ LPM (comprising just two individual linear layers) to obtain the final segmentation results without any well-designed semantic decoders, we evaluate the aforementioned two loss functions. We refer to the former as the Softmax loss and the latter as the Sigmoid loss. We select the Sigmoid loss experimentally (as shown in Table 6 ).The Softmax loss is formulated as follows:EQUATIONwhere y k and ŷk are the ground truth and the prediction vectors, respectively, and h and w represent the height and width of the input image. The Sigmoid loss is formulated as follows:EQUATIONEQUATION)EQUATIONwhere α, γ, λ 1 and λ 2 are hyperparameters. In Eq. 9, when y k equals the zero vector, y k and ŷk are substituted with 1 -y i and 1 -ŷi , respectively.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes]• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments: 
 We evaluate our method in both zero-shot and open-vocabulary settings. See Appendix D for more details on the settings. Training Strategy. We conduct all experiments on eight NVIDIA GTX 3090 GPUs using the MMSegmentation tool [50] . If not specified, we employ the pre-trained CLIP ViT-B/16 model for both the plain encoder and the text encoder. We set the batch size of 4 for each GPU and set the input resolution to 512 × 512 pixels. The data augmentation strategy adheres to the default settings in MMSegmentation, which includes random image resizing with a short-side range of [256, 1024] and a crop size of 512 × 512. The optimizer is AdamW, initialized with a learning rate of 2 × 10 -5 and a weight decay of 1 × 10 -2 . The learning rate follows a polynomial decay schedule with a power of 0.9. The number of iterations is set to 20K for the VOC dataset, 80K for the COCO dataset, and 40K for the Context dataset. We set λ 1 and λ 2 in Eq. 11 to 100 and 1, respectively.Efficiency Comparison. We present an efficiency comparison with state-of-the-art methods in Table 1 . The results of compared methods are derived from [19] . To ensure a fair comparison, we report our results based on the open-source code from [19] and evaluate them with an input resolution of 512 × 512 on a single NVIDIA GTX 1080 Ti GPU. Our method outperforms the other methods in efficiency, achieving the lowest number of trainable parameters and the smallest FLOPs.Comparison in the Zero-Shot Setting. We show the performance comparison with the state-ofthe-art methods in the zero-shot setting in Table 2 , and conduct the comparison under three scenarios: without self-training, with self-training, and fully supervised. In the absence of self-training, our method surpasses FreeSeg [10] with +6.0% mIoU u on the VOC dataset and +0.6% mIoU u on the COCO dataset. With self-training, our method outperforms ZegCLIP [19] with +3.7% mIoU u on the VOC dataset, +1.3% mIoU u on the COCO dataset, and +2.3% mIoU u on the Context dataset. Under the fully supervised scenario, our method exceeds ZegCLIP [19] with an average of +2.0% mIoU u across the three datasets. We attribute the modest improvement on the COCO dataset to the bias from the extremely unbalanced training and validation set ratios (more training data and less validation data), which contrasts with the performance on the Context dataset (less training data and more validation data), reflecting the robust zero-shot learning ability of our method.Comparison in the Open-Vocabulary Setting. We show the performance comparison with the state-of-the-art methods in the open-vocabulary setting in Table 3 . Our method does not require an additional training dataset. The results indicate that no method can consistently outperform others across all validation datasets; however, our method attains state-of-the-art performance on the A-847, A-150 and the PAS-20 datasets. As analyzed in [53] , the Context dataset and the ADE20K dataset exhibit the highest and lowest label-set similarities with the training dataset, respectively. Therefore, our method showcases a more comprehensive open-vocabulary learning ability.We conduct the ablation experiments on the VOC and the COCO datasets. If not specified, most are conducted on the VOC dataset. See Appendix D for more details on the experiments.Image-to-Pixel Attention Scheme. RPM aims to transform image-level embeddings from VLM into pixel-level semantic embeddings, enabling direct OVSS. To illuminate its functionality, we employ the Mean Attention Distance (MAD) [56, 57] as a metric, reflecting the granularity of information aggregated within the self-attention head. As illustrated in Figure 7 , we analyze MAD of each self-attention head during the initial and advanced stages of training. A higher point indicates a larger receptive field, and greater point spacing signifies richer feature diversity. In the initial training phase, shallow and deep layer information exhibit marked differences: the former concentrates on the local field with fine granularity and high diversity, while the latter focuses on the global field with coarse granularity and limited diversity. During training with RPM, deep layer information maintains attention on both local and global fields without sacrificing granularity or diversity. Clearly, deep layer information, augmented with relationship prompt learning, is more suitable for pixel-level semantic segmentation tasks, thereby diminishing the need for additional segmentation-specific networks. In addition, we visualize the relationship attention map in Figure 1 . The first line denote the relationship attention maps for seen classes across each layer; the last two lines for unseen classes. For seen classes, the model has prior pixel-level semantic, so the relationship attention map only needs to focus on a few pixels to guide the model to make predictions for these pixels (e.g., the relationship attention map for airplane has fewer highlighted areas). For unseen classes, the model lacks corresponding semantic, so the relationship attention map needs to focus on more pixels to provide the model with more sufficient pixel-level semantic (e.g., the relationship attention map for sheep highlights the complete semantic at shallow layers).Impact of Different Modules. Table 4 shows the impact of various modules. RPM (the first line) denotes the combination of M2oE, ITP and APG. RPM without M2oE (the second line) denotes the ablation about APG and ITP. Utilizing RPM and LPM yields the best performance. The performance improvement attributed to RPM (+62.6%) is significantly greater than that of LPM (+35.5%). Furthermore, integrating LPM on top of RPM yields a modest performance gain of 1.9%. In contrast, incorporating RPM based on LPM results in a substantial 29% improvement. Therefore, we conclude that VLM with relationship prompt learning is enough for OVSS without any explicit segmentation-specific networks. show that performance enhancements are marginal with increasing dimensions. Given the trade-off between performance gain and parameter increase, we adopt r = 3. The 'Multi-Scale' mode refers to removing the gating network and directly aggregating the features processed by all expert networks (See Appendix B for more details). Exploration of Different Designs in LPM. We evaluate three different LPM designs (shown in Figure 6 ) using Softmax and Sigmoid losses.The results in Table 6 reveal that LPM c with Sigmoid loss is the most effective strategy. In addition, using Sigmoid loss is significantly better than using Softmax loss. This reflects that the relationship prompt does not directly focus on pixels, but follows the image-to-pixel process. Impact of Training-Free Projection Modules. Note that our method comprises two trainable modules: RPM and LPM. To further explore the performance of RPM, we eliminate all linear layers in LPM to construct three variants of training-free projection networks. Following the sequence depicted in Figure 6 , we denote the training-free projection modules as TFPM a , TFPM b and TFPM c . The results in Table 7 demonstrate that both TFPM b and TFPM c can achieve the state-of-the-art performance. The suboptimal results with TFPM a suggests that the relationship prompt guides the plain encoder to perform pixel-level classifications by encoding semantics in the [CLS] token, which should not be disregarded when obtaining segmentation results. 
conclusion: 
 In this work, we propose the Relationship Prompt Module (RPM) to guide VLM to transform its image-level embeddings into pixel-level semantic ones. RPM and VLM combine to form Relationship Prompt Network (RPN), a VLM-based OVSS method that directly performs OVSS without any explicit segmentation-specific networks. To the best of our knowledge, we are the first to give a straightforward solution for OVSS that applies prompt learning solely to VLM. We evaluate our method on four public benchmark datasets in both zero-shot and open-vocabulary settings, and achieve the state-of-the-art performance with only about 3M trainable parameters (2% of total parameters). Therefore, it is concluded that VLM with relationship prompt learning is enough for open-vocabulary semantic segmentation without any explicit segmentation-specific networks.Limitations. Although we meticulously design an effective prompt learning method for directly using VLM to achieve pixel-level OVSS, there are several ways for prompt learning to achieve further improvement: 1) directly acting on the attention map (more direct); 2) dynamically reorganizing multi-head attention map (more lightweight). 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We provide theses information in our paper.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: Yes.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: Our paper belongs to foundational research.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: Our paper belongs to foundational research.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: They are properly credited and respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: We show the comparison of our method and the existing VLM-based methods in Figure 8 . In our method, we abandon the explicit segmentation-specific networks, i.e., the mask proposal network and the semantic decoder and guide VLM to directly output the segmentation results. In existing VLMbased methods, the entire OVSS framework consists of VLM, the mask proposal network and the semantic decoder, incurring extensive training cost. As illustrated in Section 1, the two-stage methods usually adopt knowledge distillation to transfer zero-shot learning ability to the segmentation-specific network, and the one-stage methods utilize feature adaptation to train the semantic decoder. In summary, prompt learning acts as the engine to drive VLM to directly achieve OVSS in our method.Conversely, in the existing VLM-based methods, VLM acts as the engine to drive the segmentationspecific network to indirectly achieve OVSS. Therefore, our method is more straightforward and more parameter-efficient. Details The role of the gating network is to select the right expert for each sample in a batch. Thus, we need to calculate the gating scores G(x) ∈ R B×n , i.e., the scores for B samples in a batch to n experts. For simplicity, let the input, i.e., the patch embeddings p, be denoted by x ∈ R B×N ×d , where N = h × w. The input x is first directed into a common linear to reduce the dimension d to r, and then is reshaped to the size of R B×r after global average pooling (as Eq. 12). Like in [42, 43] , the value H(x) ∈ R B×n is calculated as follows:x a = Reshape(AvgPool(LinearEQUATIONEQUATION) where W g ∈ R r×n and W noise ∈ R r×n represent the trainable gating weight and the noise term, respectively. We select only the top k values on H(x) and set the rest to -∞. After 'Softmax' operation, the gating scores G(x) can be calculated as follows:EQUATION)The role of the experts is to extract feature with multi-scale, which consist of Interpolate operation at a specific scale, DWConv 3×3 and Upsample operation to map the feature to the original scale. Let i-th expert be E i with scale s i = 1 2 i-1 , where i = 1, 2, 3, 4. The expert E i processes the input x as follows:EQUATIONFinally, we obtain the output according to Eq. 1.Multi-Scale An intuitive alternative is applying a multi-scale strategy, which utilizes n branches to extract features with different scales. The structural difference with M2oE is its lack of the gating network, as illustrated in Figure 9 . The absence brings a higher parameter consumption, due to the fact that the gating network helps selectively activate sparse experts. The motivation behind the design of RPM is inspired by LoRA [58] . As illustrated in Figure 10 (a), considering that the main components of large models lie in a low intrinsic dimension, LoRA introduces dimensionality reduction matrix A and dimensionality enhancement matrix B into the bypass to fit the frozen weights. This approach enables fine-tuning the model with minimal training costs, thereby improving performance for specific tasks without changing the original parameters.Given that VLM has two branches to process image and text data, we initially adopt a common-bypass mode as shown in Figure 10 (b) . The distinction between LoRA and VLM LoRA lies in the fact that matrix A in VLM LoRA accepts two types of modal inputs and it is training-free. It is important to note that we do not connect the output of matrix B with the embeddings from the text module for two reasons: their dimensions do not match and the relationship attention map from matrix A performs more effectively as a visual prompt rather than a textual prompt, as illustrated in Figure 1 . However, the performance of implementing VLM LoRA was suboptimal. We discovered that a direct skip connection with the output of the current image weights that bypasses matrices A and B, achieves the same effect. Our analysis suggests that merely tuning the lightweight matrix B in a post-tuning model is found to be insufficient for adapting parameter-intensive weights. Consequently, we propose a prefix-tuning architecture as illustrated in Figure 10 (c ). We position the two matrices between the preceding and current weights and established a skip connection with the original output of the preceding image weights rather than the current image weights. This circumvents the need to fit the parameter-intensive image weights. In light of this prefix-tuning architecture, we categorize this model into the category of prompt learning rather than adaptation, and thereby naming it relationship prompt learning. In addition, we consider three operations between the output of the preceding image weights and the output of matrix B: concatenation, addition, and Hadamard product. Following a quantitative analysis (shown in Table 14 ), we select the concatenation operation.Zero-Shot Semantic Segmentation Setting We adopt the popular zero-shot setting as follows in [13, 9, 4, 3, 1, 31] . In the setting, we divide all classes into seen and unseen, only the seen classes are used in training. For the VOC dataset, we select 15 seen classes and 5 unseen classes, with the 'background' class excluded. For the COCO dataset, we divide all classes into 156 seen classes and 15 unseen classes. For the Context dataset, we select 50 seen classes (including 'background') and 10 unseen classes. These unseen classes are defined as follows:Dataset Unseen Classes VOC pottedplant, sheep, sofa, train, tvmonitor COCO cow, giraffe, suitcase, frisbee, skateboard carrot, scissors, cardboard, clouds, grass playingfield, river, road, tree, wall concrete Context cow, motorbike, sofa, cat, boat, fence bird, tv monitor, keyboard, aeroplaneWe employ the open-vocabulary setting as follows in [14, 12, 51, 53] . In this setting, we train our model on the COCO dataset and evaluate it on other datasets, including the VOC dataset with 20 classes (PAS-20), the Context dataset with 59 and 459 classes (PC-59 and PC-459), the ADE20K dataset with 150 and 847 classes (A-150 and A-847).Text Prompt Templates For the VOC dataset, we apply a single template 'A photo of a {}'. For the Context and COCO dataset, we apply multiple templates as follows:'A photo of a {}.' ; 'A photo of a small {}.' ; 'A photo of a medium {}.' ; 'A photo of a large {}.' 'This is a photo of a {}.' ; 'This is a photo of a small {}.' ; 'This is a photo of a medium {}.' ; 'This is a photo of a large {}.' 'A {} in the scene.' ; 'A photo of a {} in the scene.' 'There is a {} in the scene.' ; 'There is the {} in the scene.' 'This is a {} in the scene.' ; 'This is the {} in the scene.' ; 'This is one {} in the scene.'We show the full performance comparison of existing methods in the zero-shot and open-vocabulary semantic segmentation settings in Table 8 and Table 9 .Comparison with PEFT Methods We explore some parameter-efficient fine-tuning (PEFT) methods with baseline (i.e., the CLIP with LPM): 1) fine-tuning entire baseline; 2) only fine-tuning LPM; 3) BitFit [70] , a sparse-fine-tuning method where only the bias-terms of the model (or a subset of them) are modified; 4) Adapter [71] , which inserts a trainable adapter module between the transformer layers. 5) VPT [24] , which inserts trainable tokens to the input feature of each transformer layer; 6) LST [72] , which trains a ladder-side network, a small and separate network that takes intermediate activations as input via shortcut connections (called ladders) from the backbone networks and makes predictions. 7) SSF [73] , which trains scale and shift parameters to modulate the visual features. 8) LoRA [58] , which inserts a trainable dimensionality reduction matrix and a dimensionality enhancement matrix in parallel to the frozen weights. As illustrated in Table 10 , our method shows significant improvements compared to PEFT methods. This is attributed to the fact that these PEFT methods only focus on how to train the baseline with fewer parameters, suitable only for image-level feature representation tasks, while our method not only trains the baseline with fewer parameters, but also focuses on applying capabilities suitable for image-level feature representation tasks directly to pixel-level semantic segmentation tasks. In summary, existing PEFT methods mainly focus on fine-tuning the task-specific model to improve performance on that task, while our method enables VLM to directly perform semantic segmentation.Combination with VPT We explore the combination of our method and VPT in Table 11 . When using VPT alone and in combination with our method, the number of trainable tokens in each layer is 400 and 40 respectively. Considering the impact of parameter initialization in VPT, the performance change after the combination is negligible. Impact of Different Pre-trained Weights for the Plain Encoder We explore different pre-trained weights of the plain encoder in Table 12 . These weights are from supervised learning, unsupervised learning and weakly-supervised learning of text signals. ViT [56] applies supervised learning to learn visual representation. MAE [74] applies self-supervised learning to learn rich semantics. The two methods learn uni-modal knowledge. CLIP [5] applies text as a supervisory signal and adopts contrastive learning to learn general visual representation. It is a multi-modal learning method. The results show that our method has limitations in mining uni-modal visual knowledge. In addition, compared to ViT and MAE, CLIP uses more data to pre-train the plain encoder. This also shows that even though our method is superior, it still cannot get rid of the dependence on a large amount of data.We adopt a layer-by-layer guidance mode (i.e., all) as shown in Table 13 . Our method is evaluated across various network depths, including shallow layers, deep layers, and interval layer mode. We find that the performance of single-layer mode (such as layers 1 or 11) are significantly worse than those of multi-layer modes. The layer-by-layer mode demonstrates optimal performance on both the VOC and COCO datasets.Impact of Different Prefix-tuning Operations We compare three prefix-tuning operations as shown in Figure 10 (c): concatenation, addition, and Hadamard product. Results from Table 14 indicate that the relationship prompt should not directly modify the original feature (like addition or Hadamard product), but rather should influence the model indirectly by computing the attention between the prompt and the original feature (like concatenation).Baseline represents the frozen CLIP model with LPM. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.To achieve the idea, some works train the VLM from scratch using pixel-level supervision, which relies on large-scale pixel-level pre-training. Note that we aim to employ a parameter-efficient fine-tuning method to reduce the need for additional segmentation-specific networks and large-scale pre-training. 


Decision: Overall the reviews found the contribution in terms of technique too small, and the evaluation of benefit lacking for several reasons.  There was some confusion about acronyms that seems reasonable given the authors' choice of using coining two, RPM and RPN, that are similar to each other and to other acronyms used in the field.  The ability to adapt a VLM, without an additional explicit segmentation model, to come close to the state of the art is hopefully sufficient with some careful clarification in the paper text.

Reviews: review: paper_summary: paper_summary: This paper primarily studies Open-Vocabulary Semantic Segmentation. The main contribution of this paper is the proposal of RPN, which employs relationship prompt learning solely to perform OVSS without any segmentation-specific networks. The authors claim that RPN attains state-of-the-art results on four public benchmarks.
strength_weakness: strength_weakness: The weaknesses of this paper can be summarized as follows,

- The authors claim that their method achieved state-of-the-art results, but the results in Table 3 and Table 8 do not support this claim. Additionally, the methods listed in Tables 2, 3, 7 and 8 are somewhat limited. More recent works should be included to substantiate the claim that the proposed method achieves state-of-the-art results. (refer to https://paperswithcode.com/task/open-vocabulary-semantic-segmentation)

- The authors should provide a more detailed explanation of the content in Figure 2, either in the caption or in the introduction section. (Or, they could reference this figure in a more suitable location, such as the first paragraph of the introduction section. Or, the authors could mention that a more detailed explanation is provided in the supplementary materials.)

- According to the authors' definition, there have already been many works that achieve open vocabulary semantic segmentation without segmentation-specific networks, e.g., "Image Segmentation Using Text and Image Prompts", "GroupViT: Semantic Segmentation Emerges from Text Supervision", "CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation", "SegGPT: Towards Segmenting Everything In Context", to name a few. These works, to some extent, diminish the novelty of the authors' contribution. (To be clear, I am not saying that the authors' work lacks novelty entirely, but its novelty is not as strong as claimed in the paper.)

- Did the authors conduct any ablation studies on the hyper-parameters that appear in Equations 9 to 11?

- Why do the authors use class embeddings from only text layer 12, while using image embeddings from various image layers? Can the authors provide some analysis or experimental support for this choice?

- When visualizing the attention map in Figure 1, the authors could also include a color legend to represent the response intensity.

- Figures 4 to 6 should include annotations to explain the meanings of the various symbols used in the images.

- Did the authors conduct any ablation studies on how to select the scale and the number of experts used in M2oE?

- Is the Hadamard product in Equation 2 significant? Would using a matrix product between $p^i$​ and $g^i$ instead affect the results?

- Why is pixel-level information used instead of patch-level information?

- Does the proposed algorithm produce block artifacts when generating segmentation maps for high-resolution images?

Overall, the paper is quite dull and does not stand out in the field of open vocabulary semantic segmentation. However, for a conference paper, the amount of work presented seems sufficient.
questions: questions: I have placed all the questions I want to ask in the weaknesses box. Overall, the paper doesn't seem to have any major issues, but personally, I find the work uninteresting and believe it offers limited insights to the field of open vocabulary semantic segmentation. However, if other reviewers feel that this work is worth accepting, I wouldn't strongly oppose it.
limitations: limitations: The authors did not discuss the limitations of their proposed algorithm in either the main text or the supplementary materials. 
I have highlighted some limitations of the paper in the weaknesses box. The authors can refer to these points to further improve the quality of the paper.
paper_summary: paper_summary: This paper proposes relationship prompt module (RPM), which generates relationship prompt that directs VLM to extract pixel-level semantic embeddings suitable for OVSS. Moreover, RPM integrates with VLM to construct relationship prompt network (RPN), achieving OVSS without segmentation-specific networks. RPN attains state-of-the-art performance with merely about 3M trainable parameters (2% of total parameters).
strength_weakness: strength_weakness: 1. In the experiment tables, there is no Efficiency comparison for ADE20K and Context dataset. 

2. The ablation studies in Table 4 are insufficient; they only include M2oE and LPM ablation studies, and do not cover ITP and APG.

3. The performance is not state-of-the-art compared to some previous works, such as CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation.
questions: questions: Please refer to the concerns and issues raised in the "Weaknesses".
limitations: limitations: Please refer to the concerns and issues raised in the "Weaknesses".
paper_summary: paper_summary: The paper proposes a training and inference-efficient Relationship Prompt Network (RPN). This network leverages a layer-wise Relationship Prompt Module (RPM) utilizing tuning methods similar to VLM LoRA and an improved Linear Projection Module (LPM) without relying on a segmentation model. The authors conduct extensive experiments, demonstrating the solid efficiency and effectiveness of their approach.
strength_weakness: strength_weakness: The paper lacks a discussion on region-text relationships, focusing instead on directly leveraging pixel-text relationships.
Additionally, there should be a discussion and experiments with other visual prompt tuning methods applied to the baseline.
questions: questions: See the weakness.
limitations: limitations: Please provide a more detailed analysis on the correlation between the relational abilities of the model and its performance in open-vocabulary tasks (separating seen and unseen classes).
paper_summary: paper_summary: This paper tackles open-vocabulary semantic segmentation, a task aiming at per-pixel predictions of image inputs on any given classes. Existing methods are built on pre-trained vision-language models like CLIP. Different from CLIP that achieves coarse open-vocabulary classification, open-vocabulary segmentation extends this understanding capability to pixel level. This paper claims that existing approaches take much compute for training. Authors develop a Relationship Prompt Network that features dense image-text correlations across all layers. Also, their approach includes some training techniques, including mixture-of-experts, multi-scale input and prompt learning. Experimental results show that proposed RPN achieves performance gains against published methods like ZegCLIP [19], MaskCLIP [8] and FreeSeg [10].
strength_weakness: strength_weakness: - **Unclear unique contributions.** I am not quite understand about the unique contribution from this paper. Does this paper give new insights to the community? The proposed relationship prompt is a combination of powerful tuning methods that have been proven effective, including dense image-text attentions, mixture-of-experts, multi-scale inputs and prompt learning. None of these modules are emphasized and I cannot see the unique value of applying these techniques. Performance gains from these techniques does not surprise me that much.
- **Many typos and confusing claims**. Please refer to Questions 1-11 for details. I think these issues should be taken care of before considered for publication.
questions: questions: - Line 32, “it is challenging to apply prompt learning solely to VLM for OVSS”. Has this claim been checked? Why applying prompt learning to VLM for OVSS is challenging? Does “Challenging” refer to “technically challenging”, right?
- Line 49-51, “we find they can construct an image-text relationship attention map via the attention mechanism”. Image-text relationship attention map is a confusing claim. This should refer to a well defined phrase in this field, such as language-aware deep fusion in [A], which is an influential paper in this field.
- Figure 1, how are these attention maps plotted?
- Line 58-61, we propose linear projection module comprising merely two individual linear layers, which maps the image and text feature into a shared space, and then computes their matrix product to produce the results. What should matrix product refer to? Element-wise or column-wise?
- Typos. Line 67 and 69, RPN to RPM. There is potential confusion between RPM and RPN.
- Line 113-114, reducing parameter consumption. I do not think this is a correct claim. Adding dense cross-attentions will surely increase parameters for inference. I do not see techniques in this paper that reduces parameters like quantization.
- Line 113-125, refer to a figure here is better for illustration purpose.
- Figure 2, what is M2oE, ITP and APG block? A bit confusing.
- Table 4, what does the first line mean and what is the difference between the first and the second line?
- Figure 7, only legends of head 1 and 2 are given, while others are missing. In such case, I cannot understand what is presented in this figure.
- Table 1, instead of params, this should be Learnable params, right?

[A] Grounded Language-Image Pre-training. CVPR 2022.
limitations: limitations: Authors have included limitations of proposed RPN in Appendix D.

Comments: comment: Thanks for your patient response and comments. We would like to reiterate our contribution: we only use image-level supervised VLM (note that no segmentation-specific network is introduced) and achieve OVSS with extremely low training cost. We are the first to do so. Although we did not reach a consensus in the end, thank you for your discussion and wish you all the best in your future research.
Thanks for authors prompt and active replies. For **Q2**, I think presented results convince me a bit. However, in spite of performance gains presented in this paper and many turns of debating, I am still confused on unique contributions from this paper and remain doubts on its novelty. Authors seem to claim very high-level/vague contributions (could claimed by most multi-modal learning papers) without theoretical or empirical justifications and unique insights, which do not convince me that much. So far I am inclined to reject this paper and waiting other reviewers response for a second opinion.
Dear Reviewer,

Thank you again for reviewing our manuscript. We have tried our best to address your questions, and revised our paper by following suggestions from all reviewers.

Please kindly let us know if you have any follow-up questions or areas needing further clarification. Your insights are valuable to us, and we stand ready to provide any additional information that could be helpful.
Dear Reviewer,

Thank you again for reviewing our manuscript. We have tried our best to address your questions, and revised our paper by following suggestions from all reviewers.

Please kindly let us know if you have any follow-up questions or areas needing further clarification. Your insights are valuable to us, and we stand ready to provide any additional information that could be helpful.
Dear Reviewer,

Thank you again for reviewing our manuscript. We have tried our best to address your questions, and revised our paper by following suggestions from all reviewers.

Please kindly let us know if you have any follow-up questions or areas needing further clarification. Your insights are valuable to us, and we stand ready to provide any additional information that could be helpful.
Dear Reviewer,

Thank you again for reviewing our manuscript. We have tried our best to address your questions, and revised our paper by following suggestions from all reviewers.

Please kindly let us know if you have any follow-up questions or areas needing further clarification. Your insights are valuable to us, and we stand ready to provide any additional information that could be helpful.
Thank you for your prompt reply. We would like to further discuss.

**Response for Q1**  
Compared to training a pre-trained model, training a vanilla model from scratch means introducing a large-scale training set (either pixel-level supervision or image-level supervision), so this large-scale training set is "additional" to the training process of the pre-trained model. For GroupViT, it use CC and YFCC as "additional" image-level supervised training sets; for SegGPT, it use a mixture of ADE20K, COCO, PASCAL VOC, Cityscapes and so on as "additional" pixel-level supervised training sets.

**Response for Q2**  
According to your comments, the comparison with FC-CLIP and F-VLM in #params is shown as (note that F-VLM does not provide the corresponding number of trainable parameters, so we roughly estimate the value based on their open source code for reference only):
|(ADE20K)|#Params(M)|
|:-|:-:|
|FC-CLIP|21|
|F-VLM|38|
|RPN(ours)|3.2|

**Response for Q3**  
We would like to further elaborate on our point of view.  
First, we would like to illustrate the relational modeling capabilities of RPN to answer the question “Why RPN is better than them?”. The relational capability of the model is reflected in guiding the model to make pixel-level predictions for unseen classes by establishing relationship attention between text class information and pixel-level visual information, making its performance close to that of seen classes. We visualize the relationship attention map in Fig.1. The first line denote the relationship attention maps for seen class across each layer; the last two lines for unseen classes. For seen class, the model has prior pixel-level semantic information, so the relationship attention map only needs to focus on a small number of pixels to guide the model to make class predictions for these pixels (e.g., the relationship attention map for airplane has fewer highlighted areas). For unseen class, the model lacks corresponding pixel-level semantic information, so the relationship attention map needs to focus on more pixels to provide the model with more sufficient pixel-level semantic information (e.g., the shallow relationship attention map for sheep highlights the complete semantic information). In addition, to demonstrate the guidance mechanism, we conduct related interpretability experiments (i.e., MAD evaluation), as illustrated in L.271-282. The MAD results show that with the guidance of relationship prompt, the deep features of VLM gradually have a wider MAD value range, which indicates fine-grained semantic properties. We also conduct system level comparison as shown in Tab.2,3, 7 and 8 to demonstrate the superior performance.  
Second, generating prompt from randomly initialized learnable vectors, e.g., VPT[1*], make poor performance (shown in Tab.10). We think that the randomly initialized learnable prompt cannot be associated with the VLM’s prior image-text knowledge and therefore cannot effectively guide the VLM to perform pixel-level prediction directly.

[1*] Visual prompt tuning.
Thanks authors for further detailed and thorough clarifications. My confusion remains and I think they may need further discussions.

For **Q1**, I firstly get confused because you include GroupViT in **Global Response Q1** to a category of approaches that use "additional" training sets. You use SegGPT as an example to explain this, which appears very confusing to me, because SegGPT and GroupViT do not even fall into the same category (GroupViT does not apply pixel-level supervision while SegGPT does). For training multimodal segmentation network, which dataset do you perceive as "additional", "image-caption pairs" or "pixel-wise masks"?

For **Q2**, one suggestion for authors is to include number of parameter (#param) for F-VLM, FC-CLIP and proposed RPN.  

For **Q3**, authors claim that "MaPLe does not construct a prompt with image-text relationship, but only generates a text prompt from a randomly initialized learnable vector and maps it to an image prompt with the help of a linear layer." I think "good image-text relationship" is a very vague claim and does not convince me that much, lacking theoretical or empirical justifications. Many CLIP-related papers study how to build "good or better image-text relationship". Why RPN is better than them? Also, why authors think "generates text prompt from randomly initialized learnable vectors" is a worse try, any justification?
Thanks for your exhaustive reply and suggestions. Here we have three more clarifications:

**Q1: I am not quite sure if I clearly understand “additional training sets”. Why are these training sets “additional”?**  
There are two main ways to achieve OVSS using VLM: (1) Training VLM with pixel-level supervision from scratch allows it to directly output pixel-level predictions, which means that the process needs to introduce “additional” pixel-level supervised training sets, like SegGPT[1*] use a mixture of ADE20K, COCO, PASCAL VOC, Cityscapes and so on to train ViT to directly output segmentation results. (2) Unlike training from scratch, adopting knowledge distillation or feature adaptation to transfer the frozen VLM's semantic knowledge to segmentation-specific networks dose not introduce “additional” pixel-level supervised training sets, except COCO under the OVSS setting requirements. Thus, the former usually needs “additional training sets; and the latter needs additional segmentation-specific networks.

[1*] SegGPT: Towards Segmenting Everything In Context

**Q2: Performing parameter-efficient segmentation on top of a frozen VLM (or fine-grained understanding) is very straightforward. I do not think this is a novel motivation.**  
As answered in **General Response-Q1**, training from scratch relies on additional training sets and large-scale pre-training, while only needs a simple VLM; adopting knowledge distillation or feature adaptation relies on additional segmentation-specific networks based on VLM, while only needs small-scale fine-tuning. Thus, is it possible to combine the advantages of both, using small-scale fine-tuning while only requiring a simple VLM (discarding the complex segmentation-specific networks)? This is exactly what we do. We would like to re-emphasize our contribution as: **we only use VLM with extremely low training cost (almost 3M trainable parameters) to directly achieve OVSS. (Please note that we do not rely on segmentation-specific networks and only need parameter-efficient fine-tuning)**. We would like to use the table in **General Response-Q1** again to illustrate the difference between our approach and the methods you listed.
||Additional Training Sets|Parameter-intensive Decoder Adapter|Large-scale Pixel-level Pre-training|
|:-|:-:|:-:|:-:|
|F-VLM|❌|✔|❌|
|FC-CLIP|❌|✔|❌|
|RPN(ours)|❌|❌|❌|

**Q3: Authors claim that "current prompt … a lack of an image-text relationship, which hinders the exploration of VLM potential for open-vocabulary scene understanding." This is not an accurate claim, considering paper[1\*].**  
As illustrated in L.107-108, MaPLe adopts trainable prompt to guide both visual and textual embeddings and proposes a coupling function as a bridge to build a multi-modal prompt. The coupling function is implemented as a linear layer, which maps the text prompt to the image prompt. Note that the mapping process of text-to-image prompt completely relies on text prompt without introducing the image embeddings of the VLM. In addition, the generation of text prompt also relies solely on a randomly initialized learnable vector that is independent of the text embeddings of VLM. Therefore, in essence, MaPLe does not construct a prompt with image-text relationship, but only generates a text prompt from a randomly initialized learnable vector and maps it to an image prompt with the help of a linear layer.

[1*] MaPLe: Multi-modal Prompt Learning.
Thanks authors for detailed responses to my concerns. Most typo concerns have been addressed, but surely need better clarifications and clearer presentation in the future manuscript. I do have a few previous concerns regarding contributions and novelties for this paper, which I think is more important and have not addressed by this rebuttal, and also a few new ones which need better clarifications. 

### Reply to Q1 (General Response-Q1)

1.  I am not quite sure if I clearly understand “additional training sets”. Why are these training sets “additional”?

2.  Another thing is, performing parameter-efficient segmentation on top of a frozen VLM (or fine-grained understanding) is very straightforward [A,B]. I do not think this is a novel motivation. 

### Reply to Q2

1. Authors claim that "current prompt … a lack of an image-text relationship, which hinders the exploration of VLM potential for open-vocabulary scene understanding." This is not an accurate claim, considering paper [C]. 

In all, I still do not see the unique value from this paper. Many positive factors in this paper that could lead to performance gains, as I included in **Weaknesses 1**. I will adjust my score based on responses from other reviewers and how authors clarify their unique contributions. 

[A] F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models. ICLR’23.

[B] Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP. NeurIPS’23.

[C] MaPLe: Multi-modal Prompt Learning. CVPR’23.