Open Vocabulary Panoptic Segmentation With Retrieval Augmentation

Authors: Nafis Sadeq, Qingfeng Liu, MOSTAFA EL-Khamy
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Panoptic Segmentation, Open Vocabulary, Retrieval Augmentation

Abstract:
Given an input image and set of class names, panoptic segmentation aims to label each pixel in an image with class labels and instance labels. In comparison, Open Vocabulary Panoptic Segmentation aims to facilitate the segmentation of arbitrary classes according to user input. The challenge is that a panoptic segmentation system trained on a particular dataset typically does not generalize well to unseen classes beyond the training data. In this work, we propose a retrieval-augmented panoptic segmentation method that improves the performance of unseen classes. In particular, we construct a masked segment feature database using paired image-text data. At inference time, we use masked segment features from the input image as query keys to retrieve similar features and associated class labels from the database. Classification scores for the masked segment are assigned based on the similarity between query features and retrieved features. The retrieval-based classification scores are combined with CLIP-based scores to produce the final output. We incorporate our solution with a previous SOTA method (FC-CLIP). When trained on COCO, the proposed method demonstrates 30.9 PQ, 19.3 mAP, 44.0 mIoU on the ADE20k dataset, achieving +4.5 PQ, +2.5 mAP, +10.0 mIoU absolute improvement over the baseline.

Decision: reject
Reviewer scores: 5, 5, 3, 3

Reviews and discussion:
<thread>
creation date:
1729503038802

thread_type:
official_review

<thread_object>
creator: reviewer_goti

<content>
summary:
This paper introduces a retrieval-based method to enhance the performance of open vocabulary panoptic segmentation by constructing a feature database from paired image-text data. During inference, the model uses masked segment features from the input image to query the database for similar features and associated class labels, which are then combined with CLIP-based scores. This approach leads to improvements in Panoptic Quality in both training-free and cross-dataset settings.

strengths:
The paper explains related concepts clearly and details the methodology comprehensively, making the overall article easy to understand.

weaknesses:
1. The novelty of the paper is limited, primarily building upon the feature retrieval idea from Gui et al.[1]. Compared to Gui et al. [1]., the main modifications only include using a single CLIP backbone instead of two backbone. Please explain how these contributions can meet the strict requirements of top-level conferences.

2. The authors use open vocabulary object detection combined with SAM to build the feature database, which limits the model's performance to the capabilities of the object detection component. Please explain how to handle classes that are not included in both the feature database and the fallback dataset during inference, or discuss the limitations of their approach for truly open-vocabulary scenarios.

3. The definitions of IV Classification and OOV Classification are confusing. Why is it considered that the segment features and text embedding after linear projection in Figure 1 are equivalent to IV Classification? Please provide a more detailed explanation of the distinction between these two classifications and why the linear projection is significant for IV Classification.

4. The experimental section lacks a critical component: comparisons with state-of-the-art methods, such as Gui et al. [1]., HIPIE [2], ODISE [3], OPSNet [4]. Please explain why these specific comparisons are not included and how your method compares theoretically to these state-of-the-art approaches.

5. How does this method perform on open vocabulary semantic segmentation tasks, such as testing on ADE20K-847, ADE20K-150, Pascal Context-459.

6. The paper claims to achieve performance improvement by utilizing a completely different dataset with only image level annotations. However, using the ADE20K training set to construct a feature database and evaluating it on the ADE20K validation set in the experiment lacks persuasiveness for open vocabulary. Please clarify how to ensure the open vocabulary nature when using the same dataset for both feature database construction and evaluation.

7. There is irrelevant content in the lower-left corner of Figure 2. Please redraw the figure and ensure that the image is complete and free from irrelevant content


reference:

[1] Zhongrui Gui, Shuyang Sun, Runjia Li, Jianhao Yuan, Zhaochong An, Karsten Roth, Ameya Prabhu, and Philip Torr. knn-clip: Retrieval enables training-free segmentation on continually expanding large vocabularies, 2024. URL https://arxiv.org/abs/2404.09447.

[2] Wang X, Li S, Kallidromitis K, et al. Hierarchical open-vocabulary universal image segmentation[J]. Advances in Neural Information Processing Systems, 2024, 36.

[3] Xu J, Liu S, Vahdat A, et al. Open-vocabulary panoptic segmentation with text-to-image diffusion models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 2955-2966.

[4] Chen X, Li S, Lim S N, et al. Open-vocabulary panoptic segmentation with embedding modulation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 1141-1150.

questions:
1. What is the difference between IV Classification and OOV Classification cia CLIP in cross-dataset panoptic segmentation? What is the significance of this distinction? From Figure 1, it appears that the former only differs from the latter by including a linear projection.
2. What is the fallback dataset, and how the author build it?
</content>
</thread_object>

</thread>
<thread>
creation date:
1729754660016

thread_type:
official_review

<thread_object>
creator: reviewer_xajy

<content>
summary:
This paper enhances open-vocabulary panoptic segmentation by leveraging retrieval augmentation to address the challenges of classifying unseen objects. The authors propose a framework that integrates masked segment features with a retrieval-based method to improve performance for unseen classes. The model builds a feature database using paired image-text data and retrieves similar features during inference to classify masked segments. These retrieval-based scores are combined with CLIP-based scores to enhance accuracy. When applied to FC-CLIP, the proposed method demonstrates improvements in unseen classes on the ADE20k dataset.

strengths:
Applying Retrieval Augmentation to vision tasks is a promising direction. The proposed way of constructing a database is interesting.

weaknesses:
1. While the method builds on FC-CLIP, the authors do not provide an introduction to FC-CLIP, which makes the paper hard to follow during reading.
2. The feature database should be introduced prior to discussing the retrieval method to improve the flow and clarity of the paper.
3. Since retrieval augmentation is intended to be a more general approach, the paper would benefit from presenting a more generalized framework to reflect its broader applicability.
4. The method of constructing the feature database itself serves as a strong baseline. How does the performance of the proposed retrieval-augmentation approach compare to Grounding DINO?
5. The paper lacks essential evaluations (the method is only evaluated on a single dataset with a single base model) and ablation studies.

questions:
Please refer to the weaknesses. I think the current version is not ready for publication. More experiment results are expected.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730526865845

thread_type:
official_review

<thread_object>
creator: reviewer_x1um

<content>
summary:
The paper presents a novel approach to address the challenge of segmenting arbitrary classes in images, a task known as open vocabulary panoptic segmentation. The authors propose a retrieval-augmented method that leverages a masked segment feature database constructed from image-text pairs. During inference, the system uses masked segment features from the input image to retrieve similar features and class labels from the database, combining these retrieval-based classification scores with CLIP-based scores to produce the final output. The method is evaluated on the ADE20k dataset and shows significant improvements over the baseline, particularly when fine-tuned on the COCO dataset, with absolute improvements of +4.5 PQ, +2.5 mAP, and +10.0 mIoU.

strengths:
1. The paper introduces a creative solution to the open vocabulary panoptic segmentation problem by combining retrieval-based classification with CLIP, which is an original approach not commonly seen in the literature.
2. The paper is well-structured, with clear explanations of the methodology.

weaknesses:
1. While the paper demonstrates improvements over the baseline, it does not provide a direct comparison with other state-of-the-art methods in the field, which could provide additional context for the significance of the results.
2. The discussion on how the proposed method generalizes to unseen classes could be expanded, as this is a critical aspect of open vocabulary segmentation.
3. The paper could further discuss the limitations of the retrieval-augmented approach, especially regarding the reliance on the quality of the feature database and the potential scalability issues as the number of classes increases.

questions:
1. Could the authors elaborate on how their method generalizes to completely unseen classes that are not represented in the feature database?
2. As the number of classes in the feature database grows, how does the retrieval process scale in terms of computational resources and accuracy?
3. Are there any plans to compare the proposed method with other leading approaches in the field to contextualize the improvements?
4. The paper mentions that the quality of mask proposal generation is crucial. Could the authors provide more details on how variations in mask quality affect the final segmentation results?
5. Is there potential to integrate this method with other modalities, such as depth information or video data, to further improve performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731137804728

thread_type:
official_review

<thread_object>
creator: reviewer_5aag

<content>
summary:
This paper presents an approach for open vocabulary panoptic segmentation by combining retrieval-based classification with standard image segmentation. In particular, the authors introduce a retrieval-augmented segmentation method that utilizes a database of paired image-text features. During inference to address the challenge of domain shift between masked and natural images, the model retrieves relevant features from this database using masked segment features from the input image as queries. This retrieval-based score is combined with scores from a vision-language model (CLIP) to enhance classification accuracy for unseen classes.

strengths:
- The reviewer likes the integration of retrieval-based classification with CLIP-based scores to address the domain shift issues between masked images and natural images. It clearly improves the model's ability to recognize unseen classes without additional training.

- The paper's approach to construct a feature database from widely available paired image-text data is interesting. This setup enables adaptability without requiring pixel-level annotations.

- The paper is well-organized and well-written.

weaknesses:
- The reviewer feels that the retrieval-based classification relies heavily on the quality and diversity of the feature database constructed from paired image-text data. If the database lacks sufficient variety or coverage, the method may struggle to classify certain unseen classes accurately, particularly in real-world scenarios with a wide range of objects.

- Further, the reviewer observed that the method uses Grounding DINO and SAM for generating masks in the training-free setup. However, SAM can produce suboptimal masks without human input which may degrade segmentation accuracy. This dependence on mask quality can limit the method’s effectiveness in fully automated settings.

- The authors may want to include methods such as ODISE for a more comprehensive analysis.

questions:
N/A
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a method for open vocabulary panoptic segmentation that merges retrieval-based categorization with conventional image segmentation. The manuscript was reviewed by four experts in the field. The recommendations are (2 x "3: reject, not good enough", 2 x "5: marginally below the acceptance threshold"). The reviewers raised many concerns regarding the paper, e.g., limited technical novelty, unclear motivation, unconvincing experimental evaluations, inadequate literature reviews, etc. Considering the reviewers' concerns, we regret that the paper cannot be recommended for acceptance at this time. The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.