Title: Segment Everything Everywhere All at Once
Venue: NeurIPS 2023
Accepted: yes
Keywords: Generaic segmentation

interactive segmentation

referring segmentation

multi-modality prompting.
Abstract: In this work, we present SEEM, a promotable and interactive model for segmenting everything everywhere all at once in an image. In SEEM, we propose a novel and versatile decoding mechanism that enables diverse prompting for all types of segmentation tasks, aiming at a universal interface that behaves like large language models (LLMs). More specifically, SEEM is designed with four desiderata:
i) Versatility. We introduce a new visual prompt to unify different spatial queries including points, boxes, scribbles, and masks, which can further generalize to a different referring image; ii) Compositionality. We learn a joint visual-semantic space between text and visual prompts, which facilitates the dynamic composition of two prompt types required for various segmentation tasks, as shown in Fig. 1;
iii) Interactivity. We further incorporate learnable memory prompts into the decoder to retain segmentation history through mask-guided cross-attention from the decoder to image features; iv) Semantic awareness. We use a text encoder to encode text queries and mask labels into the same semantic space for open-vocabulary segmentation. We conduct a comprehensive empirical study to validate the effectiveness of SEEM across diverse segmentation tasks. The results demonstrate that SEEM exhibits robust generalizing to unseen user intents as it learns to compose prompts of different types in a unified representation space. Our approach achieves competitive performance on interactive segmentation, generic segmentation, referring segmentation, and video object segmentation on 9 datasets with minimum 1/100 supervision in a single set of weights.

Segment Everything Everywhere All at Once

Abstract

In this work, we present SEEM , a promptable and interactive model for segmenting everything everywhere all at once in an image, as shown in Fig. 1. In SEEM , we propose a novel decoding mechanism that enables diverse prompting for all types of segmentation tasks, aiming at a universal segmentation interface that behaves like large language models (LLMs). More specifically, SEEM is designed with four desiderata: i ) Versatility. We introduce a new visual prompt to unify different spatial queries including points, boxes, scribbles and masks, which can further generalize to a different referring image; ii ) Compositionality. We learn a joint visual-semantic space between text and visual prompts, which facilitates the dynamic composition of two prompt types required for various segmentation tasks; iii ) Interactivity. We further incorporate learnable memory prompts into the decoder to retain segmentation history through mask-guided cross-attention from decoder to image features; and iv ) Semantic-awareness. We use a text encoder to encode text queries and mask labels into the same semantic space for openvocabulary segmentation. We conduct a comprehensive empirical study to validate the effectiveness of SEEM across diverse segmentation tasks. Notably, our single SEEM model achieves competitive performance across interactive segmentation, generic segmentation, referring segmentation, and video object segmentation on 9 datasets with minimum 1/100 supervision. Furthermore, SEEM showcases a remarkable capacity for generalization to novel prompts or their combinations, rendering it a readily universal image segmentation interface.

1 Introduction

Image segmentation is arguably the most important yet challenging problem in computer vision. In the past, we have witnessed significant progress in a wide range of segmentation tasks including

instance, semantic and panoptic segmentation [1, 2, 3, 4, 5, 6, 7]. Most recently, we are observing a clear trend toward more flexible segmentation models in different aspects: 1) From closed-set to open-vocabulary segmentation. Many recent works proposed to either leverage contrastive learning methods or pretrained multi-modal foundation models ( e.g. , CLIP [8]) to make the segmentation models more transferable to unseen concepts [9, 10, 11, 12]; 2) From generic to referring segmentation. In addition to generic segmentation that segments an image thoroughly given a predetermined set of concepts, language-based referring segmentation provides a user-friendly way of segmenting a specific region referred by an arbitrary text phrase [13, 14, 15, 16, 17]; and 3) From one-shot to interactive segmentation. In practice, segmentation models do not necessarily produce satisfactory masks in one round. As such, people are also studying how to progressively refine the segmentation results through intimate interactions between humans and models [18, 19, 20, 21].

Despite the aforementioned efforts taken to design more powerful and feasible segmentation models, we are still lacking a universal segmentation interface that is capable of accommodating various types of human prompts and tackling different segmentation tasks as studied in individual works. In contrast, Large Language Models (LLMs) have already emerged as such a universal interaction interface for language tasks, from early models like GPT-3 [22] and T5 [23], to conversational agent [24] augmented by advanced prompting [25, 26, 27] and chain-of-thought [28, 29, 30]. In this work, we strive for a universal interface for segmenting everything everywhere all at once in an image. On this interface, we are targeted at unifying all segmentation tasks with a single model in a promptable manner. To achieve this goal, we propose a new prompting scheme in mask decoder that has four important properties: versatility , compositionality , interactivity , and semantic-awareness . Specifically, we propose to encode points, masks, text, boxes, and even a referred region from another image into prompts in the same joint visual-semantic space. As such, our model can deal with any combination of the input prompts, leading to strong compositionality. To enable interactivity, we further introduce memory prompts for condensing the previous segmentation information followed by communication with other prompts. As for semantic awareness, our model can provide an open-set semantic label to any output segmentation.

With the proposed prompting scheme, we build a segment-everything-everywhere model called SEEM comprised of a simple Transformer encoder-decoder architecture [31, 6] with an extra text encoder [11, 32]. In SEEM , the decoding process emulates a generative LLM but with a multimodalityin-multimodality-out interface. An image encoder and text encoder are used as the prompt encoder to encode all types of queries, which are fed into the decoder. Concretely, we encode all spatial queries, namely, points, boxes, scribbles and masks into visual prompts by pooling their corresponding visual features from the image encoder, and use the text encoder to convert text queries into text prompts . By training on diverse segmentation tasks, our model learns to deal with various prompts, align the visual and text prompts, and promote their synergy via cross-attention between them. As a result, our single model after pretraining attains competitive performance across all segmentation tasks. Since the prompts of all 5 different types are mapped to the joint visual-semantic space , we can feasibly combine prompts to resolve the ambiguity to obtain better segmentation results and enable zero-shot adaptation to unseen user prompts. Furthermore, our model can immediately generalize to the case of using an exemplar image segment as the prompt and video object segmentation in a zero-shot fashion. In addition to its strong generalization capability, SEEM is also more efficient for interactive segmentation compared with the counterparts like SimpleClick [32]. Since we take the prompts as input to the decoder, when doing multi-round interactions with humans, our model only needs to run the feature extractor once at the beginning and lightweight decoding each per round. To the end, we build a segmentation interface with a single pre-trained model that can segment every object with semantics (everything), cover every pixel in the image (everywhere), and support all possible compositions of prompts (all at once). In summary, our contributions are threefold:

We design a new prompting scheme that can encode various user intents into prompts in a joint visual-semantic space , enabling strong flexibility for various segmentation tasks and generalization capability to unseen prompts or their combinations.

We build SEEM , a universal and interactive segmentation interface that integrates the newly designed prompting mechanism into a lightweight decoder for all segmentation tasks, leading to a model possessing properties of versatility, compositionality, interactivity, and semantic awareness.

We conduct extensive experiments and visualizations to show that our model has strong performance on many segmentation tasks including open-vocabulary generic segmentation, interactive segmentation, referring segmentation, and segmentation tasks with combined prompts.

Machine

Human

SEEM

Decoder

Loop

Loop

Learnable Queries

Image Features

Text Prompt

MemoryPrompt

Visual Prompt

Self Attention

SEEM-Decoder

JointImage-TextRepresentationSpace

TextEncoder

ImageEncoder

VisualSampler

Mask Pool

Memory

Prompt

Prompt

Text

Prompt

Visual

Interpolate

Class/Sentence

Image Features

(a)ModelArchitecture

(b)Human-Model Interaction

2 Related Work

Interactive segmentation. Interactive segmentation is the task of segmenting objects by interactively taking user inputs. It has been a longstanding problem and has achieved considerable progress [33, 34, 35, 20, 21, 36]. Generally, the interaction types can take various forms, such as clicks, boxes, polygons, and scribbles, among which click-based interaction models are the most prevalent. Concurrent to our work, SAM [36] proposed a promptable segmentation model trained on 11 million images and 1.1 billion masks. It takes user interactions as prompts for general segmentation. Though SAM demonstrates strong zero-shot performance, it produces segmentations without semantic meaning. In addition, its prompt types are limited to points, boxes, and text, whereas our model can also take in a referred region from another image as a prompt.

Generic segmentation. Segmentation of visual concepts has been a persistent challenge in the field of computer vision, as evidenced by its extensive literature [37, 38, 39, 40]. Generic segmentation techniques encompass several subtasks, including instance segmentation, semantic segmentation, and panoptic segmentation [4, 2, 3], each focusing on a different semantic level. For example, semantic segmentation aims to identify and label each pixel within an image based on its corresponding semantic class [41, 6, 42]. On the other hand, instance segmentation involves grouping pixels that belong to the same semantic class into separate object instances [4, 43, 7]. Recently, the Detection Transformer (DETR)[31], a model based on the Transformer [44] architecture, has made significant advances in segmentation [45, 6, 7, 46, 47] tasks. However, these approaches cannot recognize objects absent in the training set, which constrains the model to a limited vocabulary size.

Unified vision models. Unified vision models [11, 48, 49, 36, 50] have recently drawn a lot of attention because of their advantage in generalizing to various tasks and flexibility. These models can deal with multiple vision tasks or data distributions. Among them, some [11, 48, 49] train multiple tasks together with only one model and thus can deal with all training tasks without finetuning on each target task. On the other hand, SAM [36] and SegGPT [50] propose training strategies that enable their models to handle new tasks and data distributions in a zero-shot manner. The second approach is more favorable since there is no need to resolve conflicts among tasks during training.

3 Method

3.1 Model Design

SEEM employs a generic encoder-decoder architecture but also employs a sophisticated interaction scheme between queries and prompts, as shown in Fig. 2 (a). Given an input image I ∈ R H × W × 3 , an image encoder is first used to extract image features Z . Then, SEEM -Decoder predicts the masks M and semantic concepts C based on the query outputs O m h (mask embeddings) and O c h (class embeddings), which interact with text, visual, and memory prompts ⟨ P t , P v , P m ⟩ :

GenericSeg

ReferringSeg

InteractiveSeg

No

Interaction

LearnableQueries

ObjectQueries

TextQueries

VisualQueries

True

Duplication

Interaction

Attention

Tentative

Interaction

TextPrompt

VisualPrompt

MemoryPrompt

Tentative

Attention

(a)Queries and PromptInteraction

(b)Self-AttentionMask

where Q h is the learnable queries, and P t , P v , P m represent the text prompts, visual prompts, and memory prompts, respectively. During training, Q h is duplicated for generic, referring, and interactive segmentation, as shown in Fig. 3. The corresponding prompts interact with their queries through self-attention. The learnable queries can freely interact with all prompts at inference time, thereby enabling zero-shot composition. Our design is inspired by the successful practice in XDecoder [11]. However, we highlight the differences in Eq. (1), marked in red, which allow for a universal model for image segmentation with the following properties:

Versatile . In SEEM , we introduce visual prompts P v to handle all non-textual inputs, such as points, boxes, scribbles, and a referred region from another image. These non-textual queries are beneficial to disambiguate the user's intent when textual prompts alone fail to identify the correct segment. For interactive segmentation, previous works either convert spatial queries to masks and feed them into the image backbone [20] or use different prompt encoders for each input type (points, boxes) [36]. The first approach can be too heavy in applications because each interaction requires the image to go through the feature extractor. The second approach is hard to generalize to unseen prompts. To address these limitations, we propose a visual sampler (Fig. 2 (a)) to convert all kinds of non-textual queries to visual prompts that lie in the same visual embedding space:

where ˆ Z is the feature maps extracted from either the target image ( i.e. , ˆ Z = Z ) or a referred image, and s ∈ { points , box , scribbles , polygons } are the sampling locations specified by the user. We first pool the corresponding region from the image feature through point sampling [6]. For all visual prompts, we interpolate at most 512 point feature vectors uniformly from the region specified by the prompt. A notable merit of our proposed method is that the visual prompts are naturally well-aligned with the textual prompts, as our model continuously learns a common visual-semantic space through panoptic and referring segmentation.

Compositional . In practice, a user may cast their intent using different or combined prompt types. Hence, a compositional approach to prompting is essential for real-world applications. However, we confront two issues during model training. First, the training data usually only covers a single type of interaction (e.g., none, textual, visual). Second, although we use visual prompts to unify all non-textual prompts and align them with textual prompts, their embedding spaces remain inherently different. To mitigate this problem, we propose to match prompts of different types with different outputs. Considering that visual prompts P v come from image features while textual prompts P t come from the text encoder, we select matched output indices for visual and textual prompts by matching them with the mask embeddings O m h or class embeddings O c h , respectively:

where IoU mask is the IoU between ground-truth and predicted masks. The proposed separate matching method outperforms approaches that only match with either O m h or O c h for all prompts.

After training, our model becomes familiar with all prompt types and supports a variety of compositions, such as no prompts, one prompt type, or both visual and textual prompts using the same model and weights. In particular, the visual and textual prompts can be simply concatenated and fed to SEEM-Decoder, even though it was never trained in this way.

Interactive . Interactive segmentation usually cannot be completed in one shot and requires multiple interaction rounds for refinement, similar to conversational agents like ChatGPT. In SEEM , we

Algorithm 1: Pseudo code for SEEM.

propose a new type of prompt called memory prompts P m and use them to convey the knowledge of the masks from the previous iteration to the current one. Unlike previous works that use a network to encode the previous mask [20, 36], we introduce no extra module but simply a few memory prompts. These memory prompts encode the history information by using a mask-guided cross-attention layer [6]:

where M p is the previous mask, and Z is the image feature map. In this way, cross-attention only takes effect inside the regions specified by the previous mask. The updated memory prompts P l m then interact with the other prompts via self-attention to convey the historical information for the current round.

Semantic-aware . Different from previous class-agnostic interactive segmentation works such as Simple Click [20] and the concurrent work SAM [36], our model produces semantic labels to masks for all kinds of prompt combinations in a zero-shot manner, since our visual prompt features are aligned with textual features in a joint visual-semantic space . As shown in Fig. 3, semantic labels are directly computed using O c h (output of visual queries) and the text embedding. Although we do not train with any semantic labels for interactive segmentation , the calculated logits are well-aligned, benefiting from the joint visual-semantic space .

3.2 Model Pipeline and Loss Functions

We summarize the training and evaluation pipeline of the proposed method with Pytorch-style pseudocode in Algorithm 1. SEEM is trained with a linear combination of losses for panoptic segmentation, referring segmentation, and interactive segmentation:

Where α = 2 , β = γ = 5 , a = 0 . 2 , b = c = 2 , CE, BCE, and DICE denotes cross-entropy, binary cross entropy and dice loss, respectively.

4 Experiments

Datasets and Settings . SEEM is trained on three tasks: panoptic segmentation, referring segmentation, and interactive segmentation. Panoptic and interactive segmentation are trained on COCO2017 [51] with panoptic segmentation annotations. Following [11], we exclude the validation set of Ref-COCOg [52], resulting in 107K segmentation images in total. For referring segmentation, we use a combination of Ref-COCO, Ref-COCOg, and Ref-COCO+ for COCO image annotations. We evaluate generic segmentation (instance/panoptic/semantic), referring segmentation, and interactive segmentation.

Implementation Details and Evaluation Metrics . Our model framework follows X-Decoder [11] except the decoder. That is, we have a vision backbone, a language backbone, an encoder, and

SEEM -Decoder. For the vision backbone, we use FocalT [58], DaViT-d3 (B), and DaViT-d5 (L) [59]. For the language encoder, we adopt a UniCL or Florence text encoder [60, 61]. For all segmentation tasks, we use standard evaluation metrics: PQ (Panoptic Quality) for panoptic segmentation, AP (Average Precision) for instance segmentation, and mIoU (mean Intersection over Union) for semantic segmentation. For interactive segmentation, we follow previous works [20, 62] to simulate user clicks by comparing the predicted segmentation with the ground-truth one in an automatic way. After one click on the image to generate the predicted mask, the next click is placed at the center of the area with the largest segmentation error. We use the Number of Clicks (NoC) metric to evaluate interactive segmentation performance, which measures the number of clicks needed to achieve a certain Intersection over Union (IoU), i.e., 85% and 90%, denoted as NoC @ 85 and NoC @ 90, respectively. We also vary the number of maximum clicks indicated by K-NoC @ 90 (K=5, 10, 20), and evaluate the mean IoU on the single click denoted as 1-IoU to study the performance on different constraints. More qualitative evaluation with stroke, scribble, polygon, and box as prompts are illustrated in the supplementary material.

4.1 Main Results

Generic segmentation With one suite of parameters pre-trained on all the segmentation tasks, we are able to evaluate its performance on generic segmentation datasets. As shown in Table 1, SEEM maintains competitive panoptic, instance, and semantic segmentation performance against strong baselines. Compared with generalist models such as UViM [55], Pix2Seqv2 [56] and especially the recent model Painter [57] and SegGPT [50], our approach significantly outperforms those methods on generic segmentation with a margin around 10 points on panoptic segmentation metrics.

Referring segmentation As shown in Table 1, compared with other referring segmentation and generalist models, SEEM achieves competitive performance. Notably, by adding a visual compositional prompt, referring segmentation performance is improved with a large margin by 10.5 cIoU,

6.0 mIoU, and 9.3 AP50 points for the tiny model. And this gap is retained for the base and large model. Specifically, this number is computed by class embeddings O c h (Output-Q-Textual). The margin is even larger when computed with mask embeddings O m h (Output-Q-Visual) as shown in Table 5. Further, we benchmark the vanilla composition (Ensemble) that directly combines visual and text mask output probabilities as shown in Table 5 row 2.

Interactive segmentation As shown in Table 1, our approach achieves comparable performance with the specialized models, e.g. RITM, SimpleClick, and better performance than SAM [36] (B) which is trained with × 100 more segmentation data than ours. Notably, unlike existing interactive models, SEEM is the first interface that supports not only classical segmentation tasks but also a wide range of user input types, including text, points, scribbles, boxes, and images, providing strong compositional capabilities as shown in Table 2,5.

User input type of interactive segmentation In Table 2, we compare 1-IoU of SEEM with other strong baselines SimpleClick and SAM with 5 common types of prompts on three datasets. 1-IoU indicates the mean IoU of all images with a single click. The prompt types include point, stroke, scribble, and box. The results show that our SEEM achieves the best performance in the extremely limited number of clicks over all three datasets.

Video object segmentation Without any modification, our model is able to do (interactive) video object segmentation in a zero-shot manner through the visual prompt (by replacing the current image visuals prompt with the visual prompts from another image). As shown in Table 3, without any observation of DAVIS/VOS dataset [71, 72], our approach is able to achieve close performance in a zero-shot manner with a fully supervised method on DAVIS17 dataset [72]. Meanwhile, our model is able to do interactive video object segmentation on DA VIS16-Interactive [72] and achieves comparable performance with the supervised baselines with one single click of the first frame.

bpat

正

butterfly

building

smallergiri

blackdog

largesttransformer

thedog

4.2 Ablation Study

We conduct an ablation study on all the training segmentation tasks and zero-shot video object segmentation, dissecting each component of our model. The results are presented in Table 4.

LVIS mask annotation will improve interactive segmentation results. We replace the COCO mask with an overlap IoU larger than 0.7 with LVIS mask during training. This will improve the performance on interactive segmentation with 0.3 and 0.2 point gain on NoC0.9 and NoC0.85.

Training from scratch only hurts referring segmentation performance. We compare the SEEM model trained with X-Decoder pre-trained checkpoint or the checkpoint initialized with UniCL or Florence vision and language backbone (+Scratch). It indicates that training from scratch will slightly improve the performance on interactive segmentation but hurt the referring segmentation performance.

Increase interactive training iterations does help. As shown in Table 4, increasing the training iteration (the first N-1 iteration is without gradient) from 1 to 5 will gradually improve the interactive segmentation performance from 5.41 to 4.59 on NoC0.9. As the computation cost increases with more clicks, we use iteration 3 for the main paper results.

4.3 Qualitative Results

We further qualitatively evaluate SEEM . Based on the proposed prompting scheme and decoder design, with the same suite of parameters, SEEM supports a wide range of visual input types.

Visual prompt interactive segmentation . In Fig. 4, we show the visualization of using SEEM to segment objects in an interactive way. The user can segment objects of interest by simply clicking or drawing a scribble. Taking these prompts, SEEM can simultaneously produce both masks and semantic labels for the objects. Note that our model is open-vocabulary, which empowers it to label unseen categories when given the candidate vocabulary (i.e., cheetah and butterfly in Fig. 4). When no vocabulary is given, SEEM can segment in a class-agnostic manner.

Text referring segmentation . We show the text referring to segmentation visualization results in Fig. 5. The results demonstrate that our model is semantic-aware of open-vocabulary concepts and

elephant

eeph

elephant

ellephant

ReferringImage

river

river

river

river

Referringlmage

Referring(Frame.

Frame5

-rame

Frame90

Referring(Frame 0)

Frame5

Frame10

Frame20

Frame50

Referring (Frame 0)

Frame2

Frame 4

Frame6

Frame16

attributes to understand language. In addition, SEEM is able to generalize to unseen scenarios like cartoons, movies, and games.

Visual referring segmentation . In Fig.6, we show SEEM 's segmentation results when prompted with referring regions from another image. By simply drawing a click or scribble on one referring image, SEEM can take it as input and segment objects with similar semantics on other images. Notably, this referring segmentation has a powerful generalization capability to images of other domains. For example, by referring to the elephant in the forest, another object of the same category can be segmented well under drastically different scenes like cartoons, plush toys, and grassland.

Video object segmentation . In Fig. 7, we further show SEEM 's referring segmentation ability on the video object segmentation task in a zero-shot manner. By referring to the objects in the first frame with scribbles, SEEM can precisely segment the corresponding objects in the following frames, even when the following objects change in appearance by blurring or intensive deformations.

5 Conclusion

We presented SEEM , which can segment everything (all semantics) everywhere (all pixels) all at once (all possible prompt compositions). Apart from performing generic open-vocabulary segmentation, SEEM can interactively take different types of visual prompts from the user, including click, box, polygon, scribble, text, and referring region from another image. These visual prompts are mapped into a joint visual-semantic space with a prompt encoder, which makes our model versatile to various prompts and can flexibly compose different prompts. Extensive experiments indicate that our model yields competitive performance on several open-vocabulary and interactive segmentation benchmarks. Further studies revealed the robust generalization ability of our model in accurately segmenting images based on diverse user intents. We hope our work will serve as a stepping stone toward a universal and interactive interface for image segmentation and beyond.

Acknowledgements. We would like to express our gratitude to Lei Zhang for his generous support. And express the appreciated for the valuable suggestions from Zhenyuan Yang, and discussion with Xiaoyu Xiang. In addition, this work was supported in part by NSF CAREER IIS2150012, NASA 80NSSC21K0295, the Institute of Information and communications Technology Planning and Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2022-0-00871, Development of AI Autonomy and Knowledge Enhancement for AI Agent Collaboration).

References

Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE Transactions on pattern analysis and machine intelligence , 22(8):888-905, 2000.

Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence , 40(4):834-848, 2017.

Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9404-9413, 2019.

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision , pages 2961-2969, 2017.

Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Max-deeplab: End-to-end panoptic segmentation with mask transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5463-5474, 2021.

Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1290-1299, 2022.

Feng Li, Hao Zhang, Shilong Liu, Lei Zhang, Lionel M Ni, Heung-Yeung Shum, et al. Mask dino: Towards a unified transformer-based framework for object detection and segmentation. arXiv preprint arXiv:2206.02777 , 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748-8763. PMLR, 2021.

Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Open-vocabulary image segmentation. arXiv preprint arXiv:2112.12143 , 2021.

Zheng Ding, Jieke Wang, and Zhuowen Tu. Open-vocabulary panoptic segmentation with maskclip. arXiv preprint arXiv:2208.08984 , 2022.

Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. Generalized decoding for pixel, image, and language. arXiv preprint arXiv:2212.11270 , 2022.

Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open-vocabulary semantic segmentation. arXiv preprint arXiv:2302.12242 , 2023.

Chenxi Liu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, and Alan Yuille. Recurrent multimodal interaction for referring image segmentation. In Proceedings of the IEEE International Conference on Computer Vision , pages 1271-1280, 2017.

Linwei Ye, Mrigank Rochan, Zhi Liu, and Yang Wang. Cross-modal self-attention network for referring image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10502-10511, 2019.

Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. Language as queries for referring video object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4974-4984, 2022.

Shijia Huang, Feng Li, Hao Zhang, Shilong Liu, Lei Zhang, and Liwei Wang. A unified mutual supervision framework for referring expression segmentation and generation. arXiv preprint arXiv:2211.07919 , 2022.

Jiang Liu, Hui Ding, Zhaowei Cai, Yuting Zhang, Ravi Kumar Satzoda, Vijay Mahadevan, and R Manmatha. Polyformer: Referring image segmentation as sequential polygon generation. 2023.

Konstantin Sofiiuk, Ilia A. Petrov, and Anton Konushin. Reviving iterative training with mask guidance for interactive segmentation, 2021.

Ho Kei Cheng, Yu-Wing Tai, and Chi-Keung Tang. Modular interactive video object segmentation: Interaction-to-mask, propagation and difference-aware fusion. In CVPR , 2021.

Qin Liu, Zhenlin Xu, Gedas Bertasius, and Marc Niethammer. Simpleclick: Interactive image segmentation with simple vision transformers. arXiv preprint arXiv:2210.11006 , 2022.

Xi Chen, Zhiyan Zhao, Yilei Zhang, Manni Duan, Donglian Qi, and Hengshuang Zhao. Focalclick: towards practical interactive image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1300-1309, 2022.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research , 21(1):5485-5551, 2020.

OpenAI. Gpt-4 technical report, 2023.

Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980 , 2020.

Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning , pages 12697-12706. PMLR, 2021.

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 , 2021.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 , 2022.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916 , 2022.

Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761 , 2023.

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision , pages 213-229. Springer, 2020.

Hao Zhang, Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianfeng Gao, Jianwei Yang, and Lei Zhang. A simple framework for open-vocabulary segmentation and detection. arXiv preprint arXiv:2303.08131 , 2023.

Yin Li, Jian Sun, Chi-Keung Tang, and Heung-Yeung Shum. Lazy snapping. ACM Transactions on Graphics (ToG) , 23(3):303-308, 2004.

Leo Grady. Random walks for image segmentation. IEEE transactions on pattern analysis and machine intelligence , 28(11):1768-1783, 2006.

Ning Xu, Brian Price, Scott Cohen, Jimei Yang, and Thomas S Huang. Deep interactive object selection. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 373-381, 2016.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything, 2023.

King-Sun Fu and JK Mui. A survey on image segmentation. Pattern recognition , 13(1):3-16, 1981.

Pedro F Felzenszwalb, Ross B Girshick, David McAllester, and Deva Ramanan. Object detection with discriminatively trained part-based models. IEEE transactions on pattern analysis and machine intelligence , 32(9):1627-1645, 2009.

Zhengxia Zou, Zhenwei Shi, Yuhong Guo, and Jieping Ye. Object detection in 20 years: A survey. arXiv preprint arXiv:1905.05055 , 2019.

Shervin Minaee, Yuri Y Boykov, Fatih Porikli, Antonio J Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos. Image segmentation using deep learning: A survey. IEEE transactions on pattern analysis and machine intelligence , 2021.

Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 , 2017.

Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3431-3440, 2015.

Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee. Yolact: Real-time instance segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9157-9166, 2019.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Zhiqi Li, Wenhai Wang, Enze Xie, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, Ping Luo, and Tong Lu. Panoptic segformer: Delving deeper into panoptic segmentation with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1280-1289, 2022.

Jitesh Jain, Jiachen Li, MangTik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. Oneformer: One transformer to rule universal image segmentation. arXiv preprint arXiv:2211.06220 , 2022.

Hao Zhang, Feng Li, Huaizhe Xu, Shijia Huang, Shilong Liu, Lionel M Ni, and Lei Zhang. Mpformer: Mask-piloted transformer for image segmentation. arXiv preprint arXiv:2303.07336 , 2023.

Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, and Huchuan Lu. Universal instance perception as object discovery and retrieval. arXiv preprint arXiv:2303.06674 , 2023.

Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206.08916 , 2022.

Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Segmenting everything in context. arXiv preprint arXiv:2304.03284 , 2023.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV , 2014.

Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In European Conference on Computer Vision , pages 69-85. Springer, 2016.

Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip HS Torr. Lavt: Language-aware vision transformer for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18155-18165, 2022.

Qin Liu, Meng Zheng, Benjamin Planche, Srikrishna Karanam, Terrence Chen, Marc Niethammer, and Ziyan Wu. Pseudoclick: Interactive image segmentation with click imitation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VI , pages 728-745. Springer, 2022.

Alexander Kolesnikov, André Susano Pinto, Lucas Beyer, Xiaohua Zhai, Jeremiah Harmsen, and Neil Houlsby. UViM: A unified modeling approach for vision with learned guiding codes. arXiv preprint arXiv:2205.10337 , 2022.

Ting Chen, Saurabh Saxena, Lala Li, Tsung-Yi Lin, David J Fleet, and Geoffrey Hinton. A unified sequence interface for vision tasks. arXiv preprint arXiv:2206.07669 , 2022.

Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning, 2023.

Jianwei Yang, Chunyuan Li, and Jianfeng Gao. Focal modulation networks. arXiv preprint arXiv:2203.11926 , 2022.

Mingyu Ding, Bin Xiao, Noel Codella, Ping Luo, Jingdong Wang, and Lu Yuan. Davit: Dual attention vision transformers. arXiv preprint arXiv:2204.03645 , 2022.

Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Unified contrastive learning in image-text-label space. In CVPR , 2022.

Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang, Michael Zeng, Luowei Zhou, and Pengchuan Zhang. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432 , 2021.

Zheng Lin, Zheng-Peng Duan, Zhao Zhang, Chun-Le Guo, and Ming-Ming Cheng. Focuscut: Diving into a focus view in interactive segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2637-2646, 2022.

Huaijia Lin, Xiaojuan Qi, and Jiaya Jia. Agss-vos: Attention guided single-shot video object segmentation. In ICCV , 2019.

Joakim Johnander, Martin Danelljan, Emil Brissman, Fahad Shahbaz Khan, and Michael Felsberg. A generative appearance model for end-to-end video object segmentation, 2018.

Zhihui Lin, Tianyu Yang, Maomao Li, Ziyu Wang, Chun Yuan, Wenhao Jiang, and Wei Liu. Swem: Towards real-time video object segmentation with sequential weighted expectationmaximization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1362-1372, 2022.

Ho Kei Cheng and Alexander G. Schwing. XMem: Long-term video object segmentation with an atkinson-shiffrin memory model. In ECCV , 2022.

Qiang Wang, Li Zhang, Luca Bertinetto, Weiming Hu, and Philip HS Torr. Fast online object tracking and segmentation: A unifying approach. In Proceedings of the IEEE conference on computer vision and pattern recognition , 2019.

Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. Language as queries for referring video object segmentation. arXiv preprint arXiv:2201.00487 , 2022.

Jinyu Yang, Mingqi Gao, Zhe Li, Shang Gao, Fangjing Wang, and Feng Zheng. Track anything: Segment anything meets videos, 2023.

Renrui Zhang, Zhengkai Jiang, Ziyu Guo, Shilin Yan, Junting Pan, Hao Dong, Peng Gao, and Hongsheng Li. Personalize segment anything model with one shot, 2023.

Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, and Thomas Huang. Youtube-vos: A large-scale video object segmentation benchmark. arXiv preprint arXiv:1809.03327 , 2018.

Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:1704.00675 , 2017.

Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung. A benchmark dataset and evaluation methodology for video object segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 724-732, 2016.

Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Peer reviews:
comments: SUMMARY:
The paper proposes SEEM, a method that unifies several segmentation tasks. The network takes text and/or different kinds of visual prompts as inputs. The model then outputs the corresponding segmentation masks for the referred objects. In particular, SEEM trains a unified prompt embedding space. A set of learned queries then alternately attends to image features and the prompt embeddings. Finally, the output embeddings are matched to the prompts and decoded into masks.

STRENGTHS:
-	Unifying several segmentation tasks.
-	Impressive results on most tasks.
-	Similar or better than SAM in many cases, despite less training data.
-	Reasonably well written.

WEAKNESSES:
1.	Some parts of the method lack motivation and details. In particular, the attention mask used in the prompt attention is not well motivated. It is also not well explained that (and how) the output embeddings are matched to the prompts.

2.	The paper is heavily derived from X decoder. The magnitude of the contributions is not clear. The authors do not discuss the relation to X decoder. In fact it is hardly mentioned in the paper, yet the method is completely based on it. Seems like the main new addition here is the visual prompt encoder.

3.	The results for VOS are quite poor. This indicates that the method is not as robust to object appearance variations, which is crucial in many tasks.

4.	The authors should report training and inference time and memory requirements.

I am leaning positive for the paper. However, the authors need to address my concerns above.

QUESTIONS:
See weaknesses. 

Moreover:
Is the method robust to the case when the prompted text does not match any object in the image?

LIMITATIONS:
Limitations are not discussed in the main paper, but should be mentioned.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SEEM, an innovative and interactive model for segmentation. SEEM stands out with its versatility in handling various prompts, such as points, boxes, scribbles, masks, and texts. The model's design is elegantly simple yet highly effective. Additionally, SEEM incorporates memory prompts to preserve segmentation history, enhancing its interactivity and overall performance.\n\nSTRENGTHS:\n1. The paper is well-presented, showcasing clear and concise writing. The authors effectively communicate their ideas.\n\n2. The proposed idea is both innovative and highly effective. The designed model, despite its simplicity, showcases remarkable capabilities when dealing with a wide range of prompts. Its versatility and adaptability make it be applicable with more tasks.\n\n3. The experimental results are solid. Through a combination of qualitative and quantitative analyses, the authors demonstrate the effectiveness of their idea.\n\nWEAKNESSES:\n1. Do the authors observe whether training on all types of prompts performing better than on one type or a subset types of prompts? or which types of prompt performs best and which one still lacks of improvement.  It would be better to perform this ablation because it can give more insights to the society.\n\n2. since this is a very general framework supporting many different types of prompts, and it can be more easily extended with more training data.  I would also like to see its scalability with larger amount of supervised training data if there is sufficient computing resources.\n\nQUESTIONS:\nDo the authors plan to train on larger amount of dataset? (such as the one trained on SAM or other rich datasets with the semantic labels).  It would be very interesting to see how SEEM scale up with large amount of supervised or weakly supervised labels.\n\nLIMITATIONS:\nnot mentioned", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n[Task] In this work, authors introduc SEEM, a promptable and interactive model designed for comprehensive image segmentation. SEEM aims to segment all objects in an image simultaneously, addressing various segmentation tasks. The key contribution of SEEM is its novel decoding mechanism, which allows for diverse prompting and behavior similar to large language models (LLMs).\n\n[Method] SEEM is built with four key objectives: versatility, compositionality, interactivity, and semantic-awareness. The model incorporates a visual prompt that unifies different spatial queries such as points, boxes, scribbles, and masks, enabling generalization to different referring images. It also learns a joint visual-semantic space for dynamic prompt composition in different segmentation tasks. Additionally, SEEM incorporate learnable memory prompts in the decoder to retain segmentation history and utilizes mask-guided cross-attention from the decoder to image features. Moreover, a text encoder is employed to encode text queries and mask labels into the same semantic space, facilitating open-vocabulary segmentation.\n\n[Experiments] Empirical studies validate the effectiveness of SEEM across diverse segmentation tasks. The model achieves competitive performance in interactive segmentation, semantic segmentation, referring segmentation, and video object segmentation across nine datasets, requiring only minimal 1/100 supervision. SEEM demonstrates remarkable generalization capability to novel prompts or their combinations, positioning it as a versatile and universal image segmentation interface.\n\nSTRENGTHS:\n[highlight the trend towards more flexible segmentation models] In this work, the authors address the need for a universal segmentation interface capable of handling different types of human prompts and addressing various segmentation tasks. They highlight the trend towards more flexible segmentation models, including open-vocabulary segmentation, referring segmentation, and interactive segmentation. Inspired by the success of Large Language Models (LLMs) as universal interaction interfaces for language tasks, the authors propose SEEM, a promptable and interactive model for segmenting everything everywhere all at once in an image.\n\n[technical contribution] The proposed approach, SEEM, introduces a novel prompting scheme in the mask decoder with four key properties: versatility, compositionality, interactivity, and semantic-awareness. By encoding different types of prompts (points, masks, text, boxes, and referred regions) into a joint visual-semantic space, SEEM achieves strong compositionality and can handle any combination of input prompts. Memory prompts are introduced to retain previous segmentation information and enable interactivity. The model also provides open-set semantic labels for output segmentation.\n\n[tasks and results] The model is trained on diverse segmentation tasks to learn how to handle different prompts, align visual and text prompts, and promote their synergy through cross-attention. The single pre-trained SEEM model achieves competitive performance across all segmentation tasks, leveraging the joint visual-semantic space for prompt combination and zero-shot adaptation. In addition to its strong generalization capability, SEEM is efficient for interactive segmentation compared to other methods. Overall, SEEM offers a segmentation interface with a single pre-trained model that can handle all types of prompts, segment every object with semantics, and cover every pixel in the image.\n\nWEAKNESSES:\n[Model performance] The model performance of SEEM on certain benchmarks, including ADE20K for open-vocabulary panoptic segmentation and DAVIS for video instance segmentation, is often lower, and in some cases, significantly lower than the baselines X-Decoder, ODISE, and UNINEXT. It is worth noting that SEEM's framework follows a similar approach to X-Decoder. \nWhile the concept of a universal and interactive segmentation interface is intriguing, it appears that the model performance on open-vocabulary benchmarks is compromised in SEEM. It would be beneficial for the authors to provide an explanation for this discrepancy.\n\n[Unfair comparisons] In Table 2, there are some comparisons that seem unfair for two reasons. Firstly, the evaluation of SEEM on COCO, which provides instance-level annotations, aligns better with SEEM's training pipeline that includes instance-level annotations. However, this evaluation method puts SAM at a disadvantage since it is trained to respect hierarchical segmentation, including whole-part-subpart relationships. Secondly, SEEM is already trained on in-domain data from COCO's training set, while SAM is pretrained on out-of-domain data. This discrepancy in training data can impact the performance comparison between the two models. To ensure fair comparisons, it would be beneficial to evaluate both models using evaluation protocols that align with their respective training pipelines and take into account the domain of the training data.\n\nQUESTIONS:\n[Handling dataset imbalance] Since SEEM is trained on multiple datasets, there might be an imbalance in the number of samples per dataset. I'm curious to know if adjusting the frequency of training samples from each dataset has any impact on the model's performance. Can the authors shed some light on whether they have explored techniques such as data augmentation or sample weighting to address this issue? Additionally, it would be interesting to understand if adjusting the training sample frequency has any effect on mitigating the impact of dataset imbalance and improving the overall model performance.\n\nLIMITATIONS:\nYes, the authors adequately addressed the limitations", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a universal segmentation model, SEEM, for all segmentation tasks. A visual sampler module unifies different kinds of human inputs and they are encoded into a joint visual-semantic space together with image and text so that the SEEM model can learn semantic labels for masks. The proposed SEEM model achieves competitive performance across interactive segmentation, generic segmentation, referring segmentation, and video object segmentation. Besides, SEEM shows a strong ability to generalization to novel prompts or their combinations.\n\nSTRENGTHS:\n1. The proposed model has superior performance and has achieved competitive results across different types of segmentation tasks. \n2. The proposed model can accept a variety of visual prompts input types and has a strong generalization ability, which makes it possible to be applied to more scenarios.\n3. Different from the previous segmentation model, the proposed model can learn the semantic labels corresponding to the segmentation masks, thus enabling open vocabulary segmentation.\n\nWEAKNESSES:\n1. In this paper, when comparing the performance of the proposed model with the previous models, only the scales of training data are compared, without the comparison of parameters and calculation. This may cause some doubts about the comparison fairness.\n2. The analysis of some results is insufficient. In the ablation study section, the performance changes caused by the change of variables are not consistent in different segmentation tasks. Only the changing trend of the results is given without analyzing the potential reasons.\n\nQUESTIONS:\n1. Although SEEM can learn the semantic labels of masks, it is limited to the categories of COCO. Is there any way to expand the scope of semantic labels without introducing a large amount of extra data?\n2. Can you provide a comparison with other universal segmentation models in terms of the number of model parameters and calculations? If this comparison is not applicable, what are the reasons?\n\nLIMITATIONS:\nLimitations are not explicitly discussed in this paper. The scale of training data is limited, and it does not support partial segmentation as in SAM (Segmenting Anything Model).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-29"}