Segment Everything Everywhere All at Once
Authors: Xueyan Zou
Jianwei Yang
Hao Zhang
Feng Li
Linjie Li
Jianfeng Wang
Lijuan Wang
Jianfeng Gao
Yong Jae Lee
Venue: NeurIPS 2023 poster (2023)
Keywords: Generaic segmentation
interactive segmentation
referring segmentation
multi-modality prompting.
TL;DR: A universal and interactive model for all kinds of segmentation tasks with multi-modality prompts as inputs.

Abstract: In this work, we present SEEM, a promotable and interactive model for segmenting everything everywhere all at once in an image. In SEEM, we propose a novel and versatile decoding mechanism that enables diverse prompting for all types of segmentation tasks, aiming at a universal interface that behaves like large language models (LLMs). More specifically, SEEM is designed with four desiderata:
i) Versatility. We introduce a new visual prompt to unify different spatial queries including points, boxes, scribbles, and masks, which can further generalize to a different referring image; ii) Compositionality. We learn a joint visual-semantic space between text and visual prompts, which facilitates the dynamic composition of two prompt types required for various segmentation tasks, as shown in Fig. 1;
iii) Interactivity. We further incorporate learnable memory prompts into the decoder to retain segmentation history through mask-guided cross-attention from the decoder to image features; iv) Semantic awareness. We use a text encoder to encode text queries and mask labels into the same semantic space for open-vocabulary segmentation. We conduct a comprehensive empirical study to validate the effectiveness of SEEM across diverse segmentation tasks. The results demonstrate that SEEM exhibits robust generalizing to unseen user intents as it learns to compose prompts of different types in a unified representation space. Our approach achieves competitive performance on interactive segmentation, generic segmentation, referring segmentation, and video object segmentation on 9 datasets with minimum 1/100 supervision in a single set of weights.

Segment Everything Everywhere All at Once
Xueyan Zou Jianwei Yang"
Hao Zhang" Feng Li" Linjie Lit, Jianfeng Wang' Lijuan Wang', Jianfeng Gao "; Yong Jac Lee
University of Wisconsin-Madison Madison
Microsoft Research, Redmond HKUST Microsoft Cloud & Al
Equal Contribution
Equal Advisory Contribution
Project Lead
Tochnical Contribution
(xocyan, (ww.ganeee Bcc. Rcs.wisc.cdu
(jianwyar, jtgap, linjl1 /nicrceatt.com
Ibzhanges, fiiay #ccnncct. act .bk
Panoptic
Instance
Semantic
Point
Box
Scribble
Text/Audio
Cross Style
Text+Visual
SEEM
Person in blue.
No Prompt
Visual Prompts
Text Prompt
Ref Prompt
Composition
Figure 1: SEEM supports generic segmentation tasks including semantic, instance, and panoptic segmentation in an open-se set fashion when no prompt is provided. SEEM also enables the use of visual, textual, and referring region prompts in flexbile combinations, making it a promptable and interactive segmentation interface.
Abstract
In this work, present SEEM, promptable and interactive model for segmenting everything everywhere all at once an image, as shown in Fig. [ In SEEM, we propose novel decoding mechanism that enables diverse prompting for all types segmentation tasks, aiming at a universal segmentation interface that behaves like large language models (LLMs). More specifically, SEEM is designed with four desiderata: i) Versatility. We introduce a new visual prompt to unify different spatial queries including points, boxes, scribbles and masks, which further generalize to a different referring image; it) Compositionality. We learn joint iisual-semantic space between text and visual prompts, which facilitates the dynamic composition of two prompt types required for various segmentation tasks; (éz) Interactivity. We further incorporate learable memory prompts into the decoder to retain segmentation history through mask-guided cross-attention from decoder image features; and iv) Semantic-awareness. We use a text encoder to encode text queries and mask labels into the same semantic space for open- vocabulary segmentation. We conduct comprehensive empirical study validate the effectiveness of SEEM across diverse segmentation tasks. Notably, our single SEEM model achieves competitive performance across interactive segmentation, generic segmentation, referring segmentation, and video object segmentation on 9 datasets with minimum 1/100 supervision. Furthermore, SEEM showcases a remarkable capacity for generalization to novel prompts or their combinations, rendering it readily universal image segmentation interface.
Introduction
Image segmentation is arguably the most important yet challenging problem in computer vision. In the past, we have witnessed significant progress in wide range of segmentation tasks including
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

instance, semantic and panoptic segmentation 12 3 5
6 7 Most recently, we are observing a clear trend toward more flexible segmentation models in different aspects: 1) From closed-set to veeablocabulary segmentation. Many recent works proposed to either leverage contrastive learning methods or pretrained multi-modal foundation models (e.g.. CLIP S to make the segmentation
models more transferable to unseen concepts 10 2) From generic to referring segmentation.
In addition to generic segmentation that segments an image thoroughly given a predetermined set of concepts, language-based referring segmentation provides user- -rieedly way of segmenting
specific region referred by an arbitrary text phrase 13 14 15 16. 17] and 3) From one-shot to interactive segmentation. In practice, segmentation models do not necessarily produce satisfactory masks in one round. As such, people are also studying how to progressively refine the segmentation
results through intimate interactions between humans and models [8] 19 20 21].
Despite the aforementioned efforts taken to design more powerful and feasible segmentation models, we are still lacking universal segmentation interface that is capable of accommodating various types of human prompts and tackling different segmentation tasks as studied in individual works. In contrast, Large Language Models (LLMs) have already emerged such a universal interaction interface for language tasks, from early models like GPT-3 [22] and T5 [23], to conversational agent [24] augmented by advanced prompting 25 26 27 and chain-of-thought 128 29 30). In
this work, we strive for a universal interface for segmenting everything everywhere all at once in an image. On this interface, we are targeted at unifying all segmentation tasks with single model in promptable manner. To achieve this goal, we propose a new prompting scheme in mask decoder that has four important properties: versatility, compositionality, interactivity, and semantic-awarent Specifically, we propose to encode points, masks, text, boxes, and even a referred region from another image into prompts in the same joint visual-semantic space. As such, our model can deal with any combination of the input prompts, leading to strong compositionality. To enable interactivity, we further introduce memory prompts for condensing the previous segmentation information followed by communication with other prompts. As for semantic awareness, our model can provide an open-set semantic label to any output segmentation.
With the proposed prompting scheme, we build a weapeenteeerttthnn--veee SEEM comprised of a simple Transformer encoder-decoder architecture [1 6 with an extra text encoder 32]. In SEEM, the decoding process emulates a generative LLM but with a multimodality- in-multimodality-oni interface. An image encoder and text encoder are used the prompt encoder to encode all types of queries, which are fed into the decoder. Concretely, we encode all spatial queries, namely, points, boxes, scribbles and masks into visual prompts by pooling their corresponding visual features from the image encoder, and use the text encoder to convert text queries into text prompts. By training on diverse segmentation tasks, our model learns to deal with various prompts, align the visual and text prompts, and promote their synergy via cross-attention between them. As a result, our single model after pretraining attains competitive performance across all segmentation tasks. Since the prompts of all 5 different types are mapped to the joint -ssaal-emantic space, we can feasibly combine prompts to resolve the ambiguity to obtain better segmentation results and enable zero-shot adaptation to unseen user prompts. Furthermore, our model can immediately generalize to the case zero-shot
model called
of using an exemplar image segment as the prompt and video object segmentation fashion. In addition to its strong generalization capability, SEEM is also more efficient for interactive segmentation compared with the counterparts like SimpleClick [32]. Since we take the prompts as input to the decoder, when doing multi-round interactions with humans, our model only needs to run the feature extractor once at the beginning and lightweight decoding each per round. To the end, we build a segmentation interface with single pre-trained model that can segment every object with semantics (everything), cover every pixel in the image (everywhere), and support all possible compositions of prompts (all at once). In summary, contributions are threefold:
We design a new prompting scheme that can encode various user intents into prompts in a joint visual-semantic.. spuce, enabling strong flexibility for various segmentation tasks and generalization capability to unseen prompts or their combinations.
We build SEEM, a universal and interactive segmentation interface that integrates the newly designed prompting mechanism into a lightweight decoder for all segmentation tasks, leading to a model possessing properties of versatility, compositionality, interactivity, and semantic awareness.
We conduct extensive experiments and visualizations to show that our model has strong perfor- mance on many segmentation tasks including open-vocabulary generic segmentation, interactive segmentation, referring segmentation, and segmentation tasks with combined prompts.
2

---

Class Embeddings
Mask Emboddingo
Gross Acaettz
Madia
SEEM
Decoder
Leop
Hrage Norture
Phompt
Fronet
Vaulthrret
SEEM Decodor
Joint Image-Ter Representation Space
Text Incoder
Image Encoder
Visual Sampler
Menser
Frompt
Text
Prompt
Vical
Classr8entence
Image Fostuses
(a) Mochhtttccctttuur
(b) Human-Model Interaction
Figure 2: Overview of SEEM- Decoder. (a) SEEM encodes image, text, and human inputs into joint visual-semantic space as queries, features, and prompts, and then decodes queries to class and mask embeddings. (b) With the benefit of SEEM decoder, the machine loop enables memorizing history mask information, and the human loop provides new corrections to the next round.
2 Related Work
Interactive segmentation. Interactive segmentation is the task of segmenting objects by interactively taking user inputs. It has been longstanding problem and has achieved considerable progress [33 34 35 20 36] Generally, the interaction types can take various forms, such as clicks, boxes, polygons, and scribbles, among which click- interaction models are the most prevalent. Concurrent to our work, SAM (36] proposed a promptable segmentation model trained on 11 million images and 1.1 billion masks. It takes user interactions prompts for general segmentation. Though SAM demonstrates strong zero-shot performance, it produces segmentations without semantic meaning. In addition, its prompt types are limited to points, boxes, and text, whereas our model can also take in referred region from another image as a prompt.
Generic segmentation. Segmentation of visual concepts has been a persistent challenge in the field of computer vision, as evidenced by its extensive literature [37 techniques encompass several subtasks, including instance segmentation, semantic segmentation, and panoptic segmentation 4 32, each focusing on different semantic level. For example, semantic segmentation aims to identify and label each pixel within image based its corresponding semantic class [41 6 [4). On the other hand, instance segmentation involves grouping pixels that
Generic segmentation
belong the same semantic class into separate object instances 4 43 7 Recently, the Detection Transformer (DETR) 31, a model based on the Transformer 44 architecture, has made significant advances in segmentation 45
46 tasks. However, these approaches cannot recognize objects absent in the training set, which constrains the model to a limited vocabulary size.
have recently drawn lot of attention because their advantage in generalizing to various tasks and flexibility. These models can deal with multiple vision tasks or data distributions. Among them, some [11 48, 49 train multiple tasks together with only one model and thus can deal with all training tasks without finctuning on each target task. On the other hand, SAM (36) and SegGPT (50) propose training strategies that enable their models to handle new tasks and data distributions in a zero-shot manner. The second approach is more favorable since there is need to resolve conflicts among tasks during training.
Unified vision models. Unified vision models
48
49
36
50
3 Method
3.1
Model Design
SEEM employs a generic encoder-decoder architecture but also employs sophisticated interaction scheme between queries and prompts, as shown in Fig. (a). Given an input image I I R x x an image encoder is first used extract image features Z. Then, SEEM-Decoder predicts the masks M and semantic concepts C based on the query outputs O (mask embeddings) and Og (class embeddings), which interact with text, visual, and memory prompts (P,, P., P P.)):
(Og O Decoder(Qu.. (P Po, P...P...2 Im))
M Mask (O)
(2)
C = Concop:Classifer((O]]]
(3)
3

---

Generic Seg
Referring Seg
Interactive Seg
Interaction
Leamable Querles
Duplication
Object Queries
Text Querries
Queries
True
Interaction
Attention
Tentative
Interaction
Text Prompt
Visual Prompt
Memory Prompt
Tentative
Attention
(a) Queries and Prompt Interaction
(b) Self-Attention Mask
Figure 3: Queries and prompt interaction during training and evaluation. (a) Learnable queries duplicated as object, grounding, and visual queries with the same set of weights for each task. (b) Attention mask between any two kinds of tokens (denoted qpm in Algorithm. [ Tentative means the interaction is not trained but able to do inference without any modification.
where Q₂ is the learnable queries, and Pt, Pu, Pm represent the text prompts, visual prompts, and memory prompts, respectively. During training, Qn is duplicated for generic, referring, and interactive segmentation, as shown in Fig. [ The corresponding prompts interact with their queries through self-attention. The learnable queries can freely interact with all prompts at inference time, thereby enabling zero-shot composition. Our design is inspired by the successful practice in X- Decoder [Л] However, we highlight the differences in Eq. Q marked in red, which allow for universal model for image segmentation with the following properties:
Versatile. In SEEM, we introduce visual prompts P. to handle all non-textual inputs, such as points, boxes, scribbles, and a referred region from another image. These non-textual queries are beneficial to disambiguate the user's s intent when textual prompts alone fail to identify the correct segment. For interactive segmentation, previous works either convert spatial queries to masks and feed them into the image backbone [20] or use different prompt encoders for each input type (points, boxes) [36]. The first approach can be too heavy in applications because each interaction requires the image to go through the feature extractor. The second approach is hard to generalize to unseen prompts. To
address these limitations, we propose visual sampler (Fig. 2 (a)) convert all kinds of non-textual queries to visual prompts that lie in the same visual embedding space:
P VisualSampler(s. Z)
where Ż is the feature maps extracted from either the target image (i.e., 2 Z) or a referred image, and s [points, box, scribbles, polygons} are the sampling locations specified by the user. We first pool the corresponding region from the image feature through point sampling (6]. For all visual prompts, we interpolate at most 512 point feature vectors uniformly from the region specified by the prompt. notable merit our proposed method that the visual prompts are naturally well-aligned with the textual prompts, panoptic and referring segmentation.
model continuously learns common visual-semante space through
Compositional. In practice, a user may cast their intent using different or combined prompt types. Hence, compositional approach to prompting essential for real-world applications. However, we confront two issues during model training. First, the training data usually only covers a single type of interaction (e.g., none, textual, visual). Second, although we use visual prompts unify all non-textual prompts and align them with textual prompts, their embedding spaces remain inherently different. To mitigate this problem, we propose to match prompts of different types with different outputs. Considering that visual prompts P, come from image features while textual prompts Pa come from the text encoder, we select matched output indices for visual and textual prompts by matching them with the mask embeddings Og or class embeddings O% respectively:
ID, +- Match O P. + IoU mmask)
(5)
ID. Match(O5, P, + IoU mask)
(6)
where IoU mask is the IoU between ground truth and predicted masks. The proposed separate matching method outperforms approaches that only match with either O Of for all prompts.
After training, our model becomes familiar with all prompt types and supports a variety of composi- tions, such as prompts, prompt type, or both visual and textual prompts using the same model and weights. In particular, the visual and textual prompts can be simply concatenated and fed to SEEM-Decoder, even though was never trained in this way.
Interactive. Interactive segmentation usually cannot be completed in one shot and requires multiple interaction rounds for refinement, similar to conversational agents like ChatGPT. In SEEM,

---

Algorithm 1: Pscudo code for SEEM.
Inputas Inage (Lmg) [B, B, 7 Pos, Mask (pm), Meg_Maak (om) [B, 0 Text (:xt) [abc...]]
Variables: Learneble Queries (Q Attention Masks between Q and (gpm)
Funstions:
img_ancoder Text sext__sncoder sncoder Visual_Sampler feature_atton prrmppt_aten output
init
Qa Qt, Qu copy (1: initialize object, text and visual queries.
Per Pi Tin__nnodde (Ing), [oxt_Rneode (txt); Fe and P, denote Image reature, text prompt. Po Visual (Fu, pey am): Sample visual prompt from image festure, pos/aeg
def SEEM Decoder For Qu, Por Pm)):
Qo, fecture attr festure_attr
(Fr,Qor Qu, Qu,Qul;t Qul:t Croas attend queries with image features, Q., prompt_atte att_atttnnie (gpm, Qu Qu Pu, Pt, Pul; Self attard queriea prompts. Om Ocr Oc. Pm
(F ,, Qor Qe, Qulit Compute mask and class outputs.
def forward (img, txt):
10
F., Qu Qu Qu, P. init();; Pm
Initialize variablas.
11
fori in range (max_iters iter):
Om, Oc, Pm Recoddr (Fv, Qo. Qc Qu Pm))
propose new type of prompt called memory prompts P.,. and use them convey the knowledge of the masks from the previous iteration to the current one. Unlike previous works that use a network to encode the previous mask [20] 36). introduce extra module but simply few memory prompts. These memory prompts encode the history information by using a maskeguided cross-attention layer [6]:
P Masked Att Mpp Z)
(7)
where Mp is the previous mask, and Z is the image feature map. In this way, cross-attention only takes effect inside the regions specified by the previous mask. The updated memory prompts P m then interact with the other prompts via self-attention to convey the historical information for the current round.
Semantic-aware. Different from previous class-agnostic interactive segmentation works such as Simple Click [20] and the concurrent work SAM [36), our model produces semantic labels masks for all kinds of prompt combinations in zero-shot manner, since our visual prompt features are aligned with textual features a joint semaatie space. shown in Fig. 3 semantic labels are directly computed using Of, (output of visual queries) and the text embedding. Although we do not train with any semantic labels for interactive segmentation, the calculated logits are well-aligned, benefiting from the joint visual-semantic space.
3.2 Model Pipeline and Loss Functions
We summarize the training and evaluation pipeline of the proposed method with Pytorch-style pseudo- code in Algorithm I SEEM is trained with linear combination of losses for panoptic segmentation, referring segmentation, and interactive segmentation:
=afe._em pano + 3Lm BCE pano + "7Cm_DDCC,ap + aL Ce + bCm
(8)
+cCm_DICE + aLe CE_ bCm Bsc_ + cCm _iC__iCC_iseg
Where a 2, = 5, 0.2, b c = 2, CE, BCE, and DICE denotes cross-entropy, binary cross entropy and dice loss, respectively.
Experiments
Datasets and Settings. SEEM is trained on three tasks: panoptic segmentation, referring seg- mentation, and interactive segmentation. Panoptic and interactive segmentation are trained on COCO2017 7 [5T] with panoptic segmentation annotations. Following [II], we exclude the validation set of Ref-COCOg (52), resulting in 107K segmentation images in total. For referring segmentation, combination of Ref-COCO, Ref-COCOg, and Ref-COCO+ for COCO image annota- tions. We evaluate generic segmentation (instance/panopiic/seattit referring segmentation, and interactive segmentation.
Implementation Details and Evaluation Metrics. Our model framework follows X-Decoder except the decoder. That is, we have vision backbone, a language backbone, encoder, and
5

---

Table I: One model for segmentation wide range of segmentation tasks. SEEM is the first model to simultaneously support generic segmentation, referring segmentation, and interactive segmentation, as well as prompt compositionality. (#Concurrent work. indicates the model does not have capability for the task, indicates do not have reported number.)
Caseik Squnecaaiion
Refeniing Segsentation
Inksatin Segunstaion
Segromrztion
Type
mAP
miti
ctel)
miet
A150
5-NoCCS
21-10035
10-Na090 20-NAUM
LIKCOF00 2303
12N6)
4b33
01.1
Mask2Faru
C0C0
COCO (0 1250
Segrastation
35.4
(CMM)
51.
ho (13
RofCCOC--VG-.... (0, (GM)
71.
RITM LeTh
COCO-1VIS un IEMD
2.19
2.57
Toaaaa
COCC 1250
teeakack (8) 21
LUKC0-001220 12M4
2.45
2305
12M01
122
COCO+1VIS COCC--VVI (0 13V0
1551
155
COCO 1356
45.8
X-Uaander (1
L3432 12M)
41.3
12.4
X.Daodke (B: [B]
COCO--0 1766
45.8
660
66.5
UNISENT
InagoeVila (MM)
44.9
100
UNISETT 1u
Tesage-Vadon LMM
455
154
Pamari 20
CERCO+ADEANS V1 (Q.160) Gramndit
20G-AAD-+NVV2 (0.12
aSAM (b) 13
SAM (1 IM) (11N0)
2,47
2.45
3.78
2.25
3.15
4.3
[L]
188
2.15
2.01
2.46
3.12
SEESE (T)
LOCOHLVVS 12M0
53.5
390
62.2
5619
65.7
14.5
1.72
3.37
1.00
233
4.1
SEEM (8)
COCO-LVVS (U1130)
15.1
45.4
06.3
15.0
03.6
75.1
1.55
2993
1.27
247
1.70
EDCSS1VVS5 155/1
164
117
K31
179
135
4:7
SEEM (1)
1TCO 1380)
Conpodtton
76-3
778
87.8
1.56
2:01
1.77
2.66
176
COCOLLVIS
75.1
76.9
16.3
1.2
1.22
238
3.64
Table 2: One model for all kinds of mask interactions. SEEM has strong generalization capability on
different input mask types.
COCO
Open Image
ADE Scriible 1olol)
Method
Point JoU
Stroke 1:J0U
Scribble 1.oUU
Box 1
Point
Scuibble JoU
Pologgn
BoX 1
Point
Stroke 1-lol
Polygon 1
BoX
1
loU
SingleCric (B)
35.1
42.5
48.6
54.2
401.5
42.7
47.0
1900
45.3
SimpleClick
33.9
36.8
53.6
15.4
SAM
58.6
51.0
21.9
SAM
64.7
57.1
57.8
52.4
57.4
45.8
SAM (H)
30.6
50.4
52.1
25.3
SEEM (T)
78.9
$1.0
81.2
72.2
73.7
67.1
09.5
68.9
65.4
67.3
67.3
59.0
81.7
82.8
83.5
76.0
75.7
67.6
60.0
68.7
64.2
60.3
66.4
68.6
67.7
68.5
SEEM (L)
83.4
84.6
84.1
76.5
76.9
66.8
67.8
67.6
62.4
50.1
65.5
06.6
66.3
58.1
$6.1
SEEM-Decoder For the vision backbone, we use FocalT (58), DaViT-d3 (B), and DaViT-d5 (L) [59]. For the language encoder, we adopt UniCL or Florence text encoder [60 61. For all segmentation tasks, we use standard evaluation metrics: PQ (Panoptic Quality) for panoptic segmentation, AP (Average Precision) for instance segmentation, and mloU (mean Intersection over Union) for semantic segmentation. For interactive segmentation, we follow previous works 20 62 clicks by comparing the predicted segmentation with the ground-truth one in an automatic way. After one click on the image to generate the predicted mask, the next click is placed at the center of the area with the largest segmentation error. We use the Number of Clicks (NoC) metric to evaluate interactive segmentation performance, which measures the number of clicks needed to achieve a certain Interscction over Union (IoU), i.c., 85% and 90%, denoted as NoC@85 and NoC@90, respectively. We also vary the number of maximum clicks indicated by K-NoC@90 (K=5, 10, 20), and evaluate the mean IoU on the single click denoted as I-loU to study the performance on different constraints. More qualitative evaluation with stroke, scribble, polygon, and box as prompts illustrated in the supplementary material.
to simulate user
4.1 Main Results
Generic segmentation With one suite of parameters pre-trained on all the segmentation tasks, are able to evaluate its performance on generic segmentation datasets. As shown in Table T SEEM maintains competitive panoptic, instance, and semantic segmentation performance against strong baselines. Compared with generalist models such as UViM [55], Pix2Seqv2 (56) and especially the recent model Painter (57) and SegGPT [50], approach significantly outperforms those methods on generic segmentation with a margin around 10 points on panoptic segmentation metrics.
Referring segmentation As shown in Table I compared with other referring segmentation and generalist models, SEEM achieves competitive performance. Notably, by adding a visual composi- tional prompt, referring segmentation performance is improved with large margin by 10.5 cloU,

---

Table 3: Zero-shot video object segmentation. Without training with video or pairwise image data, our approach is able to do video object segmentation in a zero-shot manner. (#Concurrent work.)
Methed
Sepricstation Data
Type
Strgle
HAVISST
STTT T01S
VIISSDAAVIS of IM)
Mbat
67.4
66.
-0.4
71.5
71.3
66.5
15-3
71.1
(SymthiVi SAWISS 11M)
7010
67.3
732
06.0
51.3
XMMm 04
DDR00--- (0.253))
16.1
19.8
B013
19.2
SumMMra 67
CUKDD+UU (0-213))
Bos
56.5
15.5
710
6128
6303
58.2
45.3
47-7
s0yus
Maskficrnni
94.5
81.1
91.01
12.4
12.5
51.1
85.6
71/3
36.2
RafEOCO VOS
64.1
JAM- 20
XMEm=SAM (()
Millin Piizz
15.4
1964
LNIMETT 1
13
Trage-
Image-Vittnd (33)
Cenertle
Must
24.5
11.3
71.6
78.1
12.0
20.5
81.0
71.0
20L8
79.8
UNINENTI
Milec(M)
Wature Vade
COCO-ADE-NYYLLA (UMM
Mask
/
38.6
28.5
40.8
26.1
17.6
358
14.3
187
756
76.
67.4
#PuSAM-1
SAMHDAVIS (11M)
63.9
SEEM-T
00.4
57.
63.3
514
55
641
59.3
46.9
SREM-8
COCOHENTS (O.17M) 77N)
Made
62.8
06.1
70.9
53.8
606
65.5
47.
Table 4: Ablation study on interaction strategy. "#lter" denotes the maximum training iteration on interactive segmentation in single forward. "Negatíve" means adding negative tokens during interactive segmentation. "Scratch" means the model trains from scratch.
Ablation
Fix
Pos
Neg
COCO
Relerring Seginnnlation
Fuscal VOC
DAVIST7
PQ
mAP
mloU
cloU
miol
AP
NoC50
NoCyo
JF
]
Bascline
50.7
39,5
60.8
57.9
63.3
71.6
1.74
5.43
59.6
55.8
63.5
LVIS
39.8
62.2
58.6
72.6
1.57
59.5
55.9
63.
Negative
/
50.9
39.8
61.4
58.8
64.0
72.6
1.81
5.41
60.1
56.3
63.9
Scratch
39.5
51.4
67.0
1.45
57.7
50.7
39.7
60.5
58.3
63.44
71.3
1.76
5.14
59.2
55.4
63.07
+ fter
/
50.4
39.5
61.0
58.0
58.0
63.0
71.5
1.55
4.67
59.9
56.2
56.4
63.5
50.6
39.4
60.9
58.4
63.4
71.6
1.54
4.59
59.7
56.3
63.1
6.0 mloU, and 9.3 AP50 points for the tiny model. And this gap is retained for the base and large model. Specifically, this number is computed by class embeddings 0% (Output-Q-Texuul margin is even larger when computed with mask embeddings O,," (Output-Q-Viste as shown in Table 5 Further, we benchmark the vanilla composition (Ensemble) that directly combines visual and text mask output probabilities as shown in Table[]]oow 2.
The
Interactive segmentation As shown in Table our approach achieves comparable performance with the specialized models, e.g. RITM, SimpleClick, and better performance than SAM (36) (B) which is trained with x 100 more segmentation data than ours. Notably, unlike existing interactive models, SEEM is the first interface that supports not only classical segmentation tasks but also a wide range of user input types, including text, points, scribbles, boxes, and images, providing strong compositional capabilities as shown in Table
Table 5: The term 'Text/Visual Prompt' refers to the modality of information utilized in the study. 'Output Query' is indicative of the type of query employed to predict the output. 'Composition Approach' specifies the method through which text and visual information are integrated.
Text Prompt
Visual Prompt
Output Query
Composition Approach
Focal-Tiny
Davit-Base mloU
Davit-Large
cloC
mloU
AP@50
cloU
AP@
cloU
mloU
AP@
N
Text
N/A
58.4
63.4
71.6
63.0
68.2
76.7
62.4
67.6
75.3
Y
All
Ensemble
63.0
60.0
66.9
69.3
66.6
74.3
68.9
65.5
72.7
Y
Text
Self-Altn
66.5
69.6
78.8
75.0
76.9
86.3
73.2
76.5
85.9
Y
Visual
N/A
70.7
71.8
81.3
75.4
77.8
87.4
75.2
78.2
87.7
Y
Visual
Self-Attn
71.5
72.8
82.2
75.9
78.3
87.7
74.9
78.4
87.7
User input type of interactive segmentation In Table 2 we compare 1-loU of SEEM with other strong baselines SimpleClick and SAM with 5 common types of prompts on three datasets. 1-loU indicates the mean IoU of all images with a single click. The prompt types include point, stroke, scribble, and box. The results show that our SEEM achieves the best performance in the extremely limited number of clicks over all three datasets.
Video object segmentation Without any modification, our model is able to do (interactive) video object segmentation in zero-shot manner through the visual prompt (by replacing the current image visuals prompt with the visual prompts from another image). As shown in Table B without any observation of DAVIS/VOS dataset 77 72, our approach is able to achieve close performance in zero-shot manner with fully supervised method on DAVIS17 dataset [72.. Meanwhile, our model is able to do interactive video object segmentation on DAVIS16-Interaan comparable performance with the supervised basclines with one single click of the first frame.
72) and achieves
7

---

Figure 4: Click/seribble-basee segmentation. SEEM supports arbitrary formats of clicks or seribbles by users. Moreover, simultaneously gives the semantic label for the segmented mask, which is possible in SAM [36]
Figure 5: Text to mask or text referring segmentation. The referred text is shown on the masks. SEEM adapts to various types of input images the domain of cartoons, movies, and games.
4.2 Ablation Study
We conduct an ablation study on all the training segmentation tasks and zero-shot video object segmentation, dissecting each component of our model. The results are presented in Table
LVIS mask annotation will improve interactive segmentation results. We replace the COCO mask with an overlap IoU larger than 0.7 with LVIS mask during training. This will improve the performance on interactive segmentation with 0.3 and 0.2 point gain on NoCO.9 and NoCO.85.
Training from scratch only hurts referring segmentation performance. We compare the SEEM model trained with X-Decoder pre-trained checkpoint the checkpoint initialized with UniCL or Florence vision and language backbone (+Scratch). It indicates that training from scratch will slightly improve the performance on interactive segmentation but hurt the referring segmentation performance.
Increase interactive training iterations does help. As shown in Table increasing the training iteration (the first N-1 iteration is without gradient) from to 5 will gradually improve the interactive segmentation performance from 5.41 to 4.59 on NoCO.9. As the computation cost increases with more clicks, iteration 3 for the main paper results.
4.3 Qualitative Results
We further qualitatively evaluate SEEM. Based on the proposed prompting scheme and decoder design, with the same suite of parameters, SEEM supports a wide range of visual input types.
Visual prompt interactive segmentation. In Fig. we show the visualization using SEEM to segment objects in an interactive way. The user segment objects of interest by simply clicking or drawing scribble. Taking these prompts, SEEM can simultaneously produce both masks and semantic labels for the objects. Note that our model is open-vocabulary, which empowers it to label unseen categories when given the candidate vocabulary (i.e., cheetah and butterfly in Fig. 4). When vocabulary given, SEEM segment in class-agnostic manner.
Text referring segmentation. show the text referring to segmentation visualization results in Fig. 5 The results demonstrate that model is semantic-aware of open-vocabulary concepts and

---

Referring Image
Referring Image
Figure 6: Zero-shot visual referring segmentation with SEEM. Given a referring image with simple spatial hints, SEEM can segment the regions which are semantically similar in different target images.
Elefering (Frame
Frame
Reforinn (Frume
Frame
Frame
Frame
Frame
Rétecting (Frame
Frame
FrAmel
Frama 6
Froma 18
Figure 7: Zero-shot video object segmentation using the first frame plus one stroke. From top to bottom, the videos are "parkour" and "horsejump-low" from DAVIS (73), and video 101 from YouCook2 (74). SEEM precisely segments referred objects even with significant appearance changes caused by blurring or intensive deformations.
attributes understand language. In addition, SEEM is able to generalize unseen scenarios like cartoons, movies, and games.
Visual referring segmentation. In Fig 6 we show SEEM's segmentation results when prompted with referring regions from another image. By simply drawing a click or scribble on one referring image, SEEM can take it as input and segment objects with similar semantics other images. Notably, this referring segmentation has powerful generalization capability to images of other domains. For example, by referring to the elephant in the forest, another object of the same category can be segmented well under drastically different scenes like cartoons, plush toys, and grassland.
Video object segmentation. In Fig. 7 we further show SEEM's referring segmentation ability on the video object segmentation task zero-shot manner. By referring to the objects in the first frame with scribbles, SEEM can precisely segment the corresponding objects in the following frames, even when the following objects change in appearance by blurring or intensive deformations.
Conclusion
We presented SEEM, which can segment everything (all semanties) everywhere (all pixels) all at once (all possible prompt compositions). Apart from performing generic open-vocabulary segmenta- tion, SEEM can interactively take different types of visual prompts from the user, including click, box, polygon, scribble, text, and referring region from another image. These visual prompts are mapped into a joint disual-semanté space with prompt encoder, which makes our model versatile to various prompts and can flexibly compose different prompts. Extensive experiments indicate that our model yields competitive performance on several open-vocabulary and interactive segmentation benchmarks. Further studies revealed the robust generalization ability of our model in accurately segmenting images based on diverse user intents. We hope our work will serve as a stepping stone toward universal and interactive interface for image segmentation and beyond.

---

Acknowledgements. We would like to express our gratitude to Lei Zhang for his generous support. And express the appreciated for the valuable suggestions from Zhenyuan Yang, and discussion with Xiaoyu Xiang. In addition, this work was supported in part by NSF CAREER 10021 NASA SONSSC21K029) the Institute of Information and communications Technology Planning and Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2022-0-00871, Development of AI Autonomy and Knowledge Enhancement for AI Agent Collaboration).
References
[1] Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE Transactions on pattern analysis and machine intelligence, 22(8):888-900 2000.
[2] Liang-Chiel Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4))834-848, 2017.
[3] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9404-9413, 2019. 2019.
[4] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
[5] Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang- Chich Chen. Max-deeplab: End-to-end panoptic segmentation with mask transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5463-5474, 2021.
[6] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1290-1299, 2022.
[7] Feng Li, Hao Zhang, Shilong Liu, Lei Zhang, Lionel M Ni, Heung- Yeung Shum, et al. Mask dino: Towards a unified transformer-base based framework for object detection and segmentation. arXiv preprint arXiv:2206.02777 2022.
[8] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.
[9] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Yi Lin. Open-vocabulary image segmentation. arXiv preprint 11214:211.. 2021.
[10] Zheng Ding, Jieke Wang, and Zhuowen Tu. Open-vocabulary panoptic segmentation with maskelip. arXiv preprint arXiv:2208.08984, 2022.
[11] Xueyan Zou, Zi- Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. Generalized decoding for pixel, image, and language. arXiv preprini arXiv:22/2.
2022.
[12] Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open- vocabulayy semantic segmentation. arXiv preprint arXiv:2302.12244 2023.
[13] Chenxi Liu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, and Alan Yuille. Recurrent multi- modal interaction for referring image segmentation. In Proceedings of the IEEE International Conference on Computer Vision, pages 1271 2017.
[14] Linwei Ye, Mrigank Rochan, Zhi Liu, and Yang Wang. Cross-modal self-attention network for referring image segmentation. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 10502-10511, 2019.
10

---

[15] Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. Language as queries for referring video object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4974 2022.
[16] Shijia Huang, Feng Li, Hao Zhang. Shilong Liu, Lei Zhang, and Liwei Wang. A unified mutual supervision framework for referring expression segmentation and generation. arXiv preprint MxXi::211.009111 2022.
[17] Jiang Liu, Hui Ding, Zhaowei Cai, Yuting Zhang, Ravi Kumar Satzoda, Vijay Mahadevan, and Manmatha. Polyformer: Referring image segmentation as sequential polygon generation. 2023.
[18] Konstantin Sofiiuk, Ilia A. Petrov, and Anton Konushin. Reviving iterative training with mask guidance for interactive segmentation, 2021.
[19] Ho Kei Cheng, Yu- -i-Winng Tai, and Chi Keung Tang. Modular interactive video object segmenta- tion: Interaction-to-mase propagation and difference-aware fusion. In CVPR, 2021.
[20] Qin Liu, Zhenlin Xu, Gedas Bertasius, and Mare Niethammer. Simpleclick: Interactive image segmentation with simple vision transformers. arXiv preprint 2rXiv:2210. 1006, 2022.
[21] Xi Chen, Zhiyan Zhao, Yilei Zhang, Manni Duan, Donglian Qi, and Hengshuang Zhao. Fo- calclick: towards practical interactive image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1300-1309, 2022.
[22] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 1377-9711,,, 2020.
[23] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-555) 2020.
[24] OpenAl. Gpt-4 technical report, 2023.
[25] Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Auto- prompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010. 15980, 2020.
[26] Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pages 12697-12706. PMLR, 2021.
[27] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprini arXiv:2
2021.
[28] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning large language models. arXiv preprint 2rXiv:2201.11990 2022.
[29] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv preprint 22x::2005.11116. 2022.
[30] Timo Schick, Jane Dwivedi-Yo Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023.
[31] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. European Conference on Computer Vision, pages 213-229. Springer, 2020.
[32] Hao Zhang, Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianfeng Gao, Jianwei Yang, and Lei Zhang. simple framework for open-vocabulary segmentation and detection. arXiv preprint arXiv:2303.081111 2023.
11

---

[33] Yin Li, Jian Sun, Chi- Keung Tang. and Heung- Yeung Shum. Lazy snapping. ACM Transactions on Graphies (ToG). 23(3):303-308, 2004.
[34] Leo Grady. Random walks for image segmentation. IEEE transactions on pattern analysis and machine intelligence, 28(11): 1783, 2006.
[35] Ning Xu, Brian Price, Scott Cohen, Jimei Yang, and Thomas S Huang. Deep interactive object selection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 373-381, 2016.
[36] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything, 2023.
[37] King-Sun Fu and JK Mui. survey on image segmentation. Puttern recognition, 13(1):3-16. 1981.
[38] Pedro F Felzenszwalb, Ross B Girshick, David McAllester, and Deva Ramanan. Object detection with discriminatively trained part-based models. IEEE transactions on pattern analysis and machine intelligence, 3209):66771165 2009.
[39] Zhengxia Zou, Zhenwei Shi, Yuhong Guo, and Jieping Ye. Object detection in 20 years: A survey, arXiv preprint arXiv:1905.05053 2019.
[40] Shervin Minaee, Yuri Y Boykov, Fatih Porikli, Antonio J Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos. Image segmentation using deep learning: A survey. IEEE transactions on puttern analysis and machine intelligence, 2021.
[41] Liang-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprins arXiv:1706.05587, 2017.
[42] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for se- mantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431-3440, 2015.
[43] Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee. Yolact: Real-time instance segmentation, In Proceedings of the IEEF/CV International Conference on Computer Vision, pages 9157-9166, 2019.
[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[45] Zhiqi Li, Wenhai Wang, Enze Xie, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, Ping Luo, and Tong Lu. Panoptic segformer: Delving deeper into panoptic segmentation with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 1280-1289, 2022.
[46] Jitesh Jain, Jiachen Li, MangTik, Tik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. Oneformer: One transformer rule universal image segmentation. arXiv preprint arXiv:2211.06220, 2022.
[47] Hao Zhang, Feng Li, Huaizhe Xu, Shijia Huang, Shilong Liu, Lionel M Ni, and Lei Zhang. Mp- former: Mask-piloted transformer for image segmentation. arXiv preprint arXiv:2303.07336 2023.
[48] Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, and Huchuan Lu, Universal instance perception object discovery and retrieval. arXiv preprint 0rXiv:2303.06674, 2023.
[49] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206.000 08916, 2022.
[50] Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Segmenting everything in context. arXiv preprint arXiv:2304.03284 2023.
12

---

[51] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[52] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In European Conference on Computer Vision, pages 69-85. Springer, 2016.
[53] Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip HS Torr. Lavt: Language-aware vision transformer for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18155-18165, 2022.
[54] Qin Liu, Meng Zheng, Benjamin Planche, Srikrishna Karanam, Terrence Chen, Mare Nietham- mer, and Ziyan Wu. Pseudoclick: Interactive image segmentation with click imitation. In Computer Dision-ECC) 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Purt VI, pages 728-745. Springer, 2022.
[55] Alexander Kolesnikov, André Susano Pinto, Lucas Beyer, Xiaohua Zhai, Jeremiah Harmsen, and Neil Houlsby. UViM: A unified modeling approach for vision with learned guiding codes. arxiv preprini arXiv: 10337, 2022.
[56] Ting Chen, Saurabh Saxena, Lala Li, Tsung- Yi Lin, David Fleet, and Geoffrey Hinton. A unified sequence interface for vision tasks. arXiv preprint wrXiv22066.0766, 2022.
[57] Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: generalist painter for in-context visual learning. 2023.
[58] Jianwei Yang, Chunyuan Li, and Jianfeng Gao. Focal modulation networks. arXiv preprint arXiv:2203.11922 /1926. 2022.
[59] Mingyu Ding, Bin Xiao, Noel Codella, Ping Luo, Jingdong Wang, and Lu Yuan. Davit: Dual attention vision transformers. arXiv preprint arXiv:2204.03645, 2022.
[60] Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Unified contrastive learning in image-text-label space. In CVPR, 2022.
[61] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang. Bin Xiao, Zhen Xiao, Jianwei Yang. Michael Zeng, Luowei Zhou, and Pengchuan Zhang. Florence: A new foundation model for computer vision. 2021.
arXiv preprint 1r
[62] Zheng Lin, Zheng-Peng Duan, Zhao Zhang, Chun-Le Guo, and Ming-Ming Cheng. Focus- cut: Diving into focus view in interactive segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2637-26488 2022.
[63] Huaijia Lin, Xiaojuan Qi, and Jiaya Jia. Agss-vos: Attention guided single-shot video object segmentation. In ICCV, 2019.
[64] Joakim Johnander, Martin Danelljan, Emil Brissman, Fahad Shahbaz Khan, and Michael Felsberg. A generative appearance model for end-to-end video object segmentation, 2018.
[65] Zhihui Lin, Tianyu Yang, Maomao Li, Ziyu Wang, Chun Yuan, Wenhao Jiang, and Wei Liu. Swem: Towards real-time video object segmentation with sequential weighted expectation- maximization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern 2022.
Recognition, pages
[66] Ho Kei Cheng and Alexander G. Schwing. XMem: Long-term video object segmentation with an atkinson-shiffrin memory model. In ECCV, 2022.
[67] Qiang Wang, Li Zhang, Luca Bertinetto, Weiming Hu, and Philip HS Torr. Fast online object tracking and segmentation: A unifying approach. Proceedings of the IEEE conference on computer vision and pattern recognition, 2019.
13

---

[68] Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. Language as queries for referring video object segmentation. arXiv preprint arXiv:2201 000487 2022.
[69] Jinyu Yang, Mingqi Gao, Zhe Li, Shang Gao, Fangjing Wang, and Feng Zheng. Track anything: Segment anything meets videos, 2023.
[70] Renrui Zhang, Zhengkai Jiang, Ziyu Guo, Shilin Yan, Junting Pan, Hao Dong, Peng Gao, and Hongsheng Li. Personalize segment anything model with one shot, 2023.
[71] Ning Xu, Linjie Yang, Yuchen Fan, Dingeheng Yue, Yuchen Liang, Jianchao Yang, and Thomas Huang. Youtube-vos: A large-scale video object segmentation benchmark. arXiv preprint arXiv:1809.033
2018.
[72] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Corkine-Hormund and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv: 1704.0065, 2017.
[73] Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Dorkine-Hornune A benchmark dataset and evaluation methodology for video object segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 724-732, 2016.
[74] Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
14


Reviews:
[{"review_id": "9oY5CX4z3D", "reviewer": "Reviewer_cQYZ", "summary": "This paper proposes a universal segmentation model, SEEM, for all segmentation tasks. A visual sampler module unifies different kinds of human inputs and they are encoded into a joint visual-semantic space together with image and text so that the SEEM model can learn semantic labels for masks. The proposed SEEM model achieves competitive performance across interactive segmentation, generic segmentation, referring segmentation, and video object segmentation. Besides, SEEM shows a strong ability to generalization to novel prompts or their combinations.", "questions": "1. Although SEEM can learn the semantic labels of masks, it is limited to the categories of COCO. Is there any way to expand the scope of semantic labels without introducing a large amount of extra data?\n2. Can you provide a comparison with other universal segmentation models in terms of the number of model parameters and calculations? If this comparison is not applicable, what are the reasons?", "limitations": "Limitations are not explicitly discussed in this paper. The scale of training data is limited, and it does not support partial segmentation as in SAM (Segmenting Anything Model).", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed model has superior performance and has achieved competitive results across different types of segmentation tasks. \n2. The proposed model can accept a variety of visual prompts input types and has a strong generalization ability, which makes it possible to be applied to more scenarios.\n3. Different from the previous segmentation model, the proposed model can learn the semantic labels corresponding to the segmentation masks, thus enabling open vocabulary segmentation.", "weaknesses": "1. In this paper, when comparing the performance of the proposed model with the previous models, only the scales of training data are compared, without the comparison of parameters and calculation. This may cause some doubts about the comparison fairness.\n2. The analysis of some results is insufficient. In the ablation study section, the performance changes caused by the change of variables are not consistent in different segmentation tasks. Only the changing trend of the results is given without analyzing the potential reasons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Insufficient Analysis for ablation study.**\n\nThanks for the suggestion. We have analyzed the ablation study in Section 4.2 of the main paper. To make it more comprehensive, we further summarize and analyze the main findings in Table 4 below:\n\n*Removing LVIS mask annotations (Row 2 vs. Row 6)* from the training data will improve generic segmentation, and referring segmentation performance, but decrease the interactive segmentation performance. That is caused by the fact that although LVIS segmentation mask is more accurate than coco (improves performance on interactive segmentation), however, its mask has a domain gap with coco annotation (decrease on generic and referring segmentation).\n\n*Training from scratch (Row 4 vs. Row 7)* that (1) UniCL pretrained weight instead of X-Decoder pretrained weight is used, (2) the vision backbone and transformer encoder are unfixed. It will not influence generic segmentation performance, but decrease the referring segmentation performance, and improve interactive segmentation, vos performance. That is caused by the fact that X-Decoder weight actually provides better region-language alignment than UniCL pretrained weights. Thus the performance drop on referring segmentation is reasonable. In addition, as the weight of the vision backbone and transformer encoder weight is tuned, it will give better performance on interactive segmentation tasks that is newly proposed in SEEM as more weight can be adapted to the new task.\n\n*Increasing the training iteration (Row 5-8)* on interactive segmentation will not influence the performance on generic segmentation, referring segmentation and vos, while improving the interactive segmentation result consistently. That is caused by the fact that the first n-1 iterations will only forward interactive segmentation without any gradient. Also adding training iterations that mimic the human interaction with images will improve performance when evaluating multi-round IoU.\n\nWe will include the above detailed analysis in our revision.\n\n**Q2: Generalize to open-vocabulary interactive segmentation.**\n\nWe are able to generalize to more semantic labels without further training as shown in the example images in Fig.3 of supplementary material (e.g. Yellow Corn, Leaf, Orange Juice, etc. are never trained in coco dataset). This behavior is acquired by two reasons: \n\n(1) We leveraged language encoders that are pretrained with language-image contrastive loss (CLIP or UniCL), which renders more generalizable language embeddings for a wide range of visual concepts. \n\n(2) We used the pre-trained X-Decoder as the initialization and only fine-tuned the decoder part so that it retains open-vocabulary capability.\n\nIn addition to the qualitative results, here we evaluate on ADE20K dataset and compared it with prior work X-Decoder. From the table, we can clearly observe that although SEEM never trained towards an open-vocabulary segmentation objective, however, it retains reasonable open-vocabulary capability with X-Decoder pretrained checkpoint. In addition, for the model initialized with SAM that doesn't have any vision-language pretraining such as UniCL, it still performs comparably with X-Decoder-Seg that uses the same amount of annotations. \n\n|                    | Backbone  | COCO  |      |         | Others           | ADE20K |      |      |\n|--------------------|-----------|-------|------|---------|------------------|--------|------|------|\n|                    | Pretrain  | Class | Mask | Caption | Image-Text Pairs | PQ     | mAP  | mIoU |\n| x-Decoder-Seg (B)  | UniCL     | Y     | Y    | X       | X                | 15.3   | 8.3  | 19.5 |\n| x-Decoder-Seg+ (B) | UniCL     | Y     | Y    | Y       | X                | 16.9   | 9.5  | 23.8 |\n| X-Decoder (B)      | UniCL     | Y     | Y    | Y       | Y                | 21.1   | 11.7 | 27.2 |\n| SEEM (davit-d3)    | X-Decoder | Y     | Y    | X       | X                | 16.7   | 11.6 | 23.5 |\n| SEEM (samvit-B)    | SAM       | Y     | Y    | X       | X                | 13.3   | 9.5  | 18.9 |\n\nWe will incorporate this discussion in our revision.\n\n**Q3: Comparisons of model parameters and calculations.**\n\nThanks for pointing out this. In this work, our goal is to develop a versatile and universal image segmentation interface and thus we mainly focused on the comparisons with previous work in terms of task breadth and performance. Regarding the model parameters and calculations, we did list the rough size of backbones used in different methods (tiny (T), base (B), and large (L)), but failed to find a well-established way from the prior arts to analyze the computation of each generalist model, considering it is task-dependent while each model is used for different sets of tasks. Moreover, we faced the difficulty that the weight and inference code are not necessarily released for each specific task. That being said, we are not able to provide a thorough analysis of calculations for now but will definitely try our best to find a common-grounded way to solidly study this aspect."}, {"review_id": "UPWUlVFYqu", "reviewer": "Reviewer_R6GZ", "summary": "[Task] In this work, authors introduc SEEM, a promptable and interactive model designed for comprehensive image segmentation. SEEM aims to segment all objects in an image simultaneously, addressing various segmentation tasks. The key contribution of SEEM is its novel decoding mechanism, which allows for diverse prompting and behavior similar to large language models (LLMs).\n\n[Method] SEEM is built with four key objectives: versatility, compositionality, interactivity, and semantic-awareness. The model incorporates a visual prompt that unifies different spatial queries such as points, boxes, scribbles, and masks, enabling generalization to different referring images. It also learns a joint visual-semantic space for dynamic prompt composition in different segmentation tasks. Additionally, SEEM incorporate learnable memory prompts in the decoder to retain segmentation history and utilizes mask-guided cross-attention from the decoder to image features. Moreover, a text encoder is employed to encode text queries and mask labels into the same semantic space, facilitating open-vocabulary segmentation.\n\n[Experiments] Empirical studies validate the effectiveness of SEEM across diverse segmentation tasks. The model achieves competitive performance in interactive segmentation, semantic segmentation, referring segmentation, and video object segmentation across nine datasets, requiring only minimal 1/100 supervision. SEEM demonstrates remarkable generalization capability to novel prompts or their combinations, positioning it as a versatile and universal image segmentation interface.", "questions": "[Handling dataset imbalance] Since SEEM is trained on multiple datasets, there might be an imbalance in the number of samples per dataset. I'm curious to know if adjusting the frequency of training samples from each dataset has any impact on the model's performance. Can the authors shed some light on whether they have explored techniques such as data augmentation or sample weighting to address this issue? Additionally, it would be interesting to understand if adjusting the training sample frequency has any effect on mitigating the impact of dataset imbalance and improving the overall model performance.", "limitations": "Yes, the authors adequately addressed the limitations", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "[highlight the trend towards more flexible segmentation models] In this work, the authors address the need for a universal segmentation interface capable of handling different types of human prompts and addressing various segmentation tasks. They highlight the trend towards more flexible segmentation models, including open-vocabulary segmentation, referring segmentation, and interactive segmentation. Inspired by the success of Large Language Models (LLMs) as universal interaction interfaces for language tasks, the authors propose SEEM, a promptable and interactive model for segmenting everything everywhere all at once in an image.\n\n[technical contribution] The proposed approach, SEEM, introduces a novel prompting scheme in the mask decoder with four key properties: versatility, compositionality, interactivity, and semantic-awareness. By encoding different types of prompts (points, masks, text, boxes, and referred regions) into a joint visual-semantic space, SEEM achieves strong compositionality and can handle any combination of input prompts. Memory prompts are introduced to retain previous segmentation information and enable interactivity. The model also provides open-set semantic labels for output segmentation.\n\n[tasks and results] The model is trained on diverse segmentation tasks to learn how to handle different prompts, align visual and text prompts, and promote their synergy through cross-attention. The single pre-trained SEEM model achieves competitive performance across all segmentation tasks, leveraging the joint visual-semantic space for prompt combination and zero-shot adaptation. In addition to its strong generalization capability, SEEM is efficient for interactive segmentation compared to other methods. Overall, SEEM offers a segmentation interface with a single pre-trained model that can handle all types of prompts, segment every object with semantics, and cover every pixel in the image.", "weaknesses": "[Model performance] The model performance of SEEM on certain benchmarks, including ADE20K for open-vocabulary panoptic segmentation and DAVIS for video instance segmentation, is often lower, and in some cases, significantly lower than the baselines X-Decoder, ODISE, and UNINEXT. It is worth noting that SEEM's framework follows a similar approach to X-Decoder. \nWhile the concept of a universal and interactive segmentation interface is intriguing, it appears that the model performance on open-vocabulary benchmarks is compromised in SEEM. It would be beneficial for the authors to provide an explanation for this discrepancy.\n\n[Unfair comparisons] In Table 2, there are some comparisons that seem unfair for two reasons. Firstly, the evaluation of SEEM on COCO, which provides instance-level annotations, aligns better with SEEM's training pipeline that includes instance-level annotations. However, this evaluation method puts SAM at a disadvantage since it is trained to respect hierarchical segmentation, including whole-part-subpart relationships. Secondly, SEEM is already trained on in-domain data from COCO's training set, while SAM is pretrained on out-of-domain data. This discrepancy in training data can impact the performance comparison between the two models. To ensure fair comparisons, it would be beneficial to evaluate both models using evaluation protocols that align with their respective training pipelines and take into account the domain of the training data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Performance on open-vocabulary segmentation is lower in comparison with X-Decoder, and performance on video instance segmentation performance on DAVIS is lower than UNINEXT.** \n\nWe agree with the reviewer that our open-vocabulary performance on ADE20k is *potentially* lower than X-Decoder and ODISE, and the VIS performance is lower than UNINEXT. While all these methods are toward a unified architecture, the underlying goal and training data are very different.\n\n|           | Input Modality                  | Output Modality   | Composition      | Emerging Property                                         | Segmentation Data  | Ensemble |\n|-----------|---------------------------------|-------------------|------------------|-----------------------------------------------------------|--------------------|----------|\n| SEEM      | Vision & Language & Human Input | Vision            | Task & Embedding | Referring Image Segmentation, Video Instance Segmentation | COCO (0.12M)       | N/A      |\n| X-Decoder | Vision & Language               | Vision & Language | Task             | N/A                                                       | COCO (0.12M)       | N/A      |\n| ODISE     | Vision                          | Vision            | N/A              | N/A                                                       | COCO (0.12M)       | Y        |\n| UNINEXT   | Vision & Language & Video       | Vision            | N/A              | N/A                                                       | Image + Video (3M) | N/A      |\n\n(1) Open vocabulary Segmentation: In comparison to X-Decoder, ODISE, we agree with the reviewer that we are potentially lower. However, X-Decoder are trained on 4M more image text pairs, while ODISE have heavily fixed ensemble modules (Implicit Captionor, Mask Generator, Diffusion UNet), while only the alignment is tuned. The SEEM objective is never trained towards open-vocabulary segmentation than X-Decoder and ODISE.\n\nOne interesting thing is that we never provide any quantitative open-vocabulary evaluation in neither main paper nor supplementary material. To resolve the confusion, here we provide the full evaluation table. we can clearly observe that although SEEM never trained towards open-vocabulary segmentation objective, however, it retains resonable open-vocabulary capability with X-Decoder pretrained checkpoint. In addition, for the model initialized with SAM ViT-B backbone that doesn't have any vision-language pretraing than UniCL, it still perform comparable with X-Decoder-Seg.\n\n|                    | COCO  |      |         | Others           | ADE20K |      |      |\n|--------------------|-------|------|---------|------------------|--------|------|------|\n|                    | Class | Mask | Caption | Image-Text Pairs | PQ     | mAP  | mIoU |\n| x-Decoder-Seg (B)  | Y     | Y    | X       | X                | 15.3   | 8.3  | 19.5 |\n| x-Decoder-Seg+ (B) | Y     | Y    | Y       | X                | 16.9   | 9.5  | 23.8 |\n| X-Decoder (B)      | Y     | Y    | Y       | Y                | 21.1   | 11.7 | 27.2 |\n| SEEM (davit-B)     | Y     | Y    | X       | X                | 16.7   | 11.6 | 23.5 |\n| SEEM (samvit-B)    | Y     | Y    | X       | X                | 13.3   | 9.5  | 18.9 |\n\n(2) Video Object Segmentation: **UNINEXT is explicitly trained with video data and even on VOS dataset**, where our approach is only trained on single image and coco dataset (4th column in Table.3).  For the models that does not trained on Video dataset but with pair images (Painter-L), SEEM nearly doubled the performance (34.6 vs. 62.8 on JF Davis17, 24.1 vs 53.8 on G VOS18). Noted that we never trained on pairwise images. And even compared with the model tuned on in-domain dataset (PerSAM-L with DAVIS data), SEEM could also surpass the baseline with a margin (62.8 vs. 60.3 JF on DAVIS). These results all showed that our approach has strong zero-shot and task generalization capability. \n\n**Q2: Unfair Comparison with SAM on in-domain dataset COCO**\n\nIn both Table.1 and Table.2, we compare SEEM and SAM on VOC, OpenImage and ADE20k dataset (these are out-of-domain datasets) in addition to COCO on both interactive segmentation and first click IoU (1-IoU). COCO is only one of four dataset that we are evaluating, and our model is trained on COCO only and never see other datasets like OpenImage, VOC and etc.. \n\n**Q3: Training dataset imbalance and joint training protocol**\n\nThe problem on joint dataset training is very interesting. However, as shown in Table. 1 Main Paper (L188-189) we never trained on multiple datasets in SEEM, we only trained on COCO dataset that is 1/100 data of SAM. \n\nFor the discussion of joint dataset training itself, X-Decoder and UNINEXT are trained on multiple datasets. In X-Decoder, it trained on COCO and 4M image text pairs while it takes 32 COCO images and 1024 image text pairs in one training iteration. That strategy is chosen by task, where COCO are trained 50 epoch in Mask2Former, and image-text pair loss usually need 1024 batch size to avoid performance drop. In addition to sampling frequency, balancing loss on different tasks also plays an important role on joint training. In addition to train the model in a single batch one pass, UNINEXT are taken use of both object level and pixel level image data, while also leverage video data. In addition to balance the training data in a single batch, they also train different data in different round that means their model is trained on object detection data in the first round, segmentation data in the second round, while for video segmentation on the third round."}, {"review_id": "fw8LLzP27x", "reviewer": "Reviewer_x17t", "summary": "This paper introduces SEEM, an innovative and interactive model for segmentation. SEEM stands out with its versatility in handling various prompts, such as points, boxes, scribbles, masks, and texts. The model's design is elegantly simple yet highly effective. Additionally, SEEM incorporates memory prompts to preserve segmentation history, enhancing its interactivity and overall performance.", "questions": "Do the authors plan to train on larger amount of dataset? (such as the one trained on SAM or other rich datasets with the semantic labels).  It would be very interesting to see how SEEM scale up with large amount of supervised or weakly supervised labels.", "limitations": "not mentioned", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-presented, showcasing clear and concise writing. The authors effectively communicate their ideas.\n\n2. The proposed idea is both innovative and highly effective. The designed model, despite its simplicity, showcases remarkable capabilities when dealing with a wide range of prompts. Its versatility and adaptability make it be applicable with more tasks.\n\n3. The experimental results are solid. Through a combination of qualitative and quantitative analyses, the authors demonstrate the effectiveness of their idea.", "weaknesses": "1. Do the authors observe whether training on all types of prompts performing better than on one type or a subset types of prompts? or which types of prompt performs best and which one still lacks of improvement.  It would be better to perform this ablation because it can give more insights to the society.\n\n2. since this is a very general framework supporting many different types of prompts, and it can be more easily extended with more training data.  I would also like to see its scalability with larger amount of supervised training data if there is sufficient computing resources.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Ablation study on prompt type.**\n\nThanks for the valuable suggestion, ablation the prompt type is intuitive to study the effectiveness of each counter part, here are the results:\n\n|          | Panoptic | Grounding | Interactive |             | COCO |      |      | Ref-COCOg |      |      | VOC      |          |\n|----------|----------|-----------|-------------|-------------|------|------|------|-----------|------|------|----------|----------|\n|          | Vanilla  | Text      | Visual      | Multi-Round | PQ   | mAP  | mIoU | mIoU      | cIoU | AP50 | 20-NoC50 | 20-NoC90 |\n| SEEM (T) | Y        | Y         | Y           | X           | 51.2 | 39.8 | 61.8 | 63.8      | 58.7 | 72.6 | 1.82     | 5.51     |\n| SEEM (T) | Y        | Y         | X           | X           | 51.0 | 39.9 | 61.1 | 64.2      | 58.8 | 72.4 | -        | -        |\n| SEEM (T) | Y        | X         | X           | X           | 50.8 | 39.9 | 60.7 | -         | -    | -    | -        | -        |\n\nWe gradually remove the visual prompt and text prompt from SEEM decoder, and here we only trained on a single round on Interactive Segmentation for fast training purpose. From the table, we can clearly observe that while gradually add addition capabilities to the model, both text and visual prompt will improve panoptic segmentation. And add interactive segmentation will not influence referring segmentation task.\n\n**Q2: Scaling up training data.**\n\nThank you for your insightful suggestion, scaling up SEEM is for a good direction. Although with the short-time frame we are not able to fully train on large scale dataset such as Object365, or SA-1B. However, we are able to take advantage of the pretrained checkpoint on SA-1B. By using the SA-1B pretrained backbone, we are able out-perform our paper reported number with a good margin as shown below:\n\n|          | Backbone | Multi-Queries in Single Inference | 20-NoC@50 | 20-NoC@85 | 20-NoC90 | 1-IoU |\n|----------|----------|------------------------------------|-----------|-----------|----------|-------|\n| SAM (B)  | SAM-B    | X                                  | /         | 3.28      | 4.12     | /     |\n| SAM (L)  | SAM-L    | X                                  | /         | 2.60      | 3.12     | /     |\n| SAM (H)  | SAM-H    | X                                  | /         | 2.55      | 3.11     | /     |\n| SEEM (B) | Davit-d3 | X                                  | /         | 2.93      | 3.79     | /     |\n| SEEM (L) | Davit-d5 | X                                  | /         | 2.77      | 3.61     | /     |\n| SEEM (B) | Davit-d3 | Y                                  | 1.37      | 2.71      | 3.43     | 87.2  |\n| SEEM (L) | Davit-d5 | Y                                  | 1.38      | 2.70      | 3.47     | 88.5  |\n| SEEM (B) | SAM-B    | Y                                  | 1.29      | 2.53      | 3.21     | 86.2  |\n| SEEM (L) | SAM-L    | Y                                  | 1.30      | 2.42      | 2.97     | 84.5  |\n\nHere, Multi-Queries in Single Inference means that we are able to forward multiple interactive query in a single forward. This new capability will not add any training or inference overhead but will improve inference efficiency and accuracy. Our approach now safely surpasses SAM with a good margin, even compare with their huge model.\n\nIn addition, as acknowledged by the reviewer, our model holds the potential for scaling up data due to its compatibility with a diverse range of prompts. Some large-scale datasets that can be harnessed for training encompass SA-1B (consisting of 11 million images, each annotated with masks) and Objects365 (comprising 1.7 million images, annotated with semantic information). To effectively leverage Objects365, which exclusively offers bounding box annotations for training, we can employ SAM to generate mask annotations based on the provided bounding box in Objects365. It is important to note that accommodating such voluminous data for training demands an extended period, coupled with the intricacies of harmonizing diverse data formats inherent in disparate datasets. Therefore, we leave this to future work."}, {"review_id": "sggUsjSadk", "reviewer": "Reviewer_gMmy", "summary": "The paper proposes SEEM, a method that unifies several segmentation tasks. The network takes text and/or different kinds of visual prompts as inputs. The model then outputs the corresponding segmentation masks for the referred objects. In particular, SEEM trains a unified prompt embedding space. A set of learned queries then alternately attends to image features and the prompt embeddings. Finally, the output embeddings are matched to the prompts and decoded into masks.", "questions": "See weaknesses. \n\nMoreover:\nIs the method robust to the case when the prompted text does not match any object in the image?", "limitations": "Limitations are not discussed in the main paper, but should be mentioned.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tUnifying several segmentation tasks.\n-\tImpressive results on most tasks.\n-\tSimilar or better than SAM in many cases, despite less training data.\n-\tReasonably well written.", "weaknesses": "1.\tSome parts of the method lack motivation and details. In particular, the attention mask used in the prompt attention is not well motivated. It is also not well explained that (and how) the output embeddings are matched to the prompts.\n\n2.\tThe paper is heavily derived from X decoder. The magnitude of the contributions is not clear. The authors do not discuss the relation to X decoder. In fact it is hardly mentioned in the paper, yet the method is completely based on it. Seems like the main new addition here is the visual prompt encoder.\n\n3.\tThe results for VOS are quite poor. This indicates that the method is not as robust to object appearance variations, which is crucial in many tasks.\n\n4.\tThe authors should report training and inference time and memory requirements.\n\nI am leaning positive for the paper. However, the authors need to address my concerns above.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Lack motivation and details (e.g. self-attention mask, output embeddings).**\n\nSorry for the confusion, we explain the motivation of self-attention mask in details here.\n\n* Inputs: Text, user input interactive scribbles, and none.\n* SEEM Decoder Inputs: (1) Queries. (2) Prompts. (3) Image Features. \n* SEEM Decoder Operation: (1) Cross-Attention between Queries and Image Features. (2) Self-Attention between Queries and prompts. (3) Compute outputs using Queries.\n\nThe self-attention mask shown in Fig.3 (b) are modeling the interaction between queries and prompts. As we are computing output masks and semantics using queries, *the self-attention between queries and prompts link between inputs and queries, so that the queries understand where to attend during cross-attention with image features to output the final results.* And this is the prompt attention defined in L7 of Algorithm 1.\n\nThen, we illustrate the details in how to compute the output embeddings. Given the queries that have attended with image features and prompts, we project it separately using mask and class embedding layers (L112-123). The final mask and semantic class are computed by the similarity with 1/4 scale image features and language embeddings.\n\n**Q2: Novelty in comparison with X-Decoder.**\nGood question. We appreciate the reviewers point out this very related work that we are not discussed sufficiently. In comparison with X-Decoder, SEEM has the following difference:\n\n* Technique\n\n    (1) The model design is different. As marked with red font in Equation.1, SEEM has visual and memory prompts other than text prompts. Visual prompts can have various formats such as clicks, scribbles, boxes and referred images. Memory prompts are used to memorize history interactions and support multi-round interactive segmentation.\n\n    (2) The training method is different. Different from X-decoder which train on all the masks of an image in an iteration, SEEM trains on one mask for multiple rounds in one iteration to enable multi-round interaction.\n\n    (3) Human Interactive Animation. We construct an interaction segmentation dataset to animate human interactions such as scribbles and clicks as shown in Appendix C1.\n\n* Capability:\n\n    (1) Modality Composition. Our model is able to combine prompts of different modalities for prompting segmentation due to the joint representation space for text and visual prompts.\n\n    (2) Emerging property. Without training, our model shows the capability of image referred segmentation and VOS. Surprisingly, SEEM performs well even when source and target images for referring are in different styles.\n\n**Q3: Performance on VOS in comparison with other methods.**\n\nThanks for the interesting question, VOS indeed is a very important application. However, the main contribution of SEEM is lied on interactive segmentation, and zero-shot prompt composition, referring image segmentation. While VOS is an emerging property.\n\nAs a zero-shot method, although we are not able to out-perform the previous approach that is fully supervised on both dataset and task. However, we achieve better and comparable performance with Painter-L and SegGPT-L that are trained on paired image data. Even without supervised cross-image referencing, SEEM is able to out-perform Painter-L with a large margin (53.8 vs. 24.1 on G metrics in YT-VOS).\n\nComparing SEEM with SegGPT is an apple to pear comparison, for example in coco panoptic segmentation, we are able to out perform Painter and SegGPT with a large margin.\n\n|               | Pretrained    | Segmentation Data        | PQ   | mAP  | mIoU |\n|---------------|---------------|--------------------------|------|------|------|\n| Painter (L)   | *             | COCO+ADE+NYUv2 (0.16M)   | 43.4 | -    | -    |\n| SegGPT (L)    | Painter (L)   | COCO+ADE+VOC+... (0.25M) | 34.4 | -    | -    |\n| X-Decoder (L) | *             | COCO                     | 56.9 | 46.7 | 67.5 |\n| SEEM (L)      | X-Decoder (L) | COCO+LVIS (0.12M)        | 57.5 | 47.7 | 67.6 |\n\nIn addition, when the input prompt is not complete enough to conver the whole object, our tracking performance is much better than SegGPT, which indicates that our model can take more types of prompt and better generalization ability. We also show the visualization results in the rebuttal PDF.\n\n**Q4: Training and evaluation time and memory cost.**\n\n|          | Resolution | #Param | FPS/img | Training Memory | Inference Memory |\n|----------|------------|--------|---------|-----------------|------------------|\n| SEEM (T) | 1024       | 101.1M | 9.0     |                 | 3124M            |\n| SEEM (B) | 1024       | 140.7M | 6.6     |                 | 3184M            |\n| SEEM (L) | 1024       | 415.3M | 3.7     |                 | 4612M            |\n\nWe compute the number on 1 V100 gpu with a single image with resolution 1024 x 1024.\n\n**Q5: Generalization capability on text not included in the image.**\n\nYes, our method is robust to the case when the prompted text does not match any object in the image. Our matching is implemented by computing the similarities between the prompt embedding and object features. We are able to use a threshold to filter those object features with very low similarities."}]

Author rebuttal:
First of all, **we thank all reviewers for their valuable comments and suggestions!**

We sincerely appreciate all reviewers’ time and efforts in reviewing our paper. We are glad to find that reviewers generally recognized our contributions:

**Model.** A strong generalization ability that is able to apply to more scenarios (cQYZ). Enabling open vocabulary segmentation in addition to interactive segmentation (cQYZ). Highlight the trend towards more flexible segmentation models (R6GZ). The proposed method is technical novel (R6GZ). Innovative and highly effective (x17t). Unifying several segmentation tasks (gMmy).

**Experiments.** Superior performance across different types of segmentation tasks (cQYZ). The single pre-trained SEEM model achieves competitive performance across all segmentation tasks (R6GZ). The experimental results are solid (x17t). Impressive results on most tasks (gMmy).

**Writing.** The paper is well-presented, showcasing clear and concise writing (x17t). Reasonably well written (gMmy).

**Two Additional Contributions:**

* We add multi-instance visual prompt training and inference for interactive segmentation (Refer to Fig.1 in Rebuttal PDF). That enables query multiple objects in a single forward pass.

* We use SAM ViT backbone as the pretrained checkpoint to "scale up" our training dataset.

|          | Backbone | Multi-Queries in Single Inference | 20-NoC@50 | 20-NoC@85 | 20-NoC90 | 1-IoU |
|----------|----------|------------------------------------|-----------|-----------|----------|-------|
| SAM (B)  | SAM-B    | X                                  | /         | 3.28      | 4.12     | /     |
| SAM (L)  | SAM-L    | X                                  | /         | 2.60      | 3.12     | /     |
| SAM (H)  | SAM-H    | X                                  | /         | 2.55      | 3.11     | /     |
| SEEM (B) | Davit-d3 | X                                  | /         | 2.93      | 3.79     | /     |
| SEEM (L) | Davit-d5 | X                                  | /         | 2.77      | 3.61     | /     |
| SEEM (B) | Davit-d3 | Y                                  | 1.37      | 2.71      | 3.43     | 87.2  |
| SEEM (L) | Davit-d5 | Y                                  | 1.38      | 2.70      | 3.47     | 88.5  |
| SEEM (B) | SAM-B    | Y                                  | 1.29      | 2.53      | 3.21     | 86.2  |
| SEEM (L) | SAM-L    | Y                                  | 1.30      | 2.42      | 2.97     | 84.5  |

We found both techniques improves the performance with a good margin. 

**New Experiments:**

* Open-Vocabulary Generic Segmentation on ADE20K dataset in comparison with X-Decoder baselines. (Reviewer cQYZ, R6GZ)

* "Scaling up" with SAM pretrained checkpoint. (Reviewer x17t)

* Enable multiple interactive query in a single forward pass. 

* Ablation Study on gradually removing prompt type. (Reviewer x17t)

We carefully read all comments and attempted to address the concerns with comprehensive responses, and hope our responses could answer the questions and resolve any concerns regarding our work. Again, we thank all reviewers again for their efforts and constructive comments!

Meta-review: All reviewers reached a consensus to accept the paper. The authors are encouraged to incorporate all the feedback and suggestions from the reviewers into the final version of the manuscript.
Decision: Accept (poster)