SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xueyan Zou\*2, Jianwei Yang\*1, Hao Zhang\*#, Feng Li\*#, Linjie Li†, Jianfeng Wang†, Lijuan Wang†, Jianfeng Gao‡†, Yong Jae Lee¶§
$^{\S}$  University of Wisconsin-Madison ‡ Microsoft Research, Redmond ‡ HKUST † Microsoft Cloud & AI
* Equal Contribution ¶ Equal Advisory Contribution 1. Project Lead 2. Main Technical Contribution
{xueyan,yongjaeelee}@cs.wisc.edu {jianwyan,jfgao,linjli}@microsoft.com {hzhangcx,fliay}@connect.ust.hk
Panoptic
Text/Audio
Cross Style
Text+Visual
Figure 1: SEEM supports generic segmentation tasks—including semantic, instance, and panoptic segmentation—in an open-set fashion when no prompt is provided. SEEM also enables the use of visual, textual, and referring region prompts in flexible combinations, making it a promptable and interactive segmentation interface.
No Prompt
Visual Prompts
Ref Prompt
In this work, we present SEEM, a promptable and interactive model for segmenting everything everywhere all at once in an image, as shown in Fig. 1. In SEEM, we propose a novel decoding mechanism that enables diverse prompting for all types of segmentation tasks, aiming at a universal segmentation interface that behaves like large language models (LLMs). More specifically, SEEM is designed with four desiderata: i) Versatility. We introduce a new visual prompt to unify different spatial queries including points, boxes, scribbles and masks, which can further generalize to a different referring image; ii) Compositionality. We learn a joint visual-semantic space between text and visual prompts, which facilitates the dynamic composition of two prompt types required for various segmentation tasks; iii) Interactivity. We further incorporate learnable memory prompts into the decoder to retain segmentation history through mask-guided cross-attention from decoder to image features; and iv) Semantic-awareness. We use a text encoder to encode text queries and mask labels into the same semantic space for open-vocabulary segmentation. We conduct a comprehensive empirical study to validate the effectiveness of SEEM across diverse segmentation tasks. Notably, our single SEEM model achieves competitive performance across interactive segmentation, generic segmentation, referring segmentation, and video object segmentation on 9 datasets with minimum 1/100 supervision. Furthermore, SEEM showcases a remarkable capacity for generalization to novel prompts or their combinations, rendering it a readily universal image segmentation interface.
Image segmentation is arguably the most important yet challenging problem in computer vision. In the past, we have witnessed significant progress in a wide range of segmentation tasks including
instance, semantic and panoptic segmentation [1, 2, 3, 4, 5, 6, 7]. Most recently, we are observing a clear trend toward more flexible segmentation models in different aspects: 1) From closed-set to open-vocabulary segmentation. Many recent works proposed to either leverage contrastive learning methods or pretrained multi-modal foundation models (e.g., CLIP [8]) to make the segmentation models more transferable to unseen concepts [9, 10, 11, 12]; 2) From generic to referring segmentation. In addition to generic segmentation that segments an image thoroughly given a predetermined set of concepts, language-based referring segmentation provides a user-friendly way of segmenting a specific region referred by an arbitrary text phrase [13, 14, 15, 16, 17]; and 3) From one-shot to interactive segmentation. In practice, segmentation models do not necessarily produce satisfactory masks in one round. As such, people are also studying how to progressively refine the segmentation results through intimate interactions between humans and models [18, 19, 20, 21].
Despite the aforementioned efforts taken to design more powerful and feasible segmentation models, we are still lacking a universal segmentation interface that is capable of accommodating various types of human prompts and tackling different segmentation tasks as studied in individual works. In contrast, Large Language Models (LLMs) have already emerged as such a universal interaction interface for language tasks, from early models like GPT-3 [22] and T5 [23], to conversational agent [24] augmented by advanced prompting [25, 26, 27] and chain-of-thought [28, 29, 30]. In this work, we strive for a universal interface for segmenting everything everywhere all at once in an image. On this interface, we are targeted at unifying all segmentation tasks with a single model in a promptable manner. To achieve this goal, we propose a new prompting scheme in mask decoder that has four important properties: versatility, compositionality, interactivity, and semantic-awareness. Specifically, we propose to encode points, masks, text, boxes, and even a referred region from another image into prompts in the same joint visual-semantic space. As such, our model can deal with any combination of the input prompts, leading to strong compositionality. To enable interactivity, we further introduce memory prompts for condensing the previous segmentation information followed by communication with other prompts. As for semantic awareness, our model can provide an open-set semantic label to any output segmentation.
With the proposed prompting scheme, we build a segment-everything-everywhere model called SEEM comprised of a simple Transformer encoder-decoder architecture [31, 6] with an extra text encoder [11, 32]. In SEEM, the decoding process emulates a generative LLM but with a multimodality-in-multimodality-out interface. An image encoder and text encoder are used as the prompt encoder to encode all types of queries, which are fed into the decoder. Concretely, we encode all spatial queries, namely, points, boxes, scribbles and masks into visual prompts by pooling their corresponding visual features from the image encoder, and use the text encoder to convert text queries into text prompts. By training on diverse segmentation tasks, our model learns to deal with various prompts, align the visual and text prompts, and promote their synergy via cross-attention between them. As a result, our single model after pretraining attains competitive performance across all segmentation tasks. Since the prompts of all 5 different types are mapped to the joint visual-semantic space, we can feasibly combine prompts to resolve the ambiguity to obtain better segmentation results and enable zero-shot adaptation to unseen user prompts. Furthermore, our model can immediately generalize to the case of using an exemplar image segment as the prompt and video object segmentation in a zero-shot fashion. In addition to its strong generalization capability, SEEM is also more efficient for interactive segmentation compared with the counterparts like SimpleClick [32]. Since we take the prompts as input to the decoder, when doing multi-round interactions with humans, our model only needs to run the feature extractor once at the beginning and lightweight decoding each per round. To the end, we build a segmentation interface with a single pre-trained model that can segment every object with semantics (everything), cover every pixel in the image (everywhere), and support all possible compositions of prompts (all at once). In summary, our contributions are threefold:
- We design a new prompting scheme that can encode various user intents into prompts in a joint visual-semantic space, enabling strong flexibility for various segmentation tasks and generalization capability to unseen prompts or their combinations.
- We build SEEM, a universal and interactive segmentation interface that integrates the newly designed prompting mechanism into a lightweight decoder for all segmentation tasks, leading to a model possessing properties of versatility, compositionality, interactivity, and semantic awareness.
- We conduct extensive experiments and visualizations to show that our model has strong performance on many segmentation tasks including open-vocabulary generic segmentation, interactive segmentation, referring segmentation, and segmentation tasks with combined prompts.
(a) Model Architecture
(b) Human-Model Interaction
Figure 2: Overview of SEEM- Decoder. (a) SEEM encodes image, text, and human inputs into joint visual-semantic space as queries, features, and prompts, and then decodes queries to class and mask embeddings. (b) With the benefit of SEEM decoder, the machine loop enables memorizing history mask information, and the human loop provides new corrections to the next round.
Interactive segmentation. Interactive segmentation is the task of segmenting objects by interactively taking user inputs. It has been a longstanding problem and has achieved considerable progress [33, 34, 35, 20, 21, 36]. Generally, the interaction types can take various forms, such as clicks, boxes, polygons, and scribbles, among which click-based interaction models are the most prevalent. Concurrent to our work, SAM [36] proposed a promptable segmentation model trained on 11 million images and 1.1 billion masks. It takes user interactions as prompts for general segmentation. Though SAM demonstrates strong zero-shot performance, it produces segmentations without semantic meaning. In addition, its prompt types are limited to points, boxes, and text, whereas our model can also take in a referred region from another image as a prompt.
Generic segmentation. Segmentation of visual concepts has been a persistent challenge in the field of computer vision, as evidenced by its extensive literature [37, 38, 39, 40]. Generic segmentation techniques encompass several subtasks, including instance segmentation, semantic segmentation, and panoptic segmentation [4, 2, 3], each focusing on a different semantic level. For example, semantic segmentation aims to identify and label each pixel within an image based on its corresponding semantic class [41, 6, 42]. On the other hand, instance segmentation involves grouping pixels that belong to the same semantic class into separate object instances [4, 43, 7]. Recently, the Detection Transformer (DETR)[31], a model based on the Transformer [44] architecture, has made significant advances in segmentation [45, 6, 7, 46, 47] tasks. However, these approaches cannot recognize objects absent in the training set, which constrains the model to a limited vocabulary size.
Unified vision models. Unified vision models [11, 48, 49, 36, 50] have recently drawn a lot of attention because of their advantage in generalizing to various tasks and flexibility. These models can deal with multiple vision tasks or data distributions. Among them, some [11, 48, 49] train multiple tasks together with only one model and thus can deal with all training tasks without finetuning on each target task. On the other hand, SAM [36] and SegGPT [50] propose training strategies that enable their models to handle new tasks and data distributions in a zero-shot manner. The second approach is more favorable since there is no need to resolve conflicts among tasks during training.
3.1 Model Design
SEEM employs a generic encoder-decoder architecture but also employs a sophisticated interaction scheme between queries and prompts, as shown in Fig. 2 (a). Given an input image  $\mathbf{I} \in \mathcal{R}^{H \times W \times 3}$ , an image encoder is first used to extract image features  $\mathbf{Z}$ . Then, SEEM-Decoder predicts the masks  $\mathbf{M}$  and semantic concepts  $\mathbf{C}$  based on the query outputs  $\mathbf{O}_h^m$  (mask embeddings) and  $\mathbf{O}_h^c$  (class embeddings), which interact with text, visual, and memory prompts  $\langle \mathbf{P}_t, \mathbf{P}_v, \mathbf{P}_m \rangle$ :
$$
\left\langle \mathbf {O} _ {h} ^ {m}, \mathbf {O} _ {h} ^ {c} \right\rangle = \operatorname {D e c o d e r} \left(\mathbf {Q} _ {h}; \left\langle \mathbf {P} _ {t}, \mathbf {P} _ {v}, \mathbf {P} _ {m} \right\rangle | \mathbf {Z}\right) \tag {1}
$$
$$
\mathbf {M} = \operatorname {M a s k P r e d i c t o r} \left(\mathbf {O} _ {h} ^ {m}\right) \tag {2}
$$
$$
\mathbf {C} = \operatorname {C o n c e p t C l a s s i f i e r} \left(\mathbf {O} _ {h} ^ {c}\right) \tag {3}
$$
(a) Queries and Prompt Interaction
(b) Self-Attention Mask
Figure 3: Queries and prompt interaction during training and evaluation. (a) Learnable queries are duplicated as object, grounding, and visual queries with the same set of weights for each task. (b) Attention mask between any two kinds of tokens (denoted as qpm in Algorithm. 1). Tentative means the interaction is not trained but able to do inference without any modification.
where  $\mathbf{Q}_h$  is the learnable queries, and  $\mathbf{P}_t$ ,  $\mathbf{P}_v$ ,  $\mathbf{P}_m$  represent the text prompts, visual prompts, and memory prompts, respectively. During training,  $\mathbf{Q}_h$  is duplicated for generic, referring, and interactive segmentation, as shown in Fig. 3. The corresponding prompts interact with their queries through self-attention. The learnable queries can freely interact with all prompts at inference time, thereby enabling zero-shot composition. Our design is inspired by the successful practice in X-Decoder [11]. However, we highlight the differences in Eq. (1), marked in red, which allow for a universal model for image segmentation with the following properties:
Versatile. In SEEM, we introduce visual prompts  $\mathbf{P}_v$  to handle all non-textual inputs, such as points, boxes, scribbles, and a referred region from another image. These non-textual queries are beneficial to disambiguate the user's intent when textual prompts alone fail to identify the correct segment. For interactive segmentation, previous works either convert spatial queries to masks and feed them into the image backbone [20] or use different prompt encoders for each input type (points, boxes) [36]. The first approach can be too heavy in applications because each interaction requires the image to go through the feature extractor. The second approach is hard to generalize to unseen prompts. To address these limitations, we propose a visual sampler (Fig. 2 (a)) to convert all kinds of non-textual queries to visual prompts that lie in the same visual embedding space:
$$
\mathbf {P} _ {v} = \operatorname {V i s u a l S a m p l e r} (\mathrm {s}, \hat {\mathbf {Z}}) \tag {4}
$$
where  $\hat{\mathbf{Z}}$  is the feature maps extracted from either the target image (i.e.,  $\hat{\mathbf{Z}} = \mathbf{Z}$ ) or a referred image, and  $s \in \{\text{points}, \text{box}, \text{scribbles}, \text{polygons}\}$  are the sampling locations specified by the user. We first pool the corresponding region from the image feature through point sampling [6]. For all visual prompts, we interpolate at most 512 point feature vectors uniformly from the region specified by the prompt. A notable merit of our proposed method is that the visual prompts are naturally well-aligned with the textual prompts, as our model continuously learns a common visual-semantic space through panoptic and referring segmentation.
Compositional. In practice, a user may cast their intent using different or combined prompt types. Hence, a compositional approach to prompting is essential for real-world applications. However, we confront two issues during model training. First, the training data usually only covers a single type of interaction (e.g., none, textual, visual). Second, although we use visual prompts to unify all non-textual prompts and align them with textual prompts, their embedding spaces remain inherently different. To mitigate this problem, we propose to match prompts of different types with different outputs. Considering that visual prompts  $\mathbf{P}_v$  come from image features while textual prompts  $\mathbf{P}_t$  come from the text encoder, we select matched output indices for visual and textual prompts by matching them with the mask embeddings  $\mathbf{O}_h^m$  or class embeddings  $\mathbf{O}_h^c$ , respectively:
$$
I D _ {v} \leftarrow \operatorname {M a t c h} \left(\mathbf {O} _ {h} ^ {m} \cdot \mathbf {P} _ {v} + \mathbf {I o U} _ {\text {m a s k}}\right) \tag {5}
$$
$$
I D _ {t} \leftarrow \operatorname {M a t c h} \left(\mathbf {O} _ {h} ^ {c} \cdot \mathbf {P} _ {t} + \operatorname {I o U} _ {\text {m a s k}}\right) \tag {6}
$$
where  $\mathbf{IoU}_{mask}$  is the IoU between ground-truth and predicted masks. The proposed separate matching method outperforms approaches that only match with either  $\mathbf{O}_h^m$  or  $\mathbf{O}_h^c$  for all prompts.
After training, our model becomes familiar with all prompt types and supports a variety of compositions, such as no prompts, one prompt type, or both visual and textual prompts using the same model and weights. In particular, the visual and textual prompts can be simply concatenated and fed to SEEM-Decoder, even though it was never trained in this way.
Interactive. Interactive segmentation usually cannot be completed in one shot and requires multiple interaction rounds for refinement, similar to conversational agents like ChatGPT. In SEEM, we
Algorithm 1: Pseudo code for SEEM.
Inputs: Image(img) [B,3,H,W]; Pos_Mask(pm), Neg_Mask(nm) [B,1,H,W]; Text(txt) [abc...]; Variables: Learnable Queries  $(Q_{h})$  ; Attention Masks between  $Q$  and  $P$  (qpm) # Functions:Img Encoder(),Text Encoder(),Visual_Sampler(),feature_attn(),prompt_attn(),output();   
1 definit):  $Q_{o},Q_{t},Q_{v} = Q_{h}.copy();$  # Initialize object, text and visual queries.  $F_{v},P_{t} = \mathrm{Img\_Encoder}(\mathrm{img}),$  Text Encoder(txt);#  $F_{v}$  and  $P_{t}$  denote image feature, text prompt.  $P_{v} =$  Visual_Sampler  $(F_{v}$  , pm, nm);# Sample visual prompt from image feature, pos/neg mask.   
5 defSEEM_Decoder  $(F_{v},Q_{o},Q_{t},Q_{v},P_{v},P_{t},P_{m})$  ..  $Q_{o},Q_{t},Q_{v} =$  feature_attn  $(F_{v},Q_{o},Q_{t},Q_{v})$  ;# Cross attend queries with image features.  $Q_{o},Q_{t},Q_{v} =$  prompt_attn(qpm,  $Q_{o},Q_{t},Q_{v},P_{v},P_{t},P_{m})$  ;# Self attend queries and prompts.  $O_{m},O_{c},P_{m} =$  output  $(F_{v},Q_{o},Q_{t},Q_{v})$  ;# Compute mask and class outputs.   
9 defforward(img,pm,nm,txt):  $F_{v},Q_{o},Q_{t},Q_{v},P_{v},P_{t} =$  init();  $P_{m} =$  None;# Initialize variables. for i in range(max_iter):  $\begin{array}{rl}{\mathsf{\Pi}_{i}}&{=}\end{array}$  10  $O_m,O_c,P_m=$  SEEM_Decoder  $(F_v,Q_o,Q_t,Q_v,P_v,P_t,P_m)$
propose a new type of prompt called memory prompts  $\mathbf{P}_m$  and use them to convey the knowledge of the masks from the previous iteration to the current one. Unlike previous works that use a network to encode the previous mask [20, 36], we introduce no extra module but simply a few memory prompts. These memory prompts encode the history information by using a mask-guided cross-attention layer [6]:
$$
\mathbf {P} _ {m} ^ {l} = \operatorname {M a s k e d C r o s s A t t} \left(\mathbf {P} _ {m} ^ {l - 1}; \mathbf {M} _ {p} | \mathbf {Z}\right) \tag {7}
$$
where  $\mathbf{M}_p$  is the previous mask, and  $\mathbf{Z}$  is the image feature map. In this way, cross-attention only takes effect inside the regions specified by the previous mask. The updated memory prompts  $\mathbf{P}_m^l$  then interact with the other prompts via self-attention to convey the historical information for the current round.
Semantic-aware. Different from previous class-agnostic interactive segmentation works such as Simple Click [20] and the concurrent work SAM [36], our model produces semantic labels to masks for all kinds of prompt combinations in a zero-shot manner, since our visual prompt features are aligned with textual features in a joint visual-semantic space. As shown in Fig. 3, semantic labels are directly computed using  $\mathbf{O}_h^c$  (output of visual queries) and the text embedding. Although we do not train with any semantic labels for interactive segmentation, the calculated logits are well-aligned, benefiting from the joint visual-semantic space.
3.2 Model Pipeline and Loss Functions
We summarize the training and evaluation pipeline of the proposed method with Pytorch-style pseudocode in Algorithm 1. SEEM is trained with a linear combination of losses for panoptic segmentation, referring segmentation, and interactive segmentation:
$$
\begin{array}{l} \mathcal {L} = \alpha \mathcal {L} _ {\mathrm {c} - \mathrm {C E} - \mathrm {p a n o}} + \beta \mathcal {L} _ {\mathrm {m} - \mathrm {B C E} - \mathrm {p a n o}} + \gamma \mathcal {L} _ {\mathrm {m} - \mathrm {D I C E} - \mathrm {p a n o}} + a \mathcal {L} _ {\mathrm {c} - \mathrm {C E} - \mathrm {r e f}} + b \mathcal {L} _ {\mathrm {m} - \mathrm {B C E} - \mathrm {r e f}} \\ + c \mathcal {L} _ {\mathrm {m} - \mathrm {D I C E} _ {-} \text {r e f}} + a \mathcal {L} _ {\mathrm {c} - \mathrm {C E} _ {-} \text {i s e g}} + b \mathcal {L} _ {\mathrm {m} - \mathrm {B C E} _ {-} \text {i s e g}} + c \mathcal {L} _ {\mathrm {m} - \mathrm {D I C E} _ {-} \text {i s e g}} \tag {8} \\ \end{array}
$$
Where  $\alpha = 2, \beta = \gamma = 5, a = 0.2, b = c = 2$ , CE, BCE, and DICE denotes cross-entropy, binary cross entropy and dice loss, respectively.
Datasets and Settings. SEEM is trained on three tasks: panoptic segmentation, referring segmentation, and interactive segmentation. Panoptic and interactive segmentation are trained on COCO2017 [51] with panoptic segmentation annotations. Following [11], we exclude the validation set of Ref-COCOg [52], resulting in 107K segmentation images in total. For referring segmentation, we use a combination of Ref-COCO, Ref-COCOg, and Ref-COCO+ for COCO image annotations. We evaluate generic segmentation (instance/panoptic/semantic), referring segmentation, and interactive segmentation.
Implementation Details and Evaluation Metrics. Our model framework follows X-Decoder [11] except the decoder. That is, we have a vision backbone, a language backbone, an encoder, and