A Simple Image Segmentation Framework via In-Context Examples
Authors: Yang Liu
Chenchen Jing
Hengtao Li
Muzhi Zhu
Hao Chen
Xinlong Wang
Chunhua Shen
Venue: NeurIPS 2024 poster (2024)
Keywords: Image Segmentation; In-Context Learning; Task Ambiguity
Primary area: machine_vision

Abstract: Recently, there have been explorations of generalist segmentation models that can effectively tackle a variety of image segmentation tasks within a unified in-context learning framework. However, these methods still struggle with task ambiguity in in-context segmentation, as not all in-context examples can accurately convey the task information. In order to address this issue, we present SINE, a simple image $\textbf{S}$egmentation framework utilizing $\textbf{in}$-context $\textbf{e}$xamples. Our approach leverages a Transformer encoder-decoder structure, where the encoder provides high-quality image representations, and the decoder is designed to yield multiple task-specific output masks to eliminate task ambiguity effectively. Specifically, we introduce an In-context Interaction module to complement in-context information and produce correlations between the target image and the in-context example and a Matching Transformer that uses fixed matching and a Hungarian algorithm to eliminate differences between different tasks. In addition, we have further perfected the current evaluation system for in-context image segmentation, aiming to facilitate a holistic appraisal of these models. Experiments on various segmentation tasks show the effectiveness of the proposed method.

A Simple Image Segmentation Framework via In-Context Examples
Yang Liu',
Chenchen Jing',
Hengtao Li', Muzhi Zhu' Xinlong Wang', Chunhua Shen
Hao Chen';
'Zhejiang University, China
Ant Group
BBeijing Academy of Artificial Intelligence
Abstract
Recently, there have been explorations of generalist segmentation models that can effectively tackle variety of image segmentation tasks within unified in-context learning framework. However, these methods still struggle with task ambiguity in-context segmentation, as not all in-context examples can accurately convey the task information. order to address this issue, we present SINE, a simple image Segmentation framework utilizing in-context examples. Our approach leverages Transformer encoder-decoder structure, where the encoder provides high-quality image representations, and the decoder is designed to yield multiple task-specific output masks to eliminate task ambiguity effectively. Specifically, we introduce an In-context Interaction module complement in-context information and produce correlations between the target image and the in-context example and a Matching Transformer that uses fixed matching and Hungarian algorithm to eliminate differences between different tasks. In addition, we have further perfected the current evaluation system for in-context image segmentation, aiming facilitate a holistic appraisal of these models. Experiments on various segmentation tasks show the effectiveness of the proposed method.
Our code is released https: https:///gtthti /ighttt........ com hatmaaimmuoaa//I/W
Introduction
Image segmentation [65, 31, 24, 58, 46] involves localizing and organizing concepts at the pixel level. Different definitions of concepts, such foreground, category, and object instance, lead different types of segmentation tasks. Recent years, we have witnessed great progress in developing more accurate and faster algorithms for various segmentation tasks such as semantic segmentation [35, 64, 48], instance segmentation [19, 7, 2], panoptic segmentation [24, 4, 51, foreground segmentation [52], interactive segmentation [57, 37, 25]. Nonetheless, most existing segmentation methods are tailored for certain tasks and cannot be applied to other tasks.
More recently, few works [54, 56] have explored generalist segmentation models that capable of solving diverse and unlimited segmentation tasks via in-context learning [3]. Painter [54] performs in-context training with masked image modeling and achieve various tasks according to the in-context visual prompts. SegGPT [56] focuses on visual segmentation and introduces in-context segmentation, which unifies multiple segmentation tasks by incorporating both target image and an annotated reference image as input. To encourage the model to leverage contextual information, SegGPT employs a random coloring scheme during the task completion process. However, these models still struggle with task ambiguity in in-context segmentation.
As shown in Figure 1(a), the in-context segmentation model needs understand the task and content information conveyed by the in-context example and segments related concepts on the target image.
HC is the corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Idteniical object onginentation
00
narraric +D
ecomplee
Transfomerr Emooder
Transfomeer Docoder
eegmencation
SINE
aegresttation
target
ccample
images
predictiona
prodkdiers
Ambiguity traditional in-context segmentation framework. (b)
in-context segmentation framework
Figure 1 Illustration of ambiguity in traditional in-context segmentation framework and overview of SINE framework.
However, not all in-context examples convey the task information accurately. For example, when presented with photo of particular individual for segmenting the target image that includes him/her and others, which visual concepts should be segmented? Specifically, should be limited to the individual alone, encompass the instance segmentation of all persons, or focus on semantic segmentation? Whether the pixels of the target image need be segmented depends on their similarity to the in-context example. Ambiguous in in-context examples can make it difficult for traditional in- context segmentation models clearly define the boundaries between different tasks, resulting in undesired outputs.
To address this issue, present SINE, simple image Segmentation framework via IN-context Examples. Drawing inspiration from the segment anything model [25], which addresses point ambiguity by generating multiple masks of different granularity, SINE predicts multiple output masks, custom-made for tasks of varying complexity. This ranges from identical objects, instances to overall semantic conception. SINE unifies existing segmentation tasks with various granularities, intending to achieve a broader task generalization. SINE leverages Transformer encoder-decoder structure. The encoder contains a frozen pre-trained image encoder offering high-quality image representations and an In-context Interaction module to complement in in-context information and learn correlations between the target and the reference image features. We propose a novel Matching Transformer (M- Former) for efficient multiple-task decoding. M-Former is implemented by a dual-path Transformer. One path is used for information interaction between object queries with image features. The second path is employed to enhance the semantic prototypes for accurate matching. In addition, a fixed matching and Hungarian algorithm are used to climinate differences between different tasks.
SINE achieves impressive performance with fewer trainable parameters compared with SegGPT. Our qualitative results demonstrate that SINE can effectively address the problem of task ambiguity in in-context segmentation, while SegGPT only outputs the semantic segmentation results. SINE achieves wate-of-the-ar art or competitive performance on existing in-context image segmentation benchmarks [56]. including few-shot semantic segmentation, video object segmentation. In addition, we further introduce few-shot instance segmentation to the current evaluation system for facilitating holistic appraisal of these models. SINE provides baseline result for in-context segmentation models to promote the development of this field. Finally, comprehensive ablation studies verify the effectiveness of the proposed components. Our main contributions are as follows:
To our knowledge, our method is the first to investigate the task ambiguity of in-context segmentation, and we present simple but effective framework to address the issue.
We introduce Matching Transformer to unleash the potential of frozen pre-trained image models on various segmentation tasks with a low training cost.
Our comprehensive results demonstrate that SINE can address a broad range of segmentation tasks, including few-shot semantic segmentation, few-shot instance segmentation, video object segmentation. Ablation studies show the effectiveness the proposed components.
1.1 Related Work
Image Segmentation Segmentation involves localizing and organizing meaningful concepts the pixel level. Different definitions of concepts, such as foreground, category, and object instance, lead to different types of segmentation tasks. Specifically, semantic segmentation [65] requires

---

semantic classification at the pixel level, while instance segmentation [31] is to identify and localize various object instances. Panoptic Segmentation [24] introduces more challenging task by unifying semantic segmentation and instance segmentation. Video object segmentation [58, 46] is to segment an identical object throughout a video sequence. Most existing segmentation methods are tailored for certain tasks and cannot be applied to other tasks.
Traditional segmentation methods [35, 29, 64, 48, 19, 7, 2, 24, 4, 5, 10, 66] have been designed for identical segmentation tasks and trained on a specific dataset. These methods have limited generalization when transferring to other datasets or tasks. To solve this challenge, we unify various segmentation tasks into in-context segmentation [56, 34, 27, 39] and introduces a simple visual segmentation framework via in-context examples. Prompted by a given in-context example, our method can perform few-shot segmentation across various datasets and tasks.
In-Context Visual Learning Recently, in-context learning has been successfully used in NLP tasks and vision-and- -nn--aaaag tasks. Bar et al. [1] first introduced in-context learning in computer vision. They cast the in-context learning as an image inpainting problem, where models need to fill in hole in concatenated image containing several examples and a new input image. They demonstrate the effectiveness of the strategy for foreground segmentation, single object detection, and colorization. Painter [54] adopts masked image modeling on continuous pixels to perform in-context training with supervised datasets, and achieves competitive results on seven vision tasks. SegGPT [56] focuses on the segmentation task and uses random coloring scheme that forces the model to reference contextual information to complete the assigned task. DiffewS [67] effectively leverages in-context visual learning to unleash the potential of Stable Diffusion [49] in few-shot semantic segmentation.
Our work focuses on segmentation tasks. These aforementioned models struggle with the task ambiguity in in-context segmentation. By contrast, our SINE can effectively address the problem of task ambiguity in in-context segmentation by disentangling the specific task from the in-context example and understanding the semantic concepts of the prompts to output results at different levels of task granularity from the identical object, instance, semantics.
2 Preliminary
In this section, first formulate the problem setting of in-context segmentation. Then, revisit SegGPT, the previous in-context segmentation model.
2.1 Problem Formulation
In-context segmentation aims to identify specific task and objects within the given in-context example, including a reference image x, and its annotations y, = {(m). c )) and segment the interested objects in the target image x,. m denotes the i-th reference mask and of denotes its class label, N denotes the number of the reference masks. The interested objects are related to the in-context example, which can be an identical object for video object segmentation all objects of the same semantic concept for instance segmentation and semantic segmentation. We mainly focus on three semantic granularities in the task ambiguity, i.e., identical object (ID), instance, and semantic. The identical object segmentation can be seen as finer granularity instance segmentation, and instance segmentation can be transformed into semantic segmentation by merging instance masks belonging to the same category. Based on this relationship between these tasks, we unify them into instance segmentation.
2.2 A Revisit of SegGPT
SegGPT [56] introduces in-context segmentation, which incorporates both the images and mask annotations into an RGB image to convey the specific task to be performed and identify the objects to be segmented simultaneously. SegGPT takes stitched reference and target images as input and employs the masked image modeling algorithm [18] and smooth-f1 [15] loss train a Vision Transformer [8] encoder. During inference, SegGPT enables segmenting everything via an in-context example, including a reference image and its annotation image. Given a target image,
stitched with the reference image and fed into SegGPT to get the corresponding in-context predictions. Although SegGPT has achieved great success in various segmentation tasks, it faces two challenges: 1) In many cases, in-context examples may not accurately convey the task information. For example,
3

---

Transfomer Encoder
Transformer Decoder
Fixed Matchiig
Bipertite Matching
Lose
Image
Mask Pooling
Forward
image masika
semarte
Cross Allertion
Far
Salf Colf-Ationionn Mask
Sell Allention
Shared
Sell Allention
outputs
Image
In-Context Fusion
Encoder
Od
Gie
Puss &R
quefies
leamee insilance
quarles
semantic protohypes
In-Context Interaction
M-Former
identical
object
Instanca
somantic
Figure An overview SINE, SINE Transformer encoder-decoded structure, including frozen pre-trained image encoder, an In-Context Interaction module, and a lightweight Matching Transformer (M-Former) decoder. The top right corner is the self-attention mask of M-Former. The down right corner shows the different task outputs, from identical object, instance, semantic.
when in-context example only consists of single object and its annotation, the lack of additional task-related information can lead to incorrect outputs. 2) SegGPT utilizes single Transformer encoder structure for both feature extraction and task-specific decoding purposes. This will introduce complexity in the in-context segmentation process and lead to sub-optimal solution.
Inspired by the segment anything model [25]. which solves point ambiguity by outputting multiple different granularity masks simultaneously, we focus solely on the various content of in-context examples and endow the model with the ability to predict multiple output masks for different tasks. In addition, we disentangle the function of the Transformer encoder within SegGPT and deploy Transformer encoder and decoder to perform feature extraction and task decoding, respectively.
Method
We present SINE, a simple image Segmentation framework via IN-context Examples. SINE can effectively address the problem of task ambiguity in in-context segmentation by disentangling the specific task from the in-context example and understanding the semantic concepts of the prompts to output results at different levels of task granularity from the identical object, instance, semantics. We elaborate on our method in the following subsections.
3.1 Overview
The overview of the proposed SINE framework is depicted in Figure 2. We build SINE based on the classic Transformer structure [53, 8] and introduce some effective designs targeted for the in-context segmentation task, including a frozen pre-trained image encoder, an In-Context Interaction module, and lightweight Matching Transformer (M-Former) decoder.
We use the frozen pre-trained image encoder to encode x, and x,, resulting in the reference feature F, E Cx and target featurel', E Cx X where H. W. and C denote the image features' height, width and the number of channels, respectively. Inspired by the in-context learning in NLP [3], enable SINE to grasp the in-context correlations that coexist between the reference and the target images. Specifically, we develop an In-Context Interaction module, a component tailored to capture the semantic correlations between F, and F, and outputs the enhanced target feature F HC the ID queries 4:a E R NCC and the semantic prototypes Prem E M MC
SINE is built as query-based segmentation model, following DETR [4] and Mask2Former [5]. We employ the ID queries qıa to identify and locate objects within the target image that have identical counterparts within the reference image. Furthermore,
apply learnable instance queries aess IR. SxC to identify and locate objects within the target image that share the same semantic labels in the reference image. The M-Fomer decoder is employed to update these queries and

---

Mask Pooling
4id B NKC
In-Context Fusion
Psem B
mid MXHRK
Mmm RR MocRo
Feed Forward
Feed Forward
Merge
Cross Attention
Cross Attention
ID masks
semantic masks
Ft
Self Attention
Self Attention
Fid Eeem
shared
Pooling
Ei
Eg CCHNw
MMC
Figure 3 Illustration of the In-Context Interaction module. This module aims to complement in-context information between reference and target. The ID and semantic tokens extracted by the Mask Pooling. The enhanced target feature, the ID queries, the semantic prototypes are outputted by the In-Context Fusion module.
prototypes. Then, a prediction feed-forward network [4] is employed to predict ID and instance outputs, respectively.
3.2 In-Context Interaction
The purpose In-Context Interaction is to complement in-context information and produce correla- tions between reference and target image features. As illustrated in Figure 3. we use a Mask Pooling process to extract the ID and semantic tokens, represented as tu E IR N and 2eem R Mxx Specifically, transfer m, into ID masks mad E x x by assigning different ID labels for each mask and semantic masks M sem E xxMxXWW by merging the masks with same category label, where and M are the numbers of ID and semantic masks. Then, we use these masks to pool over the reference feature F, and obtain fa and fem, the pooling process similar to [14].
We introduce In-Context Fusion module to enable the in-context correlation between the reference and target features. The process of this module can be summarized as follows:
Mid P Psem
InConteztFussin (Lid, teem Fr;
(1)
where 0 is the parameters of the In-Context Fusion. The module is a Transformer block [53, 4] including a self-attention, cross-attention, and a feed-forward network. The tokens (t,d and feom) and target feature (F,) are fused by this shared module, where they are used keys and values for each other in the cross-attention, and the enhanced target feature F,, the ID queries qia and the semantic prototypes P Peem can be obtained.
3.3 Matching Transformer
M-Former aims decode the enhanced target feature into different task outputs, from identical object, instance, to semantic, and enables comprehensive and efficient performance for in-context segmentation. Thus, in addition to the ID queries qua and the semantic prototypes Psem also incorporate a set of learnable instance queries qine for predicting instances. To perform in-context segmentation and eliminate task ambiguity effectively, the design of M-Former needs consider 1) the semantic prototypes need to assign semantic information to the instance queries, 2) avoiding coarse- coarse-grained semantic prototypes to contaminate fine-grained ID queries, and the queries interact with the target feature.
Encouraged by the above analyses, the M-Former is implemented by alual-path Transformer decoder sharing the self-attention layers. One path is utilized to interact F; and queries (q and qins) for extracting correlated feature with the in-context example from the target image. This path consists of a series of self-attention, masked cross-attention [5], and feed-forward network. The second path employed to enhance the semantic prototypes Psem for more accurate matching. The two paths

---

idemrical object
SemAniC
reforencc o0
target
SegGPT
SINE
reforence
prodictions
(d)
predictinn
Figure Qualitative results SINE. (a) Comparison between SegGPT and SINE for addressing ambiguity in in-context segmentation. (b) Few-shot semantic segmentation. (c) Few-shot instance segmentation. (d) Video object segmentation.
share the self-attention layers for assign semantics from Psem to qins To avoid semantic prototypes contaminating fine grained ID queries, we apply a self-attention mask within the shared self-attentio layers as shown in the top right corner of Figure 2. M-Former has N blocks. The process of M-Former can be summarized as follows:
Cid Vins Premm) Former 1-1 1-1
qins P 0-, D.,FF).
Qid
where denotes the layer number of M-Former and 0 is the parameters of the /-th M-Former block. Then, a prediction feed-forward network employed to predict ID and instance outputs, respectively,
For instance segmentation, we use the updated semantic prototypes Pzem as the classifier and let Yins { J denote the set of S instance predictions. The ground truth is denoted by We use the Hungarian loss [4, 5] to learn SINE. Specifically, we compute the assignment costs between prediction Vn and ground truth y² for the matching problem, i.e., -P((c') where (cl m)' is the class and mask of the ground truth object, e² may be Ø. pr px(ce) is the probability of class e² for i-th instance query, r denotes its predicted mask. Lmaak is binary mask loss and Dice loss [40]. The Hungarian loss is
L ( m²),
Σ j= [- log
I of Cmaok (Mo m²)
Lluungrria inn
where of)) denotes the resulting index of the bipartite matching. To endow SINE with the ability to predict identical object,
the different cropped views of the same instance within the image as reference-targee image pairs. Let Y.a = (Mt 1N denote the set of ID predictions. Because the relationship between reference and target IDs is fixed and can be accurately determined, perform fixed matching between the predictions and the ground truth. The loss can be written as
NV LiD(faa))) Σn 1=1 [ (olgge( ee ( m m²)]
(4)
where (c', m²) is the ground truth class and mask, et E (1, 2), ) = denotes that an object appears in both reference and target images simultaneously. The total loss is L LHungarian + LiD- Once the training finished, the full capability of SINE is unleashed during inference. SINE can address the ambiguity within the in in-context examples and output predictions for different segmentation tasks.

---

COCO-20'
PASCAL-5'
LVIS-92'
Methods
Venue
one-shot
few-shot
one-shot
few-shot
one-shot
few-shot
specialist model
HSNet [41]
ICCV'21
41.7"
50.7*
68.7"
73.8"
17.4
22.9
VAT [20]
ECCV"22
42.9*
49.4"
72.4"
76.3"
18.5
22.7
FPTrans [62]
NeurlPS'22
56.5*
65.5*
77.7*
83.2"
gemeralist model
Painter [54]
CVPR'23 23
32.8
32.6
64.5
64.6
10.5
10.9
SegGPT [56]
ICCV'23
56.1
67.9
83.2
89.8
18.6
25.4
PerSAM-F [63]
ICLR'24
23.5
18.4
Matcher [34]
ICL.R'24
52.7
60.7
33.0
40.0
SINE
this work
64.5
66.1
85.4
86.2
31.2
35.5
Table 1 Results few-shot semantic segmentation COCO-20', PASCAL-S', LVIS-92'. Gray indicates the model is trained by in-domain datasets. * indicates that the categories in training cover the categories in testing within the same dataset.
4 Experiments
Training Data We train our model with a diverse set of segmentation datasets, including semantic, in- stance, and panoptic segmentation. Specifically, we utilize three visual perception datasets: ADE20K [65] is popular semantic segmentation dataset, providing semantic labels for 150 categories. For panoptic segmentation, the 150 categories can be divided into 100 "things" and 50 "stuff" categories. has 25K images, including 20K for training. 2K for validation, and 3K for testing. COCO [31] is widely-used dataset that supports object detection, instance segmentation, and panoptic segmentation. contains 80 "things" and 53 "stuff" categories, with 118K training and 5K validation images. Objects365 [51] is a large-scale high-quality object detection dataset. It contains 365 categories, 638K images, and 10M bounding boxes. We extend instance segmentation annotations for Objects365 by using the Segment Anything Model [25]. We convert different data annotations into the form of instance segmentation for unified mixed data training.
Training Details Following [34], we deploy the frozen DINOv2 (ViT-L) [44] with 304M parameters as the image encoder and train SINE with only 19M trainable parameters. The In-Context Fusion has one block and M-Former has six blocks. The model size of SINE is comparable with SegGPT (307M). SINE has fewer trainable parameters, leading to more efficient training. We train SINE about 50K steps with 64 batch sizes. We Adam [36] optimizer and employ 31 0.9, 82 0.999 for optimization. We use a linear learning rate scheduler with a base learning rate of le-4 and a warmup of 100 steps. The weight decay is set 0.05. For data augmentation, we use random horizontal flipping and the large-scale jittering (LSJ) [13] augmentation with a random scale sampled from range 0.1 to 2.0 followed by fixed size crop 896 896. More implementation details provided in the Appendix B.
4.1 Qualitative Results
We demonstrate that our SINE framework can effectively address the ambiguity in the in-context examples. As shown in Figure 4(a), difficult to understand which tasks should be executed via the reference image with annotation of particular Indivddull.SeGPP only outputs the semantic segmentation result. In contrast, SINE outputs multiple outputs to avoid task ambiguity. We further visualize the results of SINE on few-shot semantic segmentation, few-shot instance segmentation, and video object segmentation. SINE showcases its capability to deliver highly accurate predictions across diverse tasks while retaining exceptional flexibility in the task definition.
4.2 Few-shot Semantic Segmentation
Datasets We revisit the few-shot semantic segmentation task into two settings: in-domain (COCO-20* [43]) and out-domain (PASCAL-5* [50], LVIS-92¹ [34]). based on whether the dataset has been seen during training. COCO-20 divides the 80 classes of COCO into four cross-validation folds with 20 test classes and 60 training classes for each fold. Similarly, PASCAL-5* is built on PASCAL, including four cross-validation folds. LVIS-92 a more challenging benchmark for evaluating the generalization of model across datasets based on LVIS [16], including ten folds and 92 classes for each fold. We comprehensively verify the fhw-shot semantic segmentation performance of SINE on these datasets following the evaluation scheme of [41, 56. 34].

---

Methods
Venue
Det.
Segm.
Det.
Segm.
Methods
Venue
AP
AP50
AP50
specialiat model
Meta R-CNN [59]
ICCV'19
3.5
2.8
FGN [11]
CVPR'20
30.8
16.2
MTFA [12]
2.5
6.6
2.7
6.6
IMIFA
CVPR'21
3.3
6.2
2.8
5.2
MIFA (12)
CVPR'21
10.0
21.7
19.3
DETR [61]
TPAMU22
iMIFA
11.5
22.4
8.6
16.3
RefT [17]
arXiv'23
15.0
14.2
SINE
this work
35.9
51.9
27.6
47.6
3-shor
generalist model
SINE
this work
18.0
21.8
SINE
this work
42.8
62.1
33.3
57.7
Table 3 Results (AP and AP50) of few-shot object detection instance segmentation on COCO2VOC with K {1,5}.
Table Results (AP) few-shot object de- tection and instance segmentation on COCO-
NOVEL with K {1,5}.
Results As shown in Table 1, compare the SINE with a variety of specialist and generalist segmentation models. For COCO-20' and PASCAL-5', because SINE is trained with all data of COCO, report the performance of specialist models trained in-domain categories (marked by for fair comparison. Although SINE and SegGPT are trained on the COCO dataset, SINE achieves significant advantages in one-shot performance with simpler in-context learning framework on COCO-20'. In addition, without specific training on the PASCAL dataset, SINE achieves better performance with one-shot and comparable performance with few-shot SegGPT on PASCAL-5' On the more challenging dataset LVIS-92', SINE outperforms Painter, SegGPT, and PerSAM-F, demonstrating superior generalization and versatility. The performance of SINE on LVIS-92 is slightly weaker compared to Matcher, which benefits from the SAM [25] pre-trained on a large-scale segmentation dataset. SINE achieves competitive results independently, without relying on extensive segmentation data.
Please note that SINE aims provide insights for the research community build simple baseline for in-context segmentation, instead of SOTA. Despite having been trained on 12 diverse segmentation datasets [56]. SegGPT demonstrates a limited performance when applied to LVIS-92'. This indicates a need for developing more effective data-centric learning approaches specifically tailored for in- context segmentation. SINE the first to explore the utilization of Objects365 with automatic mask annotations. The efficient design of SINE enables the training on Objects365 at the instance level. This enables SINE to only use 19M training parameters, making the training budget much smaller than SegGPT (304M) SAM (600M) in Matcher.
4.3 Few-shot Instance Segmentation
Datasets Like few-shot semantic segmentation, evaluate the performance of SINE of few-shot instance segmentation on both settings: in-domain on COCO-NOVEL [23] and out-domain on COCO2VOC [9] settings. COCO-NOVEL includes 20 classes that intersect with VOC and 5k test images. We evaluate 1-shot and 5-shot instance segmentation performance on this dataset. The COCO2VOC dataset is used evaluate the cross-dataset generalization ability by using reference samples of COCO to test the VOC test set. We report the mean results of 10 groups of different reference images generated by different random seed for both two datasets.
Results Table shows the results of few-shot object detection and instance segmentation. SINE outperforms specialist methods by large margin both 1-shot and 5-shot settings. For the results of COCO2VOC in Table 3, SINE shows better generalization ability compared to specialist models.
Please note that the goal of this experiment is not to prove that SINE achieves better few-shot instance segmentation performance compared to specialist segmentation models. The existing in-context segmentation models, such as SegGPT, are unsuitable to perform instance segmentation. SegGPT needs to use a sliding window to traverse all grids to predict the objects. The post-processing is complicated and ineffective. SINE is the first in-context segmentation model that address few- shot instance segmentation. We hope SINE can be a baseline for in- mn-oonexxt segmentation models to promote the development of this field.
4.4 Video Object Segmentation
Datasets For video object segmentation (VOS), we focus on the semi-supervised setting where the object masks of the first frame provided, and the model is responsible for segmenting the object
8

---

Methods
Venue
DAVIS 2017
DAVIS 2016
YouTube- VOS 2018
J&
J
with video
AGAME [22]
CVPR119
70.0
67.2
72.7
66.0
66.9
61.2
AGSS
ICCV'19
67.4
64.9
69.9
71.3
71.3
65.5
75.2
73.1
AFB-URR [28]
NeurlPS 20
74.6
73.0
76.
79.6
78.8
83.1
74.1
82.6
AOT [60]
NeurlPS
85.4
82.4
88.4
92.0
90.7
93.3
84.5
84.3
89.3
77.9
86.4
SWEM [32]
CVPR'22
84.3
81.2
87.4
91.3
89.9
92.6
82.8
82.4
86.9
77.1
85.0
XMem [6]
ECCV
87.7
84.0
91.4
92.0
90.7
93.2
86.1
85.1
89.8
80.3
89.2
wnihou video data
Painter [54]
CVPR'23
34.6
28.5
40.8
70.3
69.6
70.9
24.1
27.6
35.8
14.3
18.7
SogGPT [56]
ICCV"23
75.6
72.5
78.6
83.7
83.6
83.8
74.7
75.
80.2
67.4
75.9
SEEM [68]
NeurIPS-23
58.9
55.0
62.8
50.0
57.2
38.2
61.3
43.3
DINOv [27]
CVPR 24
73.3
71.0
75.7
60.9
65.3
70.0
$2.3
$7.9
PerSAM-F [63]
ICLR'24
76.1
74,9
79.7
54.4
53.9
56.4
50.7
56.6
Matcher [34)
ICLR'24
79.5
76.5
82.6
86.1
85.2
86.7
SINE
this work
77.0
72.6
81.3
82.3
81.4
83.2
66.2
69.1
57.6
71.7
66.5
Table 4 Results of video object segmentation on DAVIS 2017. DAVIS 2016, and YouTube-VOS 2018.
Gray indicates the model is trained target datasets with video data. G is the average over the "seen" and "unseen" classes in YouTuhe- VOSe-VOS 2018.
Methods
Venue
mloU
Methods
Venue
PQ
PQT
PQST
specialist mode!
specialist model
FCN [35]
CVPR115
29.4
PanopticFPN [24] CVPR'19
40.8
48.3
29.4
RefineNet [29]
CVPR'17
40.7
SOLOv2 [55]
NeurlPS'20 20
42.1
49.6
30.7
DPT [48]
ICCV'21
49.2
Mask2Former [5]
CVPR'22
57.8
64.2
48.1
2ask2Former [5]
CVPR122
57.7
UVIM [26]
NeurlPS
45.8
generalist model
generalist model
Painter [54]
CVPR'23 23
49.9
Painter [54]
CVPR'23
43.4
SegGPT [56]
ICCV'23
39.6
SegGPT [56]
ICCY'23
34.4
SINE
this work
54.1
SINE
this work
51.0
57.8 40.8
Table 5 Transfer performance ADE20K
Table 6 Transfer performance COCO
semantic segmentation.
panoptic segmentation.
in all subsequent frames. We evaluate SINE on three validation datasets, including DAVIS 2017 [46]. DAVIS 2016 [45], and YouTube-VOSS 2018 [58]. We use the J score and the F score to evaluate the performance.
Details To effectively perform SINE on the VOS task, we introduce a group of memory banks for each object, maintaining the intermediate predictions. We determine which frame to retain in the memory according the classification and mask scores. Considering that objects to resemble those in adjacent frames, we apply a time-based decay ratio to the scores, gradually reducing its value. In addition, we store the reference image and mask in memory to solve the case where objects vanish and reappear.
more likely
Results Table 4 compares the performance of VOS between SINE and different methods trained with or without video data. Without video data, SINE achieves competitive performance compared with the models trained with video data on DAVIS 2017. In addition, SINE outperforms recent generalist segmentation methods, such as Painter, SEEM, DINOv and PerSAM-F, on YouTube-VOS 2018. These results demonstrate that SINE has the potential to address video tasks.
4.5 Transfer Learning Experiments
We investigate the performance of SINE when transferring common segmentation tasks, such as ADE20K semantic segmentation and COCO panoptic segmentation. Unlike traditional pre-trained methods, e.g., MAE [18], require to fine-tune all model parameters to downstream tasks. We verify that our method can be efficiently transferred to these tasks via parameter efficient fine-tuning (PEFT) methods [38]. Specifically, We deploy the popular PEFT method, LoRA [21], on the frozen image encoder and fix the original parameters to test the transfer ability of SINE. We select rank 32 as the default setting, We train semantic prototypes and LoRA parameters for specific segmentation tasks.
Semantic Segmentation According Table 5, our method SINE achieves 54.1 % mloU on the ADE20K semantic segmentation benchmark, outperforming other in-context segmentation models like SegGPT which trains a dataset-specific prompt using related dataset annotations. In addition, with only a few trainable parameters, our method achieves better comparable performance to specialist segmentation models. It is worth noting that, unlike 2ask2Former [5], SINE does not use multi-scale features for better performance.

---

LVIS-92'
COCO-NOVEL
DAVIS 2017
DAVIS 2017
Training Data
Frames
meaan mloU
APaee
APmost
J&PF
10
COCO
24.4
10.5
11.4
63.2
JEF
70.9
76.2
76.7
77.0
77.0
76.0
ADE20K
25.5
19.2
18.2
68.8
66.1
71.5
71.9
72.6
72.7
71.6
Objects365
28.3
22.4
21.5
77.0
P
75.8
81.0
81.4
81.3
81.3
80.3
(a) Ablation study of training data.
(b) Effect of the number of frames for VOS.
LVIS-92'
COCO-NOVEL DAVIS 2017
Rank
#Params(M)
mloU mACC.
Decoder
mean mloU
APuuz APmasi
J&cP
4
0.63
49.1
63.6
Mask2Fomer
61.1
8
1.22
50.4
65.0
16
2.40
52.4
66.7
Mask2Fomer
10.6
62.3
32
4.75
54.1
68.0
M M-Former
24.4
10.5
11.4
63.2
64
9.48
54.4
68.8
(c) Effect proposed components.
(d) Ablation study of LoRA rank on ADE20K.
Table 7 Ablation study. We report the mloU of fold0 on AVIS-92', APooo and AP APaas of seed0 on COCO-NOVEL, and J&F on DAVIS 2017. Pink is the default setting.
Panoptic Segmentation Table shows that SINE also significantly outperforms other generalist seg- mentation models the COCO panoptic segmentation task, demonstrating that the proposed method can enable the learned Transformer decoder to apply to more complicated panoptic segmentation effectively. By fixing the overall model parameters and adding only a few of LoRA parameters, SINE achieves competitive performance with the best specialist segmentation model, Mask2Former.
4.6 Ablation Study
As shown in Table we conduct ablation experiments LVIS-92' for few-shot semantic segmen- tation, COCO-NOVEL dataset for few-shot instance segmentation, DAVIS 2017 for video object segmentation, and ADE20K for the semantic segmentation task, thoroughly validate the effective- ness of the proposed components and the impact of training with different datasets. Unless specified, only fold0 and seedO are used for the ablation on the few-shot semantic/instance segmentation.
Effect of different training data. We train SINE on different data sources (Table 7a). We validate that incorporating more diverse semantic segmentation data, such ADE20K, helps improve the model. Furthermore, we demonstrate for the first time that adding additional detection data, such as Object365, greatly enhances the model's in-context segmentation capability.
Ablation study of proposed components. Table 7c shows the impact of the proposed components and all experiments only use COCO training data. The proposed In-Conter Fusion module leads to improvements in mean mloU, AP and J&F. Further enhancements are observed across all evaluation metrics when the M-Former is used. This demonstrates the effectiveness of the proposed components in improving the model's ' performance for various segmentation tasks.
Varying the number of frames Table 7b demonstrates gradual improvement in the overall segmen- tation quality as the number of frames increases from to 6. Overall, using moderate number of frames, such as 6 or 8, achieves optimal VOS performance on the DAVIS 2017 dataset.
Varying the LoRA rank Table 7d demonstrates the effect of the LoRA rank transfer performance. When the LoRA is 32, SINE achieves acceptable performance with few trainable parameters. Further increasing the rank 64 only leads marginal improvement. Considering the trade-off between performance and the number of trainable parameters, we select rank 32 as the default setting.
5 Discussion and Conclusion
Conclusion this work, we point out the task ambiguity in in-context segmentation for the first time and present SINE, which simultaneously predicts multiple task-specific masks to address this problem. Leveraging the efficient design, SINE can utilize few trainable parameters to exhibit strong segmentation abilities. We hope SINE can promote the development of in-context segmentation. Limitation and more Discussions are provided Appendix A.
Acknowledgement
This work is partially supported by the National Key R&D Program of China(NO.20222201666) and the National Natural Science Foundation China (No. 62206244).
10

---

References
[1] Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexci Efros, Visual prompting via image inpainting. In Adv. Neural Inform. Process. Syst. 2022.
[2] Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee. Yolact: Real-time instance segmentation. In Inf. Conf. Comput. Vis., 2019.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot Ieamerrs. In Adv. Neural Inform. Process. Syst. 2020.
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Eur. Conf. Comput. Vis., 2020.
[5] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. IEEE Conf. Comput. Vis. Puttern Recog.. 2022.
[6] Ho Kei Cheng and Alexander Schwing. Xmem: Long-term video object segmentation with an atkinson- shiffrin memory model. Eur. Conf. Comput. Vis., 2022.
[7] Bert De Brabandere, Davy Neven, and Luc Van Gool. Semantic instance segmentation with a discriminative loss function. arXiv preprint arXiv:1708.0223 02551, 2017.
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition scale. In Int. Conf. Learn. Represent, 2020.
[9] Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. Int. J. Comput. Vis., 2010.
[10] Chengxiang Fan, Muzhi Zhu, Hao Chen, Yang Liu, Weijia Wu, Huaqi Zhang, and Chunhua Shen. Divergen: Improving instance segmentation by learning wider data distribution with more diverse generative data. In IEEE Conf. Comput. Vis. Pottern Recog. 2024.
[11] Zhibo Fan, Jin-Gang Yu, Zhihao Liang. Jiarong Ou, Changxin Gao, Gui-Song Xia, and Yuanqing Li. Fgn: Fully guided network for few-shot instance segmentation. In IEEE Conf. Comput. Vis. Pattern Recog.. 2020.
[12] Dan Andrei Ganea, Bas Boom, and Ronald Poppe. Incremental few-shot instance segmentation. In JEEE Conf. Comput. Vis. Pattern Recog., 2021.
[13] Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Yi Lin, Ekin D Cubuk, Quac V Le, and Barret Zoph. Simple copy-paste is a strong data augmentation method for instance segmentation. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.
[14] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yo Lin. Scaling open- vocabulary image segmentation with image-level labels. In Eur. Conf. Comput. Vis., 2022.
[15] Ross Girshick. Fast R-CNN. In Int. Conf. Comput. Vis., 2015.
[16] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In IEEE Conf. Comput. Vis. Pattern Recog,, 2019.
[17] Yue Han, Jiangning Zhang, Zhucun Xue, Chao Xu, Xintian Shen, Yabiao Wang, Chengjie Wang, Yong Liu, and Xiangtai Li. Reference twice: A simple and unified baseline for few-shot instance segmentation. arXiv preprint arXiv:2.30.001 156, 2023.
[18] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In IEEE Conf. Comput. Vis. Pattern Recog., 2022.
[19] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In IEEE Conf. Comput. Vis. Pattern Recog, 2017.
[20] Sunghwan Hong. Scokju Cho, Jisu Nam, Stephen Lin, and Seungryong Kim. Cost aggregation with 4d convolutional swin transformer for few-shot segmentation. In Eur. Conf. Comput. Vis., 2022.
[21] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- -Zu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 0tt::2/06.09666 2021.
[22] Joakim Johnander, Martin Danelljan, Emil Brissman, Fahad Shahbaz Khan, and Michael Felsberg. A generative appearance model for end-to-end video object segmentation. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
[23] Bingyi Kang, Zhuang Liu, Xin Wang, Fisher Yu, Jiashi Feng, and Trevor Darrell. Few-shot object detection via feature reweighting. In Int. Conf. Comput. Vis., 2019.

---

[24] Alexander Kirillov. Ross Girshick. Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In IEEE Conf. Comput. Vis. Pattern Recog. 2019.
[25] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Int. Conf. Comput. Vis., 2023.
[26] Alexander Kolesnikov, André Susano Pinto, Lucas Beyer, Xiaohua Zhai, Jeremiah Harmsen, and Neil Houlsby. Uvim: A unified modeling approach for vision with learned guiding codes. In Ady. Neural Inform. Process. Syst., 2022.
[27] Feng Li, Qing Jiang. Hao Zhang. Tianhe Ren, Shilong Liu, Xueyan Zou, Huaizhe Xu, Hongyang Li, Jianwei Yang, Chunyuan Li, al. Visual in-context prompting. In IEFE Conf. Comput. Vis. Pattern Recog.. 2024.
[28] Yongqing Liang, Xin Li, Navid Jafari, and Jim Chen. Vidco object segmentation with adaptive feature bank and incertain-region refinement. In Adv. Neural Inform. Process. Syst., 2020.
[29] Guosheng Lin, Anton Milan, Chunhua Shen, and Ian Reid. Refinenet: Multi-path refinement networks for high-resolution semantic segmentation. In JEEE Conf. Comput. Vis. Pattern Recog. 2017.
[30] Huaijia Lin, Xiaojuan Qi. and Jiaya Jia. Aoss-vos: Attention guided single-shot video object segmentation. In Int. Conf. Comput. Vis., 2019.
[31] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In Eur: Conf. Comput. Vis., 2014.
[32] Zhihui Lin, Tianyu Yang, Maomao Li, Ziyu Wang, Chun Yuan, Wenhao Jiang, and Wei Liu. Swem: Towards real-time vidco object segmentation with soquential weighted expectation-maximaee Conf. Comput. Vis. Pattern Recog., 2022.
In IEEE
[33] Yang Liu, Xinlong Wang, Muzhi Zhu, Yue Cao, Tiejun Huang, and Chunhua Shen. Masked channel modeling for bootstrapping visual pre-training. Int. J. Comput. Vis., 2024.
[34] Yang Liu, Muzhi Zhu, Hengtao Li, Hao Chen, Xinlong Wang, and Chunhua Shen, Matcher: Segment anything with one shot using all-purpose feature matching. arXiv preprint arXiv:2305.13310 2023.
[35] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In IEEE Conf. Comput. Vis. Pattern Recog., 2015.
[36] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17Xiv:711.05101, 2017.
[37] Sabarinath Mahadevan, Paul Voigtlaender, and Bastian Leibe. Iteratively trained interactive segmentation. In Brit. Mach. Vis. Conf.,2018.
[38] Sourab Mangrulkar. Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://gither com www.cogg//ttcecoooooooooco huggingface/ peft, 2022.
[39] Lingchen Meng, Shiyi Lan, Hengduo Li, Jose M Alvarez, Zuxuan Wu, and Yu-Gang Jiang. Segic: Unleashing the emergent correspondence for in-context segmentation. In Eur. Conf. Comput. Vis., 2024.
[40] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In Int. Conf. 3D vision (3DV), 2016.
[41] Juhong Min, Dahyun Kang, and Minsu Cho. Hypercorrelation squeeze for few-shot segmentation. In JEEE
Conf. Comput. Vis. Pattern Recog., 2021.
[42] Keval Morabia, Jatin Arora, and Tara Vijaykumar. Attention-based joint detection of object and semantic part. arXiv preprint (xXiv:2007.02419 2020.
[43] Khoi Nguyen and Sinisa Todorovic. Feature weighting and boosting for few-shot segmentation. In Int. Conf. Comput. Vis., 2019.
[44] Maxime Oquab, Timothée Dareet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacidin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprini arXivv2304.07199 2023.
[45] Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Horuung A benchmark dataset and evaluation methodology for video object segmentation. In IEFF Conf. Comput. Vis. Pattern Recog., 2016.
[46] Jordi Pont-Tuuct, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine- Hoaning, and Lue Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint 1:XX::/7040007555 2017.
[47] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Int. Conf. Mach. Learn. 2021.
12

---

[48]
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Int.
Conf. Comput. Vis., 2021.
[49]
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In IEEE Conf. Comput. Vis. Pattern Recog. 2022.
[50] Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. arXiv preprint arXiv: 1709 03410, 2017.
[51] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Int. Conf. Compur. Vis., 2019.
[52] Chris Stauffer and W. Eric Grimson, Adaptive background mixture models for real-time tracking. In IEEE Conf. Comput. Vis. Pattern Recog.. 1999.
[53] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkorcit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Adv. Neural Inform. Process. Syst., 2017.
[54] Xinlong Wang. Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: generalist painter for in-context visual learning. In IEEE Conf. Comput. Vis. Pattern Recog. 2023.
[55] Xinlong Wang, Rufeng Zhang, Tao Kong, Lei Li, and Chunhua Shen. Solov2: Dynamic and fast instance segmentation. In Adv. Neural Inform. Process. Syst., 2020.
[56] Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Segmenting everything in context. In Int. Conf. Comput. Vis., 2023.
[57] Ning Xu, Brian Price, Scott Cohen, Jimei Yang, and Thomas S Huang. Deep interactive object selection. In IEEE Conf. Comput. Vis. Pattern Recog,, 2016.
[58] Ning Xu, Linjie Yang. Yuchen Fan, Dingeheng Yue, Yuchen Liang, Jianchao Yang, and Thomas Huang. Youtube-vos: large-scale video object segmentation benchmark. arXiv preprint arXiv:1809.0 100.00399900 2018.
[59] Xiaopeng Yan, Ziliang Chen, Anni Xu, Xiaoxi Wang. Xiaodan Liang, and Liang Lin. Meta r-cnn: Towards
general solver for instance-lovel low-shot learning. In Int. Conf. Comput. Vis., 2019.
[60] Zongxin Yang. Yunchao Wei, and Yi Yang. Associating objects with transformers for video object segmentation. In Adv. Neural Inform. Process. Syst., 2021.
[61] Gongjie Zhang, Zhipeng Luo, Kaiwen Cui, Shijian Lu, and Eric P Xing. Meta-detr: Image-level few-shot detection with inter-class correlation exploitation. IEEE Trans. Pattern Anal. Mach. Intell., 2022.
[62] Jian Wei Zhang, Yifan Sun, Yi Yang. and Wei Chen. Feature-proy transformer for few-shot segmentation. In Adv. Neural Inform. Process. Syst., 2022.
[63] Renrui Zhang, Zhengkai Jiang, Ziyu Guo, Shilin Yan, Junting Pan, Hao Dong, Peng Gao, and Hongsheng Li. Personalize segment anything model with one shot. arXiv preprint 2055042305.03348, 2023.
[64] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In IEEE Conf. Comput. Vis. Pattern Recog., 2017.
[65] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. Int. J. Comput. Vix., 2019.
[66] Muzhi 7hu, Chengxiang Fan, Hao Chen, Yang Liu, Weian Mao, Xiaogang Xu, and Chunhua Shen. Generative active learning for long-tailed instance segmentation. In Ini. Conf. Mach. Learn., 2024.
[67] Muzhi Zhu, Yang Liu, Zekai Luo, Chenchen Jing, Hao Chen, Guangkai Xu, Xinlong Wang, and Chunhua Shen. Unleashing the potential of the diffusion model in few-shot semantic segmentation. arXiv preprint arXiv:2410.02366 2024.
[68] Xueyan Zou, Jianwei Yang. Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. arXiv preprint arXiv:2304.06711 2023.
13

---

Appendix
A Discussions and Limitation
Comparison to SegGPT Both SINE and SegGPT are in-context segmentation models. SINE's flexibility is comparable to SegGPT for various segmentation tasks. SINE can handle instance segmentation, which fails in SegGPT. SegGPT cannot deal with task ambiguity. The pixel output of SegGPT is not the final result and necessitates complex post-processing that converts the RGB output to mask. SINE avert this problem by predicting the mask directly. Stitching reference and target in SegGPT limits its capacity to process high-resolution images. SINE does not have this problem.
Comparison to SAM SINE and SAM offer different paths to the segmentation foundation model. SAM provides semantic- Crnnantc--freeetee promptable segmentation, while SINE handles semantic in-context segmentation. With different emphases, SINE and SAM can complement each other. Take auto- labeling as an example. SAM labels objects (e-g., dog) in the first image. SINE could use that image as an in-context example to label subsequent images to reduce costs.
Limitation As the first work to study ambiguity in in-context segmentation, SINE focuses on resolving ambiguities among ID, instance, and semantic segmentation tasks (as these are more important and commonly used). More complex ambiguities, such as full objects and parts, spatial position, category, and color, can be addressed by incorporating multimodal in-context examples (e.g., image and text). In addition, SINE has a performance gap compared with SegGPT in video segmentation. We think the reason behind this performance gap is that SegGPT trains all model parameters (300M), while SINE uses a simpler in-contex, fusion module and fewer learnable parameters (19M), limiting its ability handle inter-frame relations in complex videos. Although SINE currently has limitations in learning complex inter- frame relationships in videos, we believe that by designing more suitable In-Context Interaction module, the current paradigm holds greater potential for solving in-context segmentation tasks.
Broader Impacts Our Method is built upon open-source foundation models, significantly reducing carbon emissions. We do not foresee any obvious undesirable ethical or social impacts now.
B Implementation Details
Training Details. We train our model with diverse segmentation datasets, including semantic, instance, and panoptic. The sampling weight for each dataset is 0.14 (COCO panoptic), 0.14 (ADE20K panoptic), 0.18 (COCO instance), and 0.54 (Objects365 instance). SINE uses two kinds of in-context pairs, selecting two images including the same category objects as the in-context pair or using two transformed views of the same image as the in-context pair. The probability is 0.5 for both in-context pairs.
Vision foundation models [18, 47, 44, 33] have demonstrated amazing visual representation capa- bilities, encouraging us to explore their performance in a wider range of applications. We deploy the frozen DINOv2 (ViT-L) [44] with 304M parameters as the Transformer encoder of SINE and train the In-Context Fusion module and the lightweight Transformer decoder with only 19M trainable parameters. The decoder contains six M-Former blocks. The model size of SINE is comparable with SegGPT (307M). SINE has fewer trainable parameters, leading to more efficient training. We randomly initialize the trainable parameters and train SINE about 50K steps with 64 batch sizes. We use Adam [36] optimizer and employ 81 = 0.9, _3 = 0.999 for optimization. We use a lincar learning rate scheduler with base learning rate of le-4 and warmup of 100 steps. The weight decay is set to 0.05. For data augmentation, we use random horizontal flipping and the large-scale jittering (LSJ) [13] augmentation with random scale sampled from range 0. to 2.0 followed by fixed size crop to 896 x 896. Our model is trained for days by using 8 NVIDIA V100 GPUs.
Evaluation.
The in-context examples are from the support samples for few-shot semantic segmenta- tion, the training set for few-shot instance segmentation, and the first video frame for video object segmentation. The semantic prototypes can be seen as the classifier. The post- pooseeroccssing is similar to Fask2Former [5], but we use the predictions of the ID queries for VOS and the predictions of instance queries for instance segmentation, respectively.
14

---

Additional Results
C.1 Comparison of SINE and SegGPT without Objects365.
Table 8 compares the one-shot semantic segmentation results of training SINE using only ADE20K and COCO with SegGPT. SINE outperforms SegGPT on three benchmarks. Notably, SINE achieves 10% higher mIoU than SegGPT on LVIS-92i, indicating stronger class generalization capability in real-world image segmentation compared to SegGPT.
Methods
COCO-20'
PASCAL-5'
LVIS-922
SegGPT
56.1
83.2
18.6
SINE
67.1
86.3
28.8
Table 8
Comparison of SINE and SegGPT without Objects365.
C.2 Impact Different Backbones.
We select DINOv2-S, DINOv2-B, DINOv2-L, and CLIP-L [47] to explore the impact of different follows: DINOv2 better than CLIP. DINOv2 achieves
backbones. The conclusions better performance than CLIP because it has general matching capabilities at both image and patch levels, allowing to better understand complex contextual information between images. In contrast, CLIP captures image-text similarity, making it difficult to capture relationships between images, leading to poorer performance. 2) Larger DINOv2 model performs better: larger DINOv2 models have stronger representation capabilities, making it easier to capture contextual relationships, thus improving performance. This also indicates that SINE scalable with the enhanced capabilities of the encoder.
Methods SegGPT
Backbone
COCO-20'
PASCAL-5
LVIS-92*
56.1
83.2
18.6
DINOv2-S
56.8
81.4
26.7
DINOv2-B
61.7
84.1
29.5
SINE
DINOv2-L
64.5
85.4
31,2
CLIP-L
34.8
57.3
16.1
Table Impact of Different Backbones.
C.3
Generalization of SINE on One-shot Part Segmentation.
As shown in Fig. 5, SINE can perform part segmentation like SegGPT when including PACO as training data. We evaluate SINE for one-shot part segmentation on PASCAL-PPat [42] following Matcher [34] for the data pre- me-processing and evaluation. Compared with SegGPT SINE achieves competitive performance.
Methods
animals
indoor
person
vehicles
mean
SegGPT 22.8
50.9
31.3
38.0
35.8
SINE
22.7
61.8
21.7
38.5
36.2
Table 10 Results of one-shot part segmentation PASCAL-Part.
C.4 Generalizability of SINE beyond Semantically Similar Objects.
Fig. 6 shows SINE's capability in handling complex interaction relationships. In Fig. 6(a), the reference consists of multiple images, each containing different objects (box, cup, keyboard, mouse). When using these as in-context examples, SINE can segment one or more semantically different objects on a desk. In Fig. 6(b), with a reference containing only one object, the in-context example cannot represent complex interactions, and thus no segmentation result is provided. In Fig. 6(C), replacing multiple single-object images with single image containing multiple objects yields the same effective results.
15

---

C.5 Visualizations
Fig. shows further visual comparisons. For video tasks, SINE reduces tracking failures from intersections, viewpoint changes, and occlusions. SINE addresses task ambiguity, preventing errors in semantic segmentation where SegGPT fails, as shown in the second of comparison results in Fig. 7(a). In real-world image segmentation, SINE exhibits better class generalization than SegGPT, matching the LVIS-92i comparison in Table SINE can effectively alleviate the prompt ambiguity and produce more accurate segmentation results. We provide more visualizations for example- based semantic segmentation in Fig. 8, example-base based instance segmentation Fig. video object segmentation in Fig. 10, semantic segmentation on ADE20K, and panoptic segmentation on COCO Figure 11.
12
11
10
3
5
4
H
reference (blue mask)
GT mask)
SINE (green mask)
SegGPT (purple mask)
reference mask)
GT (red mask)
SINE (green mask)
SegGPT (purple mask)
(a) Successful cases for SINE and SegGPT.
(b) Cases of failures for SegGPT.
Figure Visualization of part segmentation.
reference
Target
(b)
reference
target
reference
target
reference
target
(c)
reference
larget
target
reference
target
target
Figure 6 Generalizability of SINE beyond semantically similar objects. (a) and (c) SINE can recognize and segment another object when given successive images (or image) including different objects. (b) SINE cannot output masks when given one reference image with different object.
16

---

BegaPT
SINE
BegGPT
SegGPT
SINE
reference Diiu masik
SINE (geken
SegGPT (naple mask)
(red mask) (b) Image segmentation
(s) Video sogmeniatiin
Figure Visualization comparisons between SINE and SegGPT on video image tasks. For video task, SINE can effectively alleviate the tracking failure issues caused by crossing, perspective changes, occlusion, etc. In real-world image segmentation, SINE has stronger category generalization capability compared to SegGPT and can effectively alleviate the ambiguity problem.
reference
GT
prediction
reference
prediction
reference
prediction
Figure 8 Visualization of example- based semantic segmentation.
17

---

J.P.Mrraan
reference
predictions
Figure 9 Visualization of example-base based instance segmentation.
Figure Visualization of video object segmentation.
18

---

(a) ADE20K Semantic Segmentation
J.P.orren
COCO Panoptic Segmentation
Figure 11 Visualization of semantic segmentation on ADE20K, and panoptic segmentation on COCO.
19

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
In the abstract and the end of the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
In the limitation part of the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The dataset, model, and training procedures are clearly described in this paper and will release code upon acceptance.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [No]
Justification: Our work is based on the open-source code. The dataset, model, and training procedures are clearly described in this paper and will release code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https:
ccpps.cc/puullc//gu guwdee/iee///ee CCodeSubeesiaceecceee simiisssinn ssssonoliccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We clearly described the training and evaluation details in the experiments setting section and in the implementation details of the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive and we do not have enough computer sources.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
22

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the implementation details of the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http///eerri..ee/eee/ httpp/lii//ciiccciiieeeeeeeeeeee
Answer: [Yes]
Justification:
We follow the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: the discussions the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the things in this paper are credited and are the license and terms of use explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification:
We do not release assets
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "mR79JlxXK9", "reviewer": "Reviewer_sZmi", "summary": "SINE aims to resolve the problem of task ambiguity in in-context segmentation, where previous models struggled to accurately infer tasks based on in-context examples alone. This ambiguity arises because traditional models often fail to distinguish between different segmentation tasks like semantic segmentation, instance segmentation, or identifying specific objects based on the context provided. SINE utilizes a Transformer-based architecture where the encoder generates high-quality image representations and the decoder produces multiple task-specific output masks. In-context Interaction Module further enhances the encoder's output by establishing correlations between the target image and in-context examples, helping to better define segmentation tasks. Matching Transformer (M-Former), a dual-path Transformer decoder, updates object queries and semantic prototypes for precise task execution. It incorporates fixed matching and the Hungarian algorithm to resolve differences between tasks, ensuring that the output aligns well with the given task. SINE achieves impressive performance improvements over previous in-context segmentation models like SegGPT, particularly in handling multiple segmentation tasks simultaneously. It is shown to effectively address the issue of task ambiguity, producing relevant segmentation masks that are more aligned with the semantic content of the images.", "questions": "Minor question: Why is the target image the same as the raw image in Figure 4a)? \n\nMinor issues - typos: L307 \"Results Table 7 compares ....\" should be \"Results Table 4 compares ....\"? If it's not a typo, Table 4 was never discussed in the main paper.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "[Experimental Results]. Extensive experiments across multiple benchmarks show significant improvements in multiple downstream tasks. \n\n[Paper writing]. This paper is well-written and overall easy to follow.\n\n[Interesting task formulation]. SINE was shown to effectively address the issue of task ambiguity, producing relevant segmentation masks that are more aligned with the semantic content of the images, which was not explored by the previous works in this domain.", "weaknesses": "[Concerns on Generalizability: Complex Interaction Relationships Beyond Semantically Similar Objects] While this paper adeptly handles in-context instance or semantic segmentation, numerous open-vocabulary segmentation models already capably segment novel classes or objects. A crucial demonstration of an in-context segmentation model's comprehension of in-context samples should extend beyond merely identifying semantically similar objects provided by context information. An intriguing task would involve, for example, segmenting objects situated on a table across multiple images—such as a bottle, a plate, and a book in successive images—and understanding whether SINE can recognize and segment another object on a table in a subsequent image. This requires the model to understand the interaction, the relationships, etc, that goes beyond segmenting semantically similar objects. \n\n[Challenges with Complex In-Context Information] Often, a single pair of images suffices to provide the necessary in-context information reflecting the user's intent. However, more complex scenarios might require integrating multiple pairs of images to fully capture intricate in-context information. I am concerned about SINE's limitations in handling such complexity.\n\n[Performance Discrepancy in Video Segmentation Compared to SegGPT] While SINE excels beyond SegGPT in instance segmentation on MSCOCO, it underperforms in video instance segmentation across most evaluated benchmarks. This result is concerning, considering video instance segmentation's unique demand for the model to sustain consistent correspondence across multiple video frames, a task more complex than segmenting static images. I am interested in understanding the specific reasons behind this performance gap.\n\n[Unable to Handle Many Tasks SegGPT Supports]. SegGPT can actually support a lot more tasks, such as hierarchical in-context segmentation. I am concerned if SINE is overfitted on the instance/semantic segmentation tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your comments and the approval of our task formulation. We address your concerns here.\n___\n\n>W1: Concerns on Generalizability.\n\nFig. 1 in the attached PDF shows SINE's capability in handling complex interaction relationships. \n- In Fig. 1(a), the reference consists of multiple images, each containing different objects (box, cup, keyboard, mouse). When using these as in-context examples, SINE can segment one or more semantically different objects on a desk.\n- In Fig. 1(b), with a reference containing only one object, the in-context example cannot represent complex interactions, and thus no segmentation result is provided.\n- In Fig. 1(c), replacing multiple single-object images with a single image containing multiple objects yields the same effective results.\n\nThese experiments indicate that SINE has the potential to handle complex interaction relationships beyond semantically similar objects.\n\n\n>W2: Challenges with Complex In-Context Information.\n\nWe believe SINE can handle complex in-context infromation because the contextual relationships between complex scenarios can be captured more efficiently in the representation space of DINOv2.\nSINE leverages this characteristic of DINOv2 to accurately understand and fully capture intricate in-context information in multiple pairs of images. Extensive experiments show that SINE performs better in the few-shot setting than the one-shot setting, indicating its ability to effectively comprehend and capture in-context information from multiple samples. Figure 1(a) in the attached PDF also shows SINE's capability to understand and handle the complex relationships in in-context examples.\n\n\n>W3: Performance Discrepancy in Video Segmentation Compared to SegGPT.\n\nWe think the reason behind this performance gap is that SegGPT trains all model parameters (300M), while SINE uses a **simpler in-context fusion module** and **fewer learnable parameters (19M)**. The detailed analysis is as follows:\n\nSegGPT uses a broader dataset, concatenates references and targets spatially, and employs an ViT architecture. It trains all model parameters (300M), with self-attention effectively capturing relationships between video frames.\n\nSINE aims to learn a general and lightweight decoder to efficiently transfer representations from pre-trained DINOv2 (encoders) to in-context segmentation. The frozen DINOv2 is limited to capture in-context information. For efficiency, SINE deploys only a simple in-context fusion module (1.58M) to learn in-context relationships, limiting its ability to handle inter-frame relations in complex videos.\n\nConsidering image tasks or simple videos (e.g., DAVIS), we believe that the overall performance of SINE are satisfactory. In paiticular, compared to recent generalist segmentation models SEEM[A] and DINOv[B] (which train all parameters), SINE's efficient design shows greater potential in video tasks (see the table below).\n\nAlthough SINE currently has limitations in learning complex inter-frame relationships in videos, we believe that by designing more suitable In-Context Interaction module, the current paradigm holds greater potential for solving in-context segmentation tasks. We will explore this further in future work. These discussions will be added to the paper.\n\n||DAVIS|YouTube-VOS|\n|---|---|---|\n|SegGPT|75.6| 74.7| \n|SEEM [A] |58.9| 50.0|\n|DINOv [B] | 73.3|60.9|\n|SINE|77.0|66.2|\n\n\n[A] Segment everything everywhere all at once. NIPS 2023.\n\n[B] Visual in-context prompting. CVPR 2024.\n\n>W4: Unable to Handle Many Tasks SegGPT Supports.\n\nSINE and SegGPT have different motivations. \n- SegGPT aims to verify visual in-context learning can unify different segmentation tasks by using a broader range of datasets, such as semantic segmentation (ADE20K), instance segmentation (COCO), and part segmentation (PACO). Hence SegGPT can perform hierarchical in-context segmentation. \n- SINE aims to resolve the task ambiguity in prompts. As the first work to highlight this problem, we initially focus on resolving ambiguities among ID, instance, and semantic segmentation tasks (as these are more important and commonly used). \n\nWhen PACO is included as training data, **SINE can perform part segmentation like SegGPT**, as shown in Fig.2 of the attached PDF. This shows that our method does not overfit instance/semantic segmentation.  Additionally, the results in Table 1 of the paper (LVIS-92i) and Fig.3 of the attached PDF indicate that SINE has stronger class generalization capability compared to SegGPT in real-world image segmentation.\n\n\n>Q1: Why is the target image the same as the raw image in Figure 4a ?\n\nIn Fig. 4(a), the giraffe demo is selected from two video frames with a large temporal gap. The reference is from an earlier frame with only one giraffe, while the subsequent frame gradually introduces another giraffe. This demo tests whether SINE can identify objects with the same ID without temporal information.\n\n\n>Q2: typos: L307 \"Results Table 7 compares ....\"\n\nThanks for pointing this out. It should be Table 4. We will fix this error in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DdioGDRksA", "reviewer": "Reviewer_nRMP", "summary": "The paper proposes a generalist model for image segmentation named SINE, which unifies multiple image segmentation tasks into the common formulation of visual in-context learning. This work aims to identify and model the task of object reidentification to reduce ambiguities within the in-context examples. By incorporating the modeling of each specific segmentation task within the SINE architecture, it effectively improves upon existing generalist models and achieves strong performance across a wide range of segmentation tasks.", "questions": "1. The authors only conducted experiments on dinov2 vit-l. How do models of different sizes and different pre-training affect the results? For example, clip.\n2. How does SINE perform few-shot learning? It seems that SINE can only accept a single reference image.", "limitations": "See weaknesses.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper offers a valuable review of related works in in-context segmentation, analyzing the problems with the recent SegGPT and clarifying the differences of task setting between them.\n2. The authors discuss the relationship between different segmentation tasks, effectively unifying them as instance segmentation, which is well-motivated.\n3. This paper points out and addresses the ambiguities in visual prompting, which is currently an open research problem.\n4. The authors conduct comprehensive experiments across various segmentation tasks, demonstrating significant performance improvements over recent generalist and specialist models.", "weaknesses": "1. This paper only addresses the ambiguity between instance and semantic segmentation. However, there is a broader ambiguity in visual prompts, such as spatial position, category, color, etc. The authors need to discuss these aspects in more detail.\n2. Compared to SegGPT, SINE introduced additional Objects365 as extra training data. Although this was explained, it still seems to lack some fairness. Without using Objects365, can better performance be achieved than SegGPT? For example, by only using ADE20K, COCO, etc., and what is the performance on COCO-20i, PASCAL-5i?\n3. Although the authors conducted numerous experiments, the impact of different backbones is missing. Since the backbone is frozen, different models might bring significant performance differences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your comments and the approval of our motivation and performance. We address your concerns here.\n\n___\n\n>W1: Discussion of more ambiguity in visual prompts.\n\nThanks for your helpful suggestions. SINE is the first work to highlight task ambiguity in the visual prompts of in-context segmentation, initially focusing on resolving ambiguities among ID, instance, and semantic segmentation tasks (as these are more important and commonly used). We believe addressing ambiguities at this level is meaningful. For more complex ambiguities, such as full objects and parts, spatial position, category, and color, these can be addressed by incorporating multimodal in-context examples (e.g., image and text). We will add more discussions on this in the paper.\n\n\n>W2: Comparison of SINE and SegGPT by only using ADE20K, COCO.\n\nThe table below compares the one-shot semantic segmentation results of training SINE using only ADE20K and COCO with SegGPT. SINE outperforms SegGPT on three benchmarks. Notably, SINE achieves 10% higher mIoU than SegGPT on LVIS-92i, indicating stronger class generalization capability in real-world image segmentation compared to SegGPT. Additionally, we are the first to explore the use of Objects365 in in-context segmentation. Table 5(a) of the paper shows that SINE's effective design leads to further improvements in generalization capability with the inclusion of Objects365.\n\n|  | COCO-20i | PASCAL-5i | LVIS-92i |\n| --- | --- | --- | --- |\n| SegGPT | 56.1 | 83.2 | 18.6 |\n| SINE | 67.1 | 86.3 | 28.8 |\n\n\n>W3,Q1: Impact of different backbones.\n\n\nWe select DINOv2-S, DINOv2-B, DINOv2-L, and CLIP-L to explore the impact of different backbones. The conclusions are as follows:\n\n1. DINOv2 Outperforms CLIP: DINOv2 achieves better performance than CLIP because it has general matching capabilities at both image and patch levels, allowing it to better understand complex contextual information between images. In contrast, CLIP captures image-text similarity, making it difficult to capture relationships between images, leading to poorer performance.\n\n2. Larger DINOv2 Models Perform Better: Larger DINOv2 models have stronger representation capabilities, making it easier to capture contextual relationships, thus improving performance. This also indicates that SINE is scalable with the enhanced capabilities of the encoder.\n\n\n\n|  | COCO-20i | PASCAL-5i | LVIS-92i |\n| --- | --- | --- | --- |\n| SegGPT | 56.1 | 83.2 | 18.6 |\n| SINE DINOv2-S | 56.8 | 81.4 | 26.7 |\n| SINE DINOv2-B | 61.7 | 84.1 | 29.5 |\n| SINE DINOv2-L | 64.5 | 85.4 | 31.2 |\n| SINE CLIP-L | 34.8 | 57.3 | 16.1 |\n\n>Q2: How does SINE perform few-shot learning?\n\nMultiple reference image features and masks are concatenated in the spatial dimension. The resulting feature and mask can be treated as a single reference image, and the subsequent process remains the same.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aEvhaRWi03", "reviewer": "Reviewer_rhP3", "summary": "The paper proposes an image segmentation framework using in-context examples. To eliminate ambiguity from the in-context examples, multiple output masks are predicted. It uses a pre-trained image encoder to extract features from target and reference images, pools these features into ID and semantic tokens using the reference mask, and employs a Matching Transformer to decode the output masks. Experiments on various segmentation tasks demonstrate the effectiveness of the proposed method.", "questions": "N/A", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The motivation on task ambiguity of segmentation with in-context example is clear. \n- The paper evaluates the framework on few-shot semantic segmentation, few-shot instance segmentation, and video object segmentation on multiple datasets.", "weaknesses": "The novelty of the overall idea and network structure is limited. The idea to solve the ambiguity of prompts (in-context examples in this paper) by predicting multiple masks is from SAM. The network architectures are mainly based on DETR and Mask2Former. Though these choices are effective, they do not introduce innovations to the field. Overall, the paper shows solid practical utility,  and the limited novelty in its core idea and network design prevents me from giving it a higher rating. A stronger emphasis on introducing novel concepts or architectural advancements would enhance the impact and recognition of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your comments and the approval of our motivation and practical utility of SINE. We address your concerns here.\n\n___\n\n>W1: The novelty of the overall idea and network structure is limited. \n\n\nTo address the reviewer's concerns, we first discuss the contributions and academic insights of our paper. Then, we discuss the differences between SINE and other methods (SAM, DETR, and Mask2Former). Finally, we outline some perspectives. We hope these responses adequately address the issues and concerns raised.\n\n## Contribution and Insight\n\nIn-context segmentation is an important proxy task for unifying different segmentation tasks, and it has garnered significant attention from the research community. Our paper delves deeply into this task, and we summarize our contributions, novelty, and research insights as follows:\n\n1. **Ambiguities in Visual Prompting**: This paper is the *first to explore the task ambiguity problem in prompts within in-context segmentation* (supported by **sZmi** and **nRMP**). This is an **important** and **novel** issue that has been **underexplored**. We investigate the conflicts among multiple tasks in in-context segmentation from the perspective of task ambiguity and *provide effective solutions*, offering valuable insights to the research community.\n\n2. **Investigating the Capabilities of VFMs**: Utilizing Visual Foundation Models (VFMs) to address various tasks is becoming a research trend. Our paper explores how to efficiently transfer the visual representations of VFMs to in-context segmentation, a research paradigm that *currently lacks extensive exploration in the in-context segmentation field*.\n\n3. **Lightweight and Effective Decoder**: We thoroughly analyze the challenges brought by task ambiguity (see Lines 181-185) and design the M-Former structure to address these issues. The novel dual-path design and Shared SA in M-Former ensure efficient decoding while avoiding information confusion across tasks of different granularity. Additionally, unlike previous methods, SINE achieves significant performance improvements with fewer trainable network parameters (19M). Thus, our method also demonstrates novelty.\n\n\n## Differences between SINE and other methods\n\nBased on the above contributions, SINE fundamentally differs from SAM, DETR, and Mask2Former:\n\n**Comparison with SAM**\n\n- SAM is a promptable segmentation model that takes a point as input and outputs the corresponding mask. The ambiguity lies in the uncertainty of the segmentation granularity represented by the point. SAM addresses this by introducing different queries in the decoder to obtain masks of different granularities.\n\n- Resolving ambiguity in prompts (references) for in-context segmentation poses a more challenging task. In-context segmentation requires understanding the information in the prompt (e.g., category, position, shape) and learning the complex interaction relationships between the reference and target (approved by **sZmi**). Misunderstanding the prompt leads to incorrect outputs. Therefore, unlike SAM, SINE must learn the complex contextual relationships and avoid information confusion across different tasks.\n\n**Comparison with DETR and Mask2Former**\n\n- DETR and Mask2Former aim to train all model parameters on a specific dataset (e.g., COCO) to perform detection or segmentation on a limited number of categories.\n\n- SINE aims to effectively perform in-context segmentation in the open world by using the off-the-shelf VFMs. The parameters of the encoder are frozen. We have analyzed the difficulties in addressing task ambiguity in in-context segmentation, challenges that do not exist in the general detection and segmentation task. *Mask2Former and DETR are not designed for in-context segmentation, making it difficult to address these challenges.*\nThe dual-path design and shared SA in M-Former are introduced specifically to address these challenges. As shown in Table 5(c), *compared to the Mask2Former decoder, M-Former effectively resolves task ambiguity, resulting in improvements across various tasks.*\n\n## Perspective\n\nFinally, we want to convey our perspective to the reviewer: In the era of large models, we cannot only focus on network architecture design. Different models with the same architecture can have vastly different characteristics (e.g., DINOv2 shows excellent patch-level matching ability, while CLIP excels in image-text retrieval). We should also focus more on how to fully leverage the potential of pre-trained models with fewer parameters and computations. From this perspective, SINE provides valuable insights to the academic community.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bqMxHZRM9j", "reviewer": "Reviewer_2YmX", "summary": "This paper proposes a generalist segmentation model, dubbed as SINE, for a variety of segmentation tasks.\n\nThe general idea is to harness the in-context examples, and to alleviate the task ambiguity.\n\nSpecifically, an in-context interaction module, a matching Transformer and a Hungarian algorithm is devised to realize the above objectives. \n\nExtensive experiments on a variety of datasets and segmentation tasks show its effectiveness.", "questions": "- Q1：The motivation does not match the methodology design properly. The in-context fusion has little relevance to the task-level ambiguity. \n\n- Q2: The two losses in the M-Former are still implemented on the object level.\n\n- Q3: The proposed method clearly lacks theoretical insight on how the task ambiguity is modeled and handled in theory. The overall module designs are common in visual representation learning, which is less relevant to the task-specific guidance or the task-level ambiguity.\n\n- Q4: The Transformer design along with the Hungarian algorithm is not uncommon in modern Transformer based pipeline design such as DERT.\n\n- Q5: Limited performance: Table 1, few-shot segmentation, on three out of six experiments, SINE is weaker than either Painter, SegGPT or recent ICLR works.\n\n- Q6: Limited performance: \n\n- Q7: Table 4, video object segmentation, the propose method is clearly inferior to the state-of-the-art by a large margin on two out of three datasets.\n\n- Q8: Clarify ablation study in Table 5c. (1) In-context fusion is only a part of in-context interaction module, while in Table 5c the authors seem to only include the impact of fusion part.\n(2) The specific components in M-Former also need further study, such as the specific loss functions and the phototype length and etc.\n\n- Q9: Eq.3 Hungarian loss. Do other types of commonly-used loss functions also achieve similar performance? More discussion and comparison is needed.\n\n- Q10: The visual results are not sufficient enough. Please provide and compare the visual results from other state-of-the-art methods.", "limitations": "Though providing a discussion, the reviewer believes that the discussion is not proper. \n\nSome potential negative impacts such as job displacement brought by generalist model should be discussed.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ This paper is overall well-written and easy-to-follow.\n\n+ The proposed method and module design is rationale and effective. \n\n+ The experiments and validation are extensive.", "weaknesses": "- The motivation does not match the methodology design properly. Specifically, the authors claim that the proposed method focuses on alleviating the task ambiguity. Unfortunately, throughout the module design, the in-context fusion has little relevance to the task-level ambiguity. Besides, the two losses in the M-Former are still implemented on the object level.\n\n- The proposed method clearly lacks theoretical insight on how the task ambiguity is modeled and handled in theory. The overall module designs are common in visual representation learning, which is less relevant to the task-specific guidance or the task-level ambiguity.\n\n- The Transformer design along with the Hungarian algorithm is not uncommon in modern Transformer based pipeline design such as DERT.\n\n- From the performance perspective, compared with the prior in-context segmentation methods, the proposed SINE does not show a clear improvement in many cases, for example:\n\n(1) Table 1, few-shot segmentation, on three out of six experiments, SINE is weaker than either Painter, SegGPT or recent ICLR works. \n\n(2) Table 4, video object segmentation, the propose method is clearly inferior to the state-of-the-art by a large margin on two out of three datasets.\n\n- The ablation study Table 5c is confusing, and more details or experiments need to be clarified. For example:\n\n(1) In-context fusion is only a part of in-context interaction module, while in Table 5c the authors seem to only include the impact of fusion part.\n\n(2) The specific components in M-Former also need further study, such as the specific loss functions and the phototype length and etc. \n\n- Eq.3 Hungarian loss. Do other types of commonly-used loss functions also achieve similar performance? More discussion and comparison is needed. \n\n- The visual results are not sufficient enough. Although the authors provide a lot of segmentation predictions in the supplementary material, only the results from the proposed method along with ground truth are provided. Please provide and compare the visual results from other state-of-the-art methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">W1,Q1,Q2: Motivation does not match methodology design properly.\n\nThe motivation and methodology design are related and clear (supported by **sZmi,rhP3**). \n\n- The goal of In-Context Fusion is to establish the correlations between reference and target (see Line 163-164), understanding the complex information within the context, which is crucial for the in-context segmentation task (supported by **sZmi**). \n- The proposed M-Former is designed to address task ambiguity in prompts, with the challenges for M-Former regarding task ambiguity analyzed in Lines 181-185. \n- This paper discusses the relationships between different tasks in detail (Line 112-117) and unifies them using instance segmentation, allowing different tasks to use the same model. Therefore, the loss is reasonable (supported by **nRMP**).\n\nThe paper clearly states the motivation of all designs, and their effectiveness is verified through experiments. This has also been recognized by other reviewers, aligning our method's design with our motivation.\n\n>W2,Q3: Insights and relevance.\n\nThe contributions and insights are as follows (please see **General Response** for details):\n1. Ambiguities in Visual Prompting\n2. Investigating the Capabilities of VFMs\n3. Lightweight and Effective Decoder\n\n**We believe the module designs are relevant to our motivation.** We design M-Former to address task ambiguity. Information from different tasks can interfere, causing incorrect predictions (Lines 181-185). \n*Mask2Former and DETR are not designed for in-context segmentation, making it difficult to address these challenges.*\nM-Former's dual-path structure and shared SA enable effective information interaction and prevent task interference, balancing efficient in-context segmentation decoding and resolving task ambiguity in prompts. In Table 5c, compared to Mask2Former, M-Former effectively resolves task ambiguity, resulting in improvements across various tasks.\n\n>W3,W6,Q4,Q9: Transformer along with Hungarian algorithm is not uncommon in modern Transformer based pipeline. Eq.3 Hungarian loss.\n\nHungarian Loss: **We do not claim the loss as a contribution.** Please refer to the General Response or Lines 65-71 of the paper for our contributions. The Hungarian algorithm has been widely used[A,B,C], and our use of it aligns with these works, making it reasonable and **not a weakness**. The loss with one-to-many matching used by the traditional method[D,E] will introduce NMS, which is not suitable for Transformer based method.\n\nTransformer design: We analyze task ambiguity challenges (Lines181-185) and design M-Former to address them. The dual-path design and Shared SA ensure efficient decoding and prevent information confusion across tasks of different granularity. Unlike previous methods, SINE improves performance with fewer parameters, demonstrating our method's novelty. See our **response to Reviewer rhP3 for more details**.\n\n>W4(1),Q5,Q6: SINE is weaker than other works on FSS.\n\nOur experiments show that SINE achieves significant improvements across multiple tasks and benchmarks (supported by **sZmi and nRMP**) and demonstrates solid practical utility (supported by **rhP3**).\n\nThe table below shows the **average performance** of Table 1, SINE achieves the best performance. Therefore, SINE's improvements are significant.\n\n||one-shot|few-shot|\n|---|---|---|\n|Painter|35.9|36.0|\n|SegGPT|52.6|61.0|\n|Matcher|42.9|50.4|\n|SINE|60.4|62.6|\n\n>W4(2),Q7: VOS.\n\nSee our **response to sZmi‘s W3**.\n\n>W5(1),Q8: Impact of fusion part in Table 5c.\n\nMask Pooling part does not contain any parameters. Learning in-context information is demonstrated in In-Context Fusion (see Lines 170-171). Mask Pooling is a necessary operation for SINE and cannot be removed. Hence, the corresponding ablation was not performed in Table 5.\n\n>W5(2),Q8: Studies of specific loss, phototype length.\n\nPhototype: The number of prototypes is determined by the prompts (reference images and masks) and is not a settable hyperparameter. The number equals the total count of different semantic categories in the in-context examples.\n\nLoss: Hungarian loss has been widely validated in object detection and segmentation[A,B,C]. We use this loss to align with Mask2Former and facilitate comparison (see Table 5(c)). Loss with one-to-many matching[D,E] is not suitable. Other hyperparameters, e.g., query dimension, are also aligned with Mask2Former to eliminate their impact.\n\n>W7,Q10: More visualization comparison results.\n\nFig.3 in attached PDF shows further visual comparisons. For video tasks, SINE reduces tracking failures from intersections, viewpoint changes, and occlusions. SINE addresses task ambiguity, preventing errors in semantic segmentation where SegGPT fails, as shown in the second set of comparison results in Fig.3(a). In real-world image segmentation, SINE exhibits better class generalization than SegGPT, matching the LVIS-92i comparison in Table 1.\n___\n\n>Limitation 1:The reviewer believes that the discussion is not proper.\n\n**We believe the discussion is proper.** We discuss the connections and differences between SINE and related works (SegGPT and SAM), and the limitations of SINE(Line478-494). We believe these discussions are necessary. **If the reviewer finds any inaccuracies**, please specify them in detail so we can further address them.\n\n>Limitation 2: Some potential negative impacts such as job displacement brought by generalist model should be discussed.\n\nWe do not foresee any obvious undesirable ethical or social impacts now. We believe that generalist models are powerful tools for efficient production and will create more job opportunities rather than causing job displacement.\n\n[A] End-to-end object detection with transformers. ECCV2020.\n\n[B] Per-Pixel Classification is Not All You Need for Semantic Segmentation. NIPS2021.\n\n[C] Masked-attention mask transformer for universal image segmentation. CVPR2022.\n\n[D] Mask r-cnn. ICCV2017.\n\n[E] SOLO: Segmenting Objects by Locations. ECCV2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# **General Response**

We thank the reviewers for recognizing that our paper points out an open research problem, i.e., the ambiguities in visual prompting (**nRMP**). The motivation on task ambiguity is clear (**rhP3**), and we effectively address task ambiguity (**sZmi,nRMP**). Our method is rationale and effective (**2YmX**), significantly improving performance across various segmentation tasks (**sZmi,nRMP**), and demonstrating solid practical utility (**rhP3**). 

In addition, we will carefully address the issues and suggestions raised by the reviewers and will make further revisions and improvements to our paper. In every Official Review of the reviewers below, we have provided responses to the questions and suggestions made by the reviewers, and we hope these responses adequately address the issues and concerns raised. **If the reviewers have any further questions regarding our paper and responses, please let us know.**
___

In the General Response, we highlight the contributions and insights of this work.

In-context segmentation is an important proxy task for unifying different segmentation tasks, and it has garnered significant attention from the research community. Our paper delves deeply into this task, and we summarize our **contributions**, **novelty**, and academic **insights** as follows:

1. **Ambiguities in Visual Prompting**: This paper is the *first to explore the task ambiguity problem in prompts within in-context segmentation* (supported by **sZmi** and **nRMP**). This is an **important** and **novel** issue that has been **underexplored**. We investigate the conflicts among multiple tasks in in-context segmentation from the perspective of task ambiguity and *provide effective solutions*, offering valuable insights to the research community.

2. **Investigating the Capabilities of VFMs**: Utilizing Visual Foundation Models (VFMs) to address various tasks is becoming a research trend. Our paper explores how to efficiently transfer the visual representations of VFMs to in-context segmentation, a research paradigm that *currently lacks extensive exploration in the in-context segmentation field*.

3. **Lightweight and Effective Decoder**: We thoroughly analyze the challenges brought by task ambiguity (see Lines 181-185) and design the M-Former structure to address these issues. The novel dual-path design and Shared SA in M-Former ensure efficient decoding while avoiding information confusion across tasks of different granularity. Additionally, unlike previous methods, SINE achieves significant performance improvements with fewer trainable network parameters. Thus, our method also demonstrates novelty.

Taken together, these contributions represent novel insights to the academic community.

___

The attached PDF contains 3 additional results:

1. Generalizability of SINE beyond semantically similar objects. (**sZmi**)

2. Visualization of part segmentation. (**sZmi**)

3. Visualization comparisons between SINE and SegGPT on video and image tasks. (**2YmX**)

Meta-review: This paper received split recommendations after the post-rebuttal discussion. The rebuttal adequately addressed most reviewer concerns about technical correctness and experimental validation. During an extensive author-reviewer discussion, the authors provided additional detailed analyses and explanations that satisfied three reviewers. Only Reviewer 2YmX recommended rejection on the grounds of insufficient technical novelty. All reviewers agreed on the clarity of the motivation, the quality of writing and presentation, and the adequacy of the experimental validation. Despite the lack of technical breakthroughs, the proposed solution to the problem of task ambiguity in in-context segmentation seems ingenious and effective enough to interest the broad NeurIPS community.
Decision: Accept (poster)