A Simple Image Segmentation Framework via In-Context Examples

Authors: yang  liu
chenchen  jing
hengtao  li
muzhi  zhu
hao  chen
xinlong  wang
chunhua  shen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 05:36:06

Abstract: Recently, there have been explorations of generalist segmentation models that can effectively tackle a variety of image segmentation tasks within a unified in-context learning framework. However, these methods still struggle with task ambiguity in in-context segmentation, as not all in-context examples can accurately convey the task information. In order to address this issue, we present SINE, a simple image $\textbf{S}$egmentation framework utilizing $\textbf{in}$-context $\textbf{e}$xamples. Our approach leverages a Transformer encoder-decoder structure, where the encoder provides high-quality image representations, and the decoder is designed to yield multiple task-specific output masks to eliminate task ambiguity effectively. Specifically, we introduce an In-context Interaction module to complement in-context information and produce correlations between the target image and the in-context example and a Matching Transformer that uses fixed matching and a Hungarian algorithm to eliminate differences between different tasks. In addition, we have further perfected the current evaluation system for in-context image segmentation, aiming to facilitate a holistic appraisal of these models. Experiments on various segmentation tasks show the effectiveness of the proposed method.

Main paper: 
 introduction: 
 Image segmentation [65, 31, 24, 58, 46] involves localizing and organizing concepts at the pixel level. Different definitions of concepts, such as foreground, category, and object instance, lead to different types of segmentation tasks. Recent years, we have witnessed great progress in developing more accurate and faster algorithms for various segmentation tasks such as semantic segmentation [35, 64, 48] , instance segmentation [19, 7, 2] , panoptic segmentation [24, 4, 5] , foreground segmentation [52] , interactive segmentation [57, 37, 25] . Nonetheless, most existing segmentation methods are tailored for certain tasks and cannot be applied to other tasks.More recently, a few works [54, 56] have explored generalist segmentation models that are capable of solving diverse and unlimited segmentation tasks via in-context learning [3] . Painter [54] performs in-context training with masked image modeling and can achieve various tasks according to the in-context visual prompts. SegGPT [56] focuses on visual segmentation and introduces in-context segmentation, which unifies multiple segmentation tasks by incorporating both a target image and an annotated reference image as input. To encourage the model to leverage contextual information, SegGPT employs a random coloring scheme during the task completion process. However, these models still struggle with task ambiguity in in-context segmentation.As shown in Figure 1 (a), the in-context segmentation model needs to understand the task and content information conveyed by the in-context example and segments related concepts on the target image. However, not all in-context examples can convey the task information accurately. For example, when presented with a photo of a particular individual for segmenting the target image that includes him/her and others, which visual concepts should be segmented? Specifically, should it be limited to the individual alone, encompass the instance segmentation of all persons, or focus on semantic segmentation? Whether the pixels of the target image need to be segmented depends on their similarity to the in-context example. Ambiguous in-context examples can make it difficult for traditional incontext segmentation models to clearly define the boundaries between different tasks, resulting in undesired outputs.To address this issue, we present SINE, a simple image Segmentation framework via IN-context Examples. Drawing inspiration from the segment anything model [25] , which addresses point ambiguity by generating multiple masks of different granularity, SINE predicts multiple output masks, custom-made for tasks of varying complexity. This ranges from identical objects, instances to overall semantic conception. SINE unifies existing segmentation tasks with various granularities, intending to achieve a broader task generalization. SINE leverages a Transformer encoder-decoder structure. The encoder contains a frozen pre-trained image encoder offering high-quality image representations and an In-context Interaction module to complement in-context information and learn correlations between the target and the reference image features. We propose a novel Matching Transformer (M-Former) for efficient multiple-task decoding. M-Former is implemented by a dual-path Transformer. One path is used for information interaction between object queries with image features. The second path is employed to enhance the semantic prototypes for accurate matching. In addition, a fixed matching and a Hungarian algorithm are used to eliminate differences between different tasks. SINE achieves impressive performance with fewer trainable parameters compared with SegGPT. Our qualitative results demonstrate that SINE can effectively address the problem of task ambiguity in in-context segmentation, while SegGPT only outputs the semantic segmentation results. SINE achieves state-of-the-art or competitive performance on existing in-context image segmentation benchmarks [56] , including few-shot semantic segmentation, video object segmentation. In addition, we further introduce few-shot instance segmentation to the current evaluation system for facilitating a holistic appraisal of these models. SINE provides a baseline result for in-context segmentation models to promote the development of this field. Finally, comprehensive ablation studies verify the effectiveness of the proposed components. Our main contributions are as follows:• To our knowledge, our method is the first to investigate the task ambiguity of in-context segmentation, and we present a simple but effective framework to address the issue. • We introduce a Matching Transformer to unleash the potential of frozen pre-trained image models on various segmentation tasks with a low training cost. • Our comprehensive results demonstrate that SINE can address a broad range of segmentation tasks, including few-shot semantic segmentation, few-shot instance segmentation, video object segmentation. Ablation studies show the effectiveness of the proposed components.Image Segmentation Segmentation involves localizing and organizing meaningful concepts at the pixel level. Different definitions of concepts, such as foreground, category, and object instance, lead to different types of segmentation tasks. Specifically, semantic segmentation [65] requires semantic classification at the pixel level, while instance segmentation [31] is to identify and localize various object instances. Panoptic Segmentation [24] introduces a more challenging task by unifying semantic segmentation and instance segmentation. Video object segmentation [58, 46] is to segment an identical object throughout a video sequence. Most existing segmentation methods are tailored for certain tasks and cannot be applied to other tasks.Traditional segmentation methods [35, 29, 64, 48, 19, 7, 2, 24, 4, 5, 10, 66] have been designed for identical segmentation tasks and trained on a specific dataset. These methods have limited generalization when transferring to other datasets or tasks. To solve this challenge, we unify various segmentation tasks into in-context segmentation [56, 34, 27, 39] and introduces a simple visual segmentation framework via in-context examples. Prompted by a given in-context example, our method can perform few-shot segmentation across various datasets and tasks.In-Context Visual Learning Recently, in-context learning has been successfully used in NLP tasks and vision-and-language tasks. Bar et al. [1] first introduced in-context learning in computer vision. They cast the in-context learning as an image inpainting problem, where models need to fill in a hole in a concatenated image containing several examples and a new input image. They demonstrate the effectiveness of the strategy for foreground segmentation, single object detection, and colorization.Painter [54] adopts masked image modeling on continuous pixels to perform in-context training with supervised datasets, and achieves competitive results on seven vision tasks. SegGPT [56] focuses on the segmentation task and uses random coloring scheme that forces the model to reference contextual information to complete the assigned task. DiffewS [67] effectively leverages in-context visual learning to unleash the potential of Stable Diffusion [49] in few-shot semantic segmentation.Our work focuses on segmentation tasks. These aforementioned models struggle with the task ambiguity in in-context segmentation. By contrast, our SINE can effectively address the problem of task ambiguity in in-context segmentation by disentangling the specific task from the in-context example and understanding the semantic concepts of the prompts to output results at different levels of task granularity from the identical object, instance, to semantics. 
preliminary: 
 In this section, we first formulate the problem setting of in-context segmentation. Then, we revisit SegGPT, the previous in-context segmentation model.In-context segmentation aims to identify a specific task and objects within the given in-context example, including a reference image x r and its annotations y r = {(m i r , c i r )} N i=1 , and segment the interested objects in the target image x t . m i r denotes the i-th reference mask and c i r denotes its class label, N denotes the number of the reference masks. The interested objects are related to the in-context example, which can be an identical object for video object segmentation or all objects of the same semantic concept for instance segmentation and semantic segmentation. We mainly focus on three semantic granularities in the task ambiguity, i.e., identical object (ID), instance, and semantic. The identical object segmentation can be seen as finer granularity instance segmentation, and instance segmentation can be transformed into semantic segmentation by merging instance masks belonging to the same category. Based on this relationship between these tasks, we unify them into instance segmentation.SegGPT [56] introduces in-context segmentation, which incorporates both the images and mask annotations into an RGB image to convey the specific task to be performed and identify the objects to be segmented simultaneously. SegGPT takes stitched reference and target images as input and employs the masked image modeling algorithm [18] and smooth-ℓ1 [15] loss to train a Vision Transformer [8] encoder. During inference, SegGPT enables segmenting everything via an in-context example, including a reference image and its annotation image. Given a target image, it is stitched with the reference image and fed into SegGPT to get the corresponding in-context predictions. Although SegGPT has achieved great success in various segmentation tasks, it faces two challenges: 1) In many cases, in-context examples may not accurately convey the task information. For example, 𝐅 " ∈ ℝ #×%×& … … 𝐭'( ∈ ℝ )×# 𝐭*+, ∈ ℝ -×# 𝐅 " . ∈ ℝ #×%×& 𝐪 '( ∈ ℝ )×# 𝐪'/0 ∈ ℝ 1×# 𝐩 0+, ∈ ℝ -×#In-Context Interaction when an in-context example only consists of a single object and its annotation, the lack of additional task-related information can lead to incorrect outputs. 2) SegGPT utilizes a single Transformer encoder structure for both feature extraction and task-specific decoding purposes. This will introduce complexity in the in-context segmentation process and lead to a sub-optimal solution.Inspired by the segment anything model [25] , which solves point ambiguity by outputting multiple different granularity masks simultaneously, we focus solely on the various content of in-context examples and endow the model with the ability to predict multiple output masks for different tasks.In addition, we disentangle the function of the Transformer encoder within SegGPT and deploy a Transformer encoder and decoder to perform feature extraction and task decoding, respectively.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: In the limitation part of the appendix.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 We present SINE, a simple image Segmentation framework via IN-context Examples. SINE can effectively address the problem of task ambiguity in in-context segmentation by disentangling the specific task from the in-context example and understanding the semantic concepts of the prompts to output results at different levels of task granularity from the identical object, instance, to semantics. We elaborate on our method in the following subsections.The overview of the proposed SINE framework is depicted in Figure 2 . We build SINE based on the classic Transformer structure [53, 8] and introduce some effective designs targeted for the in-context segmentation task, including a frozen pre-trained image encoder, an In-Context Interaction module, and a lightweight Matching Transformer (M-Former) decoder.We use the frozen pre-trained image encoder to encode x r and x t , resulting in the reference feature F r ∈ R C×H×W and target featureF t ∈ R C×H×W , where H, W , and C denote the image features' height, width and the number of channels, respectively. Inspired by the in-context learning in NLP [3] , we enable SINE to grasp the in-context correlations that coexist between the reference and the target images. Specifically, we develop an In-Context Interaction module, a component tailored to capture the semantic correlations between F r and F t and outputs the enhanced target feature F ′ t ∈ R C×H×W , the ID queries q id ∈ R N ×C and the semantic prototypes p sem ∈ R M ×C . SINE is built as a query-based segmentation model, following DETR [4] and Mask2Former [5] . We employ the ID queries q id to identify and locate objects within the target image that have identical counterparts within the reference image. Furthermore, we apply learnable instance queries q ins ∈ R S×C to identify and locate objects within the target image that share the same semantic labels in the reference image. The M-Fomer decoder is employed to update these queries and prototypes. Then, a prediction feed-forward network [4] is employed to predict ID and instance outputs, respectively.𝐅 , ∈ ℝ -×%×& 𝐭 !" ∈ ℝ #×- 𝐭 '() ∈ ℝ *×- 𝐅 + ∈ ℝ -×%×& 𝐪 !" ∈ ℝ #×- 𝐩 .() ∈ ℝ *×- 𝐅 + / ∈ ℝ -×%×& 𝐭 !" , 𝐭 '() … …The purpose of In-Context Interaction is to complement in-context information and produce correlations between reference and target image features. As illustrated in Figure 3 , we use a Mask Pooling process to extract the ID and semantic tokens, represented as t id ∈ R N ×C and t sem ∈ R M ×C . Specifically, we transfer m r into ID masks m id ∈ R N ×H×W by assigning different ID labels for each mask and semantic masks m sem ∈ R M ×H×W by merging the masks with same category label, where N and M are the numbers of ID and semantic masks. Then, we use these masks to pool over the reference feature F r and obtain t id and t sem , the pooling process similar to [14] .We introduce an In-Context Fusion module to enable the in-context correlation between the reference and target features. The process of this module can be summarized as follows:EQUATIONwhere θ is the parameters of the In-Context Fusion. The module is a Transformer block [53, 4] including a self-attention, cross-attention, and a feed-forward network. The tokens (t id and t sem ) and target feature (F t ) are fused by this shared module, where they are used as keys and values for each other in the cross-attention, and the enhanced target feature F ′ t , the ID queries q id and the semantic prototypes p sem can be obtained.M-Former aims to decode the enhanced target feature into different task outputs, from identical object, instance, to semantic, and enables comprehensive and efficient performance for in-context segmentation. Thus, in addition to the ID queries q id and the semantic prototypes p sem , we also incorporate a set of learnable instance queries q ins for predicting instances. To perform in-context segmentation and eliminate task ambiguity effectively, the design of M-Former needs to consider 1) the semantic prototypes need to assign semantic information to the instance queries, 2) avoiding coarse-grained semantic prototypes to contaminate fine-grained ID queries, and 3) the queries interact with the target feature.Encouraged by the above analyses, the M-Former is implemented by a dual-path Transformer decoder sharing the self-attention layers. One path is utilized to interact F ′ t and queries (q id and q ins ) for extracting correlated feature with the in-context example from the target image. This path consists of a series of self-attention, masked cross-attention [5] , and feed-forward network. The second path is employed to enhance the semantic prototypes p sem for a more accurate matching. The two paths share the self-attention layers for assign semantics from p sem to q ins . To avoid semantic prototypes contaminating fine-grained ID queries, we apply a self-attention mask within the shared self-attention layers as shown in at the top right corner of Figure 2 . M-Former has N blocks. The process of M-Former can be summarized as follows:EQUATIONwhere l denotes the layer number of M-Former and θ l is the parameters of the l-th M-Former block.Then, a prediction feed-forward network is employed to predict ID and instance outputs, respectively.For instance segmentation, we use the updated semantic prototypes p sem as the classifier and let ŷins = {ŷ i ins } S i=1 denote the set of S instance predictions. The ground truth is denoted by y. We use the Hungarian loss [4, 5] to learn SINE. Specifically, we compute the assignment costs between prediction ŷi ins and ground truth y j for the matching problem, i.e., -p i (c j ) + L mask ( mi ins , m j ), where (c j , m j ) is the class and mask of the ground truth object, c j may be ∅. p i (c j ) is the probability of class c j for i-th instance query, mi ins denotes its predicted mask. L mask is a binary mask loss and Dice loss [40] . The Hungarian loss isEQUATIONwhere σ(j) denotes the resulting index of the bipartite matching. To endow SINE with the ability to predict an identical object, we use the different cropped views of the same instance within the image as reference-target image pairs. Let ŷid = {ŷ i id } N i=1 denote the set of N ID predictions. Because the relationship between reference and target IDs is fixed and can be accurately determined, we perform fixed matching between the predictions and the ground truth. The loss can be written asEQUATIONwhere (c i , m i ) is the ground truth class and mask, c i ∈ {1, ∅}, c i = 1 denotes that an object appears in both reference and target images simultaneously. Training Details Following [34] , we deploy the frozen DINOv2 (ViT-L) [44] with 304M parameters as the image encoder and train SINE with only 19M trainable parameters. The In-Context Fusion has one block and M-Former has six blocks. The model size of SINE is comparable with SegGPT (307M). SINE has fewer trainable parameters, leading to more efficient training. We train SINE about 50K steps with 64 batch sizes. We use Adam [36] optimizer and employ β 1 = 0.9, β 2 = 0.999 for optimization. We use a linear learning rate scheduler with a base learning rate of 1e-4 and a warmup of 100 steps. The weight decay is set to 0.05. For data augmentation, we use random horizontal flipping and the large-scale jittering (LSJ) [13] augmentation with a random scale sampled from range 0.1 to 2.0 followed by a fixed size crop to 896 × 896. More implementation details are provided in the Appendix B.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
qualitative results: 
 We demonstrate that our SINE framework can effectively address the ambiguity in the in-context examples. As shown in Figure 4 (a), it is difficult to understand which tasks should be executed via the reference image with an annotation of a particular individual.SegGPT only outputs the semantic segmentation result. In contrast, SINE outputs multiple outputs to avoid task ambiguity. We further visualize the results of SINE on few-shot semantic segmentation, few-shot instance segmentation, and video object segmentation. SINE showcases its capability to deliver highly accurate predictions across diverse tasks while retaining exceptional flexibility in the task definition. Similarly, PASCAL-5 i is built on PASCAL, including four cross-validation folds. LVIS-92 i is a more challenging benchmark for evaluating the generalization of a model across datasets based on LVIS [16] , including ten folds and 92 classes for each fold. We comprehensively verify the few-shot semantic segmentation performance of SINE on these datasets following the evaluation scheme of [41, 56, 34] . Datasets Like few-shot semantic segmentation, we evaluate the performance of SINE of few-shot instance segmentation on both settings: in-domain on COCO-NOVEL [23] and out-domain on COCO2VOC [9] settings. COCO-NOVEL includes 20 classes that intersect with VOC and 5k test images. We evaluate 1-shot and 5-shot instance segmentation performance on this dataset. The COCO2VOC dataset is used to evaluate the cross-dataset generalization ability by using reference samples of COCO to test the VOC test set. We report the mean results of 10 groups of different reference images generated by different random seed for both two datasets.Results Table 2 shows the results of few-shot object detection and instance segmentation. SINE outperforms specialist methods by a large margin at both 1-shot and 5-shot settings. For the results of COCO2VOC in Table 3 , SINE shows better generalization ability compared to specialist models.Please note that the goal of this experiment is not to prove that SINE achieves better few-shot instance segmentation performance compared to specialist segmentation models. The existing in-context segmentation models, such as SegGPT, are unsuitable to perform instance segmentation. SegGPT needs to use a sliding window to traverse all grids to predict the objects. The post-processing is complicated and ineffective. SINE is the first in-context segmentation model that can address fewshot instance segmentation. We hope SINE can be a baseline for in-context segmentation models to promote the development of this field.Datasets For video object segmentation (VOS), we focus on the semi-supervised setting where the object masks of the first frame are provided, and the model is responsible for segmenting the object in all subsequent frames. We evaluate SINE on three validation datasets, including DAVIS 2017 [46] , DAVIS 2016 [45] , and YouTube-VOS 2018 [58] . We use the J score and the F score to evaluate the performance.Details To effectively perform SINE on the VOS task, we introduce a group of memory banks for each object, maintaining the intermediate predictions. We determine which frame to retain in the memory according to the classification and mask scores. Considering that objects are more likely to resemble those in adjacent frames, we apply a time-based decay ratio to the scores, gradually reducing its value. In addition, we store the reference image and mask in memory to solve the case where objects vanish and reappear.Results Table 4 compares the performance of VOS between SINE and different methods trained with or without video data. Without video data, SINE achieves competitive performance compared with the models trained with video data on DAVIS 2017. In addition, SINE outperforms recent generalist segmentation methods, such as Painter, SEEM, DINOv and PerSAM-F, on YouTube-VOS 2018. These results demonstrate that SINE has the potential to address video tasks.We investigate the performance of SINE when transferring to common segmentation tasks, such as ADE20K semantic segmentation and COCO panoptic segmentation. Unlike traditional pre-trained methods, e.g., MAE [18] , require to fine-tune all model parameters to downstream tasks. We verify that our method can be efficiently transferred to these tasks via parameter efficient fine-tuning (PEFT) methods [38] . Specifically, We deploy the popular PEFT method, LoRA [21] , on the frozen image encoder and fix the original parameters to test the transfer ability of SINE. We select rank 32 as the default setting. We train semantic prototypes and LoRA parameters for specific segmentation tasks. specialist segmentation models. It is worth noting that, unlike Mask2Former [5] , SINE does not use multi-scale features for better performance.Panoptic Segmentation Table 6 shows that SINE also significantly outperforms other generalist segmentation models on the COCO panoptic segmentation task, demonstrating that the proposed method can enable the learned Transformer decoder to apply to more complicated panoptic segmentation effectively. By fixing the overall model parameters and adding only a few of LoRA parameters, SINE achieves competitive performance with the best specialist segmentation model, Mask2Former.As shown in Table 7 , we conduct ablation experiments on LVIS-92 i for few-shot semantic segmentation, COCO-NOVEL dataset for few-shot instance segmentation, DAVIS 2017 for video object segmentation, and ADE20K for the semantic segmentation task, to thoroughly validate the effectiveness of the proposed components and the impact of training with different datasets. Unless specified, only fold0 and seed0 are used for the ablation on the few-shot semantic/instance segmentation.Effect of different training data. We train SINE on different data sources (Table 7a ). We validate that incorporating more diverse semantic segmentation data, such as ADE20K, helps improve the model. Furthermore, we demonstrate for the first time that adding additional detection data, such as Object365, greatly enhances the model's in-context segmentation capability.Ablation study of proposed components. Table 7c shows the impact of the proposed components and all experiments only use COCO as training data. The proposed In-Context Fusion module leads to improvements in mean mIoU, AP and J&F . Further enhancements are observed across all evaluation metrics when the M-Former is used. This demonstrates the effectiveness of the proposed components in improving the model's performance for various segmentation tasks.Varying the number of frames Table 7b demonstrates a gradual improvement in the overall segmentation quality as the number of frames increases from 1 to 6. Overall, using a moderate number of frames, such as 6 or 8, achieves optimal VOS performance on the DAVIS 2017 dataset.Varying the LoRA rank Table 7d demonstrates the effect of the LoRA rank on transfer performance. When the LoRA is 32, SINE achieves acceptable performance with few trainable parameters. Further increasing the rank to 64 only leads to a marginal improvement. Considering the trade-off between performance and the number of trainable parameters, we select rank 32 as the default setting. 
discussion and conclusion: 
 Conclusion In this work, we point out the task ambiguity in in-context segmentation for the first time and present SINE, which simultaneously predicts multiple task-specific masks to address this problem. Leveraging the efficient design, SINE can utilize few trainable parameters to exhibit strong segmentation abilities.We hope SINE can promote the development of in-context segmentation.Limitation and more Discussions are provided in Appendix A. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: In the discussions of the appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: All the things in this paper are credited and are the license and terms of use explicitly mentioned and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: C.1 Comparison of SINE and SegGPT without Objects365. C.2 Impact of Different Backbones.We select DINOv2-S, DINOv2-B, DINOv2-L, and CLIP-L [47] to explore the impact of different backbones. The conclusions are as follows: 1) DINOv2 is better than CLIP. DINOv2 achieves better performance than CLIP because it has general matching capabilities at both image and patch levels, allowing it to better understand complex contextual information between images. In contrast, CLIP captures image-text similarity, making it difficult to capture relationships between images, leading to poorer performance. 2) Larger DINOv2 model performs better: larger DINOv2 models have stronger representation capabilities, making it easier to capture contextual relationships, thus improving performance. This also indicates that SINE is scalable with the enhanced capabilities of the encoder. As shown in Fig. 5 , SINE can perform part segmentation like SegGPT when including PACO as training data. We evaluate SINE for one-shot part segmentation on PASCAL-Part [42] following Matcher [34] for the data pre-processing and evaluation. Compared with SegGPT [56] , SINE achieves competitive performance. C.4 Generalizability of SINE beyond Semantically Similar Objects.Fig. 6 shows SINE's capability in handling complex interaction relationships. In Fig. 6 (a), the reference consists of multiple images, each containing different objects (box, cup, keyboard, mouse). When using these as in-context examples, SINE can segment one or more semantically different objects on a desk. In Fig. 6 (b), with a reference containing only one object, the in-context example cannot represent complex interactions, and thus no segmentation result is provided. In Fig. 6 (c), replacing multiple single-object images with a single image containing multiple objects yields the same effective results.Fig. 7 shows further visual comparisons. For video tasks, SINE reduces tracking failures from intersections, viewpoint changes, and occlusions. SINE addresses task ambiguity, preventing errors in semantic segmentation where SegGPT fails, as shown in the second set of comparison results in Fig. 7 (a). In real-world image segmentation, SINE exhibits better class generalization than SegGPT, matching the LVIS-92i comparison in Table 1 . SINE can effectively alleviate the prompt ambiguity and produce more accurate segmentation results. We provide more visualizations for examplebased semantic segmentation in Fig. 8 , example-based instance segmentation in Fig. 9 , video object segmentation in Fig. 10 , semantic segmentation on ADE20K, and panoptic segmentation on COCO in Figure 11 . Justification: In the abstract and the end of the introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This paper received split recommendations after the post-rebuttal discussion. The rebuttal adequately addressed most reviewer concerns about technical correctness and experimental validation. During an extensive author-reviewer discussion, the authors provided additional detailed analyses and explanations that satisfied three reviewers. Only Reviewer 2YmX recommended rejection on the grounds of insufficient technical novelty. All reviewers agreed on the clarity of the motivation, the quality of writing and presentation, and the adequacy of the experimental validation. Despite the lack of technical breakthroughs, the proposed solution to the problem of task ambiguity in in-context segmentation seems ingenious and effective enough to interest the broad NeurIPS community.

Reviews: review: paper_summary: paper_summary: SINE aims to resolve the problem of task ambiguity in in-context segmentation, where previous models struggled to accurately infer tasks based on in-context examples alone. This ambiguity arises because traditional models often fail to distinguish between different segmentation tasks like semantic segmentation, instance segmentation, or identifying specific objects based on the context provided. SINE utilizes a Transformer-based architecture where the encoder generates high-quality image representations and the decoder produces multiple task-specific output masks. In-context Interaction Module further enhances the encoder's output by establishing correlations between the target image and in-context examples, helping to better define segmentation tasks. Matching Transformer (M-Former), a dual-path Transformer decoder, updates object queries and semantic prototypes for precise task execution. It incorporates fixed matching and the Hungarian algorithm to resolve differences between tasks, ensuring that the output aligns well with the given task. SINE achieves impressive performance improvements over previous in-context segmentation models like SegGPT, particularly in handling multiple segmentation tasks simultaneously. It is shown to effectively address the issue of task ambiguity, producing relevant segmentation masks that are more aligned with the semantic content of the images.
strength_weakness: strength_weakness: [Concerns on Generalizability: Complex Interaction Relationships Beyond Semantically Similar Objects] While this paper adeptly handles in-context instance or semantic segmentation, numerous open-vocabulary segmentation models already capably segment novel classes or objects. A crucial demonstration of an in-context segmentation model's comprehension of in-context samples should extend beyond merely identifying semantically similar objects provided by context information. An intriguing task would involve, for example, segmenting objects situated on a table across multiple images—such as a bottle, a plate, and a book in successive images—and understanding whether SINE can recognize and segment another object on a table in a subsequent image. This requires the model to understand the interaction, the relationships, etc, that goes beyond segmenting semantically similar objects. 

[Challenges with Complex In-Context Information] Often, a single pair of images suffices to provide the necessary in-context information reflecting the user's intent. However, more complex scenarios might require integrating multiple pairs of images to fully capture intricate in-context information. I am concerned about SINE's limitations in handling such complexity.

[Performance Discrepancy in Video Segmentation Compared to SegGPT] While SINE excels beyond SegGPT in instance segmentation on MSCOCO, it underperforms in video instance segmentation across most evaluated benchmarks. This result is concerning, considering video instance segmentation's unique demand for the model to sustain consistent correspondence across multiple video frames, a task more complex than segmenting static images. I am interested in understanding the specific reasons behind this performance gap.

[Unable to Handle Many Tasks SegGPT Supports]. SegGPT can actually support a lot more tasks, such as hierarchical in-context segmentation. I am concerned if SINE is overfitted on the instance/semantic segmentation tasks.
questions: questions: Minor question: Why is the target image the same as the raw image in Figure 4a)? 

Minor issues - typos: L307 "Results Table 7 compares ...." should be "Results Table 4 compares ...."? If it's not a typo, Table 4 was never discussed in the main paper.
limitations: limitations: Yes.
paper_summary: paper_summary: The paper proposes a generalist model for image segmentation named SINE, which unifies multiple image segmentation tasks into the common formulation of visual in-context learning. This work aims to identify and model the task of object reidentification to reduce ambiguities within the in-context examples. By incorporating the modeling of each specific segmentation task within the SINE architecture, it effectively improves upon existing generalist models and achieves strong performance across a wide range of segmentation tasks.
strength_weakness: strength_weakness: 1. This paper only addresses the ambiguity between instance and semantic segmentation. However, there is a broader ambiguity in visual prompts, such as spatial position, category, color, etc. The authors need to discuss these aspects in more detail.
2. Compared to SegGPT, SINE introduced additional Objects365 as extra training data. Although this was explained, it still seems to lack some fairness. Without using Objects365, can better performance be achieved than SegGPT? For example, by only using ADE20K, COCO, etc., and what is the performance on COCO-20i, PASCAL-5i?
3. Although the authors conducted numerous experiments, the impact of different backbones is missing. Since the backbone is frozen, different models might bring significant performance differences.
questions: questions: 1. The authors only conducted experiments on dinov2 vit-l. How do models of different sizes and different pre-training affect the results? For example, clip.
2. How does SINE perform few-shot learning? It seems that SINE can only accept a single reference image.
limitations: limitations: See weaknesses.
paper_summary: paper_summary: The paper proposes an image segmentation framework using in-context examples. To eliminate ambiguity from the in-context examples, multiple output masks are predicted. It uses a pre-trained image encoder to extract features from target and reference images, pools these features into ID and semantic tokens using the reference mask, and employs a Matching Transformer to decode the output masks. Experiments on various segmentation tasks demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: The novelty of the overall idea and network structure is limited. The idea to solve the ambiguity of prompts (in-context examples in this paper) by predicting multiple masks is from SAM. The network architectures are mainly based on DETR and Mask2Former. Though these choices are effective, they do not introduce innovations to the field. Overall, the paper shows solid practical utility,  and the limited novelty in its core idea and network design prevents me from giving it a higher rating. A stronger emphasis on introducing novel concepts or architectural advancements would enhance the impact and recognition of the work.
questions: questions: N/A
limitations: limitations: N/A
paper_summary: paper_summary: This paper proposes a generalist segmentation model, dubbed as SINE, for a variety of segmentation tasks.

The general idea is to harness the in-context examples, and to alleviate the task ambiguity.

Specifically, an in-context interaction module, a matching Transformer and a Hungarian algorithm is devised to realize the above objectives. 

Extensive experiments on a variety of datasets and segmentation tasks show its effectiveness.
strength_weakness: strength_weakness: - The motivation does not match the methodology design properly. Specifically, the authors claim that the proposed method focuses on alleviating the task ambiguity. Unfortunately, throughout the module design, the in-context fusion has little relevance to the task-level ambiguity. Besides, the two losses in the M-Former are still implemented on the object level.

- The proposed method clearly lacks theoretical insight on how the task ambiguity is modeled and handled in theory. The overall module designs are common in visual representation learning, which is less relevant to the task-specific guidance or the task-level ambiguity.

- The Transformer design along with the Hungarian algorithm is not uncommon in modern Transformer based pipeline design such as DERT.

- From the performance perspective, compared with the prior in-context segmentation methods, the proposed SINE does not show a clear improvement in many cases, for example:

(1) Table 1, few-shot segmentation, on three out of six experiments, SINE is weaker than either Painter, SegGPT or recent ICLR works. 

(2) Table 4, video object segmentation, the propose method is clearly inferior to the state-of-the-art by a large margin on two out of three datasets.

- The ablation study Table 5c is confusing, and more details or experiments need to be clarified. For example:

(1) In-context fusion is only a part of in-context interaction module, while in Table 5c the authors seem to only include the impact of fusion part.

(2) The specific components in M-Former also need further study, such as the specific loss functions and the phototype length and etc. 

- Eq.3 Hungarian loss. Do other types of commonly-used loss functions also achieve similar performance? More discussion and comparison is needed. 

- The visual results are not sufficient enough. Although the authors provide a lot of segmentation predictions in the supplementary material, only the results from the proposed method along with ground truth are provided. Please provide and compare the visual results from other state-of-the-art methods.
questions: questions: - Q1：The motivation does not match the methodology design properly. The in-context fusion has little relevance to the task-level ambiguity. 

- Q2: The two losses in the M-Former are still implemented on the object level.

- Q3: The proposed method clearly lacks theoretical insight on how the task ambiguity is modeled and handled in theory. The overall module designs are common in visual representation learning, which is less relevant to the task-specific guidance or the task-level ambiguity.

- Q4: The Transformer design along with the Hungarian algorithm is not uncommon in modern Transformer based pipeline design such as DERT.

- Q5: Limited performance: Table 1, few-shot segmentation, on three out of six experiments, SINE is weaker than either Painter, SegGPT or recent ICLR works.

- Q6: Limited performance: 

- Q7: Table 4, video object segmentation, the propose method is clearly inferior to the state-of-the-art by a large margin on two out of three datasets.

- Q8: Clarify ablation study in Table 5c. (1) In-context fusion is only a part of in-context interaction module, while in Table 5c the authors seem to only include the impact of fusion part.
(2) The specific components in M-Former also need further study, such as the specific loss functions and the phototype length and etc.

- Q9: Eq.3 Hungarian loss. Do other types of commonly-used loss functions also achieve similar performance? More discussion and comparison is needed.

- Q10: The visual results are not sufficient enough. Please provide and compare the visual results from other state-of-the-art methods.
limitations: limitations: Though providing a discussion, the reviewer believes that the discussion is not proper. 

Some potential negative impacts such as job displacement brought by generalist model should be discussed.

Comments: title: Re: Official Comment by Authors
Re: Official Comment by Authors
Re: More detailed explanation
Reply
More detailed explanation
The rebuttal does not address my substantial concerns
comment: We believe that our response has answered the reviewer's concern. In addition, **the reviewer's response at the last minute of ddl is irresponsible**.
Thanks for the authors for providing another round of rebuttal, which clearly addressed my major concerns **Q1** and **Q2**.

However, I still hold my view that the novelties of this work from both theoratical perspective and representation learning respective are limited, especially for the loss design and the DERT based pipeline and the use of multiple cross-attention.

Therefore, I decide to keep my initial rating, and could not vote acceptance.
Dear reviewer,

We sincerely thank you again for your great efforts in reviewing and discussing this paper. As the Author-Reviewer discussion period is going to end soon, we want to know whether the reviewer has any additional questions based on our previous response. Please don’t hesitate to let us know if you have any remaining concerns.

Thank you,

The authors
Thank you for your feedback and your great efforts. Any further questions/suggestions would be also appreciated.
Thank you very much for your efforts and time in further discussing our work. Below, we provide a more detailed clarification regarding the remaining concerns.

___

>Q1 No rigid defintion or formulation (part of continued Q1): Is there a mechanism to consider or to compute the confidence or the weight of each individual task? From the reviewer's understanding on ambiguity and uncertainty, such amubiguity can be well resolved usually with some ways to measure the confidence of per-task. Otherwise the overall problem or the task is still deterministic, and the amubity may only exists verbally, downgradng the overall task formulation. can find a way to model the ambiguity between different tasks. I am still confused on this aspect.

Different from uncertainty, to resolve the ambiguity requires an explicit specification of the prediction type (e.g. id/instance/semantic), instead of formulating the problem as a distribution estimation. Uncertainty estimation in segmentation task usually deals with pixel level empirical prediction risks induced either by the data or the model [A].

Because of this inherent difference, our task ambiguity is not modelled in a similar way as the uncertainty. 

[A] What uncertainties do we need in bayesian deep learning for computer vision. NIPS2017.


>Object level learning (Q2): It is still objective-level, and does not connect to the relation to task-level. Especially, some saying such as * enable effective information interaction and preventing task interference* is super volid, and lack clear explanation or explanation.

The input to M-Former includes ID queries ($q_{id}$), instance queries ($q_{ins}$), and the semantic prototype ($p_{sem}$), as shown in Fig. 2 of the paper. **$q_{id}$ and $q_{ins}$ are used for predicting ID segmentation and instance segmentation, respectively, and are task-specific.** To prevent information interference between $q_{id}$ and $q_{ins}$ / $p_{sem}$, we utilize shared SA with the well-designed attention mask to avoid their interaction.

This object-level semantic segmentation approach significantly reduces fragmented masks, balances the training of masks of different sizes, and facilitates the unification of different tasks. Therefore, SINE also adopts an object-level approach for learning semantic segmentation.
By enabling interaction between instance-level $q_{ins}$ and semantic-level $p_{sem}$, we establish a stronger connection between instance segmentation and semantic segmentation. 

Thus, M-Former connects to the relation to task-level and effectively facilitates the interaction between $q_{ins}$ and $p_{sem}$ while preventing interference between $q_{id}$ and $q_{ins}$ / $p_{sem}$.


>Representation learning novelty (Q3&Q4): I still believe the novelties from both theoratical perspective and representation learning respective. This is particularly important, especially when the task formulation in Q1 and Q2 are still not well addressed.

Different from uncertainty, which models probabilistic distribution, SINE explicitly presents all prediction results to address task ambiguity, thereby enhancing the practical usability of in-context segmentation: In widely used segmentation tasks such as semantic and instance segmentation, traditional methods require assigning a category label to the object, which becomes challenging when scaled to thousands of categories. *In-context segmentation overcomes these limitations by using visual concepts as annotations.* However, ambiguous prompts in in-context segmentation can lead to incorrect predictions, limiting its practical use. We are the first to consider and resolve task ambiguity in prompts for in-context segmentation, a significant contribution in both research and application, and we believe this challenge is inherently novel.

In exploring the potential of DINOv2 for in-context segmentation, the challenge lies in how to maximize the reuse of the foundational model's capabilities across different tasks while still distinguishing the model's decisions for each task. This is a technical issue that cannot be solved by simply using methods like DETR. For example, as shown in Fig. 1 of the paper, an ID query for Prof. Geoffrey Hinton might get confused with broader information about the "person" category when using the traditional DETR decoder to process ID queries, instance queries, and semantic prototypes simultaneously. These technical challenges motivated us to propose the novel M-Former design.

As shown in the table below, compared with recent method that directly adopt the DETR structure without considering task ambiguity, SINE achieves significant performance improvements.

||DAVIS|YouTube-VOS|
|---|---|---|
|DINOv [A] | 73.3|60.9|
|SINE|77.0|66.2|

[A] Visual in-context prompting. CVPR 2024.

Based on the considerations mentioned above, our insights are not incremental but provide valuable contributions to the research community.
Thanks for your extensive effort to clarify the rest questions. 
However, there are still multiple questions need to be clarified before considering approching the borderline threshold.

**No rigid defintion or formulation (part of continued Q1):** Most of the concerns have been well addressed. However, regarding the explanation *SINE performs inference on any dataset by predicting all three tasks simultaneously*.

- Is there a mechanism to consider or to compute the confidence or the weight of each individual task?

- From the reviewer's understanding on ambiguity and uncertainty, such amubiguity can be well resolved usually with some ways to measure the confidence of per-task. Otherwise the overall problem or the task is still deterministic, and the amubity may only exists verbally, downgradng the overall task formulation.

- *can find a way to model the ambiguity between different tasks*. I am still confused on this aspect.

**Object level learning (Q2)**: The first question has been addressed, and thanks for the authors. I appreciate the author's effort to provide extensive details on *Dual-Path Structure* and *shared SA*.

- However, it is still *objective-level*, and does not connect to the relation to *task-level*. 

- Especially, some saying such as * enable effective information interaction and preventing task interference* is super volid, and lack clear explanation or explanation.

**Representation learning novelty (Q3&Q4)**: Thanks for the clarification. However, as I mentioned earlier and also raised by another reviewer today, I still believe the novelties from both theoratical perspective and representation learning respective. This is particularly important, especially when the task formulation in Q1 and Q2 are still not well addressed.

**Limited performance (Part of Q5&Q7):** I think at this stage these concerns have been well addressed. Please enrich the discussion accordingly. 

**Other Minor Issues such as losses and limitation**: Thanks for the authors to well address these issues. Yes, as I mentioned earlier, the discussion of different loss is a minor issues, which may be resolved in the future or even after the current form of the work.
I would like to thank the authors for their responses. Most of my questions have been addressed. Therefore, I will maintain my current rating as borderline accept.
Thank you very much for your efforts and time again! This rebuttal and discussion will be helpful to improve our revised manuscript.
Thank you to the author for their detailed analysis and explanation. In my opinion, the insights and novelty presented in the paper are limited, leading me to consider it as a borderline-to-rejection paper. I appreciate the extensive effort put into evaluating the proposed method. As a result, recognizing that the method is demonstrated as a strong baseline, I have adjusted my rating to borderline.
Thank you very much for your efforts and time in further discussing our work. 
Below, we provide a more detailed clarification regarding the remaining concerns.

___

>Compared with SAM, the contextual information seems heavily dependent on ground truth reference masks, whereas SAM can progressively segment objects in multiple turns. Other segmentation models, like those from co-segmentation, only require image groups as inputs. These examples make the claim confusing to me.

We will address the reviewer's question from the following two points. If our explanation is unclear, we would welcome further discussion.

1. *SINE is more cost-effective and broadly applicable.*
When using SAM for processing a large number of images, it requires human interaction to segment each image, leading to significant labor costs. Co-segmentation requires ensuring that a group of images contains objects with the same semantic conception and cannot segment multiple objects with different semantics in complex scenes simultaneously. This limitation hinders its widespread application. In contrast, SINE only needs a single in-context example containing a reference image and mask to handle target images in different tasks without human intervention, making it more cost-effective and broadly applicable.

2. *Compared with SAM, why does SINE need contextual information.*
For each image, SAM segments objects based on human interaction without learning semantics. This allows SAM to progressively segment objects in multiple turns, but its drawback is the need for human input for every image, leading to significant labor costs. In contrast, SINE can batch-process images using just one image and its mask, a practical advantage that SAM lacks. To achieve this, SINE needs to understand the contextual relationship between the reference image and target images. In fact, **SAM and SINE represent two vertically developed directions** of segmentation foundation models and can complement each other. For instance, in auto-labeling, SAM could label objects (e.g., a dog) in the first image, and SINE could use that image as an in-context example to label subsequent images, reducing costs.

>I agree with the author that we cannot solely focus on network architecture design in the era of large models. However, the insights presented appear to be only incrementally novel, as they mainly build on multiple previous works.

We appreciate the reviewer's agreement with our perspective. Below, we explain why our insights are not incremental:

In widely used segmentation tasks such as semantic and instance segmentation, traditional methods require assigning a category label to the object, which becomes challenging when scaled to thousands of categories. *In-context segmentation overcomes these limitations by using visual concepts as annotations.* However, ambiguous prompts in in-context segmentation can lead to incorrect predictions, limiting its practical use. We are the first to consider and resolve task ambiguity in prompts for in-context segmentation, a significant contribution in both research and application, and we believe this challenge is inherently novel.

In exploring the potential of DINOv2 for in-context segmentation, the challenge lies in *how to maximize the reuse of the foundational model's capabilities across different tasks while still distinguishing the model's decisions for each task*. This is a technical issue that **cannot be solved by simply using methods like DETR**. For example, as shown in Fig. 1 of the paper, an ID query for Prof. Geoffrey Hinton might get confused with broader information about the "person" category when using the traditional DETR decoder to process ID queries, instance queries, and semantic prototypes simultaneously. **These technical challenges motivated us to propose the novel M-Former design**:

- Dual-Path Structure: M-Former's input includes queries and prototypes. Queries handle instance-level predictions, while prototypes, derived from the reference image, maintain semantic-level category information. SINE processes them separately, forming a dual-path structure to enable query learning and avoid interference from prototype's semantic-level information.

- Shared SA: Queries include both ID and instance queries, representing different levels of granularity. To prevent interaction between ID queries and coarse-grained instance queries or prototypes, we introduce a well-designed attention mask (Fig. 2, top right) in the shared SA to effectively address this issue.

As shown in the table below, compared with recent method that directly adopt the DETR structure without considering task ambiguity, SINE achieves significant performance improvements.

||DAVIS|YouTube-VOS|
|---|---|---|
|DINOv [A] | 73.3|60.9|
|SINE|77.0|66.2|

[A] Visual in-context prompting. CVPR 2024.

Based on the considerations mentioned above, our insights are not incremental but provide valuable contributions to the research community.
Thank you to the author for the rebuttal. My concerns have been partially addressed.
- "SINE must learn the complex contextual relationships and avoid information confusion across different tasks"

Compared with SAM, the contextual information seems heavily dependent on ground truth reference masks, whereas SAM can progressively segment objects in multiple turns. Other segmentation models, like those from co-segmentation, only require image groups as inputs. These examples make the claim confusing to me.

- "This paper is the first to explore the task ambiguity problem in prompts within in-context segmentation"

I agree with the author that we cannot solely focus on network architecture design in the era of large models. However, the insights presented appear to be only incrementally novel, as they mainly build on multiple previous works.
>M-Former introduces a dual-path decoder and shared self-attention with a mask (Fig. 2, top right), enabling effective information interaction and preventing task interference. This explanation does not convince me...

In M-Former, interactions may occur between ID/instance level queries, semantic level prototypes, and image features. M-Former avoids the contamination of information from different granularities, thereby benefiting task performance. Specifically:

- Dual-Path Structure: M-Former's input includes both queries and prototypes. The queries are responsible for instance-level predictions, while the prototypes, derived from the reference image, need to maintain the semantic-level category information from the reference. Since their roles differ, SINE separates the processing of queries and prototypes, forming a dual-path structure. The query branch includes SA, CA, and FFN, ensuring effective information interaction between the queries and image features, allowing the model to learn object-level information. This process avoids interference from the semantic-level information contained in the prototypes.

- Shared SA: To keep the model lightweight, M-Former shares the SA between queries and prototypes. The queries include both ID and instance queries. While both operate at the object level, they represent different levels of information granularity. For example, as shown in Fig. 1 of the paper, an ID query might represent Prof. Geoffrey Hinton, whereas an instance query could represent any object categorized as a "person." The prototype, meanwhile, represents the average information for the category "person." If the ID query for Hinton were to be confused with broader information, it might lead to incorrect predictions. Therefore, it’s crucial to prevent interaction between ID queries and the more coarse-grained instance queries or prototypes. A well-designed attention mask (Fig. 2, top right) effectively mitigates this issue.

In summary, M-Former uses a dual-path decoder and shared SA to enable effective information interaction and preventing task interference. In contrast, DETR and Mask2Former's uniform approach to queries and prototypes of different granularities leads to information confusion and reduced performance.

>Representation learning novelty (Q3&Q4): There is so far no strong argument to address this perspective. 

We acknowledge that SINE uses a Transformer decoder and Hungarian loss, methods employed by DETR, Mask2Former, and other recent approaches. However, unlike previous work mainly focused on model structure improvements, our paper addresses task ambiguity in in-context segmentation and how to effectively perform in-context segmentation with the DINOv2 representation. Utilizing Visual Foundation Models for various tasks is a growing trend. For instance, DINOv2+reg (ICLR 2024, Outstanding Paper Award) explores how DINOv2 effectively performs unsupervised object discovery.Additionally, M-Former was designed to better prevent task interference. Based on these contributions, we believe our work meets top-tier conference standards.


>Limited performance (Part of Q5&Q7): The explanation does not alter the fact of limited performance on three out of six experiments. Maybe the authors have to make significant revision on this aspect...

SegGPT indeed achieved better few-shot performance on COCO-20i and PASCAL-5i due to its use of these datasets and the Context Ensemble strategy. However, SINE demonstrates superior class generalization. Evaluating both methods on LVIS-92i, which has 920 categories, shows that SINE outperforms SegGPT by about 10% in both one-shot and few-shot settings. This likely reflects SegGPT's overfitting to the 80 and 20 categories of COCO and PASCAL.

On LVIS-92i, SINE only lags behind Matcher, which uses SAM-H, a model trained with more data (SA1B including 1B masks) and more parameters (945M), making a direct comparison somewhat unfair. For a fairer comparison, we replaced SAM-H with SAM-B using Matcher's public code. The table below shows that SINE performs better with fewer parameters on LVIS-92i fold0.

||params|LVIS-92i|
|---|---|---|
|Matcher SAM-H|945M|31.4|
|Matcher SAM-B|398M|28.1|
|SINE|317M|28.3|

We will add more discussion in the revised version.


>Regarding the loss design: Can some other common mechanisms such as IoU based Non-Maximum Suppression...

The Hungarian loss ensures each query matches exactly one ground truth, eliminating the need for NMS. While SINE could theoretically achieve one-to-many matching, allowing each ground truth to match multiple queries. However, we don't have enough time for this experiment during the discussion phase. That's really not the point of this paper either. In addition, addressing task ambiguity and interference through the loss function is a novel approach we will explore in future research.
Thank you very much for your efforts and time in further discussing our work. Below, we provide a more detailed clarification regarding the remaining concerns.

>Q1 No rigid defintion or formulation (part of continued Q1):

Our goal is to address the ambiguity problem by using the following formulation, which outputs the results for all tasks simultaneously:

$f(x_r, y_r, x_t) \rightarrow \{ y_t^{ID}, y_t^{Ins}, y_t^{Sem} \}$

As the reviewer mentioned, since SINE’s learning process operates at the object level (including ID and instance), the formulation for SINE is:

$f(x_r, y_r, x_t) \rightarrow y_t \rightarrow \{y_t^{ID}, y_t^{Ins}\}$

The losses for instance and ID segmentation are clear and well-defined, as shown in Equations (3) and (4) of the paper. Therefore, the above formulation holds. We believe the primary concern here lies with semantic segmentation.

For semantic segmentation, SINE does not directly output $y_t^{Sem}$. Instead, during inference, we can obtain it by merging the instance segmentation predictions $y_t^{Ins}$. Specifically,

$y_t^{Ins} = \{ P_{mask}^{Ins} \in R^{S \times  H \times W}, P_{class}^{Ins} \in R^{S \times M} \}$  

$P_{mask}^{Ins}$ and $P_{class}^{Ins}$ represent the mask and class predictions for instance segmentation. Here, $S$ is the number of instance queries, and $M$ is the number of semantic prototypes, i.e., the number of candidate categories.

The semantic segmentation result $y_t^{Sem} \in  R^{M \times  H \times W}$ can be concisely expressed using the following matrix multiplication formula:

$y_t^{Sem} = P_{class}^T \times P_{mask}^{Ins} $

Where $P_{class}^T \in R^{M \times S}$ is the transpose of $P_{class}^{Ins} $, $ y_t^{Sem} \in R^{M \times H \times W}$ is the semantic segmentation result before applying argmax.

This formula indicates that the segmentation map for each category $m$, denoted as $y_t^{Sem}[m]$, is obtained by a weighted sum of all instance masks $P_{mask}[s]$ with the corresponding class probabilities $P_{class}[s, m]$.

Therefore, the SINE formulation can be expressed as:

$f(x_r, y_r, x_t) \rightarrow \{ y_t^{ID}, y_t^{Ins}, y_t^{Sem} \}$

**Through the above derivation, we hope to provide a clearer answer to the reviewer's question.**

>Is it really true that SINE is a one-to-three mapping?

Yes, during the inference stage, SINE can provide $ \{y_t^{ID}, y_t^{Ins},y_t^{Sem} \}$ for any input $(x_r, y_r, x_t)$.

>In the experimental table, you still do the inference on one task/dataset one time, right?

 SINE performs inference on any dataset by predicting all three tasks simultaneously. We select the relevant output based on the specific task. For instance, Table 1 uses semantic segmentation $y_t^{Sem}$ ​, Tables 2 and 3 use instance segmentation  $y_t^{Ins}$, and Table 4 uses Object ID $y_t^{ID}$ on the VOS dataset. Thank you for pointing this this. We will clarify it further in the revised version.

 >From my view, the proposed SINE can implictly learn a rank between different types of tasks. 

 The above derivation shows that SINE provides results for all tasks simultaneously, rather than learning a rank between different types of tasks.

>On top of this, perhaps the authors can find a way to model the ambiguity between different tasks, which can in turn help explain the proposed SINE from a more theortical perspective.

Thank you for the helpful suggestions in formulating SINE. We believe this makes SINE clearer and more interpretable from a theoretical perspective.

>Object level learning (Q2): There are still some remaining issues before resolving this question: In my view, instance in instance segmentation is typically object-level. Could the authors maybe justify more on Using instance/object level segmentation allows for unified loss forms across different tasks?

Beyond the previous clarification of SINE's formulation, we would like to further address the reviewer's concerns.

SINE targets three tasks: ID segmentation, instance segmentation, and semantic segmentation. During training, we focus on learning instance and ID segmentation, as defined by Equations (3) and (4). The differences in losses lies only in the matching strategy between predictions $y_t$ and ground truth, while the loss functions remain the same, making the form of the two losses unified. Semantic segmentation is derived from instance segmentation results by combining masks of instances within the same category.
First of all, I would appreciate the authors' effort, so that we could start a constructive discussion on the weakness and its improvement.

Regarding the specific responses, I've spent some time on it, and conclude that there are still multiple issues need to be clarified / resolved before raising up to the accept threshold.

**No rigid defintion or formulation (part of continued Q1)**: 

- The SegGPT formulation is fine, as it is still an one-to-one mapping. However, is it really true that SINE is a one-to-three mapping? From the reviewer's understanding, in the experimental table, you still do the inference on one task/ dataset one time, right? If so, this is not a one-to-three mapping, and the correctness remains doubted.

- From my view, the proposed SINE can implictly learn a rank between different types of tasks. Could the authors re-consider the formulation, and try to model the score of each task? In this way, it will improve the clarity and also make sense to the distingushment between different tasks. 

- On top of this, perhaps the authors can find a way to model the ambiguity between different tasks, which can in turn help explain the proposed SINE from a more theortical perspective.

I would expect if some aspects can be made on the above points, so as to improve both the clarity and the theotical insight.

**Object level learning (Q2)**: There are still some remaining issues before resolving this question:

- In my view, *instance* in instance segmentation is typically object-level. Could the authors maybe justify more on *Using instance/object level segmentation allows for unified loss forms across different tasks*?

- *M-Former introduces a dual-path decoder and shared self-attention with a mask (Fig. 2, top right), enabling effective information interaction and preventing task interference*. This explanation does not convince me. The interaction is still between instance representations, right? It just exploits the long-rang dependencies between different instances. How could it benefit the task level?

**Lightweight and Effective Decoder**: This concern has been well addressed, after having a comparison with existing paradigm.

**Representation learning novelty (Q3\&Q4)**: There is so far no strong argument to address this perspective. As the representation learning pipeline is very ordinary and common, and no theoretical insight can be demonstrated so far, I still believe this work is typically beyond the standard of top-tier conference.

**Limited performance (Part of Q5\&Q7)**: The explanation does not alter the fact of limited performance on three out of six experiments. Maybe the authors have to make significant revision on this aspect in the main text, to discuss what is the reason.
Besides, if there is some way to make a fair evaluation between them?

**Regarding the loss design**: Can some other common mechanisms such as IoU based Non-Maximum Suppression, Greedy Algorithm, Positive Sample Mining be inspected in the loss design? Anyway, this is a minor issue, but I insist that the loss and overall idea is ordinary. 

**Limited Visual Results**: Hope the authors can further address this part later, as it is still a minor weakness of this work.

**Limitation Discussion**: Thanks the authors for acknowledging this aspect and making the adapations accordingly.
Thank you for the reviewer's recognition of our work. And we promise to open-source our code.
Thank you for your comments. Due to the rebuttal's character limit, we couldn't fully address your concerns. Below, we provide a detailed explanation.

>No rigid defintion or formulation is given, especially to the level of task. Neither does the substantial relevance to the task-level ambiguity.

The formulation of SegGPT can be represented as:

$f(x_r, y_r, x_t) \rightarrow y_t,~~~   y_t \in \{task_1, task_2, task_3\}$

When the given prompt $(x_r, y_r)$ cannot precisely indicate a specific task, the prompt is ambiguous, as shown in Fig. 1 of the paper. When SegGPT performs a task (e.g., $task_1$) with an ambiguous prompt, it might incorrectly output the results of $task_2$ or $task_3$. **This is an important and unexplored problem.**

The formulation of SINE is as follows::

$f(x_r, y_r, x_t) \rightarrow \{ y_t^{task_1}, y_t^{task_2}, y_t^{task_3} \}$

By providing results for all tasks, SINE avoids incorrect predictions caused by prompt ambiguity. Fig. 4(a) in the paper presents a visual comparison.

>The text still considers the object level learning. The two losses in the M-Former are still implemented on the object level. 

We address the ambiguity between ID, instance, and semantic segmentation. These tasks can be converted into instance segmentation. Using instance/object level segmentation allows for unified loss forms across different tasks. M-Former introduces a dual-path decoder and shared self-attention with a mask (Fig. 2, top right), enabling effective information interaction and preventing task interference. This balances in-context segmentation decoding and resolves prompt ambiguity.

Our design unifies training for different tasks and prevents task information contamination during decoding.  During inference, the trained network can perform different tasks, which we believe is an advantage of our method.

>Lightweight and Effective Decoder

Compared to SegGPT's 300M training parameters, our model has only 19M. Without the encoder, Mask2Former has 21M parameters. Thus, our method is more lightweight.

> We do not claim the loss as a contribution does not necessarily mean the devised representation learning pipeline is techniqually novel. Would the authors acknowledge a paper's novelty and significance, if it claims to use U-Net with minor modifications for another task / application such as industrial segmentation for the first time?


We believe **the value of a paper lies in whether it provides academic insights to the research community.** For example, Marigold (CVPR 2024, Best Paper Award Candidate) first demonstrated stable diffusion's significant generalization in depth estimation. However, Marigold made no changes to U-Net's structure or training strategy. This does not change our view that Marigold is an outstanding and inspiring work.

In addition, our core claim is that we are the first to identify and explore task ambiguity in prompts, rather than making minor modifications for in-context segmentation. **We believe this will inspire other work.** The General Response summarizes our contributions and insights.

> Table 1, few-shot segmentation, on three out of six experiments, SINE is weaker than either Painter, SegGPT or recent ICLR works. 

1. SINE outperforms Painter on all datasets.
2. SegGPT is trained on COCO and PASCAL and uses a Context Ensemble strategy for few-shot settings. SINE is not trained on PASCAL and is not designed for few-shot settings. However, SINE achieves the best performance on 1-shot in COCO-20i and PASCAL-5i, and is comparable to SegGPT in few-shot settings.
3. On LVIS-92i, SegGPT is only weaker than Matcher because Matcher uses SAM, and its larger training set better aligns with the LVIS.

> Eq.3 Hungarian loss. Using the same loss as other works for fair evaluation is not a proper excuse to not study the impact of some common loss types. This aspect is unaddressed.

In the rebuttal, we explained that Hungarian loss is essential for Detection/Segmentation Transformer methods. These methods model detection/segmentation task as a set prediction problem, introducing one-to-one Hungarian matching. Traditional methods use a one-to-many matching for loss, which requires NMS and is not applicable here. Could the reviewer specify which loss should be used for comparison?

> Limited visual results.

The visual results in the rebuttal are limited to the **one-page** PDF requirement. We will provide more visual comparisons in the paper.

> Limitation: Efficient production definately leads to the loss of some old, tradtional and off-the-shelf jobs. This does not contradict create more job opportunities. Why is it difficult for the authors to acknowledge this aspect?

We will add this discussion in the paper. However, almost all AI model developments may cause such issues in the short term, but we cannot stop the advancement of AI because of this. We believe that, in the long run, the benefits of developing general models for human society outweigh the drawbacks.
The rebuttal has addressed my main concerns. Since the contribution of unifying segmentation tasks with in-context examples is clear, novel, well motivated, and well demonstrated, I would like to keep my original rating and recommend to accept this paper. I hope the authors can release their source code to benefit researchers in the same domain.
Thanks the authors for the rebuttal.

Unfortunately, the rebuttal keeps repeating the comments and views from other reviewers, instead of seriously addressing the specific concerns raised by me. For example:

- **Q1**：I am not convinced from the text from the submission mentioned by the authors. The text still considers the object level learning. No rigid defintion or formulation is given, especially to the level of task. Neither does the substantial relevance to the task-level ambiguity.

- **Q2**: *The two losses in the M-Former are still implemented on the object level.* How do them relate to the task-level and the ambiguity? The rebuttal does not directly answer this question. Instead, it just keeps highlighting the view from other reviewers and mentions some void and lackluster general contributions.

- Besides, the authors in the rebuttal claim *Lightweight and Effective Decoder*. But the evidence in the main text supporting this aspect is not enough. The parameter comparison is only an ablation study, not with other types of decoders.

- **Q3**: The authors keep repeating the application value and insight in the rebuttal. However, how is the task ambiguity modeled, formulated and defined, and whether some theorical support can be found, are neither clarified.
Besides, the concern *The overall module designs are common in visual representation learning, which is less relevant to the task-specific guidance or the task-level ambiguity* is unaddressed.

- **Q4**: *We do not claim the loss as a contribution* does not necessarily mean *the devised representation learning pipeline is techniqually novel*. The authors do not address ths aspect directly in the rebuttal. For example, would the authors acknowledge a paper's novelty and significance, if it claims to use U-Net with minor modifications for another task / application such as industrial segmentation for the first time?

- **Q5 \& Q7**: I acknowledged that in some tables the proposed SINE shows state-of-the-art performance. However, my question also raises. *Table 1, few-shot segmentation, on three out of six experiments, SINE is weaker than either Painter, SegGPT or recent ICLR works.* The authors do not address this in the point-by-point response.

- I acknowledge from the rebuttal that the average performance is the best, but this does not contradict to the fact that, it shows significantly inferior performance in many of my listed settings. In fact, it further raises the concerns on whether it is stable or generalized enough. 

- **Q8 \& Q9**: *Using the same loss as other works for fair evaluation* is not a proper excuse to not study the impact of some common loss types. This aspect is unaddressed.  

- **Q10**: Only limited visual results on limited datasets from SegGPT is provided. My concerns on more visual results from state-of-the-art methods are not addressed. 

- Minor issue: **Limitation**: Efficient production definately leads to the loss of some old, tradtional and off-the-shelf jobs. This does not contradict *create more job opportunities*. Why is it difficult for the authors to acknowledge this aspect?

To conclude, my substantial concerns are not properly addressed. I keep my original rating and recommend clear reject this paper.