Order-aware Interactive Segmentation

Authors: Bin Wang, Anwesa Choudhuri, Meng Zheng, Zhongpai Gao, Benjamin Planche, Andong Deng, Qin Liu, Terrence Chen, Ulas Bagci, Ziyan Wu
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Interactive Segmentation, Image Segmentation

Abstract:
Interactive segmentation aims to accurately segment target objects with minimal user interactions. However, current methods often fail to accurately separate target objects from the background, due to a limited understanding of order, the relative depth between objects in a scene. To address this issue, we propose OIS: order-aware interactive segmentation, where we explicitly encode the relative depth between objects into order maps. We introduce a novel order-aware attention, where the order maps seamlessly guide the user interactions (in the form of clicks) to attend to the image features. We further present an object-aware attention module to incorporate a strong object-level understanding to better differentiate objects with similar order.  Our approach allows both dense and sparse integration of user clicks, enhancing both accuracy and efficiency as compared to prior works.  Experimental results demonstrate that OIS achieves state-of-the-art performance, improving mIoU after one click by 7.61 on the HQSeg44K dataset and 1.32 on the DAVIS dataset as compared to the previous state-of-the-art SegNext, while also doubling inference speed compared to current leading methods.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 5, 5, 8

Reviews and discussion:
<thread>
creation date:
1729774938155

thread_type:
official_review

<thread_object>
creator: reviewer_dpql

<content>
summary:
This work studies the interactive segmentation task. Based on the prior knowledge that ground and background objects are located at different depths, this work proposes a new framework named OIS. OIS effectively takes advantage of a corresponding depth map via the proposed order- and object-aware attention. Experiments demonstrate that OIS effectively improves segmentation performance by fewer clicks and boosts inference speed.

strengths:
The manuscript is clearly written and easy to follow. The whole framework is built upon the fact that different objects should be located at various depths in the original 3D scenes, which is promising. Both qualitative and quantitative comparisons demonstrate its superior performance.

weaknesses:
Overall, I feel quite OK with this work, and there are no MAJOR weaknesses. See below for several suggestions and typos.

- It is better to visualize the five clicks in Fig. 1 for an improved presentation.
- Paragraph 2, Sec. 1, Page 1: Redundant '(' before 'RITM'.
- Paragraph 2, Sec. 1, Page 2: Better to delete 'However' before 'current methods fail to ...'.
- The blue dots in Fig. 3 are inconspicuous. Please consider another conspicuous color.

questions:
Please refer to the WEAKNESSES part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730440557456

thread_type:
official_review

<thread_object>
creator: reviewer_pfaj

<content>
summary:
Authors propose the order-aware interactive segmentation (OIS) to explicitly encode the relative depth between objects into order maps. Authors introduce a novel order-aware attention, where the order maps seamlessly guide the user interactions (in the form of clicks) to attend to the image features. Authors further present an object-aware attention module
to incorporate a strong object-level understanding to better differentiate objects with similar order. OIS achieves state-of-the-art performance, improving mIoU after one click by 7.61 on the HQSeg44K dataset and 1.32 on the DAVIS dataset

strengths:
1.	OIS can distinguish target objects based on their relative depths from one another and better differentiate objects with similar order.
2.	OIS improves the computational efficiency.
3.	OIS achieves competitive performance

weaknesses:
1. The Concept of Order. Authors should reevaluate the current definition of 'order,' which is often misleading. The term 'order' should pertain to the sequence of click instead of depth map in interactive segmentation. 
2. Insufficient ablation study. More ablation studies are required to clarify the reason of performance improvement, for example the order of object-level attention and order-level attention, and the different pretrained weights of backbone, such as Depth Anything V1.
3. Unfair comparison. OIS adopts Depth Anything V2 as backbone, while Depth Anything V2 requires much more (about 6x) pretraining images than SAM and other backbone. And MM-SAM (Table 5 in Appendix) with Depth Anything V2 achieves similar performance with OIS w/o order and object attention (Table 4). The comparison is unfair. Authors should conduct more experiments to further verify the effectiveness of order map. For example, authors can freeze the backbone of SAM and train an additional depth head to obtain the depth map. Then authors train the object and order attention to verify that the improvement comes from the order map instead of better pretrained backbone.
3. Limited novelty. OIS integrates the object attention from Cutie and depth map into interactive segmentation. However, the novelty is somewhat constrained.

questions:
1. Using the same order map for all positive points may be too naive and in may condition, the depth values of different positive points maybe different.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730521615938

thread_type:
official_review

<thread_object>
creator: reviewer_jpbl

<content>
summary:
This paper presents OIS, which explicitly encodes the relative depth between objects into order maps. It introduces an order-aware attention mechanism that guide the user interactions to attend to the image features by the order maps, and an object-aware attention module for better differentiation of objects with similar order. Experiments demonstrate the effectiveness and efficiency of OIS.

strengths:
1. The OIS integrates relative depths between objects, into interactive segmentation, improving the model’s ability to distinguish target objects.
2. Experimental results show quite good effectiveness.

weaknesses:
1. The difference between order-aware attention and object-aware attention proposed in this paper and masked attention seems to be only in the source of input mask, so both are slightly innovative.
2. In the comparative experiments of this paper, most of the traditional interactive segmentation methods (training on COCO) and SAM-based methods (training on SA-1B) have not been trained on HQSeg44K, while OIS is trained and tested on this dataset. The data quality of HQSeg44K is higher than that of COCO, so the fairness of the experiment is uncertain.

questions:
The main motivation of this paper is to enhance the ability of interactive segmentation through depth information, but the paper seems to only show the cases where the depth of different positions of the same object is not very different, and there is little discussion about the effectiveness of the proposed method when the depth of different positions of the same object is very different (such as the bridge of DepthAnything visualization).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730556208176

thread_type:
official_review

<thread_object>
creator: reviewer_qsav

<content>
summary:
1. This paper proposes order-aware attention, which integrates order (relative depths between objects) into interactive segmentation, improving the model’s ability to distinguish target objects based on their relative depths from one another. 
2. This paper introduces object-aware attention to incorporate a strong understanding of objects.
3. This paper combines both dense and sparse integration of prompts, improving the alignment between the image and prompts while maintaining efficiency. 
4. This work achieves good performance with lower latency.

strengths:
1. The method looks very good for segmenting some difficult foreground objects (tennis rackets, bicycle wheels, etc.).
2. The order-aware attention module introduced in this paper is easy to accept and effective.
3. The framework of this paper is relatively concise and the implementation is easy to understand.

weaknesses:
1. The method section of this paper is relatively easy to understand, but the motivation and problems to be solved are not concise enough. The author seems to want to solve many problems and propose many improvements, which can easily lead to readers not understanding why the author proposes certain techniques.
2. The experiments of this paper is insufficient, lacking results on typical interactive segmentation datasets, such as GrabCut, Berkeley, SBD, PascalVOC, etc. In addition, there is a lack of comparison with some recently published (CVPR 2024, etc.) interactive segmentation methods.
3. I admit the practical value of this paper, but I think it is more suitable to be published in CVPR rather than ICLR.

questions:
1. Can the author connect and refine the motivation and the problem to be solved in this paper?
2. See Weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730579706696

thread_type:
official_review

<thread_object>
creator: reviewer_r4kn

<content>
summary:
The paper proposes order-aware interactive segmentation, which utilizes the extra relative depth information from pre-trained monocular depthanything v2 to generate the order maps. Then the order maps are used to guide the sparse embeddings to attend to the image features via mask attention. Object-aware attention is also used to help boost performance. Prompts are integrated via both the sparse and dense fusion. The paper validates its experiment design on HQSeg44K and DAVIS benchmark.

strengths:
1. The paper has a good writing and structure, where the paper ideas and figures are easy to read and understand.

2. The paper validates its model design choice in Sec 4.5 and Table 4, which shows the performance gain brought by each proposed component clearly.

3. Using relative depth order to guide segmentation is an interesting idea, where the proposed order map considers both the positive and negative clicks.

weaknesses:
1. The paper utilizes additional monocular depth prediction as input. In Table 3, does the computing cost of the depth model also get included? The paper should also study the impact of using various depth prediction models to the segmentation performance.

2. The robustness of the segmentation model to the errors brought by depth prediction network is not studied. When depth prediction makes large errors, how will it influence the segmentation model? Especially, when segmenting neighboring objects with close depth distance, how much can the depth model contribute? 

3. Consider the importance of Table 4, besides DAVIS, an ablation experiment also on HQ-Seg44K should also be performed to have a more comprehensive understanding of each proposed component.

4. The paper has a limited tech novelty, where the order map guiding attention is borrowed from Mask2Former and the object-aware attention is from Cuite. The depth prediction is from a pretrained depthanything v2 model. Using depth to guide more accurate segmentation is also introduced in [a, b]. This makes the paper more like a combination of existing components and model designs.

[a] "Depth-Guided Semi-Supervised Instance Segmentation." arXiv preprint arXiv:2406.17413
[b] Unsupervised Semantic Segmentation Through Depth-Guided Feature Correlation And Sampling. CVPR, 2024.

5. The paper misses an ablation experiment on order maps designs considering both for positive and negative clicks. What if only considering the order maps for positive or negative clicks alone?

questions:
What's the advantage of using depthanything v2's pretrained backbone in model's segmentation accuracy? besides saving parameters, how does it compare to the image encoder of SAM?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692967752

thread_type:
official_review

<thread_object>
creator: reviewer_rqn7

<content>
summary:
The paper presents Order-Aware Interactive Segmentation (OIS), combining order-aware and object-aware attention to improve segmentation accuracy and efficiency. Order maps help distinguish object depths, while foreground-background separation aids object differentiation. Using both dense and sparse prompt fusion, OIS achieves state-of-the-art results on HQSeg44K and DAVIS, boosting accuracy and speed.

strengths:
1. The paper is well-written, with clear motivation.
2. It provides extensive experiments that convincingly demonstrate the proposed method's effectiveness.
3. The visualized analysis adds value by highlighting the necessity of incorporating depth information.
4. The method's performance surpasses current state-of-the-art methods.

weaknesses:
1. The paper exceeds the ICLR 2025 10-page limit with 11 pages in the main text.
2. Overall, the technical contribution and novelty of this paper are incremental, as it mainly incorporates existing priors, such as depth maps and foreground-background masks, to enhance segmentation accuracy. Since these priors have already proven effective in general segmentation tasks, their success in interactive segmentation is unsurprising. I would encourage the authors to clarify the unique benefits these priors bring specifically to interactive segmentation.
3. The rationale for the ordering of Object-level Understanding before Order-level Understanding is unclear. Could the authors explain if this order enhances performance or if alternative orderings were tested?
4. The modules for Object-level and Order-level Understanding are connected sequentially. Why was a parallel structure not considered? Could it improve performance or efficiency?
5. Similar work (see [1]) also enhances foreground-background distinction in interactive segmentation; this should be discussed.
6. The paper lacks ablation studies on Mask Guidance within the object-aware and order-aware attention modules. For example, testing the effects of different Mask Guidance qualities would be beneficial.
7. It is unclear whether OIS can handle interactive segmentation for multiple objects simultaneously. Since Object-level Understanding relies on Mask Guidance from previous interactions, if the target object changes between clicks, OIS may struggle to shift focus to the new target due to constraints from the previous mask. This limitation could reduce the method's practical utility.

[1] Object Aware Contrastive Prior for Interactive Image Segmentation

questions:
1. Would it be possible to show corresponding depth maps in Figure 6 to help readers better understand the importance of order-aware attention?
2. As the number of negative clicks increases, does the model generate a unique order map for each negative prompt? If so, this approach could result in significant memory usage. Has the author considered selectively merging these maps? Since repeated negative clicks are likely to target very localized areas, the corresponding depth maps would likely have high similarity, making unique order maps for each negative prompt potentially redundant.
3. Would lower-quality depth maps directly impact OIS performance, and conversely, would higher-quality maps improve it?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732153933040

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Highlighting Our Technical Novelty

comment:
We thank all the reviewers for their thorough analysis and detailed feedback on our work.

  We are thrilled that they found our work **“interesting idea”** (Reviewer r4kn), **“good for segmenting some difficult foreground objects”** (Reviewer QSAV), having **“no major weaknesses”** (Reviewer DpQL), **“improve computational efficiency”** (Reviewer pFaJ), **“well-written, with clear motivation”** (Reviewer rqn7), having **“quite good effectiveness”** (Reviewer jpbL) and with **“superior performance”** (Reviewer DpQL).

Some reviewers have expressed their concerns regarding our technical novelty, which we carefully address below.

---

### Technical contributions of Order-Aware Interactive Segmentation (OIS):

1. We introduce a novel concept called “order”, in the context of interactive segmentation, which is defined as the **relative depth between objects** in a scene. This concept is important and **more meaningful than interpreting the absolute depth values** for the interactive segmentation task, because, intuitively, our model only needs to leverage the relative **ordering of objects**, i.e., whether some objects are closer or further to a reference object than others. This prevents our model from suffering due to the noise and scale differences of absolute depth values.


   > a. The concept of order allows us to seamlessly integrate positive and negative clicks into the interactive segmentation task through the construction of order maps. **This is different from naively integrating depth**, which results in suboptimal performance, as demonstrated by the MM-SAM results in Table 5 and Fig. 5 of the main paper.



   > b. We design a novel order-aware attention module that utilizes order maps to selectively attend to the image features. Unlike previous approaches [1,2,3], which have only used segmentation masks for masked attention, our method, to the best of our knowledge, is the **first to leverage other modalities (order maps) to guide attention** effectively. Note that, different from Mask2Former [1] and Cutie [2], our order-aware attention mechanism incorporates a “soft” mask guidance i.e., a continuous variant of masked cross attention. It prioritizes focus on closer objects (lower order) while gradually reducing focus on the further ones (higher order). Mask2Former [1] and Cutie [2], on the other hand, employ binary masks to either completely restrict attention to certain image regions, or to fully attend to the unblocked image regions. 

---

2. We introduce the concept of “object” to ensure that our model can **distinguish objects belonging to the same order**, and to ensure that **our model is robust to erroneous order map construction** (order maps depend on the quality of depth information available). 


   >a. The concept of “objects” is incorporated in our object-aware attention module.  This module is indeed similar to Cutie. However, unlike Cutie, we adapt this design for the interactive segmentation task, which is not a straightforward adaptation. Unlike Cutie, which initializes object embeddings randomly, **our method encodes foreground and background clicks as the object embeddings**, seamlessly allowing us to use the object-aware attention module to the interactive segmentation task. The encoded foreground and background clicks as object embeddings introduce a **discriminative notion of the target object** and enable the network to distinguish different objects with similar depth.

---

3. To seamlessly integrate both object and order-aware attention modules together meaningfully to yield the best results, we follow a sequential design of cascading the object-aware and order-aware attention modules one after the other. This design is intuitive and ensures that the model **first develops a clear notion of the target object** and locates it accurately. The order-aware attention module is then used to **refine the predictions with the help of the order maps**. This step is crucial to address challenging cases, i.e., objects with occlusions, thin and intricate structures, etc, as shown in Fig. 4-6, and Fig. 8-13. 

---

4. To further improve our accuracy and speed, we combine two different approaches to integrate prompts: dense and sparse fusion. Prior interactive segmentation methods typically either use dense fusion (which is slow) or use sparse fusion (which is less accurate) and hence suffer from their respective limitations. In contrast, our approach ensures precise alignment between the image and the prompt for **high accuracy** (due to dense fusion) while maintaining **fast computational efficiency** (due to sparse fusion).

---

We have addressed the other concerns expressed by the reviewers one by one in separate comments. The reviewers' feedback has undoubtedly strengthened our paper, and we hope that our efforts have effectively addressed their concerns.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733246153852

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revised Manuscript Updates and Additional Analysis for Reviewer Comments

comment:
Dear Reviewers and Area Chairs,

We sincerely appreciate your insightful comments and valuable suggestions for our paper.  Your expert opinions have been invaluable towards the improvements we have made to our paper.

According to the points raised during the discussion, we have made the following updates in our revised paper:

1. Ablation study of image encoder backbone
    - We replaced our image encoder backbone with an MAE-pretrained ViT backbone, which is consistent with prior SOTA methods. Results in the **new Section A.13** confirm that our method significantly outperforms other methods with the same backbone, demonstrating that our performance improvements are primarily due to our proposed approach instead of a better backbone. This addresses concerns from Reviewers **r4kn** and **pFaJ**.

2. Impact of Depth Map on Model Performance
    - To address concerns raised by Reviewers **rqn7**, **r4kn**, and **jpbL**, we conducted additional ablation studies using various depth prediction models. These studies demonstrate the robustness of our model to variations in the depth map quality.
    - We provide additional qualitative results and discussion on this in **new** **Section A.9**.

3. Ablation study on the sequence of order/object-aware understanding modules
    - We have conducted experiments to validate our current sequence of the order-aware understanding module and the object-aware understanding module. We discuss the results in **Section A.10**, addressing the questions of Reviewers **rqn7** and **pFaJ**.

4. Fair Comparison with methods trained on COCO
    - To ensure fair comparisons, we trained our model solely on the COCO dataset and compared it with methods trained only on COCO. The results in the **new Section A.14** show that our model maintains a significant performance margin over other approaches, addressing the concerns raised by Reviewer **jpbL**.

5. Ablation study for order map with positive or negative clicks alone
    - We analyzed the impact of using order maps exclusively for positive or negative clicks. The results show that removing either leads to a performance drop. This is detailed in the **new Section A.12**, addressing questions from Reviewer **r4kn**.

6. Ablation Study on HQSeg44K dataset
    - To address the concern from Reviewer **r4kn**, we added an ablation study on the HQSeg44K dataset in **new Section A.11**. The results align with ablation study on the DAVIS dataset (in our original Table 4), reconfirming the effectiveness of each proposed module in our model.

7. Figures Revisions
    - We have updated **Fig. 1** to include the user prompts (positive and negative clicks) as recommended by Reviewer **DpQL**.
    - We have included depth maps in **Fig. 6,** as suggested by Reviewer **rqn7,** for better visualization.

8. Related Work Section
    - We have added and discussed the additional reference suggested by Reviewer **rqn7** in the **Detailed Related Work Section A.3.**
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper receives 4 positive and 2 negative ratings after rebuttal, with 3 upgraded scores. Initially, the reviewers had several concerns about technical motivation/contribution, ablation study, handling multiple objects, robustness to depth maps, and experimental fairness. In the post-rebuttal discussion period, three reviewers were satisfactory with the authors' comments and raised the rating. After taking a close look at the paper, rebuttal, and discussions, the AC agrees with reviewers' feedback of the proposed method being novel and effective to perform interactive segmentation. Therefore, the AC recommends the acceptance rating.